CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devices

Authors: weilin  zhao
yuxiang  huang
xu  han
zhiyuan  liu
zhengyan  zhang
kuai  li
chen  chen
tao  yang
maosong  sun
llm + lora
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 16:37:19

Abstract: Recently, there has been a demand to deploy Large Language Models (LLMs) on personal devices such as laptops and smartphones. These LLMs have different model variants when handling different tasks. However, personal devices have limited resources and require reduced storage overhead. To address this, there are two key methods available: the first is model compression, which compresses LLMs into smaller sizes; the second is LoRA, which can transfer an LLM to other tasks with very few parameters, avoiding the storage of multiple model variants in multi-task scenarios by only preserving LoRAs.
However, our experiments show that directly combining these two methods yields sub-optimal performance. Considering that the open-source community has already contributed many LoRAs to LLMs, we propose to adapt these existing LoRAs from the LLMs to their compressed version and introduce a Compression-Aware LoRA (CA-LoRA) framework.
We incorporate knowledge inheritance and recovery strategies to recover the lost knowledge caused by model compression.
Experiment results demonstrate that CA-LoRA outperforms the vanilla LoRA methods applied to a compressed LLM and achieves comparable performance to the non-compressed LLM with existing LoRA modules. The source code of CA-LoRA is available at https://github.com/thunlp/CA-LoRA.

Main paper: 
 introduction: 
 In recent years, large language models (LLMs) with billions of parameters (Brown et al., 2020; Black et al., 2022; Chowdhery et al., 2023) exhibit more powerful and comprehensive abilities, especially in terms of cognition and embodiment (Lewkowycz et al., 2022; Nakano et al., 2021; Driess et al., 2023) . To extend the LLMs to multi-task scenarios (Zhou et al., 2022; Sheng et al., 2023) , parameter-efficient fine-tuning (PEFT) (Houlsby et al., 2019; Hu et al., 2022) has been widely used, where an LLM is frozen as a backbone among different tasks and then tiny tunable PEFT modules are injected to learn task-specific knowledge. Compared with conventional full-parameter fine-tuning (FT), where a single LLM is tuned into multiple task-specific variants, PEFT tunes much fewer parameters and has lower memory overhead in multi-tasking while achieving comparable performance (Ding et al., 2023) . Among various PEFT methods, LoRA (Hu et al., 2022) is one of the most popular.Despite the success of LLMs, deploying LLMs in real-world scenarios is an important issue. Online LLM serving is well studied (OpenAI, 2022; Google, 2023) , and S-LoRA (Sheng et al., 2023) have explored the idea of serving thousands of LoRAs for thousands of tasks around a single LLM on a cloud server. However, how to deploy LLMs on personal devices, such as laptops and smartphones, is still in its early stages and faces many challenges (Ggerganov; team, 2023) , as personal devices have severely limited storage resources compared with cloud servers.To alleviate the storage constraint, model compression, such as quantization (Lin et al., 2024) , pruning (Xia et al., 2024) , and MoEfication (Zhang et al., 2022b) , propose to compress an LLM into a smaller size, which we denote as the compressed LLM (CLM). Q-LoRA (Dettmers et al., 2023) makes a preliminary attempt in the scenario of multitasking on personal devices, which proposes to leverage the advantage of LoRA in the multi-task scenario by training LoRA on the compressed LLM. From our experiment, we found two shortages of this approach. Firstly, the LoRA training from scratch on the CLM is sub-optimal. Secondly, it only supports quantization, which has a limited model compression ratio and introduces storage overhead on personal devices.Considering that the open-source community has already contributed many LoRAs to LLMs (HuggingFace), we propose to adapt those LoRAs from an LLM to its compressed version, we call it "Compression-Aware LoRA" (CA-LoRA). We introduce two mechanisms:(1) LoRA Knowledge Inheritance. A stronger LLM can make learning LoRA modules easier. Meanwhile, the LoRA modules based on the stronger LLM can better grasp task-specific knowledge. Therefore, we propose to adopt the existing LoRA modules learned on the LLM as the initialization to learn the LoRA modules for its compressed version. In this way, the task knowledge of the LLM can be inherited to obtain more effective LoRA modules for the CLM, as shown in Figure 1 .(2) Model Knowledge Recovery. Whether LoRA modules trained on an LLM can work well with its compressed version is still an open problem, especially considering that model compression techniques may introduce knowledge loss and performance degradation. To recover the lost knowledge caused by model compression, we propose to inject a low-rank non-linear recovery module into the CLM to bridge the knowledge gap that arises from the compression process. We point out that compression techniques may weaken multiple capabilities of the LLM while only a small number of the capabilities it loses are required for a specific task. Therefore, the recovery module is also task-specific.In experiments, CA-LoRA on CLM can recover the performance to the level of existing LoRA on LLM. These findings highlight the potential of CA-LoRA as an effective solution for adapting existing multi-task ability from cloud servers to personal devices, thereby advancing the development of personal AI applications. 
related work: 
 This section mainly introduces PEFT and model compression. More details on LLMs can refer to surveys (Qiu et al., 2020; Han et al., 2021; Bommasani et al., 2021; Zhao et al., 2023) .Although LLMs can acquire rich knowledge from massive pre-training data to handle complex tasks in a zero-shot or few-shot manner (Brown et al., 2020; Black et al., 2022) , there is still a need for adapting LLMs, to better stimulate the knowledge within LLMs to serve downstream tasks. For traditional PLMs, fine-tuning all the parameters of PLMs is the mainstream way to adapt them (Church et al., 2021) , yet the parameter inefficiency makes this way costly to adapt LLMs (Ding et al., 2023) .To adapt LLMs to different tasks in a more efficient manner, various PEFT methods (Lester et al., 2021; Houlsby et al., 2019; Hu et al., 2022; Li & Liang, 2021; Ben Zaken et al., 2022) have been proposed. Specifically, LLMs are frozen and some model-independent tunable modules are injected into LLMs to help the adaptation process. PEFT modules are usually tiny so that PEFT methods can significantly reduce the cost of adapting LLMs. Among PEFT methods, LoRA is the most popular.To improve the efficiency of LLM deployment, it is crucial to speed up the computation of LLMs. Model compression is a commonly used solution. Here we introduce task-agnostic model compression (Sanh et al., 2019) rather than task-specific compression (Sun et al., 2019) , because task-specific compression is not memory-friendly in multi-task settings. Task-agnostic compression techniques include Quantization, Pruning, and MoEfication.Quantization methods represent floating-point number LLM into fixed-point one, from 8-bit (Zafrir et al., 2019) , 4-bit (Frantar et al., 2023; Lin et al., 2024) to 1-bit (Bai et al., 2021; Xu et al., 2024b) . To avoid the performance degradation caused by quantization, quantization-aware training (QAT) (Stock et al., 2021) has been proposed to use a small amount of data to adjust the distribution of model parameters for quantization. Different from quantization methods that compress parameter representations, pruning methods directly discard some parameters. Commonly used pruning methods include structured pruning (Fan et al., 2020; Wang et al., 2020; Zhang et al., 2021; Xia et al., 2022; 2024) and unstructured pruning (Han et al., 2015; Chen et al., 2020; Xu et al., 2021) . MoEfication (Zhang et al., 2022b) , inspired by the mixture-of-experts (MoE) transformer (Lepikhin et al., 2021) , aims to divide LLM parameters into multiple partitions, and each time only a few partitions are used to process input data. Typically, to make a compressed LLM behave the same as its original version, distillation objectives are often used to align the LLM and CLM, including aligning both output and intermediate states (Hinton et al., 2015; Sun et al., 2019; Jiao et al., 2020; Liu et al., 2022; Park et al., 2021) . Zhang et al. (2022a) combines the above quantization, pruning, MoEfication and distillation into a mixed compression method, achieving a high compression ratio. More compression methods can refer to surveys (Liang et al., 2021; Xu & McAuley, 2022) .Maintaining multiple task-specific versions of an LLM in the storage is unacceptably resource-intensive (Zhou et al., 2022) . As shown in Figure 1 , n full-model finetunes need to be done in vanilla fine-tuning for n different tasks, posing significant challenges to the storage. In previous fine-tuning-then-compressing methods, this problem still exists.Different LoRA modules can share a unified frozen LLM as their backbone, leading to lower computation and storage overhead in multi-task scenarios. PetS (Zhou et al., 2022) and S-LoRA (Sheng et al., 2023) use LoRA for multi-task serving on cloud servers. QLoRA (Dettmers et al., 2023) applies LoRAs on CLM, which is suitable for personal devices. 
preliminary: 
 For simplicity, we denote an LLM M as Y = f (X; θ M ), where f (•) is the function of the whole transformer architecture, θ M is the parameters of the LLM M, X is the input, and Y is the output.In the FT setting, all parameters of M (i.e., θ M ) are tuned as followsEQUATIONwhere (X t , Y t ) is the data of the downstream task t, L is the loss function of the task t. θ t M is the final task-specific parameters of the LLM M.In the LoRA setting, M is frozen, and additional LoRA modules P are tuned on task-specific data. We denote the parameters of the LoRA modules injected into M as θ P (M) . As shown in Figure 2 , the computation of the transformer architecture is slightly changed due to the injected LoRA modules and becomes Y = f LoRA (X; θ M , θ P (M) ). The tuning process is formalized asEQUATIONwhere θ t P (M) is the final task-specific LoRA modules collaborating with the LLM M.We propose a more efficient LoRA framework CA-LoRA, by adapting existing LoRA on an LLM into the compressed LLM (CLM).As for the compression method, we choose task-agnostic model compression (Sanh et al., 2019) rather than task-specific compression (Sun et al., 2019) because task-specific compression is not memory-friendly in multi-task settings. After compressing the LLM M, making M have fewer parameters or lower-bit representations, we denote the compressed LLM (CLM) and its parameters as C and θ C respectively. Then the computation of the compressed model can be described as Y = f (X; θ C ) without LoRA modules, andY = f LoRA (X; θ C , θ P (C) )with LoRA modules, where θ P (C) indicates the parameters of the LoRA modules injected into the CLM C. CA-LoRA can be formalized asEQUATIONBy combining model compression and LoRA, we can deploy a unified LLM to serve multiple tasks, while only maintaining tiny task-specific LoRA modules for each task. It is not difficult to imagine that adopting task-agnostic compression methods may weaken model performance, which will inevitably affect the search for the optimal parameters θ t P (C) and the effect of the final model f LoRA (X; θ C , θ t P (C) ). Inspired by the fact that model compression preserves those capabilities of LLMs that training a smaller model from scratch cannot master, we suppose that the LoRA modules trained on the non-compressed LLM would contain certain task knowledge that the LoRA modules can hardly learn solely on the CLM. As shown in Figure 2 , to better learn LoRA modules for the CLM, we adopt the method of inheriting the LoRA knowledge from those modules trained on the non-compressed LLM. To recover the lost knowledge caused by the compressing process, in addition to the LoRA modules P, we add some knowledge recovery modules R.Instead of training LoRA modules based on the CLM from scratch, we propose to use existing LoRA modules of the LLM as their initialization, then adapting them to the CLM. The adaption can lead to learning better LoRA modules on the CLM. Intuitively, it is more effective for a teacher to teach students the fundamentals of a discipline and then let students adapt their comprehension based on their circumstances rather than letting students learn from scratch. Formally, for task t, we first use existing LoRA module θ t P (M) trained on the non-compressed LLM M as the initialization of the LoRA module θ P (C) of the CLM C. After that, θ P (C) is further tuned to θ t P (C) on the data of the task t by using Eq. (3).We propose to inject the knowledge recovery modules R into the CLM C to recover the lost of common knowledge during compression. As shown in Figure 2 , we add a bypass next to each linear layer of the transformer architecture to add a small amount of change to the output states of those linear layers. To avoid introducing too many parameters, the recovery modules R adopt a low-rank structure like LoRA. Since LoRA is low rank while the compression of the model is often not of low rank, we have added additional non-linear activation in between to express some non-low-rank structures better. Formally, we denote the recovery module asEQUATION)where D is the down projection matrix, σ(•) is the activation function, and U is the up projection matrix. D and U together form the parameter of the recovery module θ R .To help obtain the optimal parameter θ t R of the recovery module for the task t, we design a distillation objective. Specifically, we select the existing LoRA modules trained with Eq. ( 2) and the LLM as the teacher. The whole distillation loss is given asEQUATION)where X t is the input data of the task t. Instead of first learning the recovery modules and then adding the inherited LoRA modules for further adaptation, we simultaneously conduct knowledge recovery and tune LoRA modules. The overall optimization problem will become θ t P (C) , θ t R = arg min θ P (C) ,θ R L( f LoRA (X t ; θ C , θ P (C) , θ R ), Y t ) + αL DIST (X t ; θ M , θ C , θ P (C) , θ R ) .We select T5-3b (Raffel et al., 2020) as the LLM and select its CLM from Zhang et al. (2022a) , as shown in Table 1 . Details about compressing T5-3b are in Appendix A. We compare the following 4 paradigms: (1) T5-3b + LoRA: LoRA modules are attached to the original T5-3b, and only the parameters of LoRA modules are tunable while the LLM is frozen. ( 2 We set the bottleneck dimension of the LoRA modules and CA-LoRA recovery modules to 32 in all paradigms. The tunable parameters of LoRA and CA-LoRA modules are about 20M and 120M, respectively. More experimental settings are in our appendix.Table 1 : The models used in the experiments. The notation "T5-3b (X)" represents the T5-3b model with the setting "X". "M", "UP", "SP", and "Q" represent the model is compressed with MoEfication, unstructured pruning, structured pruning, and 8-bit quantization, respectively. "Q+UP+M" means "Q", "UP" and "M" are combined together to achieve higher compression ratios. "bf16" indicates the model is represented in bfloat16 floating-point rather than 32-bit floating-point format. Figure 3 shows that:(1) The CLMs cannot perform as well as the original LLM, suggesting that task-agnostic compression leads to losing some task-specific knowledge. The performance of the compressed model may decrease due to the acceleration. If a mechanism makes up the performance gap without affecting the inference speed, applying compression will be more reasonable.(2) CA-LoRA consistently outperforms vanilla LoRA methods. Such results indicate that task capabilities are effectively migrated through LoRA knowledge inheritance and model knowledge recovery mechanisms.(3) Quantization and MoEfication have relatively little degradation on the model performance, and the degradation can be completely restored using CA-LoRA. However, the pruning methods cause more degradation, especially the structured pruning method. Even though, CA-LoRA recovers most of the performance loss caused by pruning methods.To further evaluate CA-LoRA on a higher compression ratio, we use T5-3b (Q+UP+M) in Table 1 , which is a CLM obtained from mixing quantization, MoEfication, and unstructured pruning, and has a close size to T5-base. Table 2 shows the experimental results.Results show that CA-LoRA is compatible and can be easily applied to a highly compressed model that uses multiple compression methods. Meanwhile, CA-LoRA demonstrates better performance on CLMs than training a small model with a close size from scratch. 
the performance on instruction tuning: 
 The above experimental results have proven the strength of CA-LoRA for specific downstream tasks. In this section, we investigate the effectiveness of CA-LoRA when CA-LoRA is applied to a more general scenario, instruction tuning. Datasets We select Alpaca (Taori et al., 2023) as the instruction tuning dataset and use 5-shot MMLU (Hendrycks et al., 2021) as the benchmark. We also use HumanEval (Chen et al., 2021) , BBH (Suzgun et al., 2023) , and Drop (Dua et al., 2019) for evaluation.Baselines and Implementation Details Quantization is the core compression method for instruction tuning. Therefore, we compress LLaMA-13b (Touvron et al., 2023a) into an 8-bit quantization version for experiments, labeled "BMQuant", using the quantization method in Zhang et al. (2022a) . We also add a 4-bit quantization NF4 baseline, which is the quantization method of QLoRA (Dettmers et al., 2023) . Baseline methods are tested without instruction tuning on Alpaca. To evaluate CA-LoRA, we adopt 4 paradigms: QLoRA (Dettmers et al., 2023) and QA-LoRA (Xu et al., 2024a) for the previous compressing-then-tuning methods and BMQuant+CA-LoRA and QLoRA+CA-LoRA for our compression-aware tuning methods. More details of the LLaMA experiments can be found in Appendix B.The Overall Performance Table 3 shows that quantization leads to overall performance degradation. BMQuant+CA-LoRA and QLoRA+CA-LoRA achieve better overall performance than other methods, indicating that the design of CA-LoRA is effective. QLoRA+CA-LoRA obtains better performance compared with QLoRA, showing the CA-LoRA can achieve faster inference speed and better performance with fewer trainable parameters.Datasets We select CodeAlpaca-20k (Chaudhary, 2023) as the instruction tuning dataset and use 0-shot HumanEval (Chen et al., 2021) for evaluation.We compress LLaMa-2-13b (Touvron et al., 2023b) into an NF4 version, which is the quantization method of QLoRA (Dettmers et al., 2023) .We then compare LoRA, QLoRA (Dettmers et al., 2023) and QLoRA+CA-LoRA. The intermediate rank of the LoRA and Recovery modules is set to 64. We use zero-shot evaluation of LLaMa-2-13b and its NF4 version as baselines.The Overall Performance Figure 4 shows that QLoRA+CA-LoRA achieves better overall performance than baselines, indicating that the design of CA-LoRA is also effective in task-specific instruction tuning.To more clearly demonstrate the inner mechanisms of CA-LoRA, we conduct ablation studies using the CLM T5-3b (Q+UP+M), as in (1) LoRA knowledge inheritance is effective. By initializing the tunable parameters with the existing LoRA modules trained on the non-compressed LLM, the performance of combining the final LoRA modules and the compressed LLM has been significantly improved. This indicates that in the optimization space of the compressed LLM, it is difficult to obtain the optimal task-specific parameters of LoRA modules based on random initialization, but more optimal LoRA modules can be achieved more easily using LoRA knowledge inheritance.(2) Simply adding recovery modules brings a certain level of performance improvements in some circumstances. However, it can achieve further performance improvements by adopting our distillation strategy. This suggests combining recovery modules with the knowledge distillation strategy to enhance LoRA modules is necessary.More ablations can be found in Appendix C.Based on four CLMs of T5-3b (as in Figure 3 ), we test the convergence speed on BoolQ dataset. From Figure 5 , we can find that due to our LoRA knowledge inheritance mechanism, CA-LoRA is superior to the vanilla LoRA methods in terms of convergence speed and final results. Adding the recovery module will not affect the convergence speed. Furthermore, when quantization, unstructured pruning, or MoEfication is used, the inheritance mechanism gives a better starting point for tuning LoRA modules. While in the case of structured pruning, even though the initial point of CA-LoRA does not work well on tasks, it is closer to the optimal point in the optimization space and converges faster.Moreover, considering the existing LoRA modules based on the LLM may be trained by community users on their own private data and then uploaded to the Internet. When adapting these LoRA modules to the CLM, there may not be any task-specific data available for the adaptation process. In this case, we can directly use our LoRA inheritance mechanism without any fine-tuning while still achieving good performance on the CLM, when the compression method is quantization or MoEfication. 
conclusion: 
 We propose an effective LoRA framework for adapting existing LoRA on LLM to the compressed LLM (named CA-LoRA) to deploy language models for multi-tasking on personal devices. Considering task-agnostic compression may cause losing task-specific knowledge, we introduce LoRA knowledge inheritance and model knowledge recovery to recover the lost knowledge. By inheriting the prior task knowledge of the LoRA modules learned on the non-compressed LLM, searching for the optimal LoRA modules for the compressed LLM becomes easier. Moreover, by introducing knowledge recovery modules to recover task-specific capabilities lost in the compression phase, collaborating LoRA modules with the compressed LLM can achieve comparable performance to those LoRA modules based on the non-compressed LLM. 
 
 Appendix: Our work focuses on the efficiency and effectiveness, but it requires a small amount of extra training time. In this paper, we only choose LoRA as a representative of PEFT methods. In fact, our framework can be applied to any other PEFT methods. We focus on algorithmic support for running CA-LoRA on personal devices, but not the actual deployment implementation. The compression method adopted in this paper does not change the number of layers of the LLM. However, for compression methods that change the hidden dimensions of the model, how to transfer the knowledge of LoRA modules on the non-compressed LLM remains an open problem for our future work.This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.The typical NLP tasks to evaluate our methods including BoolQ (Clark et al., 2019) , CB (De Marneffe et al., 2019) , RTE (Bentivogli et al., 2009; Wang et al., 2019) , COPA (Roemmele et al., 2011 ), WiC (Pilehvar & Camacho-Collados, 2018) , SST-2 (Socher et al., 2013) , MRPC (Dolan & Brockett, 2005) , QQP (Wang et al., 2018) , MNLI (Williams et al., 2017) , QNLI (Rajpurkar et al., 2016) , and SQuAD (Rajpurkar et al., 2016) .For compressed models, we use the compressed versions of T5-3b released by Zhang et al. (2022a) , including 8-bit quantization, structured pruning, unstructured pruning, and MoEfication. Table 1 shows the models used in our experiments and their ideal inference speedup compared to T5-3b (Zhang et al., 2022a) . Under this experimental setting, all paradigms are implemented with the open-source toolkit OpenDelta (Ding et al., 2023) . We select the best learning rate among {1e -3, 5e -4, 1e -4, 1e -5}. The batch size is among {8, 16, 32, 64, 128, 256}. The weight decay is 1e -2. The distillation coefficient in Eq. ( 6) is default to α = 0.05.For Alpaca-LoRA and QLoRA, we test the released checkpoint. Alpaca-LoRA adds LoRA modules on every Q, K, V, O projection and QLoRA adds LoRA modules to every linear transformation. In order to inherit from Alpaca-LoRA in QLoRA+CA-LoRA setting, the positions where LoRA modules are added are same as Alpaca-LoRA. For BMQuant+CA-LoRA and QLoRA+CA-LoRA, we first apply the compression method to the backbone model, then we apply the LoRA modules and Recovery modules. We inherit from Alpaca-LoRA, then samely distill the modified model with Alpaca-LoRA. The positions of adding LoRA modules are the same as Alpaca-LoRA and Recovery modules are added to all linear transformations. The intermediate rank of additional modules are introduced in Table 3 . All training settings follow QLoRA (Dettmers et al., 2023) .Is the improvement only caused by increased parameters?We carry out the study in Table 5 . Each setting in this study maintains at least the same number of parameters as CA-LoRA. In CA-LoRA, the tunable parameters consist of LoRA modules and recovery modules. We introduce new settings of LoRA+LoRA and Large LoRA. More specifically, in LoRA+LoRA settings, we first inject LoRA modules to W Q and W K in attention layers, like conventional LoRA, and then inject extra LoRA modules to all linear transformations in both attention and feed-forward layers, replacing the recovery modules in CA-LoRA. Both LoRAs have an intermediate rank of 8. We then apply distillation to the modified model. In Large LoRA settings, we only inject LoRA modules with intermediate rank of 32 on W Q and W K . From Table 5 , we find that adding more parameters makes marginal improvements, showing that only adding more tunable parameters cannot bridge the knowledge gap caused by model compression.  


Decision: This paper combines two different efficiency methods, LORA and compression methods. The results are useful, but the experiments and domains are somewhat narrow.

Reviews: review: paper_summary: paper_summary: The authors present CA-LoRA, a compression-aware parameter-efficient finetuning method based on LoRA. They train LoRA modules on an uncompressed LLM and then adapt them downstream to be used with a compressed version of the same LLM, with a relatively small number of additional parameters introduced in addition to the LoRA modules. They explore two model families -- T5 and Llama 2 -- across both SuperGLUE and instruction tuning tasks, and see noticeable improvements in nearly all scenarios they tested.

The experiments are reasonable, the idea is straightforward yet novel, and motivation is sound. The significance of this work lies in their stated motivation: enabling strong, static (or rarely updated) task-specific models to be deployed in resource-constrained environments, such as mobile devices. As far as I am aware, this is a novel method for adapting compressed language models to downstream tasks. They are also clear about limitations: this method requires extra compute to adapt the full sized model before being compressed. Whether this tradeoff is worth it depends on the specific training and deployment scenario.

The structure of the paper is good, but the clarity could be improved. For example, I am not sure why the extra parameters introduced are called “knowledge recovery modules” – based on my reading, they are essentially extra parameters (+ a nonlinear function) that allow the compressed model + LoRA weights extra freedom during finetuning, but they do not explicitly encode any knowledge from the original LLM. Additionally, I believe that many of the equations written are not vital to understanding the work, and replacing them with prose and/or diagrams would improve the overall readability of the paper.
strength_weakness: strength_weakness: The first experiments focus on older models and tasks, which are less relevant for real world scenarios as described in the motivation. SuperGLUE tasks are fairly outdated, and I would prefer to see the instruction tuning experiments expanded and highlighted, because those results are also promising.

The paper could also be written more clearly. The amount of equations is – in my opinion – unnecessary, and the paper would be more straightforward to read if some of these were changed to prose (and/or diagrams) instead.
questions: questions: While the T5 experiments show value in CA-LoRA for task-specific finetuning, I believe the strength of the method lies in the instruction tuning experiments; I think they’re the most interesting and applicable to real world scenarios.

If compute allows, I have a few suggestions for extending the IFT experiments:
* Adding more evaluation domains, to cover other important properties of instruction-tuned LLMs:
  * Safety (XSTest, ToxiGen, etc)
  * Instruction-following ability (AlpacaEval 1 and/or 2, IFEval)
  * Math (GSM8K)
* If your compute budget allows, I would recommend adding another baseline:
  * Llama 2 13B finetuned (with LoRA, most likely), on Alpaca. This would be a fairer baseline vs your method than Llama 2 13B without instruction tuning.
* Try instruction tuning on a domain-specific IFT set, such as CodeAlpaca-20k, and then evaluating on domain-specific evaluations (e.g. for coding, HumanEval+ and MBPP+). 

I don’t think any of these are strictly necessary, but I think they’d help strengthen your argument (especially the evaluations I mentioned in the first bullet point).

Additionally, if you have the space, I’d recommend moving Table 6 from the appendix to section 4.

Overall, I really like your work!
paper_summary: paper_summary: The main motivation is on device personalization where having the big LLM on in device memory is expensive. Hence the authors try to understand ways this requirement can be reduced. They propose using some existing compression techniques on the base LLM and then learning Lora modules on top of it. They propose two types of Lora modules one initialized from a task-specific Lora module and another that learns how to recover the information loss that happens due to compression. However, this part if very confusing to me.
strength_weakness: strength_weakness: I have some issues with the experimental setting and the method that are listed below.
questions: questions: 1. In CLM + LORA are the LORA modules initialized from the trained checkpoint as in CLM + CA-LORA.
2. The QA-Lora method is the strongest on MMLU and authors have omitted the results on it saying they took the results from the original paper. I feel like Table-3 should be completed with the results of QA-Lora on other benchmarks. 
3. How are the recovery Lora modules initialized? 
4. Can you provide training time comparisons CA-LORA with further training an inherited lora on top of a CLM? Because it seems like the cost overhead might be too high as distillation requires forward passes on two models. 
5. Looking at Table 4, it seems like distillation is the least useful component, then it would be good to compare with inheret + recover at all the places in other tables? As it is not very clear how this combination performs at other places and if the additional training cost of distillation is worth it. 
6. It does not conceptually make sense to me why do we need two lora modules in parallel to each other as the final output is just a linear combination of the two lora's anyways and both the lora's are trained on the same joint objective. Then why do we even need to recover lora. Can you just not have a single lora and use the objective in eqn-6 to obtain similar results? I feel like this would work equally well as the CA+LORA method.
paper_summary: paper_summary: This paper proposes a method to adapt a trained LoRA module for an LLM to create a LoRA module for it's compressed version by initializing the latter with the former before training, and learning an additional a task specific low rank recovery module to recover some of the knowledge lost from the base LM in the compression process. The technique is tested both for task specific adaptation and instruction tuning with T5-3B and adapters trained with the proposed method are shown to improve over standard LoRA for compressed versions of T5-3B obtained by quantization, MoEfication, structured pruning and unstructured pruning.
strength_weakness: strength_weakness: The experiments on this paper are restricted to only one model architecture - T5. This should be acknowledged as a limitation and potential direction for future work.
It is also unclear why the paper does not compare to QLoRA.

Comments: comment: Great, thank you for your detailed response! I believe the changes you've mentioned will make the paper stronger, and I'm very interested to see the additional evaluation results in the camera ready version.