ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization
Authors: Prateek Yadav
Leshem Choshen
Colin Raffel
Mohit Bansal
Venue: Accepted by TMLR ()

Abstract: Parameter-efficient fine-tuning (PEFT) enables creation of specialized language models for diverse tasks, resulting in numerous expert modules. In many practical use cases, these expert PEFT modules are integrated into a single model that answers arbitrary queries by routing queries to different experts. However, only a few experts can be kept in GPU memory due to memory constraints. Consequently, expert modules are frequently loaded and offloaded between CPU/GPU memory or disk storage. This frequent swapping dramatically increases communication overhead, leading unacceptable latency and degrading user experience. The large size of modern PEFT modules further exacerbates this latency. For example, QLoRA experts for 65B LLaMA are 3.2GB, making swapping a major communication bottleneck, particularly in memory-constrained environments. To address these issues, we present ComPEFT (compressed PEFT), a novel method for compressing fine-tuning residuals (task vectors) of PEFT models. Reducing expert PEFT module size effectively addresses both memory and communication limitations, facilitating faster swapping and enabling a higher density of experts within a given memory footprint. ComPEFT employs sparsification and ternary quantization to reduce PEFT module size without any additional training while preserving or enhancing model performance. Extensive evaluation across T5, T0, and LLaMA-based models with 200M − 65B parameters, ComPEFT achieves compression ratios of 8x − 50x. Specifically, we show that ComPEFT improves with scale – stronger models exhibit higher compressibility and better performance. We show ComPEFT applied to LLaMA − 65B outperforms QLoRA by 4.16% on MMLU with a 26x storage size reduction. Additionally, compressed experts produced by ComPEFT maintain few-shot compositional generalization capabilities, facilitate efficient communication and computation, and exhibit enhanced performance when merged. Lastly, we provide an analysis of different method components, compare ComPEFT with other PEFT methods, and test its efficacy for compressing full finetuning residual.

Published in Transactions on Machine Learning Research (04/2025)
ComPEFT: Compression for Communicating Parameter Effi- cient Updates via Sparsification and Quantization
Prateek Yadav! Leshem Choshen Colin Raffel 4,5 Mohit Bansal'
UhC-Chapel Hill, MIT, MIT-IBM Watson AI Lab,
University of Toronto, Vector Institue
Correspondence Email: (praty@cs.unc.edu)
Reviewed on OpenRevicw:
Abstract
Paramcter-effiiien fine-tuning (PEFT) enables creation of specialized language models for diverse tasks, resulting in numerous expert modules. In many practical use cases, these expert. PEFT modules are integrated into a single model that answers arbitrary queries by routing queries to different experts. However, only a few experts can be kept in GPU memory due memory constraints. Consequently, export modules are frequently loaded and offloaded between CPU/GPU memory disk storage. This frequent swapping dramatically increases communication overhead, leading unacceptable latency and degrading user experience. The large size of modern PEFT modules further exacerbates this latency. For example, QLoRA experts for 65B LLaMA are 3.2GB, making swapping a major communication bottleneck, particularly in memory-coosttraieed environments. To address these issues, we present ComPEFT (compressed PEFT), a novel method for compressing fine-tuning residuals (task vectors) of PEFT models. Reducing expert PEFT module size effectively addresses both memory and communication limitations, facilitating faster swapping and enabling higher density of experts within given memory footprint. ComPEFT employs sparsification and ternary quantization to reduce PEFT module size without any additional training while preserving or enhancing model performance. Extensive evaluation across T5, TO, and LLaMA-based models with 200M 65B parameters, ComPEFT achieves compression ratios of 8x - 50x. Specifically, we show that ComPEFT improves with scale - stronger models cxhibit higher compressibillty and better performance. We show ComPEFT applied to LLaMA 65B outperforms QLoRA by 4.16% on MMLU with 26x storage size reduction. Additionally, compressed experts produced by ComPEFT maintain few-shot compositional generalization capabilities, facilitate efficient communication and computation, and exhibit enhanced performance when merged. Lastly, we provide an analysis of different method components, compare ComPEFT with other PEFT methods, and test its efficacy for compressing full finetuning residual."
1 Introduction
Parameter-efficient fnnetuning (PEFT) (Houlsby et al., 2019; Karimi Mahabadi et al., 2021) methods like LoRA (Hu et al., 2021) and (IA)² (Liu al., 2022) efficiently adapt language models by learning only few new parameters. QLoRA (Dettmers et al., 2023) further reduces memory needs by using 4-bit quantization for the base model. This combined efficiency has fueled a surge in specialized models for diverse tasks such multimodal understanding (Zhang et al., 2023), multilingual applications (Yang et al., 2023), and expert systems for math (Luo et al., 2023a) or coding (Luo et al., 2023b). Platforms like HuggingFace Hub (Wolf et al., 2019) now host a rapidly growing collection of these expert PEFT models.
"Cdee available http:://ffttbbbbcmmmmmmttttettes

---

Published in Transactions on Machine Learning Research (04/2025)
Quantize
Magnilude Vector
Top-k Magnitude
Constant
ComPEFT
Magnitude
Task Vector
+
+
Compressed Task Vector
+/-
Mask Signs based Magnitude
Sign Vector
Selected Sign
Figure 1: ComPEFT without any additional training compresses PEFT modules while preserving or enhancing model performance.
Serving these expert PEFT models has different strategies. LoRA, for instance, allows merging expert PEFT modules into the base model for single-expert low-latency inference and proposes expert swap- ping for sequential multi-expert serving. While efficient for single and sequential multi-expert serving, these methods become slow for concurrent multi-expert serving because swapping reduces throughput and increases latency (Sheng et al., 2023). Furthermore, LoRA's approaches don't fully utilize available GPU memory for a larger number of experts. For efficient high tigh-throughput concurrent serving, sepa- rating base model and adapter computations is crucial, as multiple experts can then share the basc. This enables efficient base model batching, but di- rectly batching expert PEFT modules remains chal- lenging. Serving numerous experts demands efficient memory management. Limited GPU memory ne- cessitates storing experts off-GPU and dynamically fetching them when needed. This dynamic load- ing of large expert modules leads communication overhead and unacceptable latency, degrading user experience (Sheng et al., 2023). Communication bottlenecks extend beyond concurrent multi-expert serving. Techniques like Model merging (Ilharco et al., 2023; Yadav et al., 2023), Model MoErging (Ya- dav et al., 2024), and compositional generalization (CG) (Huang et al., 2023; Muqeeth al., 2024) also require dynamically retrieving expert PEFT modules from cloud/ disk dlisk/cahhe cache into GPU memory based input queries to dynamically merge route through experts for improved generalization. Consequently, these methods also face communication challenges.
ComPEFT
Storage Savings
26x
Increase MMLU (over base model)
Compression Factor
20x
16x
16x
10
78
138
338
65B
LLaMA Model (# of params)
Figure 2: ComPEFT improves performance with larger base models while compressing models significantly. Left Axis: Improvement of MMLU performance over the corresponding base model. Right Axis: Compression factor achieved by COmPEFT com- pared to the original checkpoint.
For example, a 3.2 GB QLoRA adapter for LLaMA 65B (comparable to a full T5-Large model (Raffel et al., 2020b)) can make frequent. swapping a bottleneck, especially under memory constraints. Therefore reducing expert PEFT module size solves both memory and communication issues by facilitating both faster swapping and increased expert density within a given memory footprint.
To address these issues, introduce ComPEFT (compressed PEFT) method that compresses fine-tuning residuals i.c., task vectors by exploiting their inherent redundancies (Yadav et al., 2023). The task vectors represent the learned changes to the model's parameters during fine-tuning for a specific task. ComPEFT achieves this compression through a two-step process. First, it applies sparsification, aggressively sets a large portion the values within the PEFT task vector to zero. This step based on the observation that many values in task vectors are close zero and contribute minimally to the expert's behavior (Yadav al., 2023). Second, for the remaining non-zero values, ComPEFT employs ternary quantization. Instead of storing these
2

---

Published in Transactions on Machine Learning Research (04/2025)
values at full precision, it represents their magnitudes using single, shared scalar constant, and their signs (+1, -1, or 0). This results in task vectors with sparse ternary weights, drastically reducing their size (see Figure 1). ComPEFT shares similarities with the Sparse Ternary Compression (STC, Sattler et al., 2019a) method used in federated learning, howover, there are notable differences. Unlike STC, CoMPEFT retains high-pprforrmace without the need for additional training. This in stark contrast to directly applying STC to task vectors, which we find leads to performance degradation. Remarkably, ComPEFT can often restore and even surpass the original fine-tuned performance by carefully calibrating the magnitude of the shared scalar constant used in ternary quantization. Additionally, we demonstrate this effectiveness not only for PEFT modules but also for fully fine-tuned models. Moreover, we observe a beneficial trend that the optimal magnitude of this scalar constant becomes consistent across different tasks for larger models (2 13B). This eliminates the need for lask-specific tuning and simplifies the practical deployment of ComPEFT al scale, further facilitating reduced latency during model serving. Finally, the ComPEFT compression enables more efficient operations on task vectors that can facilitate faster merging of models and compute their similarity.
We perform comprehensive experiments for ComPEFT to evaluate: (1) the performance of the compressed model on its original tasks, (2) the nunber of bits needed to store the models, (3) the mergeability and composability of the compressed checkpoints, and (4) how ComPEFT compares to other existing PEFT methods. We performed experiments with T5 (Raffel et al., 2020a), TO (Sanh et al., 2021a). LLaMA (Touvron et al., 2023a), and LLaMA2 (Touvron et al., 2023b) as the base models with model sizes ranging from 200M - 70B parameters. We found that in most cases ComPEFT can provide compression of 8x - 50x (compared to 16 - bit precision checkpoints) while performing similarly or better than the uncompressed models. Additionally, we note surprising finding that the base model gets bigger, their task vectors become more compressible and these compressed checkpoints significantly outperform the original uncompressed chockpoints. Specifically, as shown in Figure 2, ComPEFT leads to improvement of 0.54%, 1.06%, 3.44%, and 4.16% on MMLU for QLoRA trained on LLaMA 7B, 13B, 33B, and 65B parameter models, respectively, while compressing model by 16x 26x. Beyond performance and size, we demonstrate that ComPEFT provides order-of-magnitude reductions in model transmission and loading latency, directly addressing communication bottlenecks in expert model serving. In addition, we show that (1) the compressed models from ComPEFT lead to bet mer-ged models; (2) for few-shot compositional generalization (CG) (Huang et al., 2023), COmPEFT checkpoints lead to similar performance on BBH (Suzgun et al., 2022); (3) ComPEFT applicd to LoRA and (IA) is Pareto-optimal in terms of storage costs vs. performance compared to wide range of existing; (4) the importance of the components of ComPEFT in an ablation study, and (5) the effect of sparsity and scaling on performance. We further show that ComPEFT's benefits extend beyond PEFT, effectively compressing fully fine-tuned models with minimal degradation. Our results and analysis establish ComPEFT as an effective method for compressing task vectors. In summary, our contributions are:
1. ComPEFT demonstrates that even efficient PEFT modules can be drastically compressed (8x- - 50x) via sparsification and quantization without performance loss, suggesting PEFT modules contain significant redundancy.
The reduced size of ComPEFT checkpoints (8x - 50x smaller) enables serving larger models or signifi- cantly more expert PEFT modules given fixed resources.
3. ComPEFT's smaller size reduces communication overhead during dynamic retrieval and load- ing/ofllodiigg, leading to improved latency in practical serving systems.
2 ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization
As discussed in the introduction ($1), serving numerous expert PEFT modules suffer from communicatio and memory bottlenecks which can be alleviated by compressing the experts. This section details ComPEFT, our method to compress experts by targeting their fine-tuning residuals. Given a pre-trained model like LLaMA (Touvron et al., 2023a) or T5 (Raffel et al., 2020b), we can create an expert model for specific task by either finetuning all model parameters or using a parameter-efficicn fine-tuning (PEFT) approach such as (IA)² (Liu et al., 2022) or LoRA (Hu et al., 2021). In both scenarios, we represent the trainable parameters
3

---

Published in Transactions on Machine Learning Research (04/2025)
8, initialized fanit which, upon fine-tuning, become 0zc. This work assumes access to the initial model parameters Oinit and the fine-tuned model parameters, 0r.. Our work focuses on two key objectives: (1) to achieve extreme compression of parameter updates for efficient and low-latency communication of expert models, and (2) to gain insights into the inherent compressibility of these updates, suggesting a lower intrinsic dimensionality of learned lask-specific knowledge.
For a given task 6, we first represent the parameter updates from fine-tuning as task vector T. Or: Iintt- This task vector encapsulates the changes applied to the base model parameters to specialize it for task t. To effectively compress these task vectors, we decompose each тı into direction and magnitude components. This decomposition allows us to treat the direction of parameter updates and their magnitude separately, enabling us to apply distinct compression strategies optimized for each component. We decompose the task vector To into a direction (sign) vector % E Fd and a magnitude vector H4 E R.. Formally, the direction vector % = sT,) captures the sign of each parameter (+1, 0 or -1), while the magnitude vector Mr is 14; = |Te] captures the absolute magnitude. This decomposition allows us to express the task vector as the Hadamard product: Tt = T/ @ Pt- Based on the intuition from Yadav et al. (2023), that the direction of parameter updates is crucial for task adaptation, while lower magnitude updates are redundant. ComPEFT achieves high compression by sparsifying the direction vector % and quantizing the magnitude vector Me to a single scalar.
2.1 Steps in ComPEFT
Algorithm 1 ComPEFT Compression Procedure.
To reconstruct expert model for task only need to communicate the compressed update over the base pre-trained model, which is represented by the task vector тς- As described carlier, we de- compose this task vector into a direction vector % and a magnitude vector (te. Given this decomposi- tion, ComPEFT compresses the task vector through two key steps of sparsification and quantization. Refer to Algorithm and Figure 1.
Input: Task vector Tr, k, and a scaling value a. Output: Compressod task vector ft 't + sgn(Te)
Me f [Tcl
D Step 1: Sparsify.
i. f keep_tople resce_reee_____ rest to zero(Te, Me, k)
D Step 2: Quantize Magnitudes to acalar.
7 = a x o(T) 7t
return R
1. Sparsify: We sparsify the direction vector " by retaining only the signs of the parameters corresponding to the top-k% largest maguitudes in μt,, and setting the signs of the remaining (1 - k)% parameters to zero. Following Yadav et al. (2023), we select the top-k% parameters based on their magnitude in Mt as larger magnitude updates generally represent more significant parameter changes learned during fine-tuning. By preserving the signs of these largest magnitude updates and zeroing out the rest, we aim to retain the most critical directional information for each lask. Formally, the sparsified direction vector, i = To O top-k(no), where top-k(.) is applied elementwise and returns 1 for indices with the top - k% magnitude values and 0 otherwise. The parameter k is referred to as the "dnsityy, and 1 - k as the sparsity.
2. Quantize Magnitudes: We then quantize the magnitude vector Pt to single scalar value. Specifically, we define the compressed task vector f, E Idd as it = * o(T,) * 2. Here, (TT)) E R is the standard deviation of the original task vector Tr, and a E R is a scaling hyper-parameter. We utilize the standard deviation of the original task vector as a scaling factor to normalize the magnitude, which helps to make the optimal a value more consistent across different lasks and models. Refer to Appendix B.5 for more discussion. The scaling factor is then chosen by evaluating performance on small validation set; importantly, is the only paramcter tuned during this process. We observe that this simple scalar scaling is sufficient to effectively miligale any performance loss from sparsification and ternary quantization. This contrasts with many model pruning methods that require computationally expensive retraining after sparsification to recover performance.
2.2 Efficient Storage of ComPEFT Models
ComPEFT's compression strategy directly addresses the communication bottleneck and latency concerns highlighted in $1 by significantly reducing the storage footprint of expert PEFT modules. This section
4

---

Published in Transactions on Machine Learning Research (04/2025)
details the storage efficiency gains and discusses practical encoding schemes for efficient communication and computation.
Entropy of the Sparsified Task Vector. A typical task vector in bfloat16 or fp16 format requires 16 d bits for m Assuming uniform value distribution, its entropy is also Hldenee = 16 d bits. ComPEFT, however, represents the compressed task vector Ft using a sparse ternary sign vector (values E 0,,,0,,+)) and a single 16-bit scalar value (a * (Tr) E R). Assuming the signs of the nonzero entries of fe are uniformly distributed, the ternarization step reduces the entropy of the update to HoComPEST - ((1 k) logg(1 - k) + k log)($)) d + 16 bits, where & is the density of the update. At a density level of k = 0.05, the resultant update has 95% of the values as O and the entropy is 0.34 + d 16 bits. Hence, with a perfect encoding-decoding scheme and 95% sparsity, our ComPEFT can reduce the nunber of bits per parameter from 16 bits to approximately 0.34 bits which is a 47x improvement in communccttio and storage costs. We now discuss two practical encoding schemes to realize these savings.
Optimal Compression via Golomb Coding. For maximal compression in communication and storage, Golomb coding is effective. This near-entropy method suits geometrically distributed data, like distances in sparse task vectors (Strom, 2015; Sattler et al., 2019a). Using Golomb coding (Golomb, 1966), we communicate the locations of non-zero elements with an additional bit indicating each element's 's sign, achieving near-optimal compression. This approach needs a total of ((11 - k) lo - k) + k log)(5)) d 16 bits for storage, and its average bits per position bpos, the calculation of which is detailed in the footnote below". Unless otherwise specified, storage costs reported in our experiments assume Golomb coding.
Efficient Computation and Communication via Two Binary Vectors. Alternatively, for scenarios prioritizing computational efficiency, ComPEFT compressed task vector it can be represented using two binary masks, one signifying positive values and the other siguifying negalive values. Formally, we need to communicate 5 = (f. == +1) E Rd and म. = (f. == -1) E Rd, and the scalar constant a * (TT.). Each binary mask needs 1 bit/parameter, resulting in 2 < + 16 bits for communicating the update. Note that this requires strictly more storage than the Golomb-based encoding described above because -(11 - k)) logg(1 - k) + & log)($)) <. However, sparse ternary vectors allow for efficient matrix operations. For example, to efficiently compute the distance between At and Ftz, we can do an XOR (e) followed by a POPCNT for each group of 64 parameters (i.c. two machine instructions for the iositive and once for the negative masks. The dol product can also be calculated by using bitwise AND operations to calculate positive contributions (both vectors have +1 or -1) and negative contributions (one vector has +1, the other- -1). The final dot product is the difference between the sum of these contributions. Similarly, other operations such as addition can also be made faster, which could reduce the time when merging models. Thus, ComPEFT offers flexibility to use Golomb coding for optimal storage, and binary vectors for efficient computation.
64-bit architecture) twice, once
3
Main Results
3.1 Compressing QLORA Trained on LLaMA Models
Experimental Setup. We first explore the utility of COMPEFT in the setting of training QLoRA adapters (Dettmers et al., 2023) for the LLaMA models (Touvron et al., 2023a) with 7B, 13B, 33B, and 65B parameters. We follow the experimental setting from the QLoRA paper (Dettmers et al., 2023) and experiment with 8 recent instruction-followe This collection includes datasets generated by language models (Alpaca (Taori et al., 2023), self-instruct (Wang et al., 2022), and unnatural-innsrruttonn (Honovich et al., 2022)), a multitask datasct (FLAN-v2 (Chung et al., 2022a)), two datasets created via human annotation and feedback (OASSTI (Köpf et al., 2023) and HH-RLHF (Bai et al., 2022)), and two hybrid datasets (Chip2 (LAION, 2023) and Longform (Köksal et al., 2023)). For each of these datasets, we reuse the checkpoints released with the QLoRA paper to perform compression using Algorithm 1 and then evaluate the 5-shot performance of the compressed QLoRA module
datasels that are diverse in terms of languages and datasel sizes.
SSimilar to Strom (2016); Sattler et al. (2019a;b), the average bits per position 6yon is calculated as follows: 6pox (10((()))))))))))0)00 ba 1-(1-p)20"> with b' =1 I [logo (ox(1-p)) ) and Φ= = v5-1 being the golden ratio.
9https://huggingee hhaggs////uggiggccce..o http://giggg.eceooo/mmeetteeeeee /twde/tteetttmms//rrrrrcccc models=qlora
5

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: Performance improvement from ComPEFT increases as models get bigger. We present the performance (storage size in GB) on the MMLU Test for the original and compressed QLoRA models. For LLaMA 65B, ComPEFT leads to a 4.16% improvement while being 26x smaller.
Model Size (-)
7B
13B
33B
65B
Dataset (1)
Original
ComPEF'
Original
ComPEFT
Original
ComPEFT
Original
ComPEFT
Self-Instruct
36.45
(7.0)
(6.20
45.15(0.61)
50.98%p.01)
(a.c2
65.34
63.43
Longform
(4.3T
(5.48
(5.77
(..8)
54.60(o.01)
57.07 57.07(0.07) (0.07)
(9.49(1.49)
(2.95
Chip2
(1.3)
(6.1)
(4.49
15.06
51. (2
(6.43
57.3 (7.30
(3.32
HH-RLHF
35.5
(..3)
44 (0.47)
(4.99
53. (0.01)
56.97 (6.97(a.07)
58. (1.49)
(..42
Unnatural Instruct
42.14
(1.82
48.98().47)
48. (8.42.0.00)
66.65
58.07 (8.07(0.09)
(9.50p..9)
(3.30
Guanaco (OASSTI)
(5.02
(o.3)
47. (7.
(7.55
60.6 (0.67().49)
63. (..25
Alpaca
(0.7
39.95 (9.96
(o.53
48, (4.00)
53. 63.66
57.68 (a.os)
(0.51
63. 63.28 (u.05)
FLAN v2
(3.)7
o4.70
50.45(0.47)
60.76().03)
56.67(a.a1)
60.01 (0.07)
62.72(1.19)
64.61 (0.11)
Average
37.88(o.3)
(210018))
46.03
47 47.09ko.024
54.15
57. 07.60
69.29(1.49)
63.45 63.45:0.058)
Increase/Comp.
+0.54 16x
+1.06 20x
+3.44 /
+4.16 / 26x
Table 2: LLaMA2-70B Results. Mirroring our main findings, ComPEFT improves average performance
Table 3: ComPEFT can compress smaller model with minimal performance loss. Test set performanco(sroner Size in MB) averaged over seven
(by 1.69%) on LLaMA2 70B, notably by 4.82% on Self-Tnstruct.
GLUE tasks when compressing (IA)² and LoRA mod- ules on different base models.
Dataset (1)
Original
ComPEFT
PEFT (L)
Method (.) T5-Base
T5- Large
TO-3B
Alpaca
67.13
67.56
(+0.43)
Original
21,3).
86 86.2
89.3
Chip2
65.18
67.00 (+1.82)
(IA)*
ComPEFT
80.0g..01)
85.9
88.4 88.4go.0)
Longform
67.63
68.50
(+0.86)
Improvement
-1.3 / 25x
0.3 / 16x
-0.9 / 17x
Guanaco
66.89
67.39
(+0.5)
Original
79.2(6.19)
84.5
89.5/33.70
Self-Instruct
62.36
67.18
(+-4.22)
LoRA
ComPEFT
78.1
84.6
80.5(2.00)
Average 65.84
67.53
(+1.69)
Improvement
/ 17x
+0.1 / 12x
/ 13x
the MMLU benchmark (Hendrycks et al., 2020). To ensure the generalizability of ComPEFT, we extend our evaluation to LLaMA2 70B model. In all experiments, we sweep both a and k in the following ranges, k E (5, 10, 20, 30,30,500 50} and E {0.5, 1, 2, 3, 4, 5, 6, 8, 10] and report the storage size based on the entropy of ComPEFT as specified in $2.2. We find that at any given value of k, you can achieve good performance (see $ 4.2). We used a single 48GB NVIDIA A6000 GPU for these experiments.
Outcomes. In Table 1, we provide results for all the task and model size combinations, comparing the performance of the ComPEFT checkpoints and the original QLoRA checkpoints along with (in subscripts) the storage size in GB assuming 16-bit precision for uncompressed models and Golomb code-based comprossion. We find that on 28 of 32 experimental configurations ComPEFT improves upon the performance of the original QLoRA models while compressing the LoRA module between 10x 50x in terms of storage costs. ComPEPT leads to an improvement of 0.54%, 1.06%, 3.44%, and 4.16% on MMLU for the LLaMA 7B, 13B, 33B, and 65B parameter models, respectively. In Table 2, we observe similar compression and improvements for LLaMA2 - 70B model. To sum, ComPEFT provides better results while also reducing the QLoRA module size. For examplc, on the 65B LLaMA basc model it reduces the storage size from 1.5GB to 110MB while improving the MMLU performance by large margin of 4.16%.
Discussion. A few important conclusions about ComPEFT can be derived from these results: (1) ComPEFT can compress all QLoRA models by factor of at least 10x. (2) Larger basc models allow for more compressible LoRA modules. We get a compression factor of approximately 16x, 20x, 16x, and 26x for 7B, 13B, 338, and 65B parameter models respectively.
(3) A similar trend is found in performance the performance gap between the original and the compressed LoRA module increases with model size from 0.54% for the 7B model to 4.16% for the 65B model. If this scaling law continues, it means that the utility of methods like ComPEFT will increase models become larger and/or their zero-shot, performance improves.
6

---

Published in Transactions on Machine Learning Research (04/2025)
Table 4: ComPEFT extends compression benefits to fully fine-tuned residuals. Average lest set performance (and storage size in GB) over 7 GLUE tasks for original and ComPEFT-compressed fully fine- tuned model's task vectors. Across various model architectures and sizes, ComPEFT compresses models by 12x - 19x with near-lossless performance, and even slight improvements for some models.
Table 5: ComPEFT enables order-off reduction in model transmission and loading latency. We report the wall clock time (mean and standard deviation) for two practical scenarios: down- loading LLaMA model checkpoints (7B-65B) from a simulated internet server a local machine, and transferring chockpoints from CPU to GPU memory, comparing original and ComPEFT-compressen
versions. ComPEFT reduces download times by up to 32x and CPU-to-GPU loading times by up to 25x, highlighting the practical advantages in deployment and serving efficiency.
Model (L)
Original
ComPEFT
Improvement
BERT Base
(7.2)
86.8(0.01)
-0.4/ / 19x
BERT- Large
86.3(0.64)
86. (0.00)
-0.2 18x
RoBERTa Base
(5.5
83.3
-2.2 / 18x
RoBERTa - Large
88.6)0.8)
89.2 89.2(0.052)
+0.6 / 13x
Model (1)
Internet -) Local (seconds)
CPD - GPU (millliseconds)
T5v.1 1 Base
(4.4
75.8 75.8(0.032:
+1.7 / 15x
Original
ComPEFT
Original
ComPEFT
T5v.1 1 Large
84.0(1.5)
11
-1.8 14x
MLaMA
11.21
11 11.23,
LLaMA 13B
16,85
1.75
186 80, 42
23 00ure
T5- Base
82.8(o.43)
78. (0.032))
-4.7
13x
LLaKA 328
32.31,m
2.88
307 222
18.00.
T5- Large
(5.2
(4.7(0.2)
-0.5 12x
65B
83.17
2.59
475.26 a1
18.60ur
3.2
Compressing Other PEFT Updates
The finding that scaling the base model makes the PEFT modules more compressible and more performant brings up the question to whether ComPEFT is still effective at smaller scales. We perform experiments two widely used PEFT methods, (TA)² (Liu et al., 2022) and LoRA (Hu et al., 2021), with three models, T5-Basc and T5-Large (Raffel et al., 2020a), and TO-3B (Sanh et al., 2021b). Specifically, we compress (IA) and LoRA modules trained on 7 classification lasks from the GLUE benchmark (Wang et al., 2018a) belonging to three categories: Natural Language Inference (MNLI (Williams et al., 2018), RTE (Bentivogli et al., 2009), QNLI (Rajpurkar et al., 2016), WNLI (Levesque et al., 2012a)), Sentiment Analysis (SST2 (Socher et al., 2013)), and Paraphrase Detection (MRPC (Dolan & Brockett, 2005), QQP (Wang et al., 2018a)). For hyperparameter selection (a, k), we use the same grid and validation procedure as described in $3.1.
Outcomes. In Table 3, we present the average performance the 7 aforementioned GLUE tasks (per-dataset results are provided in Appendix C.6) along with the average checkpoint size in MB (in subscripts) for three base models with both (TA)3 and LoRA adapters. We find that even with smaller base models, COmPEFT compress the PEFT modules by a factor of 12x 25x with minimal to no loss in performance. These results demonstrate that even at smaller scales ComPEFT can lead to substantial compression. Additionally, we performed some experiments with BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019a), and T5v1.1 (Raffel et al., 2020a) models that are not multtaask-trained and/or have weak zero-shot performance (i.e. they generally require additional finetuning to perform well on any downstream tasks). We present the results for these models in Appendix C.7, where we observe that compression works well for LoRA with minimal performance loss. However, for (TA)² we observe significant performance drops which suggest, that zero-shot performance may be important to enable ComPEFT compression of - models.
3.3 Compressing Fully Fine-tuned Models
Experimental Setup. To assess the broader applicability of ComPEFT, we investigate its effectiveness beyond PEFT modules and explore its ability to compress task voctors produced by full-model fino-tuning. We adopt the experimental setting from $ 3.2 to fine-tune the 7 classification tasks from the GLUE benchmark using full fine-tuning, and then compress the resulting fully fine-tuned task vectors using COmPEFT. We evaluate our method on four different model architectures - BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019a), T5-v1.1 (Raffel et al., 2020a), and T5 (Raffel et al., 2020a) across two model sizes (Base and Large) for each architecture.
Outcomes. Table 4 presents the average test set performance over the 7 GLUE tasks for both original and ComPEFT-compressed fully fine-tuned modelsWe observe that COMPEFT effectively compresses fully fine- tuned models, achieving 12x 19x compression ratios with minimal performance degradation. Notably, for
7

---

Published in Transactions on Machine Learning Research (04/2025)
T5v1.1-base and RoBERTa-large models, COMPEFT even leads to performance improvements of 1.7% and 0.6% respectively, while simultaneously reducing model size by 15x and 13x.
Discussion. These results demonstrate that ComPEFT is not limited to compressing parameter-eficient modules but can also be effectively applied to compress fully fine-tuned models. This broader applicability expands the potential use cases of ComPEFT and highlights its versatility as a general model compression technique. The observed performance improvements in some cases, even with full fine-tuning compression, further suggest that ComPEFT may act regularizer, potentially improving generalization.
3.4 Reduced Transmission Cost and Loading Latency
Experimental Setup. A key practical advantage of model compression is reduced storage and transmission costs. To quantify these benefits for ComPEFT, measure the wall clock time for two representative scenarios: (1) downloading model checkpoint from simulated internet server to local machine, and (2) loading a model checkpoint from CPU memory to GPU memory. We perform these measurements for both original QLoRA checkpoints and their ComPEFT-compressed counterparts for LLaMA models of sizes 7B, 13B, 33B, and 65BFor cach scenario and model configuration, we repeat the mcasurement 10 times and report the mcan and standard deviation of the wall clock times.
Outcomes. Table presents the results for transmission latency and loading timeAs expected, ComPEFT- compressed checkpoints exhibit significantly reduced transmission and loading times compared to the original checkpoints across all model sizes and both scenariosFor example, downloading the LLaMA - 65B ComPEFT checkpoint from the internet is approximately 32x faster than downloading the original checkpoint. Similarly, loading the ComPEFT-compressed LLaMA - 65B checkpoint from CPU to GPU is about 25x faster.
Discussion. These measurements showcase ComPEFT's substantial practical value beyond their size. The order- times significantly accelerate model deployment
of-magnitude speedups download and and improve multi-expert serving efficiency, especially in dynamic or resource-limited settingsComPEFT thus offers key real-world advantages for efficient model utilization.
3.5 ComPEFT vs. Other PEFT Methods
Experimental Setup. Next, we compare the (IA)² and LoRA checkpoints compressed by ComPEFT with various other PEFT methods to de- termine whether ComPEFT produces Paress Pareto-optimal parameter-efffiicnt finc-tuning in terms of Stor-
80
All paramoters
[IA]*
75
LoRA
Average Accuracy
Com(lA)a
70
ComLoRA
BitFit
age Size and Performance. For this, we use the T0-3B (Sanh al., 2021b) model and train a wide range of PEFT methods on the 11 held-out datasets from Sanh et al. (2021b)
65-
Layer Norm
Compacter
60
Compacter
Prompt Tuning
sen- tence completion (COPA (Roemmele et al., 2011), H-SWAG (Zellers et al., 2019), and Story Cloze (Sharma et al., 2018) datasets), natural language inference (three splits of ANLI (Nie et al., 2019), CB (Marneffe et al., 2019), and RTE (Dagan et al., 2005)), coreference resolution (WSC (Levesque et al., 2012b) and Winogrande (Sakaguchi et al., 2020)), and word sense disambiguation (WiC (Pile- hvar & Collad, 2019)). For cach task, from the training set, we select 200 example for the validation set and then use the first template from Prompt Source (Bach et al., 2022) both dur- ing training and evaluation. We perform experiments with 10 different PEFT methods from Liu et al. (2022) LoRA (Hu et al., 2021), (IA)3 (Liu et al., 2022), BitFiit (Zaken et. al., 2021), LayerNorm, Adapters (Houlsby
55
Prefix Tuning
Adapter
20
10a
10
Intrinsic SAID
Avg. Storage Size over 10 Tasks in KB
Figure 3: ComPEFT are Pareto-optimal. Perfor- mance vs storage size for multiple PEFT methods aver- aged over 11 tasks. A PEFT method is Pareto-optimal attains better performance (higher on the y-axis) than all methods that use less storage space (to the left. on the x-axis). In particular, Com( (IA) performance is comparable to PEFT methods that require x more storage space.
8

---

Published in Transactions on Machine Learning Research (04/2025)
Table 6: ComPEFT compressed checkpoints lead to better merged models. Average test set results on 7 GLUE tasks when employing different merging methods on the uncompressed checkpoints and compressed ComPEFT checkpoints.
BBH Pars
Method (1)
Base
Largarge
TO-3B
(IA)
LoRA
(IA)*
LoRA
(IA)*
LoRA
Method
Averaging
53.7
49.3
55.4
50.2
74.5
73.1
Figure 4: ComPEFT facilitates composi- tional gencralization. Average perfor- mance LoraHub and ComPEFT for com- positional generalization on BBH.
Task Arithmetic (TA)
60.0
52.8
62.7
61.6
77.8
75.0
ComPEFT +
59.7
53.9
61.9
64.9
75.6
MerS-Mggging
55.5
49.2
61.3
57.3
71.7
73.4
CEMPEFT +
55.6
60.4
61.4
et al., 2019), Compacter and Compactor+++ (Karimi Mahabadi et al., 2021), Prompt Tuning (Lester et. al., 2021), Prefix Tuning (Li & Liang, 2021), and Intrinsic SAID (Aghajanyan al., 2020).
Outcomes, In Figure 3, we plot the average performance over the tasks and the checkpoint sizes in KB for 10 PEFT Method and when using ComPEFT LoRA and (TA)3 checkpoints, i.e. ComLoRA and Com IA) We find that ComPEFT reduces the storage size for both LoRA and (IA)² by more than order of magnitude with minimal reduction in performance. From this plot, ComPEFT is Pareto-optimal, i.c. for any given storage budget, ComPEFT performms all other PEFT methods. Notably, Com(IA)² (IA) exhibits only a minor performance degradation compared to full-model fine-tuning while being one of the most space-efficient PEFT methods. Lastly, we note that for ComPEFT you can trade-off performance for storage cost by varying the density to obtain models of different sizes. Hence, COm(IA) and ComLoRA could be made even more space efficient.
Discussion. Figure positions ComPEFT highly competitive PEFT technique, not just compression meihodWhile methods like Prompt Tuning and Prefix Tuning offer parameter elficiency, ComPEFT, especially Com(IA) achieves significantly better balance of performance and size. Traditional full fine-tuning, while offering slightly higher peak performance, is orders of magnitude larger in sizeThis Pareto-optimality highlights the practical advantages ComPEFT in resource-constrained scenarios. Furthermore, while we focus on Pareto optimality against other PEFT methods here, ComPEFT also provides a strong compression baseline for any PEFT technique; applying ComPEFT other PEFT outputs could further enhance their storage efficiency,
3.6 Cross- Task Generalization via Dynamic LoRA Module Composition
Experimental Setup. As highlighted in $1, key motivation is to enable efficient serving of numerous experts, particularly in scenarios requiring dynamic adaptation to novel tasks. Cross-tasi generalization, using compositional methods like LoraHub which dynamically select, load, and compose expert modules, exemplifies such & scenario where communication bottlenecks are critical. Therefore, to assess ComPEFT's ability to facilitate efficient expert module serving in this demanding downstream application, we examine its impact on the composability of the resulting PEFT modules for cross-tasi task generalization. Given a set of expert models and an unscen downstream task with few training examples, the goal is combine a subset of these expert modules to allain a model that performs well on the unseen lask.
For this, we follow the LoraHuu (Huang et al., 2023) method and their experimental setting, We use the Flan-T5-large (Chung et al., 2022b) model as it exhibits strong zero-shot and few-shot capabilities. We consider nearly 200 distinct (tasks, instruction) pairs that were utilized to train the Flan-T5 model and use the LoRA modules trained on these tasks as expert Following Huang et al. (2023), when learning a new unseen task, we randomly select N LoRA modules denoted by {L, = (Ai, B.) and compose them as
Im Am Bm
Σ w/ A
Σ
we Bi
(1)
9

---

Published in Transactions on Machine Learning Research (04/2025)
Pruned Compoot
STC
Pruned STC CoMPEPT
Princd ComPERT
STC
Pruned CoMPEPT
STC
Praned COMPEPT
STC
56-
38
55.5
63
80
Avg. Perl.
55
36
62
54.5
34
66
54
53.5
60
50
50
20
Donsty sponsot
Density 1% non sparso)
Dandhy (96 sparse)
Denoty (88 non sparsa)
Dansty (16 non sporso)
(a) TO-3B
(b) LLaMA-7B
(c) LLaMA-13B
(d) LLaMA-33B
(e) LLaMA-65B
Figure 5: COmPEFT outperforms STC and all method steps are crucial. Average performance as
density k of the compressed checkpoint increases. We show results for compressing LoRA modules trained
over different models with sizes ranging from 3B - 65B and compare them with baselines and ablate method components.
where Am Bm are the matrics of the composed LoRA module and w, are parameters that are learned on the few-shot examples from the unseen tasks using the gradient-free Shiwa optimizer (Liu al., 2020). Following LoraHub, we use N 20 and treat the diverse tasks from the Big- Hard (BBH) benchmark (Suzgun et al., 2022) as unseen evaluation tasks. All the tasks multiple-choice questions and employ Exact Match (EM) as our evaluation metric. Error bars in Figure 4 represent standard deviation across the BBH tasks performance.
Outcomes. In Figure 4, we report the average performance and standard deviation (over 5 seeds) when using the LoraHub method on the original checkpoints and the ComPEFT-compressel checkpoints. We find that the compero-compressod checkpoints exhibit similar compositional abilitics as the original uncompressod checkpoints. This is a crucial finding: even with extreme compression, the modules retain the necessary properties for effective cross-tass tass- composition. Hence, ComPEFT checkpoints can be communicated quickly over high latency networks for dynamic module swapping, while maintaining their compositional abilities.
The preservation of compositional generalization performance after ComPEFT compression is significant result. It directly addresses the practical challenge of serving numerous expert modules for complex tasksBy drastically reducing module size, ComPEFT makes dynamic module composition via methods like LoraHub far more efficient and scalable, enabling faster download times and reduced memory footprint during run-time module swappingThis experiment validates that ComPEFT is not only compression technique but. also facilitator for advanced applications requiring efficient expert module management and communication.
Discussion.
3.7 Merging Compressed PEFT Modules
Experimental Sctup. Next, we examine the effectiveness of ComPEFT when merging models (Choshen et al., 2022; Matena & Raffel, 2021) by comparing the merging of compressed uncompressed models. We follow the experimental setting (including base models, PEFT methods, and datasets) from the previous section and merge the 7 GLUE tasks to produce a multitask model. We then report the average performance of the merged across all tasks. We use two methods to merge task vectors, namely, Task Arithmetic (Ilharco et al., 2023) and Terg-g (Yaday et al., 2023). We used the code from the original authors for both merging mothods.
Outcomes. As demonstrated in Table 6, in 9 out of 12 scenarios, the ComPEFT checkpoints lead to better merged models compared to the original checkpoints, with the notable exception of (LA)3 on T5 models. Notably, in stronger models like TO-3B, ComPEFT-compresse checkpoints not only reduce the size by approximately 15x but also improve the merged model's performance by 2.4% on average. One possible explanation is that ComPEFT acts as a regularizer, removing less important parameter updates and potentially leading to smoother, more generalizable loss landscape that merge more effectively. This shows ComPEFT's efficacy in both minimizing storage and communication overheads and improving the model merging performance.
10

---

Published in Transactions on Machine Learning Research (04/2025)
k=10
k=20
k=10 k:50
k-10
k=20
ko20
k:550
k::30
k::30
k:55
k2230
10:30
90
45
Aug. -0
60
Avr, Pert
80
Avg. Pert.
Aug. Perf.
Avg. Perf.
70-
30-
40
40
30
25
25
30
Alpha tel
Apha 1o)
Alpha (3)
Alpha (0)
Alphe (c)
(a) TO-3B
(b) LLaMA-7B
(c)
LLaMA-13B
(d) LLaMA-33B
(e)
LLaMA-65B
Figure 6: Larger models do not require explicit tuning of Performance vs for various denisty
levels for ComPEFT.
4 Additional Results and Analysis
4.1 Ablation of ComPEFT Components
Experimental Setup. To understand the contribution of the individual steps of ComPEFT, we now perform a brief ablation study. In ComPEFT there are two main steps: (1) Sparsifying the direction vectors, and (2) Quantizing the magnitudes to a scalar with scaling factor a. Hence, we compare with two ablated versions: Pruned (only sparsification, magnitudes reset to zero, no quantization, no scaling), and Sparse Ternary Compression (STC) (Sattler et al., 2019a) (ternary quantization with mcan magnitude scaling, no tuned a). We also include the uncompressed original model as baseline. We provide these ablations for the experimental settings from $ 3.1 and 3.5 where the model sizes range from 3B 65B.
Outcomes. In Figure 5, we plot the average validation set performance over tasks as function of the density (k) of the compressed model. From these results, we make a few observations: (1) ComPEFT almost always performs better than both STC and the Pruned version for all model sizes and sparsity levels. (2) ComPEFT almost always performs better than or similar to the original model's performance for all sparsity levels. In contrast, for smaller model sizes of 3B and 7B, STC's performance is much worse than the original models. This highlights the importance of the scaling a as proposed in ComPEFT, which allows to recover the performance lost due to pruning and ternary compression without computationally expensive retraining. (3) At low density, the performance of Pruned is much worse than ComPEFT and this gap reduces the density increases. However, note that the size of ComPEFT is much smaller than the Pruned baseline due to ternarization. (4) At larger base model sizes (2 13B), all the methods at all density levels perform similarly to or better than the original LoRA checkpoint, suggesting increased robustness to compression choices at scale.
The ablation study clearly demonstrates the contribution of each component. The superior performance of ComPEFT over the Pruned variant underscores the importance of ternary quantization and scalar scaling in maintaining performance after sparsification. The advantage over STC highlights the benefit of tuning the scaling factor a rather than using a fixed magnitude scaling like mean magnitude. These results validate our design choices and show the utility of sparsification, ternarization, and tuned scalar scaling.
Discussion.
4.2 Effect of Sparsity and Scaling on ComPEFT
Experimental Setup. For COmPEFT, we analyze the effect of different levels of sparsity and the scaling value a on the performance of the compressed checkpoints. We present this analysis for TO - 3B and LLaMA as the base models; the experimental settings are similar to $ 3.1 and 3.5 where the model sizes range from 3B 65B. We provide results for different values of the density k (sparsity 100 - k), specifically, the values k (5, 10, 20, 30, 50} and different values of a {0.5, 1, 2, 3, 4, 5, 6, 8, 10).
Outcomes. In Figure 6, we plot the average validation set performance across all tasks with respect to the scaling coelficient a. We make the following observations; (1) For smaller base-model sizes (3B and 7B) and across density values, we find a similar trend - as the value a increases, the average validation performance first increases and then drops. (2) As the value of increases, the optimal value of tends to becomes smaller. For example, for the TO-3B base model, the optimal value a for lt 50 is between 2 while for k 5 the optimal a is in the range 5 8. (3) For bigger base-models (2 13B) and low density (k E {5, 10, 20)) the
11

---

Published in Transactions on Machine Learning Research (04/2025)
variation in performance as changes is smaller. (4) Lastly, the base-model size increases, smaller values of a E (0.5, 2) and bigger range of values start to work better. Hence, for large models, the need for tuning a can be removed. For models with 2 13B parameters and high sparsity k S 20, we recommend setting a = 1.
Discussion. This analysis highlights the interplay between sparsity and scaling in COmPEFT. For smaller models, fine-tuning a important to maximize performance at a given sparsity level. However, for larger models, the method becomes more robust, and a fixed scaling factor (like a 1) can be sufficient, especially at higher sparsity. This robustness for larger models simplifies the application of ComPEFT in practice, it reduces the need for extensive hyperparameter search and makes the method more readily deployable for large language models where computational efficiency is paramount.
5 Related Work
Paremeter Efficiont Fine-Tumie Tuning. Several parameter-cfficint tochniques (Lester et al., 2021; Li & Liang, 2021; Houlsby et al., 2019; Zaken et al., 2021) have emerged as efficient alternatives to full fine-tuning in the field of pre-trained language models (PLMs). These methods introduce a small number of additional parameters to PLMs (Raffel et al., 2020a; Touvron et al., 2023a) enabling efficient fine-tuning, LoRA (Hu et al., 2021) incorporates trainable low-rank matrices into transformer layers. In Contrast, (IA)² (Liu et al., 2022) learns a new set of parameters to rescale the model activations. Recently, QLoRA (Dettmers et al., 2023) proposed training LoRA modules over a 4-bit quantized base model to further save the memory.
Network Pruning and Federated Learning. Neural network pruning techniques have garnered attention for reducing computational costs (Cheng et al.; 2017; Liang et al., 2021) by removing redundant parameters while preserving performance (Zhu & Gupta, 2018; Liu et al., 2019b; Frankle & Carbin, 2019; Gale al., 2019; Xia et al., 2022). Among these, magnitude-based pruning (Han al., 2015; Li et al., 2018; Lee et al., 2021) selects parameters based on magnitudes. Pruming is valuable in federated learning due to high communication costs over slow networks. Atomo (Wang et al., 2018b) minimizes gradient variance through unbiased sparsification, while QSGD (Alistarh et al., 2017) offers a communicalion-converge by quantizing gradients. SignSGD (Bernstein et al., 2018) further converts gradients to binary sign vectors. TernGrad (Wen et al., 2017) and STC (Sattler et al., 2019a) combine sparsification and quantization.
trade-off
Model Merging and Compositional Generalization. Various merging methods (Ortiz-Jinénez el al., 2023; Wortsman et al., 2022b;a; Ihharco et al., 2022; Ramé et al., 2022; Yu et al., 2023) aim to combine fine-tuned models for improved performance in various applications. Choshen et al. (2022) performs direct averaging of the model weights while Task Arithmetic (Ilharco et al., 2023) generates task vectors and performs arithmetic operations to create multitask checkpoints. Ortiz-Jiménez et al. (2023) offer theoretical insights into model merging by using the weight disentanglement property. TIES-Merging (Yadav et al., 2023) idintifies the issue of parameter interference in model merging and tackles it by trimminn low-magnitude parameters, resolving sign disagreements, and disjointly merging parameters with consistent signs. Ponti et al. (2023) performed CG by jointly learning adapters and routing function to allocate skills to tasks, while Caccia et al. (2023) analyzes task routing for more efficient cross-task generalization. LoraHub (Huang et al., 2023) employs gradient-free optimization to retrieve and merge expert modules for unseen tasks while Mugeeth et al. (2024) focus on zero shot compositional generalization. Pfeiffer et al. (2023) provides an overview of PEFT methods, model merging, and compositional genoralization methods.
6 Conclusion
Our PEFT compression method, ComPEFT, offers an effective solution to the latency challenges associated with retrieving expert models. By compressing finc-tuning residuals through sparsification and quantization, ComPEFT achieves high compression ratios and often enhances model performance across various NLP tasks and model sizes. Moreover, it preserves few-shot compositional generalization capabilities, facilitates efficient communication and computation, and demonstrates improved performance when merged with original models. This research contributes valuable insights into the realm of parameter-effiient fine-tuning, addressing both performance and latency concerns.
12

---

Published in Transactions on Machine Learning Research (04/2025)
References
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, arv preprint or Xrvi:2202.. 2.2.13236 13255, 2020.
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Commmicatton--ffiicntt sgd via gradient quantization and encoding. Advances in neural information processing systems, 30, 2017.
Stephen H Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht. Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. Promptsource: An integrated development environment and repository for natural language prompts. arXiv preprind arXiv:2202. (Xiu:2202.01299, 2022.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human foedback. arXiv preprint 3rXiv22204008800 2022.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7:8, 2009.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadensshcli, and Animashrec Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp. 360-569. PMLR, 2018.
Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, and Alessandro Sordoni. Multi- head adapter routing for cross-task generalization. In Thirly- Conference Newral Information Processing Systems, 2023.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. CoRR, abs/1710.092822, 2017.
Leshem Choshen, Flad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining, 2022. https://arxxi..rrgabble 04444400//////200044444..
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction finettundd language models. arXiv preprint 221::2210. 1 11416, 2022a.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-inntuuee language models. arXic preprint ar Xiv:22200. 11416, 2022b.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Ma- chine Learning Challenges Workshop, 2005. https://link: springer. com/ /ttp:ppee///...000////70000
9.
Wenlong Deng, Yize Zhao, Vala. Vakilian, Minghui Chen, Xiaoxiao Li, and Christos Thrampoulidis. Dare the extreme: Revisiting delta-parameter pruning for fine-tuned models. arXiv preprint arXiv:2410.09344 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1mss. aix preprint arXiv:2305. 14314, 2023.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04 04805, 2018.
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). 2005.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference Learning Representations. OpenReview.nee 2019.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. CoRR, 095574992.09574 2019.
Solomon Golomb. Run-length encodings (corresp.). IEEE transactions on information theory, 12(3) ((83)399-440) 401, 1966.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient. neural network. Advances in neural information processing systems, 28, 2015.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXic preprint arXiv:2009.03: 2020.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. orXiv preprint arXiv:2212 09689, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameler-effiiint transfer learning for ulp. In International Conference on Machine Learning, pp. 2790 2799. PMLR, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. ArXiv, abs/2106.09685,, 2021.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task gencralization via dynamic lora composition. arXiv preprint orXie:2307.3320, 2023.
Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. Hira: Parameter-cffiiient hadamard high-rank adaptation for large language models. In The Thirteenth International Conference on Learning Representations, 2025.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannanch Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. In Advances in Neural Information Processing Systems (NeurIPS), 2022. https://arre httpaaaaii....////a/aaaaaa//eeee a22208. 05592.
Gabricl Ilharco, Marco Tulio Ribciro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. The Eleventh International Conference on Learning Representations, 2023. URL
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercom- plex adapter layers. Advances in Neural Information Processing Systems, 34:1022-1035, 2021.
Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schütze. Longform: Optimizing instruction tuning for long text generation with corpus extraction. arXiv preprint arXiv:2304.08400 2023.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations democratizing large language model alignment. arXic preprint arXiv:2304.07327, 2023.
Dawid Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vestor-based random matrix adaptation. arXiv preprint arxw:2310011444, 2023.
LAION.
dataset.
fttps:://github.com///t/ AI/
Open-Instruce www.-nsttncctinnnneeellleee 2023.
Jacho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. In 9th International Conference on Learning Representations. OpenRevvew.net, 2021.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-cfficien prompt tuning. Xrxiv preprint Xrx::22004 00..9991,, 2021.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge represenlation and reasoning, 2012a.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. Thirteenth International Conference on the Principles of Knowledge Representation and Reasoming, 2012b.
Guiying Li, Chao Qian, Chunhui Jiang, Xiaofen Lu, and Ke Tang. Optimization based layer-wise magnitude- based pruning for DNN compression. In Proceedings of the Juenty-Sevente International Joint Conference on Artificial Intelligence, pp. 2383 2389. ijeai.org, 2018.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370- 1037770433--0000 2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit. Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950 1965, 2022.
James Liu, Guangxuan Xiao, Kai Li, Jason D Lee, Song Han, Tri Dao, and Tianle Cai. Bitdelta: Your fine-tune may only be worth one bit. Advonces in Neural Information Processing Systems, 37:13579-13600, 2024a.
Jialin Liu, Antoine Morcau, Mike Prcuss, Jeremy Rapin, Baptiste Roziere, Fabicn Teytaud, and Olivier Teytaud. Versatile black-box optimization. In Proceedinggs of the 2020 Genelic and Evolutionary Computation Conference, pp. 620 628, 2020.
Shih-Yang Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: deigpt-deeoompssed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024b.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019a. 11692.
/stps://arxiv.. http://aarivv.oo/aaa///aaa
Zhuang Liu, Mingjic Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In 7th International Conference on Learning Representations. OpenReview.net, 2019b.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXie preprint or Xiv: 09583, 2023a.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang- Wizardeoder: Empowering code large language models with evol-instruct. arXiv preprint 2rXiv:230..000 085668 2023b.
Maric-Catherin de Marneffe, Mandy Simons, and Judith Tonhauser. The BommitmentBank: Investigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung 23, 2019.
Michael Matena and Colin Raffel. Merging models with feshte- averaging. In Advances in Newral Information Processing Systems (NeurIPS), 2021. https://arxiv.ot org/ wwtp:///rg///a///////ao/// wwtp///1.///eseeaaeaaaaaaeaaeees
Mohammed Muqooth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning to route among specialized experts for zero-shot generalization, 2024.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela, Adversarial NLI: A new benchmark for natural language understanding. arXiv preprint arxiv: 1910. 1910.14499, 2019.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Guillermo Ortiz-Jiménez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. NewrIPS, 2023. https://aaxiv: https:://axxvvorr//bb/// 12827.
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulié, and Edoardo Maria Ponti. Modular deep learning. arXio preprinl WWXXi:2302.11523 2023.
Mohammad Taher Pilchvar and Josc Camacho-Collados WiC: The word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of NAACL-HLT, 2019.
Edoardo Maria Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy. Combining parameter-eflicien modules for task-level generalisation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 687 702, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 2020a. http://jmlr. http://mm...orgpppperr////// / html.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer lcarning with unified text-to-text transformer. ArXiv, abs/1910.J1 10683, 2020b.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 160: 100.00500,, 2016.
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. arXiv preprint arXiu:2212.10445, 2022.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. 2011 AAAI Spring Symposium Series, 2011.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot 10.:
task generalization. arXiv preprint 10.00771 2021a.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnad Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations (ICLRR) 2021b. https: //arxiv. wtw.vv..rggbb///111//0
Felix Sattler, Simon Wiedemann, Robert Müller, and Wojciech Samek. Robust and communication- efficient federated learning from non-iid data. IEEE transactions on newral networks and learning systems, 31 (9):3400 3413, 2019a.
Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojeiech Samek. Sparse binary compression: Towards distributed deep learning with minimal communication. In 2019 International Joint Conference Neural Networks (IJCNN) pp. 1-8. IEEE, 2019b.
Rishi Sharma, James Allen, Onid Bakhshandeh, and Nasrin Mostalazadeh. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 752-757, 2018.
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Leo, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. S-lora: Serving thousands of concurrent lora adapters. arXio preprint ar Xrv:22311 10.0285, 2023.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-12000 1642, 2013.
Nikko Strom. Scalable distributed dnn training using commodity gpu cloud computing. In Interspeech, 2015. URL attpps//aag sem.nticschollrr
//to www.eeorroooooeooo org/CorpusID : 9838808.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Scbastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether thought can solve them. orXiv preprint arXiv:2210.0920 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsuorri B. Hashimoto. Stanford alpaca: An mnstrcctionfolloving Ilama model. https://githi: com/
tatsu-lab/s swttn-/frassttsddddroooo _alacca, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv22002. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, ct al. Llama 2: Open foundation and fine-tuned chat models. ariie preprinl arXiv:2307. 099288807 2023b.
Alex Wang, Amanproct Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018, pp. 353, 2018a.
Hongyi Wang, Scott Sievert, Shengchao Liu, Zachary Charles, Dimitris Papailiopoulos, and Stephen Wright. Atomo: Communictiin--efiiieet learning via atomic sparsification. Advances in neural information processing systems, 31, 2018b.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint Xrv: 10560, 2022.
Wei Wen, Cong Xu, Fong Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. Advances in neural information processing systems, 30, 2017.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAAC pp. 1112-1122 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing, 2019.
www./og////aaa/eeeeeoooeee 037771.
Mitchell Wortsman, Gabriel Ihharco, Samir Yitzhak Gadre, Rebecca Roelofs,
Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML), 2022a. https: //tp:///aaxvvvvie www.aii..../abbbaaaaaae www.oog//////////eee 05482..
Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022b. https://arxiv. org/ www.org///bbb//eeeeeeeeeeeeeeeee abs/2109..... 01903.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Smaranda. Muresan, Preslav Nakov, and Aline Villavicencio (ods.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1513- 1528.. Association for Computational Linguistics, 2022.
17

---

Published in Transactions on Machine Learning Research (04/2025)
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging Resolving interference when merging models. In Thirly-seuenlh Conference on Neural Information Processing Systems, 2023.
Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recycling and routing among specialized experts for collaborative learning. CoRR, 2024.
Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. Bigtrans: Augmenting large language models with multilingual translation capability over 100 languages. arXiv preprint wrXi:2335518009, 2023.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilitios from homologous models free lunch. arXiv preprint arXiv:231.000000 2023.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-eficient fine-tuning for transformer- based masked maguago-models, arXiv preprint wxXi::116.101990 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? arXiv preprint arXio:1905.007 07830, 2019.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongshong Li, Peng Gao, and Yu Qiao. Tlama-adapter: Effficient. fine-tuning of language models with zero-init attention. arXiv preprinl arXiv:2303.16199 2023.
Michael Zhu and Suyog Gupta. To prunc, or not to prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Learning Representations. OpenReview.nee 2018.
A
Limitations
While ComPEFT demonstrates siguificant promise, it is important to consider several limitations of this work. Firstly, while average performance strong, certain task types might exhibit reduced effectiveness or require specific hyperparameter tuning. Secondly, we observed some performance sensitivity with ComPEFT applied (IA)* modules, especially on base models with weaker zero-shot capabilities, warranting further investigation into the interplay between base model properties, PEFT methods, and compression. From practical standpoint, while hyperparameter tuning for the scaling factor becomes less crucial for larger models, it remains relevant for smaller models, adding hyperparameter selection step to their deployment. Future work could explore automated or adaptive methods to address this. Furthermore, a rigorous theoretical understanding of ComPEFT is still lacking. We do not have a definitive explanation for the observed performance improvements some cases, nor why these improvements scale with model size, although noise reduction is a possible factor suggested by related works. A deeper understanding of how fine-tuning updates encode information and how ComPEFT interacts with this information is necessary for developing even more refined compression techniques. Finally, to fully unlock the potential wall-clock speedups promised by ComPEFT's ternary vector representations, dedicated engineering effort is needed to develop custom Triton/CUDA /CUDA kernels optimized for operations on sparse ternary data structures, as briefly discussed in our methodology section. These areas represent important avemues for future research, particularly in the context of efficiently serving and composing large numbers of expert PEFT modules for advanced applications.
B Implementation Details
B.1 Training Details
In our research, we utilized the following models, BaRT-base, BERT-Large, RoBERTa-base, RoBERTa-large 15v1.1-base, T5v1.1-larg,, T5-base, T5-large, Flan-T5- TO-3B, LLaMA 7B, 13B, 33B, 65B models. The Tl-n-e--Large and LLaMA models were not trained by us and were used by the authors of QLoRA (Dettmers et al., 2023) and LoraHub (Huang al., 2023). For the experiments in $3.2 and 53.3 on the 7 GLUE (Wang
18

---

Published in Transactions on Machine Learning Research (04/2025)
et al., 2018a) tasks, trained the large datasets (mnli, quli, sst2, qqp) for 1 epoch and the small datasets (rle, mrpe, wnli) for 10 epochs. Whereas for the experiment in $3.5, we followed most of the hyperparameter configuration from the (IA)² (Liu et al., 2022) paper and trained for 2500 steps with a batch size of 8. For each of the 11 datasets in $3.5, we selected 200 examples from the training set to be used as the validation set for best model selection as well as selecting the hyperparameters for COmPEFT. Across all experiments to obtain the trained models we selected different learning rates for cach dataset and PEFT method. For training (IA)s models we selected the learning rate from {le 2, le 3, le 4, le 5), for LoRA from {5e - 2, 5e 3, 5e 4, 5e - 5), and for full model finetuning from {5e 3, 5e 4, 5e 5, 5e 6}. During the training process, b6 was adopted to curtail GPU memory expenditure. For the purpose of evaluation, models from the T5 and TO families were evaluated using rank classification to select the correct label. In this method, the model's log probabilities for all potential label strings are ranked. The model's prediction is deemed accurate if the choice ranked highest aligns with the correct answer. It should be noted that rank classification evaluation can accommodate both classification tasks and multiple-choice tasks.
B.2 Compute Resources Used and Runtimes
We exccuted all our experiments on Nvidia A6000 GPUs equipped with 48GB RAM. Training (IA)² and LoRA models on the TO-3B model for a single ($3.2, $3.5, and $3.7) task takes about 30 minutes to hours depending on the dataset. For T5-Base and T5-Large models ($3.2, $3.7), based on dataset size, needed between 15 minutes and hours per task. Experiments with QLoRA on LLaMA models were done using the original checkpoints from QLoRA paper (Dettmers et al., 2023) for all the 8 instruction tuning datasets and are supplied the authors of QLoRA here." The ComPEFT compression experiments were efficient, with evaluations consuming between 10-30 seconds for the T5-Basc, T5-Large, and TO-3B models. For LLaMA models, following QLoRA (Dettmers et al., 2023), the hyperparameter selection is done on a small held-out subset of MMLU (Hendrycks et al., 2020) benchmark and takes about 8 minutes, 14 minutes, 28 minutes, and 49 minutes for LLaMA 7B, 13B, 33B, and 65B models respectively.
B.3 Employed Datasets and Associated Licences
We use the following datasets in the paper with the following licenses. Apache License 2.0: Flan V2, Self-Instrun Instruct, Chip2 co-by-ne-4.0: Alpaca.
MIT License: Guanaco, Unatural Instructions, HH-RLLFF, Longform
Not. Found: GLUE
B.4 Gradient Noise
Gradients are (almost) never 0 for any parameter, as all parameters somehow affect the result. Thus, we presume most updates in fine-tuning are not more than just noise, rather than learned updates. We compute the mcan and standard deviation of the task vector of a LoRA model finetuned on LLaMa (Touvron et al., 2023a) base model and compare it with the base model. We find the mean of both the LoRA task vector and the base model is close to zero, however, the LoRA task vector has a small standard deviation of 0.0007 as compared to 0.0228 for the LLaMA base. This further confirms the hypothesis that most parameters are changed very little during fine-tuning.
B.5 Why Use Multiplication Factor of Standard Deviation
This decision to use multiplication factors of std is based on a few observations: (1) the task vector parameters are typically normally distributed with almost zero means (see table below), implying that the pretrained parameters have not changed much on average and some specific parameters get huge updates while others just accumulate SGD noise.
(2) This std of task vectors can differ a lot based on the model size and the dataset. (3) We only care about the top-k fraction of the parameters (say top-20% that lie outside the first standard deviation (> o), i.e., that has a magnitude greater than a. Hence, given this different scale of
5 hhtpe///wwggiigkeeeeco//ietetttt
models=qlora
19

---

Published in Transactions on Machine Learning Research (04/2025)
Table 7: Statistics of the distribution of the task vectors for different model sizes and datasets.
Model (L)
Dataset (L)
TV
TVstd
TVaax
TVnin
storycloze
1.61E-02
0.1347
31.9909
-4.8564
TO 3B
winogrande
1.61E-02
0.1357
32.3445
-4.7301
chip2 longform 1.95E-06
8.66E-07
0.0155
0.083
-0.082
LLaMA -
7B
0.0172
0.0859
-0.085
chip2
1.69E-06
0.0106
0.0762
-0.0767
LLaMA - 13B
longform -1477-06
0.0173
0.0767
-0.0747
chip2
-2.39E-07
0.0095
0.0688
-0.0703
LLaMA 33B
longform -8.00E-88
0.0075
0.0703
-0.0708
chip2
-5.07E-07
0.0083
0.062
-0.063
LLaMA 65B
longform
-7.91E-08
0.0097
0.0635
-0.064
chip2
3.63E-07
0.0053
0.0396
-0.0393
LLaMA2 70B
longform
-1.16E-07
0.0053
0.0427
-0.043
top-k task vector parameters across different model sizes, tasks, etc., the standard deviation serves as nice unifying scale that provides us with a constant set of values to try for a, making this process simpler.
In Table 7, we provide the mean, standard deviation, maximum, and minimum values of the task vectors for models of different sizes and datasets. We observe that std, max, and min values change as the model size changes. For example, for the 3B model, the std is 0.13 while for the 70B model, the std is 0.009. Hence, we use a * as allows us try hyperparameters the correct range. However, we agree that there might. be other ways to go about selecting a, for example, learning on a small dataset.
C Additional Results
C.1 Comparision With Other Additional Pruning Methods
We performed additional experiments in setting similar to Table-2, where worked with the Llama- (Touvron et al., 2023a) 70B model and learned qLora (Dettmers et al., 2023) modules of rank 64. We then compressed these parameter updates using ComPEFT, STC, BitDelta (Liu et al., 2024a), and DAREx (Deng et al., 2024) methods. Note that the BitDelta method has two variants. The first variant does not perform any additional training for the scale parameter (referred to as "No Training"). In the "eittelta (No Training)" setting, the scale parameter (a) is set to the mean value of all the parameters in the task vector /delta weights. The "BitDelta. (Training)" variant learns the scale parameter (a) via SGD and hence is not directly comparable with ComPEFT which requires no additional training. For the DAREx method we use the DAREx-q (1/4x) variant, which uses labelled data to select the inverse scaling parameter (gx) for each per-layer separately after pruning. We DAREx, we use sparsity levels of 95% and 99% as used in their paper. The results for the experiments are provided below along the average sizes of the compressed parameters across all the tasks.
From these results, we can clearly see that: (1) ComPEFT performs better than these baseline. (2) DAREx (p=0.95) and (et Training) show slight performance loss compared to the original checkpoint while DAREx (p=0.99) results in huge drop. This is in line with the results presented in their papers. (3) BitDelta (Training) performs similar to ComPEFT, however, this method learns the scalar (a) which requires both forward and backward passes and hence more GPU memory. (4) Note that BitDelta (No Training) sets the scalar (a) as the mean of all the values in the task vector. It very similar to STC which also uses the mean value as the scalar. However, they have a critical difference which that STC also performs sparsification before performing quantization. Hence, in BitDelta the values are (+a, -a) while in STC the values are like (+b,0,-b). We note that STC performs slightly better than BitDelta (No Training), we believe
20

---

Published in Transactions on Machine Learning Research (04/2025)
Table 8: Comparing ComPEFT with other additional Pruning methods.
Dataset
Original
ComPEFT
STC
BitDelta
DAREX-9.
No Training
Training
p= 0095
p=0.99
alpaca-clean
67.13
67.56
66.57
66.27
67.43
65.85
39.57
chip2
65.18
67
64.54
64.31
67.31
63.94
50.18
longform
67.63
68.5
67.02
66.15
68.61
66.14
44.32
oasst1
66.89
67.39
66.15
65.38
67.11
65.48
45.82
self-instruct
62.36
67.18
61.94
61.52
66.82
61.97
49.39
Average 65.84
67.53
65.24
64.73
67.46
64.68
45.86
Size
1.58GB
56MB
56MB
99MB
99MB
395MB
79MB
that this is due to the sparsification step which removes redundant parameters which add noise. Similar phenomenon is also observed in TIES-Merging (Yadav et al., 2023). Lastly, we also report the storage size for the compressed checkpoints where we use different methods to store them. We use golomb coding for ComPEFT/STC, bitmask for Bitdelta, and coo coo_spaaro matrix for DAREx method. The results demonstrate that ComPEFT yields better performance/ size trade-off compared to most of these other methods.
C.2 Comparisons with Advanced PEFT Methods
Table 9: Comparison with other PEFT methods
Dataset
LORA
ComLORA
DORA
ComDORA
alpaca-clcan
67.13
67.56
68.42
69.78
chip2
65.18
67
67.21
68.32
longform
67.63
68.5
69.36
68.92
oasst1
66.89
67.39
68.89
67.63
self-instruct
62.36
67.18
65.26
66.31
Average
65.84
67.53
67.83
68.19
Size
1.58GB
56MB
1.59GB
57MB
We conducted some additional experiments with some other PEFt methods like DoRA (Liu et al., 2024b). For the experimental setting in Table-2 with rank 64 LoRA on the Llama-2 70B model. We performed additional experiments with DoRA of rank 64 and then compressed them using ComPEFT and reported the results. We omitted VeRA (Kopiczko et al., 2023) methods as based on the DoRA paper VeRA typically performs worse than both LoRA and DoRA. Lastly, we omitted HiRA (Huang al., 2025) as the method as due to its recency its code is not available. In Table 9 we present our results. Similar our other finding, we observe that ComPEFT also compress DoRA checkpoints great extent while preserving performance. Moreover, ComDoRA checkpoints slightly outperform ComLoRA's performance.
C.3 Comparison of Compressed Lora With Lower Rank Lora Modules
We perform additional experiments to compare the compressed LoRA modules with lower rank lora module which inherently have smaller sizes compression can be achieved smaller rank. We opt for the experimental setting from Table-2 where we work with the Llama-2 70B model. We perform experiments with rank 32 and 8 the results of which are attached below along with storage sizes.
From the results in Table 10, we can see that: (1) at rank 32 there is a slight drop in performance compared rank 64. We can compress the rank checkpoint as well by >5xx (2) At rank 8, we see a significant drop in performance from 65.84 to 63.77. Moreover, ComPEFT can compress rank lora as well by >25x. (3) for both rank 32 and 8, ComLoRA performs better than the original checkpoints. These results help to
21

---

Published in Transactions on Machine Learning Research (04/2025)
conclude that the observed benefits in compression and performance improvements stem from ComPEFT opposed to the overparameterrizel LoRA adapter.
Table 10:
Comparing ComPEFT with smaller rank LoRA modules.
Dataset
Lora(r=64)
ComLora(r=64)
Lora(r=32)
ComLora(r=32)
Lora(r=8)
ComLora (r=8)
alpaca-clean
67.13
67.56
66.98
67.24
64.82
65.27
chip2
65.18
67
65.24
66.75
63.35
65.18
longform
67.63
68.5
67.14
68.12
65.16
66.74
oasst1
66.89
07.39
65.42
66.92
65.56
self-instruct
62.36
67.18
62.68
67.
61.32
65.81
Average
65.84
67.53
65.49
67.30
63.77
65.71
Size
1.58GB
56MB
790MB
28MB
197MB
7MB
C.4
Validation Set Results
In Table and 12, we provide the validation set results for our main compression experiments on LLaMA, T5, experiments from Section 3.1 and 3.2 respectively.
C.5 Full Results for Compositional Generalization
In Table 13, we present the Zeroshot, ICL, LoraHub, and ComPEFT results for each the BBH tasks.
C.6 Individual Task Results
We present the task level validation and test set results along with model sizes of (IA)², LoRA, and full finetuning for T5-base (Table 20), T5-large (Table 21), TO-3B (Table 22).
C.7 Compressing Model With Smaller Models with Bad ZeroShot Performance
We present the task level validation and test set results along with model sizes for (IA)², LoRA, and full finetuning for BERT-base (Table 14), BERT-large (Table 15), RoBERTa-base (Table 16), RoBERTa-large (Table 17), T5-v1.1-base (Table 18), and T5-v1.1-lare (Table 19). These models are only trained using the pretraining objective and are not teltitass--rainnee Hence, these models have very bad zero/few-shot performance and always require explicit finetuning perform well any downstream tasks. We observe that for the LoRA method, the performance of ComPEFT is similar to the uncompressed full models while being smaller in size. This hints at the fact that the intrinsic dimensionality of the LoRA adaptation is much smaller compared to the number of parameters in the LoRA module. However, for (IA)3 method, the performance drop more, we believe that two reasons for this are: (1) The models are not good zero/few-shot models, and (2) (IA)3 adds very few parameters to perform multiplicative operation on the activations. Therefore, smooth as for good zershot models, and due to this IA3 has to scale different
the loss landscape is activations in very different manner to learn the task. Hence, compressing (IA)3 to sparse sign-vector and constant is not feasible. Whereas, In the case of Lora the updates are added and hence their impact on the final value of the parameter is not huge as the maximum of the LoRA parameter is still very small compared to the base model's parameter value.
22

---

Published in Transactions on Machine Learning Research (04/2025)
Table 11:
We present the performance (Storage GRJ on MMLU Validation for the compressed QLoRA
models.
Dataset (+)
ComPEFT
7B
13B
33B
65B
Self-Instruct
35.62
47.52
55.11
62.13
Longform
31.89
47.80
55.31
62.27
Chip2
33.49
47.15
55.02
62.21
HH-RLHF
32.37
47.19
54.78
62.06
Unnatural Instruct
42.41
49.62
56.28
62.15
Guanaco
33.92
49.52
55.35
62.00
Alpaca
39.82
49.00
55.91
62.37
FLAN v2
43.93
50.86
56.97
63.77
Average
37.88
48.58
55.59
62.37
Validation set performance(Storgee Sixe in MB) avcraged over seven GLUE tasks when compressing (IA) and LoRA modules on different base models.
Table 12:
Method (L)
Base
T5
Large
TO
(IA)
LoRA
(IA)3
LoRA
(IA)
LoRA
ORIGINAL
81.25
81.94
85.08
86.21
87.71
89.94
ComPEFT
81.04
80.96
85.28
86.54
89.14
89.95
IMPROVEMENT
-0.21
-0.98
0.2
0.33
1.43
0.01
Table 13: Task level results: Average performance over 5 seed for LoraHub and ComPEFT for compositional generalization on Big-Bench-Haad
Task
Zeroshot
ICL
LoraHub (Avg)
ComPEFT (Avg)
LoraHub (Best)
ComPEFT (Best)
Logical Deduction Three Objects
0.0
51.3
41.9
28.4
51.3
48.0
Tracking Shuffled Objects Five Objects
12.0
12.0
9.6
11.3
12.0
12.0
Web Of Lies
51.0
51.0
28.
11.7
49.3
56.01
Tracking Shuffled Objects Seven Objects
6.7
6.7
5.3
6.7
Date Understanding
15.3
22.7
29.1
42.0
38.7
Navigate
47.3
41.0
48.4
38.55
50.7
50.0
Multistep Arithmetic Two
0..7
0.7
017
0.5
1.3
0.7
Boolean Expressions
51.0
58.7
55.9
55.7
57.3
Hyperbaton
6.7
55.2
67.4
Tracking Shuffled Objects Three Objects
24.7
30.7
26.7
21.6
29.3
24.7
Sports Understanding
56.0
56.0
46.4
63.1
54.7
58.0
Logical Deduction Seven Objects
12.7
42.0
35.5
37.6
40.0
40.0
Causal Judgement
37.5
40.7
Penguins In Table
43.5
39.1
36.1
44.3
45.7
47.8
Geometric Shapes
6.7
18.7
9.6
7.3
19.3
9.3
Reasouing About Colored Objects
32,0
38.7
38.0
40.6
39.3
44.0
Dyck Languages
1.3
2.7
Disambiguation
0.0
60.3
14.3
6.5
51.3
29.3
Salient Translation Error Detection
37.3
46.0
31.3
38.5
41.7
43.3
Movie Recommendation
62.7
52.7
61.1
58.0
67.3
62.0
50.0
55.1
49.2
Formal Fallacies
51.3
$8.0
41.3
32.7
Logical Deduction Five Objects
21.3
40.0
33.6
36.3
36.7
42,0
Temporal Sequences
16.7
26.7
18.7
19.5
20.0
21.3
Word Sorting
1.3
0.7
1.2
Ruin Names
23.3
22.
Object Counting
34.7
32.0
35.5
35.3
36.7
36.0
Average
27.0
37.3
30.5
30.6
37.3
36.4
23

---

Published in Transactions on Machine Learning Research (04/2025)
Table 14: Validation and Test set performance along with storage size in MB for bert-base-uncased Model, for (IA)s, LoRA and Full model finetuning.
Original (Val)
Original (Test)
ComPEFT (Val)
ComPEFT (Test)
PEFT
Task
full mnli
84.7
83. 03.1(20s, (208.8)
84.3
82.6
mrpc
86.8
97-5(208.8)
86.8
97.0
quli
91.9
91.0(208.8)
91.9
90.9
90.1
90.3 8)
90.0
90.0(15.4)
rte
66.4
94.5
69.0
94.0(7.44
sst2
91.1
97 (2.0(208.8)
92.2
96.0(7.4)
wnli
56.3
57.0 57.0(208.8)
56.3
57.0(1.4)
ia3
mnli
79.2
(8.9(0.1)
57.6
56.2(0.0)
mrpc
84.6
94.5(0.1)
31.6
34.5(o.0)
quli
87.9
87.5(0.1)
49.3
49.6(0.0)
qqp
84.6
84.(().1)
63.2
63.2
59.2
73.5(0.1)
52.7
55.5(0.0)
sst2
91.5
91.0(p.1)
49.1
45.5(0.0)
wnli
54.9
(7.0(o.1)
56.3
57.0(0.0)
lora
mnli
82.5
81.4(2.)
76.6
76.9(0.2)
mrpc
86.3
97.5(2.))
82.8
94.0
qnli
91.7
91.0(2.6)
90.8
90.1(0.2)
qqp
89.4
89.5(2.6)
87.4
87.5
rtc
61.7
68.5(2.0)
58.8
66.0
sst2
92.4
92.5(2.6
91.6
91.5(0.2)
wnli
56.3
57.0 (7.0(2.6)
59.2
56.0
24

---

Published in Transactions on Machine Learning Research (04/2025)
Table 15: Validation and Test set performance along with storage size in MB for bert-large-uncased Model,
for (IA)s, LoRA and Full model finetuming.
Original (Val)
Original (Test)
ComPEFT (Val)
ComPEFT (Test)
PEFT
Task mnli
full
85.4
84.0 (439()33.2)
85.5
83.7
mrpc
88.2
97. 97.5()33)2)
88.5
97.5(47.2)
qnli
91.0
(9.4())3
91.2
89.6(36.8)
88.6
88. (635())))2)
88.4
88.6(36.8)
rte
71.5
(639.)3).2)
70.4
92.5
sst2
92.9
93.5 (3.5.)33
93.5
92.5
wnli
56.3
(6.0()39.2)
57.8
58.0
ia3
mnli
82.4
81.9(
59.5
59.6(p.0)
mrpe
84.6
96.0(0.3)
31.6
34.5
quli
88.6
87.6(0.3)
59.6
59.9
qqp
87.7
87.4(0.3)
73.2
73.8
rte
58.8
72.5(0.3)
52.7
35.5(0.0)
sst2
92.3
88.0
49.1
45.5
wnli
60.6
55.0
56.3
57.0(0.0)
lora mnli
83.6
(2.9(6.8)
76.4
74.8(p.6)
mrpe
88.7
94.0(6.8)
87.0
92.0
qnli
88.6
86.9(6.8)
82.6
81.1((0.6
qqp
87.0
87.1(6.8)
76.3
76.9
rte
59.9
77.0().8)
59.6
71.0
sst2
93.7
94.0(6.8)
93.6
93.0
wnli
56.3
57.0(6.8)
57.8
56.0
25

---

Published in Transactions on Machine Learning Research (04/2025)
Table 16: Validation and Test set performance along with storage size in MB for roberta-base Model, for (IA)", LoRA and Full model finetuning.
Original (Val)
Original (Test)
ComPEFT (Val)
ComPEFT (Test)
PEFT
Task mnli
full
86.4
86.4(237.8)
86.6
86.2
mrpc
87.0
89.0(237.8)
86.0
84.5(8.4)
qnli
91.8
91 (1.7
91.7
91.0(17.6)
89.1
89.4(237.8)
89.1
89.2(17.6)
rte
75.4
91.5(237.8)
78.0
93.0
sst2
95.2
95.2(237.8)
94.2
94.0(8.4)
wnli
56.3
56.0(237.8)
56.3
45.0(5.0)
ia3
mnli
84.1
83.4(1.2)
43.0
(4.7(
mrpe
88.7
98.0
71.6
(0.0
quli
89.7
88.9(1.2)
50.7
50.4(u.1)
qqp
87.0
87.1(1.2)
80.9
80.8
rte
73.3
93.0
54.9
54.5(o.1)
sst2
93.5
92.0(1.2)
76.3
72.5
wnli
56.3
57.0
56.3
57.0
lora mnli
87.0
86.1
86.2
85.6(0.3)
mrpe
89.5
98.5(3.7)
88.5
97.0
qnli
91.1
92.3(3.7)
90.0
89.9
qqp
88.8
88.8(3.7)
88.2
88.4(..))
rte
79.4
(3.0(3.7)
79.4
96.0
sst2
94.2
95.0
93.1
94.0
wnli
56.3
57.0
56.3
57.0
26

---

Published in Transactions on Machine Learning Research (04/2025)
Table 17: Validation and Test set performance along with storage size in MB for roberta-large Model, for (IA)", LoRA and Full model finetuning.
Original (Val)
Original (Test)
ComPEFT (Val)
ComPEFT (Test)
PEFT
Task
full
mnli
90.6
89.3(677.8)
90.4
89.4(50.0)
mrpc
89.2
97.5(077.8)
89.2
97.5(24.1)
qnli
93.6
(2.9
93.5
(3.3
90.1
89.9(677.8)
90.1
(5.9
rte
85.6
98.5,677.8)
85.2
98.0 98.0(50.0)
sst2
95.4
95.2(677.8)
96.3
95.0
wnli
56.3
57.0(677.8)
57.8
61.0(
ia3
mnli
89.5
88.5(2.3)
36.5
35.5
mrpe
86.8
86.5(2.3)
68.4
65.5
quli
92.3
92.1
51.6
50.4(u.1)
qqp
88.5
87.9(2.3)
63.2
63.2
rte
80.1
94.0
54.2
57.0
sst2
94.3
93.0(2.3)
55.2
39.5
wnli
56.3
57.0(2.3)
60.6
51.0
lora mnli
89.9
(8.3().8)
85.0
83.8(p.0)
mrpe
90.0
93.5(8.8)
90.4
90.0
qnli
93.4
92.9(8.8)
91.0
90.3
qqp
89.1
88.9(8.8)
86.5
85.9
rte
80.5
93.5(8.8)
79.1
89.5(p.8)
sst2
95.2
93.0(8.8)
94.8
90.5
wnli
56.3
57.0(8.8)
56.3
57.0
27

---

Published in Transactions on Machine Learning Research (04/2025)
Table 18: Validation and Test set performance along with storage size in MB for t5-v1.1-base Model, for (IA)", LoRA and Full model finetuning.
Original (Val)
Original (Test)
ComPEFT (Val)
ComPEFT (Test)
PEFT
Task mnli
full
89.8
89.8(472.2)
88.8
89.0(34.9)
mrpc
80.9
74.5(172.2)
82.6
75.5(34.9)
qnli
88.0
88.6(472.2)
86.7
87. 87.2
78.6
(8.9(172.2)
77.0
77.6(27.2)
rte
59.2
49.0(472.2)
59.2
61.0(p.9)
sst2
93.4
91.0(472.2)
93.8
91.5(27.2)
wnli
56.3
47.0(172.2)
57.8
49.0
ia3
mnli
84.6
83.8
54.3
54.4(0.0)
mrpe
82.8
81.5(0.2)
82.8
78.5
quli
85.2
86.3(0.))
60.7
61.8(u.0)
qqp
85.0
85.4(0.2)
78.6
78.8
rte
54.9
49.0
63.2
62.5
sst2
92.3
91.0
89.0
87.0
wnli
52.1
57.0
52.1
57.0
lora mnli
66.9
(6.3(1.4
56.6
57.1
mrpe
72.1
67.0(a.4)
68.4
64.0
qnli
87.1
88.8(4.4)
86.7
88.5
qqp
78.3
78.9(4.4)
72.0
72.2
rte
55.2
50.5(4.4)
53.1
49.5(0.2)
sst2
93.0
91.5(1.4
92.9
90.5
wnli
56.3
47.0(4.4)
78.9
76.0
28

---

Published in Transactions on Machine Learning Research (04/2025)
Table 19: Validation and Test set performance along with storage size in MB for t5-vi.1-large Model, for (IA)", LoRA and Full model finetuning.
Original (Val)
Original (Test)
ComPEFT (Val)
ComPEFT (Test)
PEFT
Task
full
mrpe
86.3
(143.7)
86.3
85.0
quli
94.0
94.0 (4.0(1493.7)
94.4
94.9
90.2
90.5(1493.7)
89.4
89.5
rte
74.0
(6.0.)
75.4
74.5(140.00
sst2
95.6
(140.14
95.4
92.5(86.1)
wnli
52.1
57.0(1493.7)
52.1
57.0(31.4)
ia3
mnli
92.0
92.4(0.5)
54.3
54.4(0.0)
mrpe
90.9
86.0(0.5)
77.9
77.5
quli
92.0
92.3(u.5)
79.4
78.1
qqp
87.2
87.5(0.5)
78.7
78.8(u.0)
rte
69.7
67.0
69.0
73.5
sst2
95.2
93.0
79.5
81.0
wnli
52.1
57.0
52.1
57.0
lora mnli
92.3
(3.2(11.8) (11.8)
91.8
92.4
mrpc
78.2
(1.0)
77.7
76.0
qnli
90.4
91.9
87.0
87.4(u.9)
qqp
87.1
87.6(11.8)
86.0
86.8
rte
52.7
56.0
53.8
45.5(0.4)
sst2
93.9
89.0
61.4
56.0
wnli
56.3
47.0
56.3
47.0
29

---

Published in Transactions on Machine Learning Research (04/2025)
Table 20: Validation and Test set performance along with storage size in MB for t5-base Model, for (TA)3;
LoRA and Full model finetuning.
Original (Val)
Original (Test)
ComPEFT (Val)
ComPEFT (Test)
PEFT
Task
full
mnli
91.2
91.2(425.22
89.9
90.4(31.4)
mrpc
89.7
86.0(125.2)
87.0
75.5
qnli
93.3
93.3(425.2)
91.3
91.3(39.9)
91.3
91.4 91.4(125.2)
70.4
70.6(15.1)
rte
76.2
77.0 (7.0(425.2)
74.7
77.0(39.9)
sst2
95.6
93.5,225
95.5
93.5
wnli
56.3
47.0(125.2)
56.3
48.0
ia3
mnli
91.0
90.4(0.2)
90.4
90.3
mrpe
85.5
84.0(0.2)
85.8
81.5
quli
92.6
92.9(0.2)
92.5
92.4
qqp
89.5
89.8
87.1
87.0
rte
63.9
62.0
65.0
58.5(0.0)
sst2
94.2
93.0
94.4
93.0
wnli
52.1
57.0
52.1
57.0
lora mnli
91.0
90.2(6.2)
91.3
90.5(o.5)
mrpe
90.9
84.0
84.1
77.5
qnli
93.4
93.5
93.3
93.7
qqp
90.5
90.5
90.3
90.6(u.4)
rte
52.7
53.0
57.0
53.5(o.1)
sst2
94.5
94.0
94.4
93.5(0.4)
wnli
60.6
49.0
56.3
47.0
30

---

Published in Transactions on Machine Learning Research (04/2025)
Table 21: Validation and Test set performance along with storage size in MB for t5-large Model, for (IA)3;
LoRA and Full model finetuning.
Original (Val)
Original (Test)
ComPEFT (Val)
ComPEFT (Test)
PEFT
Task
full
mnli
93.4
93.6(1407.0)
93.1
(3.2()1.1)
mrpe
91.4
88.5
90.0
84.5(131.9)
quli
94.4
94.4(14077.0)
94.5
94.7 (131.9)
qqp
91.8
91.9(1407.0)
68.8
(9.6
rte
83.8
88.0
82.0
82.0
sst2
93.5
(3.0.)
93.6
93.0(131.9)
wnli
56.3
47.0
78.9
(6.0(103.9)
ia3
mnli
93.0
(2.5(0.7)
93.0
92.7(u.1)
mrpc
90.2
88.5(
90.7
90.0
qnli
94.4
94.2(0.7)
94.3
94.1
90.6
91.1(u.7)
89.3
90.0
rte
79.8
85.0
82.0
83.5
sst2
95.5
95.0
95.6
94.0
wnli
52.1
(7.0
52.1
57.0(0.0)
lora mnli
93.0
93.5
93.0
98.5(1.2)
mrpc
90.9
87.5(16.5)
85.8
85.0(1.6)
quli
94.5
94.5(16.55
94.1
92.9(1.6)
qqp
90.9
91.4(16.5)
90.2
90.9(1.6)
rte
82.0
82.0
78.0
79.0(1.6)
sst2
95.9
95.5
95.8
94.0(1.6)
wnli
56.3
47.0
69.0
57.0
Table 22: Validation and Test set performance along with storage size in MB for TO-3B Model, for (IA)3,
and LoRA.
Original (Val)
Original (Test)
ComPEFT (Val)
ComPEFT (Test)
PEFT
Task
ia3
mnli
94.1
94.4(1.0)
93.4
93.8
mrpc
89.7
89.5(1.0)
90.4
89.0
quli
94.9
95.3
95.8
95.5
qqp
89.8
90.2(1.0)
89.6
90.0
rte
86.6
89.0
87.4
88.0(u.0)
sst2
96.8
93.0
96.9
93.0
wnli
62.0
74.0
70.4
69.0
lora
mnli
93.8
93.6(33.8)
93.5
94.2(2.5)
mrpc
90.4
90.5(33.8)
90.0
88.5(1.9)
qnli
95.8
(3.7)
95.8
96.0(2.5)
qqp
90.3
90.7 (33.8)
90.4
90.8
rte
89.2
89. 89.1 (33.8)
88.4
90.0
sst2
96.8
95.0 (33.8
96.9
93.0
wnli
73.2
73.0 (3.8)33.8)
74.6
74.0
31


Reviews:
[{"review_id": "uFxZ7qTiMA", "reviewer": "Reviewer_Jdms", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The paper addresses delta parameter compression, a crucial topic for improving the efficiency of large language models (LLMs).\n\n2. It presents experiments across different model scales and tasks, demonstrating robustness.\n\n3. The proposed ComPEFT method effectively reduces storage requirements, making PEFT more practical for deployment.\n\n**Weaknesses:**\n\n1. The paper lacks references to existing methods. For example, DAREx [1] efficiently prunes over 90% of delta parameters, and BitDelta [2] compresses delta parameters into 1-bit representations. A discussion of these approaches would provide better context.\n\n2. Limited novelty: The method appears to be a combination of delta pruning and compression, both of which have been explored in prior work [1,2].\n\n3. Limited benchmarking against delta compression methods: While the paper compares ComPEFT to STC and baseline PEFT techniques, it does not evaluate its performance against delta compression methods such as [1,2]. Adding these comparisons would strengthen the paper. Even if the combination lacks novelty, demonstrating superior performance would still make it valuable.\n\n[1] DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models, ICLR 2025.\n\n[2] BitDelta: Your Fine-Tune May Only Be Worth One Bit, NeurIPS 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces ComPEFT, a compression method for delta parameters that significantly reduces their storage and communication overhead without requiring additional training. The core idea is to leverage sparsity and quantization to achieve high compression rates while maintaining or even improving model performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add discussion on delta compression methods [1,2]\n2. Adding comparisons with   [1,2] would strengthen the paper. Even if the combination lacks novelty, demonstrating superior performance would still make it valuable.\n\n[1] DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models, ICLR 2025.\n\n[2] BitDelta: Your Fine-Tune May Only Be Worth One Bit, NeurIPS 2024.", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SiH3LaZ6Bn", "reviewer": "Reviewer_GSQx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Summary of strengths\n\n- ComPEFT leverages **sparsification and ternary quantization** to significantly reduce PEFT module size **without requiring additional training** while preserving or even enhancing model performance.\n- ComPEFT **generalizes well across multiple model architectures** and demonstrates improved efficiency as model size increases, making it particularly effective for larger models.\n- ComPEFT also improves **model merging performance** with minimized storage and communication overhead, enabling more efficient expert model serving.\n- The paper provides a detailed and comprehensive analysis with ablations, comparisons, and practical deployment scenarios.\n\nSummary of weaknesses\n\n- The approach is primarily based on **magnitude-based pruning**, a well-explored model compression technique. While effective, its novelty is somewhat limited. The authors are encouraged to elaborate on the key differences between ComPEFT and prior works in this area.\n\n- Lack of Comparisons with Advanced PEFT Methods: The experiments mainly focus on QLoRA, IA3, and LoRA, but do not include comparisons with more advanced PEFT approaches such as HiRA, DoRA, and VeRA, which have demonstrated further parameter reductions in LoRA. A comparison with these state-of-the-art techniques would strengthen the paper’s contributions.\n\n  [1] HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models (ICLR 2025 Oral)\n\n  [2] DoRA: Weight-Decomposed Low-Rank Adaptation (ICML 2024 Oral)\n\n  [3] VeRA: Vector-based Random Matrix Adaptation (ICLR 2024)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces ComPEFT, a compression technique for Parameter-Efficient Fine-Tuning (PEFT) that reduces communication and memory bottlenecks in serving expert modules for large language models. ComPEFT reduces PEFT module sizes by 8x - 50x without additional training while preserving or even improving performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- **Expand Figure 3 to Include Comparisons**: Add HiRA, DoRA, VeRA, or other recent PEFT methods to Figure 3, along with their corresponding ComPEFT-compressed versions, to better demonstrate the method’s applicability and performance across different PEFT strategies.\n\n- The paper does not seem to specify the rank settings used for LoRA and other PEFT methods in experiments. Please provide the details of these settings to ensure clarity and reproducibility.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0yILt7hX93", "reviewer": "Reviewer_XZfS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n    - **[Major]** A simple yet effective method, supported by extensive experimental evidence.\n    - **[Major]** The paper is well-written and clearly presented.\n    - **[Major]** Comprehensive experiments demonstrate the method’s applicability across various tasks, along with ablation studies that provide insights into the effects of hyperparameters.\n- Weaknesses\n    - I am uncertain whether a 3.2GB QLoRA adapter for LLaMA-65B is a good example. To achieve this size, the LoRA adapter would require approximately rank ~128 (please correct me if I am mistaken), resulting in around 1.6B parameters. This is relatively rare, and in such cases, I wonder how much the rank can be reduced while maintaining good performance. Can it also achieve a 26× storage reduction? My conjecture is that it likely cannot, but 5× is highly possible, and at least a discussion is needed. This also relates to the observation on page 3: *\"a surprising finding that as the base model gets bigger, their task vectors become more compressible...\"* In fact, Table 1 of [this paper](https://arxiv.org/pdf/2310.17513) already indicates that larger models require a lower LoRA rank to achieve the desired performance. Additionally, empirical evidence suggests that high-rank LoRA often underperforms low-rank LoRA in many scenarios, especially when the adapter is already overparameterized and the task is relatively simple. Therefore, it is unclear whether the observed benefits in compression and performance improvements stem from this specific method or just from the overparameterized LoRA adapter.\n    - Some related works are not sufficiently discussed, particularly **\"Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead.\"** Additionally, some fundamental baselines, such as rank reduction, are missing, as mentioned above.\n    - **[Minor]** Typo in Algorithm 1: In the fourth line from the bottom, *tilde γ* should be used.\n    - If I understand correctly, this method essentially performs sparse ternary compression (STE) but with an additional scaling factor, *α*. In Figure 5, STE appears significantly worse than the proposed method, but in Figure 6, the optimal *α* value is frequently 1. Could you clarify this?\n    - In Appendix Table 10, the average performance of ComPEFT appears to be lower than ICL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a general compression framework for PEFT adapters (or model updates) based on sparsification followed by magnitude quantization. The method compresses model updates into a single scalar magnitude, with each parameter retaining only its sign (-1, 0, or +1), resulting in a sparse ternary representation. While the core technique is not novel, the paper introduces a variant that incorporates a scaling factor as a hyperparameter and conducts an extensive range of experiments to evaluate its effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Include experiments comparing the method to basic baselines, such as rank reduction.\n- Discuss [1] in greater detail and add comparative experiments if necessary.\n\n[1] Brüel-Gabrielsson, Rickard, et al. \"Compress then serve: Serving thousands of lora adapters with little overhead.\" *arXiv preprint arXiv:2407.00066* (2024).", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents ComPEFT, a practical compression method significantly reducing the size of PEFT modules without additional training. Reviewer discussions mainly concerned novelty, baseline comparisons, and compatibility with recent PEFT methods. The authors resolved these by conducting comprehensive experiments comparing ComPEFT against relevant prior work, including structured pruning and LoRA variants. Reviewers ultimately recognized the practical impact, thorough experiments, and effective clarifications provided by the authors, recommending acceptance with minor revisions to clearly incorporate these additional results into the final manuscript.

Overall, the paper addresses an important practical challenge and proposes a simple yet robust approach that enables efficient deployment of LLMs. By effectively compressing PEFT modules, ComPEFT enhances flexibility and efficiency during inference, significantly benefiting real-world applications involving a large number of experts and/or frequent expert module swapping.
Decision: Accept (poster)