Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression
Authors: Xiaohui Wang
Peng Ye
Chenyu Huang
Shenghe Zheng
Bo Zhang
LEI BAI
Wanli Ouyang
Tao Chen
Venue: NeurIPS 2025 poster ()
Keywords: Delta Compression
Pruning
Quantization
Primary area: deep_learning
TL;DR: We propose an information-preserving and data-free pipeline for ultra-efficient delta compression, achieving ultra-high compression ratios and strong performance across various models.

Abstract: With the rise of the fine-tuned–pretrained paradigm, storing numerous fine-tuned models for multi-tasking creates significant storage overhead.
Delta compression alleviates this by storing only the pretrained model and the highly compressed delta weights (the differences between fine-tuned and pretrained model weights). 
However, existing methods fail to maintain both high compression and performance, and often rely on data.
To address these challenges, we propose UltraDelta, the first data-free delta compression pipeline that achieves both ultra-high compression and strong performance. 
UltraDelta is designed to minimize redundancy, maximize information, and stabilize performance across inter-layer, intra-layer, and global dimensions, using three key components:
(1) Variance-Based Mixed Sparsity Allocation assigns sparsity based on variance, giving lower sparsity to high-variance layers to preserve inter-layer information.
(2) Distribution-Aware Compression applies uniform quantization and then groups parameters by value, followed by group-wise pruning, to better preserve intra-layer distribution.
(3) Trace-Norm-Guided Rescaling uses the trace norm of delta weights to estimate a global rescaling factor, improving model stability under higher compression.
Extensive experiments across 
(a) large language models (fine-tuned on LLaMA-2 7B and 13B) with up to 50$\times$ compression, 
(b) general NLP models (RoBERTa-base, T5-base) with up to 224$\times$ compression,
(c) vision models (ViT-B/32, ViT-L/14) with up to 132$\times$ compression, and
(d) multi-modal models (BEiT-3) with 18$\times$ compression, 
demonstrate that UltraDelta consistently outperforms existing methods, especially under ultra-high compression.
Code is available at https://github.com/xiaohuiwang000/UltraDelta.

Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression
Xiaohui Wang!*
Peng Ye
Chenyu Huang'
Shenghe Zheng²
Bo Zhang?
Lei Bai
Wanli Ouyang 2.3 Tao Chen 1.4
Fudan University
Shanghai Al Laboratory
The Chinese University of Hong Kong
*Shangaai Innovation Institute
heatherwang000ggaail com setchen@fudan edu. cn
Abstract
With the rise of the fine-tuned-pretraine paradigm, storing numerous fine-tuned models for multi-tasking creates significant storage overhead. Delta compression alleviates this by storing only the pretrained model and the highly compressed delta weights (the differences between fine-tuned and pretrained model weights). How- ever, existing methods fail to maintain both high compression and performance, and often rely on data. To address these challenges, we propose UltraDelta, the first data-fre free-free delta compression pipeline that achieves both ultra-high compression and strong performance. UltraDelta is designed to minimize redundancy, maximize information, and stabilize performance across inter-layer, intra-layer and global dimensions, using three key components: (1) Variance- Based Mixed Sparsity Allo- cation assigns sparsity based on variance, giving lower sparsity to high-variance layers to preserve inter-layer information. (2) Distribution-Awart Compression ap- plies uniform quantization and then groups parameters by value, followed by group- wise pruning, to better preserve intra-layer distribution. (3) Trace-Norm- Guided Rescaling uses the trace norm of delta weights to estimate global rescaling factor, improving model stability under higher compression. Extensive experiments across (a) large language models (fine-tuned on LLaMA-2 7B and 13B) with up to 50x compression, (b) general NLP models (RoBERTa-base, T5-base) with up to 224x compression, (c) vision models (ViT-B/32, ViT-L/14) with up to 132x compression, and (d)
models (BEiT-3) with 18x compression, demonstrate that UltraDelta consistently outperforms existing methods, especially under ultra-high compression. Code is available at
Introduction
As fine-tuning pretrained models for downstream tasks becomes increasingly popular, growing number of task-specific fine-tuned models have been developed across various domains. To obtain multi-task capabilities and achieve optimal performance across tasks, deploying multiple fine-tuned models simultaneously has become a common practice. However, multi-model deployment introduces severe storage and computational overhead, since each fine-tuned model requires storing a full set of parameters. Delta Compression has emerged as promising solution to this problem by substantially reducing storage requirements. Instead of storing multiple complete fine-tuned models, delta compression works by storing a single pretrained model along with set of delta weights (i.e., the differences between each fine-tuned model and the pretrained model) and then applying aggressive compression to these delta weights to minimize storage overhead. These delta weights
"Equal Contribution.
t "Corrssonniigg Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) 138 LLMS
(b) TS-base
(c) VIT-L/14
(d) BEIT-3
Compreszion Ratio [x]
50
pmmprrssioo Retio (x)
130
152
Compression Ratio
Ratio
-
127
23-
16
16
70
60 Average Performance
10
Average Performance
Average Performance
Average Performance
DARE (Sarn Sparslly
DARE iSimlar Cumprataiion
MP (Samu Sparsity)
HP (Simalar Compresiant
BtDeha
Dalla-CoMe
UbraDalta (Qurs)
Fine-luned Mocels
Figure I: Compression vs. Performance. We compare UltraDelta with baselines, showing strong performance ultra-high compression. DARE and Magnitude Pruning are evaluated in two settings: with the same sparsity as UltraDelta, and using pruning alone to match a similar compression ratio as UltraDelta. Subfigures show average performance (a) ALaMA-2-13B fine-tuned on 3 tasks, (b) T5-base on NLP tasks, (c) ViT-L/14 on 8 vision tasks, and (d) BEÍT-3 on 3 vision- lision-lagguage tasks.
often highly redundant, allowing for substantial compression with minimal impact model performance, thereby significantly reducing storage costs.
Currently, number of works have been proposed for delta compression, which can be broadly categorized into two main types: Pruning and Quantization. Pruning methods reduce model size by removing parameters. For example, DARE [91] randomly prunes delta parameters, while Magnitude Pruning 27 28 46 (45 removes low-magni parameters. Quantization methods work by mapping full-precision parameters to low-bit representations. For example, BitDelta (53) uses masking and rescaling to achieve 1-bit quantization, while Delta-CoMe [64] applies mixed-precision quantization to matrices obtained from singular value decomposition. Other approaches, such as DeltaZip (90), combine quantization with pruning to improve deployment efficiency. However, these methods all struggle balance ultra-high compression with strong performance, mainly due to: (I) Ignoring inter- layer differences: All layers are treated equally, ignoring that different layers contribute unequally to the model, which limits information preservation. (2) Disrupting intra-layer distributions: Intra-layer weight distributions are crucial to performance but are often distorted by aggressive quantization or pruning. (3) Lack of stability under ultra-high compression: Existing methods struggle to maintain stability under ultra-high compression without relying on data, leading to severe performance drops.
To address these limitations, propose UltraDelta, the first data-free pipeline achieve ultra- efficient delta compression, delivering both ultra-high compression and strong performance. UI- traDelta focuses minimizing parameter redundancy, maximizing information preservation, and enhancing model stability across three dimensions: inter-layer, intra-layer and global. It consists of three key components: (1) Variance- Based Mixed Sparsity Allocation (MSA, inter-layer): We theoretically show that layer-wise variance reflects the amount of information in each layer. Based on this insight, we assign lower sparsity to layers with higher variance to better preserve critical informa- tion. (2) Distribution-Aue Aware Compression (DAC, intra-layer): We first apply uniform quantization, and group parameters by their quantized value, then perform random pruning within each group to maintain the relative proportions across different values and better preserve the original distribution. (3) Trace-Norm-Ce Guided Rescaling (TNGR, global): We observe that under extreme sparsity, the standard rescaling factor 1/(1 s) becomes insufficient to stabilize performance (where is the sparsity rate). We introduce a refined rescaling factor 7/(1 s), where y is heuristically estimated from the trace norm of each delta weight and is smaller for delta weights with larger trace norms, enhancing robustness under ultra-high compression.
We conduct extensive experiments across models of different scales, types, and tasks to evaluate the effectiveness and robustness of Ultraeelta. The results demonstrate its exceptional performance under -igra-highh compression: (1) Large Language Models: UltraDelta compresses LLaMA-2 series models 77 56,6 57 32x compression for 7B and 50x for 13B, consistently outperforming all baselines and even surpassing the average performance of fine-tuned models. To further demonstrate generality, we also evaluate on more recent architectures, including LLaMA-3.1 (43, Qwen2.5 [89 and Qwen3 [77]. (2) General NLP Models: UltraDelta achieves 224x compression on T5-base 67 and 32x on RoBERTa-base
54 across 8 NLP tasks, even exceeding the average performance
of fine-tuned models on T5-base. (3) Vision Models: UltraDelta achieves 50x compression on
ViT-B/32 and x32x on VIT-L/14 (66 across image classification tasks, achieving completely
lossless performance on ViT-L/14 compared with fine-tuned models. (4) Multi-modi Models: On

---

BEÎT-3 18), we achieve 18x compression on 3 vision-language tasks, outperforming all baselines across all tasks. As illustrated in Fig. IT we present compression-peree peroonnmneeeeeeee trade-off for all four model types. UltraDelta consistently achieves the hest trade-off, outperforming all baselines, and in some cases, even surpassing the performance of fine-tuned models under ultra-high compression.
The main contributions of this paper can be summarized as follows:
To break the compression ceiling of delta weights, analyze the limitations of existing methods in information preservation and model stability, and propose UltraDelta, the first data-free pipeline enabling ultra-efficient delta compression, achieving both Altra-hight compression ratios and strong performance without relying on any data.
To enhance information preservation and model stability, we introduce three novel techniques: Variance- Based Mixed Sparsity Allocation to prioritize critical layers, Distribution-Aware Compres- sion to preserve the original weight distribution, and Trace-Norm-Guun Guidedd Rescaling to stabilize model performance under extreme sparsity.
Extensive experiments across large language models, general NLP models, vision models, and multi-modal models demonstrate that UltraDelta consistently outperforms all baselines and even surpasses fine-tuned models in several settings, showcasing its effectiveness and robustness in achieving ultra-efficient delta compression.
2
Related Work
Delta Compression is a technique that compresses the delta weights. It is especially beneficial in multi-model deployment scenarios, where many task-specific models fine-tuned from the same pretrained model. By storing only the highly compressed delta weights alongside the shared pretrained model, delta compression significantly eliminates redundant storage. Existing methods for delta compression primarily fall into two categories: pruning and quantization.
Delta Weight Pruning reduces model size by removing parameters from the delta weights. Magni- tude Pruning [27 28 45 [46] removes weights with the smallest magnitudes but fails to preserve information and causes significant performance drops under high sparsity, as it disrupts the original weight distribution. (9 applics random pruning combined with global rescaling based on sparsity improve robustness, but suffers from instability under high sparsity. DAREx (00 refines DARE's rescaling using activation statistics, and DeltaDQ (38 performs group-wise pruning and selects the optimal group size based on attention error. However, both methods require data for tuning, limiting their use in data-free scenarios. Some methods also adopt layer-wist wise pruning [47] [55,, but the metrics are often computationally expensive and usually require data. Overall, pruning effectively reduces parameter redundancy but suffers from several drawbacks, such as insufficient information preservation, instability under high sparsity, and reliance additional data.
Delta Weight Quantization compresses delta weights by mapping them to low-bit representations. GPT-Zip extends GPTQ 22 and quantizes delta parameters to 2-bit. Delta-DCT [33] leverages the discrete cosine transform and compresses delta weights in the frequency domain. BitDelta [53] uses binary masks and scaling factors for 1-bit quantization, but requires calibration data and also distorts the original weight distribution due to the binary masks. Delta-Co CoMe [64] applies singular value decomposition and mixed-precision quantization the decomposed matrices to achieve 1-bit compression. Overall, although quantization effectively reduces parameter bit-width, is often limited to 1-bit precision, which prevents further minimization of parameter redundancy.
Hybrid Approach combines delta weight pruning and delta weight quantization to leverage the advantages of both. DeltaZip [90] employs structured pruning and quantization, primarily aiming to improve deployment efficiency and hardware acceleration, but it falls short in preserving critical information, limiting its effectiveness under ultra-high compression. ComPEFT (88), the other hand, applies magnitude pruning and then quantizes the remaining weights with a fixed shared value. However, it requires validation data and, due to the reliance on magnitude pruning, disrupts structural integrity and fails to preserve critical information under high sparsity.
In contrast, Ultraeelta is a data-free hybrid compression method that explicitly focuses minimizing parameter redundancy, maximizing information preservation, and enhancing model stability under uitra-high compression ratios, all without requiring any data.
3

---

(a) Overview of UltraDelta
Quanieed
Prined and Rescoled
Pree-Norma-Cuider
Rescaling
MartE) < <artE <.adg
Escinated Norm of Tink
Delu Weight Task
Distribution Aware Compression
Varse Sparsity Allocation
Weighe Yook
Lawe2
Singile Valie Decimposiion
Aiqurtiaiion (4 -el)
Laper3
Average Spanitty for Tank
Uniform Quantization
[160y]
Trace-Norm-Guile Guided Resealing
(b) Varinnce- Bused Mixed Sparsity Allocation
Distribution-Awar Compression
Loyers
Furgh
33,35
Liver
Goop
Goog
Greep
Crugg
Augo
35,3%
Yyoe
33.356
Allocation
0.062
0.043
liper
Spursity Feetd
Kandon Pruning within group
0.017
0,117
0,002
0.043
0,043
0.017
Figure 2: (a) Overview of the UltraDelta pipeline. It first assigns layer- wise sparsity based variance, and performs uniform quantization followed by group-wise pruning, and finally applies trace-norm-guice norded rescaling. (b) Variance- Based Mixed Sparsity Allocation allocates lower sparsity to high-variance layers to preserve information.
(c) Distribution- Aware Compression groups quantized parameters by value and prunes within each group to retain distribution.
(d) Frace-Norm-Ge Soidd- Rescaling estimates a rescaling factor using the trace norm of each delta weight to enhance robustness.
3 Method
3.1 Preliminaries
Consider a set of fine-tuned models that are fine-tuned from the same pretrained model. Let the weights of these models be denoted {0,, 02 ...,.0 (r). and the shared pretrained model weight as Opee. For each fine-tuned model & {1, 2, T). we define the delta weight as:
A0; = 0₂ 6pee
3.2 UltraDelta: Data-Free Pipeline for Ultra-Efficient Delta Compression
To address the limitations of existing delta compression methods, UltraDelta introduces a hybrid compression framework that minimizes parameter redundancy while maximizing information preser- vation and enhancing model stability across three dimensions: inter-layer, intra-layer, and global, enabling robustness under ultra-high compression. An overview of the overall pipeline illustrated We detail the core design of UltraDelta in Sec and provide analysis in Sec (.33
in Fig. 2
(1) Variance-Based Mixed Sparsity Allocation (MSA). Existing pruning methods often adopt uniform sparsity across layers, treating all parts of the model equally and overlooking their varying importance. However, different layers exhibit different sensitivities to pruning. We observe that layers with higher variance tend to carry more critical information and are more sensitive to pruning (see Sec. (3.3). Motivated by this, we propose MSA that categorizes all layers into three groups based on their variances (low, medium, high), each containing an equal number of parameters:
Groups = (Dow Gmid Gnigh
Each group G, is assigned a sparsity rate s, based on its variance v E (low, mid, high). with lower- variance groups assigned higher sparsity and higher-variance groups assigned lower sparsity, in order to preserve lnyer-layer information better:
8y Smid + S., 8, E {+Sspp 0, Sstep]
(3)
where 8mit is the average sparsity of the overall delta weight, and Ssep > controls the difference in sparsity across groups.

---

(2) Distribution-Aware Compression (DAC), Prior work has shown that preserving the shape of the weight distribution after compression helps maintain model performance [33]. To better preserve the intra-layer distribution shape under ultra-high compression, we propose DAC. Specifically, as delta weight exhibits a distribution well-suited uniform quantization [38], we first apply uniform quantization with lower bit width b (typically b = 4) to the delta weight. This quantization process consists of two steps: (a) mapping each value in A0; to a discrete integer within [0, 2 - 1]; (b)
mapping the quantized values back to the original value range. Let min min (0:), max max(A0, (A)),, and be the quantization step size:
od: round A+e min A
A + min, where A max min
(4)
After quantization, we group the parameters by their values and perform random pruning within each group. Let [u1, . ]n represent the unique values in the quantized delta weight At,. We denote (j, k) as the row and column indices of elements in A0, For each unique quantized value u,
we define a mask Mu, whose elements are independently sampled from a Bernoulli distribution with success probability St, where is the sparsity rate of the layer. Specifically:
Bernoulli(1 si), if A0, (J.A)
M (.)) M
"10
u4
(5)
otherwise
The pruned delta weight A0, is obtained by applying the group-wise Bernoulli masks the quantized delta weights:
Ad U. M..
(6)
DAC preserves the relative proportions of delta values, maintaining the intra-layer distribution shape. It minimizes distortion under high sparsity and ensures more stable performance. Moreover, DAC can be applied in uon-quantzedd settings with minimal modification, making it suitable for scenarios where aggressive compression not required. Instead of grouping by discrete values, we partition the weight range into several intervals, and assign parameters whose values fall within the same interval the same group. Details are provided in App. A
(3) Trace-Norm-Guider Rescaling (TNGR). As derived in Sec. 3.3 the variance of activation errors between compressed and original delta weights correlates with 8/ (1 8), causing instability at high sparsity. To address this, we introduce an additional rescaling factor " for task f, and redefine rescaling as 7 - st), where s, the overall sparsity of the delta weight for task We find that delta weights with larger trace norms (i.e., the sum of their singular values) tend to require smaller Y maintain stable performance. Motivated by this observation, we set 7 inversely proportional the trace norm of the delta weights:
1
Y
| l|ee
(7)
where ||rr denotes the trace norm. This trace-norm-guice rescaling offers simple yet effective way to adaptively stabilize pruning under extreme sparsity. In practice, "t typically falls within the range of [0.5, 1.0]. During the inference stage, the final model weight is reconstructed as:
gfnall
0pre +
7/2
A0,
(8)
1 1
3.3 Reason for Effectiveness
3.3.1 Theoretical Analysis of MSA
In lossless compression, the entropy represents the lower bound on the achievable average bit rate. A larger entropy means that layer carries more information and therefore should be more carefully preserved. We show that the variance of a layer closely related to its entropy, serving as the theoretical motivation for MSA. Following [50], model the distribution of a layer as a Gaussian Distribution L (N,, o²). The probability density function of L is given by:
12)
p(() v 2
exp
(9)
202
5

---

(a) 90% Sparsity Rate
99% Sparsity Rate
(c) Trace Norm for Each Task
100
00
(%6)
200
Performance
180
200
00
100
50
1,00
SUN2997
Beoweds
EursAr
Swi
ANST
DTD
Additional Rescaling
Additional Rescaling
SUN397
Cars
Resise45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Figure 3: Relationship between Trace Norm and Additional Rescaling Factor on 8 ViT-B/32. (a) Average performance under 90% sparsity with different additional rescaling factors. (b) Same as (a), under 99% sparsity. (c) Trace norm of delta weights across tasks.
The entropy of this distribution is (detailed derivation in App. B.1
(L) Ellog p()) log(o) log((
(10)
This demonstrates the link between variance and entropy: layers with larger variance have higher entropy, implying greater information content and thus requiring smaller sparsity to preserve it.
In lossy compression, entropy alone is not sufficient. When distortion is allowed, we employ dist-riito theory 66 [7]. which establishes the limit on how much information must be retained achieve given distortion level. Let L. denote the compressed version of L. With mean squared error (MSE) distortion, we define:
D= E J
(11)
According vate-distortion theory, the rate dite-disoo function of Gaussian source under MSE distortion (which is independent of the mean μ1) is given by:
log 0,
D<
R(D)
(12)
D2
This shows that, for fixed distortion D, larger variance o² leads to a higher rate R(D), meaning that more information must be retained. In other words, layers with larger variance require smaller sparsity in order to preserve their information under lossy compression.
3.3.2 Theoretical Analysis of the Rescaling Factor.
For given delta weight A0, let the activation be defined as AO O where z is the input feature. We introduce a Bernoulli mask ~ Bernoulli(1 - s), with sparsity rate s, and apply an additional scaling factor [0. 1]. Following (20), the activation error introduced by compression is:
A O
7
(B o Ad o 2)) a
- 1
(13)
We further derive the variance of this error (detailed derivation in App. B.2
72s a
Var(s)
(14)
As sparsity becomes higher, the variance grows rapidly and causes instability. This underscores the importance of using smaller rescaling factors to stabilize the model under high sparsity.
3.3.3 Empirical Analysis of TNGR.
We investigate how the additional rescaling factor 7 for each delta weight correlates with its intrinsic characteristics, shown in Fig. 3 At 90% sparsity, we observe that delta weights with larger trace norms tend to require smaller values of y to maintain stable performance. Moreover, their performance is more sensitive to changes in y. This phenomenon becomes even more pronounced at 99% sparsity, where the need for smaller 7 is more substantial for delta weights with large trace norms, and the performance drop-off becomes steeper when y is not well matched. These empirical trends support our trace-norm- trace-norm-guided guided heuristic estimation, which sets 7 E [0.5, 1.0) inversely proportional to the trace norm for adaptive and data-free rescaling.

---

3.3.4 Ability to Preserve Information.
To explain why UltraDelta achieves strong performance under ultra-high
[e] Layer-wise Cosine Similarity
Last-laye' Distribution
1.0
heal
compression ratios, we analyze how well it preserves the original model's
Cosine Similar ty
0.8
Delte coll 1200
Frequency
600
internal representations. Specifically, we compare the cosine similarity between the embeddings from the compressed and fine-tuned RoBERTa- base [54] models on the CoLA [83 dataset. We extract the layer-wise em- beddings of each input token and re- port the average cosine similarities. As shown in Fig. [4 our method consis- tently maintains high similarity across all layers, especially in the deeper ones that are more critical to performance. The distribution of cosine similarities in the final layer further confirms that UltraDelta retains essential information, indicating its effectiveness in preserving critical information despite ultra-high compression.
0.4
600
02
De CoNe CECO
200
0.0
0
0.4
0.6
10
Cosine Similarity
Figure 4: Cosine Similarity Analysis. (a) Layer-wise simi- larity of each layer's s embeddings. (b) Distribution of cosine similarity for the final-layer embeddings. Compression ratios for each method are indicated in parentheses.
4 Experiment
Baseline Methods. We compare UltraDelta with the following baselines: (1) Fine-tuned Models; (2) Multi-task Learning (MTL) [ which trains single model on all tasks; (3) BitDelta (53];
(4) Delta-CoMe (64;; (5) DARE [91] with either the same sparsity or similar compression ratio as UltraDelta, using pruning alone to achieve similar compression ratio; (6) Magnitude Pruning (MP) (27]. also matched by sparsity or compression ratio. Since BitDelta and Delta-CoMe target only linear layers in Transformer blocks, we restrict all methods to compress only linear layers for fairness. All original models are stored in FP16 format. Experiments are conducted on NVIDIA A800 GPU.
Compression Ratio. Unlike quantization, where compression is directly determined by bit-width, pruming methods require storing the indices of non- zerz values. To ensure a unified and fair evaluation, following 77 74 88]. we adopt Golomb coding (25 to store the compressed delta weights for
methods involving pruning, which is well-suited for encoding the zero-run lengths that approximately follow geometric distribution. Detailed derivation and calculations are App. [B.]] with results summarized in Tab. 8 Furthermore, for a complete evaluation, we present the ideal upper bound of compression ratio and practical results under different storage schemes in Sec. 4.5 storage scheme).
4.1 Performance on Large Language Models (LLMs)
Settings. We primarily evaluate Ultraeelta on the LLaMA-2 series with sizes of 7B and 13B across three types of models: math (WizardMath (56)). code (WizardCoder (57)) and chat (LLAMA-2- Chat (78).. The models are evaluated using GSM8K [15] for math (accuracy), HvmanEval [I] for code (pass@11 1), and TruthfulQA [49 for chat (accuracy). We also incorporate more recent L.Ms and
more challenging tasks: LLaMA-3, 1-1....1111. --Tulu-8B 8B [43 evaluated on MBPP+ 3 and HumanEval+
Swen2.5-7B-Insttren (89] evaluated on MATH B1 and GPQA 669, and Qwen3Guard-8B
evaluated on MMLU
30 and BBQ x
Details for the LLM IDs are presented in App. C.I
Results. We first report results the LLaMA-2 series (see Tab. I. For the 7B model, we apply 4-bit quantization and prune 95% of parameters, achieving an 32.9x compression ratio; for the 13B model, we prune 97% to reach 50.9 x compression. Despite these ultra-high compression ratios, UltraDelta achieves average scores of 45.57 (7B) and 52.05 (13B), exceeding the fine-tuned models (45.37 and 50.94, respectively). This improvement suggests that UltraDelta may introduce a regularization benefit. Compared to BitDelta (53 and Malae (64) which are limited to 16x compression, Ultraeelta delivers both higher compression and better performance. Other approaches such as DARE 191 and MP (27] fall short even at lower compression ratios, and when matched to the similar compression ratio as UltraDelta, their performance drops sharply. For newer LLMS (see Tab. 2 trade-off across baselines. In summary, Ultraeeta consistently outperforms all baselines across various tasks and model sizes, demonstrating its remarkable robustness and effectiveness under ultra-high compression.
UltraDelta achieves the best overall cammpression-perfrrmaa
7

---

Table 1: Performance comparison on large language models across 3 tasks. Ratio denotes the compression ratio (original model size compressed size). Methods marked with "(Sparsity)" use the same sparsity rate ours, while those marked with "(Compression)" match a similar overall compression ratio as our method. Best results are highlighted in bold.
higher sparsity rate to
Ratio
WathrdMath- V1.0
WizardCoder-VVu 1.0
LLAMA-2-Chat
Avg
Method
7B
13B
Individual
41.55
47.31
49.40
58.50
45.17
47
45.37
50.94
BitDela 53
16x
16x
36.54
46.12
46.30
54.30
42.84
44.85
41.89
48.42
Delm-CoMe 84
16x
16x
37.25
46.87
45.30
53.70
45.02
45.57
42.52
48.71
DARE 91
(Sparsity)
14.7x
23.7x
36.62
46.70
46.20
53.90
33.54
36.29
38.79
45.63
MP 27 Sparsity)
14.7x
23.7x
23.65
3.11
45,70
47.60
40.02
38.25
36.46
29.65
DARE 01 (Compression)
31.7x
50.1x
29.49
35.63
43.30
51.80
29.13
33.25
33.97
40.23
MP 27 (Compression
31.7x
50.1x
16.22
8.42
32,90
21.15
27.02
15.36
UltraDelta (Ours)
32.9x
50.9%
38.76
48.45
51.80
59.10
46.14
48.59
45.57
52.05
Table 2: Performance comparison on recent large language models.
-lama-3.1-Tulu-33. 3-8B
Bren2.5-7B-Instruee
Qwen
Method
Ratio
Avg
HumanFval+
MBPP+
MATH
GPQA
MMLU
BBQ
Individual
1x
57.30
56.60
36.20
36.87
72.80
50.34
$1.69
BitDelta
53
16x
55.50
52.90
38.10
37.37
73.05
49.60
$1.09
Delta-CoMe
64
16x
53.80
53.10
29.45
31.43
71.46
44.61
47.31
DARE 91
(Sparsity)
14.7x
50.60
50.00
37.76
34.85
72.18
49.69
49.18
MP 27 (Sparsity)
14.7x
45.10
40.20
11.84
31.82
70.91
50.18
41.68
UltraDelta (Ours)
32.9
56.10
54.50
40.14
38.38
73.44
49.74
52.05
4.2 Performance general NLP models
Settings. Following 76 91], we evaluate both TS-base
67 and RoBERTa-bass
54
models on
the GLUE (81) benchmark, covering CoLAA [83] SST-2
71.. MRPC (1]. STS-B 10], QQP [36].
MNLI 84), QNLI (68), and RTE [24]. For T5-base, we report Spearman's & on STS-B - and accuracy on the other tasks. Settings and results of RoBERTa-base are provided in App. C.3
Results. The results for T5-base models are presented in Tab. 3 Using 4-bit quantization combined with 99.5% pruning. UltraDelta achieves an impressive 224.6x compression ratio. Despite this extreme compression, UltraDelta attains an average accuracy of 86.74. outperforming all existing compression baselines and even slightly exceeding the full fine-tuned model's 's accuracy of 86.37. Compared to BitDelta [5X] and Delta-CoMe 664, UltraDelta achieves a much higher compression ratio while still maintaining superior performance. This demonstrates that UltraDelta successfully pushes the compression limits of delta weights while preserving model performance, highlighting its effectiveness and strong generalization ability.
Table 3: Performance comparison on T5-base models across 8 tasks.
Methods
Ratio
CoLA
SST2
MRPC
STSB
QQP
MNLI
QNLI
RTF
Avg
Individual
74.98
93.58
87.50
88.70
85.37
83.41
91.49
85.92
86.37
BitDelta
16x
70.09
93.46
84.06
86.20
85.26
83.64
90.94
83.75
84.68
74.26
93.58
87.94
85.44
83.51
91.54
84.36
85.95
DARE (Sparsity)
127.6x
74.16
93.34
87.83
87.86
85.22
83.19
91.52
84.11
MP 27 (Sparsity)
127.6x
37.87
88.76
70.59
80.4
68.55
82.88
82.67
63.97
DARE 91 (Compression)
220.5x
71.57
92.25
85.03
86.77
84.26
81.16
90.50
82.81
84,30
MP 27
(Compression)
220.5x
30.87
87.84
70.83
79.16
36.36
76.31
82.17
60.44
UltraDelta (Ours)
224.6x
76.51
93.81
88.48
88.82
85.41
83.17
91.76
85.92
86.74
4.3
Performance on Vision Models
Settings. Following 34
we evaluate VIT-B/32 and ViT-L/14 [66] models eight image classi-
fication datasets: SUN397 (87, Stanford Cars 4D), RESISC45 [12, FuroSAT (2)] SVHN (60].
GTSRB 73). MNIST 44 and DTD [3] Accuracy is used as the evaluation metric for all datasets.

---

The results of ViT-L/14 models are shown in Tab. while ViT-B/32 results are provided in With 4-bit quantization and 99% sparsity, UltraDelta achieves 132.5 compression ratio while maintaining an average accuracy of 94.4, matching the full fine-tuned model and significantly outperforming all compressed baselines, demonstrating its effectiveness on vision models.
Results.
App. C.2
Table 4: Performance comparison ViT-L/14 models across tasks.
Methods
Ratio
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Arg
Individual
1x
84.8
92.3
97.4
99.7
98.1
99.2
99.7
84.1
94.4
Traditional
96.3
BiDella 53
16x
84.0
92.1
97.2
99.7
97.9
99.0
99.7
83.0
94.1
Delta-CoMe 64
16x
84.3
92.1
97.2
99.6
98.0
99.1
99.7
83.6
94.2
(parsity)
66.5x
93.6
MP
27
Sparsity)
66.5x
26.7
43.9
83.4
92.8
24.4
62.9
77.7
66.1
59.7
DARE (Compression)
127.6x
99.6
97.7
99.2
89.1
MP
22
(Compression)
127.6x
4,9
4.2
42,6
83.2
6.4
16.7
27.2
50.7
29.5
UltraDelta (Ours)
132.5x
84.1
92.2
97.4
99.8
98.1
99.2
99.8
84.3
94.4
4.4
Performance on multi-modal models
Settings. We compress delta weights on BEIT-3 82 models fine-tuned on three datasets: VQA 26
(Visual Question Answering), NLVR2 [75] (Visual Reasoning), and COCO Captioning [51] (Image Captioning). The COCO Captioning task is evaluated using BLEU4 [6]] CIDEr 771,, METEOR [4] and ROUGE-L
48 while the other two tasks are evaluated based on accuracy.
The results of BEiT-3 models are shown in Tab. With 4-bit quantization and 90% sparsity, UltraDeita achieves a 18.4x compression ratio maintaining high accuracy and quality, outperforming all baselines across all tasks, demonstrating its effectiveness on multi-modal models.
Results.
Table 5: Performance comparison on multi-modal BEiT-3 models across vision-language tasks.
Task Metric
Ratio a(*)
COCO-Captoniing CIDER(t)
NLVR2 Accuracy(")
VQAv2 Accuracy(1)
Methods
BLEU4(()
METEOR(")
ROUGE-L(T)
Individual
0.394
1.337
0.331
84.269
BitDelta 53
16x
0.367
1,250
0.294
0.581
82.317
74.59
Delta-CoMe 64
16x
0.224
0.846
0.251
0.488
60.098
38.64
DARE 91 (Sparsity)
7.7x
0.333
1.15
0.285
0.564
80.652
73.29
(Sparsity
7.7x
0.006
0.014
49.263
DARE 91
(Compression)
18.1x
0.178
0.589
0.196
0.441
69.470
58.89
MP
20
(Compression)
18.1x
0
0.006
0.013
48.931
0.01
UltraDelta (Ours)
18.4x
0.372
1.270
0.296
0.583
83.163
75.66
4.5 Ablation Study
Effectiveness of Each Component. We evaluate the contri- bution each component on VIT-B/32 Tab. 6 DAC substantially improves compression ratio from 23.7x to 50.9x, all while preserving model accuracy. MSA yields the largest accuracy improvement, highlighting the effectiveness of mixed sparsity allocation. Together, they enable ultra-high compres- sion with strong performance.
quaritation
DARE quaribationn
DARE quentiaation
DAC quanization (Dural
DaC (Al quamiaaton rourtt
Effectiveness of DAC. To further validate the robustness of DAC, we evaluate it large-scale benchmark (32) of 30 ViT-B/16 models across 4-bit, 8-bit, and non-quantized config- urations. For fair comparison, to both methods in quantized cases. As shown in Fig, 5 (de- tailed results in App. C.4.I) DAC consistently outperforms
DaC (Alo ownntorr ond Durt
02
0.8
65
Sparshy
apply uniform quantization
Figure 5: Average Performance of DAC and DARE 30 ViT-B/16.
DARE [9T] across all configurations. Notably, under 90% sparsity and 4-bit quantization, DAC still exceeds DARE's non-quantized performance, confirming its robustness across diverse tasks.

---

Table 6: Ablation study the effectiveness of each component.
Methoes
SUN397
Cars
RESISC45
EurošAT
SVHN
GTSRB
MNIST
DTD
Ratio
Avg
[1] (97% Spanity)
23.7x
DAC
75.6
73.3
95.3
99.5
97.1
98.5
99.6
78.5
50.9% (1 27.2x)
89.7
DAC MSA
76.9
75.9
95.5
99.2
97.0
98.8
99.7
79.2
50.9%
90.3 (f 0.6)
DAC MSA TNGR
78.6
77.4
95.7
99.4
98.7
99.7
79.0
50,9x
90.7 (144
Table 7: Compression ratios across storage formats and models.
Format
LLaMA-2-7B
LLaMA-2-13B
VIT-B/32
VIT-L/14
T5-base
RoBERTa-base
BETT-3
Index-Free Storage
80.0x
133.0x
133.0x
400.0x
800.0x
80.0x
40.0x
Golomb Coding
32.9x
50.9
50.9%
132.5x
224.6x
32.9x
18.4x
CSR
19.5x
30.7x
35.4x
105.2x
188.3x
21.4x
10.8x
BCSR
12.7x
19.8x
21.7x
63.7x
121.4x
13.4x
7.3x
(a) Impact quantization bits
Impact sparsity steps
Impact of Tlyper-Parameters. We
Average Accurscy 196)
Average Accuraty 1%)
evaluate key hyper-parameters in DAC and MSA 8 ViT-B/32 mod-
els, as shown in Fig. (detailed re-
sults are in App. C.4.2). 2. For bit- width in quantization (sce Eq. 4. accuracy improves notably from 2-bit 4-bit, with marginal gains beyond, suggest- ing 4-bit as an optimal balance. For sparsity step Sstep in MSA (see Eq. 3 evaluate it under fixed 97% get sparsity and 4-bit quantization. A moderate step size (within [0.01, 0.02)) yields the best average performance. Performance remains stable as long as extremely large or small step sizes are avoided. This indicates that precise tuning of Sstep is generally unnecessary.
248
4-2t
6-tt:
0.0
0.005 0.01 0,015 0.22 6/25 0.03
Quanttization bilts (bit)
Soarsity Stop
UltraDelta (50x Oursi wee. Fine tunes Models .... Delta.Cofte (16i) ... DARE (501)
Figure 6: Impact of hyper-parameter on 8 ViT-B/32. (a)
Impact of quantization bits; (b) Impact of sparsity steps.
Storage Scheme. We report compression ratios of UltraDelta under different storage schemes. Index- Free Storage denotes the ideal upper bound where only non-zero values are stored without indices (the ideal compression ratios of other pruning baselines in Tab. [8. We also evaluate practical schemes, including Golomb coding, Compressed Sparse Row (CSR), and Block Compressed Sparse Row (BCSR), As shown in Tab. 7 BCSR performs poorly, only works when zeros are densely clustered and thus is not suitable for our setting.
5 Discussion and Conclusion
Regularization Effect. By controlling the fitting degree of models, we examine that UltraDelta particularly benefits underfitted models (see App. for detailed results and analysis). Since LLMs are typically evaluated in the zero-shot or few-shot settings, where underfitting is common, this explains why UltraDelta often yields not only strong compression but also performance gains.
Overhead. UltraDelta introduces only modest computational overhead during compression and is efficient at inference compared with baselines. The main cost arises from computing trace norms. Since only their relative magnitudes are required, fast approximation such randomized SVD can be employed to reduce the overhead. In the inference stage, delta weights only need to be dequantized and added back the base model, which can be done on CPU efficiently. In contrast, methods such [64] require additional matrix multiplications, making them less efficient.
as
Limitations and Future Work. We acknowledge that our method employs heuristic rescaling strategy and does not explicitly target deployment efficiency; detailed discussion is in App. D33
Conclusion. In this paper, we present UltraDelta, the first data-free pipeline for ultra-efficient delta compression, capable of achieving both ultra-high compression ratios and strong performance. Ultraeelta is hybrid compression method with a focus preserving information across inter- layer, lntra- and global dimensions through three novel components. Extensive experiments on language, vision, and multi-modal models, along with theoretical analysis, validate the effectiveness, robustness, and generality of UltraDelta across diverse settings.
10

---

6 Acknowledgement
This work is supported by National Key Research and Development Program of China (No. 20222D0160101), Shanghai Natural Science Foundation (No. E3ZR1402900) Shanghai Science and Technology Commission Explorer Program Project (24TS1401300) Shanghai Municipal Science and Technology Major Project performed using the CFFF platform of Fudan University.
The computations in this research were
References
[1] M Israk Ahmed, Shahriyar Mahmud Mamun, and Asif Uz Zaman Asif. Denn-based vegetable image classification using transfer learning: A comparative study. In 2021 5th International Conference on Computer. Communication and Signal Processing (ICCCSP), pages 235-243. IEEE, 2021.
[2] Sarder lftekhar Ahmed, Muhammad Ihrahim, Nadim, Md Mizanur Rahman, Maria Mchjabin Shejunti, Taskeed Jahid, and Md Sawkat Ali. Mangoleafhd: A comprehensive image dataset to classify diseased and healthy mango leaves. Data in Brief, 47:108941, 2023.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.c 1080:2208.07732. 2021.
[4] Satanjecv Banerjee and Alon Lavic. Meteor: An automatic metrie for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or sunmarization, pages 65-72, 2005.
[5] Puneet Bansal. Intel image classification. Available on https://www. kaggle. commueeedodddddd00iiimimmmmmeee classification, Online, 2019.
[6] Toby Berger. Rate- dite-iitootion theory. Wiley Encyclopedia of Telecommumications, 2003.
[7] Lukas Bossard, Matthicu Guillaumin, and Lue Van Gool. Food-10 mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
[8] Rich Caruana. Multitask learning. Machine learning, 28:41 77, 1997.
[9] CCHANG. Garbage classification. httpps:/ http:/gglll.oooo/////////999///9 2018
[10] Daniel Cer, Mona Diab, Encko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similariy-muliiiinnual and cross-lingual focused evaluation. arXiv preprint 10XXv:1708.00005:: 2017.
[11] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2 2021.
[12] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedingy of the IEEE, 105(10)1865--188 2017.
[13] Mircea Cimpoi. Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3606-3613, 2014.
[14] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings. 2011.
[15] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[16] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist handwritten letters. 2017 international joint conference on neural networks (ICNN), pages 2921-2926. IEEE, 2017.
[17] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[18] Will Cukierski. Dogs vs. cats, 2013, URL https:/kaagle. com/ccopettiioonsllssssssss
11

---

[19] DeepNets. Landscape recognition. betps://wwy
https::/ww..agggee.coomm/// htggg///.ooo/aaaaaaee////aaaaaan Maddscpe-recoomiiionn- -tw.omittiiii-iiagaddttttsstttt -ataset---- 1er-11kkimmpp images
[20] Wenlong Deng, Yize Zhao, Vala Vakilian, Minghui Chen, Xiaoxiao Li, and Christos Thrampoulidis. Dare the extreme: Revisiting delta-parameter pruning for fine-tuned models. arXiv preprint arXiv:2410009344 2024.
[21] Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (/WP2005), 2005.
[22] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post- post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17323, 2022.
[23] Leo Gao, Jonathan Tow. Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li. Kyle McDonell, Niklas Muennighoff, Chris Ociepa. Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 2024.
[24] Danilo Giampiccolo, Bernrdo Magnini, Ido Dagan, and William B Dolan. The third pascal recogniz- ing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop textual entailment and paraphrasing. pages 1, 2007.
[25] Solomon Golomb. Run-length encodings (corresp.). IEEE transactions on information theory, 12(3): 399-401, 1966.
[26] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017.
[27] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 1rX 2015.
[28] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[29] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth, Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
[30] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[31] Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.
[32] Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-free high-performance model merging. Advances in Neural Information Processing Systems, 37:122741-122769, 2024.
[33] Chenyu Huang, Peng Ye, Xiaohui Wang, Shenghe Zheng, Biqing Qi, Lei Bai, Wanli Ouyang, and Tao Chen. Seeing delta parameters jpeg images: Data-free delta compression with discrete cosine transform. arXiv preprint arXiv:2503.064 06676, 2025.
[34] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04088 2022.
[35] Berivan Isik, Hermann Kumbong, Wanyi Ning, Xiaozhe Yao, Sanmi Koyejo, and Ce Zhang. Gpt-zip: Deep compression of finetuned large language models. In Workshop on Efficient Systems for Foundation Models @ ICMI.2023, 2023.
[36] Shankar Iyer, Nikhil Dandekar, Kornél Csernai, al. First quora dataset release: Question pairs. data. quora. com. 2017.
[37] Mona Jalal, Kaihong Wang. Sankara Jefferson, Yi Zheng, Elaine O Nsoesie, and Margrit Betke. Scraping social media photos posted in kenya and elsewhere to detect and analyze food types. In Proceedings of the 5th International Workshop on Multimedia Assisted Dietary Management, pages 50 59, 2019.
12

---

[38] Yanfeng Jiang, Zelan Yang, Bohua Chen, Shen Li, Yong Li, and Tao Li. Deltadq: Ultra-high delta com- pression for fine-tuned I1ms via group-wise dropout and separate quantization. arXiv preprint arXiv:2410.08666. 2024.
[39] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, 2011.
[40] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- ine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554 004-561,22 2013.
[41] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[42] Makerere Al Lab. Bean disease dataset, 2020.
[43] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang. Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Vietoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah Smith, Yizhong Wang. Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers open language model post-training. 2024.
[44] Yann LeCun. The mnist database of handwritten digits. http://yamm. lecun. com/exdb/mmist/ 1998.
[45] Jacho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. arXiv preprint arXiv:2010.0611, 2020
[46] Guiying Li. Chao Qian, Chunhui Jiang, Xiaofen Lu, and Ke Tang. Optimization based layer- wise magnitude- based pruning for dnn compression. In IJCAI, pages
2018.
[47] Wei Li, Lujun Li, Mark Lee, and Shengjie Sun. Adaptive layer sparsity for large language models via activation correlation assessment. Advances in Neural Information Processing Systems, 37: 17:109950-109380 2024.
[48] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[49] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.0793 2021.
[50] Sihao Lin, Pumeng Lyu. Dongrui Liu, Tao Tang, Xiaodan Liang. Andy Song, and Xiaojun Chang, Mlp can he good transformer learner. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19489-19498, 2024.
[51] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 750-755. Springer. 2014.
[52] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[53] James Liu, Guangxuan Xiao, Kai Li, Jason D Lee. Song Han, Tri Dao, and Tianle Cai. Bitdelta: Your fine-tune may only he worth one bit. Advances in Neural Information Processing Systems, 37:13579-13600, 2024.
[54] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:190..11692 2019.
[55] Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. Advances in neural information processing systems, 37:9117-9152, 2024.
[56] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin. Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv: 09583, 2023.
13

---

[57] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng. Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct. arXiv preprint urXiv:2306.08855 2023.
[S8] Alexander Mamaev. flowers- - rew.gniiooiiooo
Flowers recognition.
http:///ww..aaglee..m/m///taaaaa xxxamaee
[59] Horea Muresan and Mihai Oltean, Fruit recognition from images using deep learning. Acta Universitatis Sapientiae, Informatcu, 10(1):26-42, 2018.
[60] Yuval Neizer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, page 4. Granada, 2011.
[61] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguisticy, pages 311-318, 2002.
[62] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[63] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. Bbq: A hand-built bias benchmark for question answering. arXiv preprint 10.08/9,, 2021.
[64] Bowen Ping, Shuo Wang, Hanqing Wang, Xu Han. Yuzhuang Xu, Yukun Yan, Yun Chen, Baobao Chang, Zhiyuan Liu, and Maosong Sun. Delta-come: Training-free delta-compressson with -eeccdppeeiioon for large language models. Advances in Neural Information Processing Systems, 37:3 37:31056-31077, 2024.
[65] Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrum Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Due-Tien Dang- Ngugen, Mathias Lux, Peter Thelin Schmidt, et al. Kvasir: A multi-class image dataset for computer aided gastrointestinal discase detection. In Proceedings of the 8th ACM on Multimedia Systems Conference, pages 164-169, 2017.
[66] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmL.R 2021.
[67] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 1167 2020.
[68] Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint orxiv:/606.002500 2016.
[69] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
[70] Felix Sattler, Simon Wiedemann, Klaus-Roberr Robert Müller, and Wojeiech Samek. Sparse binary compression: Towards distributed deep learning with minimal communication. In 2019 International Joint Conference on Neural Networks (IJCN)), pages 1-8. IEFF, 2019.
[71] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[72] Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurhishing unclean samples for robust deep learning. In International conference machine learning, pages 5907-5915. PMIR, 2019.
[73] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 2011 international joint conference on neural networks, pages 1453-1460. IEEE, 2011.
[74] Nikko Strom. Scalable distributed dan training using commodity gpu cloud computing. In Interspeech 2015, pages 1488- 1492, 2015.
14

---

[75] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang. Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. arXiv preprint urXiv:1111.00991 2018.
[76] Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Fusionbench: A comprehensive benchmark of deep model fusion. arXiv preprint (2d.::200000228000 2024.
[77] Qwen Team. Qwen3guard technical report. 2025.
[78] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307, 0028888800.8... 2023.
[79] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4566-45755 2015.
[80] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-uesd birds-200-2011 dataset. 2011.
[81] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 3XXiv:880.00460 2018.
[82] Wenhui Wang. Hangho Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as foreign language: BEiT pretraining for vision and vision-language tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[83] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 0451, 2019.
[84] Adina Williams, Nikita Nangia, and Samuel Bowman. A coveraee challenge corpus for sentence understanding through inference. arXiv preprint
2017.
[85] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07744 2017.
[86] Haixia Xiao. Feng Zhang, Zhongping Shen, Kun Wu, and Jinglin Zhang, Classification of weather phenomenon from images by using deep convolutional neural network. Earth and Space Science, 8(5): c2020EA001604, 2021.
[87] Jianxiong Xiao, James Hays, Krista A Ehinger. Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abhey to zoo. In 2010 IEFF. computer society conference computer vision and pattern recognition, pages 3485-3492. IEFF, 2010.
[88] Prateek Yadav, Leshem Choshen, Colin Raffel, and Mohit Bansal. Compeft: Compression for commu- nicating parameter efficient updates via sparsification and quantization. arXiv preprint arXiv:2311.13171, 2023.
[89] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang. Guanting Dong. Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu. Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng. Rui Men. Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren. Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, 7eyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint crXiv:2407.1067) 2024.
[90] Xiaozhe Yao, Qinghao Hu, and Ana Klimovic. Deltazip: Efficient serving of multiple full-modelltuneed 1ms. In Proceedings of the Twenfieth Europeon Conference on Computer Systems, pages 110-127, 2025.
[91] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024.
[92] Neizer Yuval. Reading digits in natural images with unsupervised feature learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[93] Shenghe Zheng and Hongzhi Wang. Free-merging: Fourier transform for model merging with lightweight experts, arXiv preprint arXiv.2411.16815. 2024.
15

---

Appendix for UltraDelta
A
Non-quantized Variant of Distribution-Aware Compression
While our main Distribution-Aware Compression operates on quantized delta weights, can also be adapted non-quantized variant for scenarios where quantization is not applied not desired. In this version, we replace the discrete grouping based on quantized values with continuous interval- based grouping over the full-precision delta weights.
Specifically, for a given delta weight A0, we first compute its minimum and maximum values, denoted min)) and max (A.)). We then divide the range (min(A01), (ax()A:)) into I equally spaced intervals. The width each interval is given by:
Interval max max(A0)) - min Ab.)
(15)
I
The i-th interval is then defined as:
Ii [min(A0.) (i - 1) Sinterval, min( (20,) + i. dinervat)], for I,..., (16)
To perform group-wise pruning, we define a binary mask for each interval I,, where pruning decisions and st is the sparsity rate
are sampled from Bernoulli distribution with success probability of the layer. Specifically:
Bernoulli(1 st), if A0, (,,) E 1
M
(J.) M
(17)
0,
otherwise
The pruned delta weight A0, is obtained by applying the group-wise Bernoulli masks to the quantized delta weights:
A0 Σ M I A0.
(18)
B Detailed Derivation
B.1 Derivation of the Variance-Entropy Relationship
We aim to derive the relationship between the variance and the entropy of a layer. First, we define the entropy random variable L with probability density function p(l) as:
H(L) = -Elllog p())
(19)
Assuming I. follows Gaussian distribution ((,, its probability density function is:
(1 11)²
p(()
exp
(20)
V2TT²
202
Taking the logarithm of the density function, we obtain:
(- [1- -1)²
))
(1 - 1)²
log p()) log (22T22 exp
(og(2mo²)
(21)
20
2
Substituting this into the definition of entropy, we have:
(-1 log(2TT0)
(1 - (1)= dl
H(L)
p(1)
20
1
/ ((()dl=
1
/ (D()(11 -1))
2 log(2TT?)
(22)
20
16

---

Next, we use known properties of the Gaussian distribution:
f p(()all = and
| 1) Var(L) o²
(23)
Substituting these into the expression for entropy yields:
1
HLL) log(2TT + 2
1
1og(2 + log(o) +
(24)
Therefore, we can conclude that the entropy (I.) increases logarithmically with the standard deviation of the distribution. Since the variance is x², entropy is monotonically increasing with variance. This derivation shows that the entropy of layer is positively correlated with its variance. Since entropy represents the amount of information, layers with larger variance contain more information. As result, such layers should be assigned a smaller sparsity rate to preserve more information.
B.2 Derivation of the Variance of Activation Error
According to Eq.
B the activation error is defined as:
== a @ 1 1- - B):
where = A
(25)
Here, B is Bernoulli random variable, where the probability of success is 1 - s. For Bernoulli distribution, we know that the expectation and second moment are E[ s and E B We first calculate the expectation of e. Using the linearity of expectation:
1 8.
7
Eie] ④
- s
=a0
7
E[B]
1 - 8
(26)
7 (1 8)
=a0
8
=a (1-7)
Next, we calculate the expectation of c::
27
B| = d²
1
2
7 B
7
B
8
1-
2
=
1
7
E[B]
?
E[B]
8
8
(27)
2
= 0
1 2
7
(1
8) +
7
(1 8)
8
1-8 s
22
o 0
1 2
Now, we can calculate the variance of e. The variance given by:
Var(s)
E (B()))²
22
0²
1 2 +
0² G 71-
1-8 8
22
(28)
a² 3
1 2y +
(1 (1-21-17 +22)
1
2
2 0
1 1-8
17

---

From the derived variance expression, we can observe that when the sparsity s is high, the variance will become very large, leading unstable performance. To mitigate this, we can choose value for in the range [0, 1] to reduce the impact of the original rescaling factor. However, y cannot be chosen too small. This because the expectation of the activation error the expectation of the activation error increases, which may lead to undesirable effects. Therefore, typically set in the range [0.5, 1.0] under high sparsity, which balances both the variance and the expectation of the activation error.
④ (1 - ?), and if vy is too small,
B.3 Detailed Derivation of Compression Ratio
Golomb Coding. Golomb coding 25 is run-length entropy coding scheme particularly well-suited for geometric distribution, where encodes the distances between successive non-zero entries (i.e., run-lengths of zeros). Unlike conventional formats such as Compressed Sparse Row (CSR), which store explicit indices or pointers for non-zero values, run-length coding represents zero runs instead of storing indices, which can yield a more compact representation than CSR when sparsity is high.
Compression Ratio Derivation. In setting, after pruning, most entries in the delta weight zeros, and the positions non-zeros can be modeled as a Bernoulli process with success probability where is the sparsity. Consequently, the run-lengths of zeros follow geometric distribution, for which Golomb coding achieves an average code length close to the entropy of the underlying geometric distribution. The entropy of geometric distribution is:
equal the density
Hgso = -(1 - (1 - s) lloge
(29)
When each non-zero entry can take m discrete values, the maximum entropy contribution of its symbol is log, m, which corresponds to the case where all m values are equally likely. In practice, the values of our non-zero entries follow truncated Gaussian distribution, so the actual entropy is smaller than log, m. implying that the achievable compression ratio is higher.
For simplicity of analysis, we approximate this term by log, m. which serves an upper bound, and the per-parameter entropy of the compressed model is:
H comp
- slog2 + (1 5) log2
(1-s) m
(30)
By contrast, the original uncompressed model stored in FP16 format requires Horig 16 bits per parameter. Therefore, the thcoretical compression ratio is:
Hong
16
CR
(31)
Heomp
logg (1 log. (1=s)
Note that the rescaling factor is represented by a single 16-bit value, which is negligible compared to the overall storage and is therefore ignored when reporting the compression ratio.
Detailed Compression Ratios. We also report the compression ratios and corresponding sparsity rates of DARE [91]. MP (27]. and UltraDelta under each setting. Practical compression ratios with Golomb coding are computed following Eq. 31 using m 16 for DARE and MP (16-bit precision) and = 4 for UltraDelta (4-bit quantization). For completeness, we also include Index-Free compression ratios, which denote the ideal upper bound where only non-zero values are stored. The results are shown in Tab.
C Additional Experimental Settings and Results
C.1 HuggingFace IDs for LLM Checkpoints
We provide the Hugging Face IDs of the LLMs used in our experiments, as shown in Tab. 9 All models are evaluated using the Im-evaluation-hare EvalPlus [52) framework for code generation tasks.
23 framework for general language tasks and
18

---

Table 8: Compression ratios, average performance, and corresponding sparsity rates across different models. "Practical" denotes the practical compression ratio using Golomb coding, "Ideal" the ideal upper bound, and "Avg" the average performance.
LLaMA-2-7B (Tab.
LLaMA-2-13BB (Tab. T
Method
Practical
Ideal
Avg
Sparsity
Practical
Ideal
Avg
Sparsity
DARE 91
(Spursity)
14.7x
20x
38.79
95.0%
23.7x
33.3x
45.63
97.0%
MP 27
SSprritty
14.7x
20x
36.46
95.0%
23.7x
33.3x
29.65
97.0%
DARE 91 (Compression)
31.7x
4
33.97
97.8%
50.1
74.1x
40.23
98.65%
MP
27
(Compression)
31.7x
45.5x
27.02
97.8%
50.1
74.1x
15.36
98.65%
Uelra (Ours)
32.9
80.0x
45.57
95.0%
50.9
133.0x
52.05
97.0%
Recent LLMs (Tab,
2
VIT-B/32 (Tab.
10
Method
Pructical
Ideal
Avg
Sparsity
Practical
Ideal
Avg
Sparsity
DARE 91 (Sparsity)
14.7x
20x
49.18
95 35,0%
2
33.3x
89.7
97.0%
MP 2 (Sp Sparsity)
14.7x
20x
41.68
95.0%
23.7x
33.3x
77.4
97.0%
DARE 91 (Compression)
50.1x
74.1x
83.5
98.65%
MP (Compression)
50.1x
74.1x
98.65%
UltraDetaa (Ours)
32.9
80.0x
52.05
95.0%
50.9
133.0x
90.7
97.0%
VIT-L/14 (Tab,
TS-base (Tab,
Method
Practical
Ideal
Avg
Sparsity
Practical
Ideal
Avg
Sparsity
DARE 91 (Sparsity)
66.5x
100.0x
93.6
99.0%
127.6x
200.0x
85.90
99.5%
MP 27
(Sparsity)
66.5x
100.0x
59.7
99.0%
127.6x
200.0x
63.97
99.5%
DARE 91
(Compression)
127.6x
200.0x
89.1
99.5%
220.5x
357.0x
84.30
99.72%
MP 27
(Compression)
127.6x
200.0
29.5
99.5%
220.5x
357.0x
60.44
99.72%
Uelta (Ours)
132.5x
400.0
94.4
99.0%
224.6x
800.0
86.74
99.5%
RoBERTa-base (Tab.
11
BETT-3 (Tab, 5
Method
Practical
Ideal
Avg
Sp Sparsity
Practical
Ideal
Avg
Sparsity
DARE 91 (Spursity)
14.7x
20.0x
83.23
95.0%
7.7x
10.0x
70.74
90.0%
MP 27 (Sparsity)
14.7x
20.0x
80.06
95.0%
7.7x
10.0x
16.59
90.0%
DARE 91 (Compression)
31.7x
45.5x
73.12
97.8%
18.1x
25.0x
54.49
96.0%
MP 27
(Compression)
31.7x
45.5x
69.87
97.8%
18.1x
25.0x
16.47
96.0%
Delta (Ours)
32.9x
80.0x
84.46
95.0%
18.4x
40.0
73.95
90.0%
C.2 Experiments on eight ViT-B/32 models
The results of ViT-B/32 models are shown in Tab. 10 With 4-bit quantization and 97% sparsity, UltraDelta achieves 50.9x compression ratio VIT-B/32. UltraDelta achieves an average accuracy of 90.7 across 8 diverse vision tasks, nearly matching the performance of individually fine-tuned models (91.0). Compared to all baselines, UltraDelta achieves both higher compression and better accuracy.
Results.
C.3 Experiments on eight RoBERTa-base models
Settings. For RoBERTa-base. we report Matthews correlation on CoLA, the average of Pearson and Spearman correlations STS-B, and accuracy the other tasks.
Results. The results of RoBERTa-base models are shown Tab. II For RoBERTa, we apply 4-bit quantization and prune 95% of parameters, achieving overall 32.9x compression ratio. Despite the high compression, UltraDelta achieves an average score of 84.46, closely matching the uncompressed individually fine-tuned model (85.56) and outperforming all baselines by notable margin.
19

---

Table 9: LLM checkpoint IDs.
Models
Pre-trained
Fine-tuned
Model Size 7B (Llama-2 Series)
Chat
meta- Paitt-llamaa/iaa lamm-22aaaaaa22--- 2--bhf
meta- haaie-lllammalaaaaa-------iii
Code
codellama/ Code 7b-Pythnn-
Math
meta Satit-lam//alamm-2-777
Model Size 13B (Ilama-2 Series)
Chat
meta-llamen Centu-llaaa/llamuss 2-1 3b-kf
Code
codellama/CodeLLone -a--aaamma-----P-leae 3b-Python-hff
Math
meta- Laata-llam/Lllmmm-222 2-1b-f-
vamillaOYo/Wiaardaahh-bbb
Recent Models
Llama-3,1
meta- Mlaaaaaaaaaaaaa.1....
allenai/Llama-3: ---uu-----------P-
Qwen2.5
Qwen/Owen2.5-4
Qwen/Qwwen.5-77
Qwen3
Qwen/Qwen3-8B
Queen/QwenSGuurd-GGrr
Table 10: Performance comparison on ViT-B/32 models across 8 tasks.
Methods
Ratio
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Arg
Individual
1x
79.2
77.7
96.1
99.8
97.4
98.7
99.7
79.4
91.0
Traditional MTL
8x
73.9
74.4
93.9
98.2
95.8
98.9
99.5
77.9
89.1
BitDelia
16x
78.3
75.9
96.4
98.2
Delta: CoMe 64
16x
78.5
72.3
95.6
99.6
97.3
98.6
99.5
78.3
90.0
DARE (Sparsity)
23.7%
76.0
73.3
95.1
99.7
97.0
98.5
99.6
78.1
89.7
MP 27
(Sparsity)
23.7%
58.1
44.2
91.2
96.1
91.3
73.5
98.3
66.7
77.4
DARE 01 (Compression)
50.1
57.2
55.2
91.3
99.4
96.4
98.2
99.6
71.1
83.5
MP
27
(Compression)
50.1
18.8
16.4
81.8
88.3
70.3
28.5
77.8
42.7
53.1
Uetra (Ours)
50.9%
78.6
77.4
95.7
99.4
97.0
98.7
99.7
79.0
90.7
C.4 Detailed Results of Ablation Study
C.4.1 Ablation on Distribution-Aware Compression
We present the detailed numerical results of the ablation study conducted on DAC. For brevity,
report only the results under 90% sparsity, as summarized in Tab. T2] Following [32 93], we use
ViT-B/16 the pre-trained pre-trained backbone model and evaluate its performance across 30 diverse image
classification datasets. Specifically, the datasets are: MNIST [44), CIFAR-I0 41], CIFAR-100 [41],
Cars (40) Mashion-MNIST (85), EMNIST [16] STLIO 14] GTSRB 73), SHVN 92 Oxford-IIIT
Pet (62) Cats and Dogs [18. Dogs 33 Beans [4). Food-101 77. Fruits-360 (59]. Vegetables I.
MangoLeafBD 2) Flowers Recognition 58 Landscape Recognition [9)) Weather [86), DTD [3
EuroSAT 29 RESISC45 [1]. SUN397 87,, KenyanFood13 (37, Intel Images [5], Garbage
Classification [9) Animal-101 [72), CUB-200-201 [8) and Kvasir-v2 (65] These datasets are
chosen to comprehensively evaluate generalization across various data distributions and classification
challenges. Accuracy is used as the evaluation metric for all datasets.
Table 11: Performance comparison on RoBERTa-base models across 8 tasks.
Methods
Ratio
CoLA
SST2
MRPC
STSB
QQP
MNLI
QNLI
RTE
Avg
Individual
1x
60.18
94.04
89.22
90.63
91.41
87.20
92.71
79.06
85.56
BitDelta 53
16x
36.83
93.12
88.73
84,82
90.00
85.57
91.73
70.40
80.15
Delta-CoMe
64
16x
57.67
92.48
87.99
90.53
89.17
82.66
92.57
76.9
83.74
DARE 91
(Sparsity)
14.7x
59.30
93.92
88,97
90.53
86.55
77.84
92.13
76.53
83.23
MP 27
(Sparsity)
14.7x
57.69
92,43
84.31
88. 18
86.26
74.40
86.5
70.40
80.06
DARE 91
(Compression)
31.7x
58.22
93.69
87,50
90.27
53.47
36.48
89.51
75.81
73.12
MP
(Compression)
31.7x
51.89
91.63
78.68
86.44
65.77
53.49
65.64
65.43
69.87
UltraDelta (Ours)
32.9
62.64
94.38
89,22
90.60
85.79
82.92
92.15
77.98
84.46
20

---

Table 12: Performance of DAC and DARE VIT-B/16 models across 30 tasks. "(w/ b-bit)" denotes
the quantized setting using b-bit precision, while "(w/o q.) refers to the non-quantized setting.
Methods
Asired INN
Cifa-10
Cila-100
CUB 208-200-001
EMNIST
BunDAT
Indiviiuel
5905
00.85
64.75
$2.01
81.1
91.67
DARE (orkle))
92.26
53.50
$4.25
85.35
$1.55
54,99
Ultratikia ond 4400 (fuer))
92442
W132
08.05
38.35
72.14
3316
19.34
10082
04.51
14.04
Litraliila B-bat (Quen)
$268
98447
$8.05
30.56
78.12
$4.54
90313
81.13
9461
99.00
36407
99:31
81111
04.38
Kreyssfood11
Food-101
Trate 101
Cartage
(17550
traa--nnagee
Kenstes2
Tandfcape
Tsliideel
95.36
$2.58
67.57
99.64
95.74
94.87
92.91
00
DARE (xi
99.6
7.57
95.33
14.81
Ulirstiiia ow 4460 (flue))
19.18
M6.17
$3.31
20.12
06.03
989 50
10.16
60-93
56.17
95.80
Litraliila I-tat) (Uuer))
M0.19
92499
47.00
99.39
05.30
9138
93.93
8934
73.80
g)
99.94
63.99
77.87
UltaDella (Oues)
Mangltammnn
MNIST
RESESC15
SUN
8VIN
VegroNes
Whater
Iadividal
100.00
M0.22
92.50
90.20
$5.29
99.08
$3.90
96.22
100.0
96.19
TARBS (13 burkarg
78.19
98.50
74.28
85.M4
96.35
91.67
Ulrnshika rwl 4-60 (fhur))
1545
91
$3.56
96.95
79.54
99.71
85387
96.31
91.30
B0 In
Litralelia l-ba (thert)
10300
92.37
90.11
15.94
96.31
99607
UltraPella (
1931
90.83
Intitteal
Ulraalett 4-in
DARE
Ultraleete (M000Ounn)
Uhrodela
Ave
$4.05
$1.58
$6.40
$3
90.18
$5.88
92.45
C.4.2 Ablation Hlyper-Parameet
present the detailed numerical results of our hyper-paramenn parameter ablation studies Tab. 13 The results complement the analysis provided in the main text and offer full visibility into the performance under different settings.
Table 13: Performance of different hyperparameter settings applied ViT-B/32.
Method SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg.
Distribution-Awarc Compression (quantization bit)
2-bit
73.4
70.1
94.4
99.1
96.9
98.2
99.7
74.3
88.3
4-bit
78.6
77.4
95.7
99.4
97.0
98.7
99.7
79.0
90.7
6-bit
78.7
77.5
95.7
99.5
97.1
98.8
99.7
78.6
90.7
8-bit
78.8
77.7
95.7
99.7
97.0
98.7
99.7
79.0
90.8
Variance-Baan Based Mixed Sparsity Allocation (sparsity step size)
Step = 0.000
77.3
75.1
95.5
99.7
97.1
98.8
99.6
77.6
90.1
Step = 0.005
77.9
75.7
95.4
99.7
97.0
98.8
99.7
78.9
90.4
Step = 0.010
78.4
76.7
95.6
99.4
97.0
98.9
99.6
78.9
90.6
Step = 0.020
78.6
77.4
95.7
99.4
97.0
98.7
99.7
79.0
90.7
Step = 0.025
78.3
76.4
95.1
98.1
96.6
98.6
99.7
79.2
90.3
Step = 0.030
78.5
75.3
94.5
96.5
96.7
98.2
99.7
78.7
89,8
D Discussion
D.1 Regularization Effect
We observe in experiments that UltraDelta sometimes even outperforms fine-tuned models, especially in the LLM setting. This can be attributed overfitting or underfitting during fine-tuning.
To verify this, we conducted controlled experiments on two datasets, SUN397 [7] and Cars [40] using a pretrained ViT-B/32 model. By varying the number of fine-tuning steps, we controlled the degree of model fitting. The results are shown in Tab. [4] and we find that underfitted models (early training steps, higher loss) consistently benefit from UltraDelta, while well-trained models (later steps, lower loss) show some or no improvement. These results indicate that UltraDelta mainly benefits underfitted models, while well-trained models already capture most task-specific information and thus leave little room for further improvement.
21

---

This also explains why improvements are more common in LLMs, which are typically evaluated in zero-shot or few-shot settings. such scenarios, these models are not explicitly fine-tuned on the target tasks, which often leads to weaker generalization. As result, they can be regarded as underfitted the evaluation benchmarks. this context, UltraDelta provides regularization effect, enabling underfitted models to benefit more from compression.
Table 14: Controlled experiments ViT-B/32 with varying training steps. "UltraDelta Accuracy" denotes the test accuracy after applying UltraDelta.
Training Steps
2000
3000
4000
5000
6000
7000
8000
9000
Cars [40)
Training Loss
0.1404
0.0834
0.0206
0.0069
0.0059
0.0059
0.0034
0.0077
0.0048
0.0036
Test Accuracy
72.86
75.22
78.78
77.74
78.37
78.01
75.21
77.60
77.34
UltraDelta Accurney
74.02
73.27
75.82
75.56
75.62
75.66
76.13
72.80
74.74
74.06
SUN397 (87
Training Loss
0.5475
0.1432
0.0426
0.1125
0.0526
0.0040
0.0018
0.0009
0.0006
0.0004
Test Accuracy
73.63
75.94
75.40
74.63
74.07
76.94
76.87
77.03
77.06
77.07
UltraDelta Accuracy
76.02
76.92
77.21
76.20
76.19
76.66
76.94
77.11
77.19
77.31
D.2 Compression Ratio Differences Across Models
We observe that different models can be compressed to different extents without losing performance. This variation mainly comes from the amount of redundancy in their delta weights. For instance, T5- base has relatively high redundancy, making it possible compress it up to 224x while maintaining or even improving performance.
contrast, models like BEiT-3 have delta weights with larger magnitudes and lower redundancy, leaving less room for aggressive compression. This suggests that the achievable compression ratio depends not only on the compression method itself, but also on the model architecture, its training process, and the nature of the fine-tuned tasks.
D.3 Limitations and Future Work
Heuristic Rescaling Factor. One limitation of our method lies in the heuristic nature of the additional rescaling factor 7y. We set " according to the trace norm of each delta weight, which serves lightweight proxy for activation instability under the data-free setting. While this simple strategy helps mitigate instability at high sparsity, it remains heuristic and may not be optimal. Exploring more principled or adaptive approaches for determining y under data-free constraints remains an important direction for future work.
Deployment Efficiency. Another limitation that our method not designed to directly optimize deployment efficiency, such inference acceleration. Achieving real speedup typically requires specialized GPU kernels or structured sparsity to better exploit hardware parallelism. Future work may combine our approach with optimized kernels structured pruning to realize both strong cnapressionn-erformman trade-off's and practical deployment gains.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly summarize our main contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations and potential future work in the Discussion section
and the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification:
For all theoretical results in this paper, we provide detailed derivations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of the experimental settings and evaluation metrics in Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code has been released at http://gghhiccoooocooooiiiinnnnn
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iips.cc/pill wttp:ccppppcccceeeeeeeeee..c.ett httaa//gguuiee//eeeeabiiiisssss /Cc////oddiiisssseeeeeee ssiiiiiiccyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All experimental settings, including model types, datasets, paper-earameters are reported in Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not report error bars or other statistical significance measures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the type of GPU the paper. All experiments are conducted on a single NVIDIA A800.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics ftctps://neuriin http://pww....pppoocccccccccccaa
Answer: [Yes]
Justification:
We ensure compliance with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents thcoretica//methocolliaaa contribution without applica- tion to real- world scenarios. Thus, it may not have societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve pretrained generative models other high-risk assets. Therefore, safeguards are not necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets are properly cited in the references section.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce any new assets such datasets, models, code. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing experiments involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our study does not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "45dQD2wkBY", "reviewer": "Reviewer_9VE5", "summary": "This paper presents UltraDelta, the first data-free delta compression pipeline that claims to achieve both ultra-high compression and strong (accuracy) performance. There are three key components in UltraDelta: 1) Variance-Based Mixed Sparsity allocation; 2)  Distribution-Aware Compression; 3) Trace-Norm Guided Rescaling. The resulting compression ratios across multiple different categories of models are impressive: 133x for LLMs; 800x for general NLP models; 400x for ViT models; and 40x for multi-modality models.", "questions": "- In the “Theoretical Analysis of MSA”, the authors explicitly focused on sparsity by saying “...should be assigned smaller sparsity”. I wonder if the same conclusion holds for quantization? For instance can we say: “if a layer exhibits larger standard deviation, we should use higher bit width”?\n-  I am also wondering if the variance could be a good signal for selecting sparsity level / bitwidth? So instead of saying: “should be assigned smaller sparsity”, can we suggest: “should be assigned a sparsity in the range [a,b]?”", "limitations": "I think it is fair to mention this work is primarily on compression ratio and serving performance improvement could be a future work.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strength:\n- The problem itself is important to not only the ML research community but also the system & MLSys community. It is also a timely and very relevant topic. As someone working on delta compression in the past, some of the questions that the authors try to resolve are the issues asked by reviewers in our previous papers. Overall I think this paper is a nice addition to the NeurIPS 2025 program.\n\n- The paper is very well written and easy to follow. I particularly like the figures across the paper as they are easy to read and accessible.\n- It’s great to see the authors provide not only empirical results, but also theoretically analyze the effect of MSA. I think it also suggests why model deltas are more compressible than the entire models (based on the findings from DeltaZip & DeltaDQ, the variance or standard deviation in model deltas are significantly smaller than the entire model). I have a few suggestions & follow-up questions regarding this (please see questions & weakness).\n- The experiments conducted are extensive, covering a wide range of tasks / models including not only LLMs, but also other NLP models & \n\n# Weaknesses\n\n- [NIT]: In “Theoretical Analysis of MSA”, I wonder if the authors could show some empirical results as well. For example, say we want to maintain 99% accuracy/cosine similarity, and x-axis is the standard deviation, y-axis is the sparsity we can achieve.\n- [Follow Up on above & NIT]: I think this entropy explanation is nice -- however I think there're two potential improvements: 1) If I understood it correctly, the entropy viewpoint suggests that if you apply lossless compression, then the lower the variance, the lower bit rate you can achieve. However, across your paper you are actually apply lossy compression. I think another viable approach is to try rate-distortion theory. 2) However these theories only give you lower bound -- and it does not necessarily hold for the exact compression method you used here -- some empirical results would enhance it. See [this blog post (Why Delta Compression Works)](https://about.yao.sh/posts/why-delta-works/) as a reference.\n\n- One of the main weaknesses is that it’s unclear how efficient the proposed approach will be, when deployed in a real serving engine, i.e., whether the compression ratio can be translated well into serving performance (throughput; latency) improvements.\n\n- An example with LLMs: BitDelta quantizes the model delta to 1 bit without sparsity vs. UltraDelta quantizes/sparsifies the model delta to 4 bit + 95% of sparsity -- which one is faster? Although UltraDelta achieves higher compression ratio, it won’t be faster without a properly implemented kernel to achieve the highly sparse computation. Serving performance is also a key focus in DeltaZip and therefore they chose to use structured sparsity, which only gives 50% sparsity. Ultimately, we want faster inference, instead of just high compression ratio.\n- I don’t think this should be a show-stopper, as this paper does not focus on serving performance, however, I think it is fair to mention this in the limitations section to give enough context to the readers. Ultimately I think this paper lays a good foundation for system/ml co-design.\n\n- I would also strongly suggest authors to place this work into more context in the introduction, as the ultimate goals of these works (DeltaZip, BitDelta, DeltaDQ, etc) are to improve deployment efficiency, UltraDelta offers a significant higher compression ratio which can be leverage to further improve the deployment efficiency.\n- [NIT] I am a bit surprised the proposed approach consistently outperforms the uncompressed baseline. Although I acknowledge there might be some implicit regularization effect in compression as described by the authors -- I think some more empirical results on more controlled experiments (e.g., those models are fine-tuned by the authors with varying steps, and then compressed by the authors - to show if the models are overfitting/underfitting or not) can be beneficial. \n- Re: Hyper-parameter selection. I wonder how the authors select the sparsity level in their experiments. For example, with LLMs the authors used 95%/97% sparsity; 99% for other NLP models; 90% for multi-modality models. Do the authors perform an extensive (but might be expensive) grid search of these parameters in their experiments? Or are there any other signals used in the process?\n- For LLMs, I think some more challenging tasks should be considered in the evaluations, such as AIME/GQA, etc.", "was_revised": true, "final_justification": "I maintain my score and I think this is overall a good paper.\n\nI don't understand the comment on novelty. If we execute the standard that \"if this is done by someone else in some field, then it is not novel\", we should reject most of the papers, including the paper that proposed deep learning since it is merely an application of chain rule.\n\nSome reviewers raised the baseline of ComPEFT, and I think the authors have provided evidence that their approach outperforms it, quite significantly, which is an evidence of usefulness of this work to the community.", "rebuttal": "We sincerely thank the reviewer for strong support of our work, including: **(i) important and timely topic; (ii) well written and easy to follow; (iii) theoretical and empirical analysis; (iv) extensive experiments**. We are especially grateful for **the many constructive suggestions and insightful questions**, which significantly helped us improve the depth of our paper.\n\n---\n\n# Question 1: Variance-Bitwidth Relationship\n\nWe thank the reviewer for the insightful question. We have observed that **variance can also serve as a metric for bit-width allocation**.\n1. According to the variance–entropy relationship and the rate–distortion theory, **variance is theoretically linked to compression rate**, which depends on both sparsity and bit width.\n2. We empirically validated this in 8 ViT-B/32, where bit width was allocated based on layer variance (4-bit to layers with larger variance, 2-bit to layers with smaller variance, together it obtains 3-bit/parameter quantization). The **mixed 3-bit setting achieved accuracy close to uniform 4-bit quantization**, showing that variance is an effective metric for bit width allocation.\n|  | SUN397 | Cars | Resisc45 | EuroSAT | SVHN | GTSRB | MNIST | DTD  | AVG  |\n| - | - | - | - | - | - | - | - | - | - |\n| Finetuned | 79.2   | 77.7 | 96.1 | 99.8 | 97.4 | 98.7 | 99.7 | 79.4 | 91.0 |\n| UltraDelta (uniform 4bit) | 78.6 | 77.4 | 95.7 | 99.4 | 97   | 98.7  | 99.7  | 79  | 90.7 |\n| UltraDelta (uniform 3bit) | 77.6 | 75.7 | 95.3 | 99.3 | 96.8 | 98.6  | 99.7  | 78.7 | 90.2 |\n| UltraDelta (mix 3bit)  | 78.2 | 77.2 | 95.6 | 99.3 | 96.8 | 98.8  | 99.6  | 79.0 | 90.6 |\n\n---\n\n# Question 2 : Can variance serve as a signal for sparsity/bitwidth\n\nWhile **variance provides a strong signal for allocating *relative* sparsity levels or bit widths**, it is **not sufficient to select precise numerical values** for either parameter. Our method uses **layer-wise variance** to guide allocation, but evaluating per-layer performance is impractical in experiments. Therefore, our **analysis is conducted at the model level**, and the standard deviation is calculated based on the overall weight of the model.\n\n**1. Bit width:**  We conducted experiments to test whether variance can guide the selection of an optimal quantization bit width. Specifically, we quantized 8 ViT-B-32 models using different bit widths (without sparsity, **only quantization**), due to space limits, we only show some of the results.\n- Across all datasets, **2-bit quantization is optimal**, retaining ~99% of the finetuned model performance.\n- Bit width is a **discrete variable with large step size**. Performance drops drastically from 2-bit → 1-bit.\n- Therefore, variance cannot directly predict this optimal discrete value, as **the available discrete bitwidth levels have large gaps and lack fine granularity for precise selection**.\n\n   |  | SUN397 | Cars  | Resisc45  | EuroSAT | SVHN | GTSRB | MNIST | DTD | AVG |\n   | - | - | - | - | - | - | - | - | - | - |\n   | Std  | 0.000312  | 0.000299  | 0.000271  | 0.000242  | 0.000288  | 0.000250  | 0.000261  | 0.000263  | -  |\n   | Finetuned | 79.2 | 77.7 | 96.1 | 99.8  | 97.4 | 98.7 | 99.7 | 79.4 | 91.0  |\n   | 1-bit  | 65.8  | 62.3 | 68.0 | 58.8  | 37.9 | 39.3 | 59.6  | 49.3 | 55.1 |\n   | 2-bit  | 79.2  | 77.6 | 95.5 | 99.1 | 96.8 | 98.3  | 99.6  | 79.3  | 90.7  |\n   | Optimal Bitwidth | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 |\n\n**2. Sparsity level:**  We conducted experiments on sparsity selection using different sparisty (using **only random pruning**). The optimal sparsity is the highest sparsity that preserves ~99% of the original accuracy.\n- The results show that for models with **lower original accuracy (harder tasks)**, **layers with larger variance tend to require lower sparsity**. \n- For models with **higher original accuracy (easier tasks)**, variance has **a weaker impact** on the optimal sparsity level. \n- Therefore, variance is correlated with sparsity and **serves as a reliable signal for allocating relative sparsity levels across layers**. But **the precise sparsity value for the whole model can be influenced by task complexity**. However, **task complexity is consistent across layers of a single model**, making variance a robust metric for relative layer-wise sparsity allocation.\n|  | SUN397 | Cars | Resisc45 | EuroSAT  | SVHN | GTSRB | MNIST    | DTD  |\n| - | - | - | - | - | - | - | - | - |\n| Finetuned | 79.2 | 77.7 | 96.1 | 99.8 | 97.4  | 98.7 | 99.7 | 79.4 |\n| std | 0.000312 | 0.000299 | 0.000271 | 0.000242 | 0.000288 | 0.000250 | 0.000261 | 0.000263 |\n| Optimal Sparsity | 0.88  | 0.89  | 0.97  | 0.995  | 0.99  | 0.995 | 0.995  | 0.92  |\n\n---\n\n# Weakness 1 : Analysis of MSA\n\nWe thank the reviewer for these valuable comments. We address the two points as follows:\n\n**1. Empirical upper bound**\n\nThe results can be found in the answer to Question 2. These results show that, **at the model level, the empirical upper bound is not solely determined by variance; it also depends on task difficulty**. We will include this analysis and figure in the revised version.\n\n**2. Theoretical analysis**\n\nWe initially adopted an entropy-based discussion because entropy is a well-established proxy for the **information content**. It provides a simple, intuitive link between weight variance and the amount of information in each layer.\n\nWe fully agree that **rate–distortion theory is a more appropriate framework** under lossy compression settings like pruning, as it explicitly connects **variance to information rate and distortion**. A simplified version of the proof is provided **in Reviewer 8YfX’s Question 2** (omitted here due to space limits), and we will include a more detailed proof in the revised version of the paper.\n\nThanks to the reviewer again for helping us strengthen the analysis of MSA.\n\n---\n\n# Weakness 2 : Deployment Efficiency\n\n1. We thank the reviewer for highlighting the importance of deployment efficiency, which is indeed a primary objective of many delta compression works such as **DeltaZip**.\n2. We acknowledge that UltraDelta is **not designed to directly optimize deployment efficiency**, and achieving real inference acceleration typically requires specialized kernels or structured sparsity. Our focus in this paper is orthogonal: we aim to explore **data-free delta compression** methods that maximize information preservation, **providing a foundation that can potentially complement future system-level optimizations**.\n3. Following the reviewer’s suggestion, we will **explicitly state this limitation** in the revised version and **clarify in the introduction** that works such as DeltaZip primarily target deployment efficiency.\n\n---\n\n# Weakness 3 : Controlled Experiment\n\nWe sincerely thank the reviewer for this insightful comment. To investigate this, we conducted additional controlled experiments on two datasets, **SUN397 and Cars**, using a pretrained ViT-B/32 model. We trained the models **with varying fine-tuning steps** and then applied **UltraDelta**. The results show that:\n\n1. **Underfitted models** (early training steps, higher loss) consistently benefit from UltraDelta, exhibiting **noticeable accuracy improvements** compared to the baseline.\n2. **Well-fitted models** (later steps, lower loss) do not show significant performance gains after compression. **In SUN397**, we observe only a small improvement after compression, possibly because **the task is more complex and data-rich**, making it harder for the model to fully fit the dataset.\n3. The observed effect mainly reflects that **UltraDelta benefits models that are underfitted**, while well-trained models already capture most of the task information, leaving less room for improvement.\n\n| Training Step | 1000  | 2000 | 3000 | 4000 | 5000 | 6000 | 7000 | 8000 | 9000 | 10000 |\n| - | - | - | - | - | - | - | - | - | - | - |\n| **Loss (Cars)**  | 0.1404 | 0.0834 | 0.0206 | 0.0069 | 0.0059 | 0.0059 | 0.0034 | 0.0077 | 0.0048 | 0.0036 |\n| **Acc (Cars)** | 72.86  | 68.90  | 75.22  | **78.78**  | **77.74**  | **78.37**  | **78.01**  | **75.21**  | **77.60**  | **77.34**  |\n| **UltraDelta (Cars)** | **74.02**  | **73.27**  | **75.82**  | 75.56  | 75.62  | 75.66  | 76.13  | 72.80  | 74.74  | 74.06  |\n| **Loss (SUN397)**  | 0.5475 | 0.1432 | 0.0426 | 0.1125 | 0.0526 | 0.0040 | 0.0018 | 0.0009 | 0.0006 | 0.0004 |\n| **Acc (SUN397)**  | 73.63 | 75.94 | 75.40 | 74.63 | 74.07 | **76.94** | 76.87 | 77.03 | 77.06 | 77.07 |\n| **UltraDelta (SUN397)** | **76.02** | **76.92** | **77.21** | **76.20** | **76.19** | 76.66 | **76.94** | **77.11** | **77.19** | **77.31** |\n\nThis explains why performance improvements are more commonly observed on **LLMs**, as they are typically evaluated in a **zero-shot setting** without task-specific fine-tuning, making them more likely to benefit from the regularization effect of compression.\n\n---\n\n# Weakness 4 :  Hyper-parameter Selection\n\nOur choice of sparsity levels is guided by prior findings in delta compression, which show that models can typically have **around 90% sparsity** without severe accuracy degradation. Based on this, we explored a **coarse set of candidate values** (90%, 95%, 99%, 99.9%) to establish a reasonable range, and then made **minor adjustments** for specific models when accuracy changed a lot within this range.\n\n---\n\n# Weakness 5 : More Challenging Tasks\n\nWe expanded the LLM evaluations to include more challenging tasks: **MATH** (with some AIME problems), **GPQA**, **MBPP+**, and **HumanEval+**, using updated models **Qwen2.5-7B-Instruct** and **Llama-3.1-Tulu-3-8B-SFT**. Results are shown below:\n\n| Model  |  Llama  |  | Qwen |  |\n| - | - | - | - | - |\n|  | Humaneval+   | MBPP+  | MATH  | GPQA |\n| Finetuned | 57.3  | 56.6 | 36.2  | 36.87  |\n| BitDelta (1bit)  | 55.5  | 52.9  | 38.1  | 37.37  |\n| Delta-CoMe (1bit)  | 53.8  | 53.1  | 29.45  | 31.43  |\n| DARE (95%)  |  50.6 | 50.0  | 37.76 | 34.85  |\n| UltraDelta (4bit+95%) |  **56.1** | **54.5** | **40.14**  | **38.38** |"}, {"review_id": "WkLHTOAogU", "reviewer": "Reviewer_vAk5", "summary": "The paper considers the task of delta compression, where a base model is separately finetuned for several different tasks, and the resulting models have to be stored with as little storage overhead as possible by saving only (a compressed version) of the difference between the base model’s weights and the finetuned weights. The paper proposes UltraDelta, which combines and improves upon two lines of prior work: quantization and pruning. UltraDelta consists of three main improvements over prior methods: (i) a variance-based sparsity allocation (lower sparsity for high-variance layers), (ii) a distribution-aware quantization (uniform quantization followed by group-wise pruning), and (iii) rescaling the quantized weights using the trace norm of the delta weights as a scaling factor. The results across a wide range of models, tasks, and domains show that UltraDelta achieves very high compression ratios (up to 800x) while maintaining (sometimes even increasing) strong performance.", "questions": "* Intuitively, why is it important to “maintain the relative proportions across different quantized weight values”? Is there a good explanation for why this should achieve better performance than, e.g., uniform or magnitude-based pruning?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The paper conducts an extensive empirical validation of the proposed method across models of different scales, types, and tasks (LLMs, vision models, and multi-model models), leaving no doubt about the superiority of the proposed approach.\n* The paper is very well-written and easy to follow. Figure 2 is a work of art and helps the reader understand the different components of the proposed method.\n* The proposed method addresses key weaknesses of prior approaches: (i) treating all layers equally, (ii) disrupting intra-layer weight distributions, and (iii) instability under high compression.\n* The paper provides good theoretical motivation and empirical validation, including ablation studies, for the proposed method.\n\n**Weaknesses**\n\n* The paper claims that its method does not affect the inference stage, which is not entirely true. Concretely, at inference time, the delta weights need to be added to the base model’s weights, which introduces a (non-negligible?) memory and compute overhead, especially for large models.", "was_revised": true, "final_justification": "The authors have sufficiently addressed my concern about the method's efficiency, which is why I am increasing my score.", "rebuttal": "We sincerely thank the reviewer for the constructive feedback and positive comments on our work, including: **(i) Extensive experiments; (ii) Clear and well-written; (iii) Addresses prior weaknesses; (iv) Strong theoretical and empirical validation**. We are especially grateful for the kind recognition of **Figure 2**, described as *“a work of art”*. As our method consists of multiple components, we invested great effort in designing this figure to help readers quickly grasp our approach, and we are delighted that this effort was appreciated.\nWe now address the reviewer’s comments in detail below.\n\n---\n# Question 1 : Explanation for maintaining relative proportions across values\n\nWe thank the reviewer for the question. Below, we explain why maintaining relative proportions across different quantized values is important, supported by **theoretical, methodological, and empirical perspectives**.\n\n**1. Theoretical perspective**\n\n- Our goal in maintaining the relative proportions of different quantized values is to **keep the weight distribution before and after pruning as similar as possible**. This can be formalized using **KL divergence** between the original quantized distribution $P = \\\\{p_i\\\\}$ and the pruned distribution $Q = \\\\{q_i\\\\}$:\n   $$\n   D_{\\\\mathrm{KL}}(P \\\\parallel Q) = \\\\sum_{i} p_i \\\\log \\\\frac{p_i}{q_i}\n   $$\nwhere $p_i$ and $q_i$ denote the proportion of the $i$-th quantized value before and after pruning. When the relative proportions of quantized values are better maintained, the pruned distribution $Q$ stays closer to the original quantized distribution $P$, resulting in a lower $D_{\\\\mathrm{KL}}$. **A lower $D_{\\\\mathrm{KL}}$ means the weight distribution is better preserved**.\n- Preserving the weight distribution is important because it **reflects the statistical structure of the model parameters**. Weights are not just arbitrary numbers; their distribution encodes **structural and semantic information** learned during training. Preserving distribution helps maintain model behavior after compression.\n- Furthermore, **Figure 4** provides empirical evidence supporting this claim: it shows that **UltraDelta achieves higher cosine similarity** between the compressed model’s outputs and those of the original model, showing that **preserving weight proportions helps retain the semantic information of the original model**.\n\n**2. Method comparison**\n- **DAC (ours)** uses group-wise pruning to explicitly preserve the relative proportions of quantized values, **maintaining the overall weight distribution**. This approach is **highly robust**, avoiding the randomness that could distort proportions under high sparsity.\n- **Magnitude pruning** removes weights with smaller magnitudes, **shifting the distribution toward large values**. This **ignores the balancing effect of small weights**. The imbalance weakens parameter interactions and leading to accuracy degradation.\n-  **Uniform pruning** removes weights randomly and can partially keep the distribution. However, under high sparsity it may still accidentally distort the distribution, showing **lower robustness** compared to methods that explicitly preserve weight proportions.\n\n**3. Experiment validation**\n- As shown in our experiments, under high sparsity, **magnitude pruning performs worse than almost all other methods**, showing that **disrupting the weight distribution can cause severe performance degradation**.\n|  Method      | LLM-7B    | LLM-13B   | T5-Base   | ViT-Large |\n| --------------------- | --------- | --------- | --------- | --------- |\n| **Magnitude Pruning**     | 14.19     | 7.40      | 56.08     | 11.20     |\n| **UltraDelta (Ours)** | **45.57** | **52.05** | **86.74** | **94.40** |\n\n*Note: These results are extracted from our Experiment section. Both magnitude pruning and UltraDelta are under the same compression ratios.*\n\n- Our **ablation study (Figure 5 and Table 9)** compare DAC against uniform/random pruning across 30 ViTs compression settings. In such large-scale compression scenarios, robustness is particularly important. The results show that **DAC is significantly more robust than uniform pruning**, confirming the effectiveness of preserving weight distribution.\n| Method               | 4-bit    | 8-bit     | No quantization |\n| -------------------- | -------- | --------- | --------------- |\n| **DARE**             | 81.58    | 83.85     | 85.88           |\n| **UltraDelta (DAC)** | **86.40** | **90.18** | **92.45**       |\n\n*Note: This table is extracted from the experimental results reported in our paper (Table 9, visualization in Figure 5), comparing DARE[1] and DAC under different quantization settings (4-bit, 8-bit, and no quantization).*\n\n---\n\n# Weakness 1 : Inference Overhead\n\n1. Our statement was in comparison with **other delta compression methods**, which also require adding delta weights at inference time. Our method only needs a **simple addition**, while methods like **Delta-CoMe[2] involve matrix multiplications** to project low-rank matrices back into the full parameter space before inference, which are more expensive.\n2. The addition can be done on **CPU** before inference, so it **does not increase GPU memory usage** and introduces **minimal compute overhead**. For example, we measured total inference time on WizardCoder-Python-7B-V1.0 evaluated on HumanEval+. We compared the **fine-tuned model, UltraDelta (addition on CPU), and Delta-CoMe (layer-wise matrix multiplication on CPU)**:\n| Method  | Fine-tuned | UltraDelta | Delta-CoMe |\n| ------- | ---------- | ---------- | ---------- |\n| **Runtime** | 217s       | 221s       | 341s       |\n\n3. We will **clarify this in the revised version** and add inference time comparisons. Thank you for helping us improve the precision of our claims.\n\n---\n\n[1] Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch\n\n[2] Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models"}, {"review_id": "UgcYZ9rtlo", "reviewer": "Reviewer_asSv", "summary": "This paper proposes UltraDelta, an efficient method to compress delta parameters without additional data. Extensive experiments demonstrate that by assigning layer-wise sparsity rates based on delta parameter variance and rescaling the pruning factor $\\gamma$, UltraDelta achieves comparable or better performance to the original fine-tuned model under extreme compression ratios.", "questions": "1. DAREx reports that DARE suffers a significant performance degradation when the delta parameters exhibit large mean magnitudes. Does UltraDelta encounter a similar issue? \n2. I argue that parameter grouping in the DAC component is unnecessary, as random pruning does not alter the original parameter distribution.\n3. Figure 4 employs cosine similarity to assess how well compression methods preserve internal representations. However, despite MP(80×) exhibiting lower final-layer embedding similarity than BitDelta, BitDelta underperforms on the CoLA task (43.51 vs. 36.83, Table 8). A similar pattern is observed for DARE(80×) vs. Delta-CoMe (57.90 vs. 57.67). How should this discrepancy be interpreted?\n4. P6 Equation (11), the symbol $\\odot$ means Hadamard product or matrix multiplication?", "limitations": "Yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1. The various experiments substantiate that UltraDelta can achieve better or comparable performance with the original fine-tune model even under an extreme compression ratio.\n2. The UltraDelta method is efficient because it does not use any additional data.\n\nWeakness:\n\n1. The paper employs sparsification and quantization in its methodology; however, it calculates the compression ratio using techniques from the quantization literature, overlooking the storage implications associated with the positions of non-zero elements. In contrast, conventional sparsification studies typically emphasize the sparsity ratio. Consequently, the actual compression ratio is significantly lower than what is reported.\n2. Missing Baseline: The paper does not include a comparison with a simple baseline such as DARE combined with quantization.\n3. The method of UltraDelta is not novel. Here is the reason:\n  * The idea of  MSA method assigns different sparsity rates to different layers based on their importance score, has been applied in many previous works[1,2].\n  * The methodology and analysis of TNGR closely resemble those of prior work [3], which also derived an upper bound for the error $\\varepsilon$ and proposed an estimation method for $\\gamma$ using auxiliary data. The key distinction between TNGR and DAREx[3] lies in the fact that TNGR does not rely on any additional data. However, the estimation function of $\\gamma$ and the setting of the value between 0.5 and 1.0 of TNGR are empirical, which constrains the novelty of TNGR.\n4. The models used in the experiments are kind out of dated. More recent models like Qwen2.5 should be taken into consideration.\n5. This paper presents experiments on models with easily compressible delta parameters. It is recommended that UltraDelta be evaluated in more challenging scenarios, such as delta parameters with large norms, as indicated in [3].\n\n\n---\n[1]: AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models\n\n[2]: Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment\n\n[3]: Revisiting Delta-Parameter Pruning For Fine-Tuned Models", "was_revised": true, "final_justification": "I have read through all reviewers and rebuttals. I choose to maintain my score\n\n- I fully agree with Review 8YfX. The theoretical foundation of the paper lacks rigor, and its primary methodology relies heavily on existing literature, resulting in a lack of originality.\n\n- In the experiments, the content on compression ratios is misleading. I think this is serious issue because it will affect the subsequent work.\n\nDue to the above reasons, I  I think this paper should be rejected.", "rebuttal": "We thank the reviewer for the positive feedback, including: **(i) extensive experiments with strong performance; (ii) an efficient and data-free method**. We now address the reviewer’s questions in detail below.\n\n---\n\n# Question 1: Large mean magnitude setting\n\nWe revisited the DAREx[1] experiments and found that the reported degradation occurs on **MetaMath-Llemma-7B** [8], where DAREx assumed **Llama-2-7B** as the base model. However, according to its huggingFace model card, the correct base model is **Llemma-7B**. We replicated the experiments under both settings:\n\n| Base Model | Finetuned Acc | BitDelta (1bit) | DeltaCoMe (1bit) | DARE (95%) | MP (95%) | UltraDelta (4bit+95%) |\n| - | - | - | - | - | - | - |\n| Llemma-7B  | 63 | 59.89  | 58.18  | 61.93 | 59.65 | **62.40** |\n| Llama-2-7B | 63 | 0.23  | 0 | 0 | 0.91 | 0.91 |\n\nDAREx’s conclusion that **importance-based pruning (such as MP) performs better than DARE[2] in a large mean magnitude setting**, only holds at **low sparsity levels**. At high sparsity, all delta compression methods perform poorly. When using **the correct base model (Llemma-7B)**, all methods perform as expected without severe degradation.\n\n---\n\n# Question 2: The necessity of DAC\n\nWhile random pruning may partially preserve the weight distribution, **it does not guarantee robustness**, especially under high sparsity.\n\n1. **Risk of distortion:** At high sparsity, random pruning can **accidentally distort the original parameter proportions**, making the compressed model **less robust** compared to methods that explicitly maintain weight distribution.\n\n2. **Empirical evidence:** Our ablation study (Figure 5, Table 9) compares DAC with random pruning across 30 ViT setting. DAC consistently achieves higher accuracy:\n| Method               | 4-bit    | 8-bit     | No quantization |\n| -------------------- | -------- | --------- | --------------- |\n| **DARE**             | 81.58    | 83.85     | 85.88           |\n| **UltraDelta (DAC)** | **86.4** | **90.18** | **92.45**       |\n\n\nThese results confirm that **preserving weight distribution explicitly is more robust than random pruning** in large-scale, high-sparsity compression scenarios.\n\n---\n\n# Question 3: Fig 4\n\n1. The cosine similarity metric in Figure 4 is intended to illustrate **how well each method preserves the information of the original fine-tuned model, not to predict downstream performance directly**. A higher similarity indicates closer alignment to the uncompressed model’s representations, but **final task accuracy also depends on the uncompressed model’s own performance** on that task.\n\n2. For example, beyond the two patterns highlighted by the reviewer, we also observe that **UltraDelta outperforms the fine-tuned model (62.64 vs 60.18)**, further showing that **cosine similarity alone does not perfectly reflect downstream performance**. \n\n3. In the final version, we will include **more figures on other datasets** to better illustrate this relationship and reduce the potential influence of the fine-tuned model.\n\n---\n\n# Question 4: Eq(11)\n\nIt means Hadamard product.\n\n---\n\n# Weakness 1: Compression ratios\n\nWe sincerely thank the reviewer for pointing out this issue.\n1. Our focus is on exploring **the upper bound of compression ratio achievable with minimal accuracy loss**. We believe this is a meaningful research direction as an ideal compression ratio can **benefit deployment efficiency, reduce communication costs, and storage efficiency**, as Reviewer 9VE5 noted: “*this paper lays a good foundation for system/ML co-design*”.\n\n2. We acknowledge that a more rigorous evaluation should report the effective compression ratio. Using a **CSR format**, we recalculated the practical compression ratios and found they **remain high while preserving excellent accuracy**:\n| Metric                      | Llama-2-13B | ViT-Base | ViT-Large | T5-Base |\n| --------------------------- | ----------- | -------- | --------- | ------- |\n| Original Accuracy           | 50.94       | **91.0**     | **94.4**      | 86.37   |\n| Compressed Accuracy         | **52.05**       | 90.7     | **94.4**      | **86.74**   |\n| Effective Compression (CSR) | 30.97$\\\\times$       | 36.92$\\\\times$    | 109.10$\\\\times$    | 225.50$\\\\times$  |\n\n3. We note that many ongoing works **address the efficiency gap between ideal and practical sparsity representations**, e.g., Block Sparse Row[5], SMaT (SC’24) [6], CS256 (ICLR’25 Workshop on SLLM) [7]. These advances can further improve practical deployment efficiency.\n\n4. In the revised version, we promise to **clarify our terminology and include both ideal and practical compression ratios** for completeness. We thank the reviewer again for this valuable suggestion that improves the rigor of our work.\n\n---\n\n# Weakness 2: Missing baseline\n\nWe thank the reviewer for this suggestion. **DARE itself does not natively include quantization**, which is why we did not report this baseline in the main paper. Following the reviewer’s comment, we additionally evaluated **DARE + uniform quantization**:\n\n|  | wizardmath-7b | wizardmath-13b | wizardcoder-7b | wizardcoder-13b | llama-2-7b-chat | llama-2-13b-chat |\n| - | - | - | - | - | - | - |\n| Individual  | 41.55 | 47.31 | 49.40 | 58.50  | 45.17 | 47.00 |\n| DARE (4bit+95%)  | 33.78 | 41.23 | 44.20 | 48.50  | 32.12  | 34.40 |\n| UltraDelta (4bit+95%) | **38.76**  | **48.45** | **51.80**  | **59.10** | **46.14** | **48.59**  |\n\n---\n\n# Weakness 3: Limited novelty\n\nWe appreciate the reviewer’s comment. Our novelty lies in **the design, motivation, and strict data-free constraint**.\n\n**1. The first data-free pipeline for ultra-efficient delta compression**\n\nTo the best of our knowledge, we are **the first to propose a data-free pipeline for ultra-efficient delta compression**. This makes our approach **practical in scenarios where tuning or data is unavailable**. Our method maintains strong performance under strict data-free constraints, achieving high compression ratios without additional data or tuning.\n\n**2. Novel motivation and design**\n\nOur method is guided by a clear and novel motivation: **maximizing information preservation**. All components are **specifically designed around this motivation**.\n\n- **Variance-based mix sparsity allocation** uses weight variance as a simple, data-free proxy for **information content**, theoretically supported by the variance–entropy relation and rate–distortion theory.\n\n- **Distribution-aware compression** preserves the relative proportions of quantized values to **better preserve the intra-layer information**.\n\n- **Trace-norm-guided rescaling** is designed to estimate an additional rescaling factor using trace norm to stabilize the pruned model based on trace norm, while remaining **fully data-free**.\n\n**3. Clarifications on related works**\n\n- **AlphaPruning[3]** has high computation overhead as it requires **spectral analysis** on every weight matrix, including **SVD/eigenvalue decomposition** and **power-law fitting** to estimate the heavy-tail exponent. ALS [4] requires **data access** to compute activation correlations. In contrast, **our method adopts a simple, data-free variance metric**, offering both **computational efficiency** and **no data dependency**.\n\n- DAREx[1] relies on data (labeled validation or unlabeled), and its own results show that using unlabeled data generally underperforms labeled validation data. However, using labeled validation data may cause the **risk of performance inflation**, which our approach avoids entirely.\n\n- We acknowledge that TNGR are empirically chosen. However, the design is **well-motivated**: the **trace norm captures the effective rank** of delta weights, serving as a simple, **data-free proxy for information content**. We agree this heuristic is a limitation and have explicitly stated it in our ***Limitations*** and ***Future Work*** section, where we plan to conduct **more in-depth research on estimation strategies**, while maintaining the **data-free property** of our approach.\n\n---\n# Weakness 4: Moer recent LLMs\nWe thank the reviewer for this suggestion. We have added experiments on **more recent LLMs**, including **Qwen2.5-7B-Instruct** and **Llama-3.1-Tulu-3-8B-SFT**. We also evaluate them on more challenging tasks:\n\n| Model | Llama | | Qwen | | |\n| - |  - | - | - | - | - |\n| | Humaneval+ | MBPP+ | MATH | GPQA |\n| Finetuned  | 57.3 | 56.6 | 36.2  | 36.87 |\n| BitDelta (1bit) | 55.5  | 52.9 | 38.1 | 37.37 |\n| Delta-CoMe (1bit)  | 53.8 | 53.1 | 29.45 | 31.43 |\n| DARE (95%)  | 50.6 | 50.0 | 37.76  | 34.85 |\n| **UltraDelta (4bit+95%)** |  **56.1**  | **54.5** | **40.14**  | **38.38** |\n\n---\n\n# Weakness 5: Large norm model\n\n1. Our experiments already cover a wide range of models of different types, sizes, and compression difficulties, with average L1 norms ranging from **4.1e-5 to 5.01e-3**.\n\n2. We also conducted tests on the large-norm model reported in DAREx [1] (**MetaMath-Llemma-7B**), as detailed in **Question 1**. When Llama-2-7B was incorrectly used as the base model, the L1 norm reached **1.7e-2**, and in this extreme case **all delta compression methods failed**.\n\n---\n\n[1] DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models\n\n[2] Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch\n\n[3] AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models\n\n[4] Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment\n\n[5] Automatic Performance Tuning of Sparse Matrix Kernels\n\n[6] High Performance Unstructured SpMM Computation Using Tensor Cores\n\n[7] COMPRESSED SPARSE TILES FOR MEMORY-EFFICIENT UNSTRUCTURED AND SEMI-STRUCTURED SPARSITY\n\n[8] METAMATH: BOOTSTRAP YOUR OWN MATHEMATICAL QUESTIONS FOR LARGE LANGUAGE MODELS"}, {"review_id": "0a70Oba2Bk", "reviewer": "Reviewer_8YfX", "summary": "This paper proposes UltraDelta, a new data-free compression pipeline designed to reduce the storage overhead of fine-tuned models by aggressively compressing delta weights (the difference between a fine-tuned model and its base pretrained model).", "questions": "See limitations", "limitations": "1. Limited Novelty: The core ideas—layer-wise importance for adaptive pruning [1], delta parameter quantization [2], and rescaling factor adjustments [3]—are each incremental extensions of established compression and pruning techniques. Furthermore, their combination has already been studied in works such as ComPEFT [4], which uses sparsification and ternary quantization for delta compression. This paper does not provide novel methodology beyond re-combining existing ideas, nor does it adequately acknowledge or cite these prior works (for example ComPEFT [4]).\n\n2. Weak Theoretical Foundation: The theoretical rationale, such as the variance–entropy relationship, is presented in an overly simplistic manner without rigorous proofs or formal guarantees. What's more, [3] already provides a comprehensive theoretical treatment of variance’s effect on pruning error; the current paper should reference and build on that work directly.\n\n3. The discussion in related work is not proper that methods like DAREx [3] and BitDelta [2] can operate with only small amounts of unlabeled data in an training-free unsupervised fashion, providing practical data efficiency and easy to implement, so this is not a disadvantage.\n\n4. Unclear Rescaling Factor Determination: The trace-norm-based rescaling factor γ is claimed to be inversely proportional to the trace norm, but it is not clear why it is not simply set equal to its inverse. As discussed in lines 549–552 in appendix, the factor appears manually chosen within a heuristic range. A more detailed derivation or empirical justification is necessary to clarify how γ should be set in practice.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses an important problem in delta parameter compression for large-scale fine-tuned models.\n\n2. It includes an extensive empirical evaluation covering multiple model families and tasks, demonstrating broad experimental effort.\n\n3. The manuscript is generally well-structured and easy to follow.\n\nLimitations:\n\n1. Limited Novelty: The core ideas—layer-wise importance for adaptive pruning [1], delta parameter quantization [2], and rescaling factor adjustments [3]—are each incremental extensions of established compression and pruning techniques. Furthermore, their combination has already been studied in works such as ComPEFT [4], which uses sparsification and ternary quantization for delta compression. This paper does not provide novel methodology beyond re-combining existing ideas, nor does it adequately acknowledge or cite prior works (for example ComPEFT [4]).\n\n2. Weak Theoretical Foundation: The theoretical rationale, such as the variance–entropy relationship, is presented in an overly simplistic manner without rigorous proofs or formal guarantees. What's more, [3] already provides a comprehensive theoretical treatment of variance’s effect on pruning error; the current paper should reference and build on that work directly.\n\n3. The discussion in related work is not proper that methods like DAREx [3] and BitDelta [2] can operate with only small amounts of unlabeled data in an training-free unsupervised fashion, providing practical data efficiency and easy to implement, so this is not a disadvantage.\n\n4. Unclear Rescaling Factor Determination: The trace-norm-based rescaling factor γ is claimed to be inversely proportional to the trace norm, but it is not clear why it is not simply set equal to its inverse. As discussed in lines 549–552 in appendix, the factor appears manually chosen within a heuristic range. A more detailed derivation or empirical justification is necessary to clarify how γ should be set in practice.\n\n[1] LAYER-ADAPTIVE SPARSITY FOR THE MAGNITUDE-BASED PRUNING\n\n[2] BitDelta: Your Fine-Tune May Only Be Worth One Bit\n\n[3] DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models\n\n[4] ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization", "was_revised": true, "final_justification": "------- Question 1 : Limited Novelty ----------\n\nComPEFT [4] can serve for Full Finetuning Updates, in its section 5.3, it demonstrate the effectiveness in compressing Full Finetuning Updates, the author should carefully read the references and give a comprehensive discussion, again, combining\nComPEFT [4] can indeed be data-free, as state in introduction \"ComPEFT retains high performance post-hoc compression without the need for additional training\"\nComPEFT [4] contain two steps: pruning and quantization which is followed by the paper closely.\nAs a result, without proper reference to related work is completely wrong.\n\n------- Question 2 : Weak Theoretical Foundation------\n\nDAREx[3] not only consider weight but also the activations, where its c represents $\\Delta Wx$, so they are more comprehensive in analyzing the output change.\n\nYour response on \"our analysis is based on weight variance, which is fully data-free\" is wrong, your $a= \\Delta \\theta x$ which rely on the input $x$, so your analysis is not data-free.....\n\nFinally, it is the variance being the key points rather than the entropy, you Theoretical Analysis of the Rescaling Factor eventually analyze the activation error $\\epsilon$ rather than entropy as state in line 185, this is the same as DAREx[3]'s theorem 3.1, you can completely remove the entropy term but just bound the output change as a direct signal on performance.\n\nAs a result, without proper reference to related work is wrong.\n\n-------------- Question 4 : Unclear Rescaling Factor Determination--------\n\nAs shown in above response, the author wrongly believe their analysis is \"data-free\" and it rely on the assumption of Gaussian Distribution of weight and \"ignoring\" the influence of input $x$, thus the rescaling depend on weight is not comprehensive due to ignoring the input $x$ and may perform bad when $x$ activation is large (for example massive activation[5] in LLM)\n\n[5] Massive Activations in Large Language Models\n\nDue to the wrong statement and reference on related work for novelty, lack reference to prior theory analysis, lack understanding on their own theory analysis, I think this paper should be rejected.", "rebuttal": "We sincerely thank the reviewer for the positive feedback, including: **(i) addressing an important problem for large-scale fine-tuned models; (ii) extensive experiments; (iii) well-structured and easy to follow**. We now address the reviewer’s questions in detail below.\n\n---\n\n# Question 1 : Limited Novelty\n\nWe appreciate the reviewer’s comment. Our novelty lies in **the design, motivation, and strict data-free constraint**.\n\n**1. The first data-free pipeline for ultra-efficient delta compression** :\n\nTo the best of our knowledge, we are **the first to propose a data-free pipeline for ultra-efficient delta compression**. This makes our approach **practical in scenarios where tuning or data is unavailable**. Our method maintains **strong performance** under strict data-free constraints, achieving **high compression ratios** without additional data or tuning.\n\n**2. Novel motivation and design** :\n\nOur method is guided by a clear and novel motivation: **maximizing information preservation**. All components are **specifically designed** around this motivation.\n\n- **Variance-based mix sparsity allocation** uses **weight variance** as a simple, data-free proxy for **information content**, theoretically supported by the variance–entropy relation and rate–distortion theory.\n\n- **Distribution-aware compression** preserves the relative proportions of quantized values to **better preserve the intra-layer information**. This distribution-aware compression, to our knowledge, is not used in prior delta compression work.\n\n- **Trace-norm-guided rescaling** is designed to estimate an additional rescaling factor using trace norm to stabilize the pruned model based on trace norm, while remaining **fully data-free**.\n\nWhile we understand the reviewer’s concern, we believe it is not appropriate to negate our contributions by **referring to broad topics such as layer-wise pruning or delta compression quantization**. These are widely studied areas, and many works explore them with different goals and techniques. Our method introduces **novel designs and motivation**, which we believe **provide novelty within these broader topics**.\n\n**3. Clarifications on related works** :\n\n- LAMP[1] proposes an importance score **based on weight magnitude**, whereas **we design a variance-based score** from an infomation-theoretic perspective. The metric we use for layer-wise sparsity is quite different.\n- BitDelta[2] applies **binary masks and scaling factors** to quantize delta weights. This approach is **fundamentally different from our distribution-aware compression**, which uses uniform quantization combined with group-wise pruning to preserve the relative proportions of quantized values.\n- DAREx[3] **relies on data (labeled validation or unlabeled)**, and its own results show that using unlabeled data generally underperforms labeled validation data. However, using labeled validation data may cause the **risk of performance inflation**, which our approach avoids entirely.\n- ComPEFT[4] applies **magnitude pruning and quantizes the remaining weights with a fixed shared value**. The value is determined using validation data. It is **not data-free**, it does **not implement rescaling factor adjustment**, and its **sparsity and quantization strategies differ from ours**. Moreover, ComPEFT[4] is mainly designed for **PEFT models**, making its application scope different from UltraDelta, which targets general delta compression across full fine-tuned models.\n- We have already cited [1], [2], and [3] in our paper, and we will include a citation of ComPEFT[4] in the revised version.\n\n---\n\n# Question 2 : Weak Theoretical Foundation\n\n1. We thank the reviewer for the comments on our theoretical analysis. While we understand the concerns raised here, we would like to point out that other reviewers had a different perspective. **Reviewer vAk5** mentioned that *“the paper provides good theoretical motivation and empirical validation”*, and **Reviewer 9VE5** noted that *“it’s great to see the authors provide not only empirical results, but also theoretically analyze the effect of MSA.”*\n\n2. To further strengthen this part, we have included an additional theoretical analysis based on **rate-distortion theory** (thanks to Reviewer 9VE5). Let the original weights be $W\\\\sim\\\\mathcal{N}(\\\\mu,\\\\sigma^2)$, and the compressed weights be $W^′$. The distortion is defined as the mean squared error (MSE) between them:\n   $$\n   D=\\\\mathbb{E}[(W-W’)^2]\n   $$\nAccording to classical rate-distortion theory for a Gaussian source, the minimum number of bits (information rate) required to represent $W$ under distortion $D$ is:\n$$\nR(D)=\n\\\\begin{cases}\n\\\\frac{1}{2}\\\\log\\\\frac{\\\\sigma^2}{D}, & 0 \\\\leq D \\\\leq \\\\sigma^2 \\\\\\\\\n0, & D > \\\\sigma^2\n\\\\end{cases}\n$$\nHere, $R(D)$ measures the information needed to encode the weights while maintaining a reconstruction error no larger than $D$. This relationship shows that, for a fixed distortion $D$, weight with **larger variance $\\\\sigma^2$ require more information to represent**. Therefore, **layers with high weight variance are more “information-rich”** and should be preserved more carefully during pruning to maximize information preservation.\n\n3. Regarding DAREx[3], we would like to clarify that the variance discussed in DAREx[3] (Theorem 3.1) is not the same as ours. The variance in DAREx[3] refers to **activation variance**, which requires data to obtain activations. In contrast, our analysis is based on **weight variance**, which is fully data-free. Additionally, DAREx[3] studies activation variance for the purpose of **rescaling factor adjustment**, our use of weight variance is for **layer-wise sparsity allocation**. The two approaches **differ in the definition of variance, data requirements, and intended use**.\n\n---\n\n# Question 3 : Data Usage in BitDelta[2] and DAREx[3]\n\n1. Compared with DAREx[3] and BitDelta[2], our method does **not require any labeled or unlabeled data**, nor does it involve additional tuning or retraining. This makes UltraDelta **highly generalizable**, saves computation and deployment time, and is practical for real-world applications where data availability is limited or privacy concerns exist. \n\n2. Although DAREx[3] can use unlabeled data, its experiments show that tuning with unlabeled data performs worse than using labeled validation data, while the latter **risks evaluation inflation**.\n\n3. BitDelta[2] uses 800 samples (length 128) from the C4 dataset for calibration, introducing **data and time dependencies** that limit its applicability in data-restricted environments. We also report the compression runtime comparison between BitDelta[2] and UltraDelta:\n\n   | Method  | BitDelta | UltraDelta |\n   | ------- | -------- | ---------- |\n   | Runtime | 326.96s  | **211.53s**    |\n\n---\n\n# Question 4 : Unclear Rescaling Factor Determination\n\n1. The trace norm measures **the overall magnitude and effective rank of the delta weight**, reflecting how much information it carries. A lower trace norm typically indicates lower effective rank and less information. \n\n2. Motivated by **information preservation**, we use the trace norm as **a simple and data-free proxy** for information. We also empirically validate that the optimal rescaling factor is inversely related to the trace norm (see **Figure 3**).\n\n3. Although this is a heuristic, it is **data-free, theoretically motivated, and empirically validated** in our experiments. We hope it can open **an interesting direction for future work** on data-free rescaling factor adjustment.\n\n---\n[1] LAYER-ADAPTIVE SPARSITY FOR THE MAGNITUDE-BASED PRUNING\n\n[2] BitDelta: Your Fine-Tune May Only Be Worth One Bit\n\n[3] DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models\n\n[4] ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization"}]

Author rebuttal:

Meta-review: (a) Summary of Scientific Claims and Findings

This paper introduces UltraDelta, a novel data-free pipeline for the delta compression of fine-tuned models. The primary goal is to address the significant storage overhead associated with storing numerous task-specific models. The proposed method consists of three key components designed to work in concert: (1) Variance-Based Mixed Sparsity Allocation (MSA), which assigns layer-wise sparsity based on weight variance to preserve information in high-variance layers; (2) Distribution-Aware Compression (DAC), which applies uniform quantization followed by group-wise pruning to maintain the intra-layer weight distribution; and (3) Trace-Norm-Guided Rescaling (TNGR), which uses the trace norm of delta weights to estimate a global rescaling factor for model stability. The authors claim that this pipeline achieves state-of-the-art compression ratios while maintaining strong performance across a wide variety of models, including LLMs, vision transformers, and general NLP models, all without requiring any data for the compression process.

(b) Strengths of the Paper

The paper has several notable strengths, as highlighted by the reviewers:

Practical Importance: It addresses the highly relevant and practical problem of model storage, which is a significant challenge in the era of large-scale model fine-tuning.

Data-Free Approach: The core contribution of a completely data-free compression pipeline is a major practical advantage, making the method highly applicable in real-world scenarios where data may be unavailable due to privacy, cost, or accessibility constraints.

Extensive and Strong Empirical Validation: The authors provide comprehensive experiments across a diverse set of models (LLaMA-2, RoBERTa, ViT), scales, and tasks. The results consistently demonstrate that UltraDelta outperforms existing methods, especially at very high compression ratios.

Clarity and Presentation: The paper is generally well-written, clearly structured, and easy to follow. The figures, particularly the one illustrating the pipeline, are effective in conveying the method's components.

(c) Weaknesses of the Paper

Despite its strengths, the paper has some weaknesses that were pointed out during the review process:

Limited Novelty: A primary concern, raised by multiple reviewers (asSv, 8YfX), is the novelty of the individual components. The core ideas of using layer-wise importance for pruning, quantization, and rescaling factors are not new. While their combination into a data-free pipeline is the main contribution, some reviewers felt the work was an incremental extension of established techniques.

Initial Misleading Metrics: The original submission reported idealized compression ratios that did not account for the storage overhead of sparse matrix indices. This is a significant omission that inflates the perceived performance. While the authors corrected this in the rebuttal, it was a flaw in the initial manuscript.

Theoretical Foundation: The theoretical justifications were found to be somewhat weak and not rigorously proven by one reviewer (8YfX). The analysis could have been better situated within and built upon existing theoretical frameworks from prior work.

(d) Reasons for Decision

My recommendation is to accept this paper as a poster. The decision is based on the significant practical value and strong empirical results of the proposed UltraDelta pipeline. While the concerns about the incremental nature of the technical novelty are valid, the paper's main contribution lies in the successful design and integration of these components into a fully data-free framework that demonstrably pushes the state-of-the-art in delta compression. The extensive experiments provide compelling evidence of the method's effectiveness and broad applicability. The data-free aspect, in particular, is a strong selling point that makes this work a valuable contribution to the community. A poster presentation would be an excellent venue for the authors to share their practical results and engage in discussions about the trade-offs of their approach.

(e) Summary of Rebuttal Period

The rebuttal period featured a lively discussion. Reviewers raised several key points, including concerns about novelty, the calculation of compression ratios, the theoretical underpinnings, and missing baselines.

The authors were responsive and provided a detailed rebuttal. They conducted new experiments on more recent models (Qwen2.5, Llama-3.1) and more challenging tasks, added the requested baseline (DARE + uniform quantization), and, importantly, recalculated the compression ratios using a more realistic CSR format, which addressed a major concern.

However, the discussion surrounding novelty became contentious. The authors' defense of their work, particularly against claims that it was too similar to prior art like ComPEFT, was somewhat rigid. For instance, when reviewers pointed out similarities, the authors' rebuttal focused on differentiating their work rather than acknowledging the conceptual overlap and then providing a detailed, nuanced comparison to highlight their specific innovations. A more effective approach would have been to graciously accept the connection to the related work and then clearly articulate what makes their variance-based metric superior to magnitude-based pruning in a data-free context, or how their trace-norm rescaling is a novel data-free alternative. This would have been more constructive and persuasive.

In my final decision, I have weighed the authors' thoroughness in addressing the empirical concerns more heavily than the lingering debates on novelty. The paper is empirically sound and presents a useful method. However, I strongly urge the authors to revise the final version to incorporate a more balanced and detailed discussion of the related work mentioned by the reviewers. Acknowledging the context and clearly carving out their specific contribution will significantly strengthen the paper.
Decision: Accept (poster)