Title: Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression
Venue: NeurIPS 2025
Accepted: yes
Keywords: Delta Compression

Pruning

Quantization
Abstract: With the rise of the fine-tuned–pretrained paradigm, storing numerous fine-tuned models for multi-tasking creates significant storage overhead.
Delta compression alleviates this by storing only the pretrained model and the highly compressed delta weights (the differences between fine-tuned and pretrained model weights). 
However, existing methods fail to maintain both high compression and performance, and often rely on data.
To address these challenges, we propose UltraDelta, the first data-free delta compression pipeline that achieves both ultra-high compression and strong performance. 
UltraDelta is designed to minimize redundancy, maximize information, and stabilize performance across inter-layer, intra-layer, and global dimensions, using three key components:
(1) Variance-Based Mixed Sparsity Allocation assigns sparsity based on variance, giving lower sparsity to high-variance layers to preserve inter-layer information.
(2) Distribution-Aware Compression applies uniform quantization and then groups parameters by value, followed by group-wise pruning, to better preserve intra-layer distribution.
(3) Trace-Norm-Guided Rescaling uses the trace norm of delta weights to estimate a global rescaling factor, improving model stability under higher compression.
Extensive experiments across 
(a) large language models (fine-tuned on LLaMA-2 7B and 13B) with up to 50$\times$ compression, 
(b) general NLP models (RoBERTa-base, T5-base) with up to 224$\times$ compression,
(c) vision models (ViT-B/32, ViT-L/14) with up to 132$\times$ compression, and
(d) multi-modal models (BEiT-3) with 18$\times$ compression, 
demonstrate that UltraDelta consistently outperforms existing methods, especially under ultra-high compression.
Code is available at https://github.com/xiaohuiwang000/UltraDelta.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes UltraDelta, a new data-free compression pipeline designed to reduce the storage overhead of fine-tuned models by aggressively compressing delta weights (the difference between a fine-tuned model and its base pretrained model).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper addresses an important problem in delta parameter compression for large-scale fine-tuned models.\n\n2. It includes an extensive empirical evaluation covering multiple model families and tasks, demonstrating broad experimental effort.\n\n3. The manuscript is generally well-structured and easy to follow.\n\nLimitations:\n\n1. Limited Novelty: The core ideas—layer-wise importance for adaptive pruning [1], delta parameter quantization [2], and rescaling factor adjustments [3]—are each incremental extensions of established compression and pruning techniques. Furthermore, their combination has already been studied in works such as ComPEFT [4], which uses sparsification and ternary quantization for delta compression. This paper does not provide novel methodology beyond re-combining existing ideas, nor does it adequately acknowledge or cite prior works (for example ComPEFT [4]).\n\n2. Weak Theoretical Foundation: The theoretical rationale, such as the variance–entropy relationship, is presented in an overly simplistic manner without rigorous proofs or formal guarantees. What's more, [3] already provides a comprehensive theoretical treatment of variance’s effect on pruning error; the current paper should reference and build on that work directly.\n\n3. The discussion in related work is not proper that methods like DAREx [3] and BitDelta [2] can operate with only small amounts of unlabeled data in an training-free unsupervised fashion, providing practical data efficiency and easy to implement, so this is not a disadvantage.\n\n4. Unclear Rescaling Factor Determination: The trace-norm-based rescaling factor γ is claimed to be inversely proportional to the trace norm, but it is not clear why it is not simply set equal to its inverse. As discussed in lines 549–552 in appendix, the factor appears manually chosen within a heuristic range. A more detailed derivation or empirical justification is necessary to clarify how γ should be set in practice.\n\n[1] LAYER-ADAPTIVE SPARSITY FOR THE MAGNITUDE-BASED PRUNING\n\n[2] BitDelta: Your Fine-Tune May Only Be Worth One Bit\n\n[3] DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models\n\n[4] ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization\n\nQUESTIONS:\nSee limitations\n\nLIMITATIONS:\n1. Limited Novelty: The core ideas—layer-wise importance for adaptive pruning [1], delta parameter quantization [2], and rescaling factor adjustments [3]—are each incremental extensions of established compression and pruning techniques. Furthermore, their combination has already been studied in works such as ComPEFT [4], which uses sparsification and ternary quantization for delta compression. This paper does not provide novel methodology beyond re-combining existing ideas, nor does it adequately acknowledge or cite these prior works (for example ComPEFT [4]).\n\n2. Weak Theoretical Foundation: The theoretical rationale, such as the variance–entropy relationship, is presented in an overly simplistic manner without rigorous proofs or formal guarantees. What's more, [3] already provides a comprehensive theoretical treatment of variance’s effect on pruning error; the current paper should reference and build on that work directly.\n\n3. The discussion in related work is not proper that methods like DAREx [3] and BitDelta [2] can operate with only small amounts of unlabeled data in an training-free unsupervised fashion, providing practical data efficiency and easy to implement, so this is not a disadvantage.\n\n4. Unclear Rescaling Factor Determination: The trace-norm-based rescaling factor γ is claimed to be inversely proportional to the trace norm, but it is not clear why it is not simply set equal to its inverse. As discussed in lines 549–552 in appendix, the factor appears manually chosen within a heuristic range. A more detailed derivation or empirical justification is necessary to clarify how γ should be set in practice.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes UltraDelta, an efficient method to compress delta parameters without additional data. Extensive experiments demonstrate that by assigning layer-wise sparsity rates based on delta parameter variance and rescaling the pruning factor $\\gamma$, UltraDelta achieves comparable or better performance to the original fine-tuned model under extreme compression ratios.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. The various experiments substantiate that UltraDelta can achieve better or comparable performance with the original fine-tune model even under an extreme compression ratio.\n2. The UltraDelta method is efficient because it does not use any additional data.\n\nWeakness:\n\n1. The paper employs sparsification and quantization in its methodology; however, it calculates the compression ratio using techniques from the quantization literature, overlooking the storage implications associated with the positions of non-zero elements. In contrast, conventional sparsification studies typically emphasize the sparsity ratio. Consequently, the actual compression ratio is significantly lower than what is reported.\n2. Missing Baseline: The paper does not include a comparison with a simple baseline such as DARE combined with quantization.\n3. The method of UltraDelta is not novel. Here is the reason:\n  * The idea of  MSA method assigns different sparsity rates to different layers based on their importance score, has been applied in many previous works[1,2].\n  * The methodology and analysis of TNGR closely resemble those of prior work [3], which also derived an upper bound for the error $\\varepsilon$ and proposed an estimation method for $\\gamma$ using auxiliary data. The key distinction between TNGR and DAREx[3] lies in the fact that TNGR does not rely on any additional data. However, the estimation function of $\\gamma$ and the setting of the value between 0.5 and 1.0 of TNGR are empirical, which constrains the novelty of TNGR.\n4. The models used in the experiments are kind out of dated. More recent models like Qwen2.5 should be taken into consideration.\n5. This paper presents experiments on models with easily compressible delta parameters. It is recommended that UltraDelta be evaluated in more challenging scenarios, such as delta parameters with large norms, as indicated in [3].\n\n\n---\n[1]: AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models\n\n[2]: Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment\n\n[3]: Revisiting Delta-Parameter Pruning For Fine-Tuned Models\n\nQUESTIONS:\n1. DAREx reports that DARE suffers a significant performance degradation when the delta parameters exhibit large mean magnitudes. Does UltraDelta encounter a similar issue? \n2. I argue that parameter grouping in the DAC component is unnecessary, as random pruning does not alter the original parameter distribution.\n3. Figure 4 employs cosine similarity to assess how well compression methods preserve internal representations. However, despite MP(80×) exhibiting lower final-layer embedding similarity than BitDelta, BitDelta underperforms on the CoLA task (43.51 vs. 36.83, Table 8). A similar pattern is observed for DARE(80×) vs. Delta-CoMe (57.90 vs. 57.67). How should this discrepancy be interpreted?\n4. P6 Equation (11), the symbol $\\odot$ means Hadamard product or matrix multiplication?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the task of delta compression, where a base model is separately finetuned for several different tasks, and the resulting models have to be stored with as little storage overhead as possible by saving only (a compressed version) of the difference between the base model’s weights and the finetuned weights. The paper proposes UltraDelta, which combines and improves upon two lines of prior work: quantization and pruning. UltraDelta consists of three main improvements over prior methods: (i) a variance-based sparsity allocation (lower sparsity for high-variance layers), (ii) a distribution-aware quantization (uniform quantization followed by group-wise pruning), and (iii) rescaling the quantized weights using the trace norm of the delta weights as a scaling factor. The results across a wide range of models, tasks, and domains show that UltraDelta achieves very high compression ratios (up to 800x) while maintaining (sometimes even increasing) strong performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The paper conducts an extensive empirical validation of the proposed method across models of different scales, types, and tasks (LLMs, vision models, and multi-model models), leaving no doubt about the superiority of the proposed approach.\n* The paper is very well-written and easy to follow. Figure 2 is a work of art and helps the reader understand the different components of the proposed method.\n* The proposed method addresses key weaknesses of prior approaches: (i) treating all layers equally, (ii) disrupting intra-layer weight distributions, and (iii) instability under high compression.\n* The paper provides good theoretical motivation and empirical validation, including ablation studies, for the proposed method.\n\n**Weaknesses**\n\n* The paper claims that its method does not affect the inference stage, which is not entirely true. Concretely, at inference time, the delta weights need to be added to the base model’s weights, which introduces a (non-negligible?) memory and compute overhead, especially for large models.\n\nQUESTIONS:\n* Intuitively, why is it important to “maintain the relative proportions across different quantized weight values”? Is there a good explanation for why this should achieve better performance than, e.g., uniform or magnitude-based pruning?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents UltraDelta, the first data-free delta compression pipeline that claims to achieve both ultra-high compression and strong (accuracy) performance. There are three key components in UltraDelta: 1) Variance-Based Mixed Sparsity allocation; 2)  Distribution-Aware Compression; 3) Trace-Norm Guided Rescaling. The resulting compression ratios across multiple different categories of models are impressive: 133x for LLMs; 800x for general NLP models; 400x for ViT models; and 40x for multi-modality models.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength:\n- The problem itself is important to not only the ML research community but also the system & MLSys community. It is also a timely and very relevant topic. As someone working on delta compression in the past, some of the questions that the authors try to resolve are the issues asked by reviewers in our previous papers. Overall I think this paper is a nice addition to the NeurIPS 2025 program.\n\n- The paper is very well written and easy to follow. I particularly like the figures across the paper as they are easy to read and accessible.\n- It’s great to see the authors provide not only empirical results, but also theoretically analyze the effect of MSA. I think it also suggests why model deltas are more compressible than the entire models (based on the findings from DeltaZip & DeltaDQ, the variance or standard deviation in model deltas are significantly smaller than the entire model). I have a few suggestions & follow-up questions regarding this (please see questions & weakness).\n- The experiments conducted are extensive, covering a wide range of tasks / models including not only LLMs, but also other NLP models & \n\n# Weaknesses\n\n- [NIT]: In “Theoretical Analysis of MSA”, I wonder if the authors could show some empirical results as well. For example, say we want to maintain 99% accuracy/cosine similarity, and x-axis is the standard deviation, y-axis is the sparsity we can achieve.\n- [Follow Up on above & NIT]: I think this entropy explanation is nice -- however I think there're two potential improvements: 1) If I understood it correctly, the entropy viewpoint suggests that if you apply lossless compression, then the lower the variance, the lower bit rate you can achieve. However, across your paper you are actually apply lossy compression. I think another viable approach is to try rate-distortion theory. 2) However these theories only give you lower bound -- and it does not necessarily hold for the exact compression method you used here -- some empirical results would enhance it. See [this blog post (Why Delta Compression Works)](https://about.yao.sh/posts/why-delta-works/) as a reference.\n\n- One of the main weaknesses is that it’s unclear how efficient the proposed approach will be, when deployed in a real serving engine, i.e., whether the compression ratio can be translated well into serving performance (throughput; latency) improvements.\n\n- An example with LLMs: BitDelta quantizes the model delta to 1 bit without sparsity vs. UltraDelta quantizes/sparsifies the model delta to 4 bit + 95% of sparsity -- which one is faster? Although UltraDelta achieves higher compression ratio, it won’t be faster without a properly implemented kernel to achieve the highly sparse computation. Serving performance is also a key focus in DeltaZip and therefore they chose to use structured sparsity, which only gives 50% sparsity. Ultimately, we want faster inference, instead of just high compression ratio.\n- I don’t think this should be a show-stopper, as this paper does not focus on serving performance, however, I think it is fair to mention this in the limitations section to give enough context to the readers. Ultimately I think this paper lays a good foundation for system/ml co-design.\n\n- I would also strongly suggest authors to place this work into more context in the introduction, as the ultimate goals of these works (DeltaZip, BitDelta, DeltaDQ, etc) are to improve deployment efficiency, UltraDelta offers a significant higher compression ratio which can be leverage to further improve the deployment efficiency.\n- [NIT] I am a bit surprised the proposed approach consistently outperforms the uncompressed baseline. Although I acknowledge there might be some implicit regularization effect in compression as described by the authors -- I think some more empirical results on more controlled experiments (e.g., those models are fine-tuned by the authors with varying steps, and then compressed by the authors - to show if the models are overfitting/underfitting or not) can be beneficial. \n- Re: Hyper-parameter selection. I wonder how the authors select the sparsity level in their experiments. For example, with LLMs the authors used 95%/97% sparsity; 99% for other NLP models; 90% for multi-modality models. Do the authors perform an extensive (but might be expensive) grid search of these parameters in their experiments? Or are there any other signals used in the process?\n- For LLMs, I think some more challenging tasks should be considered in the evaluations, such as AIME/GQA, etc.\n\nQUESTIONS:\n- In the “Theoretical Analysis of MSA”, the authors explicitly focused on sparsity by saying “...should be assigned smaller sparsity”. I wonder if the same conclusion holds for quantization? For instance can we say: “if a layer exhibits larger standard deviation, we should use higher bit width”?\n-  I am also wondering if the variance could be a good signal for selecting sparsity level / bitwidth? So instead of saying: “should be assigned smaller sparsity”, can we suggest: “should be assigned a sparsity in the range [a,b]?”\n\nLIMITATIONS:\nI think it is fair to mention this work is primarily on compression ratio and serving performance improvement could be a future work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}