BitDelta: Your Fine-Tune May Only Be Worth One Bit
Authors: James Liu
Guangxuan Xiao
Kai Li
Jason D. Lee
Song Han
Tri Dao
Tianle Cai
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Quantization
LLM quantization
efficient inference
multi-tenancy
Primary area: optimization_for_deep_networks
TL;DR: Compress the weight delta between a base LLM and fine-tuned LLM to 1-bit.

Abstract: Large Language Models (LLMs) are typically trained in two phases: pre-training on large internet-scale datasets, and fine-tuning for downstream tasks. Given the higher computational demand of pre-training, it is intuitive to assume that fine-tuning adds less new information to the model, and is thus more compressible. We explore this assumption by decomposing the weights of fine-tuned models into their pre-trained components and an additional delta. We introduce a simple method, BitDelta, which successfully quantizes this delta down to 1 bit without compromising performance. This interesting finding not only highlights the potential redundancy of information added during fine-tuning, but also has significant implications for the multi-tenant serving and multi-tenant storage of fine-tuned models. By enabling the use of a single high-precision base model accompanied by multiple 1-bit deltas, BitDelta dramatically reduces GPU memory requirements by more than 10x, thus reducing per-user generation latency by more than 10x in multi-tenant settings. We validate BitDelta through experiments across Llama-2, Mistral and MPT model families, and on models up to 70B parameters, showcasing minimal performance degradation in all tested settings.

BitDelta: Your Fine-Tune May Only Be Worth One Bit
James Liu't Guangxuan Xiao'
Kai Li
Jason D. Lee?
Song Han 1.
Tri Dao Tianle C
MIT
PPrinceton University
SNVIDIA
*Together AI
Abstract
Large Language Models (LLMs) are typically trained in two phases; pre-training on large internet-scale datasets, and fine-tuning for downstream tasks. Given the higher computational demand of pre- rre-training, it is intuitive to assume that fine- tuning adds less new information to the model, and is thus more compressible. We explore this assumption by decomposing the weights of fine-tuned models into their pre- praitrd components and an additional delta. We introduce a simple post- fine-tuning method, BitDelta, which successfully quantizes this delta down to I bit without compromising performance. This interesting finding not only highlights the potential redundancy of information added during fine-tuning, but also has significant implications for the multi-tenant serving and multi-tenant storage of fine-tuned models. By enabling the use of a single high-precision base model accompanied by multiple 1-bit deltas, BitDelta dramatically reduces GPU memory requirements by more than 10x, thus reducing per-user generation latency by more than 10x in multi-tenan tenant settings. We validate BitDelta through experiments across Llama-2, Mistral and MPT model families, and on models up to 70B parameters, showcasing minimal performance degradation all tested settings.
Introduction
After large-scale pretraining, foundation models are typically fine-tuned for specific downstream tasks [16 43 44] This pretrain-finetum paradigm has revolutionized machine learning: L.MSs have not only proven effective for critical tasks such as instruction following and alignment [39], but also performant on a wide array of niche yet highly impactful applications (2). Through fine-tuning, LLMs are adeptly equipped align with distinct user preferences specialized task requirements, showcasing an unprecedented level of adaptability. Thus, the prospect of serving millions of uniquely fine-tuned models, each tailored to individual tasks and user needs, presents promising vision for the future of machine learning.
Realizing this vision is challenging due to two key reasons: 1) Expensive Storage. Each new fine-tuned model is large, even if we have relatively few base models, making them expensive to store and challenging to manage on disk. 2) Expensive Serving. Distinct fine-tuned models each demand significant GPU memory, making it difficult and expensive to concurrently serve such models without noticeable downtime. To tackle these issues, decompose the fine-tuned model weights into the weights of the base pre-trained model and delta induced by the fine- te-innnnng process. By compressing this delta while maintaining model performance, aim to sidestep the prohibitive costs associated with storage and GPU memory demands.
* *Correspondence to jamesll 1 @mit edu tianle. www.bleecaipprinetonncccm edu Tianle's contribution was partially done during consulting at Together AI.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

From the delta decomposition point of view, parameter efficien fine-tuning (PEFT) methods like LoRA [25 24 46 [5 effectively enforce a highly structured and compressed form of delta during fine-tuning, a powerful insight for model serving of PEFT-based fine-tunes. Sheng et al. [49] and fine-tunes.
Chen et explore multi-tenant serving of
Fine-tuned Models
BitDalta
+1.7
-1.8
-1.0
+0.6
-1.
0.1
+12
40.5
+1.1
Base Model
+1.6
+2.
+1.9
+.0
-1.8
+0.5
+2.0
0.2
-10
+0.
+1.
#params x #models 16bits
#params (#models 1bit 16bits)
Figure 1: Overview of BitDelta. BitDelta applies 1-bit quantization to the weight delta between fine-tuned and base models. For each weight matrix, we quantize its delta as its sign bits and trainable high-precision scale factor. The scale factor is initialized to achieve the best approximation error in L2 norm and further refined with a few distillation steps. BitDelta shows minimal degradation in model performance and reduces memory consumption in multi-tenance tenancy serving by representing multiple fine-tuned models with a single
base model and multiple 1-bit deltas.
Nevertheless, recent work has shown that PEFT methods may not yet match the model quality of full parameter fine-tuning, especially on high resource tasks (6), and are fairly sensitive to hyperparameter choice and prompting methods [38]. Biderman et al. 2 show that LoRA's reduced expressivity, although providing desirable regularization, leads to significantly worse performance compared to full fine-tuning in math and programming tasks. As a result, we notice that among the 2307 LLMs the Open LLM Leaderboard [I] with valid README file, only 20% indicate that they exclusively use LoRA. Most models are full parameter fine-tunes model merges 64 28 (5) of full parameter fine-tunes, model merges of LoRA based fine-tunes (which effectively high-rank).
(as of time of writing)
It is also attractive approximate general deltas with low-rank matrices post-training (in partic- ular, post-fine-tuee fiit-inneettnn........ However, experimental results show that this challenging (Table I). as deltas from full parameter fine-tunes tend to be fairly high-rank (Figure
Cumulative Explained Variance
Layer (W₂)
0.8
0.6
We instead draw from the insight that motivates PEFT methods in general: Given the higher com- putational demand of pre-training, it is intuitive to assume that fine-tuning adds less new infor- mation to the model, and is thus much more com- pressible. In fact, we find that we can efficiently quantize the delta to merely I bit with almost no performance drop. We propose BitDelta, an efficient post-training quantization (PTQ) solu- tion that acts on the weight delta between fine- tuned model and its underlying base model.
0.4
0.2
Wasse
Wane Wease
0.0
1000
2000
3000
4000
Index
Figure 2: Cumulative Explained Variance (CEV) plot of a 4096 x 4096 weight delta between Llama 2-7B and Vicuna-7B vl.5. Deltas from full pa- rameter fine-tuning are fairly high rank, making low-rank approximations difficult.
BitDelta consists of two stages: 1) We quantize the delta between a fine-tuned model's weight matrix and base model's 's weight matrix into scaling factor multiplied by binary matrix. Specifically,

---

Table I: Comparison between BitDelta and a SVD based method, with Llama 2-7B and Llama 2-7B Chat as the base and fine-tuned models. BitDelta is performant across the board, whereas the SVD-based method fails to sufficiently capture the fine-tuned information.
Model/Method
TruthfulQA
GSM8K
MT-Bench
Adjusted Average
Llama 2-7B
38.96
13.57
60.53
Llama 2-7B Chat
45.32
22.74
6.56
59.81
BiDDelta-niital
41.10
18.27
6.31
60.70
BitDelta
44.95
20.24
6.47
59.88
SVD-Initial (r 16)
42.57
7.13
4.73
60.58
SVD (r- = 16)
42.42
5.05
4.99
60.71
SVD-Initial (r 128)
43,90
17.82
5.68
60.21
SVD (r 128)
43.32
11.83
5.85
60.58
take the sign of the weight delta form the binary matrix and initialize the scaling factor as the average of the absolute values of the delta, minimizing T.2 quantization error. 2) We further calibrate the scaling factors through model distillation over small calibration dataset while keeping the binary matrices frozen. Despite the small number of trainable parameters and calibration steps, we find that further recovering model quality. Our experiments over 17
this distillation process is effective popular fine-tuned models affirm that BitDelta can be applied across various model types and model sizes with minimal impact on performance.
BitDelta creates opportumities to efficiently serve multiple fine-tuned models with shared servers: By only storing single full-precision base model, and (dynamically) loading and performing batched inference over multiple I-bit deltas, we can efficiently represent multiple fine-tuned models. Compared naively using full precision fine-tuned models, deltas compressed by BitDelta are more than 10x smaller, and can therefore be loaded faster. This addresses the storage challenge. Moreover, since LLM inference memory-bound 32 3, the latency of each decoding step proportional to the GPU memory consumption of the model weights. With efficient CUDA kernel implementation, we can translate this memory reduction into a latency reduction, similar to other quantization methods 19 33
Using the WINTI AfpIe kernel from BitBLAS (58), we improve the multi-tenant serving latency of full full-parameter fine-tuned models by more than 10x.
Finally, we study a few extensions of BitDelta, where we quantize the base model and where we iteratively apply BitDelta. Experimental results show that our method is quite general and can be applied to various use cases.
2 Related Work
2.1 Full Model Compression
Quantization techniques are widely used to reduce memory consumption and im- prove LLMs' generation latency. Xiao et al. (60) implement technique that rescales between activations and parameters, effectively mitigating outlier activations to facilitate smoother quanti- zation. Dettmers et al. [14] develop an approach that decomposes matrix multiplications into 8-bit computations, with an additional 16-bit process for handling outliers. Exploring further, Frantar et al. 19 introduce method that iteratively rounds weight columns to 3-4 bits of precision. Similarly, Lin et al. {3] propose an activation-aware quantization scheme that selectively preserves crucial weights while compressing the majority to 3-4 bits. Kim al. (29] devise a sparse, low-precision pattern focusing on a small yet significant set of weights. Chee et al. [4] utilize incoherence processing to quantize model weights to low bits with minimal impact on performance.
Quantization.
Pruning. Pruning also aims to reduce the memory consumption of neural networks, It accomplishes this by pushing certain parameter values to zero, inducing sparsity in the model BI 21 22 67]. However, these methods may fail to take advantage of modern hardware like GPUs unless using
Adjjssedd Average is over ARC, BBH, HellaSwag, WinoGrande, and excludes TruthfulQA, GSMSK. MTTBeech.
3

---

certain structured sparsity patterns like 2:4 (50%) sparsity [36]. Frantar and Alistarh [8 demonstrate a pruning method on LLMs that successfully utilizes the 2:4 sparsity pattern and achieves a 50% sparsity ratio. It is challenging to obtain higher sparsity while being hardware-friendly
Early work on post-training delta compression. Most related to our work, a few studies explore the idea of post-training delta compression by adopting existing compression techniques like GPTQ, unstructured pruning [22], or even classic lossless compression algorithms. Isik et al. [26] focus reducing the delta size to save storage. Yu et al. [64] utilize pruning to improve model merging applications. Yadav et al. [62] reduces the size of PEFT modules to save storage. Ryu et al. [47 combines quantization with low-rank approximation reduce the delta size. The concurrent and independent work by Yao and Klimovic [65] also explores using delta compression to improve multi- tenant serving, but focuses more on reducing the model loading time from disk to GPU. Compared to existing work, we offer a much simpler and faster method, BitDelta, achieving a compression ratio of more than 10x while also being friendly to modern accelerators.
BitDelta
3.1
Method
BitDelta consists of two stages: 1) We quantize each weight matrix into a scalar multiplied by a binary matrix]] 2) We further calibrate the scalar factors using model distillation. We describe each stage in this section:
1-bit quantization. Let W base, fine E T be weight matrices from the base model and fine- tuned model respectively. We define the weight delta as A Wine Wuse representing the modification in weights post-fine- fineinuning. For efficient representation of this weight delta, we aim to obtain a binarized estimator by encoding its sign bits, denoted as Á:
Â = a @ Sign(A),
(I)
where
[+1, if W
Sign (W
(2)
1- if W
and a is a scaling factor for the entire matrix. To minimize the quantization error of in L2 norm:
A-
A
Σ ([We) - a)
(3)
we initialize a as follows:
a
Σ
Agl-
(4)
mm
Surprisingly, we find that the above quantization approach already does quite well and retains most of the fine-tuned models' performance.
significant role in the low-bit regime. Additionally, mer-matrix L2 weight error is not perfect measure of degradation in overall model quality. We further optimize these scales by performing model distillation to align the output logits of the quantized model to that of the original fine-tuned model. More concretely, we freeze the model weights and optimize for the following objective:
Scale distillation. The scaling factor intuitively plays
a = arg min E Exax [ (x) - Zua (æ; [3]
(5)
In our experiments, we only quantize the linear layers in the Transformer blocks as they contribute the majority of the parameters and computation.

---

Table 2: BitDelta works on Llama-2 and Mistral families and on wide range of model sizes ranging from 7B to 70B parameters. BitDelta works for many types of fine-tuned information, including SFT-based methods, RLHH-based methods, and context extension methods (RoPE scaling). Scale distillation is effective, raising TrutfflQAA/GSMB scores to within 1-2 points of the baseline fine-tune, and MT-Bench scores to within 0.1-0.2 points.
Model
Method
TruthfulQA
GSM8K
MT-Bench
Adjusted Average
Ilama 2-7B
38.96
13.57
60.53
Bascline
45.32
22.74
6.56
59.81
Llama 2-7B Chat
BitDelta-Inite
41.10
18.27
6.31
60.7
BitDelta
44.95
20.24
6.47
59.88
Baseline
50.38
14.18
6.06
57.50
Vicana-78 7B v1.5 16k
MuDelta-Tnitial
45.58
13.95
5.69
58.51
BitDelta
48.75
14.48
6.24
57.64
Llama 2-13B
36.90
22.74
64.68
Baseline
43.95
33.13
6.98
63.99
Llama 2-13B Chat
BitDelta- Initial
41.70
33,36
7,06
64.25
BitDelta
43.47
31.92
6.95
63.96
Baseline
50.38
29.72
6.90
57.5
Vicana-138 v1.5 16k
BitDelta-Initial
41.7
26.76
6.60
64.25
BitDelta
48.75
28.73
6.88
57.64
Baseline
47.17
42.38
6.95
61.61
WicardLM-Ia v1.2
BitDelta-Initial
44.89
42.08
6.73
61.91
BitDelta
46.67
41.62
6.93
61.86
where X a calibration dataset, and Z(-) are the logits of the respective models. Scale distillation is fairly robust to choice X. as the process extremely parameter efficient, and 2) the crucial aspect of the process is to logit match with the fine-tuned model, regardless of the actual text content.
For our experiments, we distill on the C4 dataset 45], consisting of generic internet data, using 800 samples length 128. We use the same subset C4 over all models to control for seed- based variations. We use the Adam optimizer (30] with Ir 10-4 f (0.9, 0.999), 10-8. 1x80 GB A100 GPU is used to distill 7B and 13B models, and 6x80GB A 100 GPUs are used to distill 70B models (2x for finetune, 4x for binarized). Scale distillation is fast; we can compress 70B models in roughly 10 minutes.
3.2 Methodology Cost
Compared full parameter and parameter efficient fine-tuning methods, BitDelta is extremely cheap. While fine-tuning methods require training thousands to millions of parameters, BitDelta only necessitates training single parameter per weight matrix. Moreover, BitDelta operates efficiently with input sequences of length 128, unlike fine-tuning methods that demand longer sequences to saturate the context window (4k, 8k. etc.). Crucially, BitDelta requires only 200 training steps (assuming batch size of 4), which is significantly less compared to the 10000-1000000 steps at higher batch sizes needed by fine-tunin tne-tuning methods. Thus, in terms of methodology cost, we liken BitDelta more to post-training quantization (PTQ) schemes like GPTQ [19 and AWQ [33] rather than full parameter or parameter efficient fine-tuning, while being faster than most PTQ schemes.
3.3 Implication
The ability to compress the delta to merely 1-bit opens up multiple opportunities for improving efficiency, enabling more effective model storage (26] where a single base model can be maintained alongside multiple compressed deltas and facilitating model hot-swapping 7 49 swapping, the base model remains in GPU memory, and compressed deltas are dynamically loaded in accordance incoming requests. In both cases, the compression ratio can be directly translated into reductions in storage needs and loading times.
With hot-
Moreover, BitDelta enables the possibility of multi-tenant serving system like Punica 7 or S- LoRA [49 for general fine-tuned models instead of just LoRA models. Concretely, we consider the scenario where multiple models fine-tuned from the same base model are served with the same
5

---

Table 3: Continuation of Table
Model
Method
TrutfuuQA
GSM8K
MT-Bench
Adjusted Averag
Liama 2-708
44.82
52.69
71.81
Baseline
52.77
47.61
7.12
68.82
Llama 2-70B Chat
BiiDella-Initial
42.38
BitDelta
51.37
48.82
7.06
69.14
Baseline
62.03
56.18
7.07
73.77
Solar-0-70B
BitDeltu-nniial
59.08
56.79
6.79
73.14
BitDelta
62.03
56.63
6,82
73.57
Mistral-7B v0.1
42.60
37.76
65.98
Baseline
55.93
32.75
6.86
60.36
Mistrul-7B v0.1 Instruct
BiiDella-Initial
51.27
38.82
6,54
63.83
BitDelta
55.23
31.54
61.10
Baseline
55.12
34.34
7.18
65.22
Zephyr-7B-8
BitDeltu-Inital
54.53
40.26
6.70
66.12
BitDelta
58.39
31.92
7.00
66,20
Baseline
54.02
54.28
7.36
67.31
Dolphin 2
BiiDelta-Initia
48.14
50.27
7.10
67.58
BitDelta
54.91
52.84
7,20
66.97
MPT-7B
33.37
6.22
57.95
Bascline
40.22
7.96
5,00
56.5
MPT 7B-Chat
BitDelta-Initial
38.96
10.01
4.39
57.11
BitDelta
39.87
8.11
4.94
56.52
server. This setting greatly exploits the GPU resource and saves each fine-tuned model's inference cost when their traffic is low unbalanced. With BitDelta, model with multiple compressed deltas in the GPU memory. Compared to directly serving multiple fine-tuned models, this approach greatly saves memory consumption.
keep sigh-precision base
Since LLM inference follows the memory-boume computation pattern where the generation latency proportional to the GPU memory used by the model weights, the lower memory consumption also suggests the opportunity to improve the serving latency. For example, Punica and S-LoRA exploit LoRA's structure and memory saving by computing the activation product between the shared base weight, and low-rank fine-tuned delta weights separately. Similarly, we decompose the forward pass of each linear layer follows:
X Xi W Whnee
A
(6)
Kernel
where X, and X represent input and output features to the i-th fine-tuned model, and the base model weight and the 1-bit delta are computed separately. For a batch requests, W, can be computed with the classic batched GEMM kernel. We utilize the BitBLAS [58) W WINT1APPIe kernel that allows us to calculate A in a batched setting while keeping the 1 1-bit deltas quantized until they transferred the GPU cache. This kernel fuses the dequantization operation with the GEMM calculation, reducing the data moving overhead by a large factor.
4 Experiments
4.1 Setup
Our primary baselines are the original fine-tuned models without compression. We also compare with 8-bit RTN, 4-bit GPTQ [19], and 2-bit QulP# [54] on evaluations where BitDelta on quantized base models.
Baselines.
Models and datasets. We benchmark fine-tuned models based on the Llama-2 [33, Mistral 27 and MPT [51 model families: Vicuna, Xwin-LM,, Solar-70B, Zephyr, OpenChat 3.5, Dolphin 2.2.1. and OpenOrca [T0 Challenge, 5-shot BBH, 10-shot HellaS wag, zero-shot TruthfulQA, zero-shot LAMBADA, zero-shot
52
56
55
57
23
37
We evaluate on eight tasks: MT-Bench, 25-shot ARC

---

Table 4: Comparison of model responses from Zephyr-7B-B for Question 9 in MT-Bench, a concise advertisement task. BitDelta-Initial is unable to follow the instructions, producing an advertisement that is overly formal and makes no attempt to adhere to the word limit. With the addition of scale distillation, BitDelta successfully produces concise, catchy advertisement slightly over the word limit. *Prompt slightly modified for clarity.
Please create slogan about bioenergy that satisfies the following: Make the tone sound casual 2. Embed an advertisement for a company called "FlexPower" 3. Fewer than 10 words.
Prompt
"Feex' groundbreaking bioenergy solution: Produce fuel without compromising food production!" (10 words)
BitDelta
This headline is catchy, casual, and highlights the unique sell- ing point the... GPT-4 Score:
"FlexPower's groundbreaking technology unlocks the key harnessing the power of renewable bioenergy while minimizing ethical concern'
BitDelta-Initial
GPT-4 Score:
Winogrande, and 5-shot GSM8K 666 12 50
65
We use FastChat [66] to evaluate 220 to evaluate on the other tasks. We denote
40 48
13
on MT-Bench, and use wewallaationhhrane our methodology before scale distillation is applied as BitDelta-Initial.
We primarily focus on high-margin metrics where fine-tuning is significantly impactful and aggregate the other metrics. See Tables 7 to in the Appendix for full results. BitDelta performs quite well on the aggregated metrics, even outperforming the baseline in many cases. However, it's important to contextualize these results with regard to the base model itself, which is also performant these metrics. It's difficult to attribute performance to our methodology or to the underlying base model in such cases. Because of this, we highlight TruthfulQA, GSM8K, and MTT-Bench, which base models tend to struggle on, to show that BitDelta accurately preserves fine-tune information.
4.2 Accurate Quantization
SVD comparison. We compare BitDelta to low rank approx. the weight delta on Vicuna- 7B vl.5. For the low rank approx., we decom- pose A = UEV and approximate Å AB where A UV Σ. B = VEV. During distilla- tion, we treat all entries of the low rank matrices as trainable parameters. We compare against two settings: r 16 (most commonly used) and r = 128 (memory equivalence with Bit- Delta). We find that the low rank approx. fails to fully capture the fine tune information, and underperforms across the board (Table [.. In particular, the low rank approx. heavily underperforms on MT-Bench [10. difficult multi-turn instruction following dataset fairly indicative of real world performance. Interestingly, distillation is not as effective for the low rank approx. compared to BitDelta.
Table 5: BitDelta achieves over 10x compression. We can further compress the embedding and LM head layers, but leave this to future work due to inconsistencies in tokenizer vocabularies.
Base Model
Size
Size
Comp. Factor
CB
10.87
Llama 2-13B
26.03 GB
2.09 GB
12.45
Llama 2-70B
137.95 GB
8.95 GB
15.41
Mistrol-7B vO.1
14.48 GB
1.30
11.14
Main Results. BitDelta is performant across various model families, across a wide range of model sizes, and across many fine-tuning techniques. We benchmark Llama-2, 2, Mistral, and MPT, families, and on models ranging from 7B to 70B parameters. Shown in Table 2 we find that BitDelta is very general and can recover all types of finetune information, including SFT-based methods 43 on Mistral-7B v0.l Instruct, RLHF-based methods [I] on Llama Chat, and context extension methods (RoPE scaling) 8 41 Vicuma-7B vl.5 16k.
We note that GSM8K for BitDelta-Initial on Mistral-7B v0.1 Instruct and Zephyr-7B- is abnormally high; we attribute this to how performant the base model Mistral-7B v0. is on this task in comparison. Scale distillation effective, raising TruthfulQA and GSM8K scores to within 1-2 points of the baseline fine-tune, and generally raising MT-Bench scores to within 1-0.2 points.
7

---

Table We apply BitDelta to Llama 2-7B Chat (with corresponding base model Llama 2-7B), and find it holds up when the underlying base model is quantized at various levels.
Base Model
Method
TruthfulQA
GSM8K
MT-Bench
Adjusted Average
FP16
45.32
22.74
6.56
59.81
Baseline
INT8 RTN
45.02
22.29
6.28
59.63
GPTQ QulP#
44.92
19.48
5.90
58,67
43.69
10.77
5.37
55.82
FP16 A
44.95
20.24
6.47
59.88
Llama 2-7B
INT8 RTN +
44.71
19.86
6.16
59.85
GPTQ + A
42.52
19.94
6.02
59.22
QuIPH
42,00
9.72
4.96
57.44
Case Study. We present a sample response from Zephyr-7B-5 3 in Table highlighting the efficacy of scale distillation. BitDelta-Initial does not have casual tone, and makes no attempt adhere to the word limit. With the introduction of scale distillation, BitDelta exhibits greater instruction following capabilities, producing catchy response that slightly exceeds the word limit.
Quantized base models. Because 8-bit RTN, GPTQ, and QulP# work with 16-bit bit activations, we can keep the fine-tune weights iin and scaling factors a in high precision in the compression process, only quantizing the base weights W As shown in Table 6 find that BitDelta is still performant when applied to quantized base models.
Ablation over fidelity of A. By succes- sively applying BitDelta, treating the com- pressed model from the previous iteration as our base model, we can vary the granularity over the delta, associating it with multiple 1-bit masks. One advantage of doing this is the ability to as-
50.0
47.5
TruthfulQA
45.0
sign arbitrary scale factors each 1-bit mask.
In contrast, when increasing the bit size, scale factors are implicitly fixed with respect to each other. Figure 3 shows how the TruthfulQA of Llama 2-7B plus an increasingly granular delta approaches that of Vicuna-7B vl.S. Full results are in Table 9
42.5
Llama 2-78 + A
40.0
Vicuna-78 7B v1.5
1
2
3 4
6
7
# of Bits A
Figure 3: As the fidelity of A increases, the Truth- fulQA scores of Llama 2-7B + A approaches that of Vicuna-7B v1.5.
4.3 Latency Improvement
For simplicity, consider the setting where each model receives distinct request simultaneously. would be insightful to develop more sophisticated serving systems, which we leave to future work. Following the decomposition in Eq. 6). the W, WITTI A P16 kernel is used to compute the batched matrix multiplication between B binary matrices (N x M) and B high-precs tigh-precision activations (L x N) where N, M are intermediate dimensions and L the sequence length. We focus on decoding latency which dominates runtime, as opposed to prefill latency. Tokens are generated one by one when decoding, meaning L is always For all latency experiments we use a single A100 80GB with power limit set to 500W.
Kernel latency. We benchmark the decoding latency of our kernel, batched linear operation over multiple 1-bit deltas, corresponding to the delta component of Eq. (6). We compare this to the S-LoRA kernel, a batched linear operation over multiple low-rank deltas, and also compare this to the base weight backbone shared over all deltas. We set 128 for S-LoRA, to maintain memory equivalence with BitDelta at = M = 4096.
We profile the latency of the backbone (Wosee and deltas (AX) separately. Although X's memory footprint seales with batch size, negligible compared to Whase which remains constant. For typical low to medium batch settings, which is typical for B x N & N x M. In such settings, the overall memory footprint of the backbone effectively independent of batch size, as shown in Figure (left). This is in contrast with that of the deltas, which scales with the batch size, each

---

Backbone
0.125
Backbone
BitDelta
BitDelta
Latency (ms)
S-LoRA (r-128)
Latency (ms)
0.100
S-LoRA (r=128)
0.06
0.075
0.04
0.050
0.02
0.025
0.00
2000
4000
6000
8000
5
10
15
20
Hidden dim. size
Batch Size (B)
Figure 4: Decoding latency linear layer, as Eqn. 6
Black: Shared base weight backbone W X. Blue: Batched activation-produe with B 1-bit deltas, as in BitDelta. Red: Batched
activation activation-product with low-rank deltas, as in S-LoRA. Left: Ablation over hidden size, assuming N M and B 1. Right: Ablation over batch size, assuming N M 4096.
additional client in the batch adds an additional delta. At batch size (Figure right), backbone latency dominates over delta latency (BitDelta and S-LoRA) due Wase base 16x larger memory footprint compared to single delta. As the batch size increases (Figure 4 left), the combined memory footprint of multiple deltas exceeds Wiese around
=
BitDDelta underperforms slightly compared to S-LoRA in large-batch settings as the LoRA kernel is highly optimized for GPU. We emphasize that closing example, Ma et al. 35 point out that W WNTIIA AFii requires no multiplication operations and that new hardware can be co-designed with this in mind drastically reduce energy/latency costs.
surpassing the gap is tractable. For
400
Naive
10a
Neive
Memory Usage (GB)
Naive (OOM, predicted)
Nalve (OOM, predicted)
Latency (ms)
S-LoRA
200
102
0
Batch Size
1
2
4 Batch Size
16
32
Figure 5: Memory usage of Llama 2-7B, assum- ing each sequence in the batch has a length of 128. Blue: Memory usage of the naive method, separately storing B distinct fine-tuned models. Orange: Projected values for the naive method.
Figure 6: End-to-end decoding latency of Llama 2-7B. Blue: Naive forward pass with B distinct fine-tuned models. Orange: Projected values for the naive forward pass. Green: Batched forward pass with BitDelta. Gray: Batched forward pass with S-LoRA. The naive forward pass succumbs to GPU memory issues higher batch sizes.
Green: Memory usage of BitDelta. The naive forward pass succumbs to GPU memory issues at higher batch sizes.
End-to-end latency. We benchmark the end-to-end decoding latency Llama 2-7B variants with an input length of 128 (we find the decoding latency is less sensitive to the input length), ablated across batch size. For BitDelta and S-LoRA, the forward pass consists of the addition of two components: single backbone pass (batch independent) and a delta pass (scales with batch size).
We compare BitDelta and S-LoRA with naive method that computes each W,X; separately in the forward pass. This naive approach scales poorly with batch size it effectively maintains separate backbone (W,) ) for each client in the batch. Given the substantial memory footprint of the backbone, this leads to significant memory usage as batch size increases. contrast, BitDelta and S-LoRA share single backbone across all clients in the batch, with only the 16x smaller deltas scaling with batch size. This allows for more efficient memory utilization and better performance at larger batch sizes.

---

We find that BitDelta and S-LoRA introduce overhead when the batch size is low. However, BitDelta and S-LoRA scale better and successfully translate the saved GPU memory to improved decoding latency, starting at B 2. This is exacerbated at larger batch sizes, where the naive approach succumbs out-of-memory issues and BitDelta and S-L.RRA are still performant. In the 2 16 regime, used in modern serving solutions, BitDelta has >10x lower per-user decoding latency than the naive method.
Conclusion
We propose BitDelta, simple but effective approach efficiently quantifyings the weight delta arising from the fine-tuning of LLMs down to 1 bit. BitDelta encodes the sign bits of the weight delta and a per-weight matrix scaling factor, which calibrated further through distillation. This allows for representing multiple full-parameter fine-tuned models with one base model and multiple 1-bit deltas, enhancing applications in multi-tenancy serving by reducing GPU memory requirements and improving generation latency. BitDelta is fast and accurate, showcasing minimal performance degradation, and opens new avenues for efficient model deployment and resource utilization in machine learning.
Acknowledgments and Disclosure of Funding
We thank Together AI, MyShell AI, National Science Foundation (NSF), MIT-IBM Watson AI Lab, MIT AI Hardware Program, and MIT Amazon Science Hub for supporting this research. JDL acknowledges support of NSF CCF 2002272, NSF IIS 2107304, NSF CIF 22 2212262, ONR Young Investigator Award, and NSF CAREER Award 214494. KL acknowledges the support from Meta, DataX grant from Princeton University's Center for Statistics and Machine Learning, and innovation grant from Princeton's School of Engineering and Applied Science.
References
[1] Edward Beeching. Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero. Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard. hhtpss
ingg accH4/ /ppp////ppp__aa/// 1l___eederbbardd 2023.
[2] Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John P. Cunningham. Lora learns less and forgets less, 2024.
[3] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple 11m inference acceleration framework with multiple decoding heads. arXiv preprint arXiv: 2401.10774, 2024.
[4] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees. arXiv preprint St0Kiv:2377.33000 2023.
[5] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving. Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. February 2023. doi: 10-48500/ARXIV222220133
[6] Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. Revisiting parameter- efficient tuning: Are really there yet?, 2022.
[7] Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. Punica: Multi-tenant lora serving, 2023.
[8] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023.
[9] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models, 2023.
10

---

[10] Wii-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL
[11] Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023.
[12] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018.
[13] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[14] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.0733: 2022.
[15] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. arXiv preprint 0rXiv:2305.143144 2023.
[16] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186 2019.
[17] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023.
[18] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023.
[19] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.172: 17323, 2022.
[20] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang. and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL httpps://eentt org recorss/ [0256836]
[21] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks, 2015.
[22] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. 2016
[23] Eric Hartford. Cogntttecomputttoos///pppnni222i m.misst hugging face, 2023. URL htttp::/huugiingface. htw.:h/...........ooooo.o..ooona
[24] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019.
[25] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ICLR, 2021.
[26) Berivan Isik, Hermann Kumbong, Wanyi Ning, Xiaozhe Yao, Sanmi Koyejo, and Ce Zhang. GPT-zip: Deep compression of finetuned large language models. In Workshop on Efficient Systems for Foundation Models @ ICM1.2023, 2023. URI. https: stttss:/ooperrvviee .net
Forum?2id-n00ctt022r

---

[27] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[28] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models, 2023.
[29] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-spaae and-sp quantization. arXiv preprint WrXi::2306.07629 2023.
[30] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[31] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touret- zky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan- Kaufmann, 1989. URL/attps: mwwrips. httpppps.ccppppppppppppppp httppppprr.iillsppppeee/e//////p /tll://682bbatt10000222211111633 Paperr. .dff
[32] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. November 2022. doi: 10:4550/ARXI::211.11199
[33] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for IIm compression and acceleration. arXiv preprint arXiv:2306.00998 2023.
[34] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022.
[35] Shuming Ma, Hongyu Wang. Lingxiao Ma, Lei Wang, Wenhui Wang. Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit llms: All large language models are in 1.58 bits, 2024.
[36] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius, Accelerating sparse deep neural networks. arXiv preprint arXiv: 2104.00 2104.08378, 08378, 2021.
[37] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023,
[38] Artur Niederfahrenhorst, Kourosh Hakhamaneshi, and Rehaan Ahmad. Finc-tuning 1mm:: In-depth analysis with 1lama-2, Sep 2023, https:tt::///e/ttt http::/wwwwwwwacccll...c www.aala.aaacc...a...aeeee combblgg/ f "ine-tuning- 11ms- 1almmmoor---me
lora-or or- wwwo-ffll-paaaa furl--arameterr-----eeerrr----ee an- in dent-dephh-aaa www.-iddd--aaalllsltttaee-- anw.haaaaallsssssssalsss
[39] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll I. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02 155, 2022.
[40] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context, 2016.
[41] Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022.
[42] Jianing Qiu, Lin Li, Jiankai Sun, Jiachuan Peng, Peilun Shi, Ruiyang Zhang, Yinzhao Dong, Kyle Lam, Frank P. Lo, Bo Xiao, Wu Yuan, Ningli Wang, Dong Xu, and Benny Lo. Large ai models in health informaties: Applications, challenges, and the future. IEEE Journal of Biomedical and Health Informatics, 27(12)60044-008, 2023. doi: 10. 10.1109/JBH,,2223. 3316750.
[43] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
12

---

[44] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, 1lya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
[45] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer, 2023.
[46]
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
[47] Simo Ryu, Seunghyun Seo, and Jaejun Yoo. Efficient storage of fine-tuned models via low-rank approximation of weight residuals, 2023.
[48] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.
[49] Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and lon Stoica. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285 2023.
[50] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:221000966,, 2022.
[51] MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms., 2023. BRLpttts:://ww/ www.wdatabricks.oo http////ccccccoooooooo///ttt////
[52] Xwin-LM Team. Xwin-Im, 2023. URL https:://ithubb.cem co..sthuu.com/Xaae wwwooo/wwwwwiiiieeeeoeeeooeeee LM/
LM/ TOMM/wwi--Bl
[53] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint MDNNv:2307.0928000 2023.
[54]
Albert Tseng. Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#:: Even better IIm quantization with hadamard incoherence and lattice codebooks, 2024.
[55] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
[56] Upstage. Upstage/solar 7-b0 16bit hugging face, 2023. URL https://huggingface co/ Lppttage/SOLAA---000---112- 16bit.
[57] Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data, 2023.
[58] Lei Wang, Lingxiao Ma, Shijie Cao. Quanlu Zhang, Jilong Xue, Yining Shi, Ningxin Zheng, Ziming Miao, Fan Yang. Ting Cao, Yuqing Yang, and Mao Yang. Ladder: Enabling efficient low-precision deep learning computing through hardware-aware tensor transformation. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 307-323, Santa Clara, CA, July 2024. USENIX Association. ISBN 978 939133---99333-4-3333 URL https://wwas www.i/ usenix.en.... org/ conf http://ooneeeeeesiiieeeeeeeeeeee http://oooooeeeeooo/////////aaa/
lei
[59] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022.
[60] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wa, Julien Demouth, and Song Han. Smoothquant: Aceurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
13

---

[61] Minrui Xu, Hongyang Du, Dusit Niyato, Jiawen Kang, Zehui Xiong, Shiwen Mao, Zhu Han, Abbas Jamalipour, Dong In Kim, Xuemin Shen, Victor M. Leung, and H. Vincent Poor. Unleashing the power of edge-cloud generative ai in mobile networks: A survey of aige services. IEEE Communications Surveys & Tutorials, pages 1-1, 2024. doi: 10.1 10.1109/COMST:20224 3353265.
[62] Prateek Yadav, Leshem Choshen, Colin Raffel, and Mohit Bansal. Compeft: Compression for communicating parameter efficient updates via sparsification and quantization, 2023.
[63] Xiaozhe Yao and Ana Klimovic. Deltazip: Multi-tenant language model serving via delta compression. arXiv preprini arXiv:2312.05215. 2023.
[64] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. arXiv preprint arXiv:2311.0309, 2023.
[65] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019.
[66] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
[67] Michael Zhu and Suyog Gupta. To prune, not prune: exploring the efficacy of pruning for model compression. International Conference on Learning Representations, 2017.
14

---

A Appendix
A.1 Societal Impact
Democratization of Fine-tuned Models. By dramatically reducing the hardware requirements for serving fine-tuned models, BitDelta enables smaller entities to deploy state-of-the-art models more feasibly. This can accelerate innovation and application development across various industries and academic fields, making fine-tuned models accessible wider audience.
Dealignment Mitigation. BitDelta is lossy compression method on the fine-tune information in LLMs. As such, crucial alignment information may be lost the process of compression. We believe this is an important consequence to highlight, as BitDelta democratizes multi multi-tenant applications which may exacerbate this dealignment concern. We encourage further work on evaluation techniques to detect alignment loss in BitDelta, which can lead to the creation of robust methods for its mitigation.
A.2 Additional Experiments
Table 7: We train a r 16 LoRA finetune of Llama 2-7B epoch UltraChat [17 and apply BitDelta with minimal performance degradation. This further shows the generality of BitDelta, which works on parameter-efficien fine-tunes in addition full-parameter fine-tunes.
ModelMethod
HellaSwag
TrutffuQQA
LAMBADA
WinoGrande
GSMSK
Average
MT-Bench
Liama 2-78
$2.56
33.76
78.96
38,96
68.
13.57
50:74
Liama 2-78 UlruChal
54.52
34.14
78.99
46.84
70.83
69.53
14.71
$2.79
BiDell
$4.61
34.28
79.10
46.60
70.58
69.30
15.16
52.80
4.57
Table 8: Full results of the application of BitDelta to quantized base models, corresponding to Table
[llllaSwag
IP16
$3.58
33.84
78.55
45.32
56.58
66.46
22:74
52.44
6.56
Biseline
LLM ia:8()
53.24
33.71
78.62
45.02
66.5
66.06
22.29
52.21
6.26
GPTQ
$1.88
33.54
77.17
44.92
65.32
65.43
19.48
51.11
5.90
RP16
Liona 7B
LLM 1a28()
53.67
33.45
78.57
44.71
06.7
66.35
19.86
51.98
6.16
GPTQ + & $3.45
33.90
78.06
42.52
06.85
65332
1994
$1.22
6.12
2:78 Chat
33.65
22.14
$1.80
6.11
Table 9: Full results of the ablation over the fidelity of A, corresponding to Figure 3
#
ARC
BBH
HellaSwag
TruthfulQA
LAMBADA
WinoGrande
GSM8K
Average
Llama 2-7b
52.56
33.76
78.96
38.96
68.39
68.98
13.57
50.74
1 bit
54.27
36.57
77.90
49.97
65.20
69.46
20.17
53.36
2 hits
$4.44
36.78
77.71
49.69
65.26
69.22
20.62
53.39
3 bits
54.27
36.94
77.58
49.90
65.11
70.09
19.48
53.34
4 bits
54.18
36.94
77.54
49.80
64.95
69.53
19.18
53.16
5 bits
53.67
36.78
77.63
50.15
69.69
18.57
53.10
bits
53.67
77.64
50.20
69.69
18.80
53.13
7
53.74
$0.29
65.15
69.38
18.50
$3.09
8 bits
53.84
36.94
77.51
50.15
64.95
70.17
18.80
53.19
Vicuna-bb v1.5
53.92
37.14
77.45
50.36
64.41
69.61
19.03
53.13
15

---

Table 10: Full results of BitDelta applied to fine-tuned models in the Llama-2 and Mistral families, corresponding to Table
HollaSwag
TruihraQQA
WinGrande
Average
Linmo 2:70
52.56
33.76
78.96
38.96
68.39
63.98
13.59
30.74
2-78
66.14
Baeeline
$3.90
72485
50
64.41
00.61
19.01
59.13
6.04
Rinmo-788
BiDDchsshiiill
$4,09
96
78:47
47.63
66.31
68.75
19.56
53.16
5.67
BriDeta
$4,27
36.57
77.9
49.97
65.2
09:45
20.17
53.36
3.99
35.63
52.32
Pruna-7itrd l6k
Baseline
$7.59
34.05
79.15
48.06
68.02
09:22
10.77
52.41
6.24
Xavin I.M-78 vo.7
RitDcltashiiall
$6.40
33.90
80.26
44.56
69
69. 14
16.08
52:07
5.79
BriDetta
57.94
34.19
79.36
47.62
68.29
09.53
9.02
52.28
6,50
2-138
36,90
72.22
22.74
Basetine
60.32
37.89
82.15
43.95
68.62
70:96
33.13
56.72
6.58
Liuma 2-138 Chut
BiDehta-nnial
$9.90
38.04
82.13
41.70
69.82
71.35
33.36
56.61
7.06
BaDela
59.98
38.03
81.92
43.47
68:46
71.43
31.92
56.46
6.95
Baseline
$7.34
19:47
81.14
50.86
68.48
71.67
20.72
56.95
6.48
Vinuno 12B
BrDelta Intaa
47.63
BriDelta
39.20
81.33
50.39
71.51
30,48
Basetine
54,86
25.63
71.06
30.38
32.32
67.64
29:72
52.52
6.50
Veuan-/200 /ok
59,90
38.04
82.13
41.70
69.82
71.35
26.76
55.67
6.00
BaDela
$4.61
34.68
77.14
48.75
53.89
67.85
25.73
52.24
6.88
Baseline
60.15
40.82
82.58
47.17
69.26
71:00
40.38
59.18
6.95
Wooomi M--2 38 vl.2
BeDDcttshmiiill
69.41
40.27
85.26
44.89
70.23
71.74
43.08
58.98
6.79
BrtDetta
41.30
46.67
68.97
45.54
73.00
Xven L.M-330 vo.1
Intial
63.4
40.33
83.71
43.6
91.26
23.09
26.96
57.45
6.90
BitDelta
62.99
39.81
46.19
90.74
73.50
21:76
56.94
6.83
Liamo 2-70M
67.58
51.67
$7.00
44.82
74.81
77.98
$3.69
65.22
Baseline
52.77
74.00
7.06
Baseline
71.16
55.54
81.76
62.05
75.04
79.32
56.18
69.58
7,00
Solar-9-708
BiDedta-tuiial
69.54
54.52
87.57
59.08
75.37
78.69
56.79
68.79
6.79
BaDelta
70.82
55.06
87.35
62.05
75.86
78.77
56.63
69.50
6.82
78.14
2.70
77.58
Mistral 78 HO.1
61.35
41.18
83.46
42.60
70.10
73,80
37.76
58.61
55.03
32.75
6.56
Mistrai-88
BiDeits-niiall
72.14
Baseline
63,82
39.04
84.33
55.12
66.23
72.69
34.34
59.37
7.18
2epby--78-1
Ritheltslmiiall
63.57
41:87
$3.85
54.55
67.73
79. 56
40.26
60.77
6.70
BaDelaa
65.00
41.68
84.05
58.99
66.33
73.95
$1.92
60.19
7.00
Baseline
$4.39
47.34
72.61
7.34
OpenChan 3.5
BiDeita-liiial
Baseline
64.16
44.49
83.30
$4.02
69.36
75.22
$4.28
63.55
7.36
Dulphin 2.2.1
BitDeta-Inia Inisal
64.16
44.43
84.01
48.14
69.98
75,50
50.27
62.33
7.30
BiDDlla
64.90
43.08
83.44
$4.91
68.39
75.59
$2.84
63.33
7.30
Baseline
83.58
$2.30
73.24
6.70
QpanDro--78
BiDeit--niiall
74.03
7.32
BiDeita
$1.67
74.27
7.05
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The effectiveness of BitDelta is supported by accuracy experiments, perfor- mance benchmarks on the kernel and model level, and through ablations like quantizing the base model and repeatedly applying BitDelta.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: One discussed limitation is how Section 4.3 assumes toy scenario where all B models simultaneously receive a different request, which is indicative of the worst case scenario of multi-tenant serving system, but is not necessarily the most representative scenario.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
17

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully discusses the BitDelta methodology on an algorithmic level, and how to replicate the experiments in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
The paper provides the source code to reproduce the main results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. htcp//blcc/gudeeeeeeeeeeieeeeeee
ssiobbiiiisiooccccyc. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides training details in Section 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The paper does not report error bars / statistical significant tests as of now.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The authors describe the compute resources they used to apply BitDelta to models of varying sizes in Section 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics hattps://neurip::: httpp//s///orrppssccppbbbbbbe htt.ei/eeeehhcsuueeeleeeeeeeee
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and confirm that there are no major ethical concerns with BitDelta.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The authors describe both positive and negative societal impacts of BitDelta in Section A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the creators/original owners of assets the paper uses/references.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The source code associated with the paper is well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdssuuriin//eeearch with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureinn//essarcch with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "MHEspLfDlM", "reviewer": "Reviewer_dEGq", "summary": "The paper proposes to quantize the weight delta of a fine-tuned LLM to 1-bit and observes that the model quality only drops a little.\nDuring the binarization step, it requires calibrating the scaling factor with a few hundreds of distillation steps. This is less than a full fine tuning. Evaluation show that the proposed method produces higher model quality than other post-training quantization techniques such as GPTQ and QuIP.", "questions": "1. Line 133 mentioned the scale distillation is robust to the choice of calibration dataset. Does it mean that it can also use synthetic data? It will be an improvement if the paper presents such an ablation.\n\n2. Line 226: can the quantized delta be merged with the quantized base model? For example, by synchronizing the scaling factors.", "limitations": "The paper does not have negative societal impact as far as the reviewer can tell.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper discovered a nice trade-off between fine-tuned model storage size and model quality. Trading 16x lower weight size by only storing the 1-bit delta for the limited quality drop as reported in the paper is impressive. Importantly, the binarization step has a near-post-training cost, instead of fine-tuning from the beginning. The paper did solid ablation studies on how important the scaling factor calibration is, which well described the effect of each component in the proposed method. The latency study is also appreciated.", "weaknesses": "While requiring low storage size, the proposed method introduces an extra binary-float matmul during inference. Although it is indeed a special kernel and can have much lower inference time than the float matmul, the overhead will become more significant when the base model is low-precision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the kind review! Please find below our point-by-point response regarding your feedback:\n> While requiring low storage size, the proposed method introduces an extra binary-float matmul during inference. Although it is indeed a special kernel and can have much lower inference time than the float matmul, the overhead will become more significant when the base model is low-precision.\n\nThe delta kernel overhead is indeed nontrivial in the regime where the base model is low-precision and $B$ (the number of served models) is small. Similar solutions (S-LoRA, etc.) suffer from the same issue and are actually slower than BitDelta when $B \\leq 4$. Such multi-tenant solutions work best in higher batch settings.\n\nIn terms of overall throughput, with a 16-bit base model (shown in Figure 5), BitDelta outperforms the naive method of running each model separately for all $B>1$. We expect a similar result for quantized base models, though potentially with a higher crossover point.\n\n> 1. Line 133 mentioned the scale distillation is robust to the choice of calibration dataset. Does it mean that it can also use synthetic data? It will be an improvement if the paper presents such an ablation.\n\nIntuitively this seems very doable, considering scale distillation already does well with generic internet data. We will try to include this in the final manuscript.\n\n> 2. Line 226: can the quantized delta be merged with the quantized base model? For example, by synchronizing the scaling factors.\n\nWe don’t see an easy way to do this losslessly, but we’re happy to chat more about this. To us it seems difficult to combine quantized weight matrices that have different scale factors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5yy8EOC52P", "reviewer": "Reviewer_dXr8", "summary": "This paper introduces Bitdelta, a method that enables quantization with just 1 bit. The main idea is to compress the weight delta into a scalar and a binary matrix. The experimental results demonstrate that Bitdelta achieves better performance compared to other techniques.", "questions": "- I am curious about the clear differences between Bitdelta and other compression technologies. For example, can Bitdelta, which adopts the Post-Training Quantization (PTQ) method, be used in combination with other PTQ techniques? Also, what happens if you combine Bitdelta with compression techniques like pruning? It would be beneficial to discuss the relationships between various compression technologies\n- (lines 165-167) The authors said that generation latency is proportional to the GPU memory used. How can this claim be proven? It would be helpful to mention references or provide supporting data\n- I think BitNet has a similar purpose with the design of 1-bit quantization. Comparing Bitdelta to BitNet is required", "limitations": "I think the paper will be strengthened with the measurements on GPU (or memory) consumption are added", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- Easy to read and well-structured.\n- The authors effectively demonstrate that their proposed technique, Bitdelta, performs well compared to existing quantization techniques.\n- It is expected that Bitdelta can be extended to areas such as multi-tenant serving systems.", "weaknesses": "- A significant contribution of Bitdelta lies in its ability to reduce memory consumption through 1-bit quantization. However, the paper lacks detailed evidence to support this claim. It is necessary to compare memory consumption for each compression technique, but currently, only Bitdelta is shown.\n- It would be better for the paper to mention and explain the figures within the proper location of text. For example, Figure 1 and Figure 4 are included but not mentioned in the text. It is hard to understand without sufficient explanation.\n- Table 1 lacks information about the base models for Bitdelta and SVD. It should clearly specify whether these are based on Llama-7B or Llama-7B Chat.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the kind review! Please find below our point-by-point response regarding your feedback:\n\n> A significant contribution of Bitdelta lies in its ability to reduce memory consumption through 1-bit quantization. However, the paper lacks detailed evidence to support this claim. It is necessary to compare memory consumption for each compression technique, but currently, only Bitdelta is shown.\n\nWe would appreciate further clarification on this point -- is the reviewer referencing compression techniques like quantization (GPTQ,AWQ), or potentially the SVD baseline in Table 1? Nevertheless, we are considering delta compression in this work, which differs from classic quantization methods.\n\n> It would be better for the paper to mention and explain the figures within the proper location of text. For example, Figure 1 and Figure 4 are included but not mentioned in the text. It is hard to understand without sufficient explanation.\n\n> Table 1 lacks information about the base models for Bitdelta and SVD. It should clearly specify whether these are based on Llama-7B or Llama-7B Chat.\n\nWe apologize for the unclear presentation and thank the reviewer for the suggestion. We will fix these issues in the final manuscript. The Table 1 results are based on compressing the weight delta between Llama-7B and Llama-7B Chat, so the resultant model is an approximation of Llama-7B Chat.\n\n> I am curious about the clear differences between Bitdelta and other compression technologies. For example, can Bitdelta, which adopts the Post-Training Quantization (PTQ) method, be used in combination with other PTQ techniques? Also, what happens if you combine Bitdelta with compression techniques like pruning? It would be beneficial to discuss the relationships between various compression technologies\n\nIn Table 6 we have results where we apply PTQ in conjunction with BitDelta -- we found that the two methods are fairly orthogonal. We expect other methods (pruning, etc.) that also apply to the base model to also be fairly orthogonal. \n\nRegarding further compression of the delta, it may be possible to employ techniques such as vector quantization and incoherence processing (similar to QuIP# [1]) to achieve accurate sub 1-bit deltas. However, we have to be cognizant of the hardware friendliness of these methods, and whether the reduced delta size outweighs the associated kernel overhead.\n\n> (lines 165-167) The authors said that generation latency is proportional to the GPU memory used. How can this claim be proven? It would be helpful to mention references or provide supporting data\n\nThe memory bound nature of LLM decoding is well documented [2]. During the decoding phase (on modern GPUs), the time taken to transfer weights and KV caches to GPU registers far outweighs the time needed to compute the associated skinny matrix multiplications. \n\nAWQ [3] leverages this to translate a reduction in memory footprint (through weight quantization) to a ~3x wall clock speedup. We likewise translate a reduction in memory footprint (through representing multiple fine-tuned weights with just one base weight and multiple compressed deltas) to a ~10x wall clock speedup when concurrently serving 16 models.\n\n> I think BitNet has a similar purpose with the design of 1-bit quantization. Comparing Bitdelta to BitNet is required\n\nBitNet fundamentally has a different purpose in that they propose a new architecture based on 1-bit weight entries for LLM **pretraining**, with the goal of showing superiority over conventional 16-bit pretraining. BitDelta differs in that it compresses the weight delta of two **existing pretrained** 16-bit models to 1-bit, while keeping the base model in 16-bit precision, with the goal of unlocking efficient multi-tenant serving of full-parameter fine-tuned models. The two are related only in that they both use $W_\\text{INT1}$ matrix operations.\n\n[1]: QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks: https://arxiv.org/abs/2402.04396\n\n[2]: LLM Inference Unveiled: Survey and Roofline Model Insights: https://arxiv.org/abs/2402.16363\n\n[3]: AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration: https://arxiv.org/abs/2306.00978", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eXBe3W2NTt", "reviewer": "Reviewer_dvPC", "summary": "Aiming at storage and serving overhead caused by multiple finetuned LLMs for various downstream tasks, this paper proposes a memory-friendly model compression method namely BitDelta which binaries the delta of each weight matrix and uses self-distillation to learn optimal scaling factors.", "questions": "1) The author should describe the advantages of BitDelta compared to PEFT methods, such as QLoRA, QA-LoRA, and LoftQ. Because they don't require saving the fp16 pretrained model, which results in more efficient storage utilization. \n2) More detailed description should be given, such as GPTQ+Δ in Table 6. Additionally, the performance of FP16+Δ being superior to 4-bit GPTQ and 2-bit Quip does not entirely prove that directly quantizing the base model is impractical, as INT8-RTN still shows better performance. If we consider 8-bit GPTQ, or other sota PTQ methods such as Omniquant, AWQ, it might also perform better while reducing memory usage by half. Therefore, it is necessary to provide additional arguments from other perspectives to explain why directly quantizing the base model is not preferable. \nExtensive additional experiments are not necessary, but it is important to clearly explain the aforementioned issues.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "BitDelta innovatively decomposes a fine-tuned model into its pretrained version and an additional weight matrix delta and then successfully binary delta to reduce memory overheads while preventing large performance degradation.", "weaknesses": "The paper lacks innovation and has several points that do not hold up under scrutiny. \n\nFor the first contribution, the paper proposes decomposing multiple fine-tuned models into a shared pretrained model and their respective deltas, and then binarizing the deltas. The specific decomposition method is not described in the paper. Based on experience, this can be understood as using LoRA to fine-tune LLMs and binarizing the learned low-rank matrices. This is not novel, as there are already existing low-bit model fine-tuning methods, such as Q-LoRA and QA-LoRA, which are more memory-friendly and do not require retaining an additional fp16 model.\n\nRegarding the second contribution, which involves using self-distillation to learn scaling factors, the paper's description is insufficient. For example, the initialization method of these factors is not well-explained. Furthermore, if the entire fp model's output is used to supervise the quantized model, the computational resource consumption is high. The paper could explore layer-wise optimization mechanisms to address this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the kind review! Please find below our point-by-point response regarding your feedback:\n\nWe would first like to clarify a misconception that significantly impacts the evaluation of our work: **we do not fine-tune our own models** in this paper. Rather, we take existing popular SOTA full-parameter fine-tuned models, and compress the weight delta between it and its underlying base model. This is done in a hardware friendly way, such that inference on the deltas is fast and performant.\n\nFundamentally, the goal of BitDelta is to unlock the efficient multi-tenant serving of **existing SOTA full-parameter fine-tuned models** on platforms like HuggingFace. Methods like QLoRA are impactful in that they democratize fine-tuning in resource constrained settings, trading off accuracy for decreased memory footprint. This is most useful in settings like fine-tuning your own models locally (eg. when you’re in a niche domain) and on the edge. Because of the differing target use cases, it’s hard to make a useful comparison that adequately captures the value propositions of both methods.\n\nW1:\n> The specific decomposition method is not described in the paper. \n\nWe describe the decomposition method in Section 3.1, lines 117-126. The base model and fine-tuned model weights are known apriori, and the decomposition is an element-wise subtraction. \n\n> Based on experience, this can be understood as using LoRA to fine-tune LLMs and binarizing the learned low-rank matrices. This is not novel, as there are already existing low-bit model fine-tuning methods, such as Q-LoRA and QA-LoRA, which are more memory-friendly and do not require retaining an additional fp16 model.\n\nWe do not fine-tune the LLMs ourselves, please see our beginning statement.\n\nW2:\n> Regarding the second contribution, which involves using self-distillation to learn scaling factors, the paper's description is insufficient. For example, the initialization method of these factors is not well-explained.\n\nWe apologize for the unclear presentation and will revise lines 122-126. The initialization of the scaling factors is set to the mean of the absolute values of the per-tensor weight entries. This is done to minimize weight quantization error with respect to L2 norm.\n\n>  Furthermore, if the entire fp model's output is used to supervise the quantized model, the computational resource consumption is high. The paper could explore layer-wise optimization mechanisms to address this issue.\n\nIn Section 3.2 we describe the methodology cost, and conclude that the total computational cost is fairly similar to other PTQ methods. To be clear, the methodology assumes the existence of a trained base model, and a trained fine-tuned model, and does not fine-tune in the sense that Q-LoRA + its variants do. \n\nOne potential optimization not mentioned would be to precompute the teacher model logits (which should not be a storage issue given the low sample lengths + number of samples), which would halve the memory footprint. Nonetheless, we agree that it may be possible to further optimize this. For example, we could search for the optimal per tensor scaling factor through a grid search, loading one tensor at a time, similar to AWQ [1].\n\n> Q1: The author should describe the advantages of BitDelta compared to PEFT methods, such as QLoRA, QA-LoRA, and LoftQ. Because they don't require saving the fp16 pretrained model, which results in more efficient storage utilization.\n\nWe do not fine-tune the LLMs ourselves, please see our beginning remark. Additionally, as shown in Table 6, BitDelta also works well in conjunction with quantized base models.\n\n> Q2: More detailed description should be given, such as GPTQ+Δ in Table 6. Additionally, the performance of FP16+Δ being superior to 4-bit GPTQ and 2-bit Quip does not entirely prove that directly quantizing the base model is impractical, as INT8-RTN still shows better performance. If we consider 8-bit GPTQ, or other sota PTQ methods such as Omniquant, AWQ, it might also perform better while reducing memory usage by half. Therefore, it is necessary to provide additional arguments from other perspectives to explain why directly quantizing the base model is not preferable. Extensive additional experiments are not necessary, but it is important to clearly explain the aforementioned issues.\n\nWe did not intend to suggest that directly quantizing the base model is not preferable, and we apologize for any confusion. Our goal was to demonstrate the orthogonality of BitDelta to quantizing the base model. Given a more stringent memory constraint, providers are able to quantize the base model in addition to applying BitDelta, without significant degradation in performance. In fact, applying 8-bit quantization might have such a negligible impact on accuracy that it could be preferable purely from an inference speed perspective, regardless of memory constraints.\n\nOur corollary statement ($FP16+\\Delta$ outperforms $GPTQ$) was an interesting observation we noticed. However, we acknowledge this may not necessarily be true for stronger baselines (AWQ, Omniquant, etc.). We will moderate our claims in the final manuscript to better reflect the \nscope of this finding.\n\n[1]: AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration: https://arxiv.org/abs/2306.00978", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dPQ52Hbv8T", "reviewer": "Reviewer_b1KL", "summary": "This paper introduces BitDelta which quantizes the aggregated weight updates (the authors call it “delta”) to 1-bit after full fine-tuning. The paper claims that the approach has two applications: 1. First, it shows that the delta is highly redundant and 2. It is useful in the multi-client-single-server applications where the high precision model will be saved on the server and each client only store it’s own 1-bit delta. They show that in this scenario, the generation takes up to 10x memory reduction (and similar in latency improvement).", "questions": "Please check the \"weaknesses\" section. I would be happy to discuss and change my score.", "limitations": "I think the authors should define and present the limitations of the method more clearly.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper studies an important problem for fine-tuning LLMs with a new approach where they quantizes the delta to 1-bit,\n2. The experiments are done on the most important models like LLaMa-2 and Mistral,\n3. The paper provides a kernel for INT1xFP16 matrix multiplication.", "weaknesses": "1, The author claim that BitDelta shows the potential redundancy of information added during fine-tuning. However, this is not a new finding and almost all PEFT approaches (for example LoRA) are based on this fact. \n\n2. It seems that the paper completely missed the full fine-tuning costs and just measured the memory/latency of the serving step. However, I would suggest to have “apple to apple” comparisons and compare the fine-tuning+serving cost (including memory and runtime) of “full-fine tuning+1bit optimization+serving” against “fine-tuning+serving” in PEFT schemes (like LoRA). \n\n3. In Table 6, the paper shows higher accuracy for FP+delta compared to GPTQ. Again, I would rather like to see memory Vs. accuracy tradeoff in such comparisons as a function of number of clients. The fact is FP+delta does not have the same memory as GPTQ (please current me if I am wrong).\n\n4. It would be nice to have a performance model for the latency of the decoding as a function of client numbers. This is also missed and needs to be include as this is the main claim of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the kind review! Please find below our point-by-point response regarding your feedback:\n\n> 1, The author claim that BitDelta shows the potential redundancy of information added during fine-tuning. However, this is not a new finding and almost all PEFT approaches (for example LoRA) are based on this fact.\n\nWe agree that the redundant fine-tune information angle is not new, hence our analogy that LoRA enforces structure **during training**. However, we believe there is nontrivial novelty in using this idea to accurately quantize the weight delta **post training to 1-bit** for full-parameter fine-tuned models, and successfully translating this reduced memory consumption to a >10x wall clock speedup in multi-tenant settings.\n\n> 2. It seems that the paper completely missed the full fine-tuning costs and just measured the memory/latency of the serving step. However, I would suggest to have “apple to apple” comparisons and compare the fine-tuning+serving cost (including memory and runtime) of “full-fine tuning+1bit optimization+serving” against “fine-tuning+serving” in PEFT schemes (like LoRA).\n\nWe respectfully disagree with the statement's premise -- to clarify, we do not fine-tune our own models, and instead target existing popular SOTA fine-tuned models on platforms like HuggingFace, as this setting is where our methodology’s downstream applications are most relevant. As such, the effective cost of such models is amortized over many people around the world. If we were in a different setting and needed to fine-tune our own models from scratch (eg. in a niche domain), then we would agree that a full apples-to-apples comparison including the cost of fine-tuning would be more appropriate.\n\nHowever, the reviewer raises an interesting point and we plan to clarify this in the final manuscript to ensure the value proposition of our work is more accurately understood.\n\n> 3. In Table 6, the paper shows higher accuracy for FP+delta compared to GPTQ. Again, I would rather like to see memory Vs. accuracy tradeoff in such comparisons as a function of number of clients. The fact is FP+delta does not have the same memory as GPTQ (please current me if I am wrong).\n\nThe reviewer is correct in that $FP16+\\Delta$ has a different memory footprint than $GPTQ$. The crossover point would be about 5 models. Serving separate quantized models is mainly relevant in low-batch (low number of clients) and low-memory settings. However, as shown in Table 6, BitDelta can also be applied to quantized base models, which is a viable solution in such settings. For example, when serving 3 models, it is preferable to represent them as one 8-bit base model plus three 1-bit deltas, instead of three separate 4-bit models, in terms of both accuracy and memory.\n\nNonetheless, BitDelta is not intended to be useful in this regime (low-batch + low-memory), and the quantization ablation mainly serves to show the robustness of the method in terms of accuracy. However, the remark that $FP16+\\Delta$ outperforms $GPTQ$ may mislead readers to overgeneralize, which we will address in the final manuscript.\n\n> 4. It would be nice to have a performance model for the latency of the decoding as a function of client numbers. This is also missed and needs to be include as this is the main claim of the paper.\n\nThe End-to-End Latency section (Figure 5) addresses decoding speed as a function of batch size (number of clients). We're more than happy to provide additional results if this is not what the reviewer is expecting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviews are mostly favourable, although varying in their level of support for the paper. A couple of reviewers misunderstood initially the main idea and I urge the authors to pay special attention to this aspect when revising the paper. Having said that, after reading through the paper myself I did not see any parts where the paper would be impossible to understand so I do not think any misunderstandings like those could be a base for rejection. To make it more evident, both reviewers dXr8 and dvPC revised their scores after the rebuttal (although dXr8 did so silently), suggesting there are no fundamental flaws in that regard in the paper.

Only reviewer b1KL kept their score and mentioned the paper should make it clearer that its focus is on serving, and that evaluation needs to be redesigned - however, it is not very clear to me what the reviewer here had in mind. With regard to the positioning of the paper, as mentioned above, although there might be some room for improvement I do not think it warrants rejection. With regard to evaluation, the biggest shortcoming seems to be related to the fact that the proposed method only makes sense in a multi-tenancy setting, but strictly speaking no such case has been evaluated. While this might see as a serious shortcoming, given its relevance for the central point of the paper, it is important to note that the current results are sufficient to infer the expected performance in a multi-tenancy setting. More specifically, the criticism is related to the memory consumption (dXr8, b1KL) and latency (my own comment).

With respect to the memory, while Table 5 provides the core information needed to understand and validate benefits of the method, the fact the authors do not provide any end-to-end memory, especially while varying the number of fine-tuned models, is a notable omission and should be fixed.

With respect to latency, I find the current comparison to be very confusing. After carefully reading the entire section multiple times, I do not think there is any serious mistake there, but it is unnecessarily difficult to understand. For example: "batch size" referring to the number of models is very confusing, referring to only the int1-fp16 part of the method as BitDelta in Figure 4, but encompassing the entire W_baseX + deltaX in Figure 5 does not help, etc. (in general, it took me much longer than reasonably needed to understand why BitDelta vs. Backbone in Figure 4 exhibits the exact opposite trend to BitDelta vs. Naive in Figure 5). I would urge the authors to rewrite this section thoroughly.

Having said that, both of the aspects above seem to be matters pertaining more to the presentation of the current results rather than omission of important experiments needed to support claims made by the paper. Given that and the fact that, after the rebuttal, overall motivation or contributions do not seem to be questioned by the reviewers, who largely lean towards acceptance, I also recommend acceptance. Having said that, I do trust the authors to revise their paper accordingly to the feedback from the review process, as it seems there is many minor places that should be improved before publication.
Decision: Accept (poster)