FM-Delta: Lossless Compression for Storing Massive Fine-tuned Foundation Models
Authors: Wanyi Ning
Jingyu Wang
Qi Qi
Mengde Zhu
Haifeng Sun
Daixuan Cheng
Jianxin Liao
Ce Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: model compression
lossless compression
cloud storage
Primary area: infrastructure
TL;DR: To mitigate cloud model storage overhead, we propose a novel lossless compression scheme FM-Delta to compress massive fine-tuned models stored in cloud, significantly saving cloud storage costs.

Abstract: Pre-trained foundation models, particularly large language models, have achieved remarkable success and led to massive fine-tuned variants. These models are commonly fine-tuned locally and then uploaded by users to cloud platforms such as HuggingFace for secure storage. However, the huge model number and their billion-level parameters impose heavy storage overhead for cloud with limited resources. Our empirical and theoretical analysis reveals that most fine-tuned models in cloud have a small difference (delta) from their pre-trained models. To this end, we propose a novel lossless compression scheme FM-Delta specifically for storing massive fine-tuned models in cloud. FM-Delta maps fine-tuned and pre-trained model parameters into integers with the same bits, and entropy codes their integer delta. In this way, cloud only needs to store one uncompressed pre-trained model and other compressed fine-tuned models. 
Extensive experiments have demonstrated that FM-Delta efficiently reduces cloud storage consumption for massive fine-tuned models by an average of around 50% with only negligible additional time in most end-to-end cases. For example, on up to 10 fine-tuned models in the GPT-NeoX-20B family, FM-Delta reduces the original storage requirement from 423GB to 205GB, significantly saving cloud storage costs.

FM-Delta: Lossless Compression for Storing Massive Fine-tuned Foundation Models
Wanyi Ning'
Haifeng Sun' Daixuan Cheng' Jianxin Liao' Ce Zhang²
Jingyu Wang Qi Qi Mengde Zhu'
1 Beijing University Posts and Telecommunications Pengcheng Laboratory University of Chicago (ningwanyi, wang ingyu, qiq18266, arnoldzhu, hfsun) edu daixuancheng6@gmall.. com, Liaojx@bupt.edi cn, cez@uchicago. edu
edu. cn
Abstract
Pre-trained foundation models, particularly large language models, have achieved remarkable success and led to massive fine-tuned variants. These models are commonly fine-tuned locally and then uploaded by users cloud platforms such as HuggingFace for secure storage. However, the huge model number and their billion-level parameters impose heavy storage overhead for cloud with limited resources. Our empirical and theoretical analysis reveals that most fine-tuned models in cloud have a small difference (delta) from their pre- traitrained models. To this end, we propose a novel lossless compression scheme MM-Delta specifically for storing massive fine-tuned models in cloud. FM-Delta maps fine-tuned and pre-trained model parameters into integers with the same bits, and entropy codes their integer delta. this way, cloud only needs store one uncompressed pre- trained model and other compressed fine-tuned models. Extensive experiments have demonstrated that FM-Delta efficiently reduces cloud storage consumption for massive fine-tuned models by average of around 50% with only negligible additional time in most end-to-end cases. For example, on up to 10 fine-tuned models in the GPT-NeoX-20D family, FM-Delta reduces the original storage requirement from 423GB to 205GB, significantly saving cloud storage costs.
Introduction
The widespread success of pre-trained foundational models, particularly large language models(LLM)), has led to the proliferation of fine-tur Ane-ttnnigg(] 23 4). An increasing number of end-users download pre-trained models from cloud platforms such as HuggingFace(5), and fine-tune them using their local relevant data. After muningg these fine-tuned models usually uploaded to cloud storage by users for future use. This trend imposes a heavy storage overhead in the cloud. From Figure [ we that the total number of models stored HuggingFace has rapidly increased from 33, 187 in 2022 to 574,270 in 2024, reflecting the prominent storage overhead on cloud providers. Among these models, fine-tuned models account for large proportion, which exist in two forms: full fine-tuned models and parameter-efficient fine-tuned (PEFT) m (7). PEFT model is generally a subset network of the original model and has much fewer parameters, so they do exert undue strain on cloud storage. By contrast, the full fine-tuned models which have the original size place a substantial storage burden on the cloud.
We collect the fine-tuning statistical information in HuggingFace for six popular model families in Table I which shows the numbers of full fine-tuned and PEFT models for certain pre-trained model. We also present the proportion of "inactive" full fine-tuned models with less than 10 monthly
Corresponniig authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Total Num of Models in HuggingFace
IMDb
Table I: Fine-tuning statistical informa- tion in HuggingFace for the six most popular models on different tasks. "Inac- tive" refers to models with less than 10 monthly downloads.
Pre-trained model
500k
Fine-tune
400k
My
Ste-daadd mocelo
300k
200k
Model
Model
Inact.
size
num.
num.
100k
PETT modils
Falean 40B
40B
79
48
82%
DI DI Do co
GPT-NeoX
201
84%
102249
2022-00
222.09
GPT-J
6B
284
75
88%
2022
you
Cloud
LLaMA-7B
7B
5112
1170
91%
Bert-large
336M
260
159
88%
Figure I: Pre-trained models are fine-tuned into thou- sands of model variants and stored in cloud.
Stable Diff.
800M
1606
65
64%
Approx, disk storage
1591B
41B
89%
downloads. The approximate disk storage consumption is computed under loose assumption that PEFT models hold 10% trainable parameters. It can be seen that although users tend to adopt PEFT methods for larger models, the key pain point in cloud storage lies in the storage overhead for full fine-tuned models. And 89% of these models are inactive. If cloud providers use the most popular SSD, SAMSUNG 870, for storage, which costs about $60 for 1TB, storing just the full fine-tuned models in Table | would cost $9,540, not to mention the additional costs for disk management. It is obvious that storing all the models in HuggingFace is definitely a huge expense. With the further development of LLMs, this issue will become increasingly prominent. Therefore, how to reduce the storage space of those massive fine-tuned models becomes a significant challenge for cloud providers.
The fundamental premise of reducing cloud storage costs is to safely protect the intellectual property of users (i.e., without any model alteration), since cloud providers are accountable for preserving the data integrity of all the users(8; 9). Therefore, lossless compression is a natural solution which perfectly reconstructs the original Tota
IE 14). Lossless compression has been widely used to reduce the storage size of images, sound, and text, which often contain redundant data such as transparent backgrounds in images. However, we observe that directly applying traditional lossless compression techniques on models has almost no compression effect, since model itself seems to have little redundant bytes such as high element similarity suitable for compression. To apply lossless compression to massive fine-tuned models in cloud, it is desired to figure out the characteristics of those fine-tuned models and design an efficient lossless compression scheme specifically for fine-tuned models.
In this paper, we aim at mine the relationship among massive fine-tuned models in cloud and mitigate the heavy storage overhead through lossless compression. We primarily focus on compressing full fine-tuned models in cloud since PEFT models are inherently lightweight. Despite of little redundancy of model itself, we find that most fine-tuned models in cloud are much similar with their are-trained models. Then by further analyzing the model difference theoretically. we reveal that the difference (delta) between fine-tuned and pre-trained models grows with a slow speed O(T)) as the fine-tuning steps T increase. This slow growth speed results in most fine-tuned models not differing significantly from their pre-trained models, since fine-tuning is usually conducted on a similar data domain with moderate iteration steps.
Motivated by the finding, we propose a novel lossless compression scheme FM-Delta to compress the bit-redundant delta between fine-tuned and pre- pre-trained models. Specifically, FM-Delta maps the float parameters of the two models into integers with the same bits and then entropy codes the integer delta. In this way, for a certain family, the cloud only needs to hold complete pre-trained model and other compressed fine-tuned variants, significantly reducing the storage consumption of massive full fine-tuned models. We conduct extensive experiments to demonstrate that FM-Delta ta can significantly reduce the cloud storage consumption by approximately 50% for massive full fine-tuned models, achieving a compression throughput of around 109MB/s. In our implementation. model compression and decompression processes are parallel with network transfer. This ensures that the total end-to-end time for users with a network bandwidth below 800Mbps remains nearly unchanged, with only negligible increase in pre- trained model loading time. The 800Mbps bandwidth threshold corresponds to the compression and decompression throughput. Actually, it is the common case in the real world
Our code is available in httptss//giiuub..oo//nnggggngee/ -elta
2

---

and ResNet50(36). considering the latest global average internet bandwidth (1). If compression is performed on the user side, the total time for model downloading or uploading can even be significantly reduced to as low as 70% of the original time. In addition, we give approximate cost analysis of storage and computation in cloud, concluding that the total cost savings of the cloud can be at least 40%.
To the best of our knowledge, we are the first to reduce the cloud storage consumption of massive fine-tuned models through lossless compression. With the further explosion of fine-tuned models, the demand for efficient model storage in cloud will definitely become more prominent in the future. We hope that our work will stimulate further research in this area, efficiently storing the massive large fine-tuned models with much less cost for cloud.
2 Related Work
Lossless Compression. Traditional lossless compression schemes mainly include Huffman (66) run-length coding([)), entropy (6ding(16) and (187 Based on these traditional schemes, researchers continuously propose new lossless compression algorithms to improve the com- pression rate and speed. Here introduce five common algorithms, including I.ZMA(TO), Gzip Zlib(II), Bzip2(13), and FPzip(14). LZMA, known for its high compression rate, combines LZ77 with context modeling techniques. Gzip is a combination of LZ77 and Huffman coding, widely used in network protocols and Unix-based systems. Zlib is based on the DEFLATE 19g which also combines L.777 and Huffman coding. Bzip2 is based on the Burrows- Wheeler transform (0) and Huffman coding. The above four algorithms compress byte streams directly without considering the data format. FPzip specifically targets scientific data and compresses the delta of adjacent elements through Lorenzo predictor(2I), achieving effective compression for floating- point arrays. However, when compressing models, these lossless compression algorithms have minimal effectiveness. The existing works on applying lossless compression to models have mostly focused on compressed models that have already been quantized or sparsified(22) 23). Although Hershcovitch et coneur- rently propose byte grouping method and apply the stardard compressors compress float models, its compression rate is limited without reporting the actual compression speed(24). In contrast, FM-Delta is the first lossless compression algorithm specifically for fine-tuned models, achieving significant storage reduction in cloud.
Delta Compression. Delta compression is a popular compression technique which encodes a target file relative to one or more reference files through exploiting the high redundancy between them 25 26 27
28 29 30 31.. It has been also used in the context of neural networks to compress the model updates and has demonstrated potential in accelerating model synchronization in distributed ( 28), accelerating multi-model serving 32), and reducing the transmission and storage of model checkpoints or versions 30). In the latter case which also considers storage, models are continually updated the same data domain, with the primary focus meeting the accuracy requirements of the final model checkpoint. Therefore, these researches use lossy techniques checkpoints. In contrast, cloud storage service for
like quantization to compress delta between massive tune-tuned models is a totally novel issue, where storage requires the data of users "exactly lossless". Besides, it is still yet to be mined about the numerical relationship between fine-tuned and pre-trained models which have different data domains. Therefore, we investigate delta between fine-tuned and mreitri models in detail and apply lossless compression.
3 Difference between Fine-tuned and Pre-trained Models
3.1 Empirical Results
We download four common model families for different learning tasks from the popular cloud provider HuggingFace. including Stable Diffusion(333. GPT2(34). Cert-lage-uncassed[3
Firstly, we measure the average cosine similarity between the fine-tuned and pre-trained models and show the results in Figure 2[a It can be seen the similarity on all four f'amilies is higher than 0.98, which reflects that the fine-tum fune-tuned model is much similar with the pre- pre-trained model after fine-tuning Furthermore, we show the distribution of the weight difference between meetuneed and pre- pre-trained models in Figure 2 on Pokemon Stable Diffusion(37). Wikitext103 GPT2 SST2 BERT (39), and FER2013 445 It can be seen that all the element differences are less than 1 and on SST2 Bert-large- layers on Wikitext 103 GPT2. The element of the residual is basically an order of magnitude below 10-2 All the above observations confirm that there is usually small difference between most
even less that 0.01. In Figure we show the residual matrix of different
3

---

Colina Similarity
1.000
Pony Wibitaat03
RakamBoo
Gasalp
Buresat
1.0
But lape
1.5
3.6
conil
con120033
1,0
5.90
CDF
0.5
3.4
3.2-
PeR2015
3.0
Eiiruslon om
large
ReaMt.Ci
0.0
10-4
10
Epachs
0
Epochs
Weight Difference
1e:3
1e.3
(a) Cosine similarity.
(b) Distribution of the weight difference.
avg, W. distancee
1.5-
1.0
1.0
9.
0
Epochs
Epachs
(c) Residual matrix of GPT-2 on Wikitext103,
(a) GPT2
(b) Bert-large-uncased
Figure 2: Difference information between the fine- tuned and pre-trained models.
Figure Fine-tuning different models.
fine-tuned and pre-trained models stored in cloud. For further investigation, we have fine-tuned two models on different datasets for each as shown Figure We measure the average parameter element difference A (Wp, w)) between these two models as the training processes as follows:
-
1 -
d
Á Á(w,. wy)
p.
where dj is the number of elements in the j layer and L is the number of layers. It can be seen that the difference grows with a slow speed as fine-tuning processes. Since most fine-tuning tasks use small learning rate and a limited number of steps, the difference between most fine-tuned and pre-trained models in the cloud is relatively small. We present more empirical results in Appendix B
3.2 Theoretical Analysis
Based on the above empirical observations, we next present our theoretical analysis under the following common assumptions(41] 42), to further understand the difference between fine-tuned and pre-trained model parameters.
Assumption 1. For the loss function f, there exists w E IRd such that f(w) 2 f(w"). for all w.
Assumption 2. satisfies that for all w, w Id, f(w) - f(v) (w Vf((v + - 21l || vlI?.
Assumption 3. Given a data distribution D, the variance of stochastic gradient is bounded: EgnD [[G(w;;) E) - ff(w)]l a²
Theorem 1 (Growth Rate for Model Difference.). Let W. and w are the parameters of the pre- trained and fine-tuned models, respectively. The fine-tuning stage involves T. training steps. With BVi t 1,2,..., ..., T, the distance between wp and w is
learning rate l
V30 3 CIInT)) + C,T
E [[lwy wpll <
where || is la-norm; I is the 3-smooth convex loss function the fine-tuning dataset; wt is the optimal model parameter on the fine-tuning task; Cr and O2 are the constants related to the pre-trained model, which are Ci=(a
O90" 137
f(wa)-fice f(w"))
3.2 and C2 = - 21f(We)
23
We present the full proof of Theorem iin Appendix C.I whereas here we present the key points. We start from 3-smoothness of f to connect the gradient bound [[)] with the loss function f(w)
as [/] < 2n-3, (f(w) f(w")) +
2
2-3r" where the learning rate satisfies 0 < 17 < 3.
Furthermore, we utilize the model optimization step W wp Et= G(w,)) in stochastic gradient descent (SGD)433 to connect the model weight distance with the stochastic gradient. With learning rate 3 1. 1,2, .... T, we have,

---

Table 2: Comparison of certain element value in the 7th position of the pre-trained model (wp) and the fine-tuned model (wy) re- spectively. The delta of the two original element bytes contains large number of redundant "0" bits.
Counts
1000
byte number
1234
w
0.0316
int(wyy) int( int(wp)
3d 6f 00
Significant
Up
0.0309
3c fd 21
Figure 4: Most significant bit distribution of the first
W upp
0.0007
int(wy wp) in((r))int(u,p))
3a 37 80 34
00 04 4d 01
convolutional-Layee delta.
3 a²
(4 ( In(T)) 2vT (n((T) (f(w,) (f
E [[wy -
wpll
32
2,2
(3)
Rearranging the above inequality and taking the square root, derive Theorem D which demon- strates the growth speed O(T') of model difference ||w, -w with the number of fine-tuning steps T. which is consistent with empirical results in Figure
If we further have the assumptions of domain adaptation 44), we can relate the model difference to the two data distributions of the pre-trained Tp and fine-tuned TF domains as follows:
4
4 :) + duan (Dp,Dz)+A, dlog d +log 5
2em
fT (wp) S (w)) w
(4)
where fr. (wp) is the empirical loss of the pre-trained model wp on the pre-trained dataset with m samples; fTP (wp) ) is equivalent to f(wp) in C1 and C2 of Theorem We provide more theoretical details in Appendix C.2 It can be seen that C1 and C2 increase monotonically with ST. (wp). Therefore, we conclude that there are two key factors affecting the model distance, including (1) the number of fine-tuning steps T, and (2) the data distribution divergence between the fine-tuned and pre-trained domains dz^z (Dr, Fr, Dp). Generally, the data distribution divergence between the two domains exists but is not huge 45 46). Besides, the number of fine-tuning steps T is also not large in many downstream tasks, leading to a small difference between most fine-tuned and pre-trained models as in empirical finding. This finding motivates FMt-Delta ta compress such difference for storing massive fine-tuned models.
FM-Delta
4.1 Algorithm
Mapping Float into Integer for Delta. Given a full model pair <pre-trained model MM. fine-tuned model M,>, the key idea of FM-Delta is to losslessly compress their delta, since most fine-tuned models stored in cloud are similar with their pre-trained models as analyzed in Section 3 However, directly performing parameter-leva subtraction will result in float delta that lacks bit-redundancy and could even lead to lossiness [4). As shown in Table floating-poin subtraction does not yield bit-redundant delta, while the bit-level integer subtraction yields the integer delta with a lot of "0" bits. Therefore, we determine to regard the mapped integer delta as the compression object.
As in (14), we firstly keep the same bit stream to convert the original floating-poin parameters of the fine-tuned and pre-trained models to signed integers. Furthermore, to deal with complement arithmetic implemented on most platforms, we secondly map the signed integer to the unsigned integer, by flipping the most significant bit for positive floats and flipping all bits for negative floats to scale the signed range 231 231) with 32 bits the unsigned range [0, 232). as illustrated the left half of Figure(3 5 This transformation monotonically maps floats to unsigned integers, maintaining both the order and the linearity of differences for floats with the same sign and exponent. Finally, we conduct the unsigned integer subtraction to obtain our compression object the bil-redundanl integer delta. Figure shows the most significant bit distribution of delta in the most popular fine-tuned Stable Diffusion (37) in HuggingFace. The most significant bit is the bit in binary number
5

---

(Baat22]
Sian
Exponent (8 bit)
Fraction (23 bit)
0.0016
p.
0
0
0
0
0
0
0 0
0
0
Unsigned Integer Mapping
0
000
0
0
0
0
0.0309
y 7
0 1 Unsigned Integer Mapping
1111001
k MSB A,) E [0.n]
raw bits
yg dy.s) E (,
(Aa, A A A
Mapping into Integer Delta
Probability Modeling
Update symbal probabilities dynamically
Range Coding symbols
raw bits
<-1, n>
<-1, 1>
<0,
0>
<1, 1>
<1, n>
Ao
Compressed
Step 0
row
M
Step.
Update
Output
Figure 5: The lossless compression workflow of FM-Delta. The FM-Delta scheme (1) maps the two floating-pont parameter elements at the same position fine-tuned and pre-trained models into unsigned integers, and performs integer subtraction obtain the bit bit-redundant delta element. Then it (2) regards the sign s and the most significant bit & of delta as symbols. With a quasi-static probability modeler, it encodes the symbols and scales the range to involve raw bits on all delta elements, leading to the compressed fine-tuned model.
that has the highest value position. It can be seen that the most significant bits of most delta elements are around 19, which is significantly below the full 32-bit representation, imbuing potential for further compression.
Compression with Range Coding. Since each delta element in the array is represented by fixed or 16 bit number and the redundant "O" bits appear in the high position, we determine to entropy code the most significant bit with range coding(48) and in (14) to store the range coded symbols and the remaining raw bits in the lower position as the compressed fine-tuned model delta. As with most range coding processes, our compression algorithm mainly consists of the following steps. Firstly, we set an initial range interval and use the quasistatic probability modeler in (48) for probability modeling. Secondly, we define the 2n + 1 symbols by grouping the delta elements into small set of intervals, where n is the number of bits. The 7th integer delta element can be represented as:
Å = 60 - úp.i = =8 (2 - m)
E 1-2",,2
(5)
where s encodes the sign of Á. OS S n is the position the most significant bit, and m is the s(k: + 1) for the range coding. Initially, each symbol is
remaining raw bit stream. We symbolize assigned an equal frequency. As we encode the data, the probability modeler will update the symbol frequencies dynamically based on the processed data. It should be noted that to avoid running out of finite initial range [1, +.(+r), the most significant byte will be periodically output so as to scale the range. Thirdly, determine how to encode the whole delta matrix. In general, except for the most significant bit that can be used as symbols, the remaining raw bits of delta elements do not have much regularity for further compression[14). Therefore, the compressed bytes include both the encoded symbol interval bytes through range coding and the remaining raw bits of the delta.
Similarly for the decoding procedure, with the same equal-frequency initialization, range coding maps the encoded model parameters back to the original symbol range and termly updates the probability model. Then we get the original float-poin fine-tuned model through reverse-mapping delta. Up to this point, the design of FM- FM-Delta is complete. We present its workflow in Figure which illustrates the key points discussed above.
4.2 Robustness to Difference Range
To further assess the robustness of FM-Delta across wide range of differences between fine- tuned and pre- preitrained model parameters, we present the following theorem, which relates the most significant bit position of the integer delta to the original model parameters.
Theorem 2 (Bit-Redundancy of Delta). Given a specific sauting-point encoding format, it has nh bits for the sign part, nh bits for the exponent part, and nem bits for the fraction part. Let Wy (-1)e1 x 2 my and Wp (-1)ah x 2h x my are the floating-point point parameter elements

---

Table 3: Given a base value 0.001, the most significant bit position r of the integer delta, corresponding to the range intervals of different tuned values.
Value (0.001, 0.002) (1, 23)
[0.002, 0.004) [0.004, 0.016)
[0.016, 0.256)
[0.256, 65.536)
24
25
26
27
in the same position of the fine-tuned and pre- pre-trained models, respectively. Let y and up are the unsigned integers mapped from wy. Assuming that 3 Wp let r is the most significant bit position of the integer delta, we have
log2 (2(notam)(8)
④ 8p) + 2 (ef Ep) -iig mipp)
Tbm + loge
log2
+ 2
8f + 8p
Wpp
ne + + 1m
8 B Sp
where [-] the ceiling function; the XOR function; my and my are the integer values with the same bit streams as f and my-
The full proof of Theorem is provided in Appendix C.3 which is derived based on the encoding standards for floats and integers. The impact on can be observed to diminish successively in sign, exponent, and fraction. Taking the 32- floating- point number an example, following the most common IEEE 754 standard(49), there are respectively 1, 8, 23 bits in the sign, exponent, and fractional parts (e.g. n2
The 8, and n 23). According Inequality (6), with the same sign, the upper bound for the most significant bit position of the integer delta r is 25 when "I = 2 is 26 wi
when Mi 26, and is 27 when W1 2 which is rare as experimentally and theoretically analyzed Wpp
in Section From Wip 2 to wi 2 wy differs from up with numerically 2 x scaling, while Wy
the upper bound bit r for delta only increases two bits. For the sake of clarity, we further illustrate in 3 the most significant bit position r corresponding to different ranges of fine-tuned element values, given base (i.e. pre-trained) element value of 0.001. It can be observed that FM-Delta can accommodate vast range of difference, with the most significant bit increasing by only 4 bits even in an extreme case where the value changes from 0.001 to 65,536. Therefore, we deduce that although FM-Delta benefits more with small tuning steps which has higher similarity between models, it nonetheless exhibits considerable robustness for a large range of model differences.
Experiments
Setup. We implemented FM-Delta for models in PyTorch(50) format. Our experiments were run on AMD Ryzen 9 5950X 16-Core Processors@2..1 @2.2GHz (32 logical processors) with 251GB of main memory. In our end-to-end simulation, and users through Python "socket" library. Model compression and decompression processes are parallel with network transfer through reading and writing models in chunks. We download seven popular model families in HuggingFace, including Falcon-40B(51) GPT-NeoX-20B GPT-J(53), G54-2(54), -et--arrgeunccsee(3: Stable-Diffusion (33). and ResNet50036) We collect varying number of their fine-tuned models ranging from 5 to 100, involving various downstream tasks. We primarily select the fine-tuned variants based on download counts, since the more popular model is intuitively more likely to ensure quality. We compare FM-Delta with the five state-of-the-ar lossless compression schemes, including (T)) GZip 12), Zlib(II), FTZip[14) and (3)) We apply these schemes on the fine-tuned model parameters to show their compression rates.
simulate the communication between cloud
Overall Compression Rates. We report the overall compression rate under the different number of
fine-tuned models in Table 4 which is obtained as MM MME Q(My)* where n is the number of
(1+n)-M
fine-tuned models, M the pre-trained model size, and (M,) is the 5th compressed fine-tuner model size. It can be seen that the traditional lossless compression schemes applied directly to model bytes have little compression effect with a compression rate about 90%. Surprisingly, FM-Delta significantly reduces storage consumption by approximately 50% to 60% of the original. Besides, the more fine-tuned models are stored, the more pronounced the compression effect becomes, since the proportion of storage occupied by pre-trained models correspondingly decreases.
7

---

Table 4: Overall compression rates and throughput of six lossless compression schemes on different model families.
Pretrained Size
Finetumed Num.
Original Storage (GB)
Storage after Compression (GB)
Family
LZMA
Grip
Zlib
FPZip
BZip2
PX-Delta
Falcon 40B
5
461.6
349.3
373.4
373.4
456.9
342.7
270.8 (59%)
(fp16)
40B
10
846.3
621.7
669.9
669.9
837.8
608.5
473.9 (56%)
GPT-NeoX
5
230.8
162.9
177.2
176.4
213.4
158.6
112.4 (49%)
(fp16)
20B
10
423.2
298.7
324.9
323.4
391.2
290.7
205.2 (48%)
GPT-J
5
68.4
57.2
60.6
60.6
61.2
58.7
44.6 (65%)
(fp16)
6B
10
125.3
104.8
111
111
112.2
107.6
73.8 (59%) %)
50
24.2
21.8
22
22
21.9
22.5
15 (62%)
GPT2
124M
100
43.2
43.5
43.5
43.4
44.5
28.7 (60%)
Bert-large- uncased
50
63.7
58.6
59.1
59.1
58.9
60.4
41.3 (65%)
336M
100
126.1
116.1
117.1
117.1
116.6
119.6
82.1 (65%) %)
Stable-Diffusion
5
19.2
17.7
17.8
17.8
17.8
18.3
12.8 (67 %
UNet
860M
32.7
32.7
33.5
23.5
10
0.9
0,9
0.9
0.9
0.9
0.7 (68 %)
ResNet50
26M
20
1.7
1.7
1.7
1.7
1.8
1,3 (66
Avg. Compression Throughput (MB/s)
36.1
35.6
83.5
12.1
109.7
Avg. Decompression Throgghpu (MB/s)
24.8
236.6
260.8
80.6
23.8
100.9
L2HA
F720
5200
Talzn-430
CFT-44
D1122
Reahess2
2007
2007
Figure 6: The single compression rates of the six lossless compression schemes on different down- stream fine-tuned models.
Single Compression Rates, showss the single compression rate of the top-5 most popular fine-tuned models in each family. The single compression rate r; the fine-tuned model M, is obtained as T2 = Q(M.)MM, We can observe that directly applying traditional lossless compression algorithms to the model yields similarly marginal compression effects, irrespective of the task setting. In contrast, FM-Delta always outperforms those traditional lossless compression algorithms with a significant compression rate.
Compression & Decompression Throughput. We also report the throughput of compressing from memory disk and decompressing from disk to memory in Table 4 which includes disk writing and reading process through pickling in the binary with "pickle" library. Among all the schemes, FM-Delta shows its superiority with the highest throughput 109. 9997s in compression and the third highest throughput 100.9MB/s in decompression.
Compression Rates of Baselines on Different Objects. We apply the five lossless compression base- lines to four different compression objects on the model pair <"bber-large-uucasedd,, "Jorgeutd/bert- large-uncased- gogg-unncaedd-fiettned---ee finetuned-ner"> including "float model parameters", "float delta", "signed int delta", and "unsigned int delta". FPzip is specifically for floating-por point array, so we only show its results on float parameters and delta. Table 5 shows that delta becomes more bit-redundant after the mapping from float to unsigned integer. resulting in a better compression effect.
Different Data Types. Table 6 shows the compression rate of FM-Delta on Bert-large- uncased under three different data types. We can see that the compression effectiveness ranks from high to low as follows: bfloat16, f1oat16, float32. This result is reasonable since parameter in float32 retains more fine-graine gra-ggraiedd difference between the fine-tuned and pre-trained models, and bfloat has more exponent bits compared with float16, 6. leading to a larger dynamic range with lower precision. It worth mentioning that nowadays many users fine-tune models in float16 6 or 6 to reduce computation overhead, benefits more from FM-Delta.
8

---

Table 5: Compression rates the five base- lines on different objects. The compres- sion rate of FM-Delta is 68%.
PTB
Widter2
WiNtextL03
LAMBADA
18W
Euclidean Distarcc
Compression Rate
1,00
Pupplosity
0.75
0.30
Floot
Float
Int
Uint
0.5
Compressor
Delta
Delta
0.25
0.0
0.00
LZMA
92%
78%
74%
72%
500
1000
500
500
Gzip
92%
86%
83%
82%
Steps
Steps
Steps
Zlib
92%
80%
83%
82%
HPzip
92%
Figure 7: Three metrics over the iteration steps T when fine-tuning GPT-2 on different datasets.
Bzip2
94%
79%:
77%
74%
Table 6:
Compression rates of FM-Delta under three different data types on portllarg-unncseee
Comp.
RoCamp FH-Dabs
MuComp PM-Data
200
Deccmp
200-
200
LosEl pro.
1941 Tim 20
FH-Oshal
X2
150
150
Comproaaad
Finetuned Model & Num.
FP32
FP16
BF16
50
(58
68%
55%
37%
6
50th
NL
63%
43%
27%
(a) Time for key pro- (b) Total time for up-
(c) Total download.
time for
(ST2
63%
49%
27%
cedures.
load.
MRPC (T)
57%
22%
100
65%
50%
33%
Figure 8: End-to-end time under different user bandwidths.
Robustness in Fine-tuning. To observe how the compression rate of FM-Delta changes during fine-tuning, we fine-tune GPT-2 in float32 type on five different datasets(53] [56] [6]] (4) (57). As shown in Figure 7 the Euclidean distance between the two models grows slowly as the number of fine-tuning steps increases, which is consistent with our analysis in Section 3 It can be seen that the compression rate of FM-Delta grows from around 60% once fine-tuning starts an extremely slow speed with the number of steps. For example, the compression rate on PTB only increases from 60.7% to 64.8% after 1,000 fine-tuning steps. Among these datasets, the difference in Euclidean distance is obvious while the difference in compression rate is much small, up to 2%.
Compression & Decompression in Users. We discuss another special but common case where users have cached pre-trained models and conduct compression or decompression. We name this variant FM-Delta We explore different user bandwidths on (Bert-lagge-uncasede (35), Jorgeutd/bert- aeee-uncasee-finetuee----- (58). We provide the detailed time for the key procedures in Figure 8(a) the total time for model upload and download in Figure 8(b) and Figure 8(c) It can be seen that transferring compressed model significantly decreases the transfer time from 204s to 143s under 50Mbps bandwidth. The total time with FM-Delta increases only by a negligible amount for pre-trained model loading when the bandwidth is less than 800Mbps, which is corresponding to the compression and decompression throughput around 100MB/s. The latest report shows that global average broadband speeds are 80Mbps for download and 35Mbps for upload, which are far below 800Mbps(15). Therefore, most users won't be significantly affected, and even those with higher speeds can tolerate the additional transfer time since model transfer isn't an as speed-critical as streaming media.
Table 7: RES and total runtime under different numbers of parallel decompression requests. Each decompression task is assigned one CPU processor.
Parallelism
2
24
RES
L99B
3.40B
12,5GB
24.1GB
39.7GB
Time
11.9s
12.4s
28.5s
Cloud Cost Analysis. In the real cloud system, it is inevitable that multiple users may download or upload at the same time. We assess the total runtime of decompression with varying parallelism levels in Table 7 The result shows that the total time increases slightly with the number of parallel processes due to extra I/O operations. Although we hold that the real-world cloud has powerful elastic computing (65) we give the following analysis on storage and computing hardware cost for better understanding. Assuming that the server stores model of size M with fine-

---

tuned variants, we aim to maximize loadable models n while ensuring the probability of concurrent conpresion/decomprssion tasks exceeding the server threshold below 1,, ie.. -
k) & 0.01. Regarding task concurrency as a binomial distribution, we find the maximum Tomax 35, 300 with python "scipy" library. With a compression rate of 50% for FM-Delta and 89% inactive models referring to [ we can get the saved storage cost C. for those inactive models is $5522. Considering our server purchase on Amazon costs $1179, if compressing inactive models from the 35300 6GB models, the total cost is saved at least S4343 ($5522 $1179). This reduces the total cost to 60% of the original. This underscores the significant cost benefits from FM-Del ta in real-world cloud scenarios. The detailed analysis is presented in Appendix E.7
6 Conclusion and Limitation
In this paper, we empirically and theoretically investigate the difference delta between fine-tuned and pre-trained models. Based on our analysis of the slowly-growing delta, propose FM-Delta, the first lossless compression scheme specifically for storing massive fine-tuned foundation models in cloud. Our experiments on up 100 fine-tuned models demonstrate that FM-Delta can efficiently reduce cloud storage space by around 50% (e.g., from 423GB to 205GB for GPT-NeoX). This reduction incurs only negligible pre-trained model loading time in the most common case where user bandwidth is below 800Mbps. If the fine-tuned models are compressed and decompressed on the user side, the total time for model upload and download can even be reduced under low bandwidths. In terms of limitations, our method does not provide compression support for models in GPU, as it not necessary in our target scenario of cloud storage reduction. These are natural directions for future work to dynamically compressing massive fine-tuned models during training or inference.
Acknowledgments and Disclosure of Funding
This work was done during Wanyi Ning's visiting study at ETH Zurich. We are sincerely grateful to everyone who supported us. This work is funded in part by the National Natural Science Foundation of China under Grants (U23B2001, 62101064, 62171057, 62201072, 62001054. 6207 (0071067), the Ministry of Education and China Mobile Joint Fund (MCM20200202, MCM20180101), and the China Scholarship Council program (Project ID: 202206470044).
References
[1] L. Zou, S. Zhang, H. Cai, D. Ma, S. Cheng, S. Wang, Shi, Z. Cheng, and D. Yin, "Pre-trained language model based ranking in baidu search," Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, 2021, pp. 4022
[2] J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith, "Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping," arXiv preprint arXiv:2002.06305. 2020.
[3] N. Ruiz, Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, "Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation," arXiv preprint ar arXiv:2208. /2242, 2022.
[4] M. Sun, K. Zhou, X. He, Y. Wang, and X. Wang, "Gppt: Graph pre-training and prompt tuning to generalize graph neural networks," in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 1717-1727.
[5] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz et al., "Huggingface' ' transformers: State-of-ther of-the-arttt art natural language processing," arXiv preprint arXiv: 1910::9910037711 2019.
[6] S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, and S. Paul, "Peft: State-of-the-an parameter- 2022.
efficient fine-tuning methods,"
[7] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, Wang, L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models," in The Tenth International Conference on Learning Representations, ICL.R 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. [Online]. Available: http://pww.wooo..ee////oomm////a
10

---

[8] Y. Yang, M. Cheng, Y. Ding, and W. Zhang, "A visually meaningful image encryption scheme based on lossless compression spiht coding," IEEE Transactions on Services Computing, vol. 16, no. 4, pp. 2387-2401, 2023.
[9] S. Luo, G. Zhang. C. Wu, S. U. Khan, and K. Li, "Boafft: Distributed deduplication for big data storage in the cloud," IEEE Transactions on Cloud Computing, vol. 8, no. 4, pp. (199-1211 2020.
[10] E. J. Leavline and D. Singh, "Hardware implementation of Izma data compression algorithm,"" International Journal of Applied Information Systems (IJAIS), vol. 5, no. 4, pp. 51-56, 2013.
[11] P. Deutsch and J.-L. Gailly, "Zlib compressed data format specification version 3.3," Tech. Rep., 1996.
[12] Deutsch, "Gzip file format specification version 4.3," Tech. Rep., 1996.
[13] J. Gilchrist, "Parallel data compression with bzip2," in Proceedings of the 16th IASTED in- termational conference on parallel and distributed computing and systems, vol. 16, no. 2004. Citeseer, 2004, pp. 559-564.
[14] Lindstrom and M. Isenburg. "Fast and efficient compression of floating-point data," IEEE transactions on visualization and computer graphics, vol. 12, no. 5, pp. 1245-1250, 2006.
[15] K. MacMillan, T. Mangla, J. Saxon, N. P. Marwell, and N. Feamster, "A comparative analysis of ookla speedtest and measurement labs network diagnostic test (ndt7)," Proceedings of the ACM Measurement and Analysis of Computing Systems, vol. 7, no. pp. 1-26, 2023.
[16] D. E. Knuth, "Dynamic huffman coding," Journal of algorithms, vol. 6, no. 2, pp. 163-180, 1985.
[17] Golomb, "Run-length encodings (corresp.))" IEEE transactions on information theory, vol. 12, no. 3, pp. 399-401, 1966.
[18] Rigler, W. Bishop, and Kennings, "Fpga-based lossless data compression using huffman and lz77 algorithms," in 2007 Canadian conference on electrical and computer engineering. IEEE, 2007, pp. 1235-1238.
[19] S. Oswal, A. Singh, and K. Kumari, "Deflate compression algorithm," " International Journal of Engineering Research and General Science, vol. 4, no. 1, pp. 430-436, 2016.
[20] G. Manzini, "An analysis of the wheeler transform," Journal of the ACM (JACM), vol. 48, no. 3, pp. 407-430 2001.
[21] I.. Ibarria, P. Lindstrom, Rossignac, and A. Szymezak, "Out-of-core compression and decom- pression of large n-dimensional scalar fields," Computer Graphics Forum, vol. 22, pp. 343-348, 06 2003.
[22] X. Zhang, R. Yang, D. He, X. Ge, T. Xu, Y. Wang, H. Qin, and J. Zhang, "Boosting neural repre- sentations for videos with a conditional decoder," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 2556-2566
[23] Y. Mao, W. Wang, Du, N. Guan, and C. Xue, "On the compressibility of quantized large language models," arXiv preprint arXiv:2403.01380 2024.
[24] M. Hershcovitch, L. Choshen, A. Wood, I. Enmouri, P. Chin, S. Sundararaman, and
D. Harnik, "Lossless and near-lossless compression for foundation models," arXiv preprint arXiv:2404.15 151988 2024.
[25] V. Engelson, Fritzson, and D. Fritzson, Lossless compression of high-volume numerical data from simulations. Linköping University Electronic Press, 2000.
[26] Shilane, Wallace, M. Huang, and Hsu, "Delta compressed and deduplicated storage using informe locality." in HotStorage, 2012.

---

[27] A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani, "Fedpaq: A coammuiication-fffiieee federated learning method with periodic averaging and quantiza- tion." in International Conference on Artificial Intelligence and Statistics. PMLR, 2020, pp. 2021-2031.
[28] Wang, B. Yuan, L. Rimanic, Y. He, T. Dao, B. Chen, C. Ré, and C. Zhang, "Fine-tuning language models over slow networks using activation quantization with guarantees." Advances in Neural Information Processing Systems, vol. 35, pp. 19 19215-19 230, 2022.
[29] Z. Hu, X. Zou, W. Xia, S. Jin, D. Tao, Y. Liu, W. Zhang, and Z. Zhang, "Delta-dnn: Efficiently compressing deep neural networks via exploiting floats similarity," in Proceedings of the 49th International Conference on Parallel Processing, 2020, pp. 1-12.
[30] Y. Chen, Z. Liu, B. Ren, and X. Jin, "On efficient constructions of checkpoints," arXiv preprint arXiv:2009. 13003, 2020.
[31] X. Yao and A. Klimovic, "Deltazip: Multi-tenant language model serving via delta compression," arXiv preprint arXiv:23 2023.
[32] L. Chen, Z. Ye, Y. Wu, D. Zhuo, L. Ceze, and A. Krishnamurthy, "Punica: Multi-tenant lora serving," Proceedings of Machine Learning and Systems, vol. 6, pp. 1-13, 2024.
[33] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image synthe- " in Proceedings of the IEEE/CVF Conference on Computer
sis with latent diffusion models," Vision and Pattern Recognition (CVPR), June 2022, pp. 10684-10695.
[34] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language models are unsupervised multitask learners," 2019.
[35] Devlin, M. Chang, K. Lee, and K. Toutanova, "BERT: pre-training of deep bidirectional transformers for language understanding." CoRR, vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv..or/aa/////IO... 1000004805
[36] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," " in Proceedings of the IEEE conference computer vision and pattern recognition, 2016, pp. 770-778.
[37] Lambda, "lambdilabbsc-ookemoo--ffffffffd sd- sd-pokemon-diffuset huggingFace model.
bttps//hugginnffeeccooombbbbbab
[38] N. LTV/CMU "neula//gpt--mneuun---iiiii
https//huggingfacc.co/eeaalll
gpt2- finetuned- wiittext111 huggingFace model.
[39] AssemblyAL. assemblyiineerr-------eeseeeeee [Online]. Available: Jutps://nugggnnffcc..o assembyaabbett-lng--nsssssssss
[40] C. xcx. co/Celall w0---iinetuunnee--RR011-----0010
[Online]. Available: https:/huggingfface
[41] X. Li, K. Huang, W. Yang, S. Wang, and 7., Zhang, "On the convergence of fedavg on non-iid data," arXiv preprint arXiv: 1907.02/0 2019.
[42] C. T. Dinh, N. H. Tran, M. N. Nguyen, C. S. Hong, W. Bao, A. Y. Zomaya, and V. Gramoli, "Federated learning over wireless networks: Convergence analysis and resource allocation," IEEF/ACM Transactions on Networking, vol. 29, no. 1, pp. 398-409, 2020.
[43] L. Bottou, "Stochastic gradient descent tricks," Neural Networks: Tricks of the Trade: Second Edition, pp. 421-436, 2012.
[44] S. Ben-David, Blitzer, K. Crammer, and F. Pereira, "Analysis of representations for domain adaptation," Advances in neural information processing systems, vol. 19, 2006.
[45] L. Chen, F. Yuan, J. Yang, X. He, C. Li, and M. Yang, "User-specific adaptive fine-tuning for cross-domain recommendations," IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 3, pp. 3239-3252, 2021.
12

---

[46] Y. Cui, Y. Song, C. Sun, A. Howard, and S. Belongie, "Large scale fine grained categorization and domain- specciccc transfer learning," " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4109-4118.
[47] J. H. Wilkinson, "Error analysis of floating-poin computation," Numerische Mathematik, vol. 2, pp. 319-344 340, 1960.
[48] M. Schindler, "Range encoder version 1.3, 2000." http://www..oomppessssooo//cco/m october 1999.
[49] W. Kahan, "Iece standard 754 for binary floating point arithmetic," " Lecture Notes on the Status of IEEE, vol. 754, no. 94720-1776, p. 11, 1996.
[50] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, high-performance deep learning library," Advances in neural information processing systems, vol. 32, 2019.
[51] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo, "Falcon-40B: an open large language model with state-of-the-art performance," 2023.
[52] S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, Golding, H. He, C. Leahy, K. McDonell, J. Phang. M. Pieler, U. S. Prashanth, S. Purohit, L., Reynolds, J. Tow, B. Wang, and S. Weinbach, -Gpt-neox-20b: An open-source autoregressive language model," 2022. [Online]. Available: httts://assi.oo//aa//2220.....00
[53] B. Wang and A. Komatsuzakii, "GPT--6BB A 6 Billion Parameter Autoregressive Language Model," http:///gttubbooooooggg////////s
May 2021.
[54] K. Lagler, M. Schindelegger, J. Böhm, H. Krásná, and Nilsson, "Gpt2: model for radio space geodetic techniques," Geophysical research letters, vol. 40, no. 6, pp. 1069-1073, 2013.
Empirical slant delay
[55] M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz, "Building a large annotated corpus of English: The Penn Treebank," Computational Linguistics, vol. 19, no. 2, pp. 313-330, 1993. [Online]. Available: https//wwww..ewwebbo//////oooyy/
[56] S. Merity, C. Xiong, J. Bradbury, and R. Socher, "Pointer sentinel mixture models," 2016.
[57] C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and Robinson, "One billion word benchmark for measuring progress in statistical language modeling," 2014.
[58] J. L. Grisman, Grisman,,"Jrgeeuddee htwp:////eeeetrreee-nnnndddennne bert-large-uncased hin.aeseddd-iiettndd-eeeeee huggingFace model.
httpss//huggingaacc..oooorgruu
[59] N. group. priinctoo--lyyaap--iiieeeee-rrrr
[Online]. Available:
https
Thaggnnggae...ommmmmmmcooooo//ng wareeert--rgeeuncccece uncased
[60]
priheeeo.siiiiisspsssicseeeerrer [Online]. Available: https://uuginngface
[61] Y. Matsubara. yoohitoooommatuuuu////rrtttaaaa hhn.ginghheccoyyyolooooommmmmmma
mrpp. [Online]. Available:
hhtpss
[62] E. F. Coutinho, F. R. de Carvalho Sousa, P. A. L. Rego, D. G. Gomes, and J. N. de Souza, "Elasticity in cloud computing: a survey," annals of felecommuiications-aaaddl des télécom- munications, vol. 70, pp. 289-309, 2015.
[63] A. Shahbahrami, R. Bahrampour, M. S. Rostami, and M. A. Mobarhan, "Evaluation of huffman and arithmetic algorithms for multimedia compression standards," arXiv preprint 1rXiv:1109.0216, 2011.
[64] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, "GLUE: A multi-task benchmark and analysis platform for natural language understanding." 2019, in the Proceedings of ICLR.
13

---

[65]
M. A. Lab. (2020, January) Bean disease dataset. [Online]. Available:
https://gthub.com/
AI-Lab Laaab-MMkreeer/ibbaaa///
[66]
Hugging Face, "bert-base-cased"
http://mwuggnnnaheeecooeer-aaaaa 2018
[67]
"google/vit-bae-paaceee 10oogddwii-be--ppihhh6------11-: patch16--224-m222... vit-base- 12tch16-222-11221 2020.
http:://hugeenggtcccc.oooooop
[68]
"Transformers: State-of-the-art natural language processing for PyTorch and TensorFlow," http://gtthhoooonuggngg//c/aener 2022.
[69]
KoboldAI,
"Koboldaiggpt--noxx20b- GPT-NeoX- 20B- 20B-breuus, huggingFace model.
https//huggnngaacc..kokbbbbllll
14

---

A Extended Statistics from HugeingFace
In B provide the statistical results of the ten more model families and show the proportion of full models on these families, further underscoring that the number of full fine-tuned models still occupy the majority in the HuggingFace repository. of pre-trained and fine-tuned models in HuggingFace. Specifically, based on model creation time, iterate 10,000 models respectively in both ascending (old to new) and descending (new old) order. Among these models, we only count the models that have explicitly stated their identity (i.e., pre- pre-trained or fine-tuned) in their "README.md" file. We can see that fine-tuner models occupy a significant portion (81 % and 99%) of the model hub. Furthermore, the results of the descending order indicates that fine-tuned models have become overwhelmingly dominant currently.
Table 9
we roughly calculate the portion
Table The number of full fine-tuned and PEFT models the ten additional model families, along with the proportion of full models on these families.
Model
Full
PEFT
Proportion of Full
Gemma-9b
315
121
72%
Gemma-2b
3,836
279
93%
Bloom-77 1ooom-7bl
163
105
60%
Bloom- 117
130
61
68%
Pythia-121
120
131
47%
Pythia-6.9b
316
77%
T5-xxl
106
62
63%
TS-large
1,277
203
86%
Llama-2-70b
214
96
69%
Mistral-7b
6,972
2,027
77%
AVG
71 %
Table 9: The portion of pre-trained and fine-tuned models in the 10,000 models from HuggingFace, counted in ascending and descending order.
Ascending (old to new)
Descending (new to old)
Pretrained Finetuned
# Pretrained
# Finetuned
Num.
501
2.082
52
4,295
Portion
19%
81%
1%
99%
B Detailed Empirical Results about Delta
Here present the detailed empirical results about delta between fine-tuned and pre-trained models. Figure I0 shows the detailed residual matrix of different layers on Wikitext103 GPT-2. Figure p shows that the average parameter element difference on four model families, which grows with slow speed as the fine-tuning processes.
C Theoretical Derivations
C.1 Proof of Theorem
Before deriving Theorem 1, we first relate the gradient and the loss through the following derivation.
Since f is B-smooth, for any w, Rdd, we have
3
f(w) f(v) [V() ((w v) 2 vll2.
For one step gradient descent W = w. - G(W.). we have
B
f(W+11) - -f(w.) S VV((w.) ((we) wt) 2
/
15

---

dogs
wik2
Indb
OUT osat
Lo0
1.0
fcr2013
0.50
0.5
0.25
3.0
15
20
Epochs
Epochs
Epochs
Epochs
lec
10.3
mvg. 3P distance
1.5
3.0
1.0
0.5
0.5
15
20
Epochs
Epochs
Epochs
(a) Stable Diffusion
(b) GPT2
(c) Bert-large-uncsser
(d) ResNet50
Figure 9: Fine-tuning results on different models.
Layer
Layer 2
Layer 11
Layer 12
0.10
attn.c_aate attn
0.08
attn.c_proj
0.06
mip.c_fc
0.04
mip.c_proj
0.02
0.00
Figure 10: Residual matrix of GPT-2 on Wikitex1103,
16

---

B17² ||G((:)]l
VS(w,)" (-nG)w)))
Taking the expectation before we have
E [ff(we+1) - -
8n²
S (-(ww)"(-n))ffl
[[[G(w::]l
+ B1p (I
S V-n[Vf(w.)))
2
BIf0²
2
Let g(w) f(w) f(w)- f(w")( 2 0. The gradient of g(w) is Vg(w)= Therefore, g(w) also satisfies the assumption of bounded variance EşED [[((w:)) - <f(w)ll x². Substituting u() into the above inequality, we have
VS(() Vs(w") = Vs(w).
307²
3n220
E (g( g(we)) S 7) Vg(w))
3n/
,1120²
E [g((w]] s
2 [)[[o((
2
We arrange the above inequality and get
(2n Bn²) [f(w) - Vf(w 21 < 2 (f(w) f(w-((===--> - VV((")" (ww")))(w)))))
w")) + 0n
Since <n n < I. we have 2n- -312 > 0. Therefore, the above inequality can be further written as
2
3no
[f(()]] -
2n Brp (f(w) - f((w")) 2 - ATj
Thus, we have related the gradient f((w) to the loss f(w). Then give the proof Theorem as the following:
Proof.
Since w, = Wp -Er m/. G.w,), we have
||w, - -wpl1
T
|| (G(w,) Vf((:) + V.(w:)) I|
th (o [[V[(w:))))
t=1
(0)
T
T
(f(w) - -f(w")) + 31102
11
0
+
2m:
3rF
2 BTT
1=1
T
T
1
(8)
3VV 1) (f(w) - f(w"))
f
2
t(vt
t=
(I)
(II)
where (a) follows from the above remark.
17

---

For the item (I), we have
T
0
T
a²
Σ
+
321
2B2
t(vt -2)
1=1
1=1
))
+2 2
1
Vt -
3
i
v8 1
2
20
20
1
1
82
Vi
Vē+ 1
2
20
T
20²
+
8
ga
1- 1
20
20
T
1
dt
g2
32
20
20
1 Iln )
+
In
T
82
62
4
where (a) follows from that for monotonically decreasing function f over the region [a 1,b 1].
it holds that 1 [())dx > Eisa f(i).
For the item (II), we have
3 B(V [)) (f(wi) (((w))-((w)) -f(()))
T
2 (Vt - =) (f(w:)- - f(wp)) 1 ((((W)) ((((w)))(ww)) +
1
1
(f(wp) f(w"))
(Vt 2
2=
(2)
T
2 3 (VV - -1) (f(w.) - f(wp)) + (f((p)) - f(w")( + f f(Wp) - f(ww)
{"'
TT
v8 dt
B
Σ
1 3 (Vt - 2) (f(w) f(wp))
1
-2 + ln2 + 2 + In(I 3))
(f(wp) f(w")),
8
where (a) follows from that for a monotonically decreasing function f over the region (a 1, + 1).
it holds that (.-1(()) Ei f().
Taking the expectation and summing up the inequality for & steps, we have
Σ
B[((w:))[(w'))
- s(w"))
3 8(V
1)
1=1
T
0²
-2+ In 2VT + In( VT- 1)
+
(f(W)))
3 (Vv
})
23
3
1=1
0²
0²
-2+ 1n2 2/T + In(VT
dl +
- - -((w))
T
2132
Vt
2
In2+2VT In(vT- 1 +
-2 1n2 2 In VT-VTT- -
In2
(f(w))-f(v)) - f((w))
2,
3
18

---

Combining the above two items, we can write the expectation of the inequality 8 as
IE
lww
wwpl?
20 1+22n2-
(T-2))
0²
(12+22/++nnVV)
1++
2
-2
21)
(f(wp)-f(w))
B
0²
g²
[1l(TT - 1) 2VT + InI -I((VT -)
(9)
(4
91m 4 3m +
232
2/2
-2
122
T + In(vT- -1) (f(wp) (ow))-ffr f((w))
+
8
30 0²
1
2VT 1/2 In(T)
(ln(T))+22
1n(T))
(f f(w")
B2
2,32
B
From the expression of variance D(x) = E [[[x[]]] |||x|] ? we have
V3a
9a²
+ f(wp) - f(ww)
02
2(1(wp) s(ww))
E Bllw, - wpll -
+
vIn(T)++ V
+
B T
3
4,2
2
This completes the proof of Theorem
C.2 Theorem in Domain adaptation
We first give some definitions and assumptions as the previous works(44) in domain adaptation. Let X' C IR be the input space and y C R be the output space. T denotes a domain (or task), which consists of data distribution D over X. We consider a binary classification task with the hypothesis class and the loss function is Huber loss ((}. y) = jύ - yl. Let f X' - y denote hypothesis that maps inputs to predicted labels, and H. C {h : X - } denote a hypothesis class. The H A H. == (h(z) @ h² (z), h, N' E H} is defined as the symmetric difference hypothesis space, where + denotes the XOR operator. AH^H is set of measurable subsets for (h(x)) (1' H. H. Then (DA D') == 2 Ac (Pr Pro (AA) is defined as the distribution divergence induced by the symmetric difference hypothesis space, given two distributions D and D''
From the generation bound in (44), we first give a variant based on the two data distributions of the source and target domain. Let Tg and TT be the source and target domains, whose data distributions are Ds and Df respectively. Let H C {h: :h:24 X' + }} denote a hypothesis class with VC-dimension Then with probability at least 1 8, Vh E H::
2mm
+ log )) + dHAH (Ds, DT) + X,
4
(h))== S (h( +
V doo d
(10)
where e is the base of the natural logarithm, és = EryxDs (h(z) - yl is the empirical error of the source domain based on m observable samples, and X minsen((r((( (h) cs(h) is the optimal eтror on the two domains.
We regard the pre-trained and fine-tuned domains the source and target domains respectively. The pre-trained model with parameters w, can be regarded as the hypothesis. Therefore, based on Inequality (10), we have
4
2em
+ log 4 )
IT (Wp) S TTP (w,) +
V m
dog d
dHAH (Dr, Dr) X.
(11)
Inequality g9) can be equivalently written as
E w 1 <
30
x²
(9
In + InT++vVT
(Tr(wp) (w")
32
2,32
3
In
+2VT
2
Applying 1Tp to the above inequality, we relate the model difference to the data distribution diver- gence between the fine-tuned and pre-trained domains.
19

---

C.3 Proof of Theorem 2
Proof. We first give the most widely used IEEE floating point standard representation:
n = (-1)* x m x 2",
where s is the sign, is the exponent value using a biased representation, and m is the fractional part. If we map w and wp into the integers viy and tip while keeping the same bit stream, we have
log2 A
r
logz (2 (2(netam)(
+ 8p) 2 (ef ep) + my mmn))
When w and up have the same sign (i.e., 8y @ Sp 0), we have
r= [log2 (2 (ef - Epp + my mp)1
[log2 12 (ef Ep) + 2n
19m + 1og2 (es Ep + 1)1
(a)
W m2
Aym + log2
loge
+
Wp m
(b)
W
Now + log2
log2
.2 + 1 Up
Wf
1mm
logg
log2
2
Wp'
where (a) is from that "I
2-
2
mi;
(b) is from that the fraction m of float is in my
2
[L. 2). This completes the proof of Theorem 2.
D Algorithm Details
Table I0 provides the mapping examples of FM-Delta. We detail the mapping process as the following two steps. Firstly, convert floats to signed integers while retaining the same byte string. Secondly, monotonically map signed integers into unsigned integers. For positive values, the representation range is [00000000..., 01111111111111111111111111111111 01111111...,, i.e. (0,231). Both signed and unsigned integers are monotonically increasing in this interval. We just need to map the [0, 231) of signed integers to the [2 22 of unsigned integers. Therefore, we flip the most significant bit for positive values, so the range becomes [10000000..., 11111111..). For negative values, the representation range is [11111111.., 10000000...), i.e. [-2 ,0. The monotonicity of signed and unsigned integers is opposite. Therefore, we flip all the bits of the negative integer to meet the alignment of monotonicity, so the range becomes [00000000..., 01111111....
Table 10: Example of mapping positive and negative floats.
Float
Signed Integer
Unsigned Integer
0.0316
001 100000000
101000000000000000 1011 100000000
-0.0316
1011110100000000011 (01 100000000
0100001011111 10100101111000000000000000000010 IIIIII
E Experiment Details
E.1
Model Compression Details
Table 11 presents the detailed information of the top-5 fine-tuned models in the seven families from Hugging corresponding to the compression results in Figure 6
F.2
Compression Rates on Different Layers
We present the compression rates of FM-Delta on different model layers in Figure 11 which shows
that for a specific model, the compression rate on each layer generally fluctuates around a certain value.
20

---

Table 11: Model details of the top-5 fine-tuned models in the seven families from HuggingFace, and the single compression rates of FM-Delte a on them.
Family
Model Type
HuggingFace
Comp.
Pretruined
RefinedWeb
tituze/falou 40b
Baize
Beaew/alcoon-U-innsttee 1226
50%
Dol.
OpenAssistami/lalloe
Falcon-40B
Finetuned
OASST
Boouuhhoott-ossttl-aae -alcoon-000
43%
WizardLM
chartlon//Wcarrl... Sincemsored-Faleee Falcom-40b
48%
Alpaca&Dolly
faloon 40b 400--t--bllll ftalppcc--ddlllyl dutch
48%
Pretroined
MeatherA/iggt-noo-2000
Erehus
KohoMdA//GPT NeoX 20B Erehus
30%
GPT-NooX-20B
Chat
OggcheeroomppterGGPT-NNoo -eoTTT-Neo-T----BBssee--
42%
Finetuned
Skein
GoboldAUGPT-Nooo-2000 -occ-inn
28%
OASST
dvruetielousst- gpt neox-2Ub- 1000-steps
46%
Instruction
jordiclivennsimaciionn tuned neol-neox-22 20b
53%
Pretrained
The Pile
AlgntherA
Janeway CoT&PA&NT
Janeway
52%
pgerhercompaate((PPPI.....
60%
GPT-1-6B
Finetuned
OASST
rociprocans/gptt__mm format-on
55%
Skein
KoboldAUGGPF3-68-33.. -Keein
56%
Adventure
KsboldAU/PP.... Advventure
59%
Pretrained
WebText
gpd2
IMDB
rajkunammk/ghn 2ppt 2- tuned-eee -n-ied----inn---------- revieis reviews
68%
Wikitext103
neulah/gn12 moabal/gn2--inetumeed finetuned wikitextl 03
75%
GPT-2
Finctuned
CNN-DailyMail
gavin 24/gpp2- f--ffnntnned----e
summarization-v2
78%
CommonGen
(e22--ncttund-ccomee comeeeddooooonnceooo gen mrm8 Comn8488/GPT-22
71%
SQaAD
amas-awadalla/ www.:wawllllllp--oomm htppaapppppppppnp--ppp span-head------ readdfinetuned-uoe netuned-squad
62%
Pretrained
BookCorpus
hert large uncased
CONLL2003
Jorgeutd/bert berrtutd/ber---ee werrud/bbbrrrrrrrgeeeee large ungaeed line--ned--enn
68%
MNLI&SNLI
princeton nlp/sup
large uncasse
Bert-larg-uuccssed
Finctuned
Wikipedia
princeton-n princeton-niphnssg
srap--smmee-eere hert hert-large-uncaseee
65%
SST2
aosemblyui/ber large uncased sa12
63%
MRPC
yoshiramo matsubaraaaeentt largellre---ee
www.e-uncused uncased--mrrr mipe
58%
Pretrained
Aestheettt
runway runwaymll mistaalle diffusion
Waltz
mikesmodels Waltz_ With_BBshhi Diffusion
59%
Osear
thscopeloscarvala
6.2
Stable Diffusion
Finetuned
Clons Wars
questceasticlone-ee wawpoaasssloooewwws-----------e- wawoooneeww-dffusssiaeae diffusion-vl
63%
ProGamerGiov/h Background Diff'sion
Kurzgesagt
questonssi/SD Kurzgesagt style finetume
58%
Pretrained
ImageNet-II
mierosofi/resnet.55
juyantafresnet50 finetuned
BrainTumor
Miii-Moohummeeeeeneee 50 50-finetuned www.--nnnddbbrrn-tttmmeeee -arae---------eeeer
67%
ResNet50
Finetuned
FER2013
//eeeett-50-nnn 50-finetuned-FFE 20000FF3-2220200000000000
83%
Earosat
keifbsnpaaivesnets 50 D0-finetuned-eurre
59%
NCT HE 100K
polejowska/resnet 50 finetuned net cre he 45%
69%
Rate
0.8
Compoession Comprnssionna Nane
0.8
mbb.cc
0.6
2.6
0.69
torkho77
0.68
0.0
0.0
Layer
(a) UNet of Stable Diffusion.
(b) Transformer layers of GPT2.
(c) Sublayers in transformer.
Figure 11: Compression rates of FM-Delta on different model layers.
21

---

E.3 Further Discussion on Throughput
Table 4
shows shows that FM ranks third in decompression throughput, lower than Zlib and Gzip. Here we give a further explanation about the reason.
Among all the compressors, Zlib and Gzip have similar performances since both of them use the DEFLATE algorithm, which consists of two main components: L.77 and Huffman coding. Differently, FM-Deltt ta is mainly based on the compression algorithm specifically for floating-point point data, which uses the range coding to compress array data. Since range coding involves dealing with floating-por point numbers and maintaining the current encoding interval, Huffman coding typically has a faster decompression speed than range coding(6)) (63,, resulting a higher decompression throughput of Zlib and Gzip.
E.4 Extended Fine-tuning Results
In Figure 12, we present the additional fine-tuning results on GPT2-1.5B(44) Since the model is bigger than that the main paper, it can be seen that it overfits on the relatively simple datasets, including (5)( and Wikitex(2(56), The simplicity of the two datasets results in their larger model difference due to their larger distribution divergence with original data for training GPT2-1.5B, which is also analyzed in Section
PTB
Wikitext2
Wikitext103
LAMBADA
1BW
40
Euclidean Distance
1.5
Compression Rate
1.0
Perplexity
1.0
0.5
20
0.0
0.0
500
1000
1000
500
1000
Steps
Steps
Steps
Figure 12: Three metrics over the iteration steps T when fine-tuning GPT2-1.5B on different datasets.
Table 12: Finetuning results for on MRPC and SST2, and google/vit-base-pateeee 224-in21k on Beans.
Task
Steps
Loss
Buc. Distance
Comp. Rate
0
0.95
0
400
0.38
0.66
46.9%
Bert-base-cased on MRPC
800
0.21
0.71
48.5%
1200
0.14
0.74
48.9%
0
0.74
400
0.34
0.65
45.3%
Bert-base-cased on SST2 64)
800
0.28
0.69
46.5%
1200
0.25
0.71
47.2%
0
1.03
0
400
0.15
0.21
62.5%
google/vit-base--atch 1t---p222h1h1112211111111 on Beans 65)
800
0.17
0.24
63.1%
1200
0.04
0.25
63.2%
In Table 12 we provide more fine-tuning experiment results on both Bert-base-casedee and in2 using the official example script from the "transformers" github (68). Under these two commonly used models, FM-Delta still shows good com- pression results. In fact, the small delta caused by the fine-tuning itself. The pre-trained model parameters already have strong inference capabilities, fine-tuning does not make much difference.
22

---

E.5 Compression on Atypical models
To confirm the robustness of FM-Delta, we download two user-uploaded GPT-2 models from Huggingface: "vicgalle/gp(2-alpeee alppaaggppa is popular instruction-tuned model with high quality, is an unpopular low-qualified model without any more information. Besides, we generate several atypical models by adding random tensors with noise values from 0 to {1, 10, 100, 1000) to the GPT-2 pre-trained model.
and leiepp--iietuned-bbiooo
Table 13: Compression Rates and Perplexity on the popular, unpopular, and atypical models. Model
Comp. rate of FM-Delta WikiText2 (ppll)
vicgalle/gp22aalpaccgggo
65.4%
44.7223
meckste/gpt2-fiettune---- biichmiiit gpt2 with noise (0.1)
59.4%
24564191.2008 .22077
85,3%
NaN
gpt2 with noise (0,10)
87.0%
NaN
gpt2 with noise (0, 100)
92.2%
NaN
gpt2 with noise (0,1000)
91.5%
NaN
It can be seen in Table 13 that even on the GPT-2 added with huge noise (0,1000), FM-Delta can still compress the model into 91.5% of the original. This is consistent with our robustness analysis in Section 4 which illustrates that FM-Delta can accommodate vast range of difference. Thus, FM-Delta is robust and reliable enough across diverse massive models in the cloud hub.
E.6 Extended Time Results
Figure 1 presents the detailed time for model upload and download under different user band- widths on <EleutteerA/ggg-nooo-226665
KoboldAJ/GPP--ooX--2000 220NNoXX-22B-- When the user's bandwidth is below approximately 800Mbps, the total time is nearly equivalent to that of the non-compression solution for FM-Delta, and is significantly reduced for FM-Delta" due to the decreased data transfer volume. When the user's bandwidth exceeds around 800Mbps, the total time is limited by the compression throughput due to the transmission speed being faster than the compression speed (approximately 100MB/s).
120
150
150
Comp.
NoComp.
NoComp.
100
Decomp.
125
FM-Delta
125
FM-Delta
Time (min)
Load pre.
Full
Total Time (min)
100
VM-Delta
Total Time (min)
100
FM-Delta"
Compressed
75
50-
25
50M
2003
500M
goom 1G
50M
300M
500M
goom
50M
300M
500M
goom
16
User Bandwidth (bps)
User Bandwidth (bps)
User Bandwidth (bps)
(a) Time for loading pre-trained (b) The total time for model
(c) The total time for model model, model compression, de- upload with non-compression, download with non-compression, compression and transfer.
FM- Del-eeta, and FM-Delta".
FM-Delta, and FM-Delta"
Figure 13: End-to-end time under different user bandwidths on GPT-NeoX-20B.
E.7
Cost Analysis
All the experiment results show that FM-Delta can achieve compression rate around 50% in most cases with around 100MB/s throughput. There is an interesting question raising - in practice for the cloud, what would be the cost of decompressing the models and would it be less than the cost of storing them decompressed in the cloud? We mainly discuss the cloud cost in terms of storage and computation.
23

---

We assume that in cloud there is model with size M, having fine-tuned variants. And of those fine-tuned variants are inactive (monthly downloads<10), We give a loose definition that requests received in the same minute we treat as concurrent requests. Assuming that the 10 downloads of inactive models are not concurrent, the probability of a single inactive model being downloaded in certain minute is 30-24-60" k represents the number of concurrent download requests in given minute and each request corresponds to a different fine-tuned model. Then we can think of model download as binomial distribution:
10
10
-11-
10
P(X = C*
(12)
30
24 60
30 x 24 60
Our goal is to determine the maximum total number n of loadable models on the current server, which ensures that the probability of the concurrency numher exceeding the server threshold is less than 1%, i.e., Σk= P(X = k) Σ 0.01. Referring to Table we set the threshold to 16, since its total time consumption no to compute and get the maximum value Tlmax
than twice that of single task. We utilize python "scipy" library
35300.
We refers to Table to set the inactive proportion p 89%. The model which can be compressed with FM-Delta in a minute is about 6GB, so we set the model storage size 4M = 6GB. If 1TB costs 60 dollars, give the original storage cost inactive models without compression:
Carigin 6/1024 35300 * 89% $60 $11044.
With the compression rate 50% of FM-Delta, the current storage cost is
Cimdella = 6/1024 35300 89% 50%
$60 $5522.
In addition, we directly present the purchase price of the current computing server give the reader a more intuitive sense, even though the cloud server has a lot of computing power and we assume that there idle computing resources. Our current server mainly consists of the cpu (one AMD Ryzen 9 5950X) and memory bars (Kingston Fury Beast 32GB 8), whose price in Amazon are $379 and x $100 respectively. Therefore, the purchase cost Gerver of the server is
Cerverr 379 + 8 100 $1179.
From the above analysis we can conclude that the total cost of the cloud server is at least saved 11044 5522 1179- 4343 dollars on 35,300 inactive 6GB models. The example shows the storage costs are down to 7.% Such cost savings through lossless compression is certainly promising for the cloud in the real world.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please check Section] especially the last paragraph.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please check Section for limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification:
Please check Theorem 1 and Theorem 2 along with their proof in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please check the experiment settings in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
Please check the github link http:///sshhab.ccooo///ww//////a
for our
code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.cc/pll cc/public/guu getggiddddeeeeeebbiniinn
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please check the experiment settings in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification:
Our lossless compression algorithm does not involve any training errors.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please check the experiment settings in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This research is fully with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Please check our contribution in Section Our lossless compression scheme will significantly reduce storage costs for cloud providers with minimal impact on user experience.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our lossless compression poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited all the involved researches, datasets and models.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We provide code in the github link http:////gghabbooo///ggwaa//////
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "z3YCVlCxcU", "reviewer": "Reviewer_yfTq", "summary": "This paper proposes a method to compress the differences between a pretrained model and a full fine-tuned model to save storage space on cloud servers where the full fine-tuned model is stored. For compression, the pretrained weights and full fine-tuned weights are first converted to unsigned integers and then subtracted from each other. Subsequently, range coding is applied to compress redundant zeros, while non-zero values are retained as raw bits. Consequently, this method demonstrates a higher compression ratio than existing lossless compression methods and also shows robustness across various models and datasets.", "questions": "What is the proportion of full fine-tuned models within the entire Hugging Face model repository?", "limitations": "Included in Weakness", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The proposed method for addressing the storage space issues of massive fine-tuned models is highly novel.\n- It achieves a higher compression ratio compared to existing lossless methods and demonstrates robustness across various models. Additionally, it illustrates the trade-off in compression rates resulting from fine-tuning, highlighting the robustness of the proposed method.\n- The paper provides both theoretical and experimental evidence to demonstrate the robustness of the proposed method.\n- From the perspective of practical users, the proposed method is shown to be efficient, and the paper suggests further ways to accelerate the process.", "weaknesses": "The proposed method is effective only for full fine-tuned models and not for peft (parameter-efficient fine-tuning) models. The impact of the proposed method may vary depending on the proportion of full fine-tuned models. Although Table 1 shows the number of models for the six most popular models, it does not provide information on the overall proportion of full fine-tuned models within the entire Hugging Face model repository.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer yfTq for the insightful feedback and address your concern below, and the analyses will be incorporated into our paper..\n\n> **Q1**: What is the proportion of full fine-tuned models within the entire Hugging Face model repository?\n\n**A1**: Since it is hard to distinguish the full finetuned models from all the pre-trained models in HuggingFace based on model meta information, we count the total number of PEFT models and its proportion in HuggingFace as shown below.\n\n| # PEFT Models | # All Models | Proportion of PEFT |\n| ------------- | ------------ | ------------------ |\n| 53,186        | 805,291      | 6.60%              |\n\nIt should be noted that \"All Models\" includes PEFT, full fine-tuned and pretrained models. We can see that PEFT models only occupy a small proportion of all models.\n\nFor your further review, we provide the statistical results of the following additional model families, and show the proportion of full models on these families.\n\n| Model           | # Full | # PEFT | Proportion of Full |\n| --------------- | ------ | ------ | ------------------ |\n| Gemma-9b        | 315    | 121    | 72%                |\n| Gemma-2b        | 3,836  | 279    | 93%                |\n| Bloom-7b1       | 163    | 105    | 60%                |\n| Bloom-1b7       | 130    | 61     | 68%                |\n| Pythia-12b      | 120    | 131    | 47%                |\n| Pythia-6.9b     | 316    | 93     | 77%                |\n| T5-xxl          | 106    | 62     | 63%                |\n| T5-large        | 1,277  | 203    | 86%                |\n| Llama-2-70b | 214    | 96     | 69%                |\n| Mistral-7b      | 6,972  | 2,027  | 77%                |\n| **AVG**         |        |        | **71%**            |\n\nIt can be seen that the number of full fine-tuned models still occupy the majority (71% on average) in the HuggingFace repository. Moreover, given their full size, the storage space required for these models takes up a significant portion of cloud storage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RL9U1QXTUn", "reviewer": "Reviewer_SDxT", "summary": "This paper proposes a novel lossless compression scheme FM-Delta specifically for storing massive fine-tuned models in cloud.   \n\nFM-Delta maps fine-tuned and pre-trained model parameters into integers with the same bits, and entropy codes their integer delta. In this way, cloud only needs to store one uncompressed pre-trained model and other compressed fine-tuned models.   \n\nExtensive experiments demonstrated that FM-Delta efficiently reduces cloud storage consumption for massive fine-tuned models by an average of around 50% with only negligible additional time in most end-to-end cases.", "questions": "Please refer to the weakness part.", "limitations": "The limitation is well addressed in the manuscript.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The research topic of compressing finetuned large models on the cloud is novel and has strong practical meaning in the age of foundation model. The statistics provided in Figure 1 and Table 1 clearly show the necessity of developing such a method to compress the large number of finetune models.   \n\nI like all the statistics of those large models on HuggingFace. The authors devote a lot of effort to provide a systematic study on this novel research topic.  The experiments are extensive.  \n\nTheoretical results are given in Theorem 1 to analyze the growing rate of model difference, which serves a great motivation for the proposed delta coding method.   \n\nThe bit redundancy of model difference is analyzed in Theorem 2, proving a solid theoretical foundational for the robustness of the proposed method.", "weaknesses": "It is not clear how range coding is applied. Range coding needs a probability table for the coding symbols. Did the paper use the symbol frequency as the probability? If this is the case, the probability table should also be transmitted in the compressed stream to ensure correct decoding on user side. \n\nFollowing the previous quesrtion, it would be interesting to see how the probability estimation affects the compression ratio. For example, in Boosting Neural Representations for Videos with a Conditional Decoder, CVPR 2024, a Gaussian Entropy Model is proposed to model the distributin of neural network weight for entropy coding, where only two scalar values are transmitted for each weight or embedding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer SDxT for the insightful feedback and address your concerns below, and the analyses and clarifications will be incorporated into our paper.\n\n> **Q1**: Did the paper use the symbol frequency as the probability?\n\n**A1**: Yes, we use a quasi-static probability modeler as in [1]\\[2] . **Initially, each symbol is assigned an equal frequency.** As we encode or decode the data, we update the symbol frequencies dynamically based on the processed data. This approach ensures that **the frequency table is dynamically built during the encoding and decoding processes**, without the need for transmission of the probability table. \n\nRegarding range coding, we **update the workflow figure of FM-Delta in the uploaded PDF**, involving the illustration of range coding, which includes:\n\n- **Symbolization**. We regard the sign $s$ and the most significant bit $k$ of delta as symbols $<s,k>$ for range coding.\n- **Probability model**. As mentioned above, we use a quasi-static probability modeler to termly update symbol frequencies.\n- **Encoding**. Range coding encodes the symbols and the raw bits on all delta elements through range scaling, leading to the compressed fine-tuned model.\n- **Decoding**. Range coding maps the encoded value back to the original symbol range and termly updates the probability model. Then we get the original float-point fine-tuned model through reverse-mapping delta.\n\n[1] M. Schindler. Range Encoder version 1.3, 2000. URL http://www.compressconsult.com/rangecoder/.\n\n[2] Peter Lindstrom, et al. Fast and Efficient Compression of Floating-Point Data. TVCG 2006.\n\n> **Q2**: Following the previous question, it would be interesting to see how the probability estimation affects the compression ratio. For example, in Boosting Neural Representations for Videos with a Conditional Decoder, CVPR 2024 [3], a Gaussian Entropy Model is proposed to model the distribution of neural network weight for entropy coding, where only two scalar values are transmitted for each weight or embedding.\n\n**A2**: Thanks for your valuable reference. The authors in [3] build a network-free Gaussian model for the probability estimation of the quantized weights in INR, with tiny metadata transmission overhead. In comparison, FM-Delta losslessly compresses diverse floating-point fine-tuned models using a quasi-static probability modeler without transmission of the probability table. \n\nBoth methods focus on the compression of **models**. The example[3] inspires us to further investigate model characteristics for more fine-grained probability estimation in the future work. We add discussion about [3] in the Related Work section of our final manuscript.\n\n[3] Boosting Neural Representations for Videos with a Conditional Decoder, CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JPuVFliDnm", "reviewer": "Reviewer_eKhr", "summary": "This paper proposes FM-Delta, which addresses significant storage overhead caused by fine-tuned LLMs. It maps model parameters into integers and entropy codes their small differences from pre-trained models, reducing cloud storage consumption by about 50% on average.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper proposes FM-Delta, a method for jointly compressing pre-trained and fine-tuned models, achieving a compression ratio of up to 2x for fine-tuned models.\n- Compressing large models is crucial, and the approach of jointly compressing fine-tuned and pre-trained models makes sense. Traditional compression algorithms struggle with large models, so a delta-based method intuitively provides a feasible compression rate and is an interesting direction.\n- The paper offers a formula for the distribution shift between fine-tuned and pre-trained models under given assumptions, providing valuable guidance.", "weaknesses": "- Despite the interesting concept of delta compression, I have some concerns about its practical applications. Could the author please elaborate on specific scenarios where this method can be applied? What kind of situations require frequent downloading of finetuned models from the network?\n\n- Has the author considered comparing the proposed method in this paper with quantization? For example, quantizing delta. There are existing studies showing that quantized LLM still maintains some compressibility, roughly around half the compression rate [1]. In other words, my question is: what is the necessity of performing lossless compression in this setting? Because the compression-decompression process inevitably introduces additional overhead, especially at the user end.\n\n- While the first half is well-written, the writing from the Method section onwards needs improvement. Some technical terms need to be explained in detail, such as \"most significant bit\" mentioned in 4.1 and \"range\" in \"difference range\" mentioned in 4.2. Understanding these concepts is crucial for comprehending the algorithm, yet I couldn't find corresponding explanations in the text. The subsequent method descriptions are also somewhat confusing.\n\n- Why use range coding instead of other entropy coding methods? I noticed the experiments used tools like Gzip, but currently, methods like zstd are more popular. Could the author consider trying zstd compression?\n\n- I noticed that the reported compression throughput in the final experiments is approximately 100MB/s, which is not particularly fast in practical terms. There seems to be significant room for improvement in this speed.\n\n- Some related works are not mentioned, such as [1] [2]. \n\n[1] Mao Y, Wang W, Du H, et al. On the compressibility of quantized large language models[J]. arXiv preprint arXiv:2403.01384, 2024.\n\n[2] Hershcovitch M, Choshen L, Wood A, et al. Lossless and Near-Lossless Compression for Foundation Models[J]. arXiv preprint arXiv:2404.15198, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer eKhr for the detailed and useful feedbacks. We address your concerns point by point below, and the analyses will be incorporated into our paper.\n\n> **Q1**: Could the author please elaborate on specific scenarios where this method can be applied?  What kind of situations require frequent downloading of finetuned models from the network?\n\n**A1**: Regarding application scenarios,  as shown in Figure 1 and Table 1 of the original paper, we focus on a new storage issue on cloud resulting from the rapid development of LLMs. Our method aims to compress massive finetuned large models stored on the cloud platforms like HugggingFace, which currently hosts over 800,000 models, leading to substantial storage costs.\n\n| Time    | Total Num of Models in HuggingFace |\n| ------- | ---------------------------------- |\n| 2022-03 | 33,187  |\n| 2023-03 | 157,082  |\n| 2024-03 | 574,270  |\n| 2024-07 | 805,291 |\n\nRegarding the situation of frequently downloading finetuned models, as a public model storage platform, HuggingFace enables numerous end-users to download the models they need from network for further testing. \n\nIt is important to note that **our primary concern is to reduce the cloud storage overhead** rather than downloading frequency. In fact, a large portion of models (89% as reported in Table 1) in cloud are inactive, with fewer than 10 downloads per month. If applying our method on these models, the total storage costs can be saved at least 40% as discussed in Appendix D.7.\n\n> **Q2**: Has the author considered comparing the proposed method in this paper with quantization? For example, quantizing delta. What is the necessity of performing lossless compression in this setting?\n\n**A2**: Regarding quantizing delta, we select two user-uploaded GPT2 models from Huggingface, and present the **quantization results in the Table 1 of the uploaded PDF.**\n\nWe can see from the quantization results that lossy compression inherently alters the model eval results. We believe **lossy compression goes against the original uploader's desire to store his model safely (unchanged)**, even though the change might be small (actually it is hard to ensure the performance of all models in hub due to diversity). \n\nJust like when we upload a model to HuggingFace today, we don't want the case that we download it and then find any of its eval results inconsistent with the original model. At this point, FM-Delta is exactly lossless and has a rather competitive compression rate.\n\n> **Q3**: Some technical terms need to be explained in detail, such as \"most significant bit\" mentioned in 4.1 and \"range\" in \"difference range\" mentioned in 4.2. The subsequent method descriptions are also somewhat confusing.\n\n**A3**: We explain the mentioned terms in the following.\n\n- \"most significant bit\": The most significant bit (MSB) is the bit in a binary number that has the highest value position.\n- \"difference range\": Range of differences between fine-tuned and pre-trained model parameters.\n\nWe also improve the method description with a more detailed workflow figure. Please check Figure 1 of the uploaded PDF.\n\n> **Q4**: Why use range coding instead of other entropy coding methods? I noticed the experiments used tools like Gzip, but currently, methods like zstd are more popular. Could the author consider trying zstd compression?\n\n**A4**: We use range coding since it can dynamically adapt to the probability distribution of the data and is suitable for compressing floating-points requiring the fine granularity and precision.\n\n| Model  | FM-Delta | ZSTD    | *ZSTD-Delta (on unsigned int delta)* |\n| ---------------------------------------------------- | -------- | ------- | ------------------------------------ |\n| Jorgeutd/**bert-large-uncased**-finetuned-ner  | **68%**  | 92%     | *81%*   |\n| rajkumarrrk/**gpt2**-fine-tuned-on-imdb-positive-reviews | **68%**  | 90%     | *81%*  |\n| mikesmodels/Waltz_with_Bashir_**Diffusion**| **59%**  | 93%     | *72%* |\n| **Comp. Throughput (MB/s)**| 109      | **520** | *346*  |\n| **Decomp. Throughput (MB/s)**  | 100      | **560** | *380*   |\n\nRegarding trying ZSTD, we present its results in the above table. We see that **while ZSTD has a higher throughput, its compression rate is worse than FM-Delta.** \n\nWe further present a hybrid approach ***ZSTD-Delta*** that combines the mapping of FM-Delta with ZSTD, i.e., applying ZSTD on the mapped unsigned int delta. ZSTD-Delta serves as a compromise in practice with the superiority of both FM-Delta (compression rate) and ZSTD (throughput).\n\n> **Q5**: I noticed that the reported compression throughput in the final experiments is approximately 100MB/s, which is not particularly fast in practical terms. There seems to be significant room for improvement in this speed.\n\n**A5**: The trade-off between speed and compression rate is indeed a critical consideration in practice. As mentioned in Answer **A4**, we provide a hybrid approach ZSTD-Delta that combines the mapping of FM-Delta with ZSTD as a compromise for cloud customers. \n\nWe believe that although the throughput of FM-Delta is not the highest, its best compression rate and cost saving of 40% are still valuable in practice. We will strive for greater improvements in lossless coding in our future work.\n\n> **Q6**: Some related works are not mentioned, such as [1] [2].\n\n**A6**: Thanks for your kindful suggestion. The authors in [1] investigate the compressibility of quantized LLMs, and FM-Delta focuses on the lossless compression on floating-point LLMs. The authors in [2] propose a byte grouping method and apply the stardard compressors like zstd to compress float models, while its compression rate is not as good as FM-Delta, and the actual compression speed is not reported. \n\nThese two works are contemporaneous with ours. We add discussion about them in the Related Work section of our final manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all the reviewers for dedicating their time to review our manuscript. 

The uploaded PDF includes an updated workflow figure of FM-Delta for the Reviewer SDxT, and the results of quantizing delta for the Reviewer eKhr.

Meta-review: **Summary:**

Due to their billion-level parameters, pre-trained foundation models (e.g., LLMs) often require significant cloud storage. Many fine-tuned models exhibit only small differences from their pre-trained counterparts, leading to the development of FM-Delta, a novel lossless compression scheme. FM-Delta reduces cloud storage needs by compressing fine-tuned models while keeping one uncompressed pre-trained model, cutting storage requirements by around 50% with minimal additional processing time.

**Strengths:**

- Novelty:
  - "addressing the storage space issues of massive fine-tuned models is highly novel" (yfTq)
  - "is novel and has strong practical meaning in the age of foundation model" (SDxT)
  - "is an interesting direction" (eKhr)
- Insight:
  - "both theoretical and experimental evidence" (yfTq)
  - "a systematic study on this novel research topic" (SDxT)
  - "intuitively provides a feasible compression rate" (eKhr)
- Theory:
  - "Theoretical results are given in Theorem 1 to analyze the growing rate of model difference, which serves a great motivation" (SDxT)
  - "The bit redundancy of model difference is analyzed in Theorem 2, proving a solid theoretical foundational" (SDxT)
- Results:
  - "higher compression ratio" (yfTq)
  - "achieving a compression ratio of up to 2x for fine-tuned models" (eKhr)

**Weaknesses:**

Reviewer yfTq initially contended that the proposed method might not be suitable for parameter-efficient fine-tuning (PEFT) models, as PEFT models typically require only a small portion of parameters for fine-tuning. In response, the authors presented statistical evidence showing that 71% of the models in the HuggingFace repository are full models. I believe this is a strong rationale for the motivation behind this work, and it remains a valid approach to reducing various costs. Including the specifics of this analysis, along with the date it was conducted, in the Appendix, would be beneficial for readers. (The proportional trend would be changed later.) The other issues raised were either minor or effectively addressed during the author-reviewer discussions.

**Reasons to Recommend for Acceptance:**

The AC believes this work explores a novel and important area of research focused on saving storage space on cloud servers. By potentially reducing computing costs, conserving energy, and (possibly) lowering carbon emissions, the work makes a significant contribution. Built on strong statistical motivations and theoretical analysis, the work has been well-received and agreed upon by reviewers, prompting the AC to recommend its acceptance for its innovative direction in modern computing and its emphasis on cost and energy efficiency, potentially, environmental care. 

As quantitative measure, the work received mostly positive evaluations, with good or excellent ratings in soundness (333), presentation (443), and contribution (332), resulting in an overall strong rating of 675.
Decision: Accept (poster)