Title: FM-Delta: Lossless Compression for Storing Massive Fine-tuned Foundation Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: model compression

lossless compression

cloud storage
Abstract: Pre-trained foundation models, particularly large language models, have achieved remarkable success and led to massive fine-tuned variants. These models are commonly fine-tuned locally and then uploaded by users to cloud platforms such as HuggingFace for secure storage. However, the huge model number and their billion-level parameters impose heavy storage overhead for cloud with limited resources. Our empirical and theoretical analysis reveals that most fine-tuned models in cloud have a small difference (delta) from their pre-trained models. To this end, we propose a novel lossless compression scheme FM-Delta specifically for storing massive fine-tuned models in cloud. FM-Delta maps fine-tuned and pre-trained model parameters into integers with the same bits, and entropy codes their integer delta. In this way, cloud only needs to store one uncompressed pre-trained model and other compressed fine-tuned models. 
Extensive experiments have demonstrated that FM-Delta efficiently reduces cloud storage consumption for massive fine-tuned models by an average of around 50% with only negligible additional time in most end-to-end cases. For example, on up to 10 fine-tuned models in the GPT-NeoX-20B family, FM-Delta reduces the original storage requirement from 423GB to 205GB, significantly saving cloud storage costs.

FM-Delta: Lossless Compression for Storing Massive Fine-tuned Foundation Models

Abstract

Pre-trained foundation models, particularly large language models, have achieved remarkable success and led to massive fine-tuned variants. These models are commonly fine-tuned locally and then uploaded by users to cloud platforms such as HuggingFace for secure storage. However, the huge model number and their billion-level parameters impose heavy storage overhead for cloud with limited resources. Our empirical and theoretical analysis reveals that most fine-tuned models in cloud have a small difference (delta) from their pre-trained models. To this end, we propose a novel lossless compression scheme FM-Delta specifically for storing massive fine-tuned models in cloud. FM-Delta maps fine-tuned and pre-trained model parameters into integers with the same bits, and entropy codes their integer delta. In this way, cloud only needs to store one uncompressed pretrained model and other compressed fine-tuned models. Extensive experiments have demonstrated that FM-Delta efficiently reduces cloud storage consumption for massive fine-tuned models by an average of around 50% with only negligible additional time in most end-to-end cases. For example, on up to 10 fine-tuned models in the GPT-NeoX-20B family, FM-Delta reduces the original storage requirement from 423GB to 205GB, significantly saving cloud storage costs.

1 Introduction

The widespread success of pre-trained foundational models, particularly large language models(LLM), has led to the proliferation of fine-tuning(1; 2; 3; 4). An increasing number of end-users download pre-trained models from cloud platforms such as HuggingFace(5), and fine-tune them using their local relevant data. After fine-tuning, these fine-tuned models are usually uploaded to cloud storage by users for future use. This trend imposes a heavy storage overhead in the cloud. From Figure 1, we can see that the total number of models stored on HuggingFace has rapidly increased from 33,187 in 2022 to 574,270 in 2024, reflecting the prominent storage overhead on cloud providers. Among these models, fine-tuned models account for a large proportion, which exist in two forms: full fine-tuned models and parameter-efficient fine-tuned (PEFT) models(6; 7). PEFT model is generally a subset network of the original model and has much fewer parameters, so they do not exert undue strain on cloud storage. By contrast, the full fine-tuned models which have the original size place a substantial storage burden on the cloud.

We collect the fine-tuning statistical information in HuggingFace for six popular model families in Table 1, which shows the numbers of full fine-tuned and PEFT models for a certain pre-trained model. We also present the proportion of "inactive" full fine-tuned models with less than 10 monthly

2022-03

2022-06

2022-09

2022-12

2023-03

2023-06

2023-09

2023-12

2024-03

100k

200k

300k

400k

500k

600k

Total Num of Models

in HuggingFace

Pre-trained model

Cloud

optimum on pre-trained data

optimum on fine-tuned data

...

Full fine-tuned models

PEFT models

...

Fine-tune

downloads. The approximate disk storage consumption is computed under a loose assumption that PEFT models hold 10% trainable parameters. It can be seen that although users tend to adopt PEFT methods for larger models, the key pain point in cloud storage lies in the storage overhead for full fine-tuned models. And 89% of these models are inactive. If cloud providers use the most popular SSD, SAMSUNG 870, for storage, which costs about $60 for 1TB, storing just the full fine-tuned models in Table 1 would cost $9,540, not to mention the additional costs for disk management. It is obvious that storing all the models in HuggingFace is definitely a huge expense. With the further development of LLMs, this issue will become increasingly prominent. Therefore, how to reduce the storage space of those massive fine-tuned models becomes a significant challenge for cloud providers.

The fundamental premise of reducing cloud storage costs is to safely protect the intellectual property of users (i.e., without any model alteration), since cloud providers are accountable for preserving the data integrity of all the users(8; 9). Therefore, lossless compression is a natural solution which perfectly reconstructs the original data(10; 11; 12; 13; 14). Lossless compression has been widely used to reduce the storage size of images, sound, and text, which often contain redundant data such as transparent backgrounds in images. However, we observe that directly applying traditional lossless compression techniques on models has almost no compression effect, since model itself seems to have little redundant bytes such as high element similarity suitable for compression. To apply lossless compression to massive fine-tuned models in cloud, it is desired to figure out the characteristics of those fine-tuned models and design an efficient lossless compression scheme specifically for fine-tuned models.

In this paper, we aim at mine the relationship among massive fine-tuned models in cloud and mitigate the heavy storage overhead through lossless compression. We primarily focus on compressing full fine-tuned models in cloud since PEFT models are inherently lightweight. Despite of little redundancy of model itself, we find that most fine-tuned models in cloud are much similar with their pre-trained models. Then by further analyzing the model difference theoretically, we reveal that the difference (delta) between fine-tuned and pre-trained models grows with a slow speed O ( T 1 4 ) as the fine-tuning steps T increase. This slow growth speed results in most fine-tuned models not differing significantly from their pre-trained models, since fine-tuning is usually conducted on a similar data domain with moderate iteration steps.

Motivated by the finding, we propose a novel lossless compression scheme FM-Delta 2 to compress the bit-redundant delta between fine-tuned and pre-trained models. Specifically, FM-Delta maps the float parameters of the two models into integers with the same bits and then entropy codes the integer delta. In this way, for a certain family, the cloud only needs to hold a complete pre-trained model and other compressed fine-tuned variants, significantly reducing the storage consumption of massive full fine-tuned models. We conduct extensive experiments to demonstrate that FM-Delta can significantly reduce the cloud storage consumption by approximately 50% for massive full fine-tuned models, achieving a compression throughput of around 109MB/s. In our implementation, model compression and decompression processes are parallel with network transfer. This ensures that the total end-to-end time for users with a network bandwidth below 800Mbps remains nearly unchanged, with only a negligible increase in pre-trained model loading time. The 800Mbps bandwidth threshold corresponds to the compression and decompression throughput. Actually, it is the common case in the real world

considering the latest global average internet bandwidth 80Mbps(15). If compression is performed on the user side, the total time for model downloading or uploading can even be significantly reduced to as low as 70% of the original time. In addition, we give a approximate cost analysis of storage and computation in cloud, concluding that the total cost savings of the cloud can be at least 40%.

To the best of our knowledge, we are the first to reduce the cloud storage consumption of massive fine-tuned models through lossless compression. With the further explosion of fine-tuned models, the demand for efficient model storage in cloud will definitely become more prominent in the future. We hope that our work will stimulate further research in this area, efficiently storing the massive large fine-tuned models with much less cost for cloud.

2 Related Work

Lossless Compression. Traditional lossless compression schemes mainly include Huffman coding(16), run-length coding(17), entropy coding(16) and LZ77(18). Based on these traditional schemes, researchers continuously propose new lossless compression algorithms to improve the compression rate and speed. Here we introduce five common algorithms, including LZMA(10), Gzip(12), Zlib(11), Bzip2(13), and FPzip(14). LZMA, known for its high compression rate, combines LZ77 with context modeling techniques. Gzip is a combination of LZ77 and Huffman coding, widely used in network protocols and Unix-based systems. Zlib is based on the DEFLATE algorithm(19), which also combines LZ77 and Huffman coding. Bzip2 is based on the Burrows-Wheeler transform(20) and Huffman coding. The above four algorithms compress byte streams directly without considering the data format. FPzip specifically targets scientific data and compresses the delta of adjacent elements through Lorenzo predictor(21), achieving effective compression for floating-point arrays. However, when compressing models, these lossless compression algorithms have minimal effectiveness. The existing works on applying lossless compression to models have mostly focused on compressed models that have already been quantized or sparsified(22; 23). Although Hershcovitch et al. concurrently propose a byte grouping method and apply the stardard compressors to compress float models, its compression rate is limited without reporting the actual compression speed(24). In contrast, FM-Delta is the first lossless compression algorithm specifically for fine-tuned models, achieving significant storage reduction in cloud.

Delta Compression. Delta compression is a popular compression technique which encodes a target file relative to one or more reference files through exploiting the high redundancy between them(14; 25; 26; 27; 28; 29; 30; 31). It has been also used in the context of neural networks to compress the model updates and has demonstrated potential in accelerating model synchronization in distributed training(27; 28), accelerating multi-model serving(31; 32), and reducing the transmission and storage of model checkpoints or versions(29; 30). In the latter case which also considers storage, models are continually updated on the same data domain, with the primary focus on meeting the accuracy requirements of the final model checkpoint. Therefore, these researches use lossy techniques like quantization to compress delta between two checkpoints. In contrast, cloud storage service for massive fine-tuned models is a totally novel issue, where storage requires the data of users "exactly lossless". Besides, it is still yet to be mined about the numerical relationship between fine-tuned and pre-trained models which have different data domains. Therefore, we investigate delta between fine-tuned and pre-trained models in detail and apply lossless compression.

3 Difference between Fine-tuned and Pre-trained Models

3.1 Empirical Results

Wedownload four common model families for different learning tasks from the popular cloud provider HuggingFace, including Stable Diffusion(33), GPT2(34), Bert-large-uncased(35), and ResNet50(36). Firstly, we measure the average cosine similarity between the fine-tuned and pre-trained models and show the results in Figure 2(a). It can be seen the similarity on all four families is higher than 0.98, which reflects that the fine-tuned model is much similar with the pre-trained model after fine-tuning. Furthermore, we show the distribution of the weight difference between fine-tuned and pre-trained models in Figure 2(b) on Pokemon Stable Diffusion(37), Wikitext103 GPT2(38), SST2 BERT(39), and FER2013 ResNet50(40). It can be seen that all the element differences are less than 1 and on SST2 Bert-large-uncased even less that 0.01. In Figure 2(c), we show the residual matrix of different layers on Wikitext103 GPT2. The element of the residual is basically an order of magnitude below 10 -2 . All the above observations confirm that there is usually a small difference between most

Stable

Diffusion

GPT2

Bert-

large

ResNet50

0.985

0.990

0.995

1.000

Cosine Similarity

Pokemon

Arcane

Pony

Wikitext103

IMDB

cola

conll2003

sst2

FER2013

Eurosat

memes

(a) Cosine similarity.

10

4

10

2

10 0

Weight Difference

0.0

0.5

1.0

CDF

Stable Diffusion

GPT2

ResNet50

(b)

Distribution

of

the

weight difference.

0

500

1000

1500

2000

0

100

200

300

400

500

600

700

h.0.attn.c_attn.weight

0

500

1000

1500

2000

2500

3000

0

100

200

300

400

500

600

700

h.0.mlp.c_fc.weight

0

500

1000

1500

2000

0

100

200

300

400

500

600

700

h.11.attn.c_attn.weight

0.0

0.1

(c) Residual matrix of GPT-2 on Wikitext103.

0

2

4

Epochs

3.0

3.2

3.4

3.6

Loss

wiki2

imdb

0

2

4

Epochs

0.5

1.0

1.5

avg_w_distance

1e-3

(a) GPT2

0

2

4

Epochs

0.0

0.5

1.0

1.5

Loss

conll

sst2

0

2

4

Epochs

0.5

1.0

avg_w_distance

1e-3

fine-tuned and pre-trained models stored in cloud. For further investigation, we have fine-tuned two models on different datasets for each as shown in Figure 3. We measure the average parameter element difference ¯ ∆( w p , w f ) between these two models as the training processes as follows:

where d j is the number of elements in the j th layer and L is the number of layers. It can be seen that the difference grows with a slow speed as fine-tuning processes. Since most fine-tuning tasks use a small learning rate and a limited number of steps, the difference between most fine-tuned and pre-trained models in the cloud is relatively small. We present more empirical results in Appendix B.

3.2 Theoretical Analysis

Based on the above empirical observations, we next present our theoretical analysis under the following common assumptions(41; 42), to further understand the difference between fine-tuned and pre-trained model parameters.

Assumption 1 . For the loss function f , there exists w ∗ ∈ R d such that f ( w ) ≥ f ( w ∗ ) , for all w .

Assumption 2 . f satisfies that for all w , v ∈ R d , f ( w ) -f ( v ) ≤ ( w -v ) T ∇ f ( v ) + β 2 ∥ w -v ∥ 2 .

Assumption 3 . Given a data distribution D , the variance of stochastic gradient is bounded: E ξ ∼D ∥ G ( w ; ξ ) -∇ f ( w ) ∥ 2 ≤ σ 2 .

Theorem 1 (Growth Rate for Model Difference.) . Let w p and w f are the parameters of the pretrained and fine-tuned models, respectively. The fine-tuning stage involves T training steps. With learning rate η t = 1 β √ t , t = 1 , 2 , ..., T , the distance between w p and w f is

where || · || is l 2 -norm; f is the β -smooth convex loss function on the fine-tuning dataset; w ∗ is the optimal model parameter on the fine-tuning task; C 1 and C 2 are the constants related to the

We present the full proof of Theorem 1 in Appendix C.1, whereas here we present the key points. We start from β -smoothness of f to connect the gradient bound ||∇ f ( w ) || 2 with the loss function f ( w )

Furthermore, we utilize the model optimization step w f = w p -∑ T t =0 η t G ( w t ) in stochastic gradient descent (SGD)(43) to connect the model weight distance with the stochastic gradient. With learning rate η t = 1 β √ t , t = 1 , 2 , ..., T , we have,

0

19

32

Significant bit

0

1000

2000

Counts

Rearranging the above inequality and taking the square root, we derive Theorem 1, which demonstrates the growth speed O ( T 1 4 ) of model difference || w f -w p || with the number of fine-tuning steps T , which is consistent with our empirical results in Figure 3.

If we further have the assumptions of domain adaptation(44), we can relate the model difference to the two data distributions of the pre-trained T P and fine-tuned T F domains as follows:

where ˆ f T P ( w p ) is the empirical loss of the pre-trained model w p on the pre-trained dataset with m samples; f T F ( w p ) is equivalent to f ( w p ) in C 1 and C 2 of Theorem 1. We provide more theoretical details in Appendix C.2. It can be seen that C 1 and C 2 increase monotonically with f T F ( w p ) . Therefore, we conclude that there are two key factors affecting the model distance, including (1) the number of fine-tuning steps T , and (2) the data distribution divergence between the fine-tuned and pre-trained domains d H△H ( D F , D P ) . Generally, the data distribution divergence between the two domains exists but is not huge(45; 46). Besides, the number of fine-tuning steps T is also not large in many downstream tasks, leading to a small difference between most fine-tuned and pre-trained models as in our empirical finding. This finding motivates FM-Delta to compress such a difference for storing massive fine-tuned models.

4 FM-Delta

4.1 Algorithm

Mapping Float into Integer for Delta. Given a full model pair <pre-trained model M p , fine-tuned model M f >, the key idea of FM-Delta is to losslessly compress their delta, since most fine-tuned models stored in cloud are similar with their pre-trained models as analyzed in Section 3. However, directly performing parameter-level subtraction will result in a float delta that lacks bit-redundancy and could even lead to lossiness(47; 14). As shown in Table 2, floating-point subtraction does not yield bit-redundant delta, while the bit-level integer subtraction yields the integer delta with a lot of "0" bits. Therefore, we determine to regard the mapped integer delta as the compression object.

As in (14), we firstly keep the same bit stream to convert the original floating-point parameters of the fine-tuned and pre-trained models to signed integers. Furthermore, to deal with complement arithmetic implemented on most platforms, we secondly map the signed integer to the unsigned integer, by flipping the most significant bit for positive floats and flipping all bits for negative floats to scale the signed range [ -2 31 , 2 31 ) with 32 bits to the unsigned range [ 0 , 2 32 ), as illustrated in the left half of Figure 5. This transformation monotonically maps floats to unsigned integers, maintaining both the order and the linearity of differences for floats with the same sign and exponent. Finally, we conduct the unsigned integer subtraction to obtain our compression object the bit-redundant integer delta. Figure 4 shows the most significant bit distribution of delta in the most popular fine-tuned Stable Diffusion model(37) in HuggingFace. The most significant bit is the bit in a binary number

0

0 1 1 1 1

1

0

1 1 1

1

1 1

0 1 0

0 1 1 1 1 0 0 1

....

0 0 0 0 0

0 0

....

Sign

Exponent (8 bit)

Fraction (23 bit)

0.0316

0.0309

(float32)

1

1

0

0

0 0 0

1

0 1

0 0 0 0 0 0 0 0

....

0

0

0 1 1

1

0 0

0 0 0 0 0 0 0 0

....

k=22

k=19

Compressed

s =

k =

raw bits

Range Coding symbols <s, k> + raw bits

<-1, n>

...

<-1, 1>

<0, 0>

<1, 1>

...

<1, n>

Step 0

Step

�

Mapping into Integer Delta

Probability Modeling

Update symbol probabilities dynamically

0

<s, k>

raw bits

Output

Update

2 n

that has the highest value position. It can be seen that the most significant bits of most delta elements are around 19, which is significantly below the full 32-bit representation, imbuing potential for further compression.

Compression with Range Coding. Since each delta element in the array is represented by a fixed 32 or 16 bit number and the redundant "0" bits appear in the high position, we determine to entropy code the most significant bit with range coding(48) and as in (14) to store the range coded symbols and the remaining raw bits in the lower position as the compressed fine-tuned model delta. As with most range coding processes, our compression algorithm mainly consists of the following steps. Firstly, we set an initial range interval and use the quasistatic probability modeler in (48) for probability modeling. Secondly, we define the 2 n +1 symbols by grouping the delta elements into a small set of intervals, where n is the number of bits. The i th integer delta element can be represented as:

where s encodes the sign of ˆ ∆ i , 0 ≤ k ≤ n is the position of the most significant bit, and m is the remaining raw bit stream. We symbolize g = s ( k +1) for the range coding. Initially, each symbol is assigned an equal frequency. As we encode the data, the probability modeler will update the symbol frequencies dynamically based on the processed data. It should be noted that to avoid running out of finite initial range [ l, l + r ) , the most significant byte will be periodically output so as to scale the range. Thirdly, we determine how to encode the whole delta matrix. In general, except for the most significant bit that can be used as symbols, the remaining raw bits of delta elements do not have much regularity for further compression(14). Therefore, the compressed bytes include both the encoded symbol interval bytes through range coding and the remaining raw bits of the delta.

Similarly for the decoding procedure, with the same equal-frequency initialization, range coding maps the encoded model parameters back to the original symbol range and termly updates the probability model. Then we get the original float-point fine-tuned model through reverse-mapping delta. Up to this point, the design of FM-Delta is complete. We present its workflow in Figure 5 which illustrates the key points discussed above.

4.2 Robustness to Difference Range

To further assess the robustness of FM-Delta across a wide range of differences between finetuned and pre-trained model parameters, we present the following theorem, which relates the most significant bit position r of the integer delta to the original model parameters.

Theorem 2 (Bit-Redundancy of Delta) . Given a specific floating-point encoding format, it has n s bits for the sign part, n e bits for the exponent part, and n m bits for the fraction part. Let w f = ( -1) s f × 2 e f × m f and w p = ( -1) s p × 2 e p × m p are the floating-point parameter elements

in the same position of the fine-tuned and pre-trained models, respectively. Let ˆ w f and ˆ w p are the unsigned integers mapped from w f . Assuming that w f > w p , let r is the most significant bit position of the integer delta, we have

where ⌈·⌉ is the ceiling function; ⊕ is the XOR function; ˆ m f and ˆ m p are the integer values with the same bit streams as m f and m p .

The full proof of Theorem 2 is provided in Appendix C.3, which is derived based on the encoding standards for floats and integers. The impact on r can be observed to diminish successively in sign, exponent, and fraction. Taking the 32-bit floating-point number as an example, following the most common IEEE 754 standard(49), there are respectively 1, 8, 23 bits in the sign, exponent, and fractional parts (e.g. n s = 1 , n e = 8 , and n m = 23 ). According to Inequality (6), with the same sign, the upper bound for the most significant bit position of the integer delta r is 25 when w f w p = 2 2 , is 26 when w f w p = 2 6 , and is 27 when w f w p = 2 14 , which is rare as experimentally and theoretically analyzed in Section 3. From w f w p = 2 2 to w f w p = 2 14 , w f differs from w p with numerically 2 12 × scaling, while the upper bound bit r for delta only increases two bits. For the sake of clarity, we further illustrate in Table 3 the most significant bit position r corresponding to different ranges of fine-tuned element values, given a base (i.e. pre-trained) element value of 0.001. It can be observed that FM-Delta can accommodate a vast range of difference, with the most significant bit increasing by only 4 bits even in an extreme case where the value changes from 0.001 to 65.536. Therefore, we deduce that although FM-Delta benefits more with small tuning steps which has a higher similarity between models, it nonetheless exhibits considerable robustness for a large range of model differences.

5 Experiments

Overall Compression Rates. We report the overall compression rate r under the different number of fine-tuned models in Table 4, which is obtained as r = (1+ n ) ·M M + ∑ n i =0 Q ( M ⟩ ) , where n is the number of fine-tuned models, M is the pre-trained model size, and Q ( M ⟩ ) is the i th compressed fine-tuned model size. It can be seen that the traditional lossless compression schemes applied directly to model bytes have little compression effect with a compression rate about 90%. Surprisingly, FM-Delta significantly reduces storage consumption by approximately 50% to 60% of the original. Besides, the more fine-tuned models are stored, the more pronounced the compression effect becomes, since the proportion of storage occupied by pre-trained models correspondingly decreases.

Baize

OAS.&Dol.

&Syt. OASST

WizardLM

Alpaca&Dolly

0.0

0.5

1.0

Compression Rate

Falcon-40B

Erebus

Chat

Skein

OASST

Instruction

0.0

0.5

1.0

GPT-NeoX-20B

Janeway

CoT&P3&NI

OASST

Skein

Adventure

0.0

0.5

1.0

GPT-J-6B

IMDB

Wikitext103

CNN-DailyMail

CommonGen

SQuAD

0.0

0.5

1.0

GPT-2

CoNLL2003

NLI

Wikipedia

SST2

MRPC

0.0

0.5

1.0

BERT-large-uncased

Waltz

CloneWars

SJH

0.0

0.5

1.0

SD UNet

Memes

BrainTumor

FER2013

Eurosat

NCT-CRC-

HE-100K

0.0

0.5

1.0

ResNet50

LZMA

GZip

Zlib

BZip2

FM-Delta

Single Compression Rates. Figure 6 shows the single compression rate of the top-5 most popular fine-tuned models in each family. The single compression rate r i on the fine-tuned model M i is obtained as r i = Q ( M i ) / M . We can observe that directly applying traditional lossless compression algorithms to the model yields similarly marginal compression effects, irrespective of the task setting. In contrast, FM-Delta always outperforms those traditional lossless compression algorithms with a significant compression rate.

Compression & Decompression Throughput. We also report the throughput of compressing from memory to disk and decompressing from disk to memory in Table 4, which includes disk writing and reading process through pickling in the binary with " pickle " library. Among all the schemes, FM-Delta shows its superiority with the highest throughput 109.7MB/s in compression and the third highest throughput 100.9MB/s in decompression.

Compression Rates of Baselines on Different Objects. We apply the five lossless compression baselines to four different compression objects on the model pair <"bert-large-uncased", "Jorgeutd/bertlarge-uncased-finetuned-ner">, including "float model parameters", "float delta", "signed int delta", and "unsigned int delta". FPzip is specifically for floating-point array, so we only show its results on float parameters and delta. Table 5 shows that delta becomes more bit-redundant after the mapping from float to unsigned integer, resulting in a better compression effect.

Different Data Types. Table 6 shows the compression rate of FM-Delta on Bert-large-uncased under three different data types. We can see that the compression effectiveness ranks from high to low as follows: bfloat16, float16, float32. This result is reasonable since parameter in float32 retains more fine-grained difference between the fine-tuned and pre-trained models, and bfloat16 has more exponent bits compared with float16, leading to a larger dynamic range with lower precision. It is worth mentioning that nowadays many users fine-tune models in float16 or bfloat16 to reduce computation overhead, benefits more from FM-Delta .

0

500

1000

Steps

20

40

60

Perplexity

0

500

1000

Steps

0.0

0.5

1.0

1.5

Euclidean Distance

0

500

1000

Steps

0.00

0.25

0.50

0.75

1.00

Compression Rate

PTB

Wikitext2

Wikitext103

LAMBADA

1BW

50M

300M

500M

800M

1G

User Bandwidth (bps)

0

50

100

150

200

250

Time (s)

Comp.

Decomp.

Load pre.

Full

Compressed

(a) Time for key pro-

cedures.

50M

300M

500M

800M

1G

User Bandwidth (bps)

0

50

100

150

200

250

Total Time (s)

NoComp.

FM-Delta

FM-Delta

U

(b) Total time for up-

load.

50M

300M

500M

800M

1G

User Bandwidth (bps)

0

50

100

150

200

250

Total Time (s)

NoComp.

FM-Delta

FM-Delta

U

(c)

Total

time

for

download.

Robustness in Fine-tuning. To observe how the compression rate of FM-Delta changes during fine-tuning, we fine-tune GPT-2 in float32 type on five different datasets(55; 56; 56; 34; 57). As shown in Figure 7, the Euclidean distance between the two models grows slowly as the number of fine-tuning steps increases, which is consistent with our analysis in Section 3. It can be seen that the compression rate of FM-Delta grows from around 60% once fine-tuning starts at an extremely slow speed with the number of steps. For example, the compression rate on PTB only increases from 60.7% to 64.8% after 1,000 fine-tuning steps. Among these datasets, the difference in Euclidean distance is obvious while the difference in compression rate is much small, up to 2%.

Compression & Decompression in Users. We discuss another special but common case where users have cached pre-trained models and conduct compression or decompression. We name this variant as FM-Delta U . We explore different user bandwidths on <Bert-large-uncased(35), Jorgeutd/bertlarge-uncased-finetuned-ner(58). We provide the detailed time for the key procedures in Figure 8(a), the total time for model upload and download in Figure 8(b) and Figure 8(c). It can be seen that transferring compressed model significantly decreases the transfer time from 204s to 143s under 50Mbps bandwidth. The total time with FM-Delta increases only by a negligible amount for pre-trained model loading when the bandwidth is less than 800Mbps, which is corresponding to the compression and decompression throughput around 100MB/s. The latest report shows that global average broadband speeds are 80Mbps for download and 35Mbps for upload, which are far below 800Mbps(15). Therefore, most users won't be significantly affected, and even those with higher speeds can tolerate the additional transfer time since model transfer isn't as speed-critical as streaming media.

Cloud Cost Analysis. In the real cloud system, it is inevitable that multiple users may download or upload at the same time. We assess the total runtime of decompression with varying parallelism levels in Table 7. The result shows that the total time increases slightly with the number of parallel processes due to extra I/O operations. Although we hold that the real-world cloud has powerful elastic computing resources(62), we give the following analysis on storage and computing hardware cost for better understanding. Assuming that the server stores a model of size M with n fine-

tuned variants, we aim to maximize loadable models n while ensuring the probability of concurrent compression/decompression tasks exceeding the server threshold is below 1%, i.e., ∑ n k = t P ( X = k ) ≤ 0 . 01 . Regarding task concurrency as a binomial distribution, we find the maximum n max = 35 , 300 with python " scipy " library. With a compression rate of 50% for FM-Delta and 89% inactive models referring to Table 1, we can get the saved storage cost c s for those inactive models is $5522. Considering our server purchase on Amazon costs $1179, if compressing inactive models from the 35300 6GB models, the total cost is saved at least $4343 ($5522 - $1179). This reduces the total cost to 60% of the original. This underscores the significant cost benefits from FM-Delta in real-world cloud scenarios. The detailed analysis is presented in Appendix E.7.

6 Conclusion and Limitation

In this paper, we empirically and theoretically investigate the difference delta between fine-tuned and pre-trained models. Based on our analysis of the slowly-growing delta, we propose FM-Delta , the first lossless compression scheme specifically for storing massive fine-tuned foundation models in cloud. Our experiments on up to 100 fine-tuned models demonstrate that FM-Delta can efficiently reduce cloud storage space by around 50% (e.g., from 423GB to 205GB for GPT-NeoX). This reduction incurs only negligible pre-trained model loading time in the most common case where user bandwidth is below 800Mbps. If the fine-tuned models are compressed and decompressed on the user side, the total time for model upload and download can even be reduced under low bandwidths. In terms of limitations, our method does not provide a compression support for models in GPU, as it is not necessary in our target scenario of cloud storage reduction. These are natural directions for future work to dynamically compressing massive fine-tuned models during training or inference.

Acknowledgments and Disclosure of Funding

This work was done during Wanyi Ning's visiting study at ETH Zurich. We are sincerely grateful to everyone who supported us. This work is funded in part by the National Natural Science Foundation of China under Grants (U23B2001, 62101064, 62171057, 62201072, 62001054, 62071067), the Ministry of Education and China Mobile Joint Fund (MCM20200202, MCM20180101), and the China Scholarship Council program (Project ID: 202206470044).

References

L. Zou, S. Zhang, H. Cai, D. Ma, S. Cheng, S. Wang, D. Shi, Z. Cheng, and D. Yin, 'Pre-trained language model based ranking in baidu search,' in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , 2021, pp. 4014-4022.

J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith, 'Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping,' arXiv preprint arXiv:2002.06305 , 2020.

N. Ruiz, Y. Li, V . Jampani, Y . Pritch, M. Rubinstein, and K. Aberman, 'Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,' arXiv preprint arXiv:2208.12242 , 2022.

M. Sun, K. Zhou, X. He, Y. Wang, and X. Wang, 'Gppt: Graph pre-training and prompt tuning to generalize graph neural networks,' in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2022, pp. 1717-1727.

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz et al. , 'Huggingface's transformers: State-of-the-art natural language processing,' arXiv preprint arXiv:1910.03771 , 2019.

S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, and S. Paul, 'Peft: State-of-the-art parameterefficient fine-tuning methods,' https://github.com/huggingface/peft, 2022.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, 'Lora: Low-rank adaptation of large language models,' in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

Y. Yang, M. Cheng, Y. Ding, and W. Zhang, 'A visually meaningful image encryption scheme based on lossless compression spiht coding,' IEEE Transactions on Services Computing , vol. 16, no. 4, pp. 2387-2401, 2023.

S. Luo, G. Zhang, C. Wu, S. U. Khan, and K. Li, 'Boafft: Distributed deduplication for big data storage in the cloud,' IEEE Transactions on Cloud Computing , vol. 8, no. 4, pp. 1199-1211, 2020.

E. J. Leavline and D. Singh, 'Hardware implementation of lzma data compression algorithm,' International Journal of Applied Information Systems (IJAIS) , vol. 5, no. 4, pp. 51-56, 2013.

P. Deutsch and J.-L. Gailly, 'Zlib compressed data format specification version 3.3,' Tech. Rep., 1996.

P. Deutsch, 'Gzip file format specification version 4.3,' Tech. Rep., 1996.

J. Gilchrist, 'Parallel data compression with bzip2,' in Proceedings of the 16th IASTED international conference on parallel and distributed computing and systems , vol. 16, no. 2004. Citeseer, 2004, pp. 559-564.

P. Lindstrom and M. Isenburg, 'Fast and efficient compression of floating-point data,' IEEE transactions on visualization and computer graphics , vol. 12, no. 5, pp. 1245-1250, 2006.

K. MacMillan, T. Mangla, J. Saxon, N. P. Marwell, and N. Feamster, 'A comparative analysis of ookla speedtest and measurement labs network diagnostic test (ndt7),' Proceedings of the ACM on Measurement and Analysis of Computing Systems , vol. 7, no. 1, pp. 1-26, 2023.

D. E. Knuth, 'Dynamic huffman coding,' Journal of algorithms , vol. 6, no. 2, pp. 163-180, 1985.

S. Golomb, 'Run-length encodings (corresp.),' IEEE transactions on information theory , vol. 12, no. 3, pp. 399-401, 1966.

S. Rigler, W. Bishop, and A. Kennings, 'Fpga-based lossless data compression using huffman and lz77 algorithms,' in 2007 Canadian conference on electrical and computer engineering . IEEE, 2007, pp. 1235-1238.

S. Oswal, A. Singh, and K. Kumari, 'Deflate compression algorithm,' International Journal of Engineering Research and General Science , vol. 4, no. 1, pp. 430-436, 2016.

G. Manzini, 'An analysis of the burrows-wheeler transform,' Journal of the ACM (JACM) , vol. 48, no. 3, pp. 407-430, 2001.

L. Ibarria, P. Lindstrom, J. Rossignac, and A. Szymczak, 'Out-of-core compression and decompression of large n-dimensional scalar fields,' Computer Graphics Forum , vol. 22, pp. 343-348, 06 2003.

X. Zhang, R. Yang, D. He, X. Ge, T. Xu, Y. Wang, H. Qin, and J. Zhang, 'Boosting neural representations for videos with a conditional decoder,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 2556-2566.

Y. Mao, W. Wang, H. Du, N. Guan, and C. J. Xue, 'On the compressibility of quantized large language models,' arXiv preprint arXiv:2403.01384 , 2024.

M. Hershcovitch, L. Choshen, A. Wood, I. Enmouri, P. Chin, S. Sundararaman, and D. Harnik, 'Lossless and near-lossless compression for foundation models,' arXiv preprint arXiv:2404.15198 , 2024.

V. Engelson, P. Fritzson, and D. Fritzson, Lossless compression of high-volume numerical data from simulations . Linköping University Electronic Press, 2000.

P. Shilane, G. Wallace, M. Huang, and W. Hsu, 'Delta compressed and deduplicated storage using stream-informed locality.' in HotStorage , 2012.

A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani, 'Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization,' in International Conference on Artificial Intelligence and Statistics . PMLR, 2020, pp. 2021-2031.

J. Wang, B. Yuan, L. Rimanic, Y. He, T. Dao, B. Chen, C. Ré, and C. Zhang, 'Fine-tuning language models over slow networks using activation quantization with guarantees,' Advances in Neural Information Processing Systems , vol. 35, pp. 19 215-19 230, 2022.

Z. Hu, X. Zou, W. Xia, S. Jin, D. Tao, Y. Liu, W. Zhang, and Z. Zhang, 'Delta-dnn: Efficiently compressing deep neural networks via exploiting floats similarity,' in Proceedings of the 49th International Conference on Parallel Processing , 2020, pp. 1-12.

Y. Chen, Z. Liu, B. Ren, and X. Jin, 'On efficient constructions of checkpoints,' arXiv preprint arXiv:2009.13003 , 2020.

X. Yao and A. Klimovic, 'Deltazip: Multi-tenant language model serving via delta compression,' arXiv preprint arXiv:2312.05215 , 2023.

L. Chen, Z. Ye, Y. Wu, D. Zhuo, L. Ceze, and A. Krishnamurthy, 'Punica: Multi-tenant lora serving,' Proceedings of Machine Learning and Systems , vol. 6, pp. 1-13, 2024.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, 'High-resolution image synthesis with latent diffusion models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2022, pp. 10 684-10 695.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, 'Language models are unsupervised multitask learners,' 2019.

J. Devlin, M. Chang, K. Lee, and K. Toutanova, 'BERT: pre-training of deep bidirectional transformers for language understanding,' CoRR , vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/1810.04805

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770-778.

Lambda, 'lambdalabs/sd-pokemon-diffusers.' https://huggingface.co/lambdalabs/ sd-pokemon-diffusers, huggingFace model.

N. . LTI/CMU, 'neulab/gpt2-finetuned-wikitext103.' https://huggingface.co/neulab/ gpt2-finetuned-wikitext103, huggingFace model.

AssemblyAI. assemblyai/bert-large-uncased-sst2. [Online]. Available: https://huggingface.co/ assemblyai/bert-large-uncased-sst2

C. xcx. Celal11/resnet-50-finetuned-fer2013-0.0033. [Online]. Available: https://huggingface. co/Celal11/resnet-50-finetuned-FER2013-0.003

X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, 'On the convergence of fedavg on non-iid data,' arXiv preprint arXiv:1907.02189 , 2019.

C. T. Dinh, N. H. Tran, M. N. Nguyen, C. S. Hong, W. Bao, A. Y. Zomaya, and V. Gramoli, 'Federated learning over wireless networks: Convergence analysis and resource allocation,' IEEE/ACM Transactions on Networking , vol. 29, no. 1, pp. 398-409, 2020.

L. Bottou, 'Stochastic gradient descent tricks,' Neural Networks: Tricks of the Trade: Second Edition , pp. 421-436, 2012.

S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira, 'Analysis of representations for domain adaptation,' Advances in neural information processing systems , vol. 19, 2006.

L. Chen, F. Yuan, J. Yang, X. He, C. Li, and M. Yang, 'User-specific adaptive fine-tuning for cross-domain recommendations,' IEEE Transactions on Knowledge and Data Engineering , vol. 35, no. 3, pp. 3239-3252, 2021.

Y. Cui, Y. Song, C. Sun, A. Howard, and S. Belongie, 'Large scale fine-grained categorization and domain-specific transfer learning,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 4109-4118.

J. H. Wilkinson, 'Error analysis of floating-point computation,' Numerische Mathematik , vol. 2, pp. 319-340, 1960.

M. Schindler, 'Range encoder version 1.3, 2000.' http://www.compressconsult.com/rangecoder/, october 1999.

W. Kahan, 'Ieee standard 754 for binary floating-point arithmetic,' Lecture Notes on the Status of IEEE , vol. 754, no. 94720-1776, p. 11, 1996.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , 'Pytorch: An imperative style, high-performance deep learning library,' Advances in neural information processing systems , vol. 32, 2019.

E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo, 'Falcon-40B: an open large language model with state-of-the-art performance,' 2023.

S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach, 'Gpt-neox-20b: An open-source autoregressive language model,' 2022. [Online]. Available: https://arxiv.org/abs/2204.06745

B. Wang and A. Komatsuzaki, 'GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model,' https://github.com/kingoflolz/mesh-transformer-jax, May 2021.

K. Lagler, M. Schindelegger, J. Böhm, H. Krásná, and T. Nilsson, 'Gpt2: Empirical slant delay model for radio space geodetic techniques,' Geophysical research letters , vol. 40, no. 6, pp. 1069-1073, 2013.

M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz, 'Building a large annotated corpus of English: The Penn Treebank,' Computational Linguistics , vol. 19, no. 2, pp. 313-330, 1993. [Online]. Available: https://www.aclweb.org/anthology/J93-2004

S. Merity, C. Xiong, J. Bradbury, and R. Socher, 'Pointer sentinel mixture models,' 2016.

C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson, 'One billion word benchmark for measuring progress in statistical language modeling,' 2014.

J. L. Grisman, 'Jorgeutd/bert-large-uncased-finetuned-ner.' https://huggingface.co/Jorgeutd/ bert-large-uncased-finetuned-ner, huggingFace model.

P. N. group. princeton-nlp/sup-simcse-bert-large-uncased. [Online]. Available: https: //huggingface.co/princeton-nlp/sup-simcse-bert-large-uncased

--. princeton-nlp/unsup-simcse-bert-large-uncased. [Online]. Available: https://huggingface. co/princeton-nlp/unsup-simcse-bert-large-uncased

Y. Matsubara. yoshitomo-matsubara/bert-large-uncased-mrpc. [Online]. Available: https: //huggingface.co/yoshitomo-matsubara/bert-large-uncased-mrpc

E. F. Coutinho, F. R. de Carvalho Sousa, P. A. L. Rego, D. G. Gomes, and J. N. de Souza, 'Elasticity in cloud computing: a survey,' annals of telecommunications-annales des télécommunications , vol. 70, pp. 289-309, 2015.

A. Shahbahrami, R. Bahrampour, M. S. Rostami, and M. A. Mobarhan, 'Evaluation of huffman and arithmetic algorithms for multimedia compression standards,' arXiv preprint arXiv:1109.0216 , 2011.

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, 'GLUE: A multi-task benchmark and analysis platform for natural language understanding,' 2019, in the Proceedings of ICLR.

M. A. Lab. (2020, January) Bean disease dataset. [Online]. Available: https://github.com/ AI-Lab-Makerere/ibean/

Hugging Face, 'bert-base-cased,' https://huggingface.co/bert-base-cased, 2018.

--, 'google/vit-base-patch16-224-in21k,' https://huggingface.co/google/ vit-base-patch16-224-in21k, 2020.

--,'Transformers: State-of-the-art natural language processing for PyTorch and TensorFlow,' https://github.com/huggingface/transformers, 2022.

KoboldAI, 'Koboldai/gpt-neox-20b-erebus.' https://huggingface.co/KoboldAI/ GPT-NeoX-20B-Erebus, huggingFace model.

A Extended Statistics from HuggingFace

In Table 8, we provide the statistical results of the ten more model families and show the proportion of full models on these families, further underscoring that the number of full fine-tuned models still occupy the majority in the HuggingFace repository. In Table 9, we roughly calculate the portion of pre-trained and fine-tuned models in HuggingFace. Specifically, based on model creation time, we iterate 10,000 models respectively in both ascending (old to new) and descending (new to old) order. Among these models, we only count the models that have explicitly stated their identity (i.e., pre-trained or fine-tuned) in their "README.md" file. We can see that fine-tuned models occupy a significant portion (81% and 99%) of the model hub. Furthermore, the results of the descending order indicates that fine-tuned models have become overwhelmingly dominant currently.

B Detailed Empirical Results about Delta

Here we present the detailed empirical results about delta between fine-tuned and pre-trained models. Figure 10 shows the detailed residual matrix of different layers on Wikitext103 GPT-2. Figure 9 shows that the average parameter element difference on four model families, which grows with a slow speed as the fine-tuning processes.

C Theoretical Derivations

C.1 Proof of Theorem 1

Before deriving Theorem 1, we first relate the gradient and the loss through the following derivation.

Since f is β -smooth, for any w , v ∈ R d , we have

For one step of gradient descent w t +1 = w t -1 β G ( w t ) , we have

0

1

2

3

4

Epochs

0.25

0.50

0.75

Loss

dogs

sunglasses

0

1

2

3

4

Epochs

2

4

6

avg_w_distance

1e-5

(a) Stable Diffusion

0

2

4

Epochs

3.0

3.2

3.4

3.6

Loss

wiki2

imdb

0

2

4

Epochs

0.5

1.0

1.5

avg_w_distance

1e-3

(b) GPT2

0

2

4

Epochs

0.0

0.5

1.0

1.5

Loss

conll

sst2

0

2

4

Epochs

0.5

1.0

avg_w_distance

1e-3

0

5

10

15

20

Epochs

0

2

4

Loss

eurosat

fer2013

0

5

10

15

20

Epochs

0

1

2

avg_w_distance

1e-3

(d) ResNet50

Layer 1

Layer 2

Layer 11

Layer 12

. . .

attn.c_attn

attn.c_proj

. . .

. . .

mlp.c_fc

mlp.c_proj

. . .

. . .

0

100200300400500600700

h.0.attn.c_proj.weight

0

100200300400500600

700

100

100

200

0.10

200

0.10

300

0.06

0.08

300

0.08

0.06

400

0.04

400

0.04

500

0.02

500

0.02

600

0.00

600

0.00

700

700

0.10

h.10.attn.c_proj.weight

h.11.attn.c_proj.weight

100

200

300

00900500

700

100200

0090050000

100

100

0.08

200

200

00

0.08

0.06

0.04

000

400

0.06

0.04

0.08

0.02

00

0.02

0.00

0.06

0.04

0.02

0.00

Taking the expectation before t , we have

Let g ( w ) = f ( w ) -f ( w ∗ ) ≥ 0 . The gradient of g ( w ) is ∇ g ( w ) = ∇ f ( w ) -∇ f ( w ∗ ) = ∇ f ( w ) . Therefore, g ( w ) also satisfies the assumption of bounded variance E ξ ∼D ∥ G ( w ; ξ ) -∇ f ( w ) ∥ 2 ≤ σ 2 . Substituting g ( w ) into the above inequality, we have

We arrange the above inequality and get

Since 0 < η < 2 β , we have 2 η -βη 2 > 0 . Therefore, the above inequality can be further written as

Thus, we have related the gradient ∇ f ( w ) to the loss f ( w ) . Then give the proof of Theorem 1 as the following:

Proof. Since w f = w p -∑ T η t G ( w t ) , we have

where (a) follows from the above remark.

For the item ( I ) , we have

where (a) follows from that for a monotonically decreasing function f over the region [ a -1 , b +1] , it holds that ∫ b a -1 f ( x )d x > ∑ b i = a f ( i ) .

For the item ( II ) , we have

where (a) follows from that for a monotonically decreasing function f over the region [ a -1 , b +1] , it holds that ∫ b a -1 f ( x )d x > ∑ b i = a f ( i ) .

Taking the expectation and summing up the inequality (7) for t steps, we have

Combining the above two items, we can write the expectation of the inequality (8) as

From the expression of variance D ( x ) = E [ || x || 2 ] -E [ || x || ] 2 , we have

This completes the proof of Theorem 1.

C.2 Theorem 1 in Domain adaptation

We first give some definitions and assumptions as the previous works(44) in domain adaptation. Let X ⊂ R p be the input space and Y ⊂ R be the output space. T denotes a domain (or task), which consists of a data distribution D over X . We consider a binary classification task with the hypothesis class and the loss function is Huber loss l (ˆ y, y ) = | ˆ y -y | . Let h : X → Y denote a hypothesis that maps inputs to predicted labels, and H ⊆ { h : X → Y} denote a hypothesis class. The H△H := { h ( x ) ⊕ h ′ ( x ) , h, h ′ ∈ H} is defined as the symmetric difference hypothesis space, where ⊕ denotes the XOR operator. A H△H is a set of measurable subsets for ∀ h ( x ) ⊕ h ′ ( x ) ∈ H△H . Then d H△H ( D , D ′ ) := 2 sup A∈A H△H | Pr D ( A ) -Pr D ′ ( A ) | is defined as the distribution divergence induced by the symmetric difference hypothesis space, given two distributions D and D ′ .

From the generation bound in (44), we first give a variant based on the two data distributions of the source and target domain. Let T S and T T be the source and target domains, whose data distributions are D S and D T respectively. Let H ⊆ { h : X → Y} denote a hypothesis class with VC-dimension d . Then with probability at least 1 -δ , ∀ h ∈ H :

where e is the base of the natural logarithm, ˆ ϵ S ( h )= E x, y ∼D S | h ( x ) -y | is the empirical error of the source domain based on m observable samples, and λ = min h ∈H ( ϵ T ( h ) + ϵ S ( h )) is the optimal error on the two domains.

We regard the pre-trained and fine-tuned domains as the source and target domains respectively. The pre-trained model with parameters w p can be regarded as the hypothesis. Therefore, based on Inequality (10), we have

Inequality (9) can be equivalently written as

Applying (11) to the above inequality, we relate the model difference to the data distribution divergence between the fine-tuned and pre-trained domains.

C.3 Proof of Theorem 2

Proof. We first give the most widely used IEEE floating-point standard representation:

where s is the sign, e is the exponent value using a biased representation, and m is the fractional part. If we map w f and w p into the integers ˆ w f and ˆ w p while keeping the same bit stream, we have

When w f and w p have the same sign (i.e., s ⊕ s = 0 ), we have

where (a) is from that w f w p = 2 e f · m f 2 ep · m p = 2 ( e f -e p ) · m f m p ; (b) is from that the fraction m of float is in [1 , 2) . This completes the proof of Theorem 2.

D Algorithm Details

Table 10 provides the mapping examples of FM-Delta . We detail the mapping process as the following two steps. Firstly, convert floats to signed integers while retaining the same byte string. Secondly, monotonically map signed integers into unsigned integers. For positive values, the representation range is [00000000 ..., 01111111 ... ) , i.e. [0 , 2 31 ) . Both signed and unsigned integers are monotonically increasing in this interval. We just need to map the [0 , 2 31 ) of signed integers to the [2 31 , 2 32 ) of unsigned integers. Therefore, we flip the most significant bit for positive values, so the range becomes [10000000 ..., 11111111 ... ) . For negative values, the representation range is [11111111 ..., 10000000 ... ) , i.e. [ -2 31 , 0) . The monotonicity of signed and unsigned integers is opposite. Therefore, we flip all the bits of the negative integer to meet the alignment of monotonicity, so the range becomes [00000000 ..., 01111111 ... ) .

E Experiment Details

E.1 Model Compression Details

Table 11 presents the detailed information of the top-5 fine-tuned models in the seven families from HuggingFace, corresponding to the compression results in Figure 6.

E.2 Compression Rates on Different Layers

We present the compression rates of FM-Delta on different model layers in Figure 11, which shows that for a specific model, the compression rate on each layer generally fluctuates around a certain value.

0

100

200

300

400

500

600

700

Layer

0.0

0.2

0.4

0.6

0.8

1.0

Compression Rate

pokemon

papercut

logo

voxel

sjh

(a) UNet of Stable Diffusion.

0

2

4

6

8

10

Layer

0.0

0.2

0.4

0.6

0.8

1.0

Compression Rate

imdb

goss

wiki103

covid

comp2

banking77

(b) Transformer layers of GPT2.

0

2

4

6

8

10

Layer

0.68

0.69

0.70

0.71

0.72

0.73

Compression Rate

attn.c_attn

attn.c_proj

mlp.c_fc

mlp.c_proj

(c) Sublayers in a transformer.

E.3 Further Discussion on Throughput

Table 4 shows that FM-Delta ranks third in decompression throughput, lower than Zlib and Gzip. Here we give a further explanation about the reason.

Among all the compressors, Zlib and Gzip have similar performances since both of them use the DEFLATE algorithm, which consists of two main components: LZ77 and Huffman coding. Differently, FM-Delta is mainly based on the compression algorithm specifically for floating-point data, which uses the range coding to compress array data. Since range coding involves dealing with floating-point numbers and maintaining the current encoding interval, Huffman coding typically has a faster decompression speed than range coding(63), resulting in a higher decompression throughput of Zlib and Gzip.

E.4 Extended Fine-tuning Results

In Figure 12, we present the additional fine-tuning results on GPT2-1.5B(34). Since the model is bigger than that in the main paper, it can be seen that it overfits on the relatively simple datasets, including PTB(55) and Wikitext2(56). The simplicity of the two datasets results in their larger model difference due to their larger distribution divergence with the original data for training GPT2-1.5B, which is also analyzed in Section 3.

0

500

1000

Steps

20

40

Perplexity

0

500

1000

Steps

0.0

0.5

1.0

1.5

Euclidean Distance

0

500

1000

Steps

0.0

0.5

1.0

Compression Rate

PTB

Wikitext2

Wikitext103

LAMBADA

1BW

In Table 12, we provide more fine-tuning experiment results on both Bert-base-cased (66) and google/vit-base-patch16-224-in21k (67) using the official example script from the "transformers" github repository(68). Under these two commonly used models, FM-Delta still shows good compression results. In fact, the small delta is caused by the fine-tuning itself. The pre-trained model parameters already have strong inference capabilities, so fine-tuning does not make much difference.

E.5 Compression on Atypical models

To confirm the robustness of FM-Delta , we download two user-uploaded GPT-2 models from Huggingface: 'vicgalle/gpt2-alpaca-gpt4' is a popular instruction-tuned model with high quality, and 'jacksee/gpt2-finetuned-biochemistry' is an unpopular low-qualified model without any more information. Besides, we generate several atypical models by adding random tensors with noise values from 0 to {1, 10, 100, 1000} to the GPT-2 pre-trained model.

It can be seen in Table 13 that even on the GPT-2 added with huge noise (0,1000), FM-Delta can still compress the model into 91.5% of the original. This is consistent with our robustness analysis in Section 4, which illustrates that FM-Delta can accommodate a vast range of difference. Thus, FM-Delta is robust and reliable enough across diverse massive models in the cloud hub.

E.6 Extended Time Results

Figure 13 presents the detailed time for model upload and download under different user bandwidths on <EleutherAI/gpt-neox-20b(52), KoboldAI/GPT-NeoX-20B-Erebus(69)>. When the user's bandwidth is below approximately 800Mbps, the total time is nearly equivalent to that of the non-compression solution for FM-Delta , and it is significantly reduced for FM-Delta U due to the decreased data transfer volume. When the user's bandwidth exceeds around 800Mbps, the total time is limited by the compression throughput due to the transmission speed being faster than the compression speed (approximately 100MB/s).

50M

300M

500M

800M

1G

User Bandwidth (bps)

0

20

40

60

80

100

120

Time (min)

Comp.

Decomp.

Load pre.

Full

Compressed

50M

300M

500M

800M

1G

User Bandwidth (bps)

0

25

50

75

100

125

150

Total Time (min)

NoComp.

FM-Delta

FM-Delta

U

50M

300M

500M

800M

1G

User Bandwidth (bps)

0

25

50

75

100

125

150

Total Time (min)

NoComp.

FM-Delta

FM-Delta

U

(a) Time for loading pre-trained model, model compression, decompression and transfer.

(b) The total time for model upload with non-compression, FM-Delta , and FM-Delta U .

(c) The total time for model download with non-compression, FM-Delta , and FM-Delta U .

E.7 Cost Analysis

All the experiment results show that FM-Delta can achieve a compression rate around 50% in most cases with around 100MB/s throughput. There is an interesting question raising - in practice for the cloud, what would be the cost of decompressing the models and would it be less than the cost of storing them decompressed in the cloud? We mainly discuss the cloud cost in terms of storage and computation.

We assume that in cloud there is a model with size M , having n fine-tuned variants. And p of those fine-tuned variants are inactive (monthly downloads<10). We give a loose definition that requests received in the same minute we treat as concurrent requests. Assuming that the 10 downloads of inactive models are not concurrent, the probability of a single inactive model being downloaded in a certain minute is 10 30 ∗ 24 ∗ 60 . k represents the number of concurrent download requests in a given minute and each request corresponds to a different fine-tuned model. Then we can think of model download as a binomial distribution:

Our goal is to determine the maximum total number n of loadable models on the current server, which ensures that the probability of the concurrency number exceeding the server threshold t is less than 1%, i.e., ∑ n k = t P ( X = k ) ≤ 0 . 01 . Referring to Table 7, we set the threshold to 16, since its total time consumption is no more than twice that of a single task. We utilize python " scipy " library to compute and get the maximum value n max = 35300 .

We refers to Table 1 to set the inactive proportion at p = 89% . The model which can be compressed with FM-Delta in a minute is about 6GB, so we set the model storage size 4 M = 6 GB . If 1TB costs 60 dollars, we give the original storage cost on inactive models without compression:

With the compression rate 50% of FM-Delta , the current storage cost is

In addition, we directly present the purchase price of the current computing server to give the reader a more intuitive sense, even though the cloud server has a lot of computing power and we assume that there are idle computing resources. Our current server mainly consists of the cpu (one AMD Ryzen 9 5950X) and memory bars (Kingston Fury Beast 32GB × 8), whose price in Amazon are $379 and 8 × $100 respectively. Therefore, the purchase cost c server of the server is

From the above analysis we can conclude that the total cost of the cloud server is at least saved 11044 -5522 -1179 = 4343 dollars on 35,300 inactive 6GB models. The example shows the storage costs are down to 57%. Such cost savings through lossless compression is certainly promising for the cloud in the real world.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please check Section 1, especially the last paragraph.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please check Section 6 for limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please check Theorem 1 and Theorem 2, along with their proof in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please check the experiment settings in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please check the github link https://github.com/ningwanyi/FM-Delta for our code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please check the experiment settings in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Our lossless compression algorithm does not involve any training errors.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please check the experiment settings in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research is fully with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please check our contribution in Section 1. Our lossless compression scheme will significantly reduce storage costs for cloud providers with minimal impact on user experience.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our lossless compression poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited all the involved researches, datasets and models.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We provide our code in the github link https://github.com/ningwanyi/FM-Delta. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FM-Delta, which addresses significant storage overhead caused by fine-tuned LLMs. It maps model parameters into integers and entropy codes their small differences from pre-trained models, reducing cloud storage consumption by about 50% on average.\n\nSTRENGTHS:\n- This paper proposes FM-Delta, a method for jointly compressing pre-trained and fine-tuned models, achieving a compression ratio of up to 2x for fine-tuned models.\n- Compressing large models is crucial, and the approach of jointly compressing fine-tuned and pre-trained models makes sense. Traditional compression algorithms struggle with large models, so a delta-based method intuitively provides a feasible compression rate and is an interesting direction.\n- The paper offers a formula for the distribution shift between fine-tuned and pre-trained models under given assumptions, providing valuable guidance.\n\nWEAKNESSES:\n- Despite the interesting concept of delta compression, I have some concerns about its practical applications. Could the author please elaborate on specific scenarios where this method can be applied? What kind of situations require frequent downloading of finetuned models from the network?\n\n- Has the author considered comparing the proposed method in this paper with quantization? For example, quantizing delta. There are existing studies showing that quantized LLM still maintains some compressibility, roughly around half the compression rate [1]. In other words, my question is: what is the necessity of performing lossless compression in this setting? Because the compression-decompression process inevitably introduces additional overhead, especially at the user end.\n\n- While the first half is well-written, the writing from the Method section onwards needs improvement. Some technical terms need to be explained in detail, such as \"most significant bit\" mentioned in 4.1 and \"range\" in \"difference range\" mentioned in 4.2. Understanding these concepts is crucial for comprehending the algorithm, yet I couldn't find corresponding explanations in the text. The subsequent method descriptions are also somewhat confusing.\n\n- Why use range coding instead of other entropy coding methods? I noticed the experiments used tools like Gzip, but currently, methods like zstd are more popular. Could the author consider trying zstd compression?\n\n- I noticed that the reported compression throughput in the final experiments is approximately 100MB/s, which is not particularly fast in practical terms. There seems to be significant room for improvement in this speed.\n\n- Some related works are not mentioned, such as [1] [2]. \n\n[1] Mao Y, Wang W, Du H, et al. On the compressibility of quantized large language models[J]. arXiv preprint arXiv:2403.01384, 2024.\n\n[2] Hershcovitch M, Choshen L, Wood A, et al. Lossless and Near-Lossless Compression for Foundation Models[J]. arXiv preprint arXiv:2404.15198, 2024.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel lossless compression scheme FM-Delta specifically for storing massive fine-tuned models in cloud.   \n\nFM-Delta maps fine-tuned and pre-trained model parameters into integers with the same bits, and entropy codes their integer delta. In this way, cloud only needs to store one uncompressed pre-trained model and other compressed fine-tuned models.   \n\nExtensive experiments demonstrated that FM-Delta efficiently reduces cloud storage consumption for massive fine-tuned models by an average of around 50% with only negligible additional time in most end-to-end cases.\n\nSTRENGTHS:\nThe research topic of compressing finetuned large models on the cloud is novel and has strong practical meaning in the age of foundation model. The statistics provided in Figure 1 and Table 1 clearly show the necessity of developing such a method to compress the large number of finetune models.   \n\nI like all the statistics of those large models on HuggingFace. The authors devote a lot of effort to provide a systematic study on this novel research topic.  The experiments are extensive.  \n\nTheoretical results are given in Theorem 1 to analyze the growing rate of model difference, which serves a great motivation for the proposed delta coding method.   \n\nThe bit redundancy of model difference is analyzed in Theorem 2, proving a solid theoretical foundational for the robustness of the proposed method.\n\nWEAKNESSES:\nIt is not clear how range coding is applied. Range coding needs a probability table for the coding symbols. Did the paper use the symbol frequency as the probability? If this is the case, the probability table should also be transmitted in the compressed stream to ensure correct decoding on user side. \n\nFollowing the previous quesrtion, it would be interesting to see how the probability estimation affects the compression ratio. For example, in Boosting Neural Representations for Videos with a Conditional Decoder, CVPR 2024, a Gaussian Entropy Model is proposed to model the distributin of neural network weight for entropy coding, where only two scalar values are transmitted for each weight or embedding.\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nThe limitation is well addressed in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to compress the differences between a pretrained model and a full fine-tuned model to save storage space on cloud servers where the full fine-tuned model is stored. For compression, the pretrained weights and full fine-tuned weights are first converted to unsigned integers and then subtracted from each other. Subsequently, range coding is applied to compress redundant zeros, while non-zero values are retained as raw bits. Consequently, this method demonstrates a higher compression ratio than existing lossless compression methods and also shows robustness across various models and datasets.\n\nSTRENGTHS:\n- The proposed method for addressing the storage space issues of massive fine-tuned models is highly novel.\n- It achieves a higher compression ratio compared to existing lossless methods and demonstrates robustness across various models. Additionally, it illustrates the trade-off in compression rates resulting from fine-tuning, highlighting the robustness of the proposed method.\n- The paper provides both theoretical and experimental evidence to demonstrate the robustness of the proposed method.\n- From the perspective of practical users, the proposed method is shown to be efficient, and the paper suggests further ways to accelerate the process.\n\nWEAKNESSES:\nThe proposed method is effective only for full fine-tuned models and not for peft (parameter-efficient fine-tuning) models. The impact of the proposed method may vary depending on the proportion of full fine-tuned models. Although Table 1 shows the number of models for the six most popular models, it does not provide information on the overall proportion of full fine-tuned models within the entire Hugging Face model repository.\n\nQUESTIONS:\nWhat is the proportion of full fine-tuned models within the entire Hugging Face model repository?\n\nLIMITATIONS:\nIncluded in Weakness", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}