Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models
Authors: Hui-Po Wang
Mario Fritz
Venue: NeurIPS 2024 poster (2024)
Keywords: large-scale language models
lossless gradient compression
Primary area: generative_models
TL;DR: We show that LLMs can compress gradients in a zero-shot setting, demonstrating a promising direction toward general parameter priors.

Abstract: Despite the widespread use of statistical prior models in various fields, such models for neural network gradients have long been overlooked. The inherent challenge stems from their high-dimensional structures and complex interdependencies, which complicate effective modeling. In this work, we demonstrate the potential of large language models (LLMs) to act as gradient priors in a zero-shot setting. We examine the property by considering lossless gradient compression -- a critical application in distributed learning -- that depends heavily on precise probability modeling. To achieve this, we introduce LM-GC, a novel method that integrates LLMs with arithmetic coding. Our technique converts plain gradients into text-like formats, enhancing token efficiency by up to 38 times compared to their plain representations. We ensure that this data conversion maintains a close alignment with the structure of plain gradients and the symbols commonly recognized by LLMs. Our experiments indicate that LM-GC surpasses existing state-of-the-art lossless compression methods, improving compression rates by 10% up to 17.2% across various datasets and architectures. Additionally, our approach shows promising compatibility with lossy compression techniques such as quantization and sparsification. These findings highlight the significant potential of LLMs as a model for effectively handling gradients. Code is available at https://github.com/hui-po-wang/LM-GC.

Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models
Hui-Po Wang Mario Fritz
CISPA Helmholtz Center for Information Security
Germany Chui wang, fritz)@cispa.de
Abstract
Despite the widespread use of statistical prior models in various fields, such models for neural network gradients have long been overlooked. The inherent challenge stems from their high-ding -ibbeniimniional structures and complex interdependencies, which complicate effective modeling. In this work, we demonstrate the potential of large language models (LLMs) to act as gradient priors in a zero-shot setting. critical We examine the property by considering lossless gradient compression application in distributed learning that depends heavily on precise probability modeling. To achieve this, we introduce LM-GC, a novel method that integrates LLMs with arithmetic coding. Our technique converts plain gradients into text-like formats, enhancing token efficiency by up to 38 times compared to their plain representations. We ensure that this data conversion maintains close alignment with the structure of plain gradients and the symbols commonly recognized by LLMs. Our experiments indicate that LM-GC surpasses existing state-of-the- art lossless compression methods, improving compression rates by 10% up to 17.2% across various datasets and architectures. Additionally, approach shows promising compatibility with lossy compression techniques such as quantization and sparsification. These findings highlight the significant potential of LLMs model for effectively handling gradients. Code is available at /tttts://ggiie com/hui- Sawuuu-oowwannniiil wang/LM-GG
Introduction
Statistical prior models have been applied successfully in various fields. including image denoising and super-resolution [Ulyanov et al. 2018 Gandelsman et al.] 2019]. vision task adaptation (Wang et al. 202I], and low-resource language tasks [Baziotis al. 2020 Brown et al., 2020]. However, their use modeling neural network gradients has been largely neglected. The potential reasons for this oversight might include (1) the high-dimensional nature of gradients, which makes them less intuitive to analyze; (2) the difficulty of collecting representative gradient data; and (3) the significant challenge of ensuring generalizability to unseen data, given the substantial effort required.
Instead of developing a model from scratch, this work investigates the potential of leveraging pre- trained large-scale language models (LLMs) as gradient priors in zero-shot setting. We explore this potential through the lens of lossless gradient compression, a vital application in federated and distributed learning environments. The success of this compression heavily depends precise probability modeling. An effective statistical model can significantly improve compression efficiency, whereas an inaccurate model may lead to poorer compression outcomes and could even increase the data size post-compression.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To address this, we introduce LM-GC, an innovative coding scheme for gradient compression that integrates pre-trained large language models (LLMs) with arithmetic coding. Our method involves transforming gradients into text-like formats that are easier for L.LMs to reason. Specifically, we convert the raw bit data of floating points into hexadecimal numbers and incorporate separators, such as spaces, to clarify the concept of floating points for LLMs. This serialized text is then processed by pre-trained tokenizers and LLMs to determine the probability of each token, which is subsequently utilized in arithmetic coding. Empirical evidence supports that these design choices significantly enhance gradient modeling and, consequently, compression efficiency.
Overall, our contributions are summarized below.
We introduce LM-GC, novel coding scheme that integrates LLMs with arithmetic coding. This method utilizes LLMs as powerful prior models for gradients, setting a new benchmark in state-of-the-art lossless gradient compression.
LM-GC demonstrates that transforming raw gradients into formats that LLMs can understand significantly impacts their reasoning capabilities and token efficiency. Empirical evidence indicates that this approach can affect compression rates up to 70% with recognizable symbols and 40% with proper separators. These findings underscore the critical role of effective conversion in enhancing compression performance.
Experimental results demonstrate that LM-GC significantly surpasses existing baselines, including PNG, FLAC, LZMA, GZIP, and FPZIP, by 10% to 17.2% across various architectures and datasets. Additionally, approach complements existing lossy compression methods such as quantization and sparsification, paving the way for advanced gradient compression techniques.
2 Related work
Large-scale language models. Language models aim to model the relation between texts. This problem has been extensively studied in recent decades via various approaches such as statistical models [Jelinek] 1998 and recurrent neural networks [Hochreiter and Schmidhuber 1997]. Recently, the emergence of transformer-bast has revolutionized the entire field, driving research into large- large-scale language models (LLMs). Models, such as those from GPT [Achiam al.
models [Vaswani et al.) [017]] along with large-scale text corpora
2023 Brown et al. 2020) and LLAMA (Touvron al.) 2023 Zhang et al., 2024 Geng and Liu 2023 families, are capable of solving diverse tasks in natural languages and demonstrate incredible generalizability toward unseen novel tasks, even across modalities [Mirchandani et al.) 2023 Gruver et al., [024]. Notably, recent work by Deletang et al. (2024) also explores the use of language models as general compressors. Our goal
investigate the potential of LLMs as strong prior specifically for gradients. Additionally, we offer practical considerations for handling floating-point data when structures exist within the data to be compressed.
In this work, we demonstrate for the first time that LLMs can understand the structure of network gradients, accurately modeling their probability distribution in a fully zero-shot manner. We verify our finding by taking LLMs as priors for arithmetic coding, yielding state-of- of-thh--rtt lossless gradient compression under various settings.
Deep generative priors. An ongoing research direction beyond traditional statistical modeling is learning a deep generative model from massive data and leveraging the model as a "deep" prior. The concept has been widely considered in many applications, such as image denoising and super- resolution [Ulyanov et al. 2018 Gandelsman et al.) 2019], vision task adaptation (Wang et al. Chang et al. 2019] and low-resource language tasks [Baziotis al. 2020 Brown al. Although strong priors can facilitate various downstream applications, training such models
202T
2020
for gradients can be costly and challenging due to their high dimensionality. Additionally, the generalizability of these models is often concern and may be limited to specific types of networks [Ha jet al. 2016 Wang al. 2024b). Instead of training a model from scratch, our work explores the potential of using -ietth--shiff LLMS as strong priors over gradients. This will minimize the cost of training deep prior models and may inspire applications like gradient denoising and anomaly detection.

---

Arithmetic Encoder
0100
1101
0010
Tokenizer
I
.55
I
.51
D
Pre-trained
LLMs
BOS
D
...
2
Token Prob.
0
I
40 I
.45
Grouped Text
(foss BOS)
Figure I: Overview of LM-GC. Our method initially converts every 4 bits into hexadecimal numbers and groups them with separators in between, e.g., commas in the figure. The grouped text is then input to a pre-trained, frozen tokenizer and LLM to produce the probability of each token. These probabilities are used for arithmetic encoding, where a line segment between and is repeatedly split according to the token probability until reaching predefined maximum length. Any number from that region (e.g.. the midpoint) can accurately represent the original data. We provide example of how arithmetic coding works in Sec.
Gradient compression. Gradient compression is a crucial technique, particularly in federated and distributed learning, where communication costs serve as the main bottleneck for scalability. Existing efforts have extensively studied lossy compression, which trades information precision for compression efficiency. For example, quantization [He al., 2020 Alistarh et al.) 2017 Bernstein
al.
2018 replaces floating points with fewer bits, while sparsification |aangni et al. 2018 Alistarh
ctt al. 2018 transmits only a subset the original gradients. Other approaches explore novel
optimization strategies such progressive learning (Wang al. 2022 and communicating synthetic
images [Wang et al. 2024a Xiong et al., 2023
In contrast, lossless compression, which allows compressed data to be perfectly reconstructed without sacrificing information, is rarely investigated in the field of gradient compression nowadays. The challenge lies in developing a better statistical model for gradients. In this work, we demonstrate that LLMS can model the probability distribution of gradients zero-shot setting, Building on this finding, our method combines L.LM-based modeling with arithmetic encoding and outperforms existing baselines such as PNG (Boutell| 1997], FLAAC [Coalson, 2008 1.ZMA [Pavlov] [2019), GZIP [Deutsch] 1996]. and FPZIP [Lindstrom and Isenburg [0066]. which are designed for modalities other than gradients. By integrating our approach with existing lossy compression techniques, we may pave the way for more advanced gradient compression schemes.
3 Background
In this work, we aim to explore the potential of using LLMs as prior for gradients and leverage lossless compression an examination task. We review the essential background knowledge below.
Lossless compression. The fundamental principle of lossless compression is to reduce the size of data while ensuring can fully reconstructed. This typically achieved by eliminating the statistical redundancy inherent in the data. Given a sequence of symbols so.N S drawn from probability distribution Ps. the objective is to devise a compression function g S - C. This function maps the original data 8 to unique (decodable) binary code c, ensuring that the length of denoted by É(c), is less than or equal to the length of 8, ((s). The source coding theorem [Shannon] 2001 states that the expected minimum length of coded message cannot be shorter than the Shannon entropy of the original data, denoted as ((s) > (S)). Here, H(S) = Ee- log, (s(s)) represents the entropy. This implies that any compression resulting in a length shorter than (S)
necessarily involves loss of information, preventing perfect reconstruction of the original data.
Arithmetic coding. As a means to achieve lossless compression, arithmetic coding Rissanen and Langdon 1979 provides nearly optimal message length H(S) < E(c) H(S) + 2/((s)
3

---

on average [Sayood 2017 To implement it, one must employ a statistical probability model of the data, denoted PAC. Ideally, this model PAC should closely mirror the true distribution Ps- The closer these distributions are, the more effective the compression performance will be. Conversely, significant deviations can result in a compressed data length {(c) that exceeds the original data length {(s). Notably, Most existing methods, such as CABAC, incorporate adaptive priors, meaning the probability PAc adapts based on data context. However, as we will demonstrate in Sec. 5 these methods are not optimized for gradients and are thus outperformed by zero-shot LLM prior.
The arithmetic encoder begins with an interval [0. 1) between 0 and 1. For each input symbol s, the interval is subdivided according to the probability Pac(s). The corresponding interval is then selected as the new interval. This process continues until the entire imput stream is finished or reaches the maximum length. Any number existing in the final interval suffices to represent the compressed data. Similarly, the decoder takes the encoded output as input and can perfectly reconstruct the data by repeatedly looking up the intervals.
We provide an example illustrating how arithmetic coding works given a fixed statistical prior below.
Example of arithmetic coding. Consider message consisting of only two symbols, A and B, where A occurs with probability PAc(A) = 0.8 and B with Pxc( = 0.2. The encoding interval gets subdivided into a larger part (0 to 0.8) for A and smaller part (0.8 to 1) for B. If the message is "AAAB". the interval narrows from [0, 1] to [0, 0.8), then [0, 0.64), then [0, 0.512), and finally [0.4096, 0.512] after processing the B. Any number (typically the midpoint for simplicity) within the final interval can represent the entire sequence. This number is then converted into a binary code, which is the compressed output. The final result is (0.4608) - (0.01110101),, which takes only 8 bits compared to 4 bytes of storage for the ASCII format.
Language models. Language models are designed to model the relation between text symbols. Given text stream, S = {s consisting of N symbols, modern language models typically begin with a tokenization process f: S + T (e.g.. Byte-Pair Encoding (Sennrich et al. (2015)) that maps the entire stream set of K tokens T [t) Then, the model predicts the probability follows.
K
p(t) = p(t
tk)
II
pt BOS, tzn),
k=1
where the BOS token denotes special token indicating the beginning of the sentence.
LM-GC
We introduce LM-GC, method that integrates arithmetic coding with pre-trained large language models (LLMs) to address the lack of gradient-specific priors in arithmetic coding. It's important to note that LLMs are originally trained on extensive text corpora and do not encounter gradients or model parameters during this training. A significant challenge is enabling LLMs to comprehend the structure of gradients. Our method involves two main steps: serialization and compression. In scrialization, we convert the 32-bit floating points of gradients into a format understandable by LLMs, which we call grouped text. This text is then fed into the LLMs, which predict the probability of each token in an autoregressive manner and thus accomplish compression using arithmetic coding.
Serialization. We first note that gradients are represented as 32- floating points, with values ranging from -3.40282347 x 10+38 to 1.175449435 x 10 10-38 Due to significant variations in their magnitudes and the often ambiguous importance of each gradient element, directly inputting these values into large language models (LLMs) is impractical. LLMs have fixed token limit, and representing a single gradient in plain form would consume excessive tokens, compromising the context's depth.
To address this, our method, LM-GC, initially divides the floating points into several disjoint 4-bit partitions, which are then encoded into hexadecimal numbers as illustrated in Figure [ This encoding strategy allows for a token savings of approximately 38 times compared to using plain gradients, particularly under extreme value conditions.

---

Furthermore, we organize every eight decoded hexadecimal numbers (equivalent to 4 bytes) by inserting separator between them. This format provides LLMs with structured representation of how floating point number is typically presented. Our experiments demonstrate that separators are crucial in effectively modeling gradients, especially gradients derived from sophisticated architectures and datasets.
Compression. After serializing the gradients into grouped text S. we process this text through tokenizer generate a of tokens T. These tokens are then fed into a pre-trained large language model (LLM), denoted by M, which predicts the probability of each token as follows:
K (T) = II ptx BOS, t<a).
(2)
This equation indicates that the LLM sequentially predicts the probability of the next token, starting with the BOS (beginning of sequence) token, which is used to calculate the probability of the first token, (Lm = (BOS)). The BOS token serves primarily as a contextual cue and is not included in the compression.
During compression, (TM(T) acts the statistical model, PAC, for arithmetic coding. In the decompression phase, the process begins with the BOS token, retrieving (t, to decode the first token. This decoding cycle continues until the maximum window size of the LLM is reached.
Experiments
5.1
Setup
Datasets and models. Our experiments consider three types of LLMs as the compressor, includ- ing Tinyllama 1.1B (Zhang et al.) 2024). Openllama 3B [Geng and Liu 2023), and LLAMA 7B (Touvron et al. 2023], ranging from a smaller to medium model size. All models can accept up to 4096 tokens. Unless stated otherwise, we use a context window size of 2048 by default. To ensure generalizability, we conduct experiments on four model architectures, including a three-layer convolution net (ConvNet), VGG-16 [Simonyan an and Zisserman 2015). ResNet (He et al., 2016). and vision transformer (VIT) (Dosvviskkiy et al. [021]] The models are trained on three datasets, MNIST LeCun et al.] 2010]. CIFAR-IO Krizhevsky et al. 2009]. TinyImageNe
(Le and Yang 2013]. under different settings. MNIST is digit classification task containing 10 digits and 60000 images. On the other hand, CIFAR-10 and TinylmageNet are image classification tasks. CIFAR-10 contains 50000 images of 10 classes, while TinylmageNet contains 100000 images for 200 classes. All images are rescaled to 32 by 32 in the experiments.
Evaluation protocols. If prior describes data well, lossless compression can achieve better efficiency. To test the efficiency of using LLMs priors, we first train models on different datasets for 200 epochs, collecting gradients every 200 batch steps, resulting in a data pool of approximately 400 checkpoints for compression evaluation. Due to computational time constraints, we sub-sample 10 checkpoints from the pool for the subsequent experiments unless stated otherwise. All the experiments repeated at least three times, and the standard deviations reported accordingly. We measure compression efficiency by the compression rates defined as follows.
Compression Rate (%) = 100 X Compressed Data Size
Original Data Size
Baselines. We compare our method state-of ofte-----hh--trr the-hrrr lossless compression techniques that originally targeted different data types. PNG [Boutell 1997] is one of the most common lossless compression codecs for images. On the other hand, FLAC [Coalson]
2008 a common audio compression format. Lastly, I.ZMA [Pavlov| 2019) and GZIP [Deutsch] 1996 are codecs used by 7-zip software and 7z compression format. FPZIP [Lindstrom and Isenburg (006) proposed for scientific floating-point data, particularly suitable for data with up to 4D structures.
5

---

Traditional codee
LM-GC (Ours)
Unchunked
Chunked
ISO
H,,
H.
Hot
Hammi
PNG
43.30=1.3
49.18+1.1
FLAC
52.37=0.6
50.46+0.6
GZIP
42.42=0.3
47.10:0.4
LZMA
41.91=0.0
47.36+0.1
FPZIP
41.26=0.8
49.27+0.3
Tinyllama 1.IB
117.3840.0
36.3020.8 38.83+0.4 38.4010.6 38.4620.1
43.4520.6
Openllama 3B
71.85+0.2
37.0720.1 32.32+0.3 34.3 33.0720.5 33.57:0.2
LLAMA 2 7B
109.07:0.2 72. 10=05 32.26+0.5 32.9620.3 32.21 32.78=0.4
Table 1: Gradient compression rate using PNG, FLAC, GZIP, LZMA, FPZIP, and our method with various language models. Our method considers different serializations including iso-8859-1 (ISO), hexadecimal numbers without separators (H₂)) and with spaces (H2), commas (H,), commas+spaces (Hs+e), and semicolons (Hsemi) to group every four bytes from the same floating point.
Implementation. We implement method in Pytorch and Huggingface. The checkpoints of pre-trained L.LM models are loaded from the Huggingface hub. We adapted the arithmetic coding from Torchac to fit application. We experiments cluster with NVIDIA A100 40GB GPUs and AMD EPYC 7402 24-Core Processor. All of the experiments can fit in one single A100.
5.2 Compression effectiveness
We first conduct compression experiments on gradients collected from a ConvNet trained on CIFA-10 to show that LLMs can model gradients even without seeing such data during training. Our method considers three LLMs, namely Tinyllama, Openllama, and LLAMA 2. as the priors for arithmetic coding. We also consider types of serialization, including decoding every byte with ISO-8859-J (ISO), projecting every 4 bits to hexadecimal numbers without separators (H,), and with space (H,), commas (He), space and commas (He+s). and semicolons (Haemi) as the separators. These settings outline the importance of serialization and its effect on gradient modeling. We report two settings for the baselines. The first is a chunked version, where the compressor sees a chunk of size 512 bytes every time, whereas the other one, namely the unchunked version, takes advantage of the pseudo infinitely large context length to yield the best statistical modeling.
Table T shows that our LM-GC consistently outperforms baseline codecs when serialization is properly managed. For example, ISO and H,, for LLAMA 2 perform worse than the baselines. In particular, ISO encodes gradients into symbols less familiar to LLMs, yielding up to 70% performance difference compared to settings like H,. The lack of separators may confuse language models, causing performance degradation of 40% on LLAMA 2. These results highlight the crucial role of serialization in aiding LLMs' understanding. Furthermore, compression efficiency increases as the model size grows from 1.1B to 7B, suggesting that more sophisticated models may better understand relationships between data elements, resulting more effective compression.
5.3 Ablation study
In this section, we provide a series of ablation studies to provide insights into how design choices affect LLMs and the resulting prior models.
Architectures. To further understand the generalizability to different architectures and the effect of serialization, we continue with an experiment on different architectures. We extend the experiments to three additional architectures. VGG-16 contains deeper layers compared to ConvNets. ResNet- 18 further introduces cxiipcconnctions and batch normalization, verifying our LM-GC on common design choices in modern machine learning. Lastly, ViT is built upon transformer blocks, showing that LLMS can reason beyond convolution layers. As shown in Table 2, we first observe that the performance of all methods drops as the models become more complicated, while our method remains the best among the baselines. This finding suggests that our method can better capture complex structures within the gradients. Moreover, observe that serialization with separators generally

---

Traditional codec
Ours (Tinyllama 1.1B)
PNG
FLACC
GZIP
1.7AA
FPЛIP
H
H
He
Hot
ConvNet
43.30
52.37=0.6 42,42=0.3 41.91=0.0 41 11.2660.75
36.30+0.8
38.83+0.4
38.40#0.6 38.4640.1
VGG16
95.61=0.2
91.91=0.0 91.27=0.1 89.1520.17 83.23+0.0 73.42=0.1 75.32+0.2 73.97=0.1
ResNet18
97.22
92.47=0.0 91.72=0,1 90.72+0.07 83.20#0.3 73.57=0.1 75.55+0.3 73.95#0.2
VIT
94.50=0.4
89.20=1.2 87.98#1.2 89.77=0.48 78.6543.3 70.83+1.8 72.60+2.0 71.62+1.7
Table 2: Gradient compression (%) for convolution neural networks (ConvNet), VGG-16, ResNet-18, and ViT trained on CIFAR-10.
Traditional codec
PNG
FLAC
GZIP
L.ZAA
FPZIP
LM-GC (H,)
Impr.
MNIST
50.0524.3
55.20=1.7
45.05=5.2
43.1941.3
44.6240.6
39.38+1.4
8,8%
CIFAR-10 10
43.30=1.3
52.37=0.6
42.4240.3
41.9140.0
41.2640.8
38,83=0,4
5.9%
TinylmageNet
96.08=0.1
107.36=0.0
92.18+0.0
91.06+0.1
86.88+0.1
71.90=0.0
17.2%
Table 3: Compression effectiveness on MNIST, CIFAR-1O, and TinyImageNNe datasets. We use Tinyllama the compressor compress the gradients of ConvNets. The raw data converted to hexadecimal numbers with spaces as the separator. The improvement (Impr.) over the best baseline highlights the capability of LM-GC in modeling complex gradients.
performs better than one without separators. outlines the importance of separators, especially when the data to be compressed becomes more intricate.
Datasets. The previous experiment suggests that LM-GC models gradients more accurately than the existing baselines, especially when considering complex structures. We further explore this dimension by considering two additional datasets, MNIST and TinyImageNet. Table presents the result comparing our method with Tinyllama the baselines. Datasets like EmnllmageNe introduce higher compression difficulty due to the complex task. However, LM-GC demonstrates consistently promising performance across datasets. The improvement over the best baselines (FPZIP) increases as the dataset becomes sophisticated. This finding aligns with the result in Table 2 that our method is generalizable and better at capturing complex structures than the existing codecs that are not optimized for gradient compression.
Context window size. LM-GC takes LLMs as prior over gradients. One natural question is whether the I.IMs really consider the context and yield accurate probability modeling. Ideally, similar to the traditional codec, if we provide larger context window, the statistical model should be able to
Ablation Study on Context Window Size
76
Compression Rate (%))
74
256
5i2
1024
2048
4096
Context Window Size
Figure 2: Compression rates of LLAMA 2-7B using context window sizes of 256, 512, 1024, 2048, and 4096. The compression rates improve as the context window increases.
7

---

TinyLLAMA
1e7
LLAMA2
84
Compression Rate
84
Compression Rate Tokens
Compression Rate (%)
7.0
7.0
Number of Tokens
80
80
6.5
6.5
78
78
76
6.0
76
6.0
74
74
72
5.5
5.5
70
70
5.0
68
2
4
None
1
2
None
Bytes per Group
Bytes per Group
Figure 3: Ablation study on numbers of grouped bytes. We report the compression rates and the number of tokens yielded by different serializations. The settings that closely obey the data format perform better. However, smaller numbers yield higher computation overhead.
Sparsification
Quantization
102
Sparsification only Sparsification + LZMA
102
Compression Rate (%)
Sparsification + Ours
104
Quantization only
Quantization LZMA
Quantization
100%
50%
25%
10%
32 bits
16 bits
8 bits
bit
Sparsity Level
Quantization Level
Figure 4: Compatibility analysis with sparsification (left) and quantization (right).
reason from the context and thus result in higher compression efficiency. Instead of using a default context window size of 2048 tokens, we conduct an ablation study in Fig. [ The result shows that the performance drastically improves when the context window size increases, suggesting that LLMSs indeed leverage the context. However, we note that the improvement seems to be saturated at the end. A larger context window also implies higher hardware resource demands, leaving potential trade-off in practice.
Byte grouping. In addition to the decoding schemes analyzed in the previous experiments, we demonstrate that grouping converted text significantly affects performance. Recall that floating point consists of bit for the sign, 8 bits for the exponent, and 23 bits for the mantissa. Components with the same functionality should be grouped as closely as possible. To verify this hypothesis, we conducted experiments on TinYLLAMA and LLAMA 2 with bytes per group (BPG) set to 1, 2, 3, 4, 8, and none (i.e., no grouping, denoted as H.). The results Fig. 3 show that BPG set to 1, and 4 (our default setting) perform the best, while BPG equal to 3. which covers three components, and none perform worst. It indicates that serialization should resemble the structure of data to be compressed. Notably, although BPG equal to and 2 performs well both models, smaller BPG will add more separators and increase the total amount tokens, introducing the additional computation overhead to the compression.
Comparison to run-length encoding. Lastly, compare our method to run-length encoding, the simplest adaptive compression scheme, as shown Table in the appendix. The results indicate that although serialization may slightly improve compression rates, run-length encoding is incffective

---

for compressing gradients. Combined with the earlier findings, this suggests that simple adaptive methods are unsuitable for handling complex yet structured gradient data.
5.4 Compatibility
Gradient compression, a crucial technique in federated learning, typically involves lossy compression methods. We demonstrate that LM-GC approach is compatible with lossy techniques such as quantization and sparsification. Specifically, we consider linear quantization, which uniformly discretizes value ranges according to the allotted bits. Given a vector and bits, the quantization process can be formalized as follows.
min
max U min X 11).
In practice, only the indices I E {0, n for each element are communicated. Therefore, we map the data to the indices before conducting compression. Moreover, we consider sparsification, which selectively transmits a subset of gradients based the specified proportion. When considering sparsification, it is important to note that the gradients remain as 32-bit floating points. For this experiment, we investigate quantization levels of 16, 8, and bit (i.e., SignSGD [Bernstein et al. 2018), ) and sparsification levels of 50%, 25%, and 10%.
We present compatibility analysis in Fig. 4 The results indicate that integrating lossless compression techniques such as LZMA and LM-GC enhances compression rates beyond plain lossy compression. However, LZMA shows limited improvement across all settings, particularly with sparsification. In contrast, our method consistently delivers improvements across all settings, achieving notable compression rates in addition to lossy compression. These findings underscore the potential of LLMs as a prior for gradient compression, even with the incorporation of additional compression schemes, suggesting promising new research direction in leveraging LLMs for compression.
6 Discussion and Limitation
Throughput. Despite the promising performance and generalizability, the throughput of LM-AC can be further optimized. Currently, our approach requires approximately 4 hours to compress just 28 MB. This bottleneck arises primarily from two components: LLMs and arithmetic coding, For LLMs, performance can be accelerated through techniques such as quantization [Frantar et al. 2023), faster attention mechanisms [Dao et al., 2022], KV cache Hooper et al. [2024], and model pruning Ma et al. [2023). Looking ahead, one could explore distilling language models (Hsieh] al. 2023). as many functionalities may not be necessary during compression. Additionally, our implementation is significantly hindered by arithmetic coding and CPU limitations. Adopting more
efficient implementation, such as pure C++ programs, or utilizing CPUs with superior single-thread processing speeds could effectively mitigate these constraints.
Broader impact. Our work highlights the potential of leveraging pre- trainedieed LLMs as priors for gradients. Immediately, this offers an advanced tool for gradient compression that reduces resource demands in federated and distributed learning environments. Over time, these priors could be utilized for gradient denoising, enhancing differential privacy training, or identifying adversarial gradients concealed within federated learning clients. However, this approach may also enable more subtle adversarial gradients, guided by these stronger priors.
7 Conclusion
We presented LM-GC, the first lossless gradient compressor that integrates arithmetic coding with LLMs as prior models for gradients. Our experiments show that pre-trained zero-shot LLMs are highly effective as gradient priors, setting a new state-of-th ootee--f-th-arr for gradient compression. Additionally, our findings indicate that the precise serialization of gradients substantially improves the reasoning abilities of LLMs and significantly impacts compression performance, warranting further exploration. The versatility of LM-GC sets the stage for developing more sophisticated gradient compression methods that directly incorporate LLMs. Overall, while our results in zero-shot settings are promising, the potential of expanding this approach to include few-shot learning, prompt engineering, and optimization of throughput efficiency remains open for further exploration.

---

Acknowledgements
This work partially funded by Medizininforraiik-PPatnnn Plattffrrm "Privatsphäree-schuut schutzende Analytik in der Medizin" (PrivateAIM), grant No. 01ZZ2316G, and Bundesministeriums fur Bildung und Forschung (PriSyn), grant No. 16KISA029K. The work also supported by ELSA European Lighthouse on Secure and Safe AI funded by the European Union under grant agreement No. 101070617. Moreover, The computation resources used this work are supported by the Helmholtz Association's s Initiative and Networking Fund on the HAICORE@ FZJ partition. Views and opinions expressed are, however, those of the authors only and do not necessarily reflect those the European Union European Commission. Neither the European Union nor the European Commission can be held responsible for them.
References
Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
D. Alistarh, Grubic, Li, R. Tomioka, and M. Vojnovie. Qsgd: Communication--ffiieen sgd via gradient quantization and encoding. Advances in Neural Information Processing Systems (NeurIPS), 2017.
D. Alistarh, T. Hoefler, M. Johansson, N. Konstantinov, S. Khirirat, and C. Renggli. The convergence of sparsified gradient methods. Advances in Neural Information Processing Systems (NeurIPS), 2018.
C. Baziotis, B. Haddow, and A. Birch. Language model prior for low-resource neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Bernstein, Y.-.. Wang. K. Azizzadenesheli, and A. Anandkumar. signsgd: Compressed optimi- sation for non-convex problems. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
T. Boutell. Png (portable network graphics) specification version 1.0. Technical report, 1997.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020.
W.-L. Chang, H.-P Wang, W.-H. Peng, and W.-C. Chiu. All about structure: Adapting structural information across domains for boosting semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR), 2019.
Coalson. Free lossless audio codec, 2008. URL httttss/:xxphh.rgg////// orgg
T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems (NeurIPS), 2022.
G. Deletang, A. Ruoss, P.-A. Duquenne, E. Catt, T. Genewein, C. Mattern, Grau-Moya, L. K. Wenliang, M. Aitchison, L. Orseau, al. Language modeling compression. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
P. Deutsch. Gzip file format specification version 4.3, RFC, 1996.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. Proceedings of the International Conference on Learning Representations (ICLR), 2023.
10

---

Y. Gandelsman, A. Shocher, and M. Irani. double-dip". unsupervised image decomposition via coupled deep-image-preee priors. In Proceedings of the IEEE Conference on Computer Vision and Paltern Recognition (CVPR), 2019.
X. Geng and H. Liu. Openllama: An open reproduction of Ilama, May 2023. httpttps: com/ connopenlmm. hww.emmmrreeeeeeppeeeeeeeee 1lama
N. Gruver, M. Finzi, S. Qiu, and A. G. Wilson. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems (NeurIPS), 2024.
D. Ha, A. Dai, and Q Le. Hypernetworks. arXiv preprint arXiv: 09000, 2016.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR). 2016.
Y. He, H.-P. Wang, M. Zenk, and M. Fritz. Cossgd: Communicationn-efficccal federated learning with simple cosine-based quantization. arXiv preprint arXiv:20122.66241 2020.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 1997.
C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. arXiv preprint ar arXiv:2401.18077 2024.
C.-Y. Hsieh, C.-L. Li, C.-K. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C.-Y. Lee, and T. Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint trXiv:2305.02301. 2023.
F. Jelinek. Statistical methods for speech recognition. MIT press, 1998.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Y. Le and X. Yang. Tiny imagenet visual recognition challenge. CS 23/N, 7(7):3, 2015.
Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online). Available: http://yann.L www.:.eecccccoo www.:eewww....cdeeeeeeeeece cow/exdb/mniist 2, 2010.
P. Lindstrom and M. Isenburg. Fast and efficient compression of floating-point data. IEEE transactions on visualization and computer graphics, 2006.
X. Ma, G. Fang, and X. Wang. LIm-pruner: Advances in Neural Information Processing Systems (NeurIPS), 2023.
On the structural pruning of large language models.
S. Mirchandani, F. Xia, P. Florence, B. Ichter, D. Driess, M. G. Arenas, K. Rao, D. Sadigh, and A. Zeng. Large language models as general pattern machines. In Conference on Robot Learning. 2023.
I. Pavlov. 7z format, 2019. URL http:://wwww.722/0
html
J. Rissanen and G. G. Langdon. Arithmetic coding. IBM Journal of research and development, 23(2): 149,162,11 1979.
K. Sayood. Introduction to data compression. Morgan Kaufmann, 2017.
R. Sennrich, Haddow, and Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07000 2015.
C. E. Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55, 2001.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. Proceedings of the International Conference on Learning Representations (ICLR), 2015.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, Bhosale, al. Llama Open foundation and fine-tuned chat models. arXiv preprint ar 20X1022307.09288, 2023.

---

D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. In Proceedings the on Computer Vision and Pattern Recognition (CVPR), 2018.
the IEEE Conference
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
H.PP. Wang, N. Yu, and M. Fritz. Hijack-gan, gan: Unintended-use of pretrained, black-box gans. In Proceedings of the IEEE Conference on Computer Vision and Patern Recognition (CVPR), 2021.
H.-P. Wang, S. Stich, Y. He, and M. Fritz. Progfed: effective, communication, and computation efficient federated learning by progressive training. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
H.PP. Wang, D. Chen, R. Kerkouche, and M. Fritz. Fedlap-dp: Federated learning by sharing differentially private loss approximations. Proceedings on Privacy Enhancing Technologies (PoPETs), 2024a.
K. Wang, Z. Xu, Y. Zhou, Z. Zang, T. Darrell, Z. Liu, and Y. You. Neural network diffusion. arXiv preprint 4rX1v:2402.13144 2024b.
Wangni, J. Wang. J. Liu, and T. Zhang. Gradient sparsification for communication-effiiennn distributed optimization. Advances in Neural Information Processing Systems (NeurIPS), 2018.
Y. Xiong, Wang, M. Cheng, F. Yu, and C.-J. Hsich. Feddm: Iterative distribution matching for commiiication-fficceen federated learning, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Zhang, G. Zeng. T. Wang, and Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:240..02388 2024.
12

---

Appendix
A
Run Length Encoding
RLE (bits)
RLE (H,))
RLE (ISO)
LM-GC (H,)
450.28+0.3
278.08+0.2
198.57=0.0
71.9010.0
Table 4: Run length encoding results of gradients collected from ConvNets trained on Tiny
We additionally compare our method run-length encoding (RLE). RLE compresses data by counting the consecutive symbols and replaces the original data with series of (counts, symbol) tuples. It serves as simple adaptive compression codec without knowing data characteristics. The experiment extends from Table 3, compressing gradients collected during training a ConvNet on TinylmageNer We consider three types of dictionaries: binary. hexadecimal without separators (H,, Table 1), and iso-8859-1 (extended ASCII to handle negative numbers). These methods use 1, 4, and 8 bits to represent symbols and always use 8 bits for counting. Note that this setting favorable to RLE since gradient lengths can easily exceed 256 (8 bits).
The results are presented in Table While different codebooks improve the efficacy of RLE, RLE failed to compress the data and even increase the data size. On the other hand, our method clearly outperforms RLE, indicating that simple adaptive priors ineffective for gradients.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: As described in the abstract, we explore the potential of using LLMs as prior for gradients and take compression as an examination task.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We have dedicated a section to discuss the potential limitations and solutions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: We briefly reviewed the background of lossless compression, but it's not main contribution.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provided all implementation details and plan to release the source code upon publication.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release the code upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wttsssubmiiiiooooioooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //ipps.cc/publi///guu
Codeeuumiiii wettsbbmmssiiioi ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided the required information in the main paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We repeated all experiments least three times report the mean and standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Yes we have specified it.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics luttp::/neurips: httpsccccc/c.hiccciiiiiccoooo
Answer:
[Yes]
Justification: We have closely went through the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We dedicated one section discuss it.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
We do not release any models.
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited all relevant works.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets at this point.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not include any related research mentioned in the guideline.
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not apply to IRB or any ethical reviews.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
20


Reviews:
[{"review_id": "gqi6flE5Hq", "reviewer": "Reviewer_Br5V", "summary": "This paper studies the potential of LLMs to act as gradient priors in zero-shot settings. The property is evaluated on lossless gradient compression. The proposed method is able to surpass existing compression methods and improve compression rate by 10% to 21% across various datasets and architectures.", "questions": "please see the above weaknesses.", "limitations": "provided.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper explores the potential of LLMs as gradient priors through the lens of lossless gradient compression. Given the massive model size and training datasets, LLMs are encoded with many capacities that can be used as tools in domains that are out of the text domain, such as optimizers https://arxiv.org/abs/2309.03409, and compressors https://arxiv.org/abs/2309.10668. This paper is one of these, which itself is fun and interesting. \n\n2. The effectiveness of the methodology design such as Serialization and Compression is backed up with promising results.", "weaknesses": "1. While this paper itself is fun, one concern remains what are the benefits of using LLM as a compressor if traditional approaches can sufficiently solve the problem? It is better to report the energy costs of LLMs and traditional baselines such as PNG, FLAC, LZMA, and GZIP to allow readers to have a full picture. \n\n2. What are the potentials of this paper's finding? Whether the compressed gradient can be utilized in practice? for instance, how to explore this to train neural networks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback and are encouraged that our work was found interesting. We will now address the comments and share further vision of our work.\n\n----\n***The benefit of using LLMs over traditional approaches.***\n\nWe highlight that traditional baselines struggle to compress gradients effectively in complex scenarios. As illustrated in Tables 2 and 3, our method exceeds the best baseline by 20% on ViT and 21% on TinyImageNet. This highlights the difficulty of modeling complex gradients and showcases the effectiveness of our LLM prior. Moreover, the baselines often fail to compress data adequately in some scenarios. For instance, LZMA, the best baseline, achieves only a 91% compression rate. These advantages demonstrate the benefits of our method, making it particularly favorable when communication cost or storage is the main system bottleneck. \n\n----\n***Energy consumption***\n\nWe provide the energy consumption estimation in Figure A2 of the attached PDF. The consumption is estimated by the following equation:\n\n$\nE = (\\text{CPU power consumption} \\times \\text{CPU utilization} + \\text{GPU power consumption} \\times \\text{GPU utilization}) \\times \\text{run time}\n$\n\nIndeed, LM-GC currently consumes more energy. However, as discussed in the general response, we remain positive about future efficiency improvements. Moreover, in complex scenarios, the baselines often struggle to compress the data effectively (70.98% vs. 87.98% in Tables 2 and 71.90% vs. 91.6% in Table 3), which justifies our method's investment in computational resources.\n\n----\n***Can the compressed gradient be utilized in practice?***\n\nYes. An immediate application is federated learning, where clients accumulate gradients locally and communicate periodically with the central server. One of the main bottlenecks is the communication cost as the number of clients and model size grow. Assuming both sender and receiver have access to the same LLM, LM-GC (optionally with lossy compression, as shown in Figure 4) provides superior compression rates on gradients over the existing coding schemes (Table 1-3) and thus enables better scalability. \n\n----\n***Broader impact of our findings***\n\nIn addition to LM-GC, our work highlights the potential of using large language models (LLMs) as gradient priors, which we find exciting and believe will inspire further applications. We propose several possible directions for exploration, which could be investigated in fully zero-shot, parameter-efficient fine-tuning (PEFT), or prompting settings, given the strong starting point provided by zero-shot LLMs.\n\n- **Inspiration from existing works**: With a robust prior $p(t_k|\\texttt{BOS}, t_{<k})$ over gradient tokens, it is possible to draw inspiration from existing work of different modalities. For example, one could utilize the prior model to de-noise gradients by checking the likelihood, akin to image denoising with generative priors, or explore gradient restoration from quantized low bits to higher bits, similar to image super-resolution. In the latter scenario, the lower bits provide partial information or context for the LLMs.\n\n- **Advanced lossy compression**: Most of the existing lossy compression in federated learning assumes no prior knowledge exists between servers and clients or leverages heuristically designed protocols. In contrast, one can assume both sides have access to a fixed LLM. When certain tokens consistently exhibit high probability, they can be sampled directly from the LLMs, allowing more bits to be allocated to other tokens.\n\n- **Security**: By modeling benign gradients, potentially through PEFT or prompting, one could develop a gradient-based method to guard against backdoor attacks, akin to image anomaly detection using generative adversarial networks (GANs).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zm1auKseOX", "reviewer": "Reviewer_KkqK", "summary": "The paper proposes LLMs as compressors for gradients of a neural network. The target usecase is a distributed learning setting, where the gradient updates need to be compressed before being shared.\n\nThe gradients are converted to hexadecimal representation and the LLM's outputs are used in an arithmetic coder to generate the compressed representation.\n\nPost rebuttal edit: Improved rating 3->5", "questions": "Figure 4: What does 'plain' compression entail here? Is it using a codebook with arithmetic coding?", "limitations": "The solution proposed is extremely resource-intensive for the given task, which may have societal impacts.", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "The idea is certainly novel. I do not know of prior works using LLMs to compress numerical data.\n\nIt works better than the baseline methods as demonstrated in the experiments across multiple model architectures and image datasets.\n\nThe LLM's don't require any dataset-specific training therefore they can serve as a general prior for floating-point datastreams.", "weaknesses": "The method lacks comparison to proper baseline methods. \n> Line 195: \"We compare our method to state-of-the-art lossless compression techniques that originally targeted different data types.\"\n\nThere needs to be justification why there isn't a comparison to a baseline method that is targeted at a stream of floating point values (or even specialized to gradients). I don't see significant advantages for using LLMs (or methods designed for image/audio compression) over a method specialized the task.\n* There should to be a comparison to a simple adaptive entropy coder such as run-length coding or LZW.\n* There should to be a comparison to a static codebook + arithmetic coding.\nHere are a few other works that specialize in compressing floating point values:\n* https://computing.llnl.gov/projects/fpzip Fast and Efficient Compression of Floating-Point Data\n* https://ieeexplore.ieee.org/document/1607248 Fast lossless compression of scientific floating-point data\n\nBesides, it is also clearly impractical to use LLMs for this task due to their computational costs. The LLMs use excessively more compute than the bandwidth savings could justify.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful feedback. We hope our response addresses the concerns and conveys the broader vision of our work.\n\nWe kindly emphasize that our primary contribution is demonstrating **the potential of pre-trained zero-shot large language models (LLMs) as effective gradient priors through the lens of lossless compression**, a point also supported by R-UCGG and R-Br5V. As motivated in L21-23, the lack of proper prior models has been a critical barrier for applications such as denoising or upsampling in the domain of gradients. Our work is the first to demonstrate this potential and explore its application in lossless gradient compression. We believe our findings will inspire further research beyond just arithmetic coding. The potential impact also outweighs the temporal computation overhead as we have discussed in the general response.\n\nWe now address the concerns regarding the lossless gradient compression task below.\n\n----\n\n***Justification of using LLMs.***\n\nWe first emphasize that traditional baselines struggle to compress gradients effectively in complex scenarios. As illustrated in Tables 2 and 3, our method exceeds the best baseline by 20% on ViT and 21% on TinyImageNet. This highlights the difficulty of modeling complex gradients and justifies the usage of our LLM prior. The following experiments recommended by the reviewer further stress the point.\n\n----\n***Missing baselines.***\n\nWe have compared our method to LZMA, one of the most robust **general-purpose** compression methods, commonly used in the 7z compression format. Additionally, it is important to note that **none of the suggested baselines are designed for gradients or high-dimensional data**. However, to further highlight our contribution, we have conducted additional experiments as follows.\n\n----\n**Comparison to simple adaptive coding**\n\nWe additionally compare our method to run-length encoding (RLE). This experiment extends from Table 3, compressing gradients collected during training a ConvNet on TinyImageNet. For RLE, we consider three types of dictionaries: binary, hexadecimal ($H_n$, Table 1), and iso-8859-1 (extended ASCII to handle negative numbers). These methods use 1, 4, and 8 bits to represent symbols and always use 8 bits for counting. Note that this setting is favorable to RLE since gradient lengths can easily exceed 256 (8 bits). \n\nThe results are presented in the following table. RLE failed to compress the data even with different codebooks, and our method clearly outperforms RLE, indicating that simple adaptive priors are ineffective for gradients.\n\n| RLE (bits) | RLE ($H_n$) |  RLE (ISO) | LM-GC ($H_s$) |\n|:----------:|:-----------:|:----------:|:-------------:|\n| 450.28±0.3 |  278.08±0.2 | 198.57±0.0 | **71.90±0.0** |\n----\n***Comparison to methods dedicated to floating-point compression***\n\nWhile we acknowledge that the suggested baselines are intended for scientific floating points (1D to 4D, which are relatively low-dimensional compared to gradients [1]), we have extended the comparison in Table 2 and 3 to include floating-point-specific compression. Specifically, we compare our method to FPZIP [1], as we could not find any available implementation of [2]. The results, shown in Table A1 and A2 in the attached PDF, indicate that FPZIP performs comparably to LZMA. On the other hand, our LM-GC outperforms FPZIP by up to 17.2% across three datasets and around 20% across four architectures. This highlights the challenge of using heuristically designed priors and demonstrates the effectiveness of the LLM priors adopted in our method.\n\n----\n***Practicability of LM-GC***\n\nAs previously discussed, the effectiveness of LLMs in handling complex gradients justifies their usage. We acknowledge that LM-GC requires more resources than traditional codecs. However, as detailed in the general response, significant acceleration can be achieved by implementing C++ and multi-threading. Additionally, ongoing research into more efficient LLMs suggests a promising future for further improvements. Notably, since our method does not require training, hardware can frequently achieve substantial acceleration for inference.\n\n----\n***What does 'plain' compression entail here? Is it using a codebook with arithmetic coding?***\n\nPlain means without any lossless compression. We will revise the text in the next version to avoid confusion.\n\n----\nWe hope our response has taken care of your concerns and encouraged you to reconsider the score. We would be happy to answer any other questions or concerns during the discussion phase.\n\n[1] Fast and Efficient Compression of Floating-Point Data https://computing.llnl.gov/projects/fpzip  \n[2] Fast lossless compression of scientific floating-point data https://ieeexplore.ieee.org/document/1607248", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MAPTk99ciH", "reviewer": "Reviewer_Nmkh", "summary": "This is an interesting paper that tries to solve an important problem using large language models. Authors propose to use LLMs to compress the gradients with no number representation loss. They explain their method clearly by marrying LLMs with arithmetic coding.", "questions": "Heavy tail priors are observed in training of deep models. Bring more context about number formats when the inherent distribution does not have expectation or variance, for instance Student-t distribution with 1 or 2 degrees of freedom.", "limitations": "When LLM is not a good tool for gradient compression? Under what circumstances they fail to present the prior.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper opens new views to LLMs as a tool in compression and coding. Gradient compression as stated in the paper is largely ignored. They prove LLMs can be used effectively as a prior for gradients in a zero-shot setting.", "weaknesses": "I would suggest that authors think about a proper theory to be added in the revised paper, or in the appendix. This combination is new and I suggest that some theoretical properties be developed in addition to experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and constructive feedback. We especially appreciate the recognition of our novelty and our efforts to address this timely question. We are pleased to share additional insights below.\n\n----\n***Theoretical properties***\n\nThanks for the suggestion. We agree that theoretical analysis could be insightful. The fundamental theories behind entropy coding have been well-studied in the existing literature. On the other hand, understanding the behavior of LLMs is a non-trivial ongoing research direction. We will leave it to future work at this point.\n\n----\n***Clarification on the question***\n\nWe are not entirely clear on the question. We would be more than happy to address it during the discussion phase if the reviewer could kindly provide further clarification.\n\n----\n***When is LLM not a good tool for gradient compression?***\n\nThe main limitation of our approach is its high computational resource demand, making it less suitable for weaker devices such as IoT devices or real-time systems. However, this constraint could be significantly improved with the techniques discussed in our general response, enhancing scalability. Furthermore, our method does not require any training, making it especially well-suited for hardware acceleration in production deployments.\n\n----\n***Under what circumstances do they fail to present the prior?***\n\nOur approach performs well on common benchmarks, including MNIST, CIFAR-10, and TinyImageNet, and on architectures like ConvNet, VGG, ResNet, and ViT for image classification. Further analysis may be needed for other tasks, such as generative models, particularly if different implicit biases are present. To the best of our knowledge, this remains an open question, and we will leave this analysis for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oxtypnLlND", "reviewer": "Reviewer_UCGG", "summary": "The paper explores the novel use of large language models (LLMs) as gradient priors in a zero-shot setting. The primary focus is on lossless gradient compression, which is essential for distributed learning applications. The paper introduces LM-GC, a method that integrates pre-trained LLMs with arithmetic coding to transform gradients into a text-like format that LLMs can process efficiently. This approach significantly improves compression rates, surpassing existing state-of-the-art lossless compression methods by 10% to 21%. The method also shows compatibility with lossy compression techniques like quantization and sparsification.", "questions": "1. How well does LM-GC generalize to tasks and models not covered in the experiments? Are there specific scenarios where this approach might not be as effective?\n2. How scalable is the proposed method for extremely large models or distributed training setups? Are there any known limitations in this regard?\n3. How sensitive is the performance of LM-GC to different serialization strategies? Could there be more optimal serialization methods not explored in this paper?\n4. What specific techniques or optimizations could be employed to reduce the computational overhead and improve the throughput of LM-GC?", "limitations": "see the questions.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper leverages pre-trained LLMs in an unconventional way, using them as priors for gradient compression, demonstrating creativity and novelty.\n2. The proposed method, LM-GC, consistently outperforms traditional compression methods (PNG, FLAC, GZIP, LZMA) across various datasets and architectures.\n3. Extensive experiments validate the effectiveness of LM-GC, including ablation studies and compatibility analyses with other compression techniques.\n4. The paper discusses potential applications of LLMs in gradient denoising, differential privacy training, and adversarial gradient detection, highlighting broader impacts and future research directions.", "weaknesses": "1. The method incurs significant computational overhead, with compression times being relatively high.\n2. The approach involves multiple hyper-parameters, such as context window size and byte grouping, which require careful tuning for optimal performance.\n3. The integration of LLMs with arithmetic coding and the serialization process might be complex to implement and replicate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and detailed feedback. We are also encouraged by the recognition of our novelty and creativity. We now address the comments and share our vision for further efficiency improvement below.\n\n----\n***Computation overhead and potential mitigation***\n\nWe have discussed several potential strategies in the general response. In summary, there are three possible directions: advanced implementation, efficient LLMs, and hardware acceleration.\n\n- **Advanced Implementation**: This improves our current research-oriented single-thread Python implementation.\n- **Efficient LLMs**: This is an ongoing research direction aimed at developing more lightweight LLMs.\n- **Hardware Acceleration**: This leverages the zero-shot property and is intended for production use.\n\n----\n***Hyper-parameter selection***\n\nInterestingly, our findings suggest that most hyperparameters align with human intuitions. For example, 4 bytes per group resemble floating point structures and yield the best performance, while hexadecimal encoding outperforms iso-8859-1 due to the more common characters used. Other parameters generally adhere to the scaling law, where larger models or context windows result in better performance, creating a trade-off between performance and available resources. These factors significantly reduce the complexity of hyperparameter search.\n\n----\n***Implementation efforts***\n\nOur implementation can be easily reproduced using common open-sourced packages like PyTorch, Huggingface, and TorchAC, while serialization can be done using pure Python string operations. We will release the source code upon publication.\n\n----\n***Generalizability***\n\nOur approach performs well on common benchmarks, including MNIST, CIFAR-10, and TinyImageNet, and on architectures like ConvNet, VGG, ResNet, and ViT for image classification. Further analysis may be needed for other tasks, such as generative models, particularly if different implicit biases are present. To the best of our knowledge, this remains an open question, and we will leave this analysis for future work.\n\n----\n***How scalable is the proposed method for extremely large models or distributed training setups?***\n\nWe currently target scenarios like federated/distributed learning, where gradients are compressed before sending to the server. We tested on models up to 6M parameters, a common model size for edge-device scenarios. Though the typical parameter size of LoRA fine-tuning is also around 6M, further analysis would be required on those extremely large models.\n\n----\n***How sensitive is the performance of LM-GC to different serialization strategies?***\n\nTable 1 shows that LLMs are sensitive to serialization strategies. For instance, hexadecimal numbers are up to 70% better than the extended ASCII set (iso-8859-1), and separators play a critical role in the reasoning ability of LLMs (40% difference for LLAMA 2).\n\n----\n***Could there be more optimal serialization methods not explored in this paper?***\n\nYes, there may exist better strategies. This could be similar to the existing multi-modal LLM problems, where the key challenge is to project a new modality to the token space. While we propose an easy-to-go serialization method, understanding how and why LLMs can understand gradients deserves further research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response
\
We thank all reviewers for their time and constructive feedback. It is encouraging to hear that our work has been found both fun and interesting (R-Br5V) as well as creative (R-UCGG). We particularly appreciate the recognition from all reviewers of the novelty and effectiveness of our method across various datasets and architectures (R-Br5V, R-KkqK, R-Nmkh, R-UCGG). We will begin by recapping the importance of our work and then address the comments in each thread, respectively.

----

### ***Importance of our work***

Our study is the first to investigate the applications of *general-purpose* LLMs in the field of *gradients*. In addition to traditional NLP tasks, we show for the first time that LLMs can act as a robust gradient prior model and comprehend gradients *without requiring fine-tuning or demonstrations*. This discovery will expand the potential uses of general-purpose LLMs and inspire further applications involving gradients, as discussed in L21-23. For instance, our LM-GC, an immediate application of LLM priors in lossless gradient compression, presents superior performance, especially in complex scenarios that traditional methods struggle to model. Despite the room for improvement in our implementation, our research opens up an entirely new direction that has never been explored. We are confident that our contribution outweighs these temporary computational burdens.

With that being said, we have noticed several potential strategies for further efficiency improvement. Many of them are still ongoing research problems. We anticipate that both the field and our work will mutually benefit in the near future.

- **Advanced Implementation**: Our current implementation is designed for research purposes and is a single-thread Python program that processes each window of gradients sequentially. Given that the average GPU utilization is around 10% and each window is independent, the implementation can be significantly improved by utilizing multi-threading and C++. This could potentially yield a 10 times improvement in speed.

- **Efficient LLMs**: Accelerating LLMs is a popular research problem. Techniques such as quantization [1], KV caches [2], flash attention [3], and pruning [4] are already being explored. One promising but underexplored direction is distillation [5]. While pre-trained language models are powerful, we do not need all of their functionalities for our task. Extracting the desired functions (e.g., gradient priors) from LLMs might be a significant step toward efficient LM-GC.

- **Hardware Acceleration**: LM-GC does not require any training. Inference of networks can often be significantly accelerated by AI-specific hardware such as NPUs, TPUs, or other application-specific integration circuits (ASICs).
  
[1] Lin, Ji, et al. "AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration." Proceedings of Machine Learning and Systems 6 (2024): 87-100.\
[2] Hooper, Coleman, et al. "Kvquant: Towards 10 million context length llm inference with kv cache quantization." arXiv preprint arXiv:2401.18079 (2024).\
[3] Shah, Jay, et al. "Flashattention-3: Fast and accurate attention with asynchrony and low-precision." arXiv preprint arXiv:2407.08608 (2024).\
[4] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. "Llm-pruner: On the structural pruning of large language models." Advances in neural information processing systems 36 (2023): 21702-21720.\
[5] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).

Meta-review: This paper uses LLMs as priors for arithmetic coding, showing that LLMs can be used to compress gradients of neural networks. Reviewers unanimously applauded the originality and ingenuity of the method, and I agree. By showing this unexpected use of LLMs, I believe this work is likely to lead to follow-up research.

Reviewers also raised the concern of computational cost as a limitation, which was discussed during the rebuttal. I ask the authors to please include some of the discussion and additional results from the rebuttal into the final version of the manuscript so as to make it clear throughout the manuscript that this is a limitation of the current incarnation of the method.
Decision: Accept (poster)