Title: value: LLMZip: Lossless Text Compression using Large Language Models
Authors: value: Chandra Shekhara Kaushik Valmeekam
Krishna Narayanan
Dileep Kalathil
Jean-Francois Chamberland
Srinivas Shakkottai
Venue: ICLR 2024
Keywords: value: Large Language Models
Transformers
Compression
Arithmetic Coding
Zip
Lossless Text Compression

Abstract: value: We design a lossless compression algorithm for compressing English text by using the large language model LLaMA-7B as a predictor for the next token given a window of past tokens. Specifically, the proposed LLMZip algorithm uses the conditional probabilities at the output of the large language model in conjunction with Arithmetic Coding. We show that our scheme outperforms state-of-the-art text compression schemes such as BSC, ZPAQ, and paq8h. We show that it is possible to marginally improve the compression performance further by first extracting a summary from the document and compressing the text by conditioning on the summary. Finally, we investigate the compression performance of LLMZip when the summary (side information) is available both at the encoder and decoder. We show that the LLM is able to exploit the available side information to significantly improve the compression performance. As an important byproduct, we provide new estimates of an asymptotic upper bound on the entropy of English which is significantly smaller than currently available estimates in \cite{cover1978convergent}, \cite{lutati2023focus}.


Review discussion:
- The proposed idea is intuitive and easy to follow.
- Exploring the potential applications of LLMs in compression is a promising research direction.
- The paper is well presented in general.
- The novelty of the LLMZip method seems to be limited. As mentioned in the introduction, it seems the connection between compression and prediction has been developed decades ago. Using language models for compression has also been explored before, with LSTM and/or RNN being used.
Therefore, it appears that the sole contribution of this paper is the substitution of previously explored smaller language models with large language models (LLMs)
- Only Llama2 has been used as the language model in the experiment. It is unclear how the performance of LLMZip would vary when employing LLMs of different types and sizes.
- LLMs usually require GPUs for execution, whereas traditional compression algorithms can run on CPUs, which are more widely accessible and easier to democratize.
- The LLMZip approach presented in this paper appears to be more suitable for submission to an information theory conference or journal, such as ISIT.
- How does the performance of LLMZip vary when using aligned LLMs, e.g., Vicuna or fine-tuned Llama2?
- How does the performance of LLMZip vary when using LLMs with various scales, e.g., 13B or 65/70B?
- What does the end-to-end running time of LLMZip look like when compared to Zlib?
No ethics review needed.
Dear Reviewer,
We appreciate your time and effort in reviewing our work. We would like to address each point you have raised in the review.
> 1. The novelty of the LLMZip method seems to be limited. As mentioned in the introduction, it seems the connection between compression and prediction has been developed decades ago. Using language models for compression has also been explored before, with LSTM and/or RNN being used.
Therefore, it appears that the sole contribution of this paper is the substitution of previously explored smaller language models with large language models (LLMs)
Prior to our research, the compression performance of pre-trained LLMs with Arithmetic Coding had not been reported.
While there have been works that utilized LSTMs (Deepzip) and Transformers (NNCPv2) for text compression, LLaMA1 and LLaMA2 significantly outperform earlier results using LSTMs and Transformers. We believe it is necessary to bring this to the attention of the ML community since this may have important consequences for the practice of compression in some scenarios. The idea of using the summary as side information is new.
> 2. Only Llama2 has been used as the language model in the experiment. It is unclear how the performance of LLMZip would vary when employing LLMs of different types and sizes.
We also report the performance of using LLaMa1, which is the older model, in Table 1, and we do see an improvement in performance when LLaMa2 is used. Using LLMs out of the box for LLMzip compression can be very computationally intensive. As the experiments involve processing 1M tokens for each context size , more gpus are needed for longer durations when using larger versions.
Hence we limited our focus to just the 7B version of LLaMa.
> 3. LLMs usually require GPUs for execution, whereas traditional compression algorithms can run on CPUs, which are more widely accessible and easier to democratize.
This is a valid point.
Yet, the article points to a significant gap and a new approach warrants dissemination. The approach may not be ready for widespread dissemination;
however, it is a significant step in a new direction.
> 4. The LLMZip approach presented in this paper appears to be more suitable for submission to an information theory conference or journal, such as ISIT.
The aim of this work is to disseminate this information to the ML / LLM community that LLMs can do SOTA compression. We believe it is important to disseminate this to the ML community rather than the IT community, because additional work is required from the ML community to make these practical, and this may have a significant impact on the future of compression in many scenarios. As quantization and other techniques to accelerate LLM inference become better, we believe this complexity can be a lot more manageable. There is scope to engineer better solutions.
> 5. How does the performance of LLMZip vary when using aligned LLMs, e.g., Vicuna or fine-tuned Llama2?
6. How does the performance of LLMZip vary when using LLMs with various scales, e.g., 13B or 65/70B?
We haven’t tried LLMZip with LLMs other than LLaMa1 and LLaMa2, as we believe they suffice to show that pre-trained LLMs do indeed offer substantial compression improvements.
> 7. What does the end-to-end running time of LLMZip look like when compared to Zlib?
The run times for 100K tokens is 10 hrs for 512 context length and  35 mins for a context length of 32, while zlib does it in less than a second.
While these are clearly not impressive run times, these are just out-of-the-box applications, with a lot of scope to be improved. It is also worth noting that 32 context length LLM still significantly outperforms Zlib compression. We do not have runtime comparisons for the other SOTA algorithms like paq8h or bsc.
We hope this addresses your concerns and we look forward to your further feedback.
- The authors improve the text compression performance by combining LLMs and arithmetic coding.
- The paper provides new estimates of an asymptotic upper bound on the entropy of English
- The authors should provide more background introduction about text compression and make the paper more self-contained.
- What are the baseline results mentioned in Table 1? I would suggest the authors include more introductions about the baseline methods and their performances.
- Table 3 shows the performance varies based on the summary quality. How can we determine if the side information is helpful or not before compression?
- Why does side information available in the encoder and decoder perform better than the encoder-only?
- From the perspective of evaluation metric, does a lower bpc indicate a stronger LLM model potential in common downstream NLP tasks(summarization, translation, classification, etc.)?
No ethics review needed.
N/A
Dear Reviewer,
Thank you for your valuable feedback and insightful queries regarding our work. We would like to address each point you have raised in the review.
> 1. What are the baseline results mentioned in Table 1? I would suggest the authors include more introductions about the baseline methods and their performances.
In Table 1, we presented results for different encoding schemes which have been explained in sections 2.1.1, 2.1.2, and 2.1.3, and the entropy bound was presented in section 2.2. We compare our results to ZPAQ and PAQ8h in the discussion of the results (section 4.1). These are state-of-the-art algorithms as reported in various papers on compression. Given the page limitations, we weren’t able to include more background on these algorithms, but we can include more references to these.
> 2. Table 3 shows the performance varies based on the summary quality. How can we determine if the side information is helpful or not before compression?
Thank you for your insightful question about the variability of performance based on summary quality.
While it is challenging to precisely quantify the effectiveness of a summary prior to compression, our observations indicate a trend. Particularly, a detailed and less compressible summary (like Summary 3 in our study) tends to offer more valuable information for compression, as seen in its high compression ratio (1918 characters to 1375 bits, or 0.72 bits per character).
> 3. Why does side information available in the encoder and decoder perform better than the encoder-only?
When side information is available only at the encoder, the compression ratio is computed as (compressed bits of summary + compressed bits of text)/(number of characters in text), denoted by $\tilde{\rho}$​.
When the summary is available at both ends, the compressed summary bits need not be transmitted/stored, hence we need not account for it in the compression ratio.
> 4. From the perspective of evaluation metric, does a lower bpc indicate a stronger LLM model potential in common downstream NLP tasks (summarization, translation, classification, etc.)?
We are not entirely sure whether a lower bpc indicates a stronger LLM model potential in downstream NLP tasks. We do not make any such claims in the paper. We also don’t think this possibility can be ruled out either given the consistent performance improvement of LLaMA2 over LLaMA1 (as shown in Table 1, last two columns) across all window sizes.
We hope this addresses your concerns and we look forward to your further feedback.
Utilizing a state-of-the-art Large Language Model for text compression to attain a higher compression ratio is interesting.
1. The concept of compressing text with pretrained language models is not groundbreaking. The paper's attempt to innovate using the advanced LLaMa2 as an LLM seems to lack strong novelty.
2. Sections 2.1.1 and 2.1.2, which discuss text compression methods other than arithmetic coding, appear superfluous.
3. The detailed explanation of Arithmetic coding might be redundant; perhaps it would be better placed in an appendix.
4. In Section 2.2 regarding Entropy bounds, the equation H(S) = H(X) / E[B] does not appear to be a significant finding.
5. The use of text-summary for compression seems misjudged. Theoretically, adding bits to describe a summary would only be beneficial if the probability estimation isn't flawless.
6. The claims about the entropy bounds of the English language are debatable.
For instance, Table 1 lists 0.6936, while Table 2 cites 0.7741 from a different dataset. The input data chosen for testing does not seem to be a true representation of English text, with entropy rates that fluctuate depending on the input.
7. Minor Remarks:
- The use of "It's" may not be suitable for a formal paper.
- Terminologies such as $N_{tg}$, $N_{cg}$, and others need clear and precise definitions.
Please see Weakness section.
No ethics review needed.
N/A
Dear Reviewer,
We are grateful for the time and effort you have invested in reviewing our work.
However, we believe the reviewer has overlooked the main contributions of the paper and the significance of the results.
> 1. The concept of compressing text with pre-trained language models is not groundbreaking. The paper's attempt to innovate using the advanced LLaMa2 as an LLM seems to lack strong novelty.
“Groundbreaking” is a relative term and we cannot argue with the reviewer’s view of what is groundbreaking. There are two important results in this paper that need to be disseminated to the research community.
First, we emphasize that the compression performance of pre-trained Large Language Models  (LLMs) with Arithmetic Coding has not been previously explored.
While there are studies employing LSTMs (e.g., Deepzip) and Transformers (e.g., NNCPv2) for text compression, their performance does not match the substantial results achieved using LLaMA1 or LLaMA2 for compression. The gap is not small and we believe this will have a significant influence on the future of compression, at least in some scenarios.
Secondly, the idea of using a summary as side information to compress the text is novel (please see our response to Question 4 for more details).
> 2. Sections 2.1.1 and 2.1.2, which discuss text compression methods other than arithmetic coding, appear superfluous. The detailed explanation of Arithmetic coding might be redundant; perhaps it would be better placed in an appendix.
We included a detailed explanation of Arithmetic coding in our paper to ensure it is self-contained and accessible to those unfamiliar with the concept. This decision aligns with the suggestion from reviewer 1ivf to provide more background on text compression. We believe our approach achieves a balance, but we are open to relocating this section to an appendix if you find it more appropriate.
> 3. In Section 2.2 regarding Entropy bounds, the equation H(S) = H(X) / E[B] does not appear to be a significant finding.
Section 2.2 is necessary to formally examine entropy bounds in the presence of tokenization in terms of bits per character.
In particular, it is important to realize that the token sequence has to satisfy the conditions necessary for the Shannon McMillan Breiman theorem to hold for us to meaningfully use these entropy bounds as a lower bound for compression in terms of bits/character.
> 4. The use of text summary for compression seems misjudged. Theoretically, adding bits to describe a summary would only be beneficial if the probability estimation isn't flawless.
We are not sure why the reviewer thinks that this section is misjudged.
In reality, every language model only provides a flawed estimate of the conditional probability and hence, this section is very much relevant to reality. This is an idea that, to the best of our knowledge, hasn’t been presented in any other work. We have provided empirical evidence that this is beneficial when used with a real LLM (Llama2).
> 5. The claims about the entropy bounds of the English language are debatable.
For instance, Table 1 lists 0.6936, while Table 2 cites 0.7741 from a different dataset. The input data chosen for testing does not seem to be a true representation of English text, with entropy rates that fluctuate depending on the input.
There is nothing called a “true representation of English text” that can be used for testing. The statistical nature of the English language is complicated and estimates of the bounds will vary with the realization. Our estimates of the bound (0.6936 / 0.7741) show that the upper bound estimate may be significantly lower than the estimates from Thomas & Cover (1.2) and the reviewer should focus on how large this difference is and the implications of this.
Of course, we can average the entropy over more and more inputs to get sharper estimates. That would be a pedantic exercise.
We hope that this response addresses your concerns and clarifies the contributions and intentions of our work. We remain open to further discussion and are willing to make modifications to our paper to better reflect the valuable feedback provided.

Meta-review:
role: AC
data: metareview: value: This paper presents LLMZip algorithm that uses arithmetic coding and LLM to do text compression. This is not surprising as the conditional probabilities produced by the LLM can be very informative. The authors also presents a new estimate of asymptotic upper bound for English.

Strength:
1. An important application of LLM for text compression.
2. A new estimate of asymptotic upper bound for English.

Weakness:
1. Novelty is quite limited as it is well expected that LLM can provide very accurate predictive probabilities.
2. It is computationally very expensive to do for the purpose of text compression using large language models. Unless it is significantly better than other approaches, the impact is limited as well.
justification_for_why_not_higher_score: value: See weakness.
justification_for_why_not_lower_score: value: N/A
role: PC

Decision: Reject