Compression Represents Intelligence Linearly
Authors: Yuzhen Huang
Jinghan Zhang
Zifei Shan
Junxian He
Venue: COLM (2024)
Keywords: compression, language models, linear correlation
TL;DR: Viewing language models as compressors, we show that the "intelligence" of language models as reflected by benchmark scores is linearly correlated with their compression capabilities

Abstract: There is a belief that learning to compress well will lead to intelligence. Recently, language modeling has been shown to be equivalent to compression, which offers a compelling rationale for the success of large language models (LLMs): development of more advanced language models is essentially enhancing compression which facilitates intelligence. Despite such appealing discussions, little empirical evidence is present for the interplay between compression and intelligence. In this work, we examine the relationship between compression and intelligence in the context of LLMs, treating LLMs as data compressors. Given the abstract concept of "intelligence", we adopt the average downstream benchmark scores as a surrogate, specifically targeting intelligence related to knowledge and commonsense, coding, and mathematical reasoning. Across 12 benchmarks, our study brings together 31 public LLMs that vary in size and originate from diverse organizations. Remarkably, we find that LLMs' intelligence --  reflected by benchmark scores -- almost **linearly** correlates with their ability to compress external text corpora. These results provide concrete evidence supporting the belief that superior compression indicates greater intelligence. Furthermore, our findings suggest that compression efficiency, as an unsupervised metric derived from raw text corpora, serves as a reliable evaluation measure that is linearly associated with the model capabilities. This work advocates for the adoption of compression performance as a stable, flexible, and reliable metric for evaluating LLMs. We open-source our compression datasets as well as our data collection pipelines to facilitate future researchers to assess compression properly.

Published as a conference paper at COLM 2024
Compression Represents Intelligence Linearly
Yuzhen Huang") Jinghan Zhang") Zifei Shan? Junxian He
'The Hong Kong University of Science and Technology [yhuanghj, jzhangjv, junxianh] @ese. ust. hk
Tencent
Abstract
There is a belief that learning to compress well will lead to intelligence (Hut- ter, 2006). Recently, language modeling has been shown to be equivalent compression, which offers compelling rationale for the success of large language models (LLMs): the development of more advanced language models is essentially enhancing compression which facilitates intelligence. Despite such appealing discussions, little empirical evidence is present for the interplay between compression and intelligence. In this work, examine their relationship in the context of LLMs, treating LLMs as data compressors. Given the abstract concept of "intelligence" we adopt the average downstream benchmark scores as a surrogate, specifically targeting intelligence related to knowledge and commonsense, coding, and math- ematical reasoning. Across 12 benchmarks, our study brings together 31 public LLMs that originate from diverse organizations. Remarkably, we find that LLMs' intelligence reflected by average benchmark scores almost linearly correlates with their ability to compress external text corpora. These results provide concrete evidence supporting the belief that superior com- pression indicates greater intelligence. Furthermore, our findings suggest that compression efficiency, as an unsupervised metric derived from raw text corpora, serves as a reliable evaluation measure that is linearly associ- ated with the model capabilities. We open-source our compression datasets as well as our data collection pipelines to facilitate future researchers to assess compression properly.
Overview
Knowledge and Commonsense
0.75
0.7
e: 0.031
p: -0.931
0.50
0.6
Qwenn--26
0.019
0.25
0.00
Reiwra 18x7b
Deepsniok-lmm-676
0.55
0.60
0.65
Average Score
0.5
Jamba-526
34b
Qwen-14b
0.75
0.937
Mistral-7b
0.4
Llama-1-65b
Owen-7e 7b
Llama-1-300
Falcah 406
YI-6b
0.25
0.3
Uamma 2-13b
Llama-1-13b
Deepseek-lim-7b
Uama-2-7b
Llama-1
7a
0.35
0,40
0.00
Methematical Reasoning
0.2
0.75
00-93553
0.031
0.50
0.1
0.25
0.0
0.00
Bits per character
Figure 1: Correlation between the average benchmark scores and the models' compression efficiency evaluated with bits per character (BPC). BPC is computed on external raw corpora as listed in Table 1. Each point represents an LLM. Left shows the average benchmark score v.s. average BPC across, while Right focuses on three specific areas. p and denote the Pearson correlation coefficient and the root mean square error of the linear fit respectively.
"Equal Contribution. 'https:/ http:://.tttttccoo///ss////////s

---

Published as a conference paper at COLM 2024
1 Introduction
The belief that compression is closely connected to intelligence has been held for a long time, with some researchers suggesting that they are fundamentally equivalent (Hernández-Orallo & Minaya-Collado, 1998; Mahoney, 1999; Legg et al., 2005; Hutter, 2006; Legg & Hutter, 2007). This notion becomes even more compelling in light of the recent advancements in large language models (LLMs) and their implications for AI, leading to an exploration of language modeling from a compression standpoint. According to compression theory, any predictive model can be transformed into a lossless compressor and vice versa. Thus, language modeling can be considered a form of compression, with LLMs showing strong capabilities in data compression empirically (Deletang et al., 2024).
From this viewpoint, the argument that compression leads to intelligence becomes more relevant to the current paradigm of AI based on LLMs. Yet, despite these theoretical discussions, empirical evidence on the relationship between compression and intelligence remains limited. In this work, we seek to conduct such empirical study and answer: Ifa language model can encode a text corpus with feaver bits in a lossless manner, does that indicate greater intelligence? While the precise definition of "intelligence" often ventures into philosophical territory and can vary significantly, our work approaches "intelligence" from a practical standpoint, focusing on the model's ability to complete various downstream tasks. This treatment aligns with the definition from Hutter (2006), who notes that "intelligence is an agent's s ability to achieve goals in a wide range of environments".
We measure intelligence along three key abilities: knowledge and commonsense, coding, and mathematical reasoning. Specifically, we begin by collecting external raw corpora in the respective domain (e.g. GitHub code for the coding ability) and evaluate the efficiency of various LLMs in compressing the corpus. Subsequently, we assess these models on a range of downstream tasks, using the average benchmark scores to gauge their domain-specific intelligence. The goal is to examine the correlation between the models' downstream ability and their compression efficiency. On experiments across 31 public LLMs and 12 diverse benchmarks, we uncover a remarkable finding: LLMs' downstream ability is almost linearly correlated with their compression efficiency, with Pearson correlation coefficient of around -0.95 for each evaluated domain of intelligence as shown in Figure 1. Notably, this linear relationship even extends to most individual benchmarks well.
Recent and concurrent works have explored the relationship between benchmark scores and compresiion-equiialenn metrics like validation loss iith the same model series, where the model checkpoints share most configurations such as model designs, tokenizers, and data (Xia al., 2023; Wei et al., 2023; Gadre et al., 2024; Du et al., 2024). Our study, however, is the first to document a linear correlation between compression and intelligence in LLMs across varying model sizes, tokenizers, context window lengths, and pretraining data distributions. Our findings establish the linear correlation between compression and intelligence as a universal principle, providing empirical support for the longstanding belief that superior compression is indicative of greater intelligence.
From a practical perspective, compression efficiency serves as an unsupervised metric for LLMs where the text corpora can be easily updated to avoid overfitting or test contamination. Our findings advocate for adopting compression efficiency as a stable, flexible, and reliable metric to evaluate L.MMs, which is linearly correlated with the models' abilities. We open- source our compression corpora, as well as the data collection and processing pipelines so that future researchers can easily collect and update their own corpora assess compression.
2 Background: Language Modeling is Compression
The discussion on the equivalence between language modeling and compression has long history (Shannon, 1948; 1951). Recently, Deletang et al. (2024) have echoed this connection and demonstrated that language models are general-purpose compressors empirically. Below, we briefly introduce the background to acquaint readers with such a relationship.
2

---

Published as a conference paper at COLM 2024
Lossless Compression of Language: Suppose Pdata is the underlying distribution to gen- erate the text sequence data. Given a text sequence Xts, lossless compression seeks to encode it into a code C(x1.) that is smaller in size while preserving all original information. We typically use binary bits as the coding alphabet of C. According to the source coding theorem (Shannon, 1948), the expected number of bits of an optimal encoder is equal to log2 Pautal,, which is the minimum average number of bits that one can achieve
Bx-paan 1- on compressing text from Pdata in a lossless manner. Knowing Pdatar a common approach to achieve such optimal compression is arithmetic coding (Rissanen, 1976; Pasco, 1977), which practical approach to optimally encode data with a probabílistic model. We refer the readers to Deletang et al. (2024) for more details on arithmetic coding for autoregressive language models.
Connection to Language Models: In practice, however, Patta is often unknown, and we can use a language model Pmodel to help compress x efficiently. Intuitively, we seek to utilize Pmnoee(x) (x) to encode rare sequences with more bits and the frequent ones with fewer bits. Arithmetic coding is able to achieve the optimal expected code length (in bits) as:
Optimal # Bits on Average = Exx-Panta 1= - logg Pmodel (xxl x1a-)),
(1)
which establishes the connection between Pmwodel and compressors. To achieve more efficient compression, we aim to optimize Pmodel to minimize the average code length in Eq. 1. Noting that Eq. is exactly the cross-entropy loss that is used to train language models, learning to compress well is equivalent to language modeling." 2 This implies that if a language model has a lower loss on the data to be compressed, it will be able to compress it with fewer bits in a lossless manner. Strictly speaking, one needs to access the language model parameters to recover the original data from the code, thus the bits required to encode these parameters should be counted as well. However, such a cost can be amortized and becomes negligible when the model is used to compress substantially large volume of data. There are online or offline settings to compress a data stream. In the online setting, the model is randomly initialized and trained on the data to be compressed. In the offline setting, the model is trained on external data and adopted to compress a different dataset. We focus on the offline setting in this paper, using pretrained language models to compress external corpora.
3 Examining Compression and Intelligence - -the Experimental Protocols
Our goal is to empirically examine the relationship between compression and intelligence in the context of large language models. Our method to investigate a variety of pretrained LLMs of different sizes and from diverse organizations, assessing their compression effi- ciency and "intelliience" respectively. Then we aim to summarize the potential relationship between compression and intelligence based on the evaluation results. Our overall principle is to collect diverse LLMs that are created with different training data, tokenizers, compu- tation, and even architectures (e.g., mixture of experts in MistralAI (2023)), that our conclusion on compression and intelligence is general and agnostic to specific model series. Next, we detail our experimental protocols to evaluate compression and intelligence.
3.1 Evaluating Intelligence of LLMs
The definition of "intelligence" has been a subject of debate for years. Alan Turing firstly proposed the definition based on the Turing Test, which evaluates a machine's ability to exhibit intelligent behavior indistinguishable from that of a human (Turing, 1950). Although the Turing test is probably the most commonly accepted definition for intelligence, the application of the Turing Test as a measure for Al is deemed impractical, primarily because the evaluation's outcome relies heavily on the subjective judgment of the interrogator. More recently, Legg & Hutter (2007) proposed a definition termed universal intelligence, broader interpretation than Turing's 's focus on human-like intelligence. They posited
21 "iibrrri like PoToh often implement the cross entropy loss with the log base of rather than 2, which causes constant coefficient difference. Yet, it does not influence the equivalence.
3

---

Published as a conference paper at COLM 2024
Ability
Source
Character
Time Period
Benchmark
NQ (Kwiatkuwski et al., 2019), TQA (Joshi al., 2017),
Knowledge and commonsense
Common Crawl
131M
23/09-23/10
ARC-C ARC-C((CIA (Clark et al., 2018), HellaSwag (Zellers et al,, 2019),
MMLU (Hendrycks et al., 2021a)
Coding
GitHub
98M
23/06-23/09
DS-1000 (Lai et al., 2023)
HvaanEval (Chen al,, 2021), MBPP (Austin et. 2021),
Mathematical
GSM8K (Cobbe al., 2021), MATH (Hendrycks et al,, 2021b), SAT-Math (Zhong et al,, 2023), AMLU-Math
ArXiv
101M
23/09-23/10
reasoning
Table 1: Overview of the compression corpora and benchmarks. "Time Period" refers to the start and end dates of the collected data we use the latest data at the time of the experiments to mitigate data leakage. NQ, TQA, and ARC-C denote NaturalQuestions, TriviaQA, and ARC-Challenge respectively.
that an agent's capacity achieve goals across a wide array of scenarios should define its intelligence. This practical approach to defining intelligence guides our evaluation of models, wherein we assess their intelligence based on average performance across multiple downstream tasks. Such practical evaluation is commonly adopted by most LLM developers to showcase the potential of their models (Touvron et al., 2023a;b; Jiang et al., 2023; Team et al., 2023), and Anthropic (2024) directly noted the "intelligence" of Claude 3 with the average benchmark scores. Specifically, in this paper, we study intelligence along three key abilities: knowledge and commonsense, coding, and mathematical reasoning. These abilities represent the most focused LLM areas nowadays, and we collect the well-recognized benchmarks for each of these areas, utilizing their average score to indicate domain-specific intelligence. The benchmarks are listed in Table 1.
3.2 Evaluating Compression of LLMs
According to $2, evaluate the model's compression efficiency using the average coding length per message of compressing a given corpus. Due to the variety of tokenizers employed by different LLMs, the average bits per token are not directly comparable. There- fore, we utilize the average bits per character (BPC) as the metric. BPC is a commonly used compression metric for compressing text, such as enwik8 (Mahoney, 2011). Specifically,
Metric:
BPC = - -log2 Pmodel (X)
logg Pmodel (x: (xi(
(2)
T
T
where X is the corpus to be compressed, N is the total number of tokens of X tokenized by the model's tokenizer, and T is the total number of characters of X. Note that Eq. 2 is equal to the per-character language model loss on the corpus X (with constant log base shift).
Context Window Unification: LLMs can have different context window sizes, and a longer context window gives advantages to compression efficiency. This is because a larger context window offers more information to predict the next token's distribution and allows for encoding with fewer bits, as illustrated in Eq. 2. However, in downstream benchmarks where input lengths are short, the benefit of extended context windows diminishes. This applies to all our benchmarks in Table 1, where the inputs do not exceed 2048 tokens, even in few-shot in-context learning scenarios. Basically, the downstream tasks in our study only reflect the model's intelligence when operating within relatively short contexts. To study its correlation with compression, it is crucial to assess compression using comparable context length, ensuring consistency in the models' access to information during both corpus compression and execution of downstream tasks. Therefore, for all our compression and benchmark evaluations across different LLMs, we unify the context window size to be 1900 tokens that are sufficient for all benchmark evaluation. Ideally, a more holistic evaluation of intelligence should incorporate tasks involving both short and long contexts. However, in this paper, our focus is on the short- or medium-conter regime, which encompasses the majority of benchmark tasks. We leave the exploration of long-contex: scenarios for future work. Apart from context window unification, we utilize a sliding window approach to evaluate compression more accurately, detailed in Appendix A.1.
4

---

Published as a conference paper at COLM 2024
Focusing on Base Models: In the development of LLMs, there are typically two stages: the pre-training stage and the alignment stage. The models are referred to as base models after the pre-training stage and as fine-tuned models following the alignment stage. We note that fine-tuned models are no longer general-purpon model the next token distribution for arbitrary text, but only for structured (query, response) data. Besides, it is commonly believed that the intelligence of LLMs is learned during the pretraining stage and remains relatively fixed during the alignment stage (Zhou et al., 2023). Therefore, in this work, we focus on the base models only, while in $5 we will further discuss the case of compression v.s. intelligence for fine-tuned models.
compressors since it does not
What kind of corpus shall we select to measure the models' com- pression? Firstly, it is important to recognize that different corpora can illuminate various aspects of models' abilities, and compressing one corpus well may not generalize to an- other (Magnusson et al., 2023). Therefore, we would like to select the corpora that align with the areas of our focus. Secondly, the chosen corpora should not intersect with the models' pretraining data to avoid data leakage. Given the opaque status of LLMs' pre- training datasets, we opt to use the newest corpora as a measure. Concretely, for assessing knowledge and commonsense, we have compiled texts from the latest Common Crawl dataset. To evaluate coding ability, we have sourced cataa from HutHub repositories mainly on the Python language since the downstream benchmarks focus on Python coding abilities. For mathematical reasoning, we collect academic papers from ArXiv, specifically selecting those designated with "math" as their primary category. For each data source, we ensure the recency by utilizing the latest available data at the time of our experiments. The composition of our compression corpora is detailed in Table 1. More details about data collection and processing are in Appendix A.2.
Compression Corpus:
3.3 Overfitting Benchmarks
LLMs may be overoptimized towards certain benchmarks, in this case, the benchmark scores are not a good proxy for intelligence anymore. 3 For example, Bi et al. (2024) show that when adding multi-choice QA training data, the model is able to achieve over 10 absolute points improvement on multi-choice QA benchmarks such as MMLU, while its performance on general QA benchmarks like TriviaQA remains unchanged. In the math domain, recent researches find that some models heavily optimize towards the GSM8K and MATH benchmarks, while performing poorly on held-out math tests (Paster, 2023). In such scenarios, these benchmark scores are not reliable enough to represent the models' intelligence in the respective area. Although we focus on base models in this work, where such overfitting phenomenon may be less common, it is still possible that some LLMs are specially trained on the training data of the corresponding benchmarks in the pretraining stage, or even worse, suffer from test data contamination issues as evidenced in Wei et al. (2023). To identify such cases, we adopt the MIN-K% PROB method (Shi et al., 2024) which is proposed to detect whether the given text is exposed to the model during pretraining. The MIN-K% PROB approach selects the 4% tokens in a given example with minimum probabilities, if the average probability of these tokens is high, then the example likely to be present in the pretraining data. In this approach, the MIN-K% Score is defined to be the average negative log-likelihhoo of these selected tokens. In the experiments next, we compute the MIN-K% Score for all the test splits of the benchmarks, and for the training splits as well available. We will spot and discuss the LLMs with extremely abnormal MIN-K% Scores. We choose k to be 20 in our experiments as suggested by Shi et al. (2024).
How is Intelligence Correlated with Compression?
To provide a comprehensive study of the correlation between compression efficiency and intelligence, we include 9 series of general-purpose language models, covering diverse organizations, varying in size and architectures, as shown in Table 2. Notably, in addition to dense transformer models, we further include Mixtral-8x7B (MistralAT, 2023)
General Setup:
3hheen measure becomes target, ceases to be good measure" Goodhart's law.
5

---

Published as a conference paper at COLM 2024
Series
Creator
# Parameters
General-paypose Language Model
Deepseek-IIm (Bi et al., 2024)
DeepSeck
7B, 67B
Falcon (Almazrouei et al., 2023)
7B, 40B
Jamba (Lieber al., 2024)
A121
52B
Llama-1 (Touvron et al., 2023a)
Meta
7B, 13B, 30B, 65B 7B, 13B, 70B
Llama-2 (louvron et al., 2023b)
Meta
Mistral (Jiang et al., 2023)
MistralAT
7B 8x7B
Mixtral-8x7B (MistralAI, 2023)
MistralAI
Qwen (Bai et al., 2023)
Alibaba
7B, 14B, 72B
Yi (Al al., 2024)
01.A1
6B, 34B
Code Language Model
Code (Roziere et al., 2023)
Meta
7B, 13B, 34B
Deepseek-coder (Guo et al., 2024)
DeepSeek
1.3B, 6.7B, 33B
StarCoderlaase (Li et al., 2023)
BigCode
15,5B
StarCoder2 (Lozhkov et al., 2024)
BigCode
7B, 15B
Mathematical Language Model
Deepseek-math (Shao al., 2024)
DeepSeek
Llemma (Azerbayev et al., 2023)
EleutherAl
7B, 34B
Table 2: Models evaluated in this paper.
and Jamba-52B (Lieber et al., 2024) to validate our findings across different architectures. Mixtral-8x7B utilizes a sparse mixture-of-experts (MoE) architecture, whereas Jamba-52B based on a hybrid Transformer-Mambl MoE architecture. Furthermore, we incorporate state-of-the-arr models specialized in coding and mathematical reasoning in addition to general-purpose models, as outlined in Table 2. We assess general-purpose LLMs across all benchmarks, while we include code LLMs in the code area only and mathematical LLMs in the mathematical reasoning area only. We note that our investigation focuses exclusively on well-trained LLMs, implying that the intelligence evaluated is likely already manifest, in contrast to models where such abilities have yet to emerge (Wei et al., 2022). This because LLMs that are not optimized enough tend to perform poorly, where the results are overly noisy and less meaningful, complicating the analysis. Models are evaluated using few-shot in-context learning in zero-shot manner, adhering to the established norms for the respective benchmarks. All the models are evaluated exactly the same way with the same codebase for fair comparison. More details on evaluation are included in Appendix A.3.
As described in $3.2, we utilize bits per character (BPC) as the evaluation metric for compression efficiency. To quantitatively assess the correlation between intelligence and compression, we report the Pearson correlation coefficient (Pearson p) between the average benchmark scores and BPC. Based on preliminary observations that the correlation is highly linear, we perform linear fit between the average benchmark scores and BPC, and report the Root Mean Square Error (RMSE e) that captures how much the models' benchmark scores deviate from the linear prediction on average. We extend this analysis to individual benchmarks as well in addition to the average. Qualitatively, we plot the linear regression results to visualize the relationship, with each model being a point in the figure.
Metrics:
4.1 Main Results - Compression Represents Intelligence Linearly
Overall intelligence v.s. compression: We have summarized our main results in Figure where we examine the overall relationship by computing the average benchmark scores across all three areas, and the average BPC across all the compression corpora. The average score and the average BPC demonstrate highly linear correlation qualitatively in the visualization. Quantitative results with Pearson correlation coefficient of -0.93 and an RMSE of 3.1% further verify the linear relationship. We consider such a linear correlation significant given the noisy nature of evaluating on downstream benchmarks - is known that varying the prompts or decoding hyperparameters could easily cause several points of difference for the task scores (Lu et al., 2022; Gonen et al., 2022; Sclar et al., 2024). We also report the results for the three key areas respectively which demonstrate similar phenomena qualitatively and quantitatively. We note that several previous and concurrent works have studied the correlation between benchmark scores and validation losses in limited settings-

---

Published as a conference paper at COLM 2024
Knowledge
0.8
0.8
e: 0.019
6.881
0.7
c: 0.016
0.4
0.55
0.65
Average Score
2ame
gren bmba-552b Deepseekk
Mixtral-8x7b
i-bb
6: 0.-0.0 01991
0.6
0.8
Owen-14b Laaaa 1-13b
0.6
Liama-2-13
Garma 2.7b
4-66
0.4
0.5
Plama-1
0.55
0.60
0.65
Deepseek-llm-78
40,90
0.921
0.4
0.6
0.52
0.54
0.56
0.58
0.60
0.62
0.64
0.66
0.55
0.60
0.65
Bits per character
Figure 2: Left: Correlation between benchmark scores and BPC in the knowledge and commonsense area. Right: The correlation on Hellaswag, TriviaQA and MMLU respectively. Results on NQ and ARC-C are shown in Figure (()) and Figure 9(b). BPC computed on collection of Common Crawl.
Coding
HumanEval
0.7
e: 0.040
pt D0-97.37
0.050
0.5
0.6
Despseckkcoder-33
0.20
0.25
0.30
0.35
0.40
Average Pass@1
Codellam ae
Owen
Deepsrok-
MBPP
0.4
Codellam
8x7b
-0 916
0.055
Jamba-5200
Owen-14b
Codellama
Y1448
Mistral-7b
1
0.3
Liama- 2. Tbb
0.0
0.20
0.25
0.30
0.35
0.40
Llama-2-130 2-136
D5-1000
Liama-22
2-78
0.1
0.5
0.0
0.20
0.23
0:25
0.28
0.30
0.39
0.35
0.38
0.40
0.20
0.25
0.30
0.35
0.40
Bits per character
Figure Left: Correlation between average benchmark scores and BPC in the coding area. Right: The correlation on HumanEval, MBPP, and DS-1000. BPC is computed on collection of GitHub repos.
they focus on either the model checkpoints of the same model over the course of pretraining or the same model series (Xia et al., 2023; Lu et al., 2023; Wei et al., 2023; Gadre et al., 2024; Du et al., 2024). Their studied checkpoints share designs and tokenizers and are pretrained on the same data distributions. However, our work escapes the constraints on the same model (series) and compares across models that diverge dramatically on tokenizers, model designs, and pretraining data. Our results mark the linear correlation between compression and intelligence in a general manner, establishing it as a universal principle. The specific benchmark scores and BPC for every model are reported in Appendix B. Next, we discuss the results in the three areas with more details respectively, and extend the examination to individual benchmarks as well.
We present both the average and individual benchmark results for the knowledge and commonsense area in Figure 2. Notably, the scores from individual benchmarks exhibit strong linear correlation with compression efficiency, indicated by Pearson coefficients of -0.881, -0,951, and -0.890 for HellaSwag, TriviaQA, and MMLU, respectively. The Root Mean Square Error e is approximately 2 absolute points, which falls within the expected range of variation Minor fluctuations in evaluation details, such prompt adjustments, readily account for 2-point accuracy shift (Sclar et al.,
Knowledge and Commonsense:
7

---

Published as a conference paper at COLM 2024
Mathamatical Reasoning
GSMAK
0.8
p: -0.953
0--0.922
0.7
0.031
0.068
Outlier
Owen 72b
Average Accuracy
0.5
Mixtral-Bx7b
Y1-44
Deepseek-matt-76
0.40
0.45
MATH
0.50
0.55
Deepseek- Bri
P:-0.899
Jambe $2b
Lisma.2 2
16ma-11655
Owen-7b
0.5
Yi-8b
Llame-2-130
Bec
Uaama 1-7b
0.40
0.45
0.50
0.55
0.0
Liama 2.70
Uama- 1:130
aacon-76
pt 22-0..5
0.044
0.38
0.40
0.43
0.45
0.48
0.50
0.53
0.55
0.40
0.45
0.50
0.55
Bits per character
Figure 4: Left: Correlation between average benchmark scores and BPC in the mathematical reasoning area. Right: The correlation GSM8K, MATH, and SAT-Math. Results MMLU-Math is in Figure 9c). BPC is computed on a collection of ArXiv papers with "math" tag. Outliers denote the models are likely overfitting the benchmarks, which are detected by the Min-K% PROB method as detailed $3.3.
0.5
Querr Th. 10h 775
0.
0.8
Owen 14b 8.8
0.5
Dwwn 7b, 148 72b
0.0
0.
0.6
0.6
0.4
0.4
0.4
0.4
0.2
02
0.2
0.2
0
GSHBK Train Sat
GSMBK Test Set
MATH Train Set
MATH Test Set
Figure 5: Distribution of MIN-K% scores calculated with assessed models across four datasets, displayed in linear array (k 20). The y-axis represents the density of distri- butions, where higher values indicate a greater concentration of scores, achieved through kernel density estimation.
2024). Meanwhile, observe that the linear correlation in both HellaSwag and MMLU is not as pronounced as in TriviaQA, albeit for different reasons. For HellaSwag, the average accuracy among models has exceeded 0.8, hinting at saturation point in performance that blurs the distinctions between models. Conversely, MMLU, which draws heavily from real-world examinations and textbooks, presents slight mismatch in distribution compared to the Common Crawl dataset used to measure compression efficiency. Further experiments on MMLU and textbook compression are detailed in Appendix E, where Pearson coefficient of -0.930 is observed. We note that for MMLU evaluation, we employ cloze-style input format for this popular multiple-choice question (MCQ) benchmark, presenting only the question without the options, and subsequently selecting the choice with the highest likelihood. We intentionally opted for this less conventional evaluation style to minimize the risk of models overfitting to the MCQ format, thus over-optimizing the scores. Please see Appendix for detailed discussion and ablation.
Similar to the knowledge and commonsense area, there is a strong linear correla- tion between the models' compression efficiency and its coding ability, with the Pearson correlation coefficient within 1- -0.9] for each individual benchmark. Furthermore, de- spite having significantly different architectures, both Mixtral-8x7B and Jamba-52B fit well into this linear correlation, demonstrating that our findings are agnostic to model archi- tecture. We notice the strongest linear correlation within the DS-1000 benchmark, with Pearson coefficient of -0.946, surpassing those of HumanEval and MBPP. We think that this is because we collect Python code from the popular GitHubb repositories in recent months, which are mostly related to data science and machine learning, aligned closer with DS-1000.
Coding:
8

---

Published as a conference paper at COLM 2024
0.6
0.8
Fi --
0 826
0.5
0.7
6.5
0.3
0.5
0.2
0.4
-6.0
0.027
0.1
0.55
0.00
0.05
0.3
0.25
0.30
0.35
0.40
20
200
Figure 6: Results of correlation between Math Bench and Common Crawl BPC (Left) and between knowledge and com- monsense Bench and Python BPC (Right).
Figure 7: Variation of the Pearson coeffi- cient (Left) and the RMSE (Right) with re- spect #characters of the CC corpus in knowledge and commonsense ability.
We note that the Deepseek-coder model series are consistently well-above the linear fit, demonstrating strong abilities on the HumanEval and MBPP benchmarks. We hypothesize that this due the fact that Deepseek-coder models are exposed to private, task-specific data during pretraining that help HumanEval and MBPP significantly, yet we do not have reliable methods to detect the existance such data and verify this hypotheis.
Mathematical Reasoning: Overall the correlation results in the mathematical reasoning area are similar to the other two, as shown in Figure 4. However, overfitting benchmarks in mathematical reasoning warrants particular attention. As described in $3.3, we compute the MIN-K% Score for every example- which the average negative log likelihood of the k% tokens with the lowest probabilities on the benchmark test data well training data if available. Abnormally low MIN-K% Scores imply that the example is likely to be present in the model's pretraining data. We did not observe abnormal MIN-K% patterns in all the previous experiments, yet, there are models with far lower Min-K Scores on the GSM8K and MATH datasets than all other models. We visualize the MIN-K% distribution of the training and test examples on both GSM8K and MATH in Figure 5, which implies that the Qwen model series may be exposed to the GSM8K training data, MATH training data, and even the MATH test data in the pretraining stage. Therefore, we spot the Qwen model series in our visualization and exclude them when performing the linear fit and compute the correlation metrics. As expected, the three Qwen models achieve generally higher accuracies than the predicted performance from the linear fit, and appear as outliers in Figure 4.
At last, we highlight that the strongly linear correlation between the single benchmark scores and compression efficiency originally unexpected, because individual benchmark scores are typically too noisy to represent models' general ability. However, current findings indicate that these individual scores may be predictable from the BPC with a minor error in most cases indicated by small RMSE.
4.2 Remaining Questions
How does different compression corpus influence the linear relationship? In the previ- ous experiments, we intentionally selected the appropriate compression corpora aligning with the area of focus. Herein, we further explore the effects of diverse compression corpora on the results. We investigate two representative examples where the compression corpus does not align with the specific domain, to study whether an arbitrary compression corpus would suffice for a strong linear correlation: (1) mathematical reasoning benchmarks v.s. compression on Common Crawl data, and (2) knowledge and commonsense benchmarks v.s. compression on GitHub code data. We present the findings in Figure 6. Due to substantial ability mismatch between the compression corpus and the benchmarks, the linear correlation is significantly weaker than that with in-domain data. Coupled with the MMLU case discussed in Appendix E, these instances demonstrate that linear correlation relies on the alignment degree between the compression corpus and the specific area of study. However, emphasize that our previous findings are relatively robust since our chosen compression corpora as listed in Table rely on simple heuristics at a coarse granu- larity without meticulous designing. How to automatically identify the most appropriate

---

Published as a conference paper at COLM 2024
Average hpc
Python bpc
1.0
1.0
p: -0.929
6: 60-02331
0.8
stareeger2
15c
Qwen-725
e: 0.075
Mixtral-8g77
Jan 70b Qwen-140
0.6
Deepseek
Accuracy
Jamba 528
Mistral-7t
Qwen-7b
0.4
Deense coderr
0.20
0.25
ArKiv-Math bpc
0.30
0.35
0.40
Starcoder2
Llam-221300
-0787
0.2
Starcoderbast
co
Uama 2-7b
0.0
0.30
0.33
0.35
0.38
0.40
0.43
0.45
0.48
0.40
0.45
0.50
0:55
0.60
Bits per character
Figure 8: Left: Correlation between PAL-GSM8K scores and BPC on the combined Python and ArXiv-Math corpus, Right: The correlation between PAL-GSM8K scores and BPC metrics within Python and ArXiv-Math corpus, respectively.
compression corpus that correlates with certain benchmarks is an important problem that we leave for future work.
How many characters are required to compute BPC reliably? As an example, we investi- gate the effect of size of compression corpus on the knowledge and commonsense ability, We vary the number of characters of the compression corpus from 50K to 100M, observing changes in the Pearson correlation coefficient and the RMSE depicted in Figure 7. As the number of sampled characters increases, the linear correlation gradually strengthens and reaches saturation at around 3M. This observation suggests that the compression corpus sufficiently large with only tens of millions of characters. We further provide an analysis of the impact of random sampling on compression evaluation in Appendix C.
What about extending to cross-ability tasks? The previously discussed experiments focus exclusively on tasks related to single ability. Nevertheless, numerous tasks require abilities in multiple aspects. For example, incorporating programming as an intermediate step for reasoning has been shown to enhance performance in mathematical tasks, requiring abilities in both coding and mathematical reasoning (Gao al., 2023b; Chen et al., 2023). Following Gao et al. (2023b), we apply the Program- Prdde Math Reasoning (PAL) method to GSM8K. To align with the abilities involved, we combined the Python and ArXiv-Math corpora to create mixed corpus. As illustrated in Figure 8, this mixed corpus exhibits stronger linear correlation than using either the Python or ArXiv-Math corpora alone.
Discussion
Compression as a reliable evaluation metric: Our findings provide evidence for the belief that superior compression is indicative of greater intelligence. Practically, our experiments strengthen the rationale to adopt compression efficiency as an unsupervised, flexible, and reliable metric to assess LLMs' abilities. The compression corpus can be easily updated and composed flexibly, which mitigates the data contamination and benchmark overfitting issues that most evaluations suffer from. While recent work establishes large collection of diverse corpora to evaluate LLMs with losses (Magnusson et al., 2023), our work offers strong empirical support for doing so by demonstrating the correlation with intelligence.
Limitations: Our study admits several limitations. First, we only focus on base models because fine-tuned models are not general-purpost compressors for arbitrary text. However, we posit that there are still interesting relationships to be explored about the base model compression efficiency and the benchmark scores of the corresponding fine-tuned models. Yuan et al. (2023) initially provides evidence of this correlation in mathematical reasoning ability, albeit limited to LLaMA models. Second, our focus lies on the short- to medium- context regimes, deferring the examination of long-context scenarios. Third, our conclusion and findings may only apply to well-traines models and not hold for LMs where the evaluated abilities have not emerged yet. We leave study of these problems for future work.
10

---

Published as a conference paper at COLM 2024
Acknowledgements
We thank Graham Neubig, Yao Fu, and Pengcheng Yin for their valuable feedbacks for this draft.
References
01. AI, ;, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Y1: Open foundation models by 01.ai, 2024.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxan-
dra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, al. The falcon series of open language models. arXiv preprint arXit:2311.16867, 2023.
Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, et al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. arXiv preprint ar arXiv:2402.01781, 2024.
Anthropic. Introducing the next generation of claude, 2024. URL WRR.https:/www.aathrrgg. com/news/clluude33-
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language language models. arxiv preprint arXiv:2108.07732 2021.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck, Llemma: An open language model for mathematics. arXiv preprint ar arXii:2310.10631, 2023.
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling, ar Xrv preprint ar Xiix:1809.1085: 2018.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Sheng- guang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. Xiv preprint ar Xri:::2309.1660 2023.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, al. Deepseek llm: Scaling open-source language models with longtermism. ar preprint ar Xito:2401.02954 2024.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saf- fron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2206-2240. PMLR, 17-23 Jul 2022. URL https: // http:///oooceegg../////m mlr. presss/v162/borgeaudde html.
11

---

Published as a conference paper at COLM 2024
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastr,, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo- hammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. ar Xiv preprint ar XxXx:2107.00374 2021.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL htttt::////peeereiieee net/Forum?id=>FZZZP t8zd.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arviv preprint arXiv: 1K1.:1803305457, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüchiro Nakano, et al. Training verifiers to solve math word problems. aix preprint arXir:2110. 2021.
CodeParrot. codeparrot-clean.
https://huggiingface. http:////gggnggoeee..ooooott//ss cot.aaattttess//oeeerrr..tt
codeparrot-clean, 2021.
OpenCompass ConttibtassConttibuutoosss Contributors.
Opencompass: A universal evaluation platform for foundation
models. https:// http:///./.ooooooocoooo/////c//m 2023.
Gregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. Language modeling is compression. In The Twelfth International Conference on Learning Representations, 2024. URL htttt::://ppnreeviee met/forum?ld=jjnngiiy ynus.
2024.
Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and lie Tang. Understanding emergent abilities of language models from the loss perspective, 2024.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Mar- ianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muen- nighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on downstream tasks, 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023a. URL https://zenodo. org/recoodss/12255 10256836.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aides language models. In International Conference on Machine Learning, pp- 10764-10799. PMLR, 2023b.
12

---

Published as a conference paper at COLM 2024
Hila Gonen, Srini lyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer Demystifying prompts in language models via perplexity estimation. arviv preprint arXiv:2212.04037, 2022.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming rise of code intelligence. arXiv preprint arXto:2401.14196, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. URL
d7KB 3GmQ.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b.
José Heraández-Orallo and Neus Minaya-Collado. A formal definition of intelligence based on an intensional variant of algorithmic complexity. In Proceedings of International Symposium of Engineering of Intelligent Systems (EI598), pp. 146-163, 1998.
Marcus Hutter. The hutter prize. http://ppize..e
net, 2006.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. Xiv preprint ar Xrii:23100068255, 2023.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer: TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzi- lay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 1111863/vv1/117-1/ kttps://aaclan http::://cclanthooogy.ooro///// ort/
URL
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Compulational Linguistics, 14533466, 2019.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen- Tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 18319-18345. PMLR, 23-29 Jul 2023.
Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence.
Minds and machines, 17:391-444, 2007.
Shane Legg, Marcus Hutter, et al. A universal measure of intelligence for artificial agents. In International Joint Conference on Artificial Intelligence, volume 19, pp. 1509. LAWRENCE ERLBAUM ASSOCIATES LTD, 2005.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muh- tasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashí Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Ku- nakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire
13

---

Published as a conference paper at COLM 2024
Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! arXiv preprint ar Xit::300.066611 2023.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez et al. Jamba: hybrid
Safahi, Shaked Meirom, Yonatan Belinkov, Shai transformer-mambe language model. ar preprint arXiv:2403.19888 2024.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint #xXi::2402.1913,, 2024.
Hua Lu, Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. the open-domain chatbot with human feedback. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4060-4078, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ v1/2023.acl-long wrg.//23..... 2023.02222222 acl- acl-long..22 224.
Towards boosting Jordan
URL https: / /aaaaaaaaaannoooooo/oo org
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 8086-8098, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 wwwp//22.ccllooo....... URL https://aclanthhot wwwphhholloyy.o./222222222 acl long. 556.
Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groen- eveld, Iz Beltagy, Hannaneh Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit, 2023.
Matt Mahoney. Large text compression benchmark, 2011.
Matthew V Mahoney. Text compression as test for artificial intelligence. AAAI/IAAI, 970,
1999.
MistralAI. Mixtral of experts: A high quality sparse mixture-of-experts. Mistral Blog, 2023.
URL www.mis www.mistral. oftp///ww/mmixtaallooeeeeeeppeee
R. Pasco. Source coding algorithms for fast data compression (ph.d. thesis abstr.). IEEE Transactions on Information Theory, 23(4):548-548, 1977. doi: 00.110//TTT.197.//
Keiran Paster. Testing language models on held-out high school national finals exam. httpr/iinaannnncctteeceteeeeeeee
https://huggingface co/ www.aatsstttsekkeee////aaaaaaana wtwaallsssiinaleeeeaee 2023.
Ofir Press, Noah A. Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5493-5505, Online, August 2021. Association for Computational Linguistics. doi: 10.18653 //////1//22/// v1/2021.aclloog.442 427. URL httph:tt///// http:///aaaaaahooooooooo org/ arg/222.....llloonn acl-long.4444 427.
J. Rissanen. Generalized kraft inequality and arithmetic coding. IBM Journal of Research and Development, 2003):198-03, 1976. doi: 10.1147/rd.203.011
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. ar Xiv preprint Sttf::233..29554 2023.
14

---

:ttpt//oorummiidtaaaa3rat0 Published as a conference paper at COLM 2024
Melanie Sclar, Yejin Choi, Yulia Isvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, 2024. URL htt.hhttt::/ppeeeeee /tpp///eepeeeeeeeeeeeeeeeee wwweeeewwwwweeeeeeeeeeeeeeeeeeee www///eutmmmieeeemm d=-.
C. F. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3))379-4233 1948. doi:
tb01338xx.
Claude E Shannon. Prediction and entropy of printed english. Bell system technical journal, 30(1):50-64, 1951.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint ar drXio:2422.03300, 2024.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Tweelfth International Conference Learning Representations, 2024. URL https://openreviee
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Batiiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. ar preprint ar arXio:2312.11805, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar 11X::2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et Llama Open foundation and fine-tuned chat models. arXiv preprint 1xxiv:2307.09288,. 2023b.
AM Turing. Computing machinery and intelligence. Mind, 59(23)):433 460, 1950.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Fd H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL wwwttoeummineeeeeeeeee iiw.u///iidzssssssdd..... Survey Certification.
Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, et al. Skywork: A more open bilingual foundation model. arXiv preprint ar ar1o:23310.9944 2023.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco
Guzmán, Armand Joulin, and Édouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of The 12th Language Resources and Foaluation Conference, pp. 4003-4012, 2020.
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Veselin Stoyanov. Training trajectories of language models across scales. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13711-13738, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653. /v1 201/2/23.cclloo// 767. URL htttt:://acaanttiii acl-long.767, 767.
or233023.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. Xiv preprint ar axXtt:2308.01825, 2023.
15

---

Published as a conference paper at COLM 2024
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? Anna Korhonen, David Traum, and Lluís Márquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 14791-48000 Florence, Italy, July 2019. Association for Computational Linguistics. doi: het.httts///allaantol.g.. hat.laannttooogg.rr///11-11444 org/P19-
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint ar Xtt:2330.00664 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: weweerreeiee.ne/oorrmnee KBMOKmX2he.
16

---

Published as a conference paper at COLM 2024
A Detailed Setup of the Main Experiment
A.1 Sliding Window
We evaluate models' compression with sliding window, characterized by the repetitive sliding of the context window at a fixed stride of 512. This method, widely adopted in language modeling evaluation (Baevski & Auli, 2018; Press et al., 2021; Borgeaud et al., 2022), provides more contextual information. It effectively reduces the initial spike in metric observed at the start of the input segment. Furthermore, this approach better aligns with common application scenarios, which rely on preceding context for predictions.
A.2 Data Collection
Data Processing To mitigate data leakage, we collect the most recent data available at the time of the experiments. The data sources and corresponding collection periods are detailed in Table 1. In this study, we utilize standard data processing frameworks to collect and refine our validation data. For knowledge and commonsense ability, we download the Common Crawl dump number "2023-40", which comprises web data crawled between September 21st and October 5th. And then we leverage the CCNet pipeline (Wenzek et al., 2020) to streamline the data cleaning processes. For coding ability, we access repositories containing Python code via the GitHub GraphQL API and apply the Codelarrot pipeline (CodeParrot, 2021) to filter out low-quality content. For mathematical reasoning ability, we obtain the September 2023 LaTTeX source file dump via ArXiv's S3 Bulk Source File Access². We then convert the LaTeX source files Markdown using the pandoc library and selectively retain those papers that list "math" as their primary tag. For more implementation details, please refer to our GitHub repository".
Data Sampling Through the process mentioned above, we collect more data records than needed. Therefore, it becomes necessary to downsample the data. We employ document-level random sampling strategy for our data, except Python code. Code files within the same repository frequently exhibit interdependency. To maintain certain level of interdependency while ensuring the diversity of the repository, we implement a trade-off solution segment-level sampling, which divides each repository into segments and then samples at segment level, thereby maintaining interdependency within segments.
A.3 Benchmark Evaluation
For benchmark evaluation, all models assessed following the common-used protocol, as detailed in Table 3. We utilize two types of evaluation methods: the perplexity-based method and the generation-based method. The perplexity-based method involves calculating and normalizing the perplexity for each option and selecting the option with the lowest perplexity. And for the generation-based method, we use greedy decoding uniformly. For the overall results presented in Figure 1 (Left), the average benchmark score (y-axis) calculated by averaging the scores across benchmarks, while the average bits per character (x-axis) is determined by averaging the BPC across three distinct areas.
B Detailed Results of Model Evaluation
Knowledge and Commonsense Ability: The correlations NaturalQuestion. and ARC- Challenge are shown in Figure (a) and Figure 9b),, respectively, The detailed evaluation results of knowledge and commonsense ability are shown in Table 4. The breakdown of the
"nttps://docs http:///ooc....gubbbb.... com/en/graphql
chttps://info.arxxv hwt...wwww....g//ll///...aaaaaaa html
"nttps://pandoc.org
17

---

Published as a conference paper at COLM 2024
Ability
Benchmark
N-shot
Evaluation Method
Codebase
Metric
NaturalQuestions TriviaQA
Generation-basee
LM- MM-Evaluation-Hae Harness
Exact match
Knowledge and commonsense
Generation-based
I.M-Ealuation--aan Harnesss
Exact match
ARC-Challenge
Perplexity-based
LM-Evaluation Aaaluaation-aaneess
Acc norm
Hellaswag
Perplexity-basee
IM-Evaluation-H Harness
norm
MMLU
Perplexity-bus -asedd
IM-Evaluation-Harmee
Acc.norm
HumanEival
Generation-basee
Deepseek-Coder
Pass01
Coding
MBPP
beneration-basee
InCoder
Pass01 I
DS-1000
Generation-based
Deepseek-Coder
Pass@1
GSM8K
Generation-basee
MM-Evaluation-Harress
Acc
Mathematical
MATH
Generation-base OpenCompass
Acc
reasoning
SAT-Math
Generation-base
AGlEval
Acc
MMLU-Math
Perplexity-ben bbased
IM-Evaluation-Harnees
Acc
T:ble Evaluation details of benchmarks. The perplexity calculating and normalizing the perplexity for each option and selecting the option with the lowest perplexity. For the majority of benchmarks, we utilize the LM-EEvluation- Harness framework (Gao et al., 2023a). Additionally, supplementary, including Stepseek-Coder (Guo et al., 2024), InCoder (Fried et al., 2023), OpenCompass (Contributors, 2023) and AGIEval (Zhong et al., 2023).
evaluation method involves
incorporate other sources
Model
HellaSwag
ARC-Challenge
NaturalQuessions
TriviaQA
MMLU
Average
BPC on CC (U)
6-7B Parameter Model
Deepseek-11m-76
76.1
52.5
52.9
43.1
48.9
0.635
Falcon-7b
76.3
47.5
21.4
52.3
41.9
47.9
0.649
Llama-1-7b
76.2
51.1
55.3
43.1
0.629
Llama-2-7b
76.0
54.1
24.7
56.5
44.3
51.1
0.612
Mistral-7b
81.1
Qwen--7
76.8
52,2
23.7
53.4
44.3
50.1
0.645
Yi-6b
75.0
54.9
22.9
52.8
45.8
50.3
0.638
13-15B Paramter Model
Llama-1-13b
79.1
56.1
27.8
62.1
46.1
54.2
0.609
Llama-2-13b
79.4
59.0
29.2
63.1
47.6
55.7
0.581
Qwen-114
81.3
58.3
31.8
60.6
48.6
56.1
0.620
30-40B Parameter Model
Falcon-40b
0.593
Llama-1-30b
82.6
61.9
32.8
67.9
50.3
59.1
0.577
Yi-34b
83.7
64.5
36.7
69.2
61.8
0.572
50-72B Parameter Model
Deepseek-llm-67b
84.4
66.3
35.9
71.4
54.8
62.6
0.568
Jamba-52b
84.0
72.1
0.542
Llama-1-65b
84.2
63.2
35.5
70.6
51.4
61.0
0.557
Llama-2-70b
83.8
67.6
37.6
72.1
54.6
63.1
0.527
Mixtral-8x7b
84.0
37.1
72.2
56.5
0.539
Qwen-72b 85.0
64.0
36.7
68.2
54.5
61.7
0.537
Table 4: Results of knowledge and commonsense ability. We report models' compression efficiency on the Common Crawl (CC) dataset measured by bits per character.
Benchmark Hellaswag
ARC-Chanllange
NaturalQuestions
TriviaQA
MMLU
Average
Pearson 0 RMSE e
-0.881
-0.913
-0.925
-0.951
-0.890 0.021
-0.935
0.016
0.023
0.023
0.022
0.019
Table 5: The correlation for knowledge and commonsense ability. Here, Pearson denotes the Pearson correlation coefficient, and RMSE represents the root mean square error.
correlation between benchmark scores and compression efficiency on the validation data is presented in Table 5.
Coding Ability: The detailed evaluation results of coding ability are shown in Table 6. The breakdown of the correlation between benchmark scores and compression efficiency on the validation data is presented in Table 7.
18

---

Published as a conference paper at COLM 2024
Model
HvmanEval
Mbpp
DS-1000
Average
BPC on Python Code (L)
1-7B Parameter Model
Codellama-7b
31.1
41.6
25.4
32.7
0.268
Deepseek-coder-l...
32.9
46.2
32.0
0.316
Deepseck-coderr-6
60.6
0.262
Deepseek-11m-7b
25.0
34.4
13.4
24.3
0.338
Falcon-7b
12.4
3.4
8.5
0.393
Llama-1-7b
12.2
18.6
5.4
12.1
0.379
Llama-2-7b
13.4
21.6
6.5
13.8
0.354
Mistral-7b
29.9
36.2
22.8
29.6
0.310
Qwen-7b
28.7
36.4
15.5
26.9
0.309
Starcoder2-7b
36.0
48.8
31.3
38.7
0.266
Yi-6b
15,9
22.4
11.8
16.7
0.351
13-15B Paramter Model
Codellama-13
36.0
47.6
27.7
37.1
0.251
Llama-1-13b
21.6
0.356
Llama-2-13b
26.6
19.2
0.334
Qwen- 14b
32,9
34.2
0.285
Starcoder2-15b
45.1
61.2
36.5
47.6
0.238
StarcoderBBase
33.5
43.0
25.0
33.8
0.255
30-40B Parameter Model
Codellama-34b
48.2
56.0
35.2
46.5
0.226
Deeeseek-coer--3b
54.9
60.4
40.8
52.0
0..255
Falcon-40b
25.0
23.2
13.8
20.7
0.320
Llama-1-30b
22.6
28.8
11.8
21.1
0.321
Yi-34b
25.6
39.6
25.4
30.2
0.297
50-72B Parameter Model
Deepseck-1lm-670
42.7
55.6
30.7
43.0
0.280
Jamba-52b
31.7
48.4
25.1
35.1
0.244
Lama--165b
25.6
36.2
18.7
26.8
01300
Llama-2-70b
28.7
37.0
25.4
30.4
0.287
Mixtral-8x7b
37.2
50.0
30.1
39.1
0.274
Qwen-72b
49.4
52.0
32.8
44.7
0.256
Table 6: Results of coding ability. We report models' compression efficiency on the Python dataset measured by bits per character.
Benchmark
HumanEval
Mbpp
DS-1000
Average
Pearson p RMSE
-0.903
-0.916
-0.946 0.032
-0.937
0.050
0.055
0.040
Table 7: The correlation for coding ability. Here, Pearson p denotes the Pearson correlation coefficient, and RMSE e represents the root mean square error.
Mathematical Reasoning Ability: The correlation between MMLU-Math and compression efficiency on the ArXiv-Math data is illustrated in Figure 9(c). The detailed evaluation results of mathematical reasoning ability are shown in Table 8. The breakdown of the correlation between benchmark scores and compression performance is presented in Table 9.
C Exploring the Impact of Random Sampling on Compression Evaluation
To investigate how random sampling affects results, we we studied its impact on knowledge and commonsense ability, sampling 131M characters each time, in our main experiment, for a total of three times from a pool of 1.3B characters for diversity. Results in Table 10 show minor variations between models, with average standard deviation of 0.0021.
19

---

Published as a conference paper at COLM 2024
Model
GSMSK
MATH
SAT-Math
MMLU-Math
Average
BPC on ArXiv-Math (J)
6-7B Parameter Model
Deepseek-llm-7b
0,500
meatseck-math-76
59.2
28.0
50.5
0,420
Falcon-7b
29.5
34.7
17.3
0.541
Llama-1-7b
10.7
3.0
26.4
36.1
19.0
0.510
Llama-2-7b
13.7
3.5
28.6
39.5
21.3
0.500
Llemma-7b
31.7
14.9
52.7
39.6
34.7
0.431
Mistral-7b
33.4
11.8
48.6
46.1
35,0
0.443
Qwen-7b
42.8
13.4
43.6
41.3
35.3
0.483
Yi-6b
31.1
6.5
33.6
40.8
28.0
0.483
13-15B Paramter Model
Llama-1-13b
13.6
4.1
28.6
39.0
21.3
0.487
Llama-2-13b
21.9
5.3
31.4
42.8
25.3
0.475
Qwen-14b
55.7
26.1
57.3
47.1
46.5
0.450
30-40B Parameter Model
Falcon-40b
23.0
4.0
31.8
43.0
25.5
0.482
Llama-1-30b
32.1
36.4
44.9
30.0
0.456
Llemma-34b
0,416
Yi-34b
52.3
0.421
50-72B Parameter Model
Deepscek-Ilm-676
56.0
16.6
54.1
50.1
44.2
0.430
Jamba-52b
48.3
18.3
55.0
49.4
42.7
0.417
Llama-1-65b
42.5
10.0
41.8
45.3
34.9
0.441
Llama-2-70b
50.4
12.4
50.9
48.0
40.4
0.429
Mixtral-8x7b
23.7
54.4
Qwen-72b
36.2
71.4
52.3
58.1
0.415
Table 8: Results of mathematical reasoning ability. We report models' compression efficiency on the ArXiv-Math dataset measured by bits per character.
Benchmark GSMSK
MATH
SAT-Math
MMLU-Math
Average
Pearson +0.922
-0,899
-0.933
-0.916
-0,953
RMSE
0.068
0.033
0.044
0.022
0.031
Table 9: The correlation for mathematical reasoning ability. Here, Pearson denotes the Pearson correlation coefficient, and RMSE represents the root mean square error.
0.5
0.8
0.6
00.95
0.913
4.916
Jama-2-70b
0.023
Deep 0.923
Mixtral-8x7b
0.022
0.4
hoopsenk-ilm-67
0.7
Llama 2.70b
Liama 7
Qwen: 72a
Uemmardd
Llama-2-130
gwen
14b
Mistral-7B
Llama-2-130
Uama-2-133
0.2
Llama-2-76
0.5
Llama-2-75
twen
Uemmmna 7b
Uama-2-76
Deepseek-llm-77
Deepseck-lim-11
0.1
0.35
0.60
0.65
0.4
0.55
0.65
0.3
0.40
0.55
(a) NaturalQuestions
(b) ARC-Challenge
(c) MMLU-Math
Figure 9: The correlation between benchmark score and compression. Y-axis is the bench- mark score while x-axis represents BPC. BPC in NaturalQuestions and ARC-Challenge are computed on Common Crawl data, while BPC in MMLU-Math is evaluated on the ArXiv-Math data.
D Exploring the Impact of Different Input Styles on MMLU Evaluation
Given the prominence of MMLU benchmark for LLMs, achieving exemplary scores is paramount. However, concerns arise these practices rely on overfitting. For example, Bi et al. (2024) demonstrates that incorporating multiple choice questions (MCQs) into the
20

---

Published as a conference paper at COLM 2024
Model
Average
Std
6-7B Parameter Model
Deepseek-IIm-7b
0.635
0.002
Falcon-7b
0,650
0.002
Llama-1-7b
0.630
0.003
Llama-2-7b
0.613
0.002
Mistral-7b
0.606
0.002
Qwen-7b
0.646
0.002
Yi-6b
0.639
0.002
13-15B Paramter Model
Llama-1-13b
0.610
0.003
Llama-2-13b
0.582
0.002
Qwen-144
0.620
0.002
30-40B Parameter Model
Falcon-40b
0.594
0.002
Llama-1-30b
0,579
0.002
Yi-34b
0.573
0.002
50-72B Parameter Model
Deepseek-1lm-67b
0.569
0.001
Llama-1-65b
0.559
0.003
Llama-2-70b
0.528
0.002
Mixtral-8x7b
0.560
0.002
Qwen-72b
0.558
0.002
Table 10: Results of Repeated Sampling Common Crawl Dataset. This table presents the average bits per character obtained from three independent samplings. "Std" denotes the standard deviation of the results.
training stage can significantly improve the MMLU accuracy while achieving no perfor- mance gains for tasks not formatted as MCQs. Alzahrani et al. (2024) further observes that some models on the MMLU benchmark is highly sensitive to evaluation details, indicating potential overfitting. Therefore, in our main experiments, we intentionally opted for a less conventional evaluation style, namely the cloze style. The method involves presenting only the question without the options, and subsequently selecting the choice with the highest likelihood. It aligns with the standard way of evaluating ARC-Challenge (Clark et al., 2018), also an MCQ benchmark. By omitting multiple answer options, this approach deviates from the traditional MCQ format, thereby potentially reducing overfitting. Such cloze-style format is just one of the many flavors of evaluating MMLU. And we argue that this method provides fairer assessment of MMLU's utility in gauging intelligence. For thorough analysis of the impact of input styles, we employ the L.M-Evaluation et al., 2023a) framework to evaluate the MMLU with the default style of input. The results, depicted in Figure 10, reveal markedly weaker linear correlation between the scores and compression compared to the cloze-style input. Specifically, as demonstrated in Table 11, the scores for most models increase with the default input format, except Llama-1-7b and Falcon-7b. Notably, the Yi and Qwen series models exhibit substantial score changes, aver- aging nearly 20 within the series. These changes contribute to significant deviation from the linear correlation, indicating potential overfitting on MCQ benchmarks.
the performance
(Gao
E Better Alignment between MMLU and Textbooks
The MMLU benchmark offers a comprehensive evaluation across various domains and tasks, leveraging datasets sourced from real-world examinations and textbooks. As result, textbooks from various educational levels closely align with the benchmark's distribution. We utilize a small collection of textbooks sourced from the internet as validation data. The results are shown in Figure 11. A stronger linear correlation is observed between the
21

---

Published as a conference paper at COLM 2024
Model
Default Style
Cloze Style
A
6-7B Parameter Model
Deepseck-llm-71
49.3
43.1
6.2
Falcon-7b
27.7
41.9
-14.3
Llama-1-7b
35.0
43.1
-8.1
Llama-2-7b
46.6
44.3
2.3
Mistral-7b
63.5
50.3
13.1
Qwen-7b
59.7
44.3
15.4
Yi-6b
64.0
45.8
18.2
13-15B Paramter Model
Llama-1-13b
47.1
46.1
1.0
Llama-2-13b
55.6
47.6
8.1
Qwen-14b
67.9
48.6
19.3
30-40B Parameter Model
Falcon-40b
57.1
48.9
8.2
Llama-1-30b
58.3
50.3
8.0
Yi-34b
76.4
55.0
21.5
50-72B Parameter Model
Deepseek-1lm-677
71.7
54.8
16.9
Jamba-52b
67.1
53.6
13.5
Llama-1-65b
63.6
51.4
12.3
Llama-2-70b
69.4
54.6
14.8
Mixtral-8x7b
71.7
56.5
15.2
Qwen-72b
77.4
54.5
22.9
Table 11: Comparison of different input style on MMLU. We report the average accuracy across all subjects. And we also report A as the difference of score between default style and cloze style.
p: -0.660
0.8
e: 0.098
0,930
0.8
Deepseek-1m-677
Mixtre
Liama 70b
Mistral-7b
Owen-7b
0.7
Llama-1-65bb
0.6
Llama-1-300
Llama-2-11
0.6
Llama-2-70b
Bx7b
Owen-72b
Deepseek-lm-67b
Ulma-2-7b
Deapseck lim.
Llama-1-13
0.5
Llama-14 1-63t
Llama-1-30b
Mistral-7b
Qwen-14b
@wen=7b
Liama-1 --bb
0.4
0.4
Llama-2-76
Llama-1-76
Deepseek-Im-P7
0.3
0.53
0.35
0.57
0.60
0.62
0.65
0.54
0.36
0.58
0.60
0.62
0.64
Figure 10: Results of MMLU with default style of input. The x-axis represents the models' compression efficiency on Com- mon Crawl data, while the y-axis depicts the accuracy of MMLU,
Figure 11: Correlation between MMLU ac- curacy and models' compression efficiency on textbook.
compression performance on the textbook data and the MMLU accuracy, characterized by high Pearson correlation coefficient of -0.930.
22


Reviews:
[{"review_id": "AEtah8v40s", "reviewer": "Reviewer_4KvH", "summary": "This work analyzes the relationship between the ability to compress a text corpora and the ability to solve downstream tasks associated with such corpora. This approximates the relationship between compression (treating an LLM as a compressor) and intelligence (as an ability to solve a task well). Across a wide range of models and scales, the overall message that intelligence and compression have a strong linear relationship is both novel and surprising. This work does an excellent job of dissecting their claims, providing multiple modes of analysis and giving broad context and supporting material.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and encouraging review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The linear relationship is novel, surprising and insightful. This will be extremely discussion worthy at COLM. Other reviewers have asked for impossible comparisons (e.g., to KAN, a paper which dropped literally 3 weeks ago), or to be wowed by some super unsurprising result. I respectfully disagree with all these reviews, and consider this work a solid piece of science which is important to the community. If I could hit the Best Paper recc button, I would. \n\n- The authors elagantly propose multiple modes of analysis which future practitioners can use. The usage of correlation, bit coding and statistical analyses in this paper are exemplars for the community which others should follow. \n\n- Quantifying two very abstract concepts into quantitative measures is useful for further discussion in the community when we discuss what an LM \"knows\" and \"does\".\n\n- A wide range of models are explored to make results more convincing and reliable\n\n- The domains of code, math, and commonsense are sensible and provide reliable bases to easily interpret the findings. The results are communicated excellently.", "reasons_to_reject": "- Nothing major. The authors could have explored very small model sizes to see if these findings extend to GPT2-base or BERT size models. I do not agree with the demands to explore multiple architectures, as many now consider architecture to be very much secondary to data. This paper shows the importance of data, and should be considered much more valuable than evaluation on many architectures.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3DUPAUsjCv", "reviewer": "Reviewer_pVVB", "summary": "The study explores the correlation between data compression and intelligence in the context of LLMs, positing that better compression efficiency correlates with higher intelligence. Experiments on 30 public LLMs across 12 benchmarks showed a near-linear correlation between compression efficiency and intelligence.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and encouraging review! We respond to your comments below.\n\n\n> Considering the rapid pace of innovation in the field of LLMs, is there confidence that the findings would remain applicable to novel architectures such as RWKV, Mamba, and KAN?\n\nWe hypothesize that the findings are agnostic to model architectures, while we did not test non-transformer architectures since there were few non-transformer pretrained models with strong and already-emergent abilities. To verify the findings on non-transformer archs, however, we conduct the experiment on the Jamba-52B model [1], which is based on a hybrid Transformer-Mamba MOE architecture, being the strongest open-source non-transformer pretrained model as far as we know.  The results for Jamba model in the main figure are shown in [this link](https://drive.google.com/file/d/15z6dpF_hGwP5l0w8lXVprFZxp23mm9k9/view?usp=sharing), while we report its BPC and benchmark scores below (according to the COLM rebuttal guideline, we think this is the necessary case to link such an external correlation figure that is easy to read, since the numbers below alone do not reveal much helpful information).  Despite having significantly different architectures, our findings can still be applicable to the Jamba model.\n\n\n\n| Ability                   |  BPC  | Average Benchmark Score |\n| ------------------------- | :---: | :---------------------: |\n| Knowledge and Commonsense | 0.542 |          0.624          |\n| Coding                    | 0.244 |          0.351          |\n| Mathematical Reasoning    | 0.417 |          0.413          |\n| Overall                   | 0.401 |          0.463          |\n\n> Is there an expectation that the correlation between compression efficiency and intelligence persists across different languages or modalities?\n\nThis is a good point. We do expect the correlation between compression efficiency and intelligence persists across different languages or modalities, because the long-standing discussion on their relationship has been based on data in general rather than language only. However, in this paper we focus on the single-language setting and leave further exploration of different languages and modalities as future work.\n\n[1] Lieber, Opher, et al. \"Jamba: A hybrid transformer-mamba language model.\" *arXiv preprint arXiv:2403.19887* (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work provides substantial empirical evidence supporting the hypothesis that superior compression indicates greater intelligence.\n2. The study employs a robust methodology, using a large and diverse set of LLMs and benchmarks to somewhat ensure the generalizability of the findings.\n3. The findings suggest compression efficiency as a practical and unsupervised metric for evaluating LLMs, which can be beneficial for future research and development.", "reasons_to_reject": "1. The definition of intelligence in the paper is limited to performance on specific benchmarks, which might not capture all dimensions of intelligence.\n2. As mentioned in the paper, the study focuses on short- to medium-context scenarios, which may not fully represent the models' capabilities in long-context tasks.\n3. While the paper encompasses a variety of models, it remains uncertain whether its conclusions are applicable to novel or significantly different architectures [1,2,3] not addressed.\n\n[1] Peng, Bo, et al. \"Rwkv: Reinventing rnns for the transformer era.\" arXiv preprint arXiv:2305.13048 (2023).\n[2] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).\n[3] Liu, Ziming, et al. \"Kan: Kolmogorov-arnold networks.\" arXiv preprint arXiv:2404.19756 (2024).", "questions_to_authors": "1. Considering the rapid pace of innovation in the field of LLMs, is there confidence that the findings would remain applicable to novel architectures such as RWKV, Mamba, and KAN?\n2. Is there an expectation that the correlation between compression efficiency and intelligence persists across different languages or modalities?", "extra_scores": {}}, {"review_id": "BPGH0435PJ", "reviewer": "Reviewer_hxZb", "summary": "The work is the first to document the linear correlation between compression and intelligence in LLMs.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and helpful review! We respond to your comments below.\n\n> The claimed linear correlation looks not surprising to me\n\nLM losses have indeed been an important metric, yet its relation to downstream task performance across different LLM families remains unknown. We highlight two novel findings:  \n1. In LLMs, typically, people only compare losses either between different training steps within the same model, or among models within the same family (e.g., llama-7B, 13B). These models share most design and training configurations. However, it is unclear whether comparing losses of LLMs from **different model families** has implications for downstream task results. Our work is the first to examine this. Note that these LLMs differ in tokenizer, architecture, context window size, and training data. We don’t think that it is immediately apparent that the correlation universally holds and is **agnostic** to all these factors, a fact only revealed through our experiments.\n2. The correlation between losses and downstream task scores across different LLM families, if it exists, can take many forms. Our research reveals that it is linear.\n\n> The paper is more on empirical study without proposing new idea.\n\nRight, the paper is more on empirical study, which we believe reveals novel insights. We don’t think that emphasizing empirical study is a weakness of a paper. \n\n> More interesting experiments needed … it would be better if the authors could find something across domains, e.g. how Github data correlates to NQ, MMLU, etc.\n\nFirst, as we already showed in Figure 6, the BPC on Github data is not well correlated with the average score on the knowledge domain that includes MMLU and NQ. \n\nTo add more experiments as requested, we explore cross-ability tasks where multiple abilities are required. We consider solving GSM8K questions through coding, which requires both math and coding abilities. We compute BPC on three corpora to examine their correlation with GSM8K scores: (1) on arxiv data; (2) on Github data; and (3) on the mix of arxiv and Github data. Results are shown below — as expected, the mix of arxiv and Github data demonstrates the strongest correlation, while either arxiv or Github data alone does not correlate well, underscoring that different corpora reflect diverse abilities.\n\n| Corpora | Pearson $\\rho$  | RMSE $e$ |\n| --- | --- | --- |\n| Github alone | -0.742 | 0.137 |\n| ArXiv alone | -0.787 | 0.126 |\n| Github+ArXiv | -0.940 | 0.070 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors find the linear correlation between language model loss (bits per character) and downstream tasks.\n2. The authors have a wide exploration of different LLM son three different domains.", "reasons_to_reject": "1. The claimed linear correlation looks not surprising to me. Language model loss has always been an important metric to justify the quality of LLM. The paper is more on empirical study without proposing new idea,\n2. More interesting experiments needed. It would be better if the authors could explore more domains. And it would be better if the authors could find something across domains, e.g. how Github data correlates to NQ, MMLU, etc.\n3. Figures 1,2,3,4 look duplicated. Figures 2,3,4 can be merged if some information has been shown in Figure 1.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a07usFvtFJ", "reviewer": "Reviewer_Nicz", "summary": "This paper presents an observation that bits per character (BPC) is linearly aligned with the model's capability (average score) on multiple benchmarks. The BPC can be calculated via a variation of cross-entropy loss. Basically, if the model achieves better BPC (lower XE loss) on standalone evaluation sets, the model will perform better on the corresponding benchmarks. The author argues the BPC is a more reliable metrics than the standard benchmark.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful review! We address your comments below.\n\n> So the BPC calculation is basically doing an evaluation on held-out …… If so, this is a standard ML practice. I might see limited novelty in this approach.\n\nYou are right. From a high-level view, this is indeed a standard ML practice. In LLMs, however, people typically compare losses either between different training steps within the same model, or among models within the same family (e.g., llama-7B, llama-13B). Prior to this study, there was no established norm for comparing losses **across different model families**  — for example, comparisons need to account for variations in tokenizers via the BPC metric and unify the context window size to ensure meaningful results. Additionally, it remains unclear whether comparing model losses across various model families — which differ in architecture, tokenizers, context window, and training data— is truly meaningful and has implications on downstream tasks. In this work, we establish a standard practice for making such comparisons, and provide empirical evidence that the relationship between compression and intelligence stands as a universal rule that is agnostic to the specific model and training configurations.\n\n> If the author claims that the BPC of the training set is linearly aligned with the benchmark score on the held-out dataset, I think it will be more interesting.\n\nWhile we cannot perform such experiments due to limited access to the training data, we don’t think the training losses of different models are directly comparable since the training losses are computed on different training data — this is exactly why we need to compute BPC on the same held-out corpus. \n\n> However, as the average of the benchmark score is an established/well-accepted evaluation approach. I don't see the reason we need an additional one\n\nAs an unsupervised evaluation metric, BPC has practical advantages. Specifically, as discussed in the paper, data leakage and overoptimization are major issues for these benchmarks. For instance, using the MIN-K% PROB method for detection, we discover that the Qwen series have inflated scores on math benchmarks. In such cases, BPC serves as a more reliable metric. Also, BPC can easily support dynamic evaluation to update the compression corpora automatically to mitigate the leakage or overfitting issue, and it is easy to collect new domains of corpora. However, it is expensive to update or create new downstream benchmarks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The observation that Bits per character is linearly aligned with the average benchmark performance on the corresponding categories is interesting, which I think might be the major reason to accept this paper.", "reasons_to_reject": "Concretely speaking, this paper uses standalone (held-out) set to evaluate the BPC of the LLM. And BPC is a variation of the Cross Entropy loss. So the BPC calculation is basically doing an evaluation on held-out (validation) datasets (please correct me if I understand incorrectly.) If so, this is a standard ML practice. I might see limited novelty in this approach. If the author claims that the BPC of the training set is linearly aligned with the benchmark score on the held-out dataset, I think it will be more interesting. Unfortunately, as multiple LLM models are close-sourced / only released weights, it is impossible to validate the second (more interesting in my opinion) direction.\n\nThe author also claims the BPC is a reliable evaluation metric, which I agree, given the strong linearly relationship b/w the BPC and the benchmark score. However, as the average of the benchmark score is an established/well-accepted evaluation approach. I don't see the reason we need an additional one.\n\nFurther the author didn't claim / merely claim that BPC is better / more reliable than the benchmark score. However, I didn't see (might miss) the concrete evidence to establish the connection.", "questions_to_authors": "Please address the reasons to reject.", "extra_scores": {}}, {"review_id": "OVzUdHOvjX", "reviewer": "Reviewer_Xsbe", "summary": "The paper examines the relationship between text compression efficiency and the intelligence of large language models. \"Intelligence\" is represented by performance on a selection downstream tasks such as commonsense reasoning and maths. Compression efficiency is measured as bits per character and derived from the conditional probabilities of a model on an unseen text corpora related to the benchmark tasks.\nExperiments with a variety of models and benchmarks demonstrate an almost linear correlation, i.e. a model that compresses text better also performs better on benchmarks.\nThe authors propose the use of compression efficiency as a robust, unsupervised metric for evaluating LLMs, supporting the previous use of perplexity as a metric.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging review! We address your comments below.\n‎\n> The authors did not directly compare text compression to alternative characteristics that could put the findings into more perspective. For example, is compression much more correlated to performance than, for instance, model size or training data size?  \n\nThanks for the advice! We strongly believe that model size or training data size alone does not correlate well with performance across **different model families** — other factors such as model architectures and training data quality will influence the scaling law with model size or training data size as inputs alone. Although this may already be implied, as suggested by the reviewer, in the next revisions we will supplement such comparisons to explicitly show the comparison to give more weight to our findings. \n\n> The importance of the compression aspect isn't entirely to me - as the authors mention, there is a direct link between concepts such as cross-entropy loss, average bits for compression and perplexity. Could the title equally be 'perplexity represents intelligence linearly'? Some clarification around this would help.\n\nThis is a good point. We emphasize compression in this paper due to the longstanding and recently revitalized discussion on the relation between compression and intelligence—a discussion that directly inspired our research. Indeed, there is a direct link between these concepts including losses, perplexity, and BPC. While there are various ways to present our experiments, the reviewer's suggested title, \"Perplexity Represents Intelligence Linearly,\" is also valid. We chose to focus on compression because of its historical significance in discussions about intelligence, reflecting our original motivation and preference. However, presenting our findings from the perspectives of loss or perplexity is feasible as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Interesting and relevant topic: understanding how the capabilities of LLMs relate to other signals is promising both for advancing the theory around language modelling and for practical implications.\n* Strong experimental methodology and analysis; the authors also made good efforts to avoid data contamination and overfitting.\n* The paper is very well-written and easy to follow.", "reasons_to_reject": "* Lack of comparison of compression with alternative measures that could give similar correlations (see question below)\n* Focus on the concept of compression (rather than 'level of surprisal' etc) not very clear (see question below)", "questions_to_authors": "* The authors did not directly compare text compression to alternative characteristics that could put the findings into more perspective. For example, is compression much more correlated to performance than, for instance, model size or training data size? This might already be implied but showing explicitly that other signals cannot predict performance/intelligence as well as compression would give more weight to the findings.\n* The importance of the **compression** aspect isn't entirely to me - as the authors mention, there is a direct link between concepts such as cross-entropy loss, average bits for compression and perplexity. Could the title equally be 'perplexity represents intelligence linearly'? Some clarification around this would help.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the correlation between data compression and intelligence in the context of LLMs, positing that better compression efficiency correlates with higher intelligence. After author rebuttal, it still received mixed scores of 55779. 

On one hand, three of the reviewers are positive about the paper, commenting that (1) the finding in the paper is both novel, surprising, and insightful, (2) the analysis and experiments are well conducted, and (3) the paper is well written. On the other hand, the other two reviewers commented that the claimed linear correlation looks not surprising to me, and more experiments are needed. 

The authors did a nice job of rebuttal, and added additional results of Jamba. Overall, the studied research topic in this paper is quite interesting, and will probably raise meaningful discussions in the conference and the community. The AC thinks that the merits outweigh the flaws, and thus, would like to recommend acceptance of the paper.
Decision: Accept