Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Authors: David Heurtel-Depeiges
Anian Ruoss
Joel Veness
Tim Genewein
Venue: ICML 2025 poster (2025)
Keywords: lossless compression
transformers
multimodal
Primary area: deep_learning

Abstract: Foundation models are strong data compressors, but when accounting for their parameter size, their compression ratios are inferior to standard compression algorithms. Naively reducing the parameter count does not necessarily help as it deteriorates predictions and, accordingly, compression. We conduct a large-scale empirical study to find a sweet spot where pre-trained vanilla transformers can achieve competitive compression ratios. To this end, we train models on 165GB of raw byte sequences of either text, image, or audio data (and all possible combinations of the three) and then compress 1GB of out-of-distribution (OOD) data from each modality. We find that relatively small models (millions of parameters) can outperform standard general-purpose compression algorithms (gzip, LZMA2) and even domain-specific compressors (PNG, JPEG-XL, FLAC) — even when accounting for parameter size. We achieve, e.g., the lowest compression ratio of 0.49 on  OOD audio data (vs. 0.54 for FLAC). We conduct extensive ablations and hyperparameter sweeps to study the impact of model- and dataset scale, and we investigate the effect of unimodal versus multimodal training. We find that even small models can be trained to perform well on multiple modalities,  but unlike large-scale foundation models, transfer to unseen modalities is generally weak.

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
David Heurtel-Depeiges I!
Anian Ruoss" 2
Joel Veness* 2 Tim Genewein
Abstract
ing (Pasco 1977 Rissanen, 1976 Witten al. 1987 Consequently, large pre-trained foundation models, such as LLMs, achieve high data compression on their training distributions and beyond (Delétang et al. 2024). However, when factoring these models' size into the compression ratio, too large models actually perform worse. For this reason, large foundation models with billions of parameters cannot compete with standard compression algorithms such as gzip (Deutsch) 1996) or LZMA2 (Pavlov) (019). The goal of this paper is thus to investigate whether pre-trained vanilla transformers can achieve compression ratios that are competitive with standard algorithms across a range of data modalities. This places fairly tight constraints on the maximal model size, leading us to investigate families of relatively small transformers (with millions of parameters). Note that our aim is not to build practical transformer- based data compressor, as the computational footprint (run- ning time, memory, FLOPs) of even small models is far beyond standard compressors. Instead, studying compres- sion via pre-trained models provides insight into the models' learned inductive biases, e.g., whether they are domain- general, how they depend on the training data composition, and whether there is transfer between modalities.
Foundation models are strong data compressors, but when accounting for their parameter size, their compression ratios inferior standard com- pression algorithms. Naively reducing the param- eter count does not necessarily help as it deterio- rates predictions and, accordingly, compression. We conduct a large-scale empirical study to find a sweet spot where pre-trained vanilla transformers can achieve competitive compression ratios. To this end, we train models on 165GB of raw byte sequences of either text, image, audio data (and all possible combinations of the three) and then compress IGB of out-of- -if-iiiiiiitttionn (OOD) data from each modality. We find that relatively small models (millions of parameters) can outperform standard general-purpose purpose compression algorithms (gzip, I.ZMA2) and even domain-specific com- pressors (PNG, JPEG-XL, FLAC) even when accounting for parameter size. We achieve, e.g., the lowest compression ratio of 0.49 on OOD audio data (vs. 0.54 for FLAC). We conduct ex- tensive ablations and hyperparameter sweeps to study the impact of model- and dataset scale, and we investigate the effect of unimodal versus mul- timodal training. We find that even small models can be trained to perform well multiple modal- ities, but unlike large-scale foundation models, transfer to unseen modalities is generally weak.
Recently, Reléatly.Delétang et al. (2024) stated that "language model- ing is compression". pointing out that log-loss minimization is equivalent to optimizing a lossless compression objec- tive. To illustrate this point, [Deétang et al. (2024) used billion-parameter LLMs that were exclusively trained on text Touvron al.) 2023b Hoffmann et al. 2022 com-
press 1GB of ImageNet images (Russakovsky et al. 2015) and LibriSpeech audio (Panayotov et al.] 2015). respec- tively. They found that these models compress better than gzip or LZMA2 and even domain-speciffi compressors such as PNG [Boutell 1997) or FLAC (Coalson) 2008), but only when parameter counts are not considered. However, when evaluating pre-trained neural networks as compressors, the parameter count has to be factored into the compression ra- tio. To illustrate this, imagine that Alice wants to compress and send data to Bob, who wants to decompress it. If Alice trains a neural network to compress the data but does not communicate the model's weights, Bob cannot decode the data (unless Alice communicates the training recipe and Bob retrains the same network from scratch, but that would no
1. Introduction
Strong predictive models can straightforwardly be turned into strong lossless compressors, e.g., via arithmetic cod-
'Equal contribution Work performed while the author was at Google DeepMind CChandar Research Lab. MILA - Quebec AI Institute Polytechnique Montréal Google DeepMind. Correspon- dence to: Anian Ruoss <anianr google.com>, Tim Genewein <timgen @ www.lc.cooo........
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Text
Images
Audio
Training Data Sources
UTF-8
164H-2
Wikipedia
Bytes
ImageNet
Bytes
Common Voice
Bytes
Unimodal
Multimodal
Training Data Mixtures
165GB
165GB
165GB
165GB
165GB
165GB
165GB
In-modality
Out-of-modality
Out-of-modaiity
In-modaiity
Text
Images
Audio
Evaluation (out-of-distiibuttin)
UTF-8
Fatten
1646-z
Reddit
1GB
CelebA
1GB
Libri
1GB
Speech
Figure 1: Our training and evaluation data pipelines. We consider three modalities: text, images, and audio. From these create training mixtures of 165GB that are either unimodal or multimodal. After pre-training transformers on each of these, we evaluate their compression ratio (factoring in the model size) on all three modalities. If the corresponding modality has not been seen during training, the evaluation is 'out-of-modality" otherwise it is 'in-modaiity'. Importantly, our evaluation is always performed on out-of-distributio data (different from any of the training data sources), even when is in-modality.
longer fall into the pre-trained regime). Accordingly, when evaluating pre-trained neural networks compressors, the parameter count needs to be factored into the compression ratio. For this reason Delétang et al. (2024) also trained scall-scale transformers (up to 3.2M parameters) on 1GB of Wikipedia (Hutter 2006) but found that these models were significantly worse at compressing images and audio.
Main Contributions Our key contributions are:
We conduct a large-scale empirical study (hyperpa- rameter sweeps, ablations) on the compression perfor- mance of small transformers pre-trained on raw byte sequences of text, image, and audio data (and all com- binations), across various model- and dataset sizes.
The obvious open question (see Appendix A is whether small transformers pre-trained
We are the first to show that small pre-trained trans- formers achieve better compression ratios than general- purpose and domain-specific compressors on 1GB of OOD data across different modalities, e.g., 0.49 on audio vs. 0.51 for Bellard (2021) & 0.54 for FLAAC
large (multimodal) datasets can achieve competitive compression ratios across different modalities and whether there is transfer to un- seen modalities, as observed in the large-scale model case. We therefore conduct an extensive empirical study where we train families of decoder-only transformers on 165GB of either text, image, or audio data and all combinations of the three. We then use these models (with frozen pa- rameters, i.e., offline training) to compress 1GB of out- of-distribution (OOD) data from all three modalities (see Fig. [.. We compare against transformers trained purely online, i.e., on the data stream that is being compressed (Bellard 2019: 2 Izacard et al. 2020), for which stor- age/communication of the weights not required for de- compression (unlike our pre-trained models). These online transformers are currently state-of-te of-te-of-the-rt
We show that training multiple modalities only slightly deteriorates the performance each individ- ual modality but significantly boosts the compression ratios on multimodal data, as long as all the evaluation modalities are part of the pre-training data mixture.
We show that small pre-trained transformers fail to beat standard compressors on data modalities that were unseen during training, implying weak out-of-modalle transfer (unlike LLMs, see Delétang et al. (2024)).
the Large Text Compression Benchmark (Mahoney 2006). Overall we find that our small pre-trained transformers achieve competitive compression ratios, consistently outperform domain-general and domain- -spaiiic standard compression algorithms, and are on par with the online transformers from Bellard (202T).
2. Background
Compression and prediction are "two sides of the same coin" (MacKay) 2003). This fundamental duality stems directly from Shannon's celebrated lossless source coding theorem (Shannon 1948), which states that there is a well- defined lower bound for encoding data from a probabilistic
2

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
source. For any data sequence "in == 22 Fn E A of length from a finite alphabet X' sampled from source p: x + (0. 11. a lossless compressor e: X + {0, 1 assigns code c(æ1:)). i.e., a sequence of bits, from which the original sequence is recoverable without loss of in- formation. The goal is to minimize the expected length Lp ;= [x=)]{((xr] by encoding rare sequences with more bits and frequent sequences with fewer bits. Shannon's source coding theorem states that the minimal expected length is lower-bounded by the Shannon entropy of the source: T,p 2 H (p)= == Pa-=p- logz ((x)]-
solved in information theory, with gold-standard algorithms such as Huffman coding (Huffman 1952), arithmetic cod- ing (Pasco 1977 Rissanen 1976; Witten et al. 1987), or
asymmetric numeral systems (Duda 2009). The latter two compress strings online by iteratively converting them into a single binary number with increasing precision (see Dele tang et al. (2024) for an illustration or Chapter 2 in Hutter et al. (2024)). Arithmetic coding is an example of an on- line compression algorithm since it only requires a single pass through the data and compresses on the fly (unlike offiine compressors, such as Huffman coding). Both our models and Bellard (2021), which we compare against, use arithmetic coding and compress online. However, the dif- ference that we pre-train our predictor; i.e., we perform offline training on a dataset and then freeze its parameters (non-adaptive arithmetic coding), whereas [Bellard (2019) 2021) and Izacard et al. (2020) perform online adaptation of the model parameters on the data stream that is being com- pressed (adaptive arithmetic coding). As a result, and unlike our compressors, these approaches do not communicate the trained weights for decompression but only the model archi- tecture and training algorithm (i.e., the compression ratio does not need account for the model parameters).
If the source's 's statistics are unknown, good compression becomes a statistical modeling problem, i.e,, it relies en- tirely on being able to prediet well sequentially. For any predictor ı : X + (0, 1] the expected coding length LG for data drawn from p is at least the cross entropy:
122 2 Ex logg [(²) = Egru [- -oog2 p())
-log2
p(x))
= (p) + (pl(T) 2 (p).
which is also lower-bounded by the Shannon entropy of p. A mismatch between π and p thus leads to an excess length given by their KL divergence. and minimal coding length P across the whole support of p. Accordingly, some Al researchers have argued that compressing well is fundamentally connected to intelli- gence (e.g.. Chaitin's 's famous "Compression is Comprehen- sion" (Chaitin 2006); |Rathmanner & Hutter (001 : Grau- Moya et al. (. and that building universal compressors will accelerate AI development (cf. the Hutter prize (Hutter 2006). an ongoing competition to compress (1GB of) human knowledge). The duality between compression and predic- ieformmttionthhorretee formulation of universal prediction. i.c., Solomonoff induc- tion (Solomonoff] 1964a b; Li & Vitányi 2019), one of two key ingredients for AIXI (Legg & Hutter 2007; Hutter et al. 2024), the theory of artificial superintelligence.
(maximal compression) implies
3. Related Work
Compression Without Transformers Lossless compres- sion with non-transformer) neural predictors has been ex- tensively studied, both via arithmetic coding (Lehtokan- gas et al. 1993;
Schmidhuber & Heil
1994.
1996; Ma
honey 2000 Mikolov| 2012
Knoll, 2014 van den Oord &
Schrauwen 2014 Cox
2016
Schiopu et al., 2018
Goyal
ct al. 2019 Liu et al.
2019
Mentzer et al.
2019
2020
Schiopu & Munteaau 2020
Rhee et al. 2022) and asym-
tion has also led to the (algorithmic)
metric numeral systems (Hoogeboom et al. 2019 Kingma et al. 2019: Townsend et al. 2019 Barzen et al. 2022)
Neural networks are also used for lossy compression, e.g. by overfitting tiny networks data points and transmitting
the weights rather than the data (Dupont et al.
2021 2022
Consequently, Delétang et al. (2024) argue that lossless compression performance lends itself as a comainnggnnerl metric for assessing any predictor's ss quality, including foun- dation models. They further emphasize that foundation models trained by minimizing log-loss (a.k.a., next-token prediction-eeroo or cross entropy loss) are explicitly trained to minimize the expected coding length:
Chen et al. 2021
Ladune et al. 2023
Kim et al. 2024).
Online Transformers Most of the above approaches use separate training set to pre-train models that are then used to compress a test set. Alternatively, the model can also be trained from scratch on the data stream that is being com- pressed (Bellard 2019; 2021 [zacard et al.) 2020: Goyal et al., 2020; Mao et al., 2022). The main advantage of these adaptive online compressors is that they are (quasi) parameterless (since they are initialized from scratch when compressing a data stream), i.e., the model size does not explicitly affect the compression ratio, even for large mod- els (though it implicitly affects the training performance, e.g., large models train more slowly meaning that larger chunks of the data stream are only weakly compressed).
min LG = min E Ex-pl log2 π (xr)
"log loss"
min E moup
-log log2 n ( (e))
The problem of constructing the actual codes that achieve (near) minimal expected length given a predictor is largely
3

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
The transformer-based adaptive online compressor of Bel- lard (2021) is currently state-of-the-art on the Large Text Compression Benchmark (Mahoney] 2006), and Section shows that our best models are on par across all modalities.
of 4096 (bytes), and sliding windows without overlap or memory (full details in Appendix B.3
We always train and evaluate the models with the same context size (4096 by default). We use the Adam optimizer (Kingma & Baj 2015) for 2.5 million steps with a batch size of 32, which, for 165GB of data, roughly corresponds to 2 epochs. Due to the duality of compression and prediction, we minimize the standard (sequential) log-loss (Eq. (T) during training. which is a maximum-com coxxpeum-oomppessionn objective (see Section 2 The model computes distribution over the next byte for every input byte, and the arithmetic coder then uses these predictions to losslessly compress the data. Concretely, the arithmetic coder directly uses the model's predictions over tokens (i.e., the logits) to encode/decode data (see Figure in Delétang et al. (2024) for an overview). Accordingly, no separate head or extraction procedure is necessary. As result, we can train via standard log-loss minimization to perform next-byte prediction. At inference time, we perform standard autoregressive evaluation (using teacher forcing).
Fre-Trained Transformers Most related to our work is the line of research by Valmeekam et al. (2023) Delétang iet al. )2024); Huang et al. (2024); Li et al. (2024); Mittu
let al. ) Chen et al. (2024) on lossless compression via arithmetic coding with pre-trained foundation models, i.e.,
the Llama models (Touvron et al.) 2023a b. Dubey et al.
2024) and Chinchilla (Hoffmann al., 2022). Delétang et al. (2024), in particular, also report good compression rates on unseen modalities (LLMs trained only on text com- press images and audio data well). However, these studies differ from our work as they do not take the model size into account for the compression ratios, except for Delétang et al. (2024), who report both "raw" and "adjusted" compression ratios and find that LLMs are not competitive in terms of adjusted (i.e., the actual) compression ratios. To the best of our knowledge, our paper is the first to systematically investigate the use of appropriately-size sizze pre-trained trans- formers for multimodal lossless compression in a regime where competitive performance w.r.t. standard compression algorithms is possible. In this regime, our study is the most comprehensive in that also investigates multimodal train- ing and cross-modal transfer of pre-trained transformers.
(No) Tokenization Tokenization is a commonly-used, domain-specifee pre-compression step to boost transformers' performance by increasing their vocabulary size to fit more information into their context window (Lester et al. 2024), i.c., increased information density at the cost of increased entropy. However, since we aim to be domain- gennrll,,, we do not use tokenization and instead feed byte streams to our models (we still have to choose how to flatten images/sample audio signals, i.c., minimal domain-specific preprocessing).
4. Methods
We now describe our experimental setup (cf. Appendix B
Evaluation To evaluate performance, we compute the compression ratio (lower is better):
Baselines We compare to various standard compres- sors, both general-purpose, i.e.. gzip (Deutsch) 1996) and LZMA2 (Pavlov] 2019), and domain-specific, i.c.,
compression ratio: := jcompressed datal + |compressor
juscompressed data]
FLAC (Coalson 2008 for audio data and PNG (Boutell 1997) and lossless JPEG 2000 (Skodras al., (001) and JPEG-XL (Alakuijala et al. 2019) for images. Both gzip and LZMA2 (which is used by the 7zip software) are based
which accounts for the model size and is equivalent to the "adjusted compression rate" of Delétang ct al. (2024). We always evaluate on 1GB of out-of-distribution data, i.e., Juncompressed datal 1GB. Like Delétang et al. (2024). we compute the size of the compressor by encoding the model weights with f (2 bytes per parameter) since this level of quantization has little impact on perfor- mance (Tao et al. 2022) and is standard for model inference. As a result, our model sizes range from 0.8MB to 40.3MB. Note that, similar to Delétang et al. (2024), we do not com- press the model parameters, since naive approaches (e.g.. compressing them with gzip) barely decrease the model size (only by around 7%, which corresponds to decrease in compression ratio of only 0.002821 for our largest model). However, as a result, the compression ratio we report is an upper bound, which could be improved by (losslessly) compressing the parameters (though with limited room for improvement in our regime, even in the best case).
on Huffman coding (Huffman 1952) and the Lempel-Ziv. Welch al algorithm (Welch) 1984). ) We use the default parame- ters for gzip, LZMA2, JPEG 2000, and XL, compres- sion level 12 for FLAC, and instruct PNG to find the optimal encoder settings. We also compare to the online transformer from Bellard (202T) with the default v3.3 parameters, which is currently state-of-the-art on the Large Text Compression Benchmark (LTCB) (Mahoney) 2006). We do not compare to Izacard et al. (2020). since it is conceptually very close to Bellard 22021) but worse on LTCB.
Models We focus on decoder-only transformers (Vaswani et al. 2017) with SwiGLU activations (Shazeer 2020) and post-layer normalization. Unless otherwise noted, we use heads, an embedding dimension of 64, a context size

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Baseline
Audio 000 Evalustion
Image 00D Evaluation
Text 00D Evaluation
galp
1.2
LZMA2
Bellard
Compresion Ratio
PLAC (sudio)
limages)
0.8
JPEG 2000 (images)
JPEG XL (images)
0.4
0.0
Amdio
Evailuttion
Audio Text Evaluation
OOD Evallsttao
Audion Image Evaluatton
1.0
Comprezzion Ratio
0.4
0.0
image
aud winageaaeeeeeeeemm toxe
Image
audio
box
andio toxt
Imagen boxt
audio image
Image Movt
image
sudio toxt
audio
mndssimoo Nost
Imagee fest
audio image
sudio toot
audio
text
aud mudiseimgg--too
aud
audio
Figure 2: Small pre-trained transformers domain-geenn general compressors (panels: evaluation data mixtures, bars: training data mixtures). Our method (bars) outperforms standard compression algorithms (horizontal lines) and is par with the online adaptive transformers from Bellard (2021) (blue line) - as long as the evaluation modality is in the training mixture. There is very little cross-modal transfer to unseen modalities (unlike foundation models (Delétang et al. 2024)). Unimodal models are good for their respective modality, but multimodal models perform almost as well across all their training modalities (despite seeing much less data per modality than the unimodal models), i.c., one can trade off a small amount of performance on each individual modality to obtain strong domain gennral compressor via multimodal training (gray bar).
Training Datasets key point of our investigation is to compress
parison), where the compressor is designed with only mini- mal statistical assumptions about the data (domain-specifie compressors make stronger assumptions), we evaluate on unseen, out-of-distribution (OOD) datasets for each modal- ity and not on in-distribution held-out datasets (as common in machine learning). do so, we create a single OOD dataset of 1GB for each modality (details in Appendix B.2]
evaluate how well pre-trained transformers data from different modalities - both if the modality was or was not part of the training data (see Fig. [ We create three different unimodal training datasets with audio, images, and text, and four multimodal training sets (full details in Ap- pendix B.I]. This yields seven 165GB pre- pre-training datasets: (i) 165GB audio; (ii) 165GB images; (iii) 165GB text; (iv) 82.5GB audio and 82.5GB images; (v) 82.5GB audio and 82.5GB text; (vi) 82.5GB images and 82.5GB text; and (vii) 55GB audio, 55GB images, and 55GB text. By train- ing models on all seven data mixtures, we can investigate in-modality and out-of-modality compression ratios. For example, for model trained on text (iii), the in- modality compression ratio given by evaluating on text, while audio image data provide oft-ol-mydality compression ratios.
5. Results
We present our extensive experimental evaluation (see also Appendix C). Unless otherwise noted, we report the best
results over two hyperparameter sweeps (see Appendix(B.3 (i) model- vs. dataset size, and (ii) model- vs. context size.
Small Transformers Are Domain-General Compressors Figure shows the best compression ratio attained on each of the seven out-of- distrii-ittiittoonn evaluation datasets when training a model on each of the seven training data mix-
OOD Evaluation Datasets To mimic the standard com- pression algorithm setting (and thereby ensure a fair com-
5

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Training Mixture
audio
image
text
aud audimmimmage
audio+text
image+text
audio+image+tex
Audio Evaluation
Image OOD Evaluation
Text OOD Evaluation
0.325
Compression Ratio
0.52
0.320
0.29
0.51
0.315
0.50
0,310
0.28
0,305
0.49
50
100
150
50
100
150
50
100 150
Audio GBs Training Mixture
Image GBS Training Mixture
Text GBs Training Mixture
Figure 3: What you see is what you get. Each panel visualizes the compression ratios for one of our modalities when training models varying dataset mixtures and sizes. Although one replace a large proportion of the unimodal training datasets with data from other modalities without incurring significant losses on the original modality (note the scale of the y-axis), transformers (at our tested model sizes) do not exhibit improved transfer from the out- ooo-ooffmotttyy data (i.e., the multimodal models are worse than the unimodal ones, even when trained on much more data from that particular modality). Nevertheless, multimodal training data significantly improves multimodal compression performance (as shown in Fig.
tures (we report the best-performing model from our two sweeps for each training-evaluatinn pair). We observe that pre-trained transformers achieve state-of-the-ee in tn-modality compression ratios, regardless of the concrete composition of the training mixture, outperforming standard compression algorithms (even domain-specific ones) in all cases where all evaluation modalities are part of the training mixture. In these cases, transformers thus learn the modality's 's prototyp- ical statistical patterns during pre- pre-training. Importantly, by multimodal data,
dataset size, and then evaluate as before. Figure 3 shows that, for our models and datasets, there is little transfer between modalities. For all cases of audio, text, and (less clearly) images, it is better to train a smaller unimodal dataset to get the best unimodal performance, as opposed to training on a much larger multimodal dataset. For example, training on a pure text dataset of 33GB (20% of 165GB) outperforms training on a dataset consisting of 82.5GB (i.e., more than twice as much) text and of 82.5GB images/audio.
comparing models trained on unimodal we observe that multimodal training only slightly decreases performance compared to the unimodal models on their respective modalities (despite only having half or a third amount of data from that modality). This means that it is possible to trade off a small amount of performance on each individual modality to obtain a very strong gomain-generni compressor via multimodal training (the gray bar in Fig. (
Scaling Analysis Since there is non-trivial relationship between model- and dataset size, we analyze the scaling of these factors (details in Appendix [B.).. Figure 4 shows trends akin to the scaling laws observed for LLMs (Kaplan et al. 2020), which state that better prediction (in our case compression) is only possible by scaling both models and datasets in a particular way. Unlike traditional scaling laws for models trained on internet-scale datasets, the distribution shift in our evaluation makes it easier for the model overfit to the training distribution (since we always evaluate on OOD data). However, as the number of parameters and the training flops of our small models increase, the adjusted compression ratio improves, eventually beating standard compression algorithms. We do observe gradual overlitting on the image dataset for our models trained only on images. However, this phenomenon can be mitigated by including other modalities in the training mixture (see Fig. Al).
What You See What You Get While Fig. 2 shows that substituting half or two thirds of the training set with data from other modalities only leads to small performance loss compared to the unimodally trained models, it is unclear whether simply training on smaller amount of unimodal data (i.c., decreasing the unimodal training dataset size to, e.g., 82.5GB, and not substituting 82.5GB with another modality) would give the same performance, or whether there is some transfer between modalities (as suggested by Mirchandani et al. (2023)) that compensates for the smaller amount of data per individual modality. To investigate this, we run an ablation where we subdivide each of our seven training sets into 5 different sizes: 20%, 40%, 60%, 80%, and 100% of the respective dataset (uniformly subsampled). We train a series of models (sweeping over their number of layers; see Appendix B33) on each dataset mixture and each
Model Size Context Size The previous two experi- ments investigated the impact of training dataset- and model size, revealing a complex, "scaling lawe-like, relationship between the two and the overall FLOPS training budget. We now investigate the impact of the context window length. Since the context window length has a large impact on the
6

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Audio 0D0 Evaluation
Image ODD Evaluation
Text DOD Evaluation
0.40
0.42
Model Size
0.38
412K
Compression Ratio
0.54
Complession Ratio
Compression Rabio
546K
0.53
679K
0.34
813K
0.36
947K
0.32
0.51
0.34
0.30
0.28
0.49
0.00
0.25
0.50
0.75
1.00
1.25
0.00
0.25
0.50
0.75
1.00
1.25
0.00
0.25
0.50
0.75
1.00
1.25
FLOPS
1e19
FLOPS
1e19
FLOPS
1e19
Figure 4: Scaling training dataset- and model size (for unimodal training and evaluation). Colors indicate the model size; lines correspond to dataset size. We train for epochs regardless of dataset size (i.e., smaller datasets require fewer FLOPS). Increasing the model- and dataset size boosts compression (at the cost of FLOPS). Our OOD evaluation makes models more prone to overfitting (e.g.. our largest image models), making scaling more complex than traditional LLM scaling laws.
Evalustion
Image OOD Evaluation
Text OOD Evaluation
00D Evaluation
0.50
0.515
Model Size Guide
Context Size
Compresion Rato
0.510
1M
Compression Ralio
Compressin Ratio
1024
Comprestion Retio
Audio
0.45
0.28
0.505
20M
8192
0.40
0.500
0:40
0.26
0.35
0.35
0.24
0.30
1
1
5
2
5
FLLPSS
1+19
FLOPS
1w19
FLOPS 1e19
Sliding Window Overisp
Figure 5: Context- vs. model size. Both context size (measure in bytes) and model sizes affect the training compute budget (in FLOPS), leading to non-trivial trade-off. Our results show that this trade-off is highly modality-dependel the effect varies significantly with modality). For text, shorter context sizes and larger models are beneficial (short-term dependencies most important). For images, larger context is generally beneficial, given that single image consists of 512-512- 3 786432 bytes, far exceeding our models' contexts, i.e., models with larger context can process larger fractions of an image at once. For audio, the relationship is complex with intermediate context length and larger models performing better (the reverse is true for short contexts).
Figure 6: Impact of the slid- ing window overlap (for uni- modal training/evaluation)) Increasing the overlap be- tween context windows only marginally improves the per- formance (most significantly for images) but comes at a huge computational cost.
(note the different y-axis scales, i.c.,
overall FLOPS footprint (attention scales quadratically with input sequence length), we also vary the model size to ex- plore whether there is sweet spot in terms of training compute budget allocation (details in Appendix B.3). Fig. shows that the optimal trade-off strongly depends on the data modality. The best models for text have a context win- dow> < 2048 bytes, indicating that short term dependencies are more important than long ones this case. For images, the best compromise overall is to choose a larger context window of 8192, which means decreasing the model size. For audio data, the trade-off is more complex. Overall these results highlight the difficulty of tuning architectures to achieve best performance across many modalities.
early the context window are not conditioned on a lot of data (conditioning on more data should help with prediction and thus compression, according to transformers' in-context learning abilities (Brown et al. 2020 Genewein et al.) 2023 Ge et al.) 2024)). Sliding the context window with more overlap requires more forward passes to process the same amount of data, significantly increasing the computational cost. Figure 5 shows that increasing the overlap window (for a context length of 4096) has relatively little effect. The strongest effect is observed for image data, since 4096 bytes only captures small fraction of an image and there obvious long-range dependencies between image channels.
Evaluation Dataset Size Figure 7 shows the relationship between the compression ratio and the evaluation dataset size for all three modalities and our bert-perforning model
Sliding Window
So far we used a sliding window without overlap to process the evaluation data. Consequently, bytes

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Audio 0OD Evaluation
Image OOD Evaluation
Text OOD Evaluation
Compression Ratio
1.0
Compression Rabio
Compression Ratio
4
0.8
2
0.4
0.6
76-
107
10a
10
107
10
10
107
10a
10
Evaluation Dataset Size (Bytes)
Evaluation Dataset Size (Bytes)
Evaluation Dataset Size (Bytes)
Compressors
-X- FLAC (audio)
X- qzp
-K-
LZMA2
X-
JPEG 2000 (images)
-K- JPEG XL (images)
- PNG (images)
Offline Transformer (ours)
Figure 7: Compression ratio vs. evaluation dataset size. The numerator of the compression ratio consists of the size of the compressed data and the size of the compressor (Eq. (3). For standard compressors (e.g., gzip), the size of the compressor (a few thousand lines of code) is negligible given sufficient evaluation data (i.e., the compression ratio is unaffected by the evaluation dataset size). For neural compressors trained offline (i.e., where the size of the compressor is dominated by the model parameters). the compression ratio improves with increasing data since the model size influence decreases. If the model size 2 evaluation dataset size (e.g., 500M parameters and 1GB of data), one cannot achieve a compression ratio < 1.
(as determined on 1GB of OOD data in Table A2). For offline (i.e., pre-) trained neural compressors, the model parameters are factored into the compression ratio, which means that their compression performance will improve with increasing evaluation data (as long as the model gen- eralizes well to the additional data). In contrast, the size of standard compressors is negligible compared to the amount of evaluation data, which means that their compression ratios are largely unaffected by the evaluation dataset size. FLAC cannot losslessly compress more than 4.2GB of data.
more test data. We chose 1GB of test data as a regime where standard compression algorithms are hard to beat. Moreover, evaluations on more test data, and/or without accounting for model parameters, have been conducted (Delétang et al.) 2024 Valmeekam et al. 2023 Li et al., 2024) (finding sig- nificant cross-domail transfer, unlike our experiments).
Like Xue et al. (2022). we do not use a tokenizer, which has two reasons. First, tokenizers are typically pre-trained per modality, and we want to rule out bad cross-modality transfer resulting from a bad tokenizer. Second, tokeniza- tion acts "pre-compression" step (cf. Delétang al. (2024)) This pre-compression increases information den- sity in the context window at the cost of increasing entropy, which can make the prediction problem harder: Lester et al. (2024) even show that when using a strong neural- compeesers (together with arithmetic coding) to train LLMs, training performance can collapse catastrophically.
6. Discussion
Our main goal is to investigate whether pre-trained trans- formers can compete with standard compressors, even when taking their parameter size into account. In contrast to previous work, this places our models into a relatively small regime, where it is unclear whether models will learn well from large datasets at all and have non-trivial out-of- distribution and mross-modality transfer. One could try to train larger models and compress the model parameters themselves, but we chose not to do this since naive lossless compression of model parameters leads to S 10% reduction (see Table A4 and even best-case scenarios would only lead to marginal improvements in compression ratio given the size of our models. For very large (e.g., foundation) models, compressing weights to achieve competitive com- pression ratios may be interesting, though it would require lossy compression (Tao et al. 2022). leading to non-trivial trade-offs between high (lossy) compression and maintain- ing strong predictor performance (the two summands in the numerator of Eq. (3) Exploring these trade-off's is an inter- esting direction for future research but beyond the scope of our work. Another option for larger models is to evaluate
Limitations All our claims regarding the universality of our compressors (or the lack thereof) are limited to the model size, regime, and the particular modalities and datasets we studied. We cannot rule out that there are cases where even in in-modality transfer is weak (e.g., using another OOD image evaluation dataset with very different statis- tics), or that there are cases of non-trivial cross-modal trans- fer (which we have not observed). We did not investigate transfer learning approaches to improve the out-of-modality performance of our neural compressors, but we consider this an interesting avenue for future work. Our claims re- garding outperforming standard compression algorithms are limited to our experiments. We cannot rule out that there are datasets where no pre-trained transformer outperforms, e.g., LZMA2 (in fact, we think its plausible that such datasets can be constructed synthetically). Moreover, we cannot rule out
8

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
that more sophisticated architectures (e.g., Perceivers (Jae- gle et al. ),21) MegaByte Yu al. 2023), or Byte Latent Transformers Pagnoni et al. 2024g) would outperform
Acknowledgments
We thank Anna Mitenkova, Benjamin Beyret, Emilien Dupont, Daniele Calandriello, Grégoire Delétang, Jordi Grau-Moya, Li Kevin Wenliang, Laurent Orseau, Mar- cus Hutter, Matthew Aitchison, Sarath Chandar, Satin- der Baveja, Theophane Weber, Zhengdong Wang, Zoubin Ghahramani. and the anonymous reviewers for their helpful feedback and insightful discussions.
our models, and we consider neural architecture optimiza- tion for lossless compression an interesting direction for future research. Finally, our goal is not to build practical transformer-bbsed universal compressors to compete with standard compressors in terms of computational footprint. As Table A3 shows, models are orders of magnitude slower for encoding data (with significantly larger memory- and FL.OPS-demande and s 3x slower than Bellard's ss on- line adaptive transformer. When decoding data with our models, which has to be done token-by-toker to obtain the correct conditioning, our running time even worse.
References
Alakuijala, J., van Asseldonk, R., Boukortt, S., Bruse, M.,
Comşa, L-M.,, Firsching, M., Fischbacher, T., Kliuch- nikov, E.,, Gomez, S., Obryk, R., Potempa, K., Rhatush- nyak, A., Sneyers, J., Szabadka, Z., Vandevenne, L., Ver- sari, L., and Wassenberg, JPEG XL next-generation image compression architecture and coding tools. In Applications of Digital Image Processing XLII, 2019.
Future Work Even our largest models that were trained on multiple different modalities failed to achieve out-of- modality transfer (unlike LLMs, which do achieve cross- modal transfer when used as compressors, albeit using the "unajdusted" compression ratio; see Delétang et al. (2024)). Accordingly, investigating more sophisticated training strate- gies to improve cross-modal transfer, e.g., transfer learning, presents an interesting direction for future work.
Ardila, R., Branson, M., Davis, K., Kohler, M., Meyer, J.,
Henretty, M., Morais, R., Saunders, L.,, Tyers, F. M., and Weber, G. Common voice: A mussively-multilinge speech corpus. In LREC, 2020.
Barzen, B. L. C., Glazov, F., Geistert, J., and Sikora, T. Accelerated deep lossless image coding with unified par- alleleized GPU coding architecture. In PCS, 2022.
7. Conclusion
Bellard, F. Lossless data compression with neural networks.
We showed that pre-trained vanilla transformers are com- petitive "zero-shot" compressors on OOD evaluation data, achieving better compression ratios than domain -gnneal and domain-specifie standard compression algorithms. We showed this for text, images, and audio data, and for all pos- sible combinations of the three- as long as the correspond- ing modalities are in the training mixture. Despite their rela- tively small size, our models compress well across multiple modalities, without losing much performance compared to a purely unimodal model. On the other hand, multimodal training does not lead to strong compression performance on unseen modalities. This is in contrast to LLMs (Delétang let al. 2024), indicating a qualitative difference between small and (very) large models, even when the small models are trained on large datasets. Our results suggest that small transformers can be pre-trained to exploit statistical regular- ities on par or better than standard compressors and current state-of-the-an adaptive online neural compressors, but we do not observe the emergence of universal compression.
Technical report, Amarisoft, 2019.
Bellard, F. NNCP v2: Lossless data compression with
transformer. Technical report, Amarisoft, 2021.
Boutell, T. PNG (portable network graphics) specification version 1.0. RFC, 1997.
Brown, T. B., Mann, B.,, Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In NeurlPS, 2020.
Chaitin, G. J. The limits of reason. Scientific American, 2006.
Chen, H., He, B., Wang, H., Ren, Y., Lim, S., and Shri- vastava, A. Nerv: Neural representations for videos. In NeurlPS, 2021.
Impact Statement
This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, K. Zhang, P., Liu, H., Liu, J., Liu, Y., Huang, J., Wang, S., Yan, H., and Li, H. Large language models for lossless image compression: Next-pixel prediction in language space all you need. arXiv:2411. /2.4444 2024.

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Coalson, J. Free lossless audio codec, 2008. URL https; flac
Genewein, T., Delétang, G., Ruoss, A., Wenliang, L. K., Catt, E.,, Dutordoir, V., Grau-Moya, J., Orseau, L., Hutter, M., and Veness, J. Memory-based meta-learning non- stationary distributions. In ICML, 2023.
Cohan, A., Dernoncourt, F,, Kim, D. S., Bui, T., Kim, S., Chang, W., and Goharian, N. A discourse-awert aware attention model for abstractive summarization of long documents. In NAACL-HLT (2). 2018.
Goyal, M., Tatwawadi, K., Chandak, S., and Ochoa, I. Deepzip: Lossless data compression using recurrent neu- ral networks. In DCC, 2019.
Cox, D. Syntactically informed text compression with re- current neural networks. arXiv: 2016.
Goyal, M., Tatwawadi, K., Chandak, S., and Ochoa, I. Dzip: general-purpose purpose lossless compression based on
Improved general novel neural network modeling. DCC, 2020.
Delétang, G., Ruoss, A., Duquenne, P., Catt, E., Genewein, T., Mattern C., Grau-Moya, J., Wenliang, L. K., Aitchi- son, M., Orseau, L., Hutter, M., and Veness, J. Language modeling is compression. In ICL.R 2024.
Grau-Moya, J., Genewein, T., Hutter, M., Orseau, L., Delé- tang, G., Catt, E., Ruoss, A., Wenliang, L. K., Mattern, C., Aitchison, M., and Veness, J. Learning universal predictors. In ICML, 2024.
Deutsch, P. GZIP file format specification version 4.3. RFC, 1996.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A.. Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. Training compute-optimal large language models. arxiv:2203. 155566 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A.,
Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku-
mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B.. Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C.,, Keller, C., Touret, C., Wu, C., Wong. C., Ferrer, C. C., Nikolaidis, C., Al- lonsius, D. Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia
Hoogeboom, E.. Peters, J. W. T., van den Berg, R., and Welling, M. Integer discrete flows and lossless compres- sion. In NeurIPS, 2019.
D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, M., Radenovic, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Nail, G.. Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I. M., Misra, I., Evtimov, I., Copet, J., Lee, J., Geffert, J. Vranes, J.. Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang. J., Liu, J., Wang. J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Al- wala, K. V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., and et al. The llama 3 herd of models. arXiv:2407.21783, 2024.
Huang, Y., Zhang, J., Shan, Z., and He, J. Compression represents intelligence linearly, arXiv:2404, 09937, 2024.
Huffman, D. A. A method for the construction of minimum- redundancy codes. Proceedings of the IRE, 1952.
Hutter, M. 500'0000 prize for compressing human knowl- edge, 2006. URL http: httt:://ppiree.httetl.eeee.
Hutter, M.. Quarel, D., and Catt, E. An Introduction to Universal Artificial Intelligence. Chapman & Hall, 2024.
Izacard, G., Joulin, A., and Grave, E. Lossless data compression with transformer, 2020. URL https!
poppenreviewnet/fooooe //openreview. net/ ftrtt/forum2a00 Hyy Hygi/ 7xStvS
Duda, J. Asymmetric numeral systems. arXiv:0902.0271, 2009.
Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J. Perceiver: General perception with iterative attention. In ICML, Proceedings of Machine Learning Research, 2021.
Dupont, E., Golinski, A., Alizadeh, M., Teh, Y. W., and Doucet, A. COIN: compression with implicit neural representations. arXiv:2103.03123, 2021.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv:2001.0836 2020.
Dupont, E., Loya, H., Alizadeh, M., Golinski, A., Teh, Y. W., and Doucet, A. COIN++: neural compression across modalities. Trans. Mach. Learn. Res., 2022.
Kim, H., Bauer, M., Theis, L., Schwarz, J. R., and Dupont, E. C3: high-performance and low-complexity neural compression from a single image or video. In CVPR, 2024.
Ge, T., Hu, J., Wang, L., Wang, X., Chen, S., and Wei, F. In-context autoencoder for context compression in large language model. ICLR, 2024.
10

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Mikolov, T. Statistical Language Models Based on Neural Networks. PhD thesis, Brno Universtiy of Technology, 2012.
Kingma, F. H., Abbeel, P., and Ho, Bit-swap: Recursive bits-back coding for lossless compression with hierarchi- cal latent variables. In ICML, 2019.
Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models as general pattern machines. In CoRL, 2023.
Knoll, B. CMIX, 2014.
URL http://wwi
byronknoll cwmmcoo/c//oo..oo. cmi x. html
Mishra, S., Khashabi, D., Baral, C.,, and Hajishirzi, H. Cross- task generalization via natural language crowdsourcing instructions. In ACL (1), 2022.
Ladune, T., Philippe, P., Henry, F., Clare, G., and Leguay, T. COOL-CHIC: foordinate-basee low complexity hierar- chical image codec. In ICCV, 2023.
Mittu, F., Bu, Y., Gupta, A., Devireddy, A., Ozdarendeli, A. E., Singh, A., and Anumanchipalli, G. Finezip Push- ing the limits of large language models for practical loss- less text compression. arXiv:2409. 171441 2024.
Legg. S. and Hutter, M. Universal intelligence: A definition of machine intelligence. Minds Mach., 2007.
Lehtokangas, M., Saarinen, J., Huuhtanen, P., and Kaski, K. Neural network optimization tool based on predictive MDL principle for time series prediction. In ICTAI, 1993.
Pagnoni, A., Pasunuru, R., Rodríguez, P., Nguyen, J., Muller, B., Li, M., Zhou, C., Yu, L., Weston, J., Zettle- moyer, L., Ghosh, G., Lewis, M., Holtzman, A., and Iyer, S. Byte latent transformer: Patches scale better than tokens. CoRR, abs/2412.09871, 2024.
Lester, B., Lee, J., Alemi, A., Pennington, J., Roberts, A., Sohl-Dickstein, J., and Constant, N. Training Iims over neurally compressed text. arxiv:2404.03626, 2024.
Li, M. and Vitányi. P. M. B. An Introduction to Kolmogorov Complexity and Its Applications, 4th Edition. Springer, 2019.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: An ASR corpus based on public domain audio books, In ICASSP, 2015.
Li, Z.., Huang, C., Wang. X., Hu, H., Wyeth, Bu, D., Yu, Q., Gao, W., Liu, X., and Li, M. Understanding is
Pasco, R. C. Source coding algorithms for fast data com- pression (ph.d. thesis abstr.). IEEE Trans. Inf. Theory, 1977.
compression. arxiv:2407.0772. 2024.
Liu, Q., Xu, Y., and Li, Z. DecMac: A deep context model for high efficiency arithmetic coding, In ICAIIC, 2019.
Pavlov, I. 7z Format, 2019. URL http:// www. 7 zip org/7z 72 html
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In ICCV, 2015.
Pot, E., Mohiuddin, A., Ruyssen, P., Michalski, M., Simsa, R. S. and Wicke, M. TensorFlow Datasets, a collection of ready-to-use datasets, 2019. URL https: ///w: /www
MacKay, J. C. Information theory, inference, and learn- ing algorithms. Cambridge University Press, 2003.
Mahoney, M. V. Fast text compression with neural networks. In FLAIRS, 2000.
Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C.,, and Lillicrap, T. P. Compressive transformers for long-range sequence modelling, In ICLR, 2020.
Mahoney, M. V. Large text compression benchmark, 2006. URL httpp::/wwwt www.wa.attahhoee...... n Cc/text.oo html
Rathmanner, and Hutter, M. A philosophical treatise of universal induction. Entropy, 2011.
Mao, Y., Cui, Y., Kuo, T., and Xue, J. TRACE: A fast transformer-basse general- pprrpose lossless compressor. In WWW, 2022.
Rhee, H., Jang, Y. I., Kim, S., and Cho, N. I. LC-FDNet: Learned lossless image compression with frequency de- composition network. In CVPR, 2022.
Mentzer, F., Agustsson, E., Tschannen, M., Timofte, R., and Gool, L. V. Practical full resolution learned lossless image compression. In CVPR, 2019.
Rissanen, J. Generalized kraft inequality and arithmetic coding. IBM J. Res. Dev. 1976.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. S., Berg, A. C., and Fei-Fei, L. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 2015.
Mentzer, F., Gool, L. V., and Tschannen, M. Learning better lossless compression using lossy compression. In CVPR, 2020.

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Schiopu, I. and Munteanu, A. Deep-leaming-basee lossless image coding. IEEE Trans. Circuits Syst. Video Technol., 2020.
A., Kuan, J. X., Xu, P., Yan, Z.., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.099: 307.0988.. 2023b.
Schiopu, I., Liu, Y., and Munteanu, A. CNN-based predic- tion for lossless coding of photographic images. In PCS, 2018.
Townsend, J., Bird, T., and Barber, D. Practical lossless compression with latent variables using bits back coding. In ICLR (Poster), 2019.
Schmidhuber. J. and Heil, S. Predictive coding with neural nets: Application to text compression. In NIPS, pp. 1047- 1054. MIT Press, 1994.
Valmeekam, S. K., Narayanan, K., Kalathil, D., Cham- berland, J., and Shakkottai, S. Llmzip: Lossless text com- pression using large language models. (XXv:23306040000 2023.
Schmidhuber, J. and Heil, S. Sequential neural text com- pression. IEEE Trans. Neural Networks, 1996.
van den Oord, A. and Schrauwen, B. The student-t mixture natural image patch prior with application image compression. J. Mach. Learn. Res., 2014.
Shannon, C. E. A mathematical theory of communication. Bell Syst. Tech. J., 1948.
Sharma, E., Li, C., and Wang, L. BIGPATENT: A large- scale dataset for abstractive and coherent summarization.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS, 2017.
In ACL (1), 2019.
Shazeer, N. 202.:220002.0200,, 2020.
GLU variants improve transformer.
Völske, M., Potthast, M.. Syed, S., and Stein, B. TL;DR:
Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Sum- marization, 2017.
Skodras, A., Christopoulos, A., and Ebrahimi, T. The JPEG 2000 still image compression standard. IEEE Sig- nal Process. Mag., 2001.
Wang. Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., Pathak, E., Karamanolakis, G., Lai, H. G., Purohit, I., Mondal, I., Anderson, J., Kuz- nia, K., Doshi, K., Pal, K. K., Patel, M., Moradshahi, M., Parmar, M., Purohit, M., Varshney, N., Kaza, P. R., Verma, P., Puri, R. S., Karia, R., Doshi, S., Sampat, S. K., Mishra, S., A, S. R., Patro. S., Dixit, T., and Shen, X. Supper-naturalinsttuttiio Generalization via declarative instructions on 1600+ NLP tasks. In EMNLP, pp. 5085- 5 5109. Association for Computational Linguisties, 2022.
Solomonoff, R. J. A formal theory of inductive inference. part I. Inf. Control., 1964a.
Solomonoff, R. A formal theory of inductive inference. part II. Inf. Control., 1964b.
Tao, C., Hou, L., Zhang, W. Shang, L., Jiang, X., Liu, Q., Luo. P., and Wong, N. Compression of generative pre- trained language models via quantization. ACL (1), 2022.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. arXiv:2302. 3971, 2023a.
Welch, T. A. A technique for pression. Computer, 1984.
data com-
Wikimedia. Wikimedia downloads, 2023. URL https
//dumps.wwi www.wediaiii......mmmm.m........ org
Witten, H., Neal, R. M., and Cleary, J. G. Arithmetic coding for data compression. Commun. ACM, 1987.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi,
A.. Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Feeree, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J. Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Sub- ramanian, R., Tan, X. E.,, Tang, B., Taylor, R., Williams,
Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., and Raffel, C. Byt3: Towards a token-free future with pre-trained byte-to-byte models. Trans. Assoc. Comput. Linguisties, 2022.
Yu, L., Simig, D., Flaherty, C., Aghajanyan, A., Zettlemoyer, L.,, and Lewis, M. MEGABYTE: predicting million- byte sequences with multiscale transformers. In NeurIPS, 2023.
12

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Table AI: Similarities and differences of our work w.r.t. Bellard 2021) and Delétang et al. 2024). Unlike Bellard (022))
we use offline (i.e., pre-) training and consider multimodal data. Online training imposes different constraints on the optimal model size, since the compression ratio does not have to account for the model parameters, which is why |Bellard (202T) were able to study somewhat larger models. We also conduct comprehensive scientific ablations over dataset-, model-, context-size, etc. Unlike Delétang al. (2024), (only) consider the adjusted (i.e., the actual) compression ratio, which includes the model parameters, and therefore, cannot use massive foundation models. Moreover, while Delétang et al. (2024) studied the cross-domain transfer of foundation models trained on text, they did not study multimodal training.
Hellard 2021
Delétang et al. 2024
Ours
# of Parameters
187M
70B
20M
Training
online
offiine evaluation
offline training & evaluation
Multimodal
Actual (Adj.) Compression Ratio
X
Model Size
Context Size
Tokenizer
Ablations
Data Mixture
Train Dataset Size
Evaluation Dataset Size
Sliding Window Overlap
A. Relation to Prior Work
As stated in Section I and shown in Table we study the open question of whether small transformers pre-trained on multimodal data can achieve competitive compression ratios across different modalities and whether there is transfer to unseen modalities (as observed in the large-scale model case). Consequently, the fact that they can do so cannot be predicted from the strong compression performance of L.MSs (e.g., results by Delétang et al. (2024)).
In contrast to Bellard (02T), we use offline adaptive arithmetic coding, which induces entirely different constraints on the optimal model size (since the model parameters do not have be factored into the compression ratio). Bellard (202T) relies on "test-time" gradient updates on in-distribution data, whereas we pre-train models and then leverage in-context learning on cuttof-diitribution data. It is, therefore, impossible to conclude from the results in [ellard (202T) whether results are possible. The training data and protocol are incomparable, making it all the more interesting that our results are generally very close- another surprising finding that cannot be trivially explained.
In contrast to Delétang et al. (2024), we do not evaluate off- thell, large-scale, text-based foundation models but pre- pre-train small-scale transformers on audio, image, and text data. As a result, the compressors proposed by Delétang et al, (2024) are not competitive w.r.t. standard compressors (which they acknowledge) unlike our models (we beat standard compressors across the board). Delétang et al. (2024) do conduct pilot experiment by pre-training a small transformer on text data, but, in contrast to our work, they do not perform (i) a comprehensive study of multi- modal training. (ii) nut-of-distribution evaluation, (iii) ablations over context-, dataset-, and model-size, (iv) comparison to Bellard (2021). (v) an investigation of the sliding window overlap, and (vi) a test dataset size ablation. As our work shows, all of these ingredients are necessary to obtain the strong performance we reported, and, as a result, Delétang et al. (2024) do not manage to pre-train a transformer- based compressor that beats standard compressors across multiple modalities - unlike our work. The main contribution of Delétang et al. (2024) is conceptual, with relatively simple experimental evaluation to illustrate the conceptual arguments. Our work, on the other hand, performs rigorous and comprehensive empirical study.
B. Experimental Details
B.1. Training Data Sources
We source all of our data from the following open-source TensorFlow datasets (oot et al. 2019
Text Since most of TensorFlow's ' text datasets are quite small, we concatenate the following five datasets into a single collection of 165GB: (i) Wikipedia (Wikimedia] 2023), the filtered UTF-8 encoded text from an XMI. XMI.dump from 2023-06-01,
13

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
containing all languages but predominantly English and western languages (113.9GB); (ii) PG- 19 RRae et al. 2020), books from the Project Gutenberg, also encoded in UTF-8 (9.4GB); (iii) Big Patent (Sharma al. 2019), dataset of patents in English (30.2GB); (iv) Scientific Papers (Cohan et al. 2018), from arXiv and PubMed, containing the raw text including the LaTeX code (8.1GB); and (v) Natural Instructions (Mishra et al. 2022
Wang et al. (022). tasks formulated in English
covering different domains and lanugages (B.1GB).
Image We collect subset of 165GB of the ImageNet dataset (Russakovsky et al. 2015), uniformly sampled across the 1000 classes, which contains 14 197 122 annotated images (of varying resolutions) from the WordNet hierarchy. We decode the images into RGB arrays (three uint& 8 channels), flatten them, and concatenate them into a byte stream of flattened images. As a consequence, we ignore image boundaries when sampling from this data source (i.e., sequences are not guaranteed to start or end at the start or end of an image).
Audio We create a subset of 165GB from the Common Voice dataset (Ardila et al. 2020). multilingual dataset of voice recordings. We downsample the dataset from 48 kHz to 16 kHz and encode the waveform as int 6, i.e., with two bytes per sample. As for images, we concatenate all individual audio samples into a single byte stream. Accordingly, there no guarantee that a sequence sampled from our dataset starts or ends the beginning of a recording.
B.2. nnstr--Disttibution Evaluation Data Sources
We source all of our data from the following open-source TensorFlow datasets (Pot et al. 2019
Text
We use 1GB subset of Reddit (Völske et al. 2017), which contains 3.8 million Reddit posts encoded in UTF-8.
Images We create 1GB subset of the CelebA HQ dataset (Liu al. 2013) with resolution of 512 x 512. We process the images in the same way as for our image training set, i.e., flattening and concatenation, and we subsample uniformly across classes of CelebA
Audio We use 1GB from the LibriSpeech (Panayotov et al. 2015) dataset, which contains roughly 1000 hours of English speech data derived from audiobooks that have been segmented and aligned in the LibriVox project. The data is already in 16kHz (with a sample size of 2 bytes), and we simply concatenate samples into a single byte stream.
Multimodal Evaluations For our evaluations multimodal data, described above and average the results accordingly (both for models but also all standard compression algorithms, and Bellard's online adaptive transformer), either over two or three evaluations depending on the evaluation mixture composition.
the unimodal evaluations 1GB of data as
Sweeps
Model Size vs. Dataset Size The experiment to investigate the impact of training dataset- and model size, with results shown in Fig. 4 used the following model parameters. Dataset sizes were 20%, 40%, 60%, 80%, and 100% of the full 165GB for each training set mixture (uni- and multimodal). All models used a context size of 4096, 8 attention heads per layer, a widening factor of 4 and the number of layers was either 2, 4, 6, 8, or 10. Models were trained with a batch size of 32. The learning rate was 10-4, and a sinusoid positional encoding was used.
Model Size vs. Context Size Fig. 5 in the main paper shows the relationship between context length and model size. For this experiment we performed a large-scale sweep with the goal of covering a good range of training FLOPS budget with models that make various trade-off's between model size and context length (given the same model size, compute demand increases with inereasing context length). The main question was whether there a qualitatively similar relationship across parameters, and whether there is a clear sweet spot- see the main paper for results and discussion. For our sweep we used the same model parameters as in the previous paragraph (the training data size was always at 100%) and sweep over the following four context sizes (with training batch size in brackets): [1024 (128), 2048 (64), 4096 (32), 8192 (16) For each context size we train five models (XS, S, M, L. and XL) on all three unimodal datasets, respectively. Each model has a different combination of embedding dimension and number of layers for each different context size. The XS models have embedding dimensions [112, 96, 80, 64] and numbers of layers [11, 7, 5, 3] for the different context sizes respectively (i.e., wider and deeper models for shorter contexts and more narrow and more shallow models for long context size).
14

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Table A2: Best compression ratios for each compressor. This table shows the same results as Fig. 2 but as precise numerical
values to facilitate detailed comparison.
Out-of-Distriuune Compression Ratio
Evaluation Modality
Ours
Bellard
gzip
1.ZMA2
FLACC
PNG
JPEG 2000
JPEG--X
Audio
0.487
0.509
0.813
0.699
0.538
Image
0.285
0.281
0.698
0.545
0.426
0.390
0.260
Text
0.217
0.204
0.394
0.286
Audio + Image
0.393
0.395
0.756
0.622
Audio Text
0.362
0.357
0.604
0.493
Image- Text
0.270
0.243
0.546
0.415
Audio + Image Text
0,349
0.331
0.635
0.510
Table A3: Running times compress 1GB of data for all compressors used in study. We the best model per modality, which have different sizes and thus different running times.
Running Times [s]
Evaluation Modality
Ours
Bellard
gzip
LZMA2
FLAC
PNG
JPEG 2000
JPEG-XL
Audio
305 609
101 178
169
Image
222 065
103 391
47
436
174
495
99
871
Text
452
100
102
881
184
The models have embedding dimensions [192, 160, 112, 96] and numbers of layers [10, 8, 6, 6, The M models have embedding dimensions [224, 192, 144, 112] and numbers of layers [12, 9, 7, The L models have embedding dimensions (272, 240, 176, 144] and numbers layers [13, 10, 8, 5). The XL models have embedding dimensions [320, 304, 240, 160) and numbers of layers [12, 9, 7, 6]. The main goal with these settings is create families of models that have roughly the same demand in terms of FLOPS (iso-FLOPS) but very different trade-offs in terms of model- and context size.
B.4. Computational Resources
We trained every model on 16 NVIDIA A100 GPUs from our internal cluster. We trained 315 models in total, yielding computational footprint of 5040 A100s. We ran Bellard's code NVIDIA GeForee RTX 4090 GPU with 24-core Intel 19-13900KF CPU @ 3Ghz.
C. Additiona Results
C.1. Compression Ratios
Table shows the optimal compression ratios that each of the compressors achieve on the different evaluation modalities (note that all evaluations are on out-of-disrriutiion data). The same values as shown Fig. 2 in the main paper and given here precise numerical values for completeness.
C.2. Running Times
Computing the FLOPS for standard compressors (e.g., gzip or PNG) is much more involved than for neural networks and requires intricate knowledge of the algorithm, which can consist of more than 100K lines of code (standard compression algorithms have also been highly optimized and run CPU, whereas the neural compressors need GPU). Therefore, instead compare the wall-clock running times in seconds when compressing GB of data from each of the three modalities for our models, Bellard's online adaptive transformer (Bellard 2021), and the standard compression algorithms used in our work. As Table clearly shows, our models and Bellard's 's model are orders of magnitudes slower (let alone the increased computational demand and GPU requirements). Running times for our models differ, because we pick the best model per modality, which are models of different sizes.
15

---

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Table A4: Compression ratios for model parameters. We losslessly compress the trained model parameters with standard compressors. For each modality we choose the best- bert-perf model. As is shown, the maximal compression is 1 11%, which would affect the overall compression ratio on the corresponding evaluation data only very marginally.
Model Parameter
Compression Ratio
Evaluation Modality
gzip
LZMA2
Audio
0.93
0.90
Image
0.93
0.90
Text
0.92
0.89
Audio OOD Evaluation
Image OOD Evaluation
Text OOD Evaluation
0.50
0.40
Model Size
Compression Ratio
0.59
Compression Ratio
0.46
Compression Ratio
0.38
412K
0.36
679K
0.57
0.42
813K
0.34
947K
0.55
0.38
0.32
0.53
0.34
0.30
0.00
0.25
0.50
0.75
1.00
1.25
0.00
0.25
0.50
0.75
1.00
1.25
0.00
0.25
0.50
0.75
1.00
1.25
FLOPS
1e19
FLOPS
1e19
FLOPS
1e19
Figure A1: Similar Fig. 4 in the main paper, but here the models trained on uniform mixture over all three modalities (55GB per modality). The plot shows compression performance evaluated the unimodal datasets as training progresses for various model- and training set sizes (models are different colors, each line is different training set size of either 20%, 40%, 60%. 80%, and 100%). We always train for 2 epochs, regardless of dataset size, i.e., smaller datasets require fewer FLOPS. In contrast to Fig. 4 where models are trained on unimodal data, we observe no overfitting, e.g., on images, even for the largest models tested. However, the compression ratios are slightly worse than for unimodal training, which line with our other expriments that show small losses when training on multimodal data.
C.3. Compressing Model Parameters
Throughout our paper we report compression rates that take uncompressed model parameters into account. As diseussed in the main paper, compression ratios could be improved by also compressing model parameters. However, as Table A4] shows, naively compressing model parameters with lossless compressor does not lead to much compression, which would translate into very marginal gains on the overall compression ratio. While it is possible to investigate more sophisticated compression schemes, in particular lossy compression of network weights (though this opens the problem of having to solve a trade-off between increasing weight compression and maintaining compression performance), this is beyond the scope of our paper. Accordingly, our compression rates can be understood (somewhat) conservative estimates that give (in our case fairly tight) upper bounds on compression performance. Compressing network weights achieve competitive compression ratios would be of greater significance in regime where models are significantly larger than ours (but the evaluation data stays roughly at the same size).
C.4. Scaling Analysis for Multimodal Training
Fig. A1 shows the results of simultaneously scaling dataset- and model size across training. In contrast to the similar Fig. in the main paper, where models were trained on unimodal data, Fig. A1 shows models trained on multimodal data (i.e., the uniform mixture across all three modalities, with 55GB per modality). The multimodal training mixture acts as a regularizer, which can clearly be seen by the lack of overfitting of the largest models on images. Compare this against the unimodal training results in Fig. where overfitting can be observed. line with our other main results Fig, 2 and Fig. S the overall compression ratios are slightly worse for the models trained on multimodal data compared unimodal training.
16


Reviews:
[{"review_id": "bPaqq6fhsq", "reviewer": "Reviewer_5Jx7", "summary": "1. This paper empirically examines the effectiveness of small pre-trained transformers (with millions of parameters) as multimodal data compressors for text, images, and audio.\n2. Trained on 165GB of data per modality, these models achieve compression ratios that surpass both general-purpose and domain-specific compression algorithms while remaining competitive with state-of-the-art online adaptive transformers. However, despite their strong performance on in-domain data, traditional compressors outperform them on out-of-distribution data. \n3. The study also investigates the effects of multimodal training, revealing limited cross-modal transfer in small transformers—contrasting with trends observed in large-scale models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and insightful comments. We are pleased that they think that our `paper presents a comprehensive empirical study`, which `leverages appropriate benchmarks`, and that our `writing is clear and well-structured`.\n\n\n**What is the novelty over Delétang et al. (2024)?**\n\nWhile our work is indeed inspired by Delétang et al. (2024), we believe that the unadjusted compression ratio that they primarily use to evaluate their models is not well suited to evaluate pre-trained neural networks as compressors since it does not capture the entire output of the compressor. The adjusted compression ratio, which is the one we investigate, induces an entirely different regime where large-scale models are no longer viable. Note that Delétang et al. (2024) are aware of this and, therefore, also report the adjusted compression ratio for their models, with the difference that the large language models (LLMs) they evaluate are not competitive in this regime.\n\nMoreover, Delétang et al. (2024) focus on off-the-shelf LLMs and only perform a single experiment where they train small transformers on enwik8 (100MB of text data) to obtain strong compression performance on enwik9. In contrast, we perform a `comprehensive empirical study` (in the reviewer’s words) in the multimodal setting with much larger datasets (165GB), where we not only show that small-scale transformers can achieve compression ratios on audio and image data that are on par with general-purpose and domain-specific compressors but also investigate the cross-modality transfer (in addition to a variety of ablations). We also improve the evaluation by always evaluating on out-of-distribution data (even within the same modality) to ensure a fair comparison with standard compression algorithms (overfitting to the training distribution is no longer possible).\n\nAccordingly, our contributions compared to the previous literature are (see also Table A1):\n* Small transformers can achieve competitive compression ratios on audio and image data (Delétang et al. (2024) only showed this for text data), even when comparing against domain-specific compressors.\n* Small transformers can achieve competitive compression ratios across multiple domains, pointing towards the generality of relatively small models.\n* Unlike the results on LLMs by Delétang et al. (2024), small transformers do not achieve transfer outside the training modalities. Moreover, unlike our work, Delétang et al. (2024) did not investigate in-modality out-of-distribution data (e.g., training on ImageNet and evaluating on CelebA) to show in-modality generalization.\n\n\n**The conclusions of the paper seem intuitive and straightforward.**\n\nWhile we agree that our results may not challenge the expectations and intuitions of experts in the field who know the literature well, we present novel quantitative results, which are significant and address an important *open* problem, i.e., whether small transformers pre-trained on large (multimodal) datasets can achieve competitive compression ratios across different modalities and whether there is transfer to unseen modalities, as observed in the large-scale model case (see Section 1). A priori, it was unclear whether we would attain similar conclusions to existing literature on multimodal generalization of small and large models since the compression viewpoint introduces an entirely different set of constraints on the models (most importantly with respect to their size). Finally, even some of our qualitative findings, e.g., the lack of out-of-modality transfer even for our largest models that we trained on multiple different modalities, were somewhat surprising to us.\n\nWe agree that investigating more sophisticated training strategies to improve cross-modal transfer, e.g., via transfer learning, presents an interesting direction for future work and have mentioned this in the updated version of our manuscript.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "Yes, The evaluation is thorough, leveraging appropriate benchmark datasets and comparing performance against both domain-specific and general-purpose compression algorithms.", "theoretical_claims": "Key findings:\n•\tSmall pre-trained transformers can achieve compression rates comparable to both domain-specific and general-purpose algorithms.\n•\tEven with multimodal training, small transformers do not develop a universal compression ability that enables strong performance on unseen modalities.\nThe authors present two key findings (as highlighted in the summary). Building on prior work [1][2], they conduct a comprehensive study on multi-modal training and perform extensive ablations over context size, dataset composition, and model size, ultimately deriving conclusions specific to small transformers.\n[1] Izacard G, Joulin A, Grave E. Lossless Data Compression with Transformer[J]. 2019.\n[2] Yu L, Simig D, Flaherty C, et al. Megabyte: Predicting million-byte sequences with multiscale transformers[J]. Advances in Neural Information Processing Systems, 2023, 36: 78808-78823.", "experimental_designs_or_analyses": "Yes, The authors compare to various standard compressors, both general-purpose, and domain-specific for audio data and PNG and lossless JPEG2000  for images. They also compare to the online transformer with the default v3.3 parameters, which is currently SOTA on the Large Text Compression Benchmark(LTCB).", "relation_to_broader_scientific_literature": "This is my main concern: the conclusions of the paper seem intuitive and straightforward. While there may not be prior work that explicitly derives these specific findings, existing literature on the multimodal generalization capabilities (in this case, compression abilities) of both large and small models already points toward similar conclusions.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths\n1.The writing is clear and well-structured, making the study easy to follow.\n2. This paper presents a comprehensive empirical study on the compression performance of small transformer models, particularly for multi-modal data.\n3.Key findings:\n•\tSmall pre-trained transformers can achieve compression rates comparable to both domain-specific and general-purpose algorithms.\n•\tEven with multimodal training, small transformers do not develop a universal compression ability that enables strong performance on unseen modalities.\n\nWeaknesses:\n1.\tLimited novelty: The study appears to be an extension of [1], applying the pre-training of small transformers for text compression to a multimodal setting with additional ablation experiments. However, it does not seem to derive significantly more valuable insights or practical applications beyond this extension.\n2.\tIntuitive conclusions: The conclusions of the paper seem intuitive and straightforward. While there may not be a prior work that explicitly derives these specific findings, existing literature on the multimodal generalization capabilities (in this case, their compression abilities) of both large and small models  already points toward similar conclusions. The paper could be strengthened by going beyond empirical observations and offering optimization strategies—such as transfer learning—to improve small transformers' performance on unseen modalities, making the findings more actionable and impactful.\n[1] Delétang G, Ruoss A, Duquenne P A, et al. Language modeling is compression[J]. arXiv preprint arXiv:2309.10668, 2023.", "other_comments_or_suggestions": "no", "questions_for_authors": "Please refer to weakness part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "no"}}, {"review_id": "ftW8Be0K8Y", "reviewer": "Reviewer_Bivq", "summary": "This paper studies byte language models as a means to compress data on multimodal data (text, audio, images).  They show that by training small scale transformers data can be compressed better than with standard compression algorithms like gzip or domain-specific standards (JPEG). This is achieved without online training, i.e., the Transformers are not trained to compress the data they are evaluated on, unlike competing methods like Bellard.\n\nThe obvious disadvantage of this method is speed (much slower than standard compression). Thus the practical utility of the method is limited to very specific cases (where compression ratio has to be minimized, no matter the computational cost). Yet the authors are open about this, thus this is more of a feasibility study.\n\n## update after rebuttal\nI had initially given an \"accept\", conditional on the authors' response to clarify my questions (and concerns). The authors answered my questions, thus I retained my score. The work appears sound, novel and non-incremental, thus I am clearly in favor of acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review and constructive feedback. We are pleased that they think our `interesting findings will impact the future of compression via LMs`, our `results are rich with multiple ablation studies`, and our `paper is clearly written`.\n\n**How do you ensure that your compression is lossless? Is this by construction?**\n\nYes, it is lossless by construction as we use arithmetic coding (AC), a lossless compression algorithm that uses the next-symbol probabilities of a sequential predictor (i.e., our transformers) to compress (see Figure 1 in Delétang et al. (2024) and Hutter et al. (2024)). For any predictor, AC is optimal in terms of coding length, i.e., the compression performance only depends on the predictor. If the predictor is bad (e.g., an untrained network), the compressed size would be larger than the original — but the data would still be losslessly recoverable.\n\nTo decompress, AC must use the same predictor, so we also have to communicate its parameters (we use half precision). The precision used to encode the parameters does not affect the compression's “losslessness”, only the performance.\n\n**How can a text-only model losslessly compress images?**\n\nAs we convert all data into byte streams (no tokenization) and train our models on next-byte prediction, they can process any byte data. Images are simply byte streams with different statistics and can also be fed to models trained only on text bytes. If the model has learned general patterns, it will also predict the image byte stream well (and losslessly compress it well via AC).\n\n**What is considered lossless compression (half, single, double precision)?**\n\nOur compression is lossless by construction, and the numerical precision depends entirely on the AC, which can be arbitrarily high (at the expense of compression performance). We use the data’s precision, i.e., single precision (we compress bytes). The underlying data may be encoded at higher precision (e.g., `float32` images), which is losslessly recoverable from our byte streams.\n\n**Do you have a train/validation/test split, or did you cherry-pick the best models?**\n\nOur setup differs slightly from the standard train/validation/test splits as we always evaluate OOD data (e.g., train on ImageNet, evaluate on CelebA) to mimic the standard compressor setup. As the reviewer states, Figure 2 is a feasibility result, which is why we show the best performance. Figures 3 to 6 show *all* the models (the full hyperparameter sweeps), as the reviewer requested.\n\n**How are the bytes extracted from the model’s head during training and inference?**\n\nAC directly uses the model’s predictions over tokens (i.e., the logits) to encode/decode data (see Figure 1 in Delétang et al. (2024) for an overview). The model computes a distribution over the next byte for every input byte, and the AC then uses these predictions to losslessly compress the data. No separate head or extraction procedure is necessary.\n\nWe train via standard log-loss minimization to perform next-byte prediction. At inference time, we perform standard autoregressive evaluation (using teacher forcing).\n\nWe have added the above clarifications to our revised manuscript.\n\n**Why are LLMs worse at compression when the parameter count is considered?**\n\nLLMs are viable compressors depending on whether their parameter count is factored into the *compression ratio* (i.e., adjusted vs. unadjusted). Imagine that Alice wants to compress and send data to Bob, who wants to decompress it. If Alice trains a neural network to compress the data but does not communicate the model’s weights, Bob cannot decode the data. Thus, the parameter count needs to be factored into the compression ratio. We believe that the unadjusted compression ratio is not well suited to evaluate *pre-trained* neural networks as compressors.\n\nWe have reformulated Section 1 to make this more explicit.\n\n**What is the context window's impact on compression performance?**\n\nWe investigated this question in Figure 5. For text, increasing the model size is more beneficial than increasing the context length (short-term dependencies are most important). In contrast, large context sizes are generally beneficial for images (models with larger contexts can process larger fractions of an image). For audio, the relationship between model and context size is more complex.\n\nWe expect MegaByte (Yu et al., 2023) or multiscale byte language models (Egli et al., 2025) to improve performance on some but not all domains.\n\n**Why do you only report the compression ratio?**\n\nAs we train our models for lossless compression via AC (which is equivalent to log-loss minimization), the compression ratio is the only relevant metric (apart from more “practical” metrics such as running time, which we report in Table A3). The compression ratio and the log-loss are proportional (the additive factor is the model size) and can be recovered from each other.\n\n**L196 misses \"is\".**\n\nFixed, thanks!", "claims_and_evidence": "There are four claims in the introduction which are all supported well by evidence:\n(1) Empirical study on compression of small Transformers\n(2) transformers are better at compressing than general-purpose compressors\n(3) Multimodal boosts compression on multimodal data\n(4) Poor transfer to unseen modalities\n\nHowever, there is one implicit claim where I see zero support: Can you demonstrate that the model is actually doing lossless compression and the original sequence can be recovered exactly? How do you ensure that compression is lossless? Is this by construction? E.g., if you ran an inference before even starting the training, would the \"compressed\" file still be lossless, just exactly as big (or bigger) than the source bytestream? For a text-only model, I fail to see how it can losslessly compress 1GB of image.", "methods_and_evaluation_criteria": "The compression ratio metric makes sense, but it is the only metric reported throghout the paper. Results may be slightly richer if some additional metrics (like train/validation loss) were also shown", "theoretical_claims": "What is considered lossless compression? Are we talking about half, single, double precision? (I only saw that model weights are counted with half precision)", "experimental_designs_or_analyses": "Apologies for my lack of knowledge on compression but I would like to see more details on the training/inference workflow for the model. Maybe a figure could help? \nTraining: If training is done with pure CE objective on next byte prediction, how is the compressed output extracted from the model? Training with standard next token (aka byte) prediction would yield a byte sequence of identical length to the input bytestream,.\nInference: If you compress 1GB of data, with a context size of 4096 and no overlap, then, in practice you sequentially run 1M/4096 forward steps. How many bytes are you predicting in each of those steps?\n\n\nIt's problematic to (1) cherry-pick and report the best performance over hyperparameter sweeps on the validation data (it seems that you did not do a train/validation/test split) and (2) report single-run results without any indication of confidence intervals. If you did only either (1) or (2) I wouldn’t raise this, but in combination, I find the results a bit brittle. I acknowledge that (1) prior art didn’t even make this distinction between train/validation and (2) the computational costs for such methods are very high. For the rebuttal I would like to see performance of all models, rather than only the best ones", "relation_to_broader_scientific_literature": "Well related to broader literature in LM-compression space (Deletang, Bellard etc) and the byte-LM space (Megabyte, BLT etc). Out of curiosity, could the authors comment on the impact of larger context windows for compression algorithms? Megabyte has a context window up to 1M bytes, multiscale byte LMs (https://arxiv.org/pdf/2502.14553) can even go to 5M bytes -- would you expect that compression ratio can be increased significantly with a model that can fit the entire bytestream into the context window?", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n- Interesting findings that will impact the future of compression via LMs \n- Paper is clearly written and relatively easy to follow\n- Results are rich and multiple ablation studies were conducted\n\nWeaknesses:\n- The extraction of the bytes from the LM head is not clearly defined (see my comment above)", "other_comments_or_suggestions": "Minor things:\n\t- Introduction: \"LLMs compress better when parameter count is not considered\" -- according to which metric are LLMs worse at compression when the count is considered? I only understood later from equation 3 that you include the size of the compressor by encoding model weights in half precision. Maybe you can be more explicit in the intro\n\t- L196 - missing \"is\"", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M5Y6lQVAOu", "reviewer": "Reviewer_L7Gi", "summary": "The paper shows that small decoder-only transformers trained on *multimodal data* are effective data compressors. This happens when the modality of the data being compressed (text, audio, video) belongs to the training set. In this multimodal setup, the authors show that it is possible to achieve compression ratios that are competitive with state-of-the-art algorithms (on 1GB of data), even considering the transformer parameters.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough assessment of our work. We are pleased that they think that our `work addresses a relevant open question backed by a convincing set of experiments` with `methods and evaluation criteria that are appropriate and sound`, and that they consider our `paper well-written`.", "claims_and_evidence": "The paper is well written the claims are supported by convincing evidence. See the experimental design section.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate and sound.", "theoretical_claims": "No theoretical claims to discuss.", "experimental_designs_or_analyses": "A convincing and comprehensive set of experiments backs the paper. \nThe paper examines various effects influencing the compression rations, including the training set composition Fig.2, 3 model size Fig. 4. It explores how the context size and sliding window approaches affect the data compression and how the compression ratio is affected by the size of the evaluation set Fig. 7. Taken together these are a convincing set of experiment supporting the claims of the paper.", "relation_to_broader_scientific_literature": "The relation to the broader scientific literature is thoroughly discussed in Sec. 6 and expanded in Sec. A of the appendix. The literature itself is well presented in Sec. 3.", "essential_references_not_discussed": "Essential references are discussed.", "other_strengths_and_weaknesses": "The work is good and well-written, it is slightly incremental with respect to the previous studies on compression but addresses a relevant open question backed by a convincing and comprehensive set of experiments.", "other_comments_or_suggestions": "None; the paper is well written.", "questions_for_authors": "I do not have any relevant concerns that could change my evaluation of this paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tL0iXQ6xNy", "reviewer": "Reviewer_YU4t", "summary": "This paper investigates the compression efficiency of foundation models when accounting for parameter size. Through extensive experiments on 165GB of text, image, and audio data, the authors demonstrate that relatively small transformer models (millions of parameters) can outperform standard compression algorithms like gzip and even domain-specific compressors like FLAC—achieving a 0.49 compression ratio on out-of-distribution audio data compared to FLAC's 0.54. While these smaller models can handle multiple trained modalities effectively, they show limited transfer capabilities to unseen modalities unlike their larger counterparts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful assessment and constructive feedback. We are pleased that they think that our `topic is interesting and an important direction`, that our `experimental settings are comprehensive`, and that our `paper is well-written`.\n\n\n**Could you also compare to state-of-the-art lossless image codecs such as JPEG-XL?** \n\nYes, thank you for the suggestion! We evaluated JPEG-XL on our image data and obtained a compression ratio of 0.26, which, as the reviewer correctly predicted, is slightly better than our model (and Bellard’s):\n\n| Evaluation Modality | Ours | Bellard | gzip | LZMA2 | PNG | JPEG 2000 | JPEG XL |\n| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| Image | 0.285 | 0.281 | 0.698 | 0.545 | 0.426 | 0.390 | **0.260** |\n\nWe have updated all the tables and figures with these results in our revised manuscript. Note that the point of our paper is not so much that neural networks can outperform any domain-specific compressor but that they can be on par with some of the most widely used general-purpose and domain-specific compressors.\n\n\n**I think that the theoretical contributions are too weak for the ICML audience**\n\nOur work is a comprehensive empirical study, which is in scope for and frequently gets published at ICML.\n\nThe theory underlying our work (and Delétang et al. (2024)) is standard textbook material in information theory. Accordingly, Delétang et al. (2024), which was published at ICLR 2024, did not introduce this theory. Instead, they provided a concise review of that theory, as we did in Section 2.\n\n\n**What is the contribution compared to prior work?**\n\n We investigated the *open* problem of whether *small* transformers *pre-trained* on large *multimodal* datasets can achieve competitive compression ratios across different modalities and whether there is *transfer to unseen modalities* (see Section 1), as observed in the large-scale model case. Prior work either considered large-scale, pre-trained models (Delétang et al., 2024) or models trained online (Bellard, 2021). As a result, we investigated a different regime that induces entirely different constraints on the optimal model architecture (in particular, its size) and for which it was a priori unclear whether competitive compression ratios would be achievable across different modalities.\n\nAccordingly, our contributions compared to the previous literature are (see also Table A1):\n* Small transformers can achieve competitive compression ratios on audio and image data (Delétang et al. (2024) only showed this for text data), even when comparing against domain-specific compressors, which the reviewer considers an `interesting finding`.\n* Small transformers can achieve competitive compression ratios across multiple domains, pointing towards the generality of relatively small models.\n* Unlike the results on large language models by Delétang et al. (2024), small transformers do not achieve transfer outside the training modalities. Moreover, unlike our work, Delétang et al. (2024) did not investigate in-modality out-of-distribution data (e.g., training on ImageNet and evaluating on CelebA) to show in-modality generalization.\n\n\n**How do you process RGB images for lossless compression?**\n\nAs described in Appendix B.1, we use all three (RGB) channels, each represented as `uint8` (unlike Delétang et al. (2024), who compress grayscale images, i.e., use a lossy representation of the original image). We flatten the image and concatenate all flattened images to form our image byte stream from which we iteratively sample chunks where the chunk size depends on the model’s context size (typically only parts of a single image fit into our models’ context, and, occasionally, there may be parts of two different images in the context).", "claims_and_evidence": "Although I appreciate the author's interesting findings that indicate a small transformer with massive training data can compress well to out-of-distribution data than general-purpose and even domain-specific compressors, the theoretical contribution is relatively weak for the ICML audience. Meanwhile, Deletang et al. actually have shown the performance of a transformer compressor trained from the scratch. Compared with Deletang et al., the main contributions of this paper lie on the usage of more multi-modality data and corresponding performance on cross-modality data, if I understand correctly. I still think the theoretical contributions are still weak to ICML audience.", "methods_and_evaluation_criteria": "I think the method is very easy to follow. Authors leverage a small transformer with multiple domains for training. The out-of-distribution compression performance is focused on.", "theoretical_claims": "This paper lacks theoretical claims and main results are derived from empirical experiments.", "experimental_designs_or_analyses": "Indeed, there are some SOTA image lossless codecs are used for comparison. Please consider some representative SOTA image lossless codecs, such as JPEG-XL and WebP.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- This paper is well-written and easy to follow\n\n- The topic is interesting and the capacity of transformer-based model for lossless data compression is an important direction \n\n- The experimental settings are comprehensive, and the authors choose multiple modalities for lossless compression comparison.\n\nWeaknesses:\n- Although I appreciate the author's interesting findings that indicate a small transformer with massive training data can compress well to out-of-distribution data than general-purpose and even domain-specific compressors, the theoretical contribution is relatively weak for the ICML audience. Meanwhile, Deletang et al. actually have shown the performance of a transformer compressor trained from the scratch. Compared with Deletang et al., the main contributions of this paper lie on the usage of more multi-modality data and corresponding performance on cross-modality data, if I understand correctly. I still think the theoretical contributions are still weak to ICML audience.  \n\n- I think it is necessary to compare with state-of-the-art lossless image compressor such as JPEG-XL.\n\n- How the authors to process the RGB images for lossless compression? Deletang et al.  give the scheme of grayscale image. It is very interesting to claim this detail.\n\n- Overall, I think this paper needs to further justify the contribution compared to previous literature. I am happy to increase my score upon nice answers.", "other_comments_or_suggestions": "Please see the weakness.", "questions_for_authors": "Please see the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Novel work on studying byte-level multimodal model. The work is motivate,  and the experiments are solid and convincing. Reviewers concerns on the incremental contribution, results / conclusions are intuitive and straightforward, and lack of theoretical contribution.
Decision: Accept (poster)