ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment

## Abstract

Abstract Data selection is crucial for optimizing language model (LM) performance on specific tasks, yet most existing methods fail to effectively consider the target task distribution.
Current approaches either ignore task-specific requirements entirely or rely on approximations that fail to capture the nuanced patterns needed for tasks like Autoformalization or code generation.
Methods that do consider the target distribution often rely on simplistic, sometimes noisy, representations, like hashed n-gram features, which can lead to collisions and introduce noise.
We introduce ZIP-FIT , a data selection framework that uses gzip compression to directly measure alignment between potential training data and the target task distribution.
Our key insight is that compression-based similarity captures both syntactic and structural patterns relevant to the target task, enabling more precise selection of truly task-relevant data.
In extensive evaluations on Autoformalization and Python code generation, ZIP-FIT significantly outperforms leading baselines like DSIR and D4.
Models trained on ZIP-FIT -selected data achieve their lowest cross-entropy loss up to 85.1% faster than baselines, demonstrating that better task alignment leads to more efficient learning.
In addition, ZIP-FIT performs selection up to 65.8% faster than DSIR and two orders of magnitude faster than D4.
Notably, ZIP-FIT shows that smaller, well-aligned datasets often outperform larger but less targeted ones, demonstrating that a small amount of higher quality data is superior to a large amount of lower quality data.
Our results imply that task-aware data selection is crucial for efficient domain adaptation, and that compression offers a principled way to measure task alignment.
By showing that targeted data selection can dramatically improve task-specific performance, our work provides new insights into the relationship between data quality, task alignment, and model learning efficiency.

## 1 Introduction

Choosing training data is crucial for the performance of language models (LMs) in both general-purpose and domain-specific applications (Brown et al., ; Gururangan et al., ; Hoffmann et al., ).
To date, most research on data curation has focused on creating diverse pre-training datasets to enhance model performance across a wide range of tasks (Sorscher et al., ; Xie et al., ; Tirumala et al., ; Abbas et al., ; Xie et al., ; Lee et al., ; Wettig et al., ; Penedo et al., ; Li et al., ; Sachdeva et al., ), and while these methods have been demonstrated to work well for general pre-training, they fall short in domain-specific fine-tuning, where data relevance is crucial. This raise a key question:
How should we, in a general purpose manner, effectively select fine-tuning data for a domain-specific target task?

One approach is to train binary classifiers to identify relevant data.
For example, a mathematical language model called DeepSeekMath (Shao et al., ) utilized OpenWebMath (Paster et al., ), a compilation of high-quality mathematical texts, to train a FastText classifier to retrieve analogous texts from the Web (Bojanowski et al., ).
Although effective, this method relies on the availability of large and well-annotated data sets, something that is often missing in niche tasks where relevant data are scarce.

Figure: Figure 1: ZIP-FIT selects task-specific data for efficient finetuning. (0) Obtain both the source and target datasets. (1) Calculate ZIP-FIT Alignment of each source example with the target dataset using gzip compression. (2) Rank all source examples based on these alignment scores. (3) Select the top-K most aligned examples for fine-tuning. (4) Fine-tune a large language model using the selected top-K examples to improve performance on the target task.
Refer to caption: extracted/5949657/diagrams/zip-fit-final-schematic.png

Figure: Figure 2: Code Generation: ZIP-FIT accelerates cross-entropy loss reduction, even in code-specialized models like CodeGemma-2B. The plots show cross-entropy test loss versus the number of training tokens for Gemma2-2B (top row) and CodeGemma-2B (bottom row) across different token selection sizes. ZIP-FIT (blue) consistently reduces loss faster than DSIR (green) and D4 (red), achieving up to $85.11\%$ speed improvement at lower token counts. These results demonstrate ZIP-FIT’s efficiency in data selection for fine-tuning models on code-geneation tasks.
Refer to caption: extracted/5949657/plots/code_all_models.png

Another common approach is to use neural embeddings to measure the similarity between data points and a reference corpus (Xie et al., ).
Although this improves relevance, embedding-based methods are computationally expensive and sensitive to the choice of embedding space (Muennighoff, ).
Alternatively, DSIR (Data Selection via Importance Resampling) (Xie et al., ) utilizes unigrams and bigrams to select data points without the need for pre-trained embeddings, with the aim of matching the hashed n-gram distributions of the target data. Although DSIR is effective in capturing direct word correlations, it may not capture structured patterns of syntax that unfold across sentences or paragraphs, such as nested function calls in code or embedded clauses in formal language translation (Moura et al., ). Additionally, the hashing introduces noise due to collisions. These shortcomings highlight the need for alternative data selection strategies better suited to domain-specific tasks.

To address these challenges, we propose ZIP-FIT, a novel data selection framework that leverages the classic compression algorithm gzip.
Recent research suggests that language modeling and data compression are fundamentally equivalent tasks (Delétang et al., ), and the intelligence of large language models (LLMs) is closely related to their ability to compress external corpora (Huang et al., ).
This insight suggests that compression algorithms can encode information in ways similar to neural networks.
For example, Jiang et al. () found that the use of normalized compression distances for text classification outperformed traditional neural embeddings.
Inspired by this, ZIP-FIT selects aligned training data with a target data set based on compression-based alignment, providing a lightweight and embedding-free method for selecting high-quality data.

We evaluated ZIP-FIT in two domains: Autoformalization and Python code generation.
ZIP-FIT outperforms existing data selection methods, consistently improving model performance cross-entropy test loss.
Smaller, well-aligned datasets selected by ZIP-FIT lead to faster convergence and better performance than larger, less aligned datasets, highlighting the importance of data quality.

Our contributions are as follows:

- 1.
Methodology: The introduction of ZIP-FIT, an embedding-free data selection method based on gzip compression.
- 2.
Superior Performance: ZIP-FIT consistently outperforms leading baselines (DSIR, D4) in Autoformalization and Python code generation, achieving up to 85.1% faster convergence and lower cross-entropy loss.
- 3.
Computational Efficiency: ZIP-FIT is computationally efficient, running up to 65.8% faster than DSIR. This makes it scalable for low-resource environments without compromising performance.

## 2 ZIP-FIT : an Embedding-Free Data Selection Algorithm via Compression-Based Alignment for LM Fine-Tuning

Figure: Figure 3: Higher ZIP-FIT alignment correlates with lower cross-entropy loss. The relationship between ZIP-FIT alignment and cross-entropy (CE) loss for (a) GPT-2 trained on 22k tokens ($R^{2}=0.90,p=0.001$) and (b) Mistral7B trained on 22k tokens ($R^{2}=0.75,p=0.025$). Each point represents a dataset, with its position reflecting the dataset’s ZIP-FIT alignment score against the ProofNet test set and the resulting CE loss. The dashed red line indicates the linear regression fit, while the dashed grey line shows the pretrained CE loss. Higher alignment scores correspond to lower CE losses, demonstrating that training on better aligned data yields better performance.
Refer to caption: extracted/5949657/plots/ce_loss_combined.png

Before introducing ZIP-FIT, it is essential to understand the desired attributes of effective data selection algorithms. Ideally, such algorithms should be performant, computationally economical, fast, scalable, and designed to improve the efficiency of model training. These characteristics ensure that the data filtering process can be applied broadly and effectively in various machine learning contexts, particularly when computational resources are limited. By setting these criteria, we can better appreciate the innovations ZIP-FIT introduces in the realm of data selection.

### 2.1 Background

gzip compression: uses two main techniques for compression, LZ77 and Huffman coding.
Together, these methods compress sequences by exploiting repeated patterns in the data.
LZ77 works by identifying repeated substrings and replacing them with references to their earlier occurrences.
Huffman coding further compresses the data by assigning shorter binary codes to more frequent symbols, optimizing the overall length of the compressed text.
For more details, see Appendix A.

AutoFormalization (AF): refers to the task of translating natural language mathematical statements into a formal mathematical programming language, like Lean4 Moura et al. ().
This process requires precise understanding and representation of mathematical formal syntax, making the selection of well-aligned training data crucial for effective model training.

### 2.2 ZIP-FIT Algorithm

Setup: Given a set of examples $\{x^{\prime}_{1},x^{\prime}_{2},\ldots,x^{\prime}_{n}\}$ from a target distribution $p$ and a large source dataset $\{x_{1},x_{2},\ldots,x_{N}\}$ from an arbitrary distribution $q$, ZIP-FIT aims to select a subset of $k$ examples (where $k\ll N$) from $q$. The selected subset is used for model training, in order to improve performance for tasks associated with $p$. This approach is intended to maximize the efficacy and efficiency of model training by focusing on the most relevant data samples.

Method: ZIP-FIT uses gzip compression as a metric to measure the alignment of each example in $q$ with the target $p$, focusing on capturing patterns and redundancies.

To address the challenge of selecting highly aligned data, we propose the ZIP-FIT algorithm:

Figure: Algorithm 1 ZIP-FIT Data Selection Algorithm

## 3 Higher alignment interventionally achieves better model performance

Experiment: We validate compression as an alignment metric by evaluating the impact of a model fine-tuned on more ZIP-FIT-aligned data with a target task and the corresponding cross-entropy (CE) loss.
We chose ProofNet (test) as the target benchmark and then fine-tuned GPT-2 Radford et al. () and Mistral7B Jiang et al. () ) LMs on datasets with varying ZIP-FIT alignment.

Results: Figure 3 shows a strong negative correlation ($R^{2}$ of 0.90) between gzip alignment scores and CE loss for GPT-2 and 0.75 for Mistral7B.
This implies that data alignment plays a crucial role in improving model performance
We control for the number of tokens in each dataset, setting it to 100k tokens, except for datasets that do not contain this many tokens (e.g., ProofNet validation set).
Data sets such as LeanDojo Yang et al. () and the ProofNet validation set, which exhibit high alignment scores, resulted in significantly lower CE loss compared to less aligned data sets such as C4 and WikiText (Raffel et al. (), Merity et al. ()).
Data sets with high alignment like LeanDojo Yang et al. () and the ProofNet (validation) resulted in a significantly lower CE loss compared to less aligned data sets like C4 and WikiText (Raffel et al. (), Merity et al. ()).

## 4 Higher Alignment leads to more efficient training

Experiment: We fine-tuned GPT-2 (124M) and Mistral7B for the AutoFormalization task using different datasets scored with ZIP-FIT alignment. We used ProofNet (test) for the evaluation.
The curves represent different datasets with varying alignment to the target domain (ProofNet validation).

Results: More aligned data reduces CE loss quickest, as shown by the steep decline for high-alignment datasets.
This is most evident as ProofNet (validation). Less aligned data require significantly more tokens to achieve similar performance.
This demonstrates that targeted data selection with ZIP-FIT accelerates fine-tuning and improves performance, reducing computational costs.

Figure: Figure 4: Highly aligned data lowers cross-entropy loss more efficiently. The x-axis shows the number of training tokens, and the y-axis represents the cross-entropy (CE) test loss. Different curves correspond to datasets filtered by different alignment scores, indicating their relevance to the target domain. The most aligned data reduce Test CE loss significantly faster than less aligned data. The left panel depicts results using GPT-2, and the right panel uses Mistral7B, demonstrating that using highly aligned data not only accelerates training but also achieves better model performance, validating the effectiveness of ZIP-FIT for data selection in fine-tuning.
Refer to caption: extracted/5949657/plots/more_alignment_lower_loss.png

## 5 Comparative Evaluation of ZIP-FIT for Efficient Fine-Tuning

Figure: Figure 5: AutoFormalization: ZIP-FIT consistently achieves lower test loss more quickly than D4 and DSIR, demonstrating its efficiency in data selection. The plots show cross-entropy test loss versus the number of training tokens for three models (InterLM-Math-Plus-1.8B, Gemma2-2B, and Mistral7B) across different token selection sizes. ZIP-FIT (blue line) consistently outperforms both DSIR (green line) and D4 (red line) across all model and token size configurations, highlighting its ability to process data more efficiently. The percentage labels in each plot indicate the relative speedup of ZIP-FIT over DSIR in reaching the lowest cross-entropy loss, reinforcing the method’s scalability and adaptability for domain-specific fine-tuning.
Refer to caption: extracted/5949657/plots/af_all_models3.png

We evaluate ZIP-FIT on two domain-specific tasks: Autoformalization and Python Code Generation.
Our goal is to show ZIP-FIT’s data selection leads to superior fine-tuning.

### 5.1 AutoFormalization

#### Experiment:

Our source dataset comprised approximately 185,000 sequences from LeanDojo, Proof-Pile 2, C4, and WikiText Yang et al. (); Azerbayev et al. ().
For details, refer to Appendix A.2.
Alignment was computed using ZIP-FIT, DSIR and D4 with the ProofNet’s validation split (our target distribution).
For a fair comparison, we did not modify how any of the methods rank sequences.
To compare each method, we selected the n sequences ranked highest for several values of n (353k, 695k tokens, etc.).
For each selected data set at each value of n we fine-tune InterLM-Math-Plus-1.8B, Gemma2-2B, and Mistral7B (Ying et al. (); Team et al. ()).
Performance was evaluated using the CE loss ProofNet’s test split.

Results:
Figure 5 shows that ZIP-FIT significantly outperforms DSIR and D4 in reducing cross-entropy (CE) loss across all token selection sizes (353k, 695k).
The steep decline in the blue curves (ZIP-FIT) highlights its ability to achieve faster convergence, resulting in up to 62.79% improvements in convergence speeds compared to DSIR.
Notably, ZIP-FIT demonstrates up to a 65.8% faster data selection process than DSIR.
An interesting observation is ZIP-FIT’s efficiency in selecting highly-aligned data and improving even specialized mathematical models like InternLM-MAath-Plus-1.8B.
While one might expect diminished returns on a model already adept in a related domain, the improvements suggest that the model still benefits from the AutoFormalization data.
Similar results were observed at other token counts, as detailed in Appendix C.

### 5.2 Code Generation

#### Experiment:

We conducted code generation experiments using ZIP-FIT, DSIR, and D4 to select data from a mix of sources:
MBPP (Austin et al. (), Python docstrings, Proof-Pile 2, C4, WikiText.
The latter two are included to study whether the data selection methods considered are robust to misaligned data.
For details, refer to Appendix A.3.
The datasets were utilized to fine-tune both CodeGemma-2B and Gemma2-2B models, with the focus on translating function signatures and docstrings into executable Python code.
For the selection process, we used HumanEval for validation and a separate hold-out portion for final testing.
For a fair comparison, we did not modify how any of the methods rank sequences.
To compare each method, we selected the n sequences ranked highest for several values of n (800k, 930k, 1.6M tokens, etc.).

#### Results:

Across all tested n values, ZIP-FIT consistently outperformed DSIR and D4 in reducing cross-entropy loss, demonstrating faster and more effective fine-tuning.
In particular, the CodeGemma-2B model, already optimized for code-related tasks, showed the most improvements with ZIP-FIT, confirming its ability to select highly relevant and beneficial training data.
Rapid loss reduction under ZIP-FIT emphasizes its efficiency, especially noted in its 25% faster data processing compared to DSIR.
Most notably, the flattening of the DSIR and D4 curves indicate diminishing returns, suggesting that additional tokens would not achieve the performance of ZIP-FIT.
In general, these findings emphasize that ZIP-FIT accelerates model training and optimizes resource usage, making it a superior choice for code generation tasks.

## 6 Impact of Data Misalignment on Model Performance

Existing research showed that data alignment plays a critical role in improving model performance and learning efficiency for downstream tasks. In this section, we explore how misalignment in data can affect model performance and how ZIP-FIT addresses this issue with data selection.

Experiment: We fine-tuned the Mistral7B model on the same source dataset we used for the AutoFormalization experiment (see Appendix 5.1), filtering data with ZIP-FIT at different alignment thresholds ($>$0.1, $>$0.2, $>$0.3). Each threshold creates a progressively more aligned dataset, where the $>$0.3 dataset is the most aligned, and the $>$0.2 dataset is a superset of the $>$0.3 dataset, including less aligned data. Similarly, the $>$0.1 dataset is a superset of both $>$0.2 and $>$0.3.
Figure 6 shows cross-entropy test loss (y-axis) versus the number of training tokens (x-axis).
The dashed line marks the pretrained Mistral7B baseline.

Results: ZIP-FIT selected data achieves lower cross-entropy loss faster than training on all data, showing improved performance and efficiency. Higher alignment thresholds result in a steeper loss reduction, confirming that filtering out misaligned data enhances fine-tuning.Misalignment in training data can introduce noise and irrelevant patterns, which typically require more training data and computational resources to overcome. By applying higher alignment thresholds, ZIP-FIT ensures that only the most relevant and helpful examples are used for training. This targeted selection leads to a more efficient learning process as evidenced by the sharper decline in cross-entropy loss for higher alignment thresholds. Such efficiency is crucial in scenarios where computational resources are limited or costly.

Practical Considerations: For practitioners, these results suggest that investing in better data curation and alignment tools can significantly cut down the cost and time of model training without compromising performance. It also highlights the potential pitfalls of using large, uncurated datasets that might slow down the learning process or lead to poorer generalization on specific tasks.

Future Directions: Further research could explore adaptive alignment thresholds based on real-time validation performance, potentially automating the selection process to optimize both speed and accuracy during training.

These results further validate the empirical performance gains and computational efficiency achieved by ZIP-FIT, as outlined in our contributions. By filtering out misaligned data, ZIP-FIT accelerates fine-tuning and reduces computational overhead, confirming its utility in low-resource settings.

Figure: Figure 6: Selective data filtering with ZIP-FIT allows us to achieve better cross-entropy test loss faster than training on all the data, resulting in improved performance and efficiency. The x-axis represents the number of training tokens, while the y-axis shows the cross-entropy test loss. The curves represent models fine-tuned (FT) on datasets filtered by varying alignment thresholds ($>$ 0.1, $>$ 0.2, $>$ 0.3). The dashed line indicates the baseline performance of the pretrained Mistral7B model. Training on data filtered with higher alignment thresholds leads to superior performance, demonstrating the effectiveness of removing misaligned data in fine-tuning.
Refer to caption: extracted/5949657/plots/remove_misaligned_data2.png

## 7 Related Works

Curating pre-training data for Language Models often involves using classifiers to filter high-quality data from large corpora like Common Crawl, as done for models like GPT-3 and PaLM2 (Brown et al., ; Google, ; Shao et al., ). While effective, this process requires significant computational resources and large volumes of curated data. In contrast, ZIP-FIT efficiently selects relevant data without relying on external models, making it especially useful in data-scarce environments.

Deduplication techniques, such as SemDeDup (Abbas et al., ) and D4 (Tirumala et al., ), improve data efficiency by removing duplicate or semantically similar examples using embedding-based clustering. However, these methods are computationally expensive and not tuned to the target task. ZIP-FIT is embedding-free and task-aware, making it both scalable and more effective at selecting relevant data.

Mixture weights are essential when drawing from multiple domains, as they significantly influence the performance of language models (Du et al., ; Xie et al., ). While Domain Reweighting with Minimax Optimization DoReMi) (Xie et al., ) proposes a robust domain-level reweighting strategy suitable for diverse target distributions. DoReMi is not designed for example-level data selection, as it primarily focuses on domain-level reweighting. Adapting it to select individual data points for specific target distributions would require substantial modifications to its foundational algorithm.
One possibility be to effectively transform each data point into a ’mini-domain,’ a process that would stray significantly from DoReMi’s original purpose and scope.
Therefore, we did not use DoReMi in our comparisons because it does not directly address the fine-grained selection needs that ZIP-FIT fulfills.

Autoformalization refers to the process of translating natural language mathematics into formal language (Wang et al., ; Wu et al., ), which is advantageous because formal proofs can be verified for correctness.
However, the ability of current models to autoformalize text is limited by the scarcity of human-curated formal data. ZIP-FIT provides a framework for selecting the most relevant data, ensuring that models are trained on aligned datasets that enhance their performance.

## 8 Limitations

While ZIP-FIT provides a computationally efficient method for data selection, it has several limitations.
First, the gzip compression-based alignment may not fully capture nuanced semantic relationships that dense representations can, potentially affecting its effectiveness for complex domains like natural language understanding, where paraphrasing is important.
Second, ZIP-FIT’s reliance on gzip means that its performance could vary depending on the nature of the textual data, particularly in highly diverse datasets where compression gains are less apparent.

## 9 Discussion and Future Work

ZIP-FIT introduces an efficient, embedding-free approach for data selection in language model fine-tuning.
By leveraging gzip compression to capture redundancies in data, ZIP-FIT enables the alignment of large-scale datasets with a target domain without the computational burden of neural embeddings.
Our results show that using compression-based alignment leads to faster convergence and lower cross-entropy loss compared to existing methods like DSIR and D4 (Tirumala et al., ; Xie et al., ).

However, this approach highlights the trade-off between simplicity and the ability to capture complex semantic relationships.
While compression-based methods offer a lightweight alternative, they might not fully replace embedding-based techniques for highly intricate domains, such as natural language understanding or paraphrases.
Nonetheless, ZIP-FIT’s promising results suggest that leveraging compression as a data selection tool can be highly effective, especially in resource-constrained scenarios and economically crucial tasks like code generation, where gzip can leverage the syntactic structure of the data.

Future work could explore hybrid models that combine the strengths of compression-based techniques with neural embeddings to further enhance data selection.
Additionally, extending ZIP-FIT to support more diverse data modalities and investigating its robustness across various domains would provide a more comprehensive understanding of its capabilities and limitations.
We plan for future work to study its application to complex natural language-only tasks and mathematics, where paraphrasing and semantics are important.

We also plan to explore the use of ZIP-FIT for synthetic data generation. While generating synthetic data is straightforward, selecting high-value samples for training presents challenges, especially when managing limited token budgets Villalobos et al. ().
Autoformalization is a fantastic task for this exploration, as it inherently has a limited number of tokens, thus simulating the critical challenge of token scarcity.
Additionally, studying synthetic data selection is crucial for developing self-improving agents that can avoid model collapse (Gerstgrasser et al., ; Kazdan et al., ) by ensuring high-quality data accumulation.

Furthermore, diversity was identified as an important meta-data property that can influence model performance (Miranda et al., ).
Therefore, we aim to address this in future work by either:
(1) developing an algorithm that balances diversity with alignment in data selection, or
(2) creating a metric that incorporates diversity as part of its evaluation process.

[backgroundcolor=blue!10, linecolor=blue!50!black, linewidth=2pt, innertopmargin=nerbottommargin=nerrightmargin=20pt, innerleftmargin=20pt, roundcorner=10pt]
Key Takeaways:

- •
Efficiency in Data Selection: ZIP-FIT utilizes gzip compression for alignment, demonstrating significant efficiency in selecting domain-specific data, enhancing model fine-tuning.
- •
Resource Optimization: It outperforms traditional methods like DSIR and D4 by speeding up training and reducing computational demands, beneficial in resource-limited settings.
- •
Domain-Specific Improvements: Exhibits superior performance in tasks like AutoFormalization and code generation, where precise data alignment is crucial.
- •
Practical Application: Effective in identifying and using the most relevant data from mixed datasets, proving critical for achieving better domain-specific results.

## 10 Conclusion

In this work, we introduced ZIP-FIT, an efficient and scalable data selection method that leverages gzip-based compression to enhance the downstream performance of language models for domain-specific tasks. Our experiments demonstrate that ZIP-FIT not only accelerates the fine-tuning process but also significantly improves downstream performance by aligning training data more closely with target tasks. By comparing against established methods like DSIR and D4, ZIP-FIT proved superior in selecting highly-aligned data, especially in complex tasks such as Autoformalization and code generation. This methodology sets a new standard for resource-efficient and effective data selection for model training, providing a step in understanding the choice of training
data for downstream transfer in LMs.

## Appendix A gzip Compression Details

gzip is a lossless data compression algorithm that combines two primary techniques: LZ77 compression and Huffman coding. Here, we provide additional technical details on how gzip works.

#### LZ77 Compression:

LZ77 works by identifying repeated substrings in the input text and replacing them with backward references. Mathematically, LZ77 can be described as follows:

Given an input sequence $S=s_{1},s_{2},\dots,s_{n}$, the algorithm searches for the longest prefix of the remaining sequence $S^{\prime}=s_{i},s_{i+1},\dots,s_{n}$ that matches a substring within a predefined window of previous characters. If a match is found, it is replaced by a tuple $(d,l,c)$, where:

- •
$d$ is the distance from the current position to the start of the matching substring,
- •
$l$ is the length of the matching substring, and
- •
$c$ is the character following the match (if any).

For example, the substring $s_{i},s_{i+1},\dots,s_{i+l-1}$ can be replaced by the tuple $(d,l,c)$, thereby reducing redundancy in the data.

#### Huffman Coding:

After applying LZ77, gzip employs Huffman coding to further reduce the size of the compressed data. Huffman coding assigns variable-length codes to symbols based on their frequency of occurrence, with shorter codes assigned to more frequent symbols.

The expected length $L(X)$ of the Huffman code for a sequence of symbols $X=x_{1},x_{2},\dots,x_{n}$ is calculated as:

$$ $L(X)=\sum_{i=1}^{n}p(x_{i})\cdot\text{len}(C(x_{i})),$ $$

where:

- •
$p(x_{i})$ is the probability of symbol $x_{i}$,
- •
$\text{len}(C(x_{i}))$ is the length of the Huffman code for $x_{i}$.

This further minimizes the size of the compressed data by leveraging the statistical properties of the input.

#### Combined gzip Compression:

The total compressed size $C(S)$ after applying both LZ77 and Huffman coding can be approximated as the sum of the lengths of the backward references and the Huffman-coded symbols:

$$ $C(S)=\sum_{(d,l,c)}\text{len}(d,l,c)+\sum_{i=1}^{n}\text{len}(C(x_{i})).$ $$

#### Normalized Compression Distance (NCD):

gzip’s effectiveness in data selection stems from its ability to measure the alignment between two sequences $A$ and $B$ based on how efficiently they compress together. The Normalized Compression Distance (NCD) is given by:

$$ $NCD(A,B)=\frac{C(A\oplus B)-\min(C(A),C(B))}{\max(C(A),C(B))},$ $$

where $C(A)$ and $C(B)$ are the compressed lengths of sequences $A$ and $B$, and $C(A\oplus B)$ is the length of the compressed concatenation of both sequences.
A lower NCD indicates greater alignment between the sequences.

### A.1 Why Use Compression?

Compression algorithms, such as gzip, provide a computationally efficient way to detect patterns and minimize redundancy in data.

#### Limitations of n-grams:

Many traditional methods, including hashed n-grams, focus on capturing immediate textual correlations by simplifying text into discrete, fixed-size buckets.
Although these techniques are computationally efficient, they may not adequately capture syntactic or structural relationships within the data. Additionally, the introduce noise due to collisions during hashing.

#### Challenges with Neural Embeddings:

Neural embeddings offer a powerful tool for capturing semantic relationships, but they come with significant computational costs. These embeddings are typically pre-trained on large corpora and fine-tuned for specific tasks, which requires substantial resources. Given the scalability challenges of embedding-based methods, we conjecture that a simpler method like compression can provide a more scalable and resource-efficient alternative.

We hypothesize that compression – in this case gzip, but perhaps a different compression algorithm –serves as a strong proxy for capturing syntactic and structural relationships in textual sequences.
gzip’s ability to compress data based on redundancy minimization can be leveraged as a metric to align text with a target distribution.

### A.2 Composition of the Source Dataset for AutoFormalization

The source dataset for the AutoFormalization task was compiled from a variety of datasets to ensure a diverse mix of mathematical, general textual, and code-related content. Below are the details of the datasets included:

- •
UDACA/AF: 4,300 samples from informal formalization statements.
- •
C4: 10,000 samples from the clean crawl of the internet, ensuring a broad linguistic variety.
- •
LeanDojo: 10,000 samples from task-oriented proofs and tactics.
- •
LeanDojo Informalized: 10,000 samples combining traced tactics with informal descriptions, aiming to bridge formal reasoning and natural language.
- •
UDACA/AF-split: 10,000 samples, a variant of the UDACA/AF dataset with split annotations.
- •
WikiText: 10,000 samples from a collection of professionally curated articles, providing a rich linguistic framework.
- •
Algebraic Stack: Samples from various subsets of mathematical and programming languages, capped at 10,000 samples per subset or fewer if the total subset size was under this threshold.

Each dataset was selected to complement the others by covering different aspects of language use, from technical to informal, ensuring the model’s exposure to a wide range of linguistic structures and contents. The total dataset size aggregated to approximately 185,000 sequences, which were then subjected to alignment scoring and further processing for model training.

### A.3 Composition of the Source Dataset for Code Generation

The source dataset for the Code Generation task was assembled from various data sources to provide a diverse range of coding and natural language contexts. Below are the details of the datasets included:

- •
MBPP (Google Research): A total of 964 samples focusing on Python coding challenges.
- •
Python Code Instructions (18k Alpaca): 5,000 sequences providing natural language prompts for Python code, fostering a practical approach to code generation.
- •
Python Docstrings (Calum/The Stack): 5,000 sequences each of Python function docstrings integrating detailed natural language documentation of python functions.
- •
Python Docstrings (Calum/The Stack): 5,000 sequences each of Python function code bodies, integrating raw python code without documentation.
- •
C4 (AllenAI): 10,000 samples from a clean web crawl.
- •
WikiText: 10,000 samples from a collection of curated articles, providing rich natural language training material.
- •
Algebraic Stack: A selection of sequences from various programming language subsets, each capped at 10,000 samples or the total subset size if less than this threshold.

This combination of datasets was specifically chosen to challenge our methods ’s ability to choose syntactically correct and functionally accurate Python code, while also responding appropriately to natural language prompts.

### A.4 Hyperparameters for Model Fine-Tuning

All models in our experiments were fine-tuned with the following unified setup, aimed at ensuring a consistent evaluation across different models and data selection strategies.

#### Models and Tokenizer:

The fine-tuning was performed using the following models:

- •
InterLM-Math-Plus-1.8B
- •
Gemma2-2B
- •
Mistral7B

#### Training Settings:

The key hyperparameters used across all models are as follows:

- •
Block Size: 1024 tokens
- •
Learning Rate: $7.5\times 10^{-7}$
- •
Batch Size: 4 (per device)
- •
Number of Epochs: 1
- •
Weight Decay: 0.01
- •
Maximum Gradient Norm: 1.0

Training was facilitated using the Trainer class from Hugging Face’s Transformers library, with the Accelerate library handling model parallelism to efficiently utilize available computational resources.

#### Evaluation Metrics:

For model evaluation, we employed:

- •
Cross-Entropy Loss at the end of training to measure the effectiveness of the fine-tuning.

Fine-tuning was performed under controlled conditions to ensure fair comparison between data selected by ZIP-FIT, DSIR, and manual curation methods. The effectiveness of each method was assessed based on how the models performed on the ProofNet and HumanEval.

#### Data Handling and Logging:

All logs, model checkpoints, and tokenizer settings were systematically saved in designated directories for thorough analysis post-experiment

This comprehensive and standardized approach to fine-tuning ensures that our experimental results are robust, reproducible, and transparent, providing clear insights into the effectiveness of the data selection methodologies employed in our study.

## Appendix B Rationale for the Method Name ZIP-FIT

We chose the name ZIP-FIT for two reasons:

- 1.
ZIP refers to the use of gzip compression for data selection, where compression aligns the data for better future fine-tuning (or FITting).
- 2.
The name also references scaling laws, as ZIP-FIT consistently reduces loss faster than competing methods, implying better power-law scaling parameters, drawing a parallel to Zipf’s law Piantadosi (), which describes similar scaling behavior in language models.

Remark: Zipf’s law Piantadosi () describes the inverse relationship (thus power law $f(r)\propto 1/r^{s}$, where $r$ is the rank and $f(r)$ is the frequency of the word with rank $r$) between a word’s frequency and its rank in natural language, a pattern that reflects scaling behavior.
Rank in this context is the position of the word after sorting with respect to frequency in the text.

## Appendix C Additional Experimental Results: data selection for efficient fine-tuning using ZIP-FIT

Figure: Figure 7: ZIP-FIT consistently achieves a lower test loss at a faster rate compared to D4 and DSIR for Autoformalization. The plots show the cross-entropy test loss against the number of training tokens for three models (InterLM-Math-Plus-1.8B, Gemma2-2B, and Mistral7B) across various token selection sizes. ZIP-FIT (blue line) consistently surpasses both DSIR (green line) and D4 (red line) across all model and token size configurations, emphasizing its superior data processing efficiency. The percentage labels in each plot denote the relative speedup of ZIP-FIT over DSIR in attaining the lowest cross-entropy loss, further underscoring the method’s scalability and adaptability for domain-specific fine-tuning.
Refer to caption: extracted/5949657/plots/af_remaining_plot.png

## Appendix D Data Selection Profiling (Run Times)

ZIP-FIT performs selection up to 65.8% faster than DSIR and 21,076% (=5h/85s=211, which is 2 orders of magnitude) faster than D4.
Experimental results comparing ZIP-FIT vs DSIR profiling/run time for Code data selection can be found in figure 8. Note that depending on the dataset and number of samples these numbers may not hold. Compression may not scale well to long-context datasets and depending on the source dataset, our run times varied widely. However, on average we observed that ZIP-FIT is comparable to DSIR and generally faster. More experiments across a wider range of datasets need to be conducted in order to infer more.

Figure: Figure 8: ZIP-FIP demonstrates lower cross-entropy and lower run time during data selection than competing DSIR and D4 methods. ZIP-FIT is cheaper, faster, and better performing. The run times do no include fine-tuning time, since it’s a constant offset across all models. D4’s data selection (not shown) takes 5hs because it uses an embedding model (opt-125m Zhang et al. ()), the same one as the original paper Tirumala et al. ().
Refer to caption: extracted/5949657/plots/runtime_zip_fit_vs_dsir_code.png

## Appendix E Qualitative Analysis

Qualitative results show top 20 examples can be found it table 1.

Table: Table 1: Beginning characters of the top 20 samples selected by ZIP-FIT when the target task is code generation.

Table: Table 2: Beginning characters of the top 20 samples selected by DSIR when the target task is code generation. DSIR does not easily provide alignment scores, so instead we report the ZIP-FIT scores, which reveals that ZIP-FIT doesn’t score highly the DSIR examples which might explain why ZIP-FIT achieves better CE loss.

## Appendix F Future Work (Cont.)

Lossless Compression for Alignment: While ZIP-FIT has demonstrated substantial efficiency for data selection, there are several promising directions for future exploration.
One potential enhancement is leveraging faster compression algorithms, such as LZ4 and Snappy, which offer rapid processing speeds at the cost of lossy compression.
In our current approach, we utilize gzip for compression-based alignment, which is lossless and provides a robust foundation.
However, LZ4 and Snappy are optimized for speed and could potentially offer even greater computational efficiency without the need for decompression in our pipeline.
Given that our primary goal is efficient data selection rather than perfect data recovery, these faster algorithms might be more suitable.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: ### Minor comments

Figure 3, page 5:  
The color bar is labeled "Gzip Alignment" instead of
"ZIP-FIT-Alignment" from Algorithm 1; it may be confusing to readers.

Figure 3, page 5, line 231:  
Please mention also in the figure caption that the test loss is
calculated on ProofNet data.
soundness: 4
strengths: The paper is concise, sound, well written, and the experimental section shows promise for the method, especially with regard to other embedding-free methods.

The conceptual simplicity combined with the empirical results of the method is an especially strong point of the work.
summary: The paper introduces a new data selection mechanism based on text
compression distances. The concept of using compression methods for
deep learning follows several modern practical results and theoretical
motivations that language modeling is fundamentally based in text
compression. The method's conceptual simplicity combined with strong
empirical results make it stand out as a modern way for filtering for
aligned data.
weaknesses: Ideally, it would be shown how the size of $n$ (i.e., number of
samples from the target domain $p$) influences the performance of the
method. If it is possible to pick $n$ just sufficiently large enough,
it would greatly improve the computational efficiency of the method
for large target datasets.

Experiments in other domains would be really nice to better
demonstrate the generalization capabilities of the method. Possibly
there is data that is not well-suited to compression and accordingly
ZIP-FIT, or where the data's compression factor varies too much
between samples?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could you provide further insights into how ZIP-FIT might perform with data that have higher variability and diverse syntactic structures, such as conversational datasets?
2. Can you clarify the theoretical basis for using gzip compression over other compression methods that might exploit redundancy differently? Would alternative compression algorithms affect the performance of ZIP-FIT?
soundness: 3
strengths: 1. ZIP-FIT’s embedding-free approach is a refreshing deviation from common embedding-based methods, offering a novel solution by leveraging gzip compression. The concept of using normalized compression distance (NCD) as an alignment metric is insightful and could inspire future research in embedding-free methodologies for various data selection tasks.
2. The empirical results support the claims, showing that ZIP-FIT achieves faster convergence and better performance than established methods. The experiments were conducted on both AutoFormalization and code generation tasks, demonstrating ZIP-FIT's versatility across different domains.
3. The paper is well-structured, with a clear exposition of the algorithm, experimental setup, and results. The figures effectively illustrate the performance benefits of ZIP-FIT.
4. ZIP-FIT could represent a significant advancement in data selection for machine learning, particularly in computationally constrained environments. Its potential to optimize model fine-tuning with minimal resource requirements makes it highly applicable for real-world use cases, especially in domain-specific and low-resource applications.
summary: This paper introduces ZIP-FIT, an embedding-free data selection method leveraging gzip compression to measure the alignment between training and target domains. Unlike existing approaches that rely on neural embeddings, ZIP-FIT uses a computationally efficient compression-based alignment metric, enabling faster data selection while maintaining high relevance to the target task. Empirical evaluations demonstrate ZIP-FIT’s superiority over baselines DSIR and D4 in AutoFormalization and code generation tasks, achieving significantly faster convergence and lower cross-entropy loss with reduced computational costs. ZIP-FIT’s promise lies in its scalability and effectiveness, particularly in low-resource settings, where traditional embedding-based methods may be impractical.
weaknesses: 1. While ZIP-FIT achieves excellent results on the tasks tested, its reliance on gzip compression may limit its effectiveness in complex semantic domains where relationships are nuanced and less compressible. Embedding-free approaches, while efficient, may not be ideal for tasks that require deep semantic understanding or complex syntactic relationships.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: As specified in the "Weaknesses" section:
- What is the score of the fine-tuned LLM using ZIP-FIT on benchmarks like HumanEval and PubMedQA compare to LLMs fine-tuned without using ZIP-FIT?
- How does ZIP-FIT compare to prior method like https://arxiv.org/abs/2405.00705 in terms of both running time and final model score?
soundness: 2
strengths: - This paper is well-presented and well-motivated.
- Studying computation-efficient methods for data selection in LLM instruction fine-tuning is a promising research direction.
- The proposed ZIP-FIT is intuitive and easy to follow.
- The proposed approach bypasses the need for LLM forward computation to obtain embeddings, making it computationally efficient.
- The presented experimental results seem promising.
summary: This paper proposes ZIP-FIT, an efficient, embedding-free method for selecting high-quality, domain-specific fine-tuning data for language models (LMs). Prior methods often rely on computationally expensive neural embeddings or classifiers to filter aligned datasets, while those based on N-gram similarity may lack the structural depth needed for complex tasks like code generation. In contrast, ZIP-FIT leverages gzip compression to evaluate data alignment with target domains, based on the idea that compression algorithms encode information in a way similar to neural networks. The ZIP-FIT approach eliminates the need for LM forward passes to obtain embeddings, making it efficient and particularly suitable for low-resource environments. Experimental results show that ZIP-FIT outperforms prior data selection methods, such as DSIR and D4, as measured by test loss.
weaknesses: - [Major] The proposed method seems very simple and straightforward; using a gzip-style method to embed data appears to be a relatively standard approach.
- [Major] All experimental results are based on test loss, which may not be very reliable. It would be essential to conduct evaluations on some standard benchmarks, such as HumanEval and MBPP for code evaluation, to demonstrate the scores the model can achieve.
- It is unclear how the proposed ZIP-FIT compares to prior, more complex data selection methods in terms of both running speed and final model quality (e.g., [1]), aside from deduplication approaches like D4.
- [Minor] The paper seems to be written somewhat in rush, the figure quality of Figure 2 does not seem to be very high.

[1] https://arxiv.org/abs/2405.00705

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: (1) Could the authors provide additional evidence to support the claim that gzip is effective in capturing syntactic and structural relationships in textual sequences?

(2) Would the authors be able to demonstrate the effectiveness of their approach using evaluation metrics beyond cross-entropy test loss, and compare it to relevant baselines, such as those mentioned earlier?

(3) Could you provide more insight into why D4 was excluded from the code generation experiments, and specifically how it affected model performance?
soundness: 1
strengths: (1) Problem Significance: The author tackles a crucial problem in low-resource settings, addressing the challenge of fine-tune data selection without relying on GPU-intensive and embedding-based methods. This is a highly relevant and impactful research direction.

(2) Innovative Filtering Criterion: The authors' inspiration from gzip compression methods has led to the proposal of a novel and intriguing selection criterion. This approach is not only interesting but also demonstrates out-of-the-box thinking, making it a notable contribution to the field.
summary: This paper introduces an innovative, embedding-free data selection method for efficient fine-tuning of large language models. Drawing inspiration from gzip compression techniques, the authors propose utilizing Normalized Compression Distance as a metric to filter and prune fine-tuning datasets. The authors conduct a comparative analysis with prior embedding-free methods, originally designed for filtering pre-training datasets, on Autoformalization and Python coding tasks.
weaknesses: (1) **Inadequate Baselines**: The authors propose a data selection method for model alignment, but only compare it with prior works such as DSIR and D4, which were primarily designed for data selection during the pre-training phase. A more comprehensive literature review on data pruning methods for model alignment is lacking, including embedding-based methods [1], LLM model response metrics [2], Gradient-based metrics [3],Quality metrics judged by LLMs [4], inference loss on evaluation sets [5].

(2) **Evaluation Metrics**: The authors primarily use test data cross-entropy loss as the evaluation metric, results are thus not surprising given that the data selection method uses the test data to anchor the selection criteria. However, the authors do not compare their results with widely accepted metrics in the research community for the studied downstream tasks, such as:

(a). Autoformalization: proof success rates on miniF2F [6,7]

(b). Python coding: functionality pass rates (pass@k on HumanEval) based on unit-tests [8,9]


(3) **Clarifications on Motivation**: In Section 2.3, the authors argue that n-grams fail to capture syntactic or structural relationships within the data, while hypothesizing that gzip does. However, this hypothesis is not supported by theoretical or empirical evidence, weakening the motivation for the proposed approach. It is also not compared on if the proposed approach is better or worse than high-resource methods, such as embedding-based methods.

References: 

[1] DEFT-UCS: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection

[2] From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning

[3] LESS: Selecting Influential Data for Targeted Instruction Tuning

[4] Alpagasus: Training a better alpaca with fewer data

[5] Instruction Mining: Instruction Data Selection for Tuning Large Language Models

[6] Autoformalization with Large Language Models

[7] LEGO-Prover: Neural Theorem Proving with Growing Libraries

[8] Evaluating Large Language Models Trained on Code

[9] Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

Meta-review: The paper introduces ZIP-FIT, a data selection framework that utilizes gzip compression to measure alignment between potential training data and target tasks, aiming to enhance language model performance in specific domains. The authors reported strong results (by test losses) on two downstream tasks. The method is also scalable, thanks to efficient data compression techniques to align train-test data distributions. However, there are some major concerns still in the current work: (1) Task Specificity: While effective in tasks like autoformalization and Python code generation, the applicability of ZIP-FIT to other domains remains to be explored. (2) Major experimental results are based on test loss, which may not be very reliable. The authors should provide the final results according to the downstream task performance.

Decision: Reject