ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment

Authors: Elyas Obbad, Iddah Mlauzi, Brando Miranda, Kamal Obbad, Rylan Schaeffer, Suhana Bedi, Sanmi Koyejo
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords: data centric machine learning, autoformalization, large language models, reasoning

Abstract:
Data selection is crucial for optimizing language model (LM) performance on
specific tasks, yet most existing methods fail to effectively consider the target task
distribution. Current approaches either ignore task-specific requirements entirely
or rely on approximations that fail to capture the nuanced patterns needed for tasks
like Autoformalization or code generation. Methods that do consider the target
distribution often rely on simplistic, sometimes noisy, representations, like hashed
n-gram features, which can lead to collisions and introduce noise. We introduce
ZIP-FIT, a data selection framework that uses gzip compression to directly
measure alignment between potential training data and the target task distribution. Our key insight is that compression-based similarity captures both syntactic
and structural patterns relevant to the target task, enabling more precise selection of truly task-relevant data. In extensive evaluations on Autoformalization and
Python code generation, ZIP-FIT significantly outperforms leading baselines
like DSIR and D4. Models trained on ZIP-FIT-selected data achieve their lowest cross-entropy loss up to 85.1% faster than baselines, demonstrating that better
task alignment leads to more efficient learning. In addition, ZIP-FIT performs
selection up to 65.8% faster than DSIR and two orders of magnitude faster than
D4. Notably, ZIP-FIT shows that smaller, well-aligned datasets often outperform larger but less targeted ones, demonstrating that a small amount of higher
quality data is superior to a large amount of lower quality data. Our results imply that task-aware data selection is crucial for efficient domain adaptation, and
that compression offers a principled way to measure task alignment. By showing
that targeted data selection can dramatically improve task-specific performance,
our work provides new insights into the relationship between data quality, task
alignment, and model learning efficiency.

Decision: reject
Reviewer scores: 8, 6, 3, 1

Reviews and discussion:
<thread>
creation date:
1730152464921

thread_type:
official_review

<thread_object>
creator: reviewer_ctcn

<content>
summary:
This paper introduces an innovative, embedding-free data selection method for efficient fine-tuning of large language models. Drawing inspiration from gzip compression techniques, the authors propose utilizing Normalized Compression Distance as a metric to filter and prune fine-tuning datasets. The authors conduct a comparative analysis with prior embedding-free methods, originally designed for filtering pre-training datasets, on Autoformalization and Python coding tasks.

strengths:
(1) Problem Significance: The author tackles a crucial problem in low-resource settings, addressing the challenge of fine-tune data selection without relying on GPU-intensive and embedding-based methods. This is a highly relevant and impactful research direction.

(2) Innovative Filtering Criterion: The authors' inspiration from gzip compression methods has led to the proposal of a novel and intriguing selection criterion. This approach is not only interesting but also demonstrates out-of-the-box thinking, making it a notable contribution to the field.

weaknesses:
(1) **Inadequate Baselines**: The authors propose a data selection method for model alignment, but only compare it with prior works such as DSIR and D4, which were primarily designed for data selection during the pre-training phase. A more comprehensive literature review on data pruning methods for model alignment is lacking, including embedding-based methods [1], LLM model response metrics [2], Gradient-based metrics [3],Quality metrics judged by LLMs [4], inference loss on evaluation sets [5].

(2) **Evaluation Metrics**: The authors primarily use test data cross-entropy loss as the evaluation metric, results are thus not surprising given that the data selection method uses the test data to anchor the selection criteria. However, the authors do not compare their results with widely accepted metrics in the research community for the studied downstream tasks, such as:

(a). Autoformalization: proof success rates on miniF2F [6,7]

(b). Python coding: functionality pass rates (pass@k on HumanEval) based on unit-tests [8,9]


(3) **Clarifications on Motivation**: In Section 2.3, the authors argue that n-grams fail to capture syntactic or structural relationships within the data, while hypothesizing that gzip does. However, this hypothesis is not supported by theoretical or empirical evidence, weakening the motivation for the proposed approach. It is also not compared on if the proposed approach is better or worse than high-resource methods, such as embedding-based methods.

References: 

[1] DEFT-UCS: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection

[2] From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning

[3] LESS: Selecting Influential Data for Targeted Instruction Tuning

[4] Alpagasus: Training a better alpaca with fewer data

[5] Instruction Mining: Instruction Data Selection for Tuning Large Language Models

[6] Autoformalization with Large Language Models

[7] LEGO-Prover: Neural Theorem Proving with Growing Libraries

[8] Evaluating Large Language Models Trained on Code

[9] Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

questions:
(1) Could the authors provide additional evidence to support the claim that gzip is effective in capturing syntactic and structural relationships in textual sequences?

(2) Would the authors be able to demonstrate the effectiveness of their approach using evaluation metrics beyond cross-entropy test loss, and compare it to relevant baselines, such as those mentioned earlier?

(3) Could you provide more insight into why D4 was excluded from the code generation experiments, and specifically how it affected model performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730421367189

thread_type:
official_review

<thread_object>
creator: reviewer_39ck

<content>
summary:
This paper proposes ZIP-FIT, an efficient, embedding-free method for selecting high-quality, domain-specific fine-tuning data for language models (LMs). Prior methods often rely on computationally expensive neural embeddings or classifiers to filter aligned datasets, while those based on N-gram similarity may lack the structural depth needed for complex tasks like code generation. In contrast, ZIP-FIT leverages gzip compression to evaluate data alignment with target domains, based on the idea that compression algorithms encode information in a way similar to neural networks. The ZIP-FIT approach eliminates the need for LM forward passes to obtain embeddings, making it efficient and particularly suitable for low-resource environments. Experimental results show that ZIP-FIT outperforms prior data selection methods, such as DSIR and D4, as measured by test loss.

strengths:
- This paper is well-presented and well-motivated.
- Studying computation-efficient methods for data selection in LLM instruction fine-tuning is a promising research direction.
- The proposed ZIP-FIT is intuitive and easy to follow.
- The proposed approach bypasses the need for LLM forward computation to obtain embeddings, making it computationally efficient.
- The presented experimental results seem promising.

weaknesses:
- [Major] The proposed method seems very simple and straightforward; using a gzip-style method to embed data appears to be a relatively standard approach.
- [Major] All experimental results are based on test loss, which may not be very reliable. It would be essential to conduct evaluations on some standard benchmarks, such as HumanEval and MBPP for code evaluation, to demonstrate the scores the model can achieve.
- It is unclear how the proposed ZIP-FIT compares to prior, more complex data selection methods in terms of both running speed and final model quality (e.g., [1]), aside from deduplication approaches like D4.
- [Minor] The paper seems to be written somewhat in rush, the figure quality of Figure 2 does not seem to be very high.

[1] https://arxiv.org/abs/2405.00705

questions:
As specified in the "Weaknesses" section:
- What is the score of the fine-tuned LLM using ZIP-FIT on benchmarks like HumanEval and PubMedQA compare to LLMs fine-tuned without using ZIP-FIT?
- How does ZIP-FIT compare to prior method like https://arxiv.org/abs/2405.00705 in terms of both running time and final model score?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730578422085

thread_type:
official_review

<thread_object>
creator: reviewer_vfcf

<content>
summary:
This paper introduces ZIP-FIT, an embedding-free data selection method leveraging gzip compression to measure the alignment between training and target domains. Unlike existing approaches that rely on neural embeddings, ZIP-FIT uses a computationally efficient compression-based alignment metric, enabling faster data selection while maintaining high relevance to the target task. Empirical evaluations demonstrate ZIP-FIT’s superiority over baselines DSIR and D4 in AutoFormalization and code generation tasks, achieving significantly faster convergence and lower cross-entropy loss with reduced computational costs. ZIP-FIT’s promise lies in its scalability and effectiveness, particularly in low-resource settings, where traditional embedding-based methods may be impractical.

strengths:
1. ZIP-FIT’s embedding-free approach is a refreshing deviation from common embedding-based methods, offering a novel solution by leveraging gzip compression. The concept of using normalized compression distance (NCD) as an alignment metric is insightful and could inspire future research in embedding-free methodologies for various data selection tasks.
2. The empirical results support the claims, showing that ZIP-FIT achieves faster convergence and better performance than established methods. The experiments were conducted on both AutoFormalization and code generation tasks, demonstrating ZIP-FIT's versatility across different domains.
3. The paper is well-structured, with a clear exposition of the algorithm, experimental setup, and results. The figures effectively illustrate the performance benefits of ZIP-FIT.
4. ZIP-FIT could represent a significant advancement in data selection for machine learning, particularly in computationally constrained environments. Its potential to optimize model fine-tuning with minimal resource requirements makes it highly applicable for real-world use cases, especially in domain-specific and low-resource applications.

weaknesses:
1. While ZIP-FIT achieves excellent results on the tasks tested, its reliance on gzip compression may limit its effectiveness in complex semantic domains where relationships are nuanced and less compressible. Embedding-free approaches, while efficient, may not be ideal for tasks that require deep semantic understanding or complex syntactic relationships.

questions:
1. Could you provide further insights into how ZIP-FIT might perform with data that have higher variability and diverse syntactic structures, such as conversational datasets?
2. Can you clarify the theoretical basis for using gzip compression over other compression methods that might exploit redundancy differently? Would alternative compression algorithms affect the performance of ZIP-FIT?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667629898

thread_type:
official_review

<thread_object>
creator: reviewer_1ukb

<content>
summary:
The paper introduces a new data selection mechanism based on text
compression distances. The concept of using compression methods for
deep learning follows several modern practical results and theoretical
motivations that language modeling is fundamentally based in text
compression. The method's conceptual simplicity combined with strong
empirical results make it stand out as a modern way for filtering for
aligned data.

strengths:
The paper is concise, sound, well written, and the experimental section shows promise for the method, especially with regard to other embedding-free methods.

The conceptual simplicity combined with the empirical results of the method is an especially strong point of the work.

weaknesses:
Ideally, it would be shown how the size of $n$ (i.e., number of
samples from the target domain $p$) influences the performance of the
method. If it is possible to pick $n$ just sufficiently large enough,
it would greatly improve the computational efficiency of the method
for large target datasets.

Experiments in other domains would be really nice to better
demonstrate the generalization capabilities of the method. Possibly
there is data that is not well-suited to compression and accordingly
ZIP-FIT, or where the data's compression factor varies too much
between samples?

questions:
### Minor comments

Figure 3, page 5:  
The color bar is labeled "Gzip Alignment" instead of
"ZIP-FIT-Alignment" from Algorithm 1; it may be confusing to readers.

Figure 3, page 5, line 231:  
Please mention also in the figure caption that the test loss is
calculated on ProofNet data.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732162571264

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Responses

comment:
We thank all reviewers for their thoughtful feedback. For calibration purposes, we’d like to note that the ICLR 2025 rubric differs slightly from previous similar conferences. For example:

- To indicate **"Accept"**, the NeurIPS 2024 rubric says to use 7 whereas the ICLR 2025 rubric says to use 8
- To indicate **"Strong Accept"**, the NeurIPS 2024 rubric says to use 9 whereas the ICLR 2025 rubric says to use 10

The reviewers highlight several **strengths** of our work:

- "A refreshing deviation from common embedding-based methods" [vFcf]
- "Method's conceptual simplicity combined with strong empirical results make it stand out" [1UKb]
- "Tackles a crucial problem in low-resource settings" [CtcN]
- "Well-presented and well-motivated" [39ck]


The main concerns raised were:

- Evaluation beyond cross-entropy loss [CtcN, 39ck]
- Additional baselines [CtcN, 39ck]
- Justification for compression choice [vFcf]
- Performance on diverse data types [1UKb, vFcf]
- Impact of target domain sample size on performance [1UKb]

During the rebuttal period, we conducted additional experiments to address these concerns:

- We evaluated ZIP-FIT on HumanEval, showing that it doubles Pass@1 performance (12.19%) compared to a pre-trained Gemma2-2B (6.09%) and outperforms DSIR (9.14%) and D4 (6.09%).
- We performed a comprehensive analysis of different compression algorithms, demonstrating that compression parameter choice significantly impacts performance. The results validate our approach while providing insights into how compression characteristics affect data selection quality.
- We performed initial experiments evaluating the effect of the size of the target domain on performance, which suggests we can maintain or even improve performance while significantly reducing the number of target samples required, directly addressing computational efficiency concerns for large target datasets.
- We acknowledge limitations with highly variable data as discussed in section 8 of our paper and look forward to exploring this in future work.
- We are currently running comparisons with additional baselines (LESS and SHED) and will update our rebuttal with these results as soon as we have them.

We also improve clarity by:

- Moving Figure 5 (now Figure 2) to page 2
- Adding ProofNet test specification to Figure 3’s (now Figure 4) caption
- Improving quality of Figure 2 (now Figure 3)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732231714513

thread_type:
official_comment

<thread_object>
creator: reviewer_ctcn

<content>
title:
Questionable Baseline Results on Gemma-2 for HumanEval

comment:
During the review process, two reviewers (Reviewer 39ck, CtcN) had concerns regarding the evaluation metrics (of only using cross-entropy test loss) and requested for additional evaluation experiments, particularly pass@k metrics on HumanEval and proof success rates on miniF2F:

- Reviewer 39ck: *All experimental results are based on test loss, which may not be very reliable. It would be essential to conduct evaluations on some standard benchmarks, such as HumanEval and MBPP for code evaluation, to demonstrate the scores the model can achieve.*

- Reviewer CtcN: *The authors primarily use test data cross-entropy loss as the evaluation metric. However, the authors do not compare their results with widely accepted metrics in the research community for the studied downstream tasks*

During discussion phase, the authors responded to several reviewers the pass@1 HumanEval scores on Gemma2-2B of 6.1%. The authors reported scores have huge discrepancies with what the Gemma 2 technical report [1] states (20.1%) as well as what EvalPlus Leaderboard [2] reports (25%). When Reviewer CtcN questioned the authors, the authors responded with: 

*Our current experiments use quantized LoRA fine-tuning via unsloth for resource efficiency, which explains the lower baseline numbers compared to [7]. We are currently running full fine-tuning experiments with Gemma2-2B to provide direct comparisons with the reported 20.1% Pass@1.*

Reviewer CtcN responded with:

*[7] clearly states the released model to have results of 20.1%. I do not understand how your baseline results are so poor. I also do not understand how your finetuning method quantized or full precision, LoRA or full parameter (which is not disclosed in the paper) could affect the baseline released pretrained models evaluation scores.*

As reviewer CtcN, I question the creditability of the authors evaluation results. The authors responses have raised my doubt on whether experiments and evaluations are properly conducted, and if the authors are familiar with relevant works on LLMs for Coding or Autoformalization. Due to the authors responses, I have downgraded my rating for the paper. I also summarize the issue above and highlight it here.

[1] Gemma 2: Improving Open Language Models at a Practical Size

[2] https://evalplus.github.io/leaderboard.html
</content>
</thread_object>

</thread>
<thread>
creation date:
1733079568278

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additional Baseline Results for HumanEval - Comparison with Resource-Intensive Baselines

comment:
In response to Reviewers CtcN and 39ck requests for comparisons against more computationally expensive baselines, we have completed additional experiments comparing ZIP-FIT against recent resource-intensive baselines (LESS) using full parameter fine-tuning on Gemma2-2B. All methods were evaluated under identical training settings using the same number of training tokens (top 1M):

| Fine-tuning | Data Selection Method | Pass@1 (%) | Pass@10 (%) | Selection Time |
|-------------|----------------|------------|-------------|----------------|
| None | None: Pre-trained Gemma2-2B | 15.24 | 38.81 | - |
| Full FT | ZIP-FIT | **18.86** | 41.78 | **32s** |
| Full FT | LESS | 18.06 | 40.19 | 19h |
| Full FT | DSIR | 17.98 | **44.27** | 97s |
| Full FT | D4 | 14.37 | 40.66 | 7h 40m |
| None | None: Pre-trained Gemma2-2B (4-bit quantized) | 6.09 | - | - |
| QLoRA | ZIP-FIT | **12.19** | - | **32s** |
| QLoRA | DSIR | 9.14 | - | 97s |
| QLoRA | D4 | 6.09 | - | 7h 40m |

For transparency and comprehensive evaluation, we include both our new full parameter fine-tuning results and our previous QLoRA results to demonstrate ZIP-FIT's effectiveness across different computational settings.

Key findings:
1. With full parameter fine-tuning, ZIP-FIT achieves competitive Pass@1 (18.86%) and Pass@10 (41.78%) scores compared to resource-intensive methods
2. ZIP-FIT maintains significantly faster selection times (32s) compared to LESS (19h) and DSIR (97s)
3. Even with QLoRA fine-tuning, ZIP-FIT shows meaningful improvements over baselines while maintaining efficiency

These results demonstrate that ZIP-FIT can match or exceed the performance of more computationally expensive methods while maintaining its core advantage of efficiency.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces ZIP-FIT, a data selection framework that utilizes gzip compression to measure alignment between potential training data and target tasks, aiming to enhance language model performance in specific domains. The authors reported strong results (by test losses) on two downstream tasks. The method is also scalable, thanks to efficient data compression techniques to align train-test data distributions. However, there are some major concerns still in the current work: (1) Task Specificity: While effective in tasks like autoformalization and Python code generation, the applicability of ZIP-FIT to other domains remains to be explored. (2) Major experimental results are based on test loss, which may not be very reliable. The authors should provide the final results according to the downstream task performance.