ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment
Authors: name: Elyas Obbad; name: Iddah Mlauzi; name: Brando Miranda; name: Rylan Schaeffer; name: Kamal Obbad; name: Suhana Bedi; name: Sanmi Koyejo
Published: 2024-10-23 18:01:06

Abstract: Data selection is crucial for optimizing language model (LM) performance on
specific tasks, yet most existing methods fail to effectively consider the
target task distribution.
  Current approaches either ignore task-specific requirements entirely or rely
on approximations that fail to capture the nuanced patterns needed for tasks
like Autoformalization or code generation.
  Methods that do consider the target distribution often rely on simplistic,
sometimes noisy, representations, like hashed n-gram features, which can lead
to collisions and introduce noise.
  We introduce ZIP-FIT, a data selection framework that uses gzip compression
to directly measure alignment between potential training data and the target
task distribution.
  In extensive evaluations on Autoformalization and Python code generation,
ZIP-FIT significantly outperforms leading baselines like DSIR and D4.
  Models trained on ZIP-FIT-selected data achieve their lowest cross-entropy
loss up to 85.1\% faster than baselines, demonstrating that better task
alignment leads to more efficient learning.
  In addition, ZIP-FIT performs selection up to 65.8\% faster than DSIR and two
orders of magnitude faster than D4.
  Notably, ZIP-FIT shows that smaller, well-aligned datasets often outperform
larger but less targeted ones, demonstrating that a small amount of higher
quality data is superior to a large amount of lower quality data.
  Our results imply that task-aware data selection is crucial for efficient
domain adaptation, and that compression offers a principled way to measure task
alignment.
  By showing that targeted data selection can dramatically improve
task-specific performance, our work provides new insights into the relationship
between data quality, task alignment, and model learning efficiency.