Evaluating Sample Utility for Data Selection by Mimicking Model Weights
Authors: name: Tzu-Heng Huang, name: Manjot Bilkhu, name: Frederic Sala, name: Javier Movellan

Foundation models rely on large-scale web-crawled datasets, which frequently
contain noisy data, biases, and irrelevant content. Existing data selection
techniques typically use human heuristics, downstream evaluation datasets, or
specialized scoring models, and can overlook samples' utility in the training
process. Instead, we propose a new approach, Mimic Score, a data quality metric
that uses a pretrained reference model as a guide to assess the usefulness of
data samples for training a new model. It relies on the alignment between the
gradient of the new model parameters and the vector pointing toward the
reference model in weight space. Samples that misalign with this direction are
considered low-value and can be filtered out. Motivated by the Mimic score, we
develop Grad-Mimic, a data selection framework that identifies and prioritizes
useful samples, automating the selection process to create effective filters.
Empirically, using Mimic scores to guide model training results in consistent
performance gains across six image datasets and enhances the performance of
CLIP models. Moreover, Mimic scores and their associated filters improve upon
existing filtering methods and offer accurate estimation of dataset quality.