ICon: In-Context Contribution for Automatic Data Selection
Authors: name: Yixin Yang, name: Qingxiu Dong, name: Linli Yao, name: Fangwei Zhu, name: Zhifang Sui

Data selection for instruction tuning is essential for improving the
performance of Large Language Models (LLMs) and reducing training cost.
However, existing automated selection methods either depend on computationally
expensive gradient-based measures or manually designed heuristics, which may
fail to fully exploit the intrinsic attributes of data. In this paper, we
propose In-context Learning for Contribution Measurement (ICon), a novel
gradient-free method that takes advantage of the implicit fine-tuning nature of
in-context learning (ICL) to measure sample contribution without gradient
computation or manual indicators engineering. ICon offers a computationally
efficient alternative to gradient-based methods and reduces human inductive
bias inherent in heuristic-based approaches. ICon comprises three components
and identifies high-contribution data by assessing performance shifts under
implicit learning through ICL. Extensive experiments on three LLMs across 12
benchmarks and 5 pairwise evaluation sets demonstrate the effectiveness of
ICon. Remarkably, on LLaMA3.1-8B, models trained on 15% of ICon-selected data
outperform full datasets by 5.42% points and exceed the best performance of
widely used selection methods by 2.06% points. We further analyze
high-contribution samples selected by ICon, which show both diverse tasks and
appropriate difficulty levels, rather than just the hardest ones.