ICML 2025

NICE
Data Selection for Instruction Tuning in LLMs
with Non-differentiable Evaluation Metric
Jingtan Wang 1 2 Xiaoqiang Lin 1 Rui Qiao 1 3 Pang Wei Koh 4 Chuan-Sheng Foo 2 5 Bryan Kian Hsiang Low 1 3
Abstract
Curating data for instruction tuning is crucial
for enhancing the performance of large language
models (LLMs). This work aims to select training
data for instruction tuning to improve the LLM
performance on specific tasks. Existing methods
often rely on next-token prediction (NTP) loss as
a proxy for target task performance due to the non-
differentiable nature of performance evaluation
metrics. They select training data points that are
most helpful in reducing validation loss. However,
there is a discrepancy between minimizing NTP
loss and maximizing performance (e.g., code pass
rate in code generation). To remedy this, we intro-
duce a novel Non-differentiable evaluation metric-
based InfluenCe Estimation (NICE), which lever-
ages the policy gradient to select the training data
that improves the performance. Moreover, NICE
can perform data selection in the absence of labels
(ground-truth responses) when the evaluation met-
rics do not require labels (e.g., a reward model can
output reward scores without supervision from
labels). Experimental results show that our ap-
proach outperforms existing data selection base-
lines that use NTP loss in diverse and realistic sce-
narios. Notably, subsets selected by NICE often
produce models that outperform those trained on
the full dataset. Our code is available at https:
//github.com/JTWang2000/NICE.
1. Introduction
Instruction tuning (Bai et al., 2022; Ouyang et al., 2022) is
a fine-tuning paradigm that enables large language models
1National University of Singapore 2Institute for Infocomm Re-
search (I2R), A*STAR, Singapore 3Singapore-MIT Alliance for
Research and Technology 4University of Washington 5Centre for
Frontier AI Research (CFAR), A*STAR, Singapore. Correspon-
dence to: Bryan Kian Hsiang Low <lowkh@comp.nus...