NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric
Authors: Jingtan Wang
Xiaoqiang Lin
Rui Qiao
Pang Wei Koh
Chuan-Sheng Foo
Bryan Kian Hsiang Low
Venue: ICML 2025 poster (2025)
Keywords: data selection
data curation
instruction tuning
large language models
Primary area: deep_learning->large_language_models
TL;DR: We leverage policy gradients to directly optimize non-differentiable evaluation metrics for LLM instruction tuning data selection.

Abstract: Curating data for instruction tuning is crucial for enhancing the performance of large language models (LLMs). This work aims to select training data for instruction tuning to improve the LLM performance on specific tasks. Existing methods often rely on next-token prediction (NTP) loss as a proxy for target task performance due to the non-differentiable nature of performance evaluation metrics. They select training data points that are most helpful in reducing validation loss. However, there is a discrepancy between minimizing NTP loss and maximizing performance (e.g., code pass rate in code generation). To remedy this, we introduce a novel Non-differentiable evaluation metric-based InfluenCe Estimation (NICE), which leverages the policy gradient to select the training data that improves the performance. Moreover, NICE can perform data selection in the absence of labels (ground-truth responses) when the evaluation metrics do not require labels (e.g., a reward model can output reward scores without supervision from labels). Experimental results show that our approach outperforms existing data selection baselines that use NTP loss in diverse and realistic scenarios. Notably, subsets selected by NICE often produce models that outperform those trained on the full dataset. Our code is available at https://github.com/JTWang2000/NICE.

NICE O
Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric
Jingtan Wang" Xiaoqiang Lin Rui Qiao 13
Pang Wei Koh* Chuan-Sheng Foo 25 Bryan Kian Hsiang Low
Abstract
(LLMss) to follow specific human instructions, improving their performances on target downstream tasks. The effec- tiveness of instruction tuning heavily relies on the quality of the instruction dataset (Chen al., 2023; Li et al., 2024a; Zhou et al., 2024). However, the instruction dataset is usu- ally collected from mixed sources, and some data points may not be directly relevant to the target tasks (Wang et al., 2023; Xia et al., 2024). In addition, the data points often vary in quality and may contain noisy labels (Carlini et al., 2024; Frénay Verleysen, 2013; Wang et al., 2024a). These challenges underline the importance of data selection meth- ods, which enhance instruction tuning by systematically choosing relevant, high-quality data to cultivate specific tar- get capabilities in LLMs. In practice, LLMs fine-tuned on selected subsets of data can outperform those trained on the full dataset (Wang et al., 2023; Xia et al., 2024).
Curating data for instruction tuning is crucial for enhancing the performance of large language models (LLMs). This work aims to select training data for instruction tuning to improve the LLM performance on specific tasks. Existing methods often rely on next-token prediction (NTP) loss as proxy for target task performance due to the non- differentiable nature of performance evaluation metries. They select training data points that are most helpful in reducing validation loss. However, there is a discrepancy between minimizing NTP loss and maximizing performance (c.g., code pass rate in code generation). To remedy this, we intro- duce a novel Non-differentiable evaluation metric- based InfluenCe Estimation (NICE), which lever- ages the policy gradient to select the training data that improves the performance. Moreover, NICE can perform data selection in the absence of labels (ground-truth responses) when the evaluation met- rics do not require labels (e.g.. a reward model can output reward scores without supervision from labels). Experimental results show that our ap- proach outperforms existing data selection base- lines that use NTP loss in diverse and realistic see- narios. Notably, subsets selected by NICE often produce models that outperform those trained on the full dataset. Our code is available at https: //git hug..uu..... cow wwt.m//wwaagg000 200.///00000/OII......O /NICE.
Loss-based influence estimation methods (Kwon et al., 2024; Xia et al., 2024; Yeh et al., 2022) have been demon- strated to be effective in data selection. It estimates the effect of each training data on the validation loss (e.g.. Next-Token Prediction (NTP) loss) via the gradient of the validation loss, then selects the subset of data with the most positive influ- ence. However, many instruction instruction-following tasks require generating long-form responses, which are evaluated using non- din-differentiable metrics (instead of the differentiable val- idation loss). These evaluation metrics, such as the code pass rate (Chen et al., 2021), LLM-judge (Dubois al., 2023; Zheng et al., 2023) and reward model (Ouyang et al., 2022), cannot directly provide useful gradient information to estimate the influences due to their non-differentiable nature. Moreover, minimizing NTP loss may poorly align with maximizing the evaluation metrics due to overfitting to surface-level patterns (e.g., n-grams) and ignoring alterna- tive correct generations (Brown et al., 2020; Gloeckle et al., 2024; Tay et al., 2021; Zhou et al., 2024). For instance, in code generation (Chen et al., 2021), there are multiple ways of writing 'correct' code for a problem, but NTP loss is only measured w.r.t one such way. This mismatch between the NTP loss and the true evaluation metric poses significant challenge to data selection for instruction tuning. Therefore. existing approaches that rely on the influence of NTP loss may fail to select the dataset that improves the metrics used in specific tasks.
1. Introduction
Instruction tuning (Bai et al., 2022; Ouyang et al., 2022) is a fine-tuning paradigm that enables large language models
'National University of Singapore "Institute for Infocomm Re- search (f²R). S Singapore "Singapore-MIT Alliance for Research and Technology "University of Washington 'Centre for Frontier Al Research (CFAR), A STAR, Singapore. Correspon- dence to: Bryan Kian Hsiang Low <lowkh wwwwkh@comp.mus.coom wwwm.muu..ddu.s...
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Data Selection with NICE
To tackle this challenge, we introduce a novel Non- differentiable evaluation metric-based InfluenCe Estimation (NICE) method. NICE selects data that directly optimizes commonly used yet non-differemtiable evaluation metries of long-form generation tasks. Inspired by reinforcement learning (RL) (Williams, 1992; Wu et al., 2018; Sutton & Barto, 2018), NICE treats the evaluation metric as the reward function and the LLM as policy. This formulation allows us to overcome the non-differeniiabiiiyy by computing the policy gradient of the metric w.r.t. the model parameters. In particular, the policy gradient is calculated based on the gradients of the likelihood of the model-generated responses, weighted by their corresponding rewards. By using the policy gradients. NICE directly quantifies the influence of training data on validation performance measured by the metric. Therefore, NICE-selected data can better align with the evaluation metrics than the data selected by loss-based influence estimation. Moreover, NICE has two additional advantages: First, NICE supports data selection with unlabeled validation data when the reward function only requires the input and the model-generraed response (e.g.. the reward model in Bai et al. (2022)), rendering wider applicability and lower annotation costs compared to loss-based influence estimation. Second, NICE is able to use responses generated from petter-performing LLMs on the target tasks to further improve the data selection performance as NICE can make use of these high-quality generated responses (instead of only the label used in loss-based influence estimation).
ground truth sequence of words or tokens). Similar notations apply the validation set Dv == {2e (F yo ]=) +=n++1 Dv can contain different subtasks: DV. Di:..., Dr The LLM generates a sequence of words, denoted as 9; = [0 h- Here, oi is the p-th word (or token) in the generated re- sponse, and the autoregressive generation process can be described recursively as: of ~ [(y' [.a, i/...... .......... ".." ")).
The NTP loss for the training data point is defined
[ 0) -PP log f'y (2a vt. -...
- vf 0)
The NTP loss for the validation data point is defined like- wise. In the rest of the section, we first restate two rep- resentative loss-based influence estimation frameworks: TracIn (Pruthi et al., 2020) and Influence Function (Koh & Liang, 2017). Then we review how to use these influence scores to select training data points.
2.1. TracIn and Influence Function
TracIn quantifies the influence of a training data point 2, on the loss of a validation data point 2y during training. Denote " the learning rate used in the parameter update. At each step t, the influence is expressed as
L(z;; - ~ - m (VVVo(( VoT(e;; ('))
which is the gradient similarity between 2n and 2, derived in App. E.I. To measure the influence of 2; over the en- tire training run, TracIn aggregates the influence at every training step that uses z,. As z; is used once per epoch, it is natural to express this as a summation over epochs:
We perform comprehensive analyses to demonstrate the advantages of NICE. First, we empirically show the effec- tiveness of NICE across diverse and realistic scenarios for instruction tuning. This includes (1) selection from a task- agnostic training data pool that is mixed-source instruction tuning datasets and (2) selection from a task-aware training data pool whose datasets are closely aligned with down- stream tasks. Our experiments show that models trained data subsets selected by our approach generally outperform those trained using either data subsets selected by other baselines or the full dataset. Second, we demonstrate the generality of NICE by applying it to multiple bass- influence estimation frameworks and empirically verifying their resulting improved performance.
In (21, 2p) Ee= Tee VoL((z A*), VoL(2;
where Te denotes the average learning rate applied in the e-th epoch, E' is total number of training epochs, and ge represents the model parameters after the e-th epoch.
Influence Function (IF) measures the influence of down- weighting 24 on the loss of the validation data point zn:
infur(z,, x.) VoL ) H V LoL (5)
where F is the model parameters after the last epoch (total E epochs) and Hos n. > LZz 0²) is the Hessian matrix of the average training loss over the training set. The derivation can be found in App. E.2.
2. Preliminaries
2.2. Targeted Data Selection
Denote an LLM parameterized by 0 as f(; 0). Let x, y be the random variables (RVs) for the input (prompt) and the output (response) of the LLM, respectively. Let y/ be the RV for a single-token output. Let DN = { (z,, ) denote the training set where z; consists of the prompt 2; (a sequence of words or tokens) and the label response y4 (the
The objective of data selection is to identify an optimal sub- set Ds C D2 such that training model f on Dg achieves comparable or superior performance on downstream tasks
'In this paper, we examine several datasets, including AlpacaB- val, which exhibit this characteristic. Additionally, we investigate datasets that lack explicit subtasks where q = 1.
2

---

Data Selection with NICE
compared to training on the full dataset. It is achieved by se- lecting training data that maximizes the performance of the larget task's 's validation set Dv. thereby enhancing model performance on target tasks.
as follows:
r (Tos Mos Mos Bo)
when yo is required, e.g.,
LLM judge in (Dubois et al., 2024);
r(zu. Do) =
r Do)
when yo is not required, e.g..
Loss-based influence estimation methods quantify the influ- ence of the individual training data point on validation loss. The influence scores are typically higher for training data which reduces the validation loss more. When loss serves as a proxy for the validation performance, higher scores indicate greater helpfulness for the target task when they are included in the training. To apply influence estimation for data selection, it is necessary to aggregate the scores across the validation set, which may consist of multiple subtasks. Specifically, the influence score of a training data point 2i on each subtask is first computed by averaging the influence scores across the validation data within that subtask. The overall influence for the validation set Dv is then calculated as the maximum influence score across all subtasks:
reward model in (Bai et al., 2022).
Note that the ground truth response ye is not always required by the reward function, depending on the evaluation metric used here. We will use the terms "reward function" and "evaluation metric" interchangeably in the rest of our paper.
To calculate the influence of a training data point on non- differentiable r, we cannot directly apply the same formula as the loss-based influence estimation such as TracIn or IF, because they require the gradient of r(z, No) w.r.t. the model parameters, which is not available. To address this, we propose to use the policy gradient from RL (Wu et al., 2018; Sutton & Barto, 2018). Specifically, we adopt the RL objective function for validation data point.?
Inf(zi Dy) max; |DV1 Ex.epfo Inf(z;, zo)
I,, (Zu: 0) Egj. (gyle,))) (r(z,, Bo)]
where Inf ze) denotes an influence estimation (such as InfTractn Infir) that aims assign higher scores to more helpful training data points. The of the max function ensures that training data improving performance on at least one validation subtask are prioritized (Xia et al., 2024). Based on these scores, the top-ranked training data points are selected to construct the training subset Ds. This subset then used to fine-tune the target model.
where f(: 0) denotes the policy defined by the LLM with pa- rameter 0, which is used to generate response d. for z.. Sub- sequently, the policy gradient of L, w.r.t. the model param- eters can be derived using the log derivative trick (Williams, 1992; Meyer, 2023):
VeL, (zpi 0) = Eg.-foll..)))) |- Ve log(s ( ())r (F) Bo)l
This can be estimated using Monte-Carlo sampling on the responses generated by fzz: 0), a technique also known as the Monte-Carlo (MC) policy gradient. Intuitively, policy gradient optimizes the model by increasing the probability of generating responses with high and positive rewards. By using the policy gradient, the influence of a training data point 2; on the model performance on a validation data point z, measured by the reward function r at time step / calculated as
3. Methodology
3.1. Non-differentiable evaluation metric-based InfluenCe Estimation (NICE)
boss-based influence estimation methods quantify the effect of a training data point on the validation loss (e.g., NTP loss), which is a diffcrentiable proxy for the validation per- formance. However, there are two major drawbacks of loss- based influence estimation: (a) There is discrepancy be- tween the NTP loss and the evaluation metrics of instruction- following tasks, especially those that require long-form gen- erations (c.g., LLM-judge (Dubois et al., 2023) and code generation benchmarks (Chen et al., 2021)). In other words, selecting training data that minimizes NTP loss on valida- tion data does not necessarily improve the performance for these tasks (as shown in Fig. 1). (b) While obtaining the prompt of a validation data point T. is relatively easy, the high-quality label yo may not always be available.
Lz
- (VoLizz 0'), (zoj 0²))
The approximation above is derived using similar logic as Eq. I in App. E that uses the first-order Taylor approxima- tion. Then, we can measure the non-differentiable evalua- tion metric-based influence of z, on z 's performance over the entire training run as
Inf (2,, 20) Ee= To (V
/g [- log( ((nv (In; 0c ())))))))) dv)l)
We propose to directly compute the influence of each train- ing data point on the evaluation metric instead of the loss. Specifically, denote the reward function as r (defined by an evaluation metric), which calculates the model performance
"W use a negative sign in front of r to make the notations in the influence estimation in the rest of our paper consistent with loss-based influence estimation, i.c. the lower the [ze; 0), the better the model (consistent with validation loss).
3

---

Data Selection with NICE
To interpret, NICE assigns higher influence score to training data point 2; if its gradient (of the training loss) is more similar to the policy gradients of the validation performance evaluated by the reward function r. The higher the influence score, the larger the performance measured by the evaluation metric improves when including training data point ž5. To apply NICE for data selection, the same aggregation of the influence scores in Sec. 2.2 is applied. We then select the data subset D3 by including the training data points with top-ranked aggregated influence scores.
by first quantifying the influence of the training data on the parameter, then using the chain rule to calculate the impact of this influence higher InfNICEP means a larger increase in L, and hence a larger decrease in performance measured by when down- weighting the training data (i.e., removing the data point makes the model perform worse), indicating a higher quality of that training data point. The same aggregation method described in Sec. 2.2 is used for aggregating INfNI
Lr at the validation data. A
Our implementation uses similar approach as DataInf to improve the efficiency of NICEIF by using the first-order derivatives to estimate the Hessian inverse, which is required in the calculation of the IF (Kwon et al., 2024).
To summarize, NICE enables data selection to directly opti- mize the oon-iffferentiable evaluation metrics via influence estimation using policy gradient. It also enables data selec- tion with unlabeled validation data when the metric does not require the label as input.
3.3. Assisted Monte-Carlo Sampling
Although there are various alternatives to compute the gra- dient of L. from policy optimization research (Schulman et al., 2017; Rafailov et al., 2023), we use the MC policy gradient as it is easy to implement and has been shown effective in many applications. We further demonstrate in App. 1.1 that gradients computed by other policy optimiza- tion approaches can also be used in NICE achieve better performance than loss- based influence estimation.
We use MC sampling to estimate the policy gradient used in NICE. Recall that NICE uses the MC policy gradient. Specifically, for a prompt 2. of a validation data point, we sample multiple responses from the LLM and use the sample mean to estimate the policy gradient. There are two major advantages of using this MC policy gradient compared to the gradient used in loss-based influence estimation: 1) Pol- icy gradient estimated using MC utilizes multiple different responses, offering diverse guidance compared to the label response; 2) The generated response can be better than the label response (as demonstrated in Tab. where the label response is incorrect and less detailed) and hence result in better data selection performance.
Remark 3.1 (Equívalence to TracIn). When the label re- sponse consists of single token and the evaluation metric is accuracy, NICE is equivalent to TracIn. Thus, for tasks that do not require generating long responses, vanilla loss- based influence estimation performs similarly to NICE. Remark 3.2 (Empirical consideration). To improve the per- formance of NICE, we integrate the two improvements proposed by LESS (Xia al., 2024), which adapts TracIn for influence estimation on LLM. Specifically, we use the Adam gradient for training data instead of the SGD gradient and replace the inner product with cosine similarity (i.c., equivalent to normalizing the gradient before the inner prod- uet) in the definition of influence to mitigate the bias toward short sequences. The explicit form of NICE used in our implementation is elaborated in App. F.
Despite these advantages, the MC policy gradient has its practical limitations. Specifically, when the model (policy) is too weak for the task, the MC samples may not contain high-quality responses with high rewards. As a result, the corresponding estimated policy gradient will not contain signals for improving the policy's performance. To ensure the quality of the generated responses, we propose an alter- native approach named assisted Monte-Carlo (AMC) sam- pling, which uses a model ((;;,) ()) parameterized by 1a that is better at the target task to assist the response generation:
3.2. Generalization to Other Loss-based Influence Estimation Methods
Inf InfNICEAMe (21, 20) = Ee To (VoL(2;;
Eg:~o(1(x.:)) [- V logo(f((( (0] 8*)) r(ze. 0:31)
We have discussed the use of policy gradient specific loss-based influence estimation method - TracIn to esti- mate the influence of data points on the non non-differemtiaalee evaluation metrics. However, our approach is not limited to TracIn alone. The policy gradient can be applied to other methods, such as the influence function:
The generated responses j ~ g( v)) with the assistance from better-performing LLMs have better quality assurance than vanilla MC sampling (Tab. 1). With AMC sampling, NICEAMC is more likely to select a data subset that guides the model to generate better responses, improving the per- formance.
InfNICEr 20) = V (zu; BB) Hoe (oL(z;
4. Experiments
where E is the number of epochs and VoL, R8) Eg.- Egg,-(pla, I- V (jg(f(jj( (r [. The
We experimentally analyze model performance under two different procedures for forming the initial training data
derivation follows from a similar logic as Eq. App. E
4

---

Data Selection with NICE
Table Generated respouses from different models for a validation data point in AlpacaEval task. Llama2-7B or Mistral-7 is model f used in NICE. GPT4 is the betteer-erformen
LLM used in NICEAMC. As the model improves, its generated response becomes more detailed and can better than the label response (the label response is incorrect while Mistral-7B and GPT's responses are correct). If the endpoints of a line segment are (2, -2) and (10, 4), what is the length of the segment?
Question:
Label Response: The length of the line segment is 12 units.
Llama2-7B Generated Response: The length of the segment is 10 - (-2) = 12 units.
Mistral-7B Generated Response: 'The length is the distance between the endpoints. To find this distance, use the following formula: (final x init x²² + (final y - init 2)² = distance?
In this case, the values are (10, 4) and (2, 2). The length is sqrt(8² + 6²) = sqrt (00) 10.
GPT4 Generated Response: To find the length of a line segment given its endpoints, you can use the distance formula. The distance formula between two points (x1 y1) and (#2: W2) in the coordinate plane is: d v(02 x1)² + (y2 )1) For the
points given, (1). = (2,-2) 22) and (22; 1/2) = (10, 4). Plugging these into the formula: d = /110- 2) + (4 - (-)))² d V8² 62
v 64 + d <100 d 10. Therefore, the length of the line segment is 10 units.
pool: task-agnostic and task-a We evaluate our method on tasks that require long-form generation and use evalua- tion metrics that are commonly used evaluate the corre- sponding tasks. In this section, we describe the evaluation tasks (i.e., the task that we wish to improve on via instruc- tion tuning) in Sec. 4.1, experimental settings in Sec. 4.2, experimental details in Sec. 4.3, main results in Sec. 4.4, and additional analysis in Sec. 4.5.
tains polished text summaries. The evaluation metric is the reward model (OpenAssistant, 2023) trained on human feed- back, measuring the the quality of summaries and alignment with human preference. RLHF consists of prompt-responss pairs where each includes a "chosen" response that aligns better with human preferences (we use only the "chosen" columns). We use a trained reward model (Ray2333, 2024) as the evaluation metric to evaluate the helpfulness of the responses. For the dataset splits, we randomly select 10 examples from both AlpacaEval and HumanEval as the val- idation set, with the remainder as the test set. For RLHF, we sample 5% from the training dataset as the validation set since the original dataset only contains train and test splits. For TLDR, we use 5% of the original validation set, given that the original validation set is too large. We study the robustness of our method to different validation splits in App. H.2. We provide a summary of these evaluation tasks in Tab. 2 and additional details these tasks in App. B. Unless specified, the results reported are evaluated on the test set.
4.1. 1. Evaluation Tasks
We use four evaluation tasks, including AlpacaEval (Dubois et al., 2024), TLDR (Stiennon et al., 2020), RLHF (Bai et al., 2022), and HumanEval (Chen et al., 2021).
Table 2. Detailed information about the evaluation datasets.
Dataset
|Dy]
[Deeal
Evaluation metrics
AlpacaEival
10
795
length-controlle win rate
TLDDR
322
6553
reward model
RLHF
2192
2354
reward model
HumanEval
10
154
pass@ k:
4.2. Experimental Settings
AlpacaEval is a compilation of prompt-response pairs aimed at assessing language models' instruction following capability. We use the ennghh-oontrolled win rate to ensure fair evaluation (Dubois al., 2024). HumanEval evaluates code generation from natural language instructions using the pass @ k metric, which is the probability of having at least one correct solution (pass specified unit test) when sampling k responses from the model. @ with & > is practical in real-world scenarios when expected behaviors are known and test cases are readily available. It evaluates the test-time scaling capability of the model by allowing multiple candidate solutions (k) to be generated and val- idated to effectively identify the correct code. We adopt pass @ 100 because it achieves substantially higher accuracy than smaller l (Chen et al., 2021), making it more practical for real-world scenarios. For completeness, we also provide detailed results for different h in App. H.1 TLDR con-
Our problem setup focuses on targeted data selection, as- suming access to validation data during data selection. We further consider two distinct settings where we perform selection from "a aask-agnostic training data pool" and "a task-aware training data pool", distinguished by whether the knowledge of the downstream task available when forming the initial training data pool before data selection. Specifically:
Selection from a task-agnostic training data pool. In this setting, a large, diverse, mixed-source pool of instruction tuning training set is collected without the knowledge of the downstream task before data selection. We use Tulu (Wang et al., 2023) as the training dataset, which consists of Chain of Thought (COT) (Wei et al., 2022), Databricks Dolly (DOLLY) (Conover et al., 2023), Open Assistant 1 (OASST) (Köpf et al., 2023), and FLAN V2 (Longpre et al.,
5

---

Data Selection with NICE
2023). Intuitively, this mixed-source pool of data may con- tain irrelevant data (e.g., assistant-style conversations) w.r.t. the targeted task (e.g., coding task). Additional details for the training set are in App. A.I.
points are randomly sampled from the training set for in- struction tuning. We also employ BM25 (Robertson et al., 2009), which ranks training data based on relevance to the validation data, and then selects the top-ranked data points to form Ds. Another baseline, DSIR (Xie et al., 2023), se- lects Dg based on n-gram lexical feature matching between training and validation distributions. Representation- Data Selection (RDS) (Zhang et al., 2018) ranks the train- ing instances using the cosine similarity of the features between the training and the validation data, and we adopt BRTT (Reimers, 2019) embedding as the features. LESS (Xia et al., 2024) uses loss- influence estima- tion to select the training data with top influence scores. TSDS (Liu et al., 2024b) also leverages loss-based gradient features, and further optimizes for distribution alignment and diversity via optimal transport and kernel density esti- mation, respectively. Note that all baselines, except random, are calculated using complete data points (i.e., the con- catenation of prompt and response). More implementation details about the baselines are in App. D.
Selection from a task- training data pool. In this setting, the training set is collected specifically for the down- stream evaluation task, hence being task awae. We consider two evaluation tasks here: RLHF and HumanEval, with the evaluation datasets the same in Sec. 4.1. For RLHF, the training data is 95% of the original training set used for the helpfulness assistant, as provided in Bai et al. (2022). For HumanEval, we adopt the CodeAlpaca 20k (Chaudhary, 2023) dataset as the training set, which is a growd-souredd collection of code-related instruction-responee pairs, de- signed to fine-tune language models for better performance in code generation and understanding. Additional training set details are in App. A.
4.3. Experimental Details
Efficient Data Selection for LLM. To improve the effi- ciency of data selection, we train models with LoRA (Low- Rank Adaptation). We adopt the warmup training which trains the LLM on randomly selected subset of training data for influence estimation and the number of warmup epochs the F. Additionally, random projections are ap- plied to the LoRA gradients, preserving the essential inner products while reducing the dimensionality of the gradient to reduce the memory requirement (Johnson, 1984).
4.4. Results
The results of NICE and NICEAMC against baselines are presented in Tab. 3. We summarize the findings below.
NICE Outperforms Loss-based Influence Estimation. Our results show that NICE consistently outperforms LESS, which uses loss-based influence estimation for data selec- tion, across various models, settings, and datasets. This result verifies that using the estimated influence on the eval- uation metric is more helpful for data selection than that on the loss.
Models and Hyperparameters Our primary evaluation of NICE focuses on two LLMs: Llama2-7B (Touvron et al., 2023), and Mistral-7B (Jiang et al., 2023), with performance averaged over three seeds. Larger and state-of-the-ee mod- els, including Llama2-13B and Llama3-8B, were also tested on the RLHF task, presented in App. H.3. We perform warmup training for 4 epochs on 5% of the training set for the task-agnostic training data pool, and 20% for the task- aware training data pool due to its smaller training set size. We project the LoRA gradient into an 3192-dimensional vector. The influence estimates for data points are obtained by the respective data selection approaches, with the top 5% data points (ranked by influence) to be the selected data sub- set Ds for the task-agnostic setting (20% for the task-aware setting). Multinomial sampling (Chatterjee & Cancedda, 2010) is used to generate the MC samples for NICE. We generate 20 MC samples for all evaluation tasks, except for HvmanBval where we generate 500 samples due to the difficulty of the task (i.c., responses having low code pass rates). For NICEAMC, we use GPT-4 16). Note that when the reward function does not require labels, the ground truth label is not used by our approach, detailed in App. C.
No Labels? No Problem! NICE Outperforms Base- lines that Utilize the Label Response. For tasks like TLDR, RLHF, and HumanEval, NICE NICEAMC uses only unlabeled validation data (i.e., only prompts). Surpris- ingly, they outperform baselines that use labeled data (both prompts and label responses).
Less Is More: Subset Outperforms the Full Dataset. We find that the subset selected by NICE or NICEAMC can outperform the full dataset, demonstrating the value of carefully curated data over larger, less refined datasets.
Assisted Monte-Carlo Sampling Can Boost Data Selec- tion. Adopting AMC sampling in policy gradient has the potential to further improve the performance of NICE, es- pecially when the initial pool of training data is large. Re- sponses generated from better- perttrrperformng models can effee- tively guide data selection, enabling models trained on the selected subset to achieve better performance.
However, NICEAMC does not always have improved per- formance compared to NICE, particularly under a smaller selection pool of training data (e.g., a task-aware training
Baselines. We evaluate NICE against a variety of baselines. The most straightforward baseline is Random where data
6

---

Data Selection with NICE
Table Comparison of NICE and NICEAMC in both rask-agnostic and task-aware settings for Llama2-7B and Mistral-7B. Bold numbers indicate the top-performing selected subset. A purple cell suggests that NICE outperforms LESS which uses based influence estimation. Underlined numbers show that the subset selected by our approach exceeds the performance of the full dataset. Subscript
numbers represent standard deviations.
Model & Dataset
Full
Random
RDS
BM25
DSIR
TSDS
LESS
NICE
NICEAMC
Task-agnostic training data pool.
AlpacaEval
22.59
16. 16.13
14.70
19.60
20.27
17.40 17.40+2.44
26.94
27.61+2.13
30.45 22.40
TIDRR
2.40
1.80
2.08
2.15
1.53
2.19
3.37
3.61 20.78
3.55
Llama2-7B
RLHF
2.31
2.05
1.87
2.83
2.57
=.01
1.4440.07
2.82++0.0 1010
3.03 L0.02
HumanEval
47.44
44.30 44.30
45.29
46.19
42.22
43. 43.68+1.82
47.50
48. 18.5994..0m .08
45. 45.10
AlpacaEval
33.77
24.99
21.70
28.47
29.31
35.84
1..09
43.00
47. 47.40+4.94
TLDRR
2.79
3.06
2.90
2,41
3.48
3.28
4.40
4.80_o.12
4.59
Mistral-7B
RLHF
2.56
2. 2.13
1.78
2.88
2.94
1.83=0:15
1.70+4009
3.10 06
3.42 20.05
HvmanEval
83.63
85.56
84.15
84.09
79.17
82.78+1.25
85.24400.5
85.59 1.41
85.67 40.34
Task-aw aware training data pool.
Llama2-7B
RLHF
1.01
1.0120.04
0,66
1.29
1.43
0.97
1.62
1.69 1.69_2..5
20.05
1.32
HvaanEval
51.27
51.91
54.74
52.23
53.10
19.85
52.67
55.00 1.86
50.67
RLHF
0.99
1.05
0,56
1.31
1.31
1.15
1.29
1.7 =0.01
1.35
Mistral-7B
HumanEval
84.27
83.3442.54
86.75
84.81
79.91
85.51
85.2641.13
87 41.35
84.1841.63
data pool often smaller). When the training set is small, there may not be enough training data points with gradi- ents close to the policy gradient of NICEAMC. We per- form a simple experiment to verify this intuition: restricting NICEAMC to compute score from the RLHF training set yields a performance of 1.26, but expanding the selection pool to the combination of RLHF set and a large instruction tuning set (COT, DOLLY, OASST, Flan V2) increases the performance to 3.35. Note that the additional controlled ex- periment only expanded the selection pool without altering the initial warmup process. In contrast, the performance of NICE improves from 1.68 to 2.44 with a larger pool. This comparison shows a clear advantage of NICEAMC when the training data to select from is large in size. Consequently, when the size of training data is large (e.g., task-agnostic training data pool), we can prioritize using NICEAMC.
Table 4. Comparison of asymptotic time complexity and wall-clock time (in GPU hours) for each stage in data selection. The time for Warmup training with LoRA is measured on H100, and the others are measured on L40.
Remark
Asymptotic
Compute
Warmup Training Training Grad
NICE LESS NICE = LESS
O( (E)) O((Da(E)
48h
LESS: O( Dy (E);
LESS: 0,1 on avg: NICE: 14.67h avg <0.02h
Validation Grad
NICE > LESS
NICE: O([Dy|EM)
Data Selection
NICE= LESS
Table S. Validation gradient computation time across tasks for NICE and LESS single L40 GPU hours.
NICE
NICE
LESS
Task
M
(MC Sumpling)
(Val Grad)
(Val Grad)
AlpacaFival (DDy) - 10)
20
0.17h
0.05h
<0.02h
TLDR (Dyy = 322)
20
8h
1.47h
0.08h
RLJI ([De) 2192)
32h
10h
0.33h
HumanFval (IDv)
500
5h
2h
<0.02h
4.5. Additional Analysis
Unless specified, the experiments in the section below are conducted on the Llama2-7B model.
dimension of the projected gradients, and M the number of Monte Carlo (MC) samples. Let [Dxl, and Dvv denote the warmup. training, and validation set sizes, re- spectively. When [Dy] and M are small, NICE adds only marginal overhead to LESS (e.g.. AlpacaEval).
Time Complexity Analysis. We provide a comparative analysis of the computational costs between NICE and LESS, an approach that adopts loss-based influence estima- tion, showing that NICE remains within a practical com- putational range. Tab. 4 lists the asymptotic complexity and wall-clock runtime (the time for warmup training with LoRA is measured in single H100 GPU hours, others are measured in single L40 GPU hours) for each stage in the data selection procedure. Tab. 5 highlights the validation gradient computation where NICE differs from LESS. Let E denote the number of epochs (saved checkpoints), the
While NICE utilizing Monte Carlo sampling can indeed increase the computational cost, this trade-off is justified by our approach not needing validation labels- key mo- tivation of our work. NICE fills a gap left by existing loss-based basclines by supporting data selection with unla- beled validation data in cases where the evaluation metrics are label-indeeenddent Furthermore, we can observe the performance improvement over other methods in Tab. 3.
7

---

Data Selection with NICE
Generalizing NICE to the Influence Function (IF). We demonstrate the effectiveness of the main idea of NICE be- yond the TracIn framework by extending it to IF. We adopt Datalnf as an efficient implementation for IF. We compute the Datalnf on the last checkpoint of the warmup model and adopt random projection to reduce the dimensionality of gra- dients and store the projected gradients, eliminating the need for computing the gradients again after the computation of the Hessian. More details on the implementation of Datalnf are provided in the App. E.3. We compared the performance of selecting data with the vanilla (loss-based) Datalnf and the DataInf enhanced by the policy gradient: NICEIF when the training data pool is task-agnostic (Sec. 4.2). As shown in Tab. 6, NICEIF consistently outperforms DataInf, demon- strating the effectiveness of our approach and showing that the concept of NICE of applying policy gradient for influ- ence estimation can be readily applied to other loss-based influence estimation methods to improve the performance.
2,8
Negative Loss
-1.724
Reward
2.7
-1.726
2,0
-1.728
250
300
350
400
Step
Figure I. Discrepancy between NTP loss and performance (i.c., measured by the reward model here) of the validation set for RLHF task in the last few training steps. The checkpoint with the mini- mized loss (highest negative loss) corresponds to the checkpoint with relatively worse performance (having the lowest reward among the five checkpoints). The performance may continue to increase even if the loss increases (negative loss decreases).
and the number of MC samples, which shows the potential to further improve NICE by using more MC samples. How- ever, generating additional MC samples is computationally expensive. We use 20 MC samples for the majority of tasks since it is relatively less computationally expensive, while sufficient to achieve better performance than other baselines. The number of MC samples can also affect the stability of our approach, as elaborated in App. H.5.
Table Comparison between Influence Function (DataInf) and NICEIF on Llama2-7B when the training data pool is task- agnostic. NICEIF consistently outperforms the DataInf.
Method
AlpacaEval
TLDR
RLHF
HumanEval
IF (Datainf) 11.11
2.01
0.83
37.40
NICElF 20.44
3.97
1.89
39.68
Data Addition. In Fig. 4, we plot the performance (mea- sured by reward) against the percent of data points selected by NICE and Random for RL.FF task. The performance of the model trained on a randomly selected subset increases gradually as more data is used. In contrast, for the NICE- selected subset in the task- aask- setting (right of Fig. 4), performance rises slightly from 5% to 25% but drops sharply beyond 25%. The task-aware training data pool is more rele- vant to the downstream task, so a small percentage can miss useful data, while a large percentage can include irrelevant or harmful data, harming performance. When the training data pool is task-agnostic, performance declines as the se- lection percentage increases, likely because only a small fraction (no more than 5%) is relevant. This experiment also demonstrates the importance of data selection to exclude data points that are not useful to model performance.
The Discrepancy Between NTP Loss and Evaluation Metrics. Previous works have discussed the discrepancy between validation loss and downstream performance in in- struction tuning for LLMs (Tay et al., 2021; Xia et al., 2024). In these scenarios, minimizing validation loss does not nec- essarily correspond improving validation performance, especially when the task requires long-form generations. Empirical observations described in Fig. 1 and further re- sults in App. H.4 verify this discrepancy: The minimized validation loss is achieved at step around 250. However, that checkpoint is the worst- corst-performing checkpoint (lowest validation reward). The reward can be further increased in later steps, even if the loss increases.
We additionally plot the validation performance, measured by the reward of the last few checkpoints, for models trained on NICE-selecter subsets and LESS- seleetel subsets in Fig. 2 for the RLHF task. NICE-selected data optimizes in the direction of increasing validation performance, whereas LESS-selected data prioritizes loss reduction, which may improved
5. Related Works
Various approaches are proposed to estimate the influence of training data in fine-tuning LLMs. LESS (Xia et al., 2024) adapts the TracIn framework to estimate the influence of data points in instruction tuning. Kwon et al. (2024) and Choe al. (2024) scale up the IF by speeding up the computation of the Hessian inverse. Lin et al. (2024) stud- ies the token-level influence function for LLMs. These approaches above are all loss-based influence estimation methods that aim to approximate the influence of data on the validation loss. Consequently, they fall short for genera-
not necessarily lead to improved validation performance.
The Effect of the Number of Monte-Carlo Samples. We perform additional experiments to study the effect of the number of Monte-Carlo (MC) samples used in approxi- mating policy gradient on the data selection performance (measured by reward for the RLHF task). The results in Fig. 3 indicate a positive correlation between performance
8

---

Data Selection with NICE
Task- aakkosics:ic: NICE
Task-agnostic: LCSS
Irend (Slope: 5. 5.25e-04)
Trend (Siope: 1 2222-05
Reward
Reward
1.4
225
250
275
325
350
400
225
250
275
300
325
350
375
400
Step
Step
Task-aware: NICE
Task-aware: LESS
1.65
1.70
Thend (Slope: 3.34
Thend (Slope:
Reward
Reward
1.60
1.64
1.55
1.62
140
160
130
200
220
240
140
150
180
200
220
240
Step
Step
Figure 2. Validation performance, measured by the reward model of the last few checkpoints, for models trained subsets selected by NICE and LESS for the RLHF task. NICE-selected data optimizes for improved validation performance, whereas LESS-selected data focuses loss reduction, which may always enhance validation performance.
Task-agnostic
Task-aware
agnostic
Task-aware
Reward 2
Rewaro
Reward
Rewand
1.50
40
60
80
100
20
40
60
80
100
10
Number samples
Number samples
25
50
75
100
75
100
Original Values
Fitted Trend
LESS Baseline
Percent
Points Used NICE
Percent Points Used (%)
Random
Figure 4. Performance versus percent of data points selected by NICE Random for the RLHF task. As the percentage selected by NICE increases, performance may drop due to the inclusion of low-score, irrelevant, harmful points. When using data lection, the trained model consistently outperforms those trained on randomly selected data, even outperforming the full dataset, underscoring the importance of data sclection.
Figure Performance of LLMs trained on data selected by NICE for RHLF task when different numbers of MC samples are used. There a positive correlation between performance and generated MC samples. Using the sampling size of 20 provides good per- formance while increasing the sampling size has the potential to improve the performance.
tions tasks whose evaluation metrics align poorly with NTP loss, as elaborated in Sec. 4.5. On the other hand, the works of Kwon et al. (2024); Choe et al. (2024); Lin et al. (2024) focus on data attribution and hence do not optimize for data selection. In addition, Park et (2023) approximates the data model estimate the influence; Wang et al. (2024a) scales up the Shapley value to estimate the influence; and Qiao et al. (2025) uses the influence function to reweight the training samples. However, these methods only focus on classification tasks. contrast, our work focuses on the influence estimation for tasks that require extensive genera- tions. More related works on data curation for instruction tuning discussed in App. G.
than relying NTP loss. When using a reward function that does not require label response, NICE can perform data selection without relying on costly annotated labels. Experimental results show that our approach outperforms cxisting data selection methods across diverse scenarios. Of note, despite the superior performance achieved by NICE and NICEAMC, the computational cost of these approaches is not negligible of LoRA and random projection). Further explorations can be done to study other computationally efficient ways of computing the gradients.
with the acceleration (e.g., the use
6. Conclusion
We propose NICE, a novel influence estimation ap- proach that selects training data to directly optimize non- differentiable evaluation metrics via policy gradient, rather

---

Data Selection with NICE
Impact Statement
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in neural information processing systems, 2020.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Bukharin, A. and Zhao, T.
Data diversity matters for ro- bust instruction tuning. arXiv preprint arXiv:2311.14736. 2023.
Acknowledgments
This research/projectii is supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: AISG-NMLP- 2024-001). Any opinions, findings and conclusions rec- ommendations expressed in this material those of the author(s) and do not reflect the views of National Research Foundation, Singapore. This research is supported by the National Research Foundation Singapore and the Singapore Ministry of Digital Development and Innovation, National AI Group under the AI Visiting Professorship Programme (award number AIVP-2024-001). This research is supported by the National Research Foundation (NRF). Prime Min- ister's Office, Singapore, under its Campus for Research Excellence and Technological Enterprise (CREATE) pro- gramme. The Mens, Manus, and Machina (M3S) is an in- terdisciplinary research group (IRG) of the Singapore MIT Alliance for Research and Technology (SMART) centre. Jingtan Wang supported by the Institute for Infocomm Research of Agency for Science, Technology and Research (A*STAR). We would like to thank the anonymous review- ers and AC for the constructive and helpful feedback.
Cao, Y., Kang, Y., Wang, C., and Sun, L. Instruction min- ing: Instruction data selection for tuning large language models. arXiv preprint vrXiv:2307.06290 2023.
Carlini, N., Jagielski, M., Choquette-Choo C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., and Tramèr, F. Poisoning web-scale training datasets is practical. In 2024 IEEE Symposium Security and Privacy (SP), pp. 407-425, IEEE, 2024.
Chatterjee, S. and Cancedda, N. Minimum error rate train- ing by sampling the translation lattice. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 606-615, Cambridge, MA, Oc- tober 2010. Association for Computational Linguistics.
Chaudhary. S. Code alpaca: An instruction model for code generation. GitHub repository, 2023.
llama
Chen, L.,, Li, S., Yan, J., Wang. H., Gunaratna, K., Yadav, V., Tang, Z., Srinivasan, V., Zhou, T., Huang, H., et al. Alpagasus: Training a better alpaca with fewer data, arXiv preprint arXiv:2307.08700 2023.
References
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, D. Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377. 2021.
Albalak, A., Elazar. Y., Xie, S. M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H., Raffel, C., Chang, S., Hashimoto, T., and Wang. W. Y. A survey on data selection for language models. Transac- tions on Machine Learning Research, 2024. ISSN 2835- 8856. URL https: http///eeeevvieeeee n forum? id 1d=XEHMcNTSHp. Survey Certification.
Choe, S. K., Ahn, H., Bae, J., Zhao, K., Kang, M., Chung, Y., Pratapa, A., Neiswanger, W.. Strubell, E., Mitamura, T., al. What is your data worth to GPT? LLM--scale data valuation with influence functions. arXiv preprint 2405 13954, 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05882. 2022.
Conover, M., Hayes, M., Mathur, A., Xic, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free dolly: Introducing the world's first truly open instruction- tuned 1lm. Company Blog of Databricks, 2023.
Bartlett, M. Approximate confidence intervals. Biometrika, 40(1/2):12-19, 1953.
Du, Q., Zong, C., and Zhang, J. Mods: Model-oriented data selection for instruction tuning. arXiv preprint 1.Xi::331.15666 2023.
Bhatt, G., Chen, Y., Das, A., Zhang, J., Truong, S., Muss- mann, S., Zhu, Y., Bilmes, J., Du, S., Jamieson, K., et al. An experimental design framework for label-efficient su- pervised finetuning of large language models. In Findings of the Association for Computational Linguistics ACL 2024, pp. 6549-656655 66660,6600000 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
10

---

Data Selection with NICE
Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C.,, Liang, P. S., and Hashimoto, T. B. Alpacafarm: A simulation framework for methods that learn from human feedback. In Advances in Neural Infor- mation Processing Systems, pp. 30039-30069, 2023.
Boosting LLM performance with self-guided data selec- tion for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Associ- ation for Computational Linguisties: Human Language Technologies (Volume I: Long Pupers), June 2024a.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.044.
Li, Y., Hui, B., Xia, X., Yang, J., Yang, M., Zhang, L.,, Si, S., Chen, L.-H., Liu, J., Liu, T., Huang, F., and Li, Y. One-shot learning as instruction data prospector for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics (Volume I: Long Papers), pp. 4586 4601, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 1
2024.
Frénay, B. and Verleysen, M. Classification in the presence of label noise: survey. IEEE transactions on neural networks and learning systems, 25(5):845 869, 2013.
Gloeckle, F., Idrissi, B. Y., Roziere, B., Lopez-Paz, D., and Synnaeve, G. Better and faster large language models via multi-token prediction. In Forty-first International Conference on Machine Learning, 2024.
Lin, H., Long, J., Xu, Z., and Zhao, W. Token- influ- ential training data retrieval for large language models. arXiv preprint arXiv:2405.
2024.
Liu, W., Zeng. W., He, K.,, Jiang, Y., and He, J. What makes good data for aligmment? comprehensive study of auto- matic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations, 2024a.
Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., et al. Study- ing large language model generalization with influence functions. arXiv preprini arXiv:2308.03326, 2023.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.099 09685, 2021.
Liu, Z., Karbasi, A., and Rekatsinas, T. TSDS: Data se- lection for task-specific model finetuning. In The Thirty- eighth Annual Conference on Neural Information Pro- cessing Systems, 2024b.
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effec- tive instruction tuning. In International Conference on Machine Learning, pp. 22631-226444 PMLR, 2023.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. 1., Bressand, F., Lengyel. G., Lample, G., Saulnier, I.., et al. Mistral 7b. arXiv preprint mxXiv:2310.06822 2023.
Johnson, W. B. Extensions of lipshitz mapping into hilbert space. In Conference modern analysis and probability, 1984, pp. 189-206, 1984.
Lu, K., Yuan, H., Yuan, Z., Lin, R., Lin, J., Tan, C., Zhou, C., and Zhou, J. #instag: Instruction tagging for analyz- ing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Repre- sentations, 2024.
Koh, P. W. and Liang, P. Understanding black-box predic- tions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Martens, J. et al. Deep learning via hessian-free optimiza- tion. In Icml, volume 27, pp. 735-742, 2010.
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D., Stanley, O., Nagyfi, R., ES, S., Suri, S., Glushkov, D., Dantuluri, A., Maguire, A., Schuhmann, C., Nguyen, H., and Mattick, A. Openassistant conversations democratizing large language model alignment. In Advances in Neural In- formation Processing Systems, pp. 47669-47681 Curran Associates, Inc., 2023.
Meyer, D. Notes on policy gradients and the log derivative trick for reinforcement learning, 2023. Lecture Notes.
OpenAssistant.
Reward model deberta
v3 large v2, 2023.
URL https://
huggingface co ctt.://////o///e hhop/ppenssistant// rewa rd-adel -ee--e-deeeerb--aaeeeee Accessed: 2025-01-06.
Kwon, Y., WU, E., Wu, K., and Zou, J. DataInf: Efficiently estimating data influence in lora-tuned LLMs and diffu- sion models. In Proc. ICLR, 2024.
Ouyang, I., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In Proc. NeurIPS, pp. 27730- 27744, 2022.
Li, M., Zhang, Y., Li, Z., Chen, J., Chen, L., Cheng, N., Wang, J., Zhou, T., and Xiao, From quantity to quality:
11

---

Data Selection with NICE
Park, S. M., Georgiev, K., Ilyas, A., Leclerc, G., and Madry, A. TRAK: Attributing model behavior at scale. In Proc. ICML, 2023.
Wang, J., Lin, X., Rui, Q., Chuan-Sheng, F,, and Bryan Kian Hsiang, L.. Helpful harmful data? fine-tuning- free shapley attribution for explaining language model predictions. In Proc. ICML, 2024a.
Pruthi, G., Liu, F., Kale, S., and Sundararajan, M. Estimat- ing training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems, volume 33, pp. 19920-19930. Curran Associates, Inc., 2020.
Wang, P., Shen, Y., Guo, Z., Stallone, M., Kim, Y. Golland, P., and Panda, R. Diversity measurement and subset selection for instruction tuning datasets. arXiv preprint arXiv:2402.02311 2024b.
Qiao, R., Wu, Z., Wang. J., Koh, W., and Low, B. K. H. Group-robust sample reweighting for subpopulation shifts via influence functions. arXiv preprint arXiv:2503.07315, 2025.
Wang. Y., Ivison, H.,, Dasigi. P., Hessel, J., Khot, T., Chandu, K., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36.74744-74786. 2023,
Rafailov, R., Sharma, A., Mitchell, E.,, Manning, C. D.,
Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E., Le, Q. V., and Zhou, D. Chain-of-thouglte prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Informa- tion Processing Systems, volume 35, pp. 24824-24837. Curran Associates, Inc., 2022.
Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Ray2333. Gpt-2 large harmless reward model. https: httpss//hhugginaaceeec Co/Ray2333/ 9pt2-1arge-ha haechharmlleeee rmless reward_ 2024.
Williams, R.
Simple statistical rithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
algo-
Accessed: 2024-06-18.
Reimers, N. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
Wu, L.., Tian, F., Qin, T., Lai, J., and Liu, T.-Y. A study of reinforcement learning for neural machine translation. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, Brussels. Belgium, Netober-November 2018. Association for Computational Linguistics.
Robertson, S., Zaragoza, H., et al. The probabilistic rele- vance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-383) 2009.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 1rXiv: 0034070..677.... 2017.
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. Less: Selecting influential data for targeted instruction tuning. In Proceedings of the Forty-first International Conference on Machine Learning, 2024.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. In Ad- vances in Neural Information Processing Systems, pp- 3008 1028-3021..00 Curran Associates, Inc., 2020.
Xie, S. M., Santurkar, S., Ma, T., and Liang, P. S. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34227, 2023.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. technical report. arXiv preprint 2xXx:2412..5115: 2024.
Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D. Scale efficiently: Insights from pre- training and fine-tuning transformers. arXiv preprint arXiv: 10686, 2021.
Yeh, C.-K., Taly, A., Sundararajan, M., Liu, F., and Raviku- mar, First is better than last for language data influence. Advances in Neural Information Processing Systems, 35: 32285-32298, 2022.
Touvron, H., Martin, L.., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
12

---

Data Selection with NICE
Zhao, Y., Yu, B., Hui, B., Yu, H., Li, M., Huang. F., Zhang, N. L., and Li, Y. Tree-instruct: A preliminary study of the intrinsic relationship between complexity and alignment. In Proceedings of the 2024 Joint Inter- national Conference on Computational Linguistics, Lan- guage Resources and Evaluation (LREC-COLING 2024), pp. 16776-16789, Torino, Italia, May 2024. ELRA and ICCL.
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing. E., et al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Proc. NeurlPS, volume 36, 2023.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Data Selection with NICE
A. Training
A.1. Training Dataset
For the task-agnostic setting, the four processed training datasets that are utilized are described Wang et al. (2023). These datasets, annotated or authored by humans, are detailed in Tab. 7. FLAN V2 and COT are based on existing NLP datasets, while DOLLY and OASST feature open-ended responses from humans, demonstrating diverse formats, lengths, and tasks.
For the task-aware setting, we use 95% of the RLHF task's s training data as training set, while the remaining 5% serves as the validation set. The RLHF dataset is constructed from conversations between humans and Al assistants. Specifically, given an existing (potentially multi-turn) conversation between an LLM and human, the LLM provides two responses, which are then evaluated by human annotators. Each data point contains pair of responses, with one marked as 'chosen' by human annotators for being more helpful than the other. Since only the last responses from the assistant are compared during human labeling, the helpfulness is evaluated only for the last-turn response of each conversation. Consequently, we use only the last-turn response labels. We only use these "chosen" responses for training, validation, and test. For the HumanEval task's training data, adopt the CodeAlpaca 20k (Chaudhary, 2023) dataset with the addition of the original instruction format for each data point.
For fine-tuning Llama2, we adopt the 'Tulu' format following from the study by Wang et al. (2023).
Tulu Instruction Format
<user>
What can you help me with?
<assistant>
I'd like to show off how chat templating works!
For fine-tuning the Mistral and Llama3 models, we utilize the respective instruction formats predefined by each model, as detailed below:
Mistral Instruction Format
(INST) What can you help me with? 
 I'd like to show off how chat templating works!
Llama3 Instruction Format
<I
slsttattheederr hnader
What you help with?< eot_id| start >assistant <eendhhedderidd
I'd like to show off how chat templating works!
A.2. Training Details
All experiments utilized the parameter-efficient LoRA (Hu et al., 2021) approach. A linear warm-up with 0.03 warmup ratio was employed, peaking at learning rate of 2 x 10-5 We trained for epochs on each dataset with a batch size of 128. The LoRA module had rank of 128, an a of 512, a dropout rate of 0.1, and learned matri- ces for all attention layers. Specifically, the Llama2-7b model used is meta-llama//Laama -a-e-laaaa--7---a--- 2-7b-hf, Llama2- 13b used is meta--llaaaLLeee - Mistral-7B meta-llamal Met Bate/MMeaa--aaa2222222
Bestralaa/MMstra---22
Llama3-8B is
Each main experiment was repeated three times with random seeds 0, 1, and 2. Under random selection methods, three different random subsets from the training set were chosen for each seed. For LESS, TSDS, NICE, and NICEAMC, we first performed warmup training on subsets chosen by each seed, then selected distinct subsets from the resulting warmup model for each trial. We used the same optimization seeds as those used for the warmup model. For experiments without reported standard deviation, we used seed Full training was conducted on seed 0 only, due heavy computation.
14

---

Data Selection with NICE
Table Detailed information about the training The task-agnostic training data pool is the in Xia et al. (2024) and Wang (2023).
Task-agnose agnostic
Dataset FLAN V2
Size
Sourced from
Turns
Prompt Len.
Response Len.
100,000
Based on Existing NLP
355.7
31.2
Datasets
COT
100,000
Based on Existing NLP
266.0
53.2
Datasets
DOLLY
15,011
Tluman-writter
from
1
118.1
91.3
scratch
OASST
55,668
Human-written
from
1.6
34.8
212.5
scratch
Task-aware
RLHF
41.643
Human feedback data
2.46
145.0
517.3
Code Alpaca
20,022
Code-related human- written instructions
294.8
197.0
B. Evaluation
B.1. Evaluation Dataset
AlpacaEval sources its data from self-instruct, OASST, Anthropic's helpful dataset, Vicuna, and Koala, widely used for understanding model behavior in structured, instrection-driven settings. The evaluation metrie is length-controlled win rate, adjusting for biases in response length using regression model to ensure fair and accurate assessments (Dubois et al., 2024). Each response is compared to baseline model, text fext_davinci_00. using weighted www.acapaeallappppppaaaa the annotator. We use text davinii .003 as baseline because its relatively lower baseline performance can more clearly highlight the performance difference between selected subsets. By contrast, using a stronger baseline (e.g., 8p14 turbo) could mask the differences among the model trained on the subset selected by different selection strategies. For annotation, we employ weighted alpacaaaeeaaaaaaaaaaaa eval gpt4 _pt4_turbo (https: / //gitttt.... com ttttss-liee a alpacaa_eeee eval), chosen for its high agreement with human annotations, large context capacity, and cost-efficttiveness.
as
HvmanEval uses set of programming challenges evaluate code correctness and functionality, measured by the pass @k metric which is how many correct solutions appear within specified number of attempts & (e.g.. we use 100 in the main experiment). This metric assesses the ability of models to generate correct solutions within a limited number of attempts, reflecting the model's efficiency in code generation. We additionally use the unbiased estimator of pass@ to avoid the high variance vanilla pass@k (Chen et al., 2021).
TLDR uses cleaned data from Stiennon et al. (2020), focusing on well-structured input-output pairs for summarization tasks. It is evaluated against reward model (OpenAssistant,, 2023) that is trained based human feedback to ensure the generation of high-quality summaries.
RLHF is designed for training and evaluating language models using human feedback to optimize response generation. It consists of prompt-response pairs and we only use the "chosen" response whose last-turn response aligns best with human preferences, according to specific criteria such as relevance and safety. The evaluation metric reward model (Ray2333, 2024) which can measure the helpfulness of model responses.
B.2. Evaluation Details
For AlpacaEval and HumanEval, we set the number of tasks to be = and g = 10, while for TLDR and RLHF have 1. The reason is: for AlpacaEval, two samples are drawn from each of the five subtasks (Self-Instruct, OASST, Anthropic's helpful dataset, Vicuna, Koala), resulting in a total of ten samples in the validation set. For HumanEval, since pass @ aims check if there is at least one functional code within the le generations of each test data point, select training data points that enhance performance across all validation data. Each validation data point is treated as an individual task. Hence, AlpacaFval and HumanEval are multi-subtask scenarios with g 5 and 10, while TLDR and RI.FF are single-subtask scenarios with q = 1. The aggregation of each training data point's influence score w.r.t the validation set follows from Sec. 2.2.
For TLDR and RLHF, the reward function used during the computation of the policy gradient is the reward model, and the
15

---

Data Selection with NICE
evaluation metric for the validation set is the average reward for each validation point. For AlpacaEval, the reward function for each generated response is the annotator's decision for that response, and the evaluation metric for the validation set is not simply average but an average of debiased version of each annotator's decision (Dubois al., 2024). For HumanEval, the reward function r for each generated response (code) is the boolean result of all unit tests for that code, while the evaluation metric for the validation set is pass@ k. This metric measures functional correctness by generating codes for each test problem and considering the problem solved if any code passes all unit tests of that problem. The pass@ score can be viewed as the average each boolean result, whereas pass and pass @ 100 are computed via a problem-level "OR" across the & generated codes, followed by an average across all problems. Additionally, address numerical instability and reduce variance, we use the unbiased estimator version of pass@ following Chen al. (2021).
We evaluate the model on the validation set Dv (the same set used for data selection) after each epoch, and assess the best-performing checkpoint on the test set.
C. NICE and NICEAMC Details
NICE and NICEAMC rely solely on the probability of generated responses and the score from reward function, enabled by the policy gradient mechanism. For AlpacaEval, labels are provided to all approaches because the reward function requires ground-truth labels. For the other three tasks, including TLDR, RLHF, and HumanEval, the ground-truth labels of the validation dataset are not used by our method, although they are available to the other baselines (such as LESS). This is because in these three tasks, the reward function does not require ground-truth labels. Specifically:
For RLHF and TLDR, the reward function is learned reward model that outputs scores based the prompt and generated response.
For HumanEval, the reward is defined by whether the generated code passes unit tests, not requiring reference solutions.
C.I. NICE Details
We generate 20 MC samples for all evaluation datasets, except for HumanEval where generate 500 samples due to the task's s difficulty (i.e., a lower code pass rate). Generally, we set the sampling temperature to 1.2 to promote diversity, except for AlpacaEval where we use 1.0. The generated responses of the validation set under temperature of 1.0 yield higher win rate on the final checkpoint compared to 1.2. We employ multinomial sampling with top k 50 and top = 0.95.
C.2. NICEAMC Details
For NICEAMC, we use gpt-4-ttubbo2224--0000 Regarding the GPT API hyperparameters, we set the fre- quency quency-penalty to 0, pessence_penalyy to O, and temperature to 0.8 for all tasks.
For TLDR tasks, we add a prompt A brief summary of my post is (TL; : after the prompt of the data point, before generating the response, to enhance generation quality.
For HumanEval, prepend a prompt:
Complete the following python function to return only the function body (completion). Do not include the function header or docstring.
before the coding question. This ensures the model outputs only the necessary code, avoiding chain-of-thought content that could fail unit tests.
D. Baseline Details
For LESS and TSDS, the hyperparameter settings are the same as in their official repos. For BM255, DSIR, and RDS, these methods warmup model-agnostic, meaning the selection process does rely on the warmup models' randomness. Hence, we only run them seed 0. To avoid the instruction format's effect the representation retrieval, we use the format of:
Quest : [Question \nAnAnswer: [Answer]
16

---

Data Selection with NICE
for the majority of the training data and validation data. We use TULU format for RLHF training and validation data, as RLHF contains many turns.
For BM25, we use the cank_bm2 treat each validation data point as a query to retrieve the BM25 scores of each training data point. For RDS, we adopt the sentence-trrass www.eranssoorrrsooooooommmeerrer Matteeerrra---Miii------ sentence trans formers Mateeerrral---MMiMM---- 11ALLM--6--900 L6-v2). We compute the cosine similarity of embeddings between each validation data point and each training data point. Overall, for each validation point, BM25 and RDS will have a score vector of dimension n (the size of the training dataset n)). We then follow the same aggregation steps as in Sec. 2.2 to aggregate the scores. For DSIR, validation data cnttps://githun com/p pp lambda/dss ddsir..
__2n).. We
model (https:/ http://ugggggggggeeeeee co/
the official Github repo and match the distribution between training data and
E. Preliminary
E.1. Derivation The Change in the Validation Loss
The detailed derivation of the change in the validation loss is explained here: When training data point 2; is included in the training step f. the model parameters are updated accordingly, leading change in the validation loss. Assuming a small learning rate n/e is used in the parameter updates with the Stochastie Gradient Descent (SGD) optimizer, this one-step change step t can be approximated using first-order Taylor expansion (Pruthi al., 2020):
[zz; 1 ((=;00))
Vol(z0; 0¹) (0 8t) + 0(110+1 (²113)
= VoL(zu; 01) (0++1- - 8²) -8¹)
(I)
VoL(zo; 0') (-Th, VoL(z;;0) 0*))
(VoL(zz 0'), V L 0'))
E.2. Derivation of Influence Function
Influence Function (IF) (Koh & Liang, 2017) measures the influence of down- weighting the training data point by some small e, on the new parameter 0f := arg mingge 1 Ei= L(z;;) 0) + (z;; The parameter change is given by:
005 c.
Towwn.parame (24) = de
= =H gE VoL(z;;
c=0
where Hgp = Σ (z; is the Hessian matrix of the average loss over the training set and is positive definite by assumption. Then, by applying the chain rule, IF can measure the influence of down-weighting z; on the loss of the validation data point 2,: Infir 22, 2v) aL(zo; F C.R.
t=0
00 C.
(2)
VoL(zu;0
e=0
VoL(zv; 08) "iah V VoL(z;
E.3. Efficient Influence Function
DataInf makes use of Bartlett's second identity (Bartlett, 1953) to approximate the Hessian. That is, the Hessian can be replaced with the second moment of the first-order gradients: G(05) = - VoL(zz; ( BE)T, which simplifies the computation of the Hessian matrix. The 8 here is the last checkpoint of the warmup model. In the later section, the computation of DataInf is all w.r.t. to the last checkpoint of the warmup model. We denote the gradient of the loss of training data point w.r.t. Φ (ó can be single layer's parameter) by V Tu == L 05), while the gradient of the loss of validation data point z, w.r.t. & (Ø can be a single layer's parameter) by V VoLv == V (zL(z,0%).
To further address the potential large dimension of and computational challenges, DataInf adopts another two techniques: Damping: small positive constant, X, is added the diagonal elements of G( enhancing its positive definiteness
17

---

Data Selection with NICE
and invertibility (Martens al., 2010). 2. Block Diagonal Matrix Representation: G(OE) is further approximated using its
block diagonal matrix where each block is layer of the deep neural network (Grosse al., 2023). The influence function then becomes
7 Vo Vo Vo
Vo Lo
+ +ATa
Vo
where 8r E IR: is the model parameter in I-the layer, Id. E Rd xdi is identify matrix of size di, and is the total number of layers.
Following these transformations, DataInf tweaks the order of matrix inversion and the average calculations, and then leverages the Sherman-Morrisse formula:
Va, Vo + +laa
V 91 Lg Va DJ Aaa)
j=1
VoL,VoLL X+ Vo Vol.
Id.
nd
The computation of the Hessian and the later computation the influence function can require retrieving the training gradient VoL; twice. To additionally optimize the time, we apply random projections each gradient vector V and store them. Later,
retrieve the stored projected gradients and compute Hessian and the influence function score.
Besides Datalnf. other methods have been proposed for efficiently computing the influence function for large language models. Grosse et al. (2023) improved the computation of the block-diagonal Hessian using the Kronecker product of uncentered forward and backward covariances of each layer (EK-FAC). TRAK projects gradients into a low-dimensional space and calculates influence scores within the subspace (Park et al., 2023). LORGA further improves the projection step with an efficient gradient projection strategy that leverages the gradient structure in backpropagation (Choe et al., 2024). However, neither EK-FAC nor LORGA discussed their applicability LoRA fine-tuned models, leading us exclude them for experiment. Additionally, TRAK's approach of treating the multi-class classification problem a single binary logistic regression may result in information loss. Therefore, we also did not apply it, our task is purely generative and each token's 's prediction is multi class classification prediction whose prediction space is vocabulary size. For the hyperparameter in terms Datainf, the projected gradient is 2048 dimension. The smaller dimension due to the fact that we need to project gradients for each layer. The À 0.0001.
F. Explicit form of NICE During Implementation
As discussed in Sec. 3.1, Xia et al. (2024) replaces the SGD with Adam gradient and replaces the inner product with the cosine similarity of the original TracIn for performance consideration. We integrate these two enhancements on NICE as well, leading to:
E
E ~ f(ula, V log( (o (r))) Bo)] 1'o L' 00) ]w)] 1o(2s: 0=)
Inf 20)
Σ
Te
E j.n f
Vo log log(f 0-) ))r
m ++
To(2;:0)
vvetl +
mm e + (1 - V
m m=+1
1- 3
vet1_ Bzve +11 1 -82) (V (2;; 1²) 1 8
where every operation is applied elementwise. Here, 81 and 82 represent the hyperparameters for the first and second moments, respectively, with e serving as a small constant.
18

---

Data Selection with NICE
G. Related Work: Data curation for instruction tuning
Curating high-quality, diverse, and complex instruction tuning data has been shown to improve the instruction-followingg ability of LLMs. Researchers have proposed different strategies to measure and improve different aspects of instruction- following ability: Cao et al. (2023) utilize natural language indicators to evaluate quality; Zhao al. (2024) employ GPT-transformed instructions to measure complexity; and Chen et al. (2023) adopt LLM-annotated scores to assess both quality and complexity. Bukharin & Zhao (2023); Du et al. (2023); Lu et al. (2024); Wang et al. (2024b) optimize instruction data by emphasizing diversity in tandem with quality or complexity. Liu et al. (2024a) offers a comprchensive comparison of existing methods regarding these three properties. They further train a model to predict complexity and quality and iteratively filter out the most diverse points using embeddings. These approaches typically select data without a validation set. Our work extends this line of research to targeted data selection with guidance from validation set, selecting data that aligns the most with downstream tasks to enhance specific model capabilities. Our setting is more similar Xia et al. (2024), Liu et al. (2024b) and Li et al. (2024b). However, Xia et al. (2024) selects data based on the influence validation loss, which can lead discrepancies between minimizing loss and maximizing performance, while we select data based on their influence on validation performance. Liu et al. (2024b) uses loss-based gradients to measure data distance when optimizing for distribution alignment and diversity, which can also suffer from the aforementioned discrepancy to some extent. Li et al. (2024b) utilizes a perplexity- -bsse scoring system to select the most advantageous data for a defined anchor set, but their methodology is limited to single-turn training data. Another line of work, exemplified by Bhatt et al. (2024), frames curation as active learning by selecting the most informative prompts for predicting the label (i.e., generate responses) through uncertainty or diversity maximization. This active learning paradígm differs from our setting. A more comprehensive review of data curation methods is provided by Albalak al. (2024).
H. Additional Analysis
Unless otherwise specified, the experiments in the section below are conducted on the Llama2-7B model.
H.1. Additional Results of @ass@k on HumanEyal
We provide additional evaluation metrics for HumanExal dataset, specifically pass@1 and pass 10, in Tab. 8. Overall, they align with our main findings: NICE and NICEAMC generally outperform loss-based influence estimation (LESS) and baselines, and the subsets selected by NICE and NICEAMC have the potential to outperform the full dataset. Note that BM25 performs well in terms of pass@1 1 and pass @ 10 on the Mistral-7B model and even outperforms other baselines under task-agnostic training data pool. Empirically, we observe that BM25 is more likely to select data points that contain codes. This is likely because BM25 is based on TF-IDF, which assigns higher scores to training data that is more relevant in terms of word frequency, and certain words appear more frequently than others in the codes. Consequently, BM25 performs well, especially when the training data contains data from multiple different domains that are not coding-related. However, this good result does not transfer to either the Llama2-7B model or under a task-aware training data pool due to two main reasons: 1) BM25 does not use the information from the models, meaning that the same data subset will be selected for different models. Intuitively, different models require different data to achieve better performance. Therefore, selecting data using BM25 is sub-optimal; 2) Selection based the word frequency is not enough for the task-aware setting. In a task-aware data pool, training data points are more relevant to the task, possibly resulting in comparable BM25 scores for all data points. Consequently, a more careful selection based on other criteria (i.e., not just word frequency) is needed, Additionally, while BM25 achieves 31.81 pass@ @1 for Mistral-7B. 7B,, pass@] is not the sole evaluation criterion in practice, because there are situations where multiple responses can be generated from the LLM and checked by a verifier (e.g., test cases). Therefore, for HumanEval, focusing on pass@k k with larger (and higher accuracy) is more desirable.
Additionally, we argue that NICE can improve pass@) I performance with lower temperature while sacrificing the perfor- mance of @ 10 and @ass@ 100. We analyze the performance on the HumanEval task for models trained on NICE-selected subsets when using different temperatures to generate MC samples. As shown in Tab. 9, the pass @ 1 performance on downstream tasks is improved by using a lower temperature, while at the cost of reduced performance on pass@1 10 and pass@ 100 metrics. A lower temperature reduces uncertainty during generation. If the model is good at certain problems, it increases the probability of answering these problems correctly. However, this reduction in uncertainty comes at the expense of diversity, as the generated responses tend to be very similar to one another. Consequently, for difficult questions, all generated responses are incorrect, pass @k (for larger k) suffers. This trade-off implies that using a lower temperature to generate responses improves pass@1 @ performance. When these responses generated under a lower temperature are used to
19

---

Data Selection with NICE
Table Additional Results Pass@k on HvmanEval both task-agnostic lask-aware settings Llama2-7B and Mistral-7B. Bold numbers indicate the top- cop-eerfrnnnng selected subset. A purple cell suggests that NICE outperforms LESS which uses bass-based influence estimation. Underlined numbers show that the subset selected by our approach exceeds the performance full dataset. Numbers in small font represent standard deviations.
Task-agnostic training data pool.
Llama2-7B Pass @ 10
Mistral-7B Pass @ 10
Pass @
Pass @ 100
Pass @
Pass @ 100
Full
7.61
25.52
47.44
29.47
59.68
83.63
Random
8.34_0.34
23.8540.1 40
44.30
19.9941.35
62.04
85.5641.27
RDS
10.00
25.55
45.29
30.31
62.00
84.15
BM25
8.27
24.51
46.19
31.81
62.43
84.09
DSIR
9.53
24.02
42.22
27.71
56.81
79.17
TSDS
10.30 10.30+1.58
2..47+1.27
43.68+1.82
27. 27.50+1.41
59.78+1.86
82. 82.78+1122
LESS
9.24=0.77
26.12 10.17
17.50
26.85+0.58
60.66+0.39
85.2440.45
NICE
10.33 1.72
27.37+1.56
48.59
48.5942.08 +2.08
29. 19,48 4..93
62.05 4:2.33
85,59 1.41
NICEAMC
9.04 -2.5
25. 25.1141.72
45.1042.84
29.96 29.96
62.10 +1.8
85.67 #0.31
Task-aware training data pool.
Llama2-7B
Mistral-7B
Pass
Pass@
Pass@ 100
Pass 1
Pass 10
Pass 100
Full
13.27
30.30
51.27
33.14
64.09
84.27
Random
20.22
20.86.20.22
51.91
33.15
63.62.11922
83.344+2.54
RDS
12.40
31.34
54.74
33.32
63.23
86.75
BM25
13.66
31.22
52.23
33.58
64.35
84.81
DSIR
11.98
30.43
53.10
32.30
59.07
79.91
TSDS
12.85
28. 10.991
49.85
31.7441.63
63.04 1.06
85.51 1:1.8
LESS
13.55
30.53 +0.57
52.67 10.71
34.05
64.12 00.37
85.26
NICE
13.43+0.33
31.70, +0.66
55. 09
33.6 61 1.29
65.56 +..32
87. 87.35 +7.35+100
NICEAMC
12.87
30.39 10.26
50.67+1.24
34.1 13 20.88
63.91 =0.81
84.18
Table Performance on the HvmanEval task under a task-aware training data pool for models trained on NICE-sclocted subsets with different temperatures for generating MC samples. Wc decrease the temperature to improve pass@) metrie, while at the cost of decreased performance on pass@ 10 and pass @ 100.
Temperature 1.0
Temperature 1.2
Model
p I
pass
pass 100
pass
pass
pass @ 100
Llama2-7B
14.13
31.21
49.95
13.12
31.41
53.96
Mistral-PP 7B
35.70
63.60
81.04
35.23
63.85
85.69
compute policy gradients for data selection, the resulting selected subset also favors the pass metric, while cost of decreased performance pass@1 @ k (for larger k).
H.2. Robustness across Validation Splits
We demonstrate the robustness of NICE across different validation splits and address concerns about potential overfitting to specific validation set. When the training data pool is task-agnostic, we randomly selected an alternative validation set D as a reference and re-selected a subset D/s. The performance of models retrained on Ds (selected based on the original validation split Dv) and D's (selected based on the new validation split D',) are shown in Fig. 5. Importantly, models trained on the selected subsets, whether Ds or D's, consistently outperform models trained on randomly selected subsets. This verifies the robustness of our approach and confirms that its effectiveness does not depend on specific validation split.
20

---

Data Selection with NICE
AlpacaEval
TLDR
RLHF
HumanEval
30
20-
Reword
Reword
2.0
Poss-100
13
0.5
0.0
voldation 2e13
vnlidttoon seta
reddon
validation sal
Vildedon set2
aandon
velitation 3017
velatation set
random
vntidaation set
valitation set2
random
Figure 5. Performance of models trained on different NICE-selectee subsets using different validation sets as references. Models trained on NICE seleetele subsets consistently outperform those trained on randomly selected subsets, regardless the validation set used.
Table 10, Additional Results on the RLHI task for Llama2-13B and Llama3-8B, Bold numbers indicate the top-performing selected subset, purple cell suggests that NICE outperforms LESS which uses loss-based influence estimation.
Llama2-13B
Llama3-88 -BB
RLHF
Task-agnostic
Task-aware
Task-agnostic
Task-aware
Random
2.0640.01
1.20
1.97
1.12
RDS
1.77
0.70
1.75
0.81
BM25
2.72
1.34
2.84
1.43
LESS
1.52+0.09
1.65+0.04
1.64
1.65
NICE
2.87 =0.04
1.76
3. 3.22++000
1.99
H.3. Additional results on Llama3-8B and Llama2-13B
evaluate NICE against various data selection baselines using the state-of-the-an model, Llama3-8B (Dubey et al., 2024), and a larger model, Llama2-13B, on the RLHF task (see Tab. 10). The superiority of NICE underscores our method's generalizability across different model sizes and state-of-the-art models.
H.4. Discrepancy between NTP Loss and Evaluation Metrics
We additionally include the NTP loss and performance (i.e., measured by each task's evaluation metric here) of the validation for the remaining three tasks in several training checkpoints. The results in Fig. 6 are similar those Fig. 1: checkpoints with minimal loss (highest negative losses) do not correspond to checkpoints with the best performance; the performance can continue to increase even if the loss increases (negative loss decreases).
AlpacaEval
HumanEval
42.2
-0.960
-2.750
52
Length controlled Win
Negative Loss
42.0
Neqative Loss
2,753
4.25
-0.440
50
Reward
Negative
Park
-40.8
7,760
4.00
43
-0.970
-6.765
3.75
48
40.6
-2.770
3,50
-0.150
44
100
200
300
400
100
200
300
400
100
200
300
400
Step
Figure 6. Discrepancy observed between the NTP loss and performance (as measured by each task's evaluation metric) the checkpoints. The checkpoint with the lowest loss (i.c., most negative) can exhibit relatively poorer performance. Notably, performance can continue to improve as the loss worsens (i.c., the negative loss becomes higher).
21

---

Data Selection with NICE
Sample Size vs Dev for RLHF dataset
0.120
Standard Deviation
0.115
0.110
0.105
0.100
16
Monte Carlo Samples
Figure 7. Standard deviation of different w.r.t the number of MC samples. Increasing the number of MC samples generally lowers the standard deviation across runs, indicating better stability.
H.S. Stability of MC Sampling
We provide an ablation study (Fig. 7), varying MC samples from 5 to 20 on the RLHF task under the task-agnostic training data pool, for more in-depth discussion on stability. Results show that increasing the number of MC samples generally lowers the standard deviation across runs with different seeds, indicating better stability. The benefit of reduced standard deviation diminishes as increases. This validates that our chosen MC (MC=20 for RLHF task) provides a good trade-off, offering sufficient stability without excessive computation.
H.6. On the Additional Cost of NICEAMC
Note that NICEAMC an optional enhancement -NCCE itself does not require GPT-4. We list the projected GPT-4 cost for NICEAMC in Tab. 11. The costs low for the majority the tasks, except for RLHF, due to its large validation set (which can be addressed by using alternative models discussed in the next paragraph).
Table 11. Projected GPT-4 cost for NICEAMC across different tasks. The cost for RLHI is high due to large validation set.
Task
AlpacaEval
TLDR
RLHF
HumanEval
Avg
GPT Cost (S)
1.70
14.26
291.17
6.34
78.37
Use of Spen-llurreeSmmaller LLMs. To reduce cost, we can use high-performing open-source models. On the RLHF task in the task-agnostic setting, we use Qwen 2.5-3B/7B-Instruct (Yang et al., 2024) for AMC. Both outperform NICE. Notably, even a small model like Qwen -----rnsttuct performs better due to its better alignment training, despite its smaller size. These models offer comparable performance to GPT-4 without incurring the additional API cost.
Table Performance on the RLHF task using different models for NICEAMC. Qwen models offer competitive performance without the API cost of GPT-4. Model
NICE
NICE AMC (GPT-4)
NICE AMC (Qwen2.5 7B)
NICE AMC (Qwen2.5 3B)
RLHF 2.82+0.10
3.03
3.00
2.97+0..3
I. Ablation Studies
Unless otherwise specified, the ablation studies in the section below are conducted on the Llama2-7B model.
22

---

Data Selection with NICE
I.1. Using an Alternative Way to Compute Policy Gradient
Table 13. Performance comparison between loss-based influence estimation (LESS) and NICE which uses different approaches to compute policy gradient when selecting from task-aware training data pool. PG refers to using the MC, policy gradient described in Sec. 3.1, while PPO denotes using Proximal Policy Optimization. Employing policy gradient computed from either policy optimization approach during data selection results in generally better-sclected subset compared to loss-based influence estimation.
dataset
RLHF
HvmanEval
metrics
reward
pass
pass 10
pass @ 100
LESS
1.6210.005 10.05
13.55 10.28
30.53 10.57
52.67 10.71
PG
1..05
13.43 1.333
31.7040.66
55.0941.66
PPO
1.73+0.02
13.5841.90
30.84+1.82
52.08
Besides the vanilla Monte-Carlo policy gradient, we conducted an ablation study of using another policy optimization methodology to compute the policy gradient used in NICE. We tried computing the policy gradient using Proximal Policy Optimization (PPO) (Schulman et al., 2017), with results presented in Tab. 13 for the task-aware setting. These results demonstrate that compared the loss-based influence estimation (LESS), which selects training data by optimizing the direction of decreasing validation loss, integrating either the MC policy gradient PPO gradient select training data by optimizing in the direction of improving downstream task performance leads to better-selected subset. A "better-selected subset" refers to subset of training data that, when used for model training, results in improved performance on downstream tasks.
1.2. Introducing Additional knowledge on Loss-based Influence Estimation
Tab. 14 shows the performance of LESS using GPT--gnnertedd labels (LESS+GPT), which is generally worse than our approaches and can even be worse than LESS + true labels (LESS). Hence, simply using GPT-generated labels with loss-based approaches cannot always address the unavailability of labels for validation data.
Table 14. Performance of LESS using GPT-generated labels (2nd row) in the task-agnostic setting, which is generally worse than our approaches and can be worse than LESS + true labels (Ist row).
Table D
Alpaca
TLDR
RLHF
HumanEval
LESS
26.9 26.9442.37
3.37
1.44.0.07
47.50
LESS+GPT
27.35+1.86
3.41
3.03
43.04+1.39
NICE
27 27.61+2.12
3.61
48.5942.08 3.03
2.82+0.10
NICEAMC
30.45
3.55
45.1042.84
I.3. Effect of reward score
We conduct simple ablation study to evaluate the effectiveness the reward score within the context of the policy gradient methodology. For the RLHF task under task-agnostic training data pool, compare the performance between the continuous reward score (our current methodology) and discrete reward score where the reward is set to if positive and 0 if non-positive. This discrete reward setup can also be interpreted as form of rejection sampling. The subset selected using the continuous reward yields a model with a performance score of 2.82=0.10 10. whereas the subset selected using the discrete reward yields a performance of 2.25 Although less effective than the original policy gradient method, the policy gradient with a discrete reward still demonstrates some effectiveness, outperforming both LESS 11.4410.07) and random (2.05+0.11) selection strategies. This ablation study highlights that subsets selected with the guidance of reward scores produce models with better performance than those trained on subsets selected by LESS random sampling.
J. Qualitative analysis
J.1. Generated Responses
We present three generated responses from GPT-4 on an AlpacaEval validation data point in Tab. 15 and Tab. 16, demonstrat- ing that by using a better cetter-performing model, we can gather responses of comparable or even higher quality than the original label. This supports the use of the better-performing model during data selection to guide further performance improvements.
23

---

Data Selection with NICE
By comparing the original labels with the GPT-4 responses, partieularly in Tab. 15, observe that the generated responses exhibit higher quality and more accurate results. By observing Tab. 16, we find that the generated responses are not only better in quality, provide richer detail, and more vivid imagery for monologues, but they also offer diverse responses. These monologues differ in style and narrative focus, ranging from a grand epic tone to reflective introspection. The first explores ancient ruins and lost civilizations, the second emphasizes nature and mysticism, and the last tells mythic tales.
J.2. Highest/Lowest Scoring Training Data for RLHF Task under a Task-agnostic Training Data Pool
In task-agnostic setting, we present the highest- and lowest-scoring training data points in Tab. 17. The sighest-scoring training data points complete the task effectively and offer additional knowledge, while the lowest-scoring ones tend to be vague, minimal, or fail to address the task meaningfully. Considering the RLHF task targets helpfulness, the high-scoring data points can potentially boost the model's helpfulness. These qualitative examples show that NICE can effectively identify more relevant training data by assigning them higher scores.
J.3. Highest/Lowest Scoring Training Data for RLHF Task under a Task-aware Training Data Pool
In task-aware setting, we present the highest- and lowest- seoorstscoring.e training data points in Tab. 18. The lowest-scoring training data points whose last-turn responses are either unhelpful fail to provide additional meaningful information, potentially degrading model performance (we only use the last-turn response during training, explained in App. A.1). Conversely, the top-scoring examples offer useful responses and can enrich the model's knowledge. Overall, NICE effectively filters out noisy data while favoring high-qualit training examples.
24

---

Data Selection with NICE
Table 15. We compare three GPT-4-generated responses with the original labels of an AlpacaEval validation data point, observing that GPT-4 consistently produces higher-quality and more accurate outputs.
Question: "If the endpoints of line segment are (2, -2) and (10, 4), what is the length of the segment?"
Baseline Response: The length of the linc segment is 12 units (generated by text daxiicivin))_003)
Generated Response 1: To find the length a line segment given its endpoints, you can use the distance formula. The distance formula between two points (I1, yn) and (22, y2) in the coordinate plane is:
d = / (m2 x1)² + (12 v1)²
For the points given, (11, = (2, -2) and (22, y2) = (10, 4). Plugging these into the formula:
d v (10 2) + (1 (-2))²
d / 1 62
d d= /64 36
d v100
d = 10
Therefore, the length of the line segment 10 units.
Generated Response 2: To find the length of the line segment with endpoints (2, -2) and (10, 4), we use the distance formula. The distance d between two points (21. y1) and (#2, 1/2) given by:
V - x1)² + (y2 31)²
In this case, the coordinates of the endpoints are (#1. = (2, 22) and (2g, U2) = (10, 4). Substitute these values into the formula:
10 2
V2 y1
2) = 2
Now substitute these differences into the distance formula:
d= -=V8+6'
64 V100
So, the length of the segment is:
Therefore, the length of the line segment is 10 units.
Generated Response 3: find the length of a line segment with endpoints (I1, y1) and (#2, U2), we can use the distance formula, which is derived from the Pythagorean theorem. The formula is:
d V(2 x1)² + (y2 y1)²
Given the endpoints of the line segment are -2) and (10, 4), we substitute these coordinates into the formula:
= 2, y = -2, 10, =
Now, plug these values into the formula:
d = / 2)00-))
d /82 +62
d == V61 36
= V100
d = 10
Thus, the length of the segment 10 units.
25

---

Data Selection with NICE
Table 16. We present three diverse and vividly detailed generated monologues that answer an AlpacaEval validation question, demonstrat- ing how GPT-4 can vary tone, style, and narrative focus in generated responses. Question: "Write monologue. Do not offer any explanation or out of character notes. The setting is in the Elder Scrolls universe."
"Ah, what a glorious day! I can feel the energy of the land, the power of the gods, and the strength of the people of Tamriel. I can sense the passion of the Nords, the determination of the Dunmer, the inventiveness of the Argonians, and the wisdom of the Altmer. The air is filled with the sounds of battle, adventure, and hope. I am so proud to be part of this world, to be a part of the Elder Scrolls. have seen many things in my travels, but nothing compares to the freedom and beauty of this land. hope that can do something to help protect it and ensure its future."
Baseline Response:
so blessed be able share in its secrets and experience its wonders. can only
Generated Response I: "Ah, listen, traveler, to the whispers of this old, timeworn place. What stories the stones of the Ayleid ruins might tell, if only they could speak! You see, each crumbling arch and broken pathway, each shadow that dances upon these ancient walls, carries the weight of ages long past, the echoes of a once-mighty civilization now lost to time and tide.
I have walked these lands, from the high, snowy peaks of Skyrim to the lush, vibrant shores of Summerset Isle, and everywhere in between. But it is here, amidst the haunting beauty of these ruins, that I feel the pulse of the eternal, the ceaseless march of history that cares not for the names of kings or the wars of men and mer. Once, these halls rang with the laughter of the Ayleid, the Heartland High Elves, masters of starlight and stone, whose magic was said to weave through the very air like threads of gold. They built their cities high as their ambitions, soaring and majestic. But pride, ah, pride it festers, and it corrupts. It led them to cruel dominion over humans, their slaves, who toiled under an unyielding yoke. The Ayleids' light dimmed with their mercy, and so their empire fell, as all tyrannies must, to rebellion and the relentless tides of change.
Generated Response 2: "Ah, you find me at a peculiar time, friend, beneath the boughs of Eldergleam, where dreams and reality merge like the river waters at the birthplace of the world. I am but humble servant of Kynareth, tending to the whispers and the weeping of the wind through these ancient branches. You seek wisdom, or perhaps power? Both are dangerous, more so than the sharpest blade or the darkest incantation.
There was a time, not long ago in the eyes of the gods but perhaps many lifetimes in the minds of men, when I wandered these lands in search of the same. From the chilled breath of Skyrim to the sun-kissed shores of Hammerfell, walked. I have slept under the stars that wheel unchanging over the Imperial City, and I've faced the dragons' fury, their breath the fire of thousand forges.
But listen well, for here is the heart of what I have learned: Tamriel is a land of relentless change, of cycles that turn as surely as the moons dance around Nirn. Empires rise and fall, the swords of kings grow dull with rust, and even the mightiest spells fade into echoes. Yet, the trees, the stones, and the rivers-they remember. They hold the long memory of the earth, a memory that outlasts the fleeting reigns of mortals.
Do you feel the roots of the Eldergleam? Do you hear their slow, ceaseless whisper? They speak of the Green Pact and the oaths sworn by the first Bosmer, of the Red Mountain where fire married sky, of the Snow Elves driven deep into the earth's 's dark embrace. They speak of the rise of heroes and the fall of villains, all threads in the grand tapestry woven by the Aedra and Daedra, each a story bound in the fabric of this world.
Generated Response 3: "Ah, traveler! Welcome to the warmth of my hearth. Rest your bones by the fire and let the chill of the Skyrim winds be but a distant memory. You must have many tales of your own, but spare an old Mer a moment to regale you with story of the ancient days, a tale passed down through generations to keep alive the spirit of our people. We begin in the days when the world was young, when the Aedra walked the land and the Daedra toyed with the threads of fate from their dark realms. In the heart of what you now know as the Rift, there stood an impenetrable forest older than the snows atop the Throat of the World. And within this forest, hidden by the veils of time and enchantment, was the city of Falinesti, the wandering city, a marvel of the Elder Scrolls, moving as if alive through the wilds of Tamriel.
Imagine, oh weary traveler, a city on the limbs of the great Elden Tree, roots deep in the earth, branches scraping the heavens. It was there, in that moving city, that the tale of Aclarion was born. Aclarion, son of no one, a child of the shadows who could speak with the birds and run as swiftly as the West Wind.
Now, Aelarion was a creature of curiosity and mischief. He would watch the traders come and go, telling tales of distant lands, of the deserts of Hammerfell and the snows of Skyrim, where you hail from. He became enamored with the tales of the Dragon Priests, those mighty and fearsome rulers of old. Enthralled by the power they wielded through their masks, relics gifted by dragons, he dreamt of nothing but to behold such mask himself.
26

---

Data Selection with NICE
Table 17. We list the training data points with the highest lowest NICE score for the RLHF task under task agno-sgi training data pool, illustrating that highest-scoring responses are helpful and can convey additional knowledge. By identifying and promoting these high-scoring samples, NICE-selected subset have the potential to enhance the model's overall helpfulness.
Dataset Dialogue
Training data points with the highest NICE score
Flan V2 User: Detailed Instructions: In this task, you'l re given an ambiguous question (which can be answered in more than one way). Your task is to provide one question which clarifies the input question and it has one unique answer, and also provide an answer to the generated question. Generated question and answer should be separated with a new line. Problem: When was free health care introduced in Canada? Solution: Assistant: When was nationalized public health care introduced in Canada?
1947
Flan V2
User: Q: In this task, you given an ambiguous question (which can be answered in more than one way). Your task is to provide question which clarifies the input question and it has unique answer, and also provide an answer to the generated question. Generated question and answer should be separated with a new line.
Who got the biggest NBA contract right now?
Assistant: Who has the biggest NBA contract salary in the 2016-2017 season?
LeBron James
DOLLY
User: Task: What team did Bobby Orr play for in his 5th season?
Input: Robert Gordon Orr OC (born March 20, 1948) is a Canadian former professional ice hockey player, of the greatest of all time. Orr used his ice skating speed, scoring, and play-
widely acknowledged making abilities revolutionize the position of defenceman. He played in the National Hockey League (NHL) for 12 seasons, the first 10 with the Boston Bruins, followed by two with the Chicago Black Hawks. OπT remains the only defenceman to have won the league scoring title with two Art Ross Trophies. He holds the record for most points and assists in single season by a defenceman. Orr won record eight consecutive Norris Trophies as the NHL's best defenceman and three consecutive Hart Trophies as the league's 's most valuable player (MVP). Orr was inducted into the Hockey Hall Fame in 1979 at age 31, the youngest be inducted that time. In 2017, Orr was named by the National Hockey League as one of the "100 Greatest NHL Players" in history. Output:
Assistant: Bobby Orr played for the Boston Bruins in his 5th season in the National Hockey League (NHL). He played in the NHL for 12 seasons, the first 10 with the Boston Bruins, followed by two with the Chicago Black Hawks.
Training data points with the lowest NICE score
Flan V2
User: Translate" " " to Finnish?
Assistant:
DOLLY
User: Is tap water safe drink in Japan?
Assistant: Yes, tap water is safe drink in Japan.
OASST
User: have trouble staying on track when need do schoolwork. What some ways that can keep
myself on track?
Assistant: Getting rid of anything that distracts you could be helpful.
27

---

Data Selection with NICE
Table 18. We show training data points with the highest or lowest NICE score for the RLHF task under a task-aware training data pool, demonstrating that top-scoring samples enrich model knowledge while low-scoring can be noizy. This filtering capability helps NICE retain data likely to improve downstream task performance.
Training data points with the highest NICE score
User: Mustard seems pretly common these days, in most people's ' refrigerators. When did people start using mustard as a condiment on foods, I' I'm curious about its history.
Assistant: don't It really know about mustard, so I found an article that you might like, in which the author writes, "Mustard is known to have originated in India but was long ago used as a condiment in ancient China and Egypt." Here's a quote from that article that sounds relevant: "It appears that mustard was used as an ingredient cooking in several ancient civilizations, including those of India, China, and Egypt. In fact, the oldest known picture of mustard a carving of a jar on the wall of a tomb in ancient Egypt. This artifact, which dates to the 15th century B.C., shows the Egyptians enjoying a dish that may well have included mustard as a condiment." I'd be happy to read that article for you if you want.
User: What are some good stretches should do before going to the gym?
Assistant: Stretches are an important part workout regimen. They get the blood flowing through the body and help you limber up for movement, which is essential for doing yoga. What types of stretching are you looking to do? User: I was hoping you could tell me.
Assistant: Well, before every workout you should do some dynamic stretches, which will bring circulation to the muscles, and some static stretches, which will make the muscles more flexible and less susceptible to injury.
User: Ok, which stretches specifically?
Assistant: Ok, what about calf raises?
User: Can you teach me how to make candles?
Assistant: Sure, just follow these directions and it should be easy!
Training data points with the lowest NICE score
User: How fast trains go?
Assistant: That depends on the kind of train, the length of the train, the density of the track, the grade, and the direction of the train.
User: What about coal trains?
Assistant: I'm sure what you mean by "coal trains."
User: What's the best brand of TV?
Assistant: I'm not sure what you mean by "best brand."
User: How can make my clothes whiter?
Assistant: You can use vinegar to "bleach" the clothes.
User: Vinegar can bleach clothes?
Assistant: Yes.
28


Reviews:
[{"review_id": "6zgdJnIpGU", "reviewer": "Reviewer_znd4", "summary": "The paper introduces NICE (Non-differentiable evaluation metric-based InfluenCe Estimation) for selecting training data to improve the performance of large language models (LLMs) on specific tasks. The method leverages policy gradient techniques to optimize non-differentiable evaluation metrics directly, addressing the limitations of existing loss-based influence estimation methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the effectiveness of NICE and the soundness of our methods and evaluation.\n\n**Claims And Evidence:**\n\nWe defer the computational analysis between LESS and NICE to our rebuttal for Reviewer waTE. For BM25, it is indeed an efficient retrieval method based on lexical matching, ranking training data by relevance to the validation data. However, **BM25 is model-agnostic** (i.e., it selects the same data for different models). Thus, a subset selected may perform well with one model but not necessarily be optimal with another (lines 286-94, R). While being **model-agnostic contributes to BM25's efficiency, it also limits its performance**. In contrast, NICE enables model-aware data selection, optimizing training data specifically for the target model’s validation performance.\n\n**Theoretical Claims:**\n\nThe theoretical suitability of policy gradient for influence estimation can be justified as follows. Loss-based influence estimation methods (e.g., TracIn, influence function), which our method builds upon, estimate the influence of a training point on validation loss. In particular, these methods measure the influence via the “gradient of the validation loss” (the change in validation loss w.r.t. the model parameters / policy), which by the **chain rule**, can be combined with the the first-order gradient or the Hessian of the training loss (w.r.t. the model parameters). In contrast, we estimate the influence on **validation performance**, measured by evaluation metrics. However, since the evaluation metrics are non-differentiable, the policy gradient, which measures the change in validation performance caused by the corresponding training data point, is a direct replacement of the “gradient of the validation loss”. It also allows the estimating the influence of the training data via chain rule. \n\nMoreover, we show elaborate the derivations for NICE (based on TracIn) and NICE_IF (based on influence functions) in lines 141–52 R and lines 201-19 L, respectively. Both of them also relies on the derivations shown in App A.5.1 and App A.5.2, respectively. \n\nTherefore, our NICE is theoretically-grounded since it uses the principled influence estimation framework from prior work and extends it to calculate the influence of data on the non-differentiable metrics using policy gradient.\n\n**Experimental Designs Or Analyses:**\n\n**Cost of GPT-4.**\n\nNote that NICE AMC is an optional enhancement—NICE itself does not require GPT-4. We list the projected GPT-4 cost for NICE AMC in the table below. The costs are low for the majority of the tasks, except for RLHF due to its large validation set (which can be addressed as below).\n\n||AlpacaEval|TLDR|RLHF|Humaneval|Avg|\n|-|-|-|-|-|-|\n|GPT Cost($)|1.70|14.26|291.17|6.34|78.37|\n\n**Use of open-source/smaller LLMs.**\n\nAs suggested, to reduce cost, we can use high-performing open-source models. On the RLHF dataset, we use Qwen 2.5-7B/3B-Instruct for AMC. Both outperform NICE. Notably, even Qwen 2.5-3B performs better due to its better alignment training, despite its smaller size. These models offer comparable performance to GPT-4 without the API cost.\n\n||NICE|NICE AMC (GPT4)|NICE AMC (Qwen2.5 7B)|NICE AMC (Qwen2.5 3B)|\n|-|-|-|-|-|\n|RLHF|2.82±0.10|3.03±0.02|3.00±0.03|2.97±0.03|\n\n**500 Monte Carlo samples on HumanEval.**\n\nWe use 500 samples because HumanEval is challenging—correct generations are rare, and we aim to estimate pass@100, whose evaluation requires 100 generated code pieces. In contrast, other tasks are evaluated using one sample, and we use 20 samples (i.e., a multiplier of 1) to approximate expected performance under the current policy.\n\n**MC sample size vs. performance trade-off.**\n\nWe perform experiments on HumanEval with MC sample sizes ranging from 200 to 500 (see Fig. in [link](https://postimg.cc/ZCdzgpBM)), showing a clear performance improvement with more samples. We also analyze this trade-off in Sec. 4.5 (lines 374-84, R). Both HumanEval and RLHF results show performance can improve with sample size. On RLHF, 20 samples already yield strong performance, with additional gains as the size increases.\n\n**Other Comments Or Suggestions:**\n\nWe provide qualitative examples in App. A.10 (pages 23–24). These examples illustrate that “harmful” data can include paraphrased versions of the questions or fail to provide useful information.\n\nWe will include additional experiments in the revised paper and hope our justifications will address your concerns and improve your opinion of our work.", "claims_and_evidence": "Supported: \n1. Effectiveness of NICE: The paper claims that NICE outperforms existing data selection baselines across diverse scenarios. \n2.  The authors argue that optimizing non-differentiable evaluation metrics directly leads to better data selection than traditional loss-based methods. The experimental results, particularly in tasks requiring long-form generation, support this claim by demonstrating improved performance metrics.\n3. The paper claims that NICE can perform data selection without labeled validation data when the reward function does not require labels.\n\nProblematic: \n1. While the paper acknowledges the computational cost of NICE, it claims that the method is efficient due to the use of LoRA and random projection. However, there is a lack of detailed analysis or evidence comparing the computational cost of NICE with other methods LESS or BM25.", "methods_and_evaluation_criteria": "Overall, the proposed methods and evaluation criteria are well-aligned with the problem of optimizing data selection for instruction tuning in LLMs. They address the key challenges and provide a comprehensive framework for evaluating the effectiveness of the proposed approach.", "theoretical_claims": "The theoretical claims in the paper are based on sound principles from reinforcement learning and influence estimation.\nI found that the connection between policy gradients and influence estimation is intuitive but lacks formal grounding. The paper does not clarify why policy gradients are theoretically suitable for influence estimation.", "experimental_designs_or_analyses": "1. The computational cost of Monte Carlo sampling (e.g., 500 samples for HumanEval) and reliance on GPT-4 for AMC are mentioned but not quantified. I also wonder about the cost of use of GPT4 API. Can your method also work well with other open-source or smaller LLMs? You may also discuss trade-offs between sample size and performance.", "relation_to_broader_scientific_literature": "1. NICE advances the field by directly optimizing for task-specific performance metrics, which is a novel approach compared to traditional methods that rely on proxy metrics like next-token prediction loss.\n2.  By applying policy gradient techniques to data selection, the paper introduces a new way to estimate the influence of training data on model performance, bridging the gap between reinforcement learning and data selection in NLP.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Please see the comments in previous sections.", "other_comments_or_suggestions": "1. While the paper notes that larger subsets can harm performance (Fig. 3), it does not systematically characterize \"harmful\" data (e.g., via qualitative examples).", "questions_for_authors": "Please see the comments in previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FjZ0RxoxVe", "reviewer": "Reviewer_waTE", "summary": "This paper proposes NICE, which provides an innovative and label-efficient approach to data selection by using policy gradients for non-differentiable evaluation metrics, outperforming several baselines in many benchmarks, including both task-specific and -agnostic settings. While its computational cost and complexity may limit its scalability, it seemly shows promise for improving model performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our insight and our RL-based data selection approach. Please find our responses below.\n\n**Essential References Not Discussed:**\n\nWe have included TSDS (as suggested) and an additional baseline, DSIR in Tab. C of our response to Reviewer d8As. Our method outperforms the new baselines across diverse datasets and models. \n\n**Weaknesses:** \n\n1.1 **Computation cost**\n\nAlthough Monte Carlo sampling increases the computational cost, **this trade-off is justified by NICE not needing validation labels** - a key motivation of our work. NICE **fills a gap left by existing loss-based baselines by supporting data selection with unlabeled validation data** in cases where the evaluation metrics are label-independent. Furthermore, we can observe the performance improvement over other methods in Tab.3. \n\nNevertheless, we provide a comparative analysis of the computational costs between NICE and LESS, showing **NICE remains within a practical computational range:** Tab. A lists the asymptotic complexity and wall-clock runtime (training is measured on a single H100 hour, others are on L40 GPU hours) for each stage in the data selection procedure. Tab. B highlights the validation gradient computation where NICE differs from NICE. Let $E, d, M$ denote number of epochs (saved checkpoints), dimension of the projected gradients, number of MC samples. Let $D_W, D_N, D_V$ denote the warmup, training, validation set. When the sizes of the validation set and the MC samples are small, NICE adds only marginal overhead to LESS (e.g., AlpacaEval).\n\n|Table A|Warmup LoRA|Training Grad Comp|Validation Grad Comp|Data Selection|\n|-|-|-|-|-|\n|Remark|LESS=NICE|LESS=NICE|NICE>LESS|LESS=NICE|\n|Compute|$O(\\|D_W\\|E)$ 3h|$O(\\|D_N\\|E)$ 48h|LESS:$O(\\|D_V\\|E)$ 0.11h on average; NICE: $O(\\|D_V\\|EM)$ 14.67h on average|$O(\\|D_N\\|\\|D_V\\|d)$ < 0.02h|\n\n|Table B| |NICE|NICE|LESS|\n|-|-|-|-|-|\n|$\\|D_V\\|​$|$M$​|MC Sample|Val Grad|Val Grad|\n|Alpaca=10|20|0.17h|0.05h|<0.02h|\n|TLDR=322|20|8h|1.47h|0.08h|\n|RLHF=2192|20|32h|10h|0.33h|\n|HumanEval=10|500|5h|2h|<0.02h|\n\n1.2 **Stability** \n\nWe provide an ablation study with results in https://postimg.cc/Mf0N6b76, varying MC samples from 5 to 20 on the RLHF dataset for a more in-depth discussion on stability. Results show that increasing the number of MC samples $M$ generally lowers the standard deviation across runs with different seeds, indicating better stability. The benefit of reduced standard deviation diminishes as $M$ increases. This validates that our chosen $M$ provides a good trade-off, offering sufficient stability without excessive computation.\n\n2. We ensure there is no data leakage in our experimental setup, since the test data has no overlap with the validation data. Additionally, we clarify that similar to LESS and TSDS, our work focuses on targeted data selection (lines 102-5 R), which explicitly assumes access to a validation set that is distributionally aligned with the target task. Our “task-agnostic” & “task-specific” settings in the experiments pertain to the **data preparation** stage (to form the initial pool of training data without & with the knowledge of the target task, resp.) **prior** to data selection (lines 245-8 R, 255-7 R). \n\nWe believe that selecting training data without the knowledge of the downstream evaluation data falls under a different setting—task-agnostic data selection—where alternative approaches based on perplexity or coreset are applicable. However, these directions are outside the scope of this work, focusing on targeted data selection. \n\n3. NICE_AMC generally performs well under two conditions: (a) when a stronger model is available for the target task—one that can generate higher-quality Monte Carlo samples, and (b) when the training pool is sufficiently large, allowing better gradient alignment with NICE_AMC (lines 313–5, R). This explains its success in our task-agnostic (data preparation) setting with abundant data.\n\nAs for guidance on when to use NICE vs. NICE_AMC:\n\n- NICE_AMC: a stronger assistance model is available + a large, diverse training pool can be leveraged.\n- NICE: resources or training data are limited or when the base model is already powerful enough.\n\nIn the revision, we will include additional baselines and analyses, and clarify the use case of NICE_AMC. We hope the discussion above will address your concerns and improve your impression of our work.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Please refer to the Questions.", "essential_references_not_discussed": "By following LESS, this paper introduces a data influence selection strategy based on evaluation metrics rather than loss or gradients that can be applied to both task-specific and -agnostic settings. I believe it would be beneficial to incorporate a discussion and experimental performance comparison with [1], even though [1] approaches data selection from the perspective of distribution matching. \n\n[1]TSDS: Data Selection for Task-Specific Model Finetuning", "other_strengths_and_weaknesses": "Strengths:\n\nThe authors' insight that there is no direct correlation between the validation set loss and the final test performance is reasonable and profound. Instead, directly using the performance reward for data selection and solving it through reinforcement learning is highly promising and adds innovation to this paper.\n\nWeaknesses:\n\n1. In the paper, RL is used to solve the data selection problem based on performance rewards, which involves computationally expensive steps such as MC sampling. This introduces significant computational cost and instability. Is there a more in-depth discussion on the computational cost and stability? This could affect the reproducibility of the work and its subsequent use in future research.\n\n2. Is there any potential data leakage in the experiments? For example, in Line 258, it is mentioned that the validation set of AlpacaEval and HumanEval share the same distributions as the corresponding test sets. If we were to construct a validation set from a subset of the training dataset without any validation data and without knowledge of the downstream evaluation data, how would the model perform in the task-agnostic setting?\n\n3. There appears to be some inconsistency in the performance between NICE and NICE_AMC in Table 3. How can this be explained? Additionally, could the authors provide some theoretical guidance on when to choose NICE over NICE_AMC or vice versa?", "other_comments_or_suggestions": "Please refer to the Questions.", "questions_for_authors": "Please refer to the Questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C1CRUPfLdz", "reviewer": "Reviewer_d8As", "summary": "The paper proposes NICE, a RL based framework for choosing instruction tuning data targeted for downstream tasks. The proposed method uses reward signals such as loss function or influence function on the validation data. The policy gradient is then used to estimate the training data point's influence on the given validation sets. The authors conducted experiments on instruction tuning of LLMs such as llama and mistral models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback on our writing and the thoroughness of our methods and evaluations. We wish to make the following clarification.\n\n**Claims And Evidence:**\n\nTo clarify, our “task-agnostic” & “task-specific” settings pertain to the data preparation stage (to form the initial pool of training data without & with the knowledge of the target task, resp.) **prior** to data selection (lines 245-8 R, 255-7 R). For data selection, this work only focuses on targeted (task-specific) data selection (lines 102-5 R).\n\nHowever, methods based on DPP [https://arxiv.org/pdf/2402.02318] and submodular optimization [https://arxiv.org/abs/2401.06692] focus on non-targeted (task-agnostic) data selection without validation data, which is a different problem setting that falls outside the scope of our work. We will improve the clarity in the revised paper. \n\nTo improve the comprehensiveness of the experiments for targeted data selection, we add TSDS [https://arxiv.org/abs/2410.11303] (as suggested) and DSIR (highlighted in the suggested survey [https://arxiv.org/pdf/2402.16827] [1] ) for the setting where the initial pool of training data does not have the knowledge of the target task (our \"task-agnostic\" setting). The results are shown in Tab. C below.\n\nAcross multiple datasets and models, **our method generally continues to outperform the newly added baselines**. Although DSIR performs the best on AlpacaEval with the LLaMA2-7B model, it selects data based on n-gram lexical feature matching (similar to BM25) between training and target distributions, independent of model-specific signals. As a result, it selects the same subset regardless of the model used for instruction tuning, which can be sub-optimal. We can see it is no longer the best with Mistral-7B on the same task. This highlights the advantage of our model-aware data selection strategy. We will incorporate these baselines and citations in the revised paper.\n\n|Table C|Alpaca|TLDR|RLHF|pass@1|pass@10|pass@100|\n|-|-|-|-|-|-|-|\n|**Llama**|\n|TSDS|11.42±1.33|1.42±0.07|1.01±0.12|0.1030±0.02|0.2547±0.01|0.4368±0.02|\n|DSIR|20.27|1.53|2.57|0.0953|0.2402|0.4222|\n|NICE|16.79±0.85|1.61±0.39|2.82±0.10|0.1035±0.02|0.2737±0.01|0.4859±0.02|\n|NICE AMC|17.16±3.73|3.55±0.40|3.03±0.02|0.0904±0.02|0.2511±0.02|0.4510±0.03|\n|**Mistral**|\n|TSDS|25.91±2.49|3.47±0.00|1.83±0.15|0.2750±0.01|0.5978±0.02|0.8278±0.01|\n|DSIR|29.31|3.48|2.94|0.2771|0.5681|0.7917|\n|NICE|26.20±4.13|3.31±0.35|3.10±0.06|0.2948±0.01|0.6205±0.02|0.8559±0.01|\n|NICE AMC|31.05±1.23|4.60±0.20|3.42±0.05|0.2996±0.02|0.6210±0.02|0.8567±0.00|\n\n**Experimental Designs Or Analyses:**\n\nWe would like to clarify that for three tasks—TLDR, RLHF, and HumanEval—the ground-truth labels of the validation dataset are not used by our method, although they are available to the baselines (lines 305–18, L). For AlpacaEval, labels are provided to all approaches. Specifically, NICE relies solely on the probability of generated responses and the score from a reward function, as enabled by the policy gradient mechanism (lines 130–132, R). In these three tasks, the **reward function does not require ground-truth labels**. Specifically:\n\n- For RLHF and TLDR, the reward function is a learned reward model that outputs scores based on the prompt and generated response (lines 160-162, L).\n- For HumanEval, the reward is defined by whether the generated code passes unit tests, not requiring reference solutions (lines 245-250, L).\n\nWe will make this distinction clearer in the experiment section to avoid confusion.\n\n**Essential References Not Discussed:**\n\nWe have checked the survey paper in https://arxiv.org/pdf/2402.16827 and added a related work DSIR as a baseline for comparison. We will incorporate the citations of the survey paper and works that are not directly comparable but related like DPP, SKILL-IT[2] in the related work section. \n\nWe thank the reviewer for the valuable comments. We hope the additional baselines make our comparison more thorough and improve your impression of our work.\n\n[1] Xie, Sang Michael, et al. \"Data selection for language models via importance resampling.\" *Advances in Neural Information Processing Systems* 36 (2023): 34201-34227.\n\n[2] Chen, Mayee, et al. \"Skill-it! a data-driven skills framework for understanding and training language models.\" Advances in Neural Information Processing Systems 36 (2023): 36000-36040.", "claims_and_evidence": "The authors claims their selection strategy is better than loss based strategy and other existing data selection strategies. There has also been other recent literature in this space such as https://arxiv.org/abs/2410.11303. The authors should compare to the methods proposed in this paper.\n\nThe authors also claim their method works well in task agnostic settings, but fail to compare against some of the existing literature in this space, such as the methods that use DPP (https://arxiv.org/pdf/2402.02318) and submodular optimization (https://arxiv.org/abs/2401.06692).", "methods_and_evaluation_criteria": "The methods and evaluation make sense and are quite thorough other than the baselines they compare against.", "theoretical_claims": "I did check the theoretical derivations.", "experimental_designs_or_analyses": "I did not find the specific details when the labels are unavailable. I think the paper could benefit from a better description what what was used in place of the labels $y$ in the experiment section.", "relation_to_broader_scientific_literature": "The paper falls under the general data selection for LLM instruction tuning.", "essential_references_not_discussed": "In addition to the papers mentioned above, the authors are also encouraged to check out other methods mentioned in https://arxiv.org/pdf/2402.16827.", "other_strengths_and_weaknesses": "Overall, the paper is very well written. I am willing to raise my score given the experiment comparisons with other baselines are conducted.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NtQytppRdT", "reviewer": "Reviewer_pZkV", "summary": "The paper proposes a data selection framework that computes the evaluation metric (e.g., reward) for validation samples and employs the Monte-Carlo policy gradient to calculate the influence of each training data point on the validation data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting the importance of our work on data efficiency in instruction tuning and for finding the paper easy to understand. Below are clarifications to address your concerns: \n1. Since discrepancy between NTP loss & eval. metrics has been shown in previous works (lines 380-1 L for refs.), it explains why we only consider a single experiment to further validate this claim. Nevertheless, we conduct experiments on 3 other datasets with different reward functions (eval. metrics) to show that the discrepancy between loss & metrics can be generalized. The results in [link](https://postimg.cc/GBMkCw0q) are similar to that in Fig.1: checkpoints with minimal loss (highest negative loss) do not correspond to checkpoints with the best performance (measured by the eval. metric) (lines 394-6 L); the performance can continue to increase even if the negative loss decreases. \n2. Our work can be motivated using two limitations of loss-based data selection: (a) discrepancy between NTP loss & eval. metrics as discussed in point 1 (lines 19-43 R) & (b) dependency on high-quality labels which can be unavailable in practice (lines 149-51 L). In contrast, NICE can select training data to directly optimize commonly used eval. metrics of generation tasks (lines 46-9 R) & alleviate the dependency on validation labels when the eval. metric is independent of labels (lines 64-8 L).  \n\n2.1. High-quality labels (ground-truth responses) are often unavailable or available only in small quantities for generation tasks. So, a common practice is to use an LLM judge or train a reward model based on the small quantity of high-quality labels, and assume they are reliable, high-quality metrics that generalize to other inputs. Thus, this does not contradict the importance of high-quality labels. In fact, prior works have shown that these eval. metrics can track the response quality & align well with humans (Bai et al. 2022; Dubois et al., 2024; Stiennon et al., 2020; Zheng et al., 2023) and are hence widely adopted for evaluating generation quality.\n\n2.2. Tab.D below shows the performance of LESS using GPT-generated labels (as suggested, 2nd row), which is generally worse than LESS + true labels (1st row) and our approaches. Hence, the performance of loss-based data selection is hurt by the unavailability of true labels, highlighting their importance. Furthermore, using GPT-generated labels alone does not address the issue discussed in point 1.  \n|Table D|Alpaca|TLDR|RLHF|pass@1|pass@10|pass@100|\n|-|-|-|-|-|-|-|\n|LESS|15.44±2.74|1.54±0.21|1.44±0.07|0.092±0.01|0.261±0.00|0.475±0.02|\n|LESS+GPT|15.07±0.91|1.47±0.60|3.03±0.01|0.043|0.188|0.417|\n|NICE|16.79±0.85|1.61±0.39|2.82±0.10|0.104±0.02|0.274±0.01|0.486±0.02|\n|NICE AMC|17.16±3.73|3.55±0.40|3.03±0.02|0.090±0.02|0.251±0.02|0.451±0.03|\n\n3. Advantage of NICE (policy gradients) over LESS (loss-based gradients) is supported by experimental evidence in Tab.3: NICE generally outperforms LESS (lines 321-9 L). About the concern on MC sampling & the eval. metric $r$, note that the policy gradient is used to optimize $r$ with data selection and approximated by MC sampling. So, it won’t be meaningful if we consider MC sampling as a standalone component and remove the eval. metric (e.g., for an ablation study). To see this, not using the eval. metric is equivalent to treating all sampled responses equally, i.e., $r$ is a constant function, resulting in a policy gradient of 0, as derived in [link](https://postimg.cc/5jGSyF71). \n\n4. We use Johnson-Lindenstrauss random projections (see lines 275-8 L).\n\n5.1. In Tab.D above, we have provided the results of LESS + GPT-generated labels, making the comparison fair. However, as mentioned in point 2.2, LESS is using the true labels to outperform that with GPT-generated labels and thus has an unfair advantage (yet poorer performance) over our approach in TLDR, RLHF & HumanEval (lines 278-81 R). Moreover, NICE (without AMC) already outperforms LESS without the help of additional models (Tab.3).\n\n5.2. To clarify, our “task-agnostic” & “task-specific” settings pertain to the data preparation stage (to form the initial pool of training data without & with the knowledge of the target task, resp.) **prior** to data selection (lines 245-8 R, 255-7 R). For data selection, this work focuses on targeted (task-specific) data selection (lines 102-5 R). Hence, all our baseline choices are fair and valid. We will clarify this in the revised paper. In addition, we have included 2 more baselines TSDS & DSIR in Tab.C of reviewer d8As’s rebuttal.\n\n6. While NICE incurs more computational cost due to MC sampling, this trade-off is justified by not needing validation labels—a key motivation of this work—and its improved performance over other methods (Tab.3). We defer the computational analysis to reviewer waTE’s rebuttal.\n\nWe will include the above discussions in the revised paper. We hope we have addressed your concerns and improved your opinion of our work.", "claims_and_evidence": "See Other Strengths And Weaknesses", "methods_and_evaluation_criteria": "See Other Strengths And Weaknesses", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "See Other Strengths And Weaknesses", "relation_to_broader_scientific_literature": "See Other Strengths And Weaknesses", "essential_references_not_discussed": "See Other Strengths And Weaknesses", "other_strengths_and_weaknesses": "### Strengths\n1. The issue of data efficiency in instruction tuning datasets is an important topic in scaling LLMs.\n2. The paper is easy to understand\n\n\n### Weaknesses\n1.  The claim about validation loss vs. evaluation metrics may be imprecise. The authors primarily base their argument on a single experiment with a reward model (Figure 1). This raises concerns that the relationship between validation loss and evaluation metrics may not be generalizable and cause misunderstandings.\n\n2. The motivation seems unclear. \n\n(1) The paper argues that high-quality labels are crucial yet often unavailable for loss-based data selection methods. It then relies on an additional model (e.g., a reward model or LLM judge) to generate the evaluation metric. How do the authors ensure the quality and reliability of these metrics?\n\n(2) There is insufficient evidence in the paper showing that the lack of labels is the main limitation of previous loss-based data selection methods. How will existing state-of-the-art loss-based data selection methods perform if established LLMs or [1] can be used to generate validation labels or according to the teacher-forcing loss? \n\n3. Insufficient experiments to support the claim about the advantage of policy gradient compared to the gradient used in previous loss-based methods. Note that previous gradient-based methods might also be improved by incorporating multiple label responses from additional LLMs. Thus, it remains unclear whether the benefit arises from Monte Carlo sampling or from the introduced evaluation metric itself.\n\n4. Some technical details remain unclear. For example, the paper projects the LoRA gradient into an 8192-dimensional vector in Line 290, but it does not illustrate the projection method used.\n\n5. The choices of baselines may be outdated and less than ideal. \n\n(1) The paper introduces additional models to help data selection, which may create an unfair comparison against other baselines.\n\n(2) Some baselines (e.g., LESS) are specifically designed for target data selection, so applying them in a task-agnostic context may be inappropriate. I suggest the author further compare and discuss more recent data selection methods in each setting.\n\n6. Monte Carlo sampling can be expensive, and the paper does not offer a comparative efficiency analysis of the computational cost for the proposed method and other baselines.\n\n[1] The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators. In NIPS, 2024\n\n## update after rebuttal\nI appreciate that most of my concerns have been addressed. Consequently, I have increased my score to 3. I expect the authors to incorporate new results, analyses, and clarifications in the revised manuscript.", "other_comments_or_suggestions": "See Other Strengths And Weaknesses", "questions_for_authors": "See Other Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes NICE for selecting instruction-tuning data by leveraging policy gradients to estimate the influence of training data on validation performance. 

The main strengths of the paper lie in the novelty compared to existing data selection approach for instruction tuning, as agreed by all reviewers.

While reviewers acknowledged the method’s novelty and practical utility, concerns centered on computational overheads, baseline comparisons, generalizability. Post-rebuttal, the authors strengthened empirical claims with expanded baselines (e.g., TSDS, DSIR) and computational analysis but left unresolved questions about generalizability.
Decision: Accept (poster)