Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning
Authors: Ling Zhang
Xianliang Yang
Juwon Yu
Park Cheonyoung
Lei Song
Jiang Bian
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Fine-tuning (SFT
DPO
SimPO); Data selection; Holdout loss; In-context learning; Gradient reweighting
Primary area: foundation or frontier models, including LLMs
TL;DR: We present a holdout-loss-based data selection framework that leverages in-context learning for efficient computation.

Abstract: Fine-tuning large pretrained language models is a common approach for aligning them with human preferences, but noisy or off-target examples can dilute supervision. While small, well-chosen datasets often match the performance of much larger ones, systematic and efficient ways to identify high-value training data remain underexplored. Many current methods rely on heuristics or expensive retraining. We present a principled, resource-efficient framework for data selection and reweighting. At its core is an In-Context Approximation (ICA) that estimates the holdout loss a model would incur after training on a candidate example by conditioning on a small, curated holdout set in context. ICA requires no reference model and no additional finetuning. 
We define the resulting estimate as the ICA score, and  derive per-example weights that dynamically reweight gradient updates as model parameters evolve. Across SFT, DPO, and SimPO, and over diverse backbones and datasets, ICA-based reweighting consistently improves model alignment with minimal overhead. We analyze sensitivity to score update frequency and the number of in-context holdout examples. We also discuss 
limitations in rapidly drifting on-policy settings, highlighting directions for future work.

Published as a conference paper at ICLR 2026
HOLDOUT-LOSS-BASHR DATA SELECTION FOR LLM FINETUNING VIA IN-CONTEXT LEARNING
Ling Zhang" & Xianliang Yang"
Juwon Yu & Park Cheonyoung
Microsoft Research Asia Beijing, China (zhangling, xianya] cmtos/oftcrooocccoom
Korean KT
Scoul, Korea
{juwon1.yu yu, cheonyoung)ekt.coo
Lei Song & Jiang Bian Microsoft Research Asia Beijing, China (Lei. Song, Jiang wim.niiicoooooofottooooooooooo..
ABSTRACT
Fine-tuning large pretrained language models is common approach for align- ing them with human preferences, but noisy or off-target examples can dilute supervision. While small, well-chosen datasets often match the performance of much larger ones, systematic and efficient ways to identify high-value training data remain underexplored. Many current methods rely on heuristics expen- sive retraining. We present principled. resource-efficient framework for data selection and reweighting. estimates the holdout loss model would incur after training ample by conditioning on small, curated holdout set context. ICA requires reference model and no additional finetuning. We define the resulting esti- mate as the ICA score, and derive per-example weights that dynamically reweight gradient updates model parameters evolve. Across SFT, DPO, and SimPO, and over diverse backbones and datasets, ICA-based reweighting consistently im- proves model alignment with minimal overhead We analyze sensitivity to update frequency and the number of in-context holdout examples. We also discuss limitations in rapidly drifting on-policy settings, highlighting directions for future work.
core an In-Context Approximation (ICA) that
candidate ex-
INTRODUCTION
Fine-tuning has become the standard approach for adapting large pretrained language models to downstream applications and aligning them with human intent (Wei al. 2021 Ouyang et al. 2022). This process typically starts with supervised fine-tuning (SFT) on instruction-response
pairs al. followed by preference- based alignment using pairwise preference data, e.g., RLHF (Christiano et al. 2017 Ouyang et al.] 2002)). DPO (Rafailov et al. 2023), and simPO (Meng al. 2024). Since fine-tuning effectively steers the pretrained model toward desired behaviors, the quality of training data plays central role: high-quality examples provide clear alignment signals, while noisy or inconsistent data can severely degrade performance.
However, training data for fine-tuning, typically collected from human annotators or model- generated outputs, often contain errors, inconsistencies, or redundancies. For example, [iao et al. (224) report that 20-40% of preference pairs in LM alignment are noisy, and that alignment per- formance highly sensitive to such noise. addition, recent studies have shown that smaller but carefully curated datasets yield alignment performance comparable to much larger ones (Zhou (et al. 2023 Chen et al., 2023). Despite the recognized importance and promise effective data selection for fine-tuning, systematic yet principled approaches remain underexplored.
"Equal contribution.
Cooee and prompts available at http::/://.ttto..coo/rrrtttt/t/t
dpo

---

Published as a conference paper at ICLR 2026
A central challenge is the absence of consensus what constitutes "valuable" data. While some approaches adopt hand-crafted heuristics or leverage large models as judges, a more principled per- spective is to define data value by its impact on downstream performance, which is the ultimate goal of fine-tuning. Nevertheless, directly measuring each example's contribution to downstream perfor- mance would require retraining (and evaluating) the model across all possible candidates, a process that is computationally infeasible.
To overcome this challenge, prior work has explored different strategies, including influence- function-based methods that approximate model performance change using Taylor expan- sions (Pruthi al. 2020; Xia et al.] 2024, Wang et al.) 2024). surrogate models that fit a linear approximation of the loss (Engstrom et al. 2024), and bilevel optimization or meta-learning ap-
proaches (Shen et al. 2024 Grangier et al. 2023; Calian et al. 2025). Beyond these analytical and
learning-basee approaches, empirical studies examine correlations between hand-picked picced notions of data quality and downstream performance to inform heuristic data selection criteria (Liu et al. [023
Bukharin & Zhao [2023] Zhao et al. 20224] Lu et al., 2023 Li et al.) 2023a| Cao et al.) 2023 Li et al.
2023b). These methods, while promising, are often computationally expensive or lack theoretical grounding.
In this work, we propose a principled, rexource-efficien framework for data selection in model fine- tuning. Following prior work (Mindermann et al. 2022 Xia et al. [024] Calian et al., 2025), we aim to select a subset of training data such that the model trained on it minimizes the holdout loss, i.e., the loss the holdout set. Each example's 's value is quantified based on the holdout loss the model would incur if trained with that example. Computing this naively is intractable, so we build on a framework introduced by RHO-Loss (Mindermann al. 2022), which uses Bayesian probability theory to derive a more tractable expression for the holdout loss. Nonetheless, efficiently computing this expression remains challenging, as evaluating would normally require retraining at each step.
RHO-Loss mitigates per-step retraining by relying on a separate reference model trained on the holdout set. We introduce an in-context approximation (ICA) that eliminates the need for both additional fine-tuning and a fixed reference model. Motivated by the insight from (Dai et al.) (003)) we provide the holdout set as in-context demonstrations at each training step to approximate the effects of explicit parameter updates model performance. The resulting data values derived from ICA, termed ICA scores, enable dynamic evaluation of each example's ' utility as the model evolves. These scores then used to reweight gradient updates during tinettunig, prioritizing examples that most reduce holdout loss. Experiments show that training with ICA- -bssed reweighting consistently improves model alignment for SFT, DPO, and SimPO across diverse datasets and backbone models.
2 RELATED WORK
A central question in data selection is how to determine what makes a training example valuable. This is often done by quantifying each example's impact on downstream proxy, typically measured as the loss on small, high-quality holdout set. Methods in this category include celttennc--funtiion
formulations (Pruthi et al.] [2200] Xia et al.) 2024; Wang et al. 2024)) Data Shapley (Ghorbani &
Zou
2019 Wang et al. 2025). and learned scorers such Datamodels (Engstrom al.] 20224) mettannerning frameworks ks (Calian et al. (025), and optimization based approaches (Grangier et al.
meta-
2023
Shen et al., 2024 Gu al. [0225] Pan et al. 2025). Particularly relevant to our work RHO- Minderraann et al., (022). which estimates the holdout loss a model would incur if trained on a particular example, but requires separate reference model. Another related approach is One-Shot Learning (Li et al. 2023b), which also leverages in-context learning, though in a different manner from our method.
Loss
In addition to analytic or learned approaches for quantifying data value, hand-crafted notions of data quality have been studied, with empirical analyses assessing how these proxies correlate with fine- tuned model performance and inform data selection (Liu et al. 2023 Bukharin & Zhao, 2023 Zhao
ct al. 2024 Lu et al. 2023 Li et al. 2023a Cao et al. 2023 Huang & Goyal 2025) Yu et al. 2025] Deng et al. 2025 Morimura et al. 2024). Beyond quantifying data value with respect to downstream performance, prior work has also explored selecting data to directly match the target distribution, using techniques such as gradient aligmment (Killamsetty et al. 2021), importance resampling (Xie) et al., 2023; Katharopoulos & Fleuret 2018). or optimal transport (Kang et al., (204)). In contrast to these data-centric methods, separate line adopts an algorithmic perspective, directly modify-
2

---

Published as a conference paper at ICLR 2026
ing learning objectives and proposing extensions to SFT DPO account for instance- instance-specifie differences or improve generalization (Wu et al. 2024 D'Oosterlinck et al. 2025 Wu et al. 2025)
PRELIMINARY
In-context learning LLMs have demonstrated strong in-context learning (ICL.) capabili- ties (Brown et al. 2020 Dong et al. 2022). In ICL, few demonstration examples are concatenated with a query to form the model input. The model then identifies patterns from these examples and makes predictions without any parameter updates.
Formally, given a query x, the model predicts an output y conditioned on a demonstration set C. In this work, we adopt demonstration set of the following form, which includes optional task instruction I followed by demonstration examples:
C = [T, 8 yI), (x.. y+)],
where s(.))) .) is a formatting function (Dong et al. 2022).
Supervised fine-tuning SFT adapts pretrained LLMs to produce responses with desired charac- teristics using instruction-response pairs. Let D (x), denote the instruction dataset,
where y the reference response for query X5. SFT updates the model parameters by minimizing the sentence-lever entsopy:
(0) (xxx)*>>D log (y
x)]
where m x) the probability of generating y given under the model parameterized by
Preference-based alignment methods RLHF (Christiano et al. 2017: Ouyang et al. 2022 is widely used method to align LLMs with human preferences. uses data of the form D
{(x,, Yw.s, Vt,i) yt,)), j==1 where Vw,t and Yı, are the preferred and dispreferred responses for prompt x,. The standard RLHF pipeline first learns reward model and then optimizes policy using RL algorithms such as PPO.
DPO (Rafailov et al. 2023) provides an off off-policy alternative that bypasses the RL step. learning directly from preference data without training a reward model. Specifically, DPO solves:
Cppo (0) = -E(x.yw.))>> log o To (Yw
x)
Te (y: x)
(y w.
x) (re x)
where Teet is reference policy, is the sigmoid function, and 3 controls the trade-off between adhering to the reference and incorporating new preference data.
A recent variant, SimPO (Meng et al.) 2024). extends DPO by eliminating the need for reference model and introducing a length-normalized implicit reward based on the average log probability of sequence. Its objective is:
Camro (0) = -x(xxy log 3
-7)]
(y
x))
Toly x)
yywl
yyl
where 7 is target reward margin ensuring that the reward difference between winning and losing responses exceeds this threshold.
4 HOL.DOUT-LOSS-AASEE DATA SELECTION VIA IN-CONTEXT LEARNING
In this section, we formally introduce the problem formulation and present a holdout-loss lostt-los--bbeed data selection framework that leverages in-context learning for efficient computation.
4.1 PROBLEM FORMULATION
We consider the problem of fine-tuning pretrained model on a large, but potentially noisy, training dataset D {(x,, !2) where (x, y) may represent insructionnresppnse pair preference
3

---

Published as a conference paper at ICLR 2026
pair. The goal is to optimize model performance smaller, higher-quality holdout set Dno {(xto, )) |/ 1 For example, Dho could be a corrected subset of generated by a stronger model or manually curated set from target domain.
However, training on all of DD may be inefficient and suboptimal due to the presence of noise. While one could train directly on Dho its small size often leads overfitting and fails to leverage the information in the larger set. Therefore, we aim to select a subset DD C 'D such that a model trained on D minimizes the loss Dho (i.e., the holdout loss). Formally, the problem can be framed as
D arg min (Dno: 0* (D)), where 0* (D) arg min (): DCD
We denote by 0 (S) the model parameters obtained by training any dataset S and L(S; 0) the loss on S under model 9.
Equation T would be prohibitively expensive to solve naively, it requires training every didate subset C 'D and evaluating the holdout loss. Instead of solving it directly, we recast the problem in terms of quantifying the contribution of each training example to reducing holdout loss. Concretely, we assign a score to each example that reflects this contribution. These scores can then be used either select the example that most reduces the holdout loss, or to reweight gradient updates in the optimizer according to each example's s contribution.
In the following section, we describe in-context learning-based approach for approximating these scores, removing the need for auxiliary retraining.
4.2 COMPUTING APPROXIMATED HOLDOUT LOSS VIA IN-CONTEXT LEARNING
Bayesian formulation of the holdout loss Consider sequential (greedy) data selection, where points are added one time. Let D, be the dataset selected up to step and O, == 0 (D.) the model trained on it. Then, the subset selection problem in Equation Treduces to selecting, at each step, the example (x, y) D that, when added to Dr, minimizes the holdout loss:
(x",y") = arg min L(Dro: 0- (D, U ((x,))))).
(X.y)ED
Accordingly, each training example's 's contribution can be measured by this holdout loss.
To avoid explicit retraining when evaluating the holdout loss in equation we build on the Bayesian formulation introduced by Mindermann et al. (2022). Specifically, considering the negative log-
likelihood as the loss function (Ely x; 0) log x; 0)), and, applying Bayes' rule under the conditional independence assumption, the holdout loss of the model trained with particular example can be expressed as (see Appendix AA]or a reproduction of the derivation in Mindermann
al. (2022) P:
L(Dro; (Dx,
l(y x; (D, Dn.))
ely x: xx L(Dno; 0.)
where we L for losses set, and e for the per-example loss. Omitting the term independent of (x, y) and reversing the sign, we define the remaining expression as the holdout-loss score of each example at step t:
Sbo y: 0:) fly x: 0.)
f(y x; 0 (D, U Dno)).
(4)
The optimal example can then be approximately found by maximizing this score:
= arg max (x, (x
(5)
The holdout-loss score provides a tractable tool for data selection. However, since D, is updated with each newly added example, the model trained on D, U Dho must also be updated at every selection step, which still incurs substantial computational overhead. To mitigate this cost, prior work (Mindermann et al. 2022) approximates the retraining by training a model only on Dno and
TTii Bayesian framework also extends to pairwise preference data (e.g., for DPO and simPO). where represents preference pair (t): with y yı, and the loss is defined correspondingly for the chosen preference learning model (e.g., Bradley-Terry used DPO).
4

---

Published as a conference paper at ICLR 2026
reusing it across all selection steps, ie., f(y x: (D; U Pho)) f(y (Dho)) This results in the reducible holdout loss (RHO-Loss) criterion employed for data selection in their work.
Nonetheless, RHO-Loss still incurs the cost of training separate reference model. Moreover, using a fixed model in place one that would be updated at each selection step can introduce bias in estimating each example's contribution (Wang et al. 2024). To enable efficient and adaptive data selection criterion, we introduce an in-context approximation that removes auxiliary training entirely by approximating E(y x; 0* (D, Dho)) via in-context learning. This technique described in detail below.
Efficient computation via in-context learning Motivated by the finding that in-context learning can induce model behaviors similar to those obtained through gradient-based updates (Dai et al. 2023), we provide the holdout set as in-context demonstrations to a model trained on D; to approxi- mate the effect of training on the combined set D, U'Doo. Specifically, the second term in equation approximated as
fyy 0- (D, )no)) Eyy x, Doo 0t).
(6)
We call the approximation in equation 6 the in-context approximation (ICA).
Applying this approximation to equation 4 yields the in-context approximation score (ICA score):
SICA (x; 0.) := f(y x; 0.) E(y x, Dmo; 0₂).
We now the ICA to approximately identify the data point that maximizes the holdout- loss score in equation [ This approach is not only more computationally efficient, but also enables dynamic re-evaluation of each example's impact on the holdout loss as the model evolves.
In practice examples are often selected in batches rather than sequentially. We next describe how to leverage the ICA score for batch selection using a reweighting strategy.
4.3 BATCH SELECTION VIA REWEIGHTING STRATEGY
ICA score-based reweighting For batch selection, we use reweighting strategy that up- weights high-scoring examples and downweights lower-scoring ones. Unlike hard selection, which only chooses top examples and may reduce batch diversity or ignore interactions among data points (Wang et al. 2024). reweighting leverages the gradient signal from the entire batch, improv- ing training stability. We apply ICA score-based reweighting in main experiments and ablate this choice in Section 5.3 comparing it to percentile-based filtering.
Concretely, consider gradient-based training (e.g., Adam SGD) with mini-batch updates. At each iteration f, a batch B, C 'D size ng is sampled. For each example in the batch, we compute its ICA score defined in equation and convert these scores into continuous weights in the range [0, 1] via max-min normalization. We adopt max-min instead of the softmax used by |Wang et al. (2024) Calian al. (2025) because it preserves the relative differences between scores and avoids the exponential amplifictiion that can distort the contribution of low- and high-scoring examples. The per-example weights are given by
(X,, yii 0.) minjeB, (x 0,)
w (x(x, yı; 0.) maxjen, (x,, y 0.) minjen, 8(x,, 0.)
(8)
These weights, reflecting the relative utility of each example for minimizing the holdout loss, used to scale their contributions to the batch gradient:
BBd Σ
ge
wo(x; y:; O.) Vol(x,,
The resulting weighted gradient g: is used to update the model parameters via standard optimizer.
Because the ICA score, and hence the weights, evolve throughout training. this reweighting strategy adaptively adjusts each example's contribution the gradient updates according to the holdout loss the model would achieve trained on that example. Algorithm [ outlines how the ICA score is computed and used reweight gradient updates during fine-tuning.
5

---

end for Published as a conference paper at ICLR 2026
Algorithm Reweighting training examples using ICA scores for LLM fine-tuning
1: Input: Training set D; Holdout set Dhoi Pre-trained model parameters O; Number of training
steps T: Batch size ng; Optimizer OPTIMIZER
2: Initialize 0 +
do 4: Sample candidate set B, C DD of size n
3: for = 0, ...
5: for 1,...................... BB do
6:
ConditioralLose Loss; + [(y x 0,)
+ {(y: xi, Dho: 0.)
7:
8:
Score; + Loss; ConditionalLoss,
9:
end for
10:
Compute per-example weights within the batch using equation g
Compute the reweighted batch gradient g: on B, using equation
12: 0.+1 OPTIMIZER g:)
13:
Return Finetuned model parameters Or
Practical implementation the efficiency of Algorithm
In practice, we apply the following two techniques to further improve
When computing the in-context approximation in equation 6 including the entire holdout set as demonstrations is often infeasible due to prompt length constraints. straightforward approach to partition the holdout set into multiple subsets, compute scores for each subset, and average the results; however, this is computationally expensive. Instead, we select the top-k holdout examples most similar to each candidate using k-nearest neighbor (kNN) search in embedding space. In addition, rather than computing scores at every training iteration, we update them only R times periodically over the course of training. Experimental results show that, despite these practical approximations, our method consistently improves alignment performance.
The complete procedure incorporating these techniques is presented Algorithm in Ap- pendix B.4] We ablate the effects different choices of and R in Section 5.3 and analyze the computational overhead of our implementation in Section 5.5
EXPERIMENTAL RESULTS
We apply method to both SFT and preference-based alignment (DPO and SimPO), comparing ICA score-based reweighting to standard training (i.e., without reweighting) and to reweighting using scores computed by baseline methods, across multiple models and datasets.
5.1 EXPERIMENTAL SETUP
Datasets We consider two data selection scenarios. High-quality selection prioritizes expert-level data using smaller curated holdout set: for SFT, we use Alpaca as training data and sample high- quality holdout examples from its curated version, Alpaca-cleaned (Taori et al. (003);; for preference
optimization (DPO and SimPO), we use UltraFeedback-binarizee (Cui et al.] (023), which provides preference pairs with scalar quality scores, enabling the construction of high-quality holdout and test sets. Domain-relevant selection prioritizes examples relevant to a target domain using a domain- specific holdout set: for SFT, we use Yahoo Yahoo-Answen Answerss [7 and for preference optimization, we use SHP-2 (Ethayarajh et al.] (2022); both datasets contain domain labels, allowing selection of target domain for evaluating out-of-domain alignment. Dataset split details are provided in Appendix B.I

---

Published as a conference paper at ICLR 2026
Within the high-quality data selection scenario, additionally evaluate method reasoning- intensive benchmark, GSM8K (Cobbe et al. 2021), which consists of nugh-quality schdol math word problems. To test the method' ability to identify high-quality examples in noisy data, random fraction of chain-of-thought (Co'T) examples in the training split are intentionally perturbed. Experimental setup details provided in Section 5.4 with full results deferred Appendix C.4
Evaluation protocol and metries The objective of data selection in this paper is to choose a subset of training data such that model trained on it minimizes the holdout loss. Accordingly, we evaluate our method by measuring how closely the model's outputs align with the test set targets. To this end, we define the win rate as the percentage of pairwise comparisons in which response from a model trained with our method is judged more semantically aligned with the target output than that of model trained with standard (non-reweighted) or baseline methods. Comparisons are performed by GPT-4o (2024-08-06).
In the main experiments, each model is trained and evaluated once due to computational constraints. An analysis of training and evaluation variability is provided in Appendix B.2
Models and training We evaluate our approach on multiple model families and scales, includ- ing MAMMA---8B-Instee tuned using two parameter-updating paradigms: full cull-parameter fine-tuning and parameter-efficien LoRA (Hu al. 202T). Detailed training configurations are provided in Appendix B.3] and LoRA fine-tuning results with our method are reported in Appendix C.T
CAMMM-----B-nntruuue Qwen-3-8B, and Qwen-3-4B. Models are fine-
Default setting By default, we perform full- pal-merrreterr training and use the ICA score to reweight training examples. We compute ICA scores using the two practical techniques described in Sec- tion 4.3 (see Appendix [B.4 for additional implementation details). Specifically, for each candidate, holdout examples as in-context demonstrations and update scores for all train- ing examples R = 1 time (i.e., computing scores for all training examples only at the initialization step 0). When selecting the top k holdout examples in embedding space, we adopt all-mpnet- base-v2 Reimers & Gurevych, 2019) to compute embeddings.
use the top
Baselines In main experiments, we compare our method with standard training (without reweighting) and two closely related basclines using alternative scores: (1) RHO-Loss (Minder- mann et al. 2002). which approximates the holdout loss score using a model trained on the holdout set; (2) One-shot learning (Li et al. 2023b), which scores each candidate as the difference between the one-shot loss with the candidate included as context and the zero-shot loss without it. Detailed formulas for computing scores with these two methods are provided in Appendix B.5
In the SFT setting, we further compare against LESS Xia et al. (2024) and GREATS Wang et al. (2024), two data selection methods based on influence functions. For fair comparison, adopt the original selection scheme of each method: when comparing to LESS, we select the top 5% of examples based on precomputed scores before training; when comparing to GREATS, we select the top 50% of examples within each batch for gradient updates. Comparison results are summarized in Section 5.2 and provided full in Appendix
5.2
MAIN RESULTS
We report the performance of our method across SFT. DPO, and SimPO, comparing against standard training and baseline approaches, in Tables 2 and
Comparison to standard training Across all datasets and model families, incorporating our reweighting method leads to consistently better aligmment than standard training without reweight- ing. The improvements hold for both SFT and preference-basee aligmment, demonstrating that our method provides robust advantage across different learning paradigms.
Comparison to baselines Compared to baseline reweighting methods, our approach consistently outperforms one-shot learning across all settings, with win rates above 50%, and often exceeding 60%. Against RHO-Loss, which approximates the same selection criterion in equation but re- quires training auxiliary reference model, our method achieves comparable-and in some cases
7

---

Published as a conference paper at ICLR 2026
Table 1: Win rates (%) of our method against each baseline when applied to SFT across models and datasets (higher values indicate better performance of our method).
Alpaca
Yahoo Answers Topics
Model
w/o
RHO-Loss
One-Shot
w/o
RHO-Loss
One-Shot
LLaMA 3B
77.81
48.96
57.03
78,90
46.73
62.33
LLaMA 8B
80.55
49.92
62.11
85.10
54.03
66.93
Qwen 4B
71.21
50.56
56.82
80.30
49.93
58.73
Qwen 8B
82.92
51.21
58.33
82.93
54.43
63.13
Table Win rates (%) our method against each baseline when applied to DPO across models and datasets (higher values indicate better performance of our method).
UltraFFeedaac---innriie
StanfordNLP/SHP-2
Model
w/o
RHO-Loss
One-Shot
w/o
RHO--Lss
One-Shot
LLaMA 3B
61.05
46.85
57.60
79.90
56.70
60.20
LLaMA 8B
64.00
48.25
58.40
77.20
48.70
55.10
Qwen 4B
64.30
51.90
54.60
79.20
49.90
57.00
Qwen 8B
64.85
49.90
60.45
70.40
47.60
54.40
anperior-performanee while avoiding the additional cost of reference model training. These results hold for both SFT and preference-based training paradigms and across LLaMA and Qwen models different scales (3B and 8B).
For SFT on the method achieves comparable performance LESS and GREATS, with win rates slightly above 50% on both 8B LLaMA and Qwen models, while being more computationally efficient (see Ap- pendix C33 for full results). This efficiency advantage arises because LESS and GREATS rely on linaarizinni based influence functions, whose first-order assumptions be violated under batch data selection and require additional computation to control approximation error, whereas our method avoids this overhead by deriving scores from Bayesian formulation rather than first-order expansion.
A dataset, and using the same data selection scheme, our
5.3 ABLATION STUDIES
We conduct ablation studies under the SFT setting on LLaMA-3B-Instrucc trained on Ya- Answers.Topen dataset to examine the effect of key design choices in our method model performance. Results are detailed in Appendix B.6 with key findings presented below.
Large is not required for good performance To reduce computational cost, we use the top-k holdout examples selected via kNN in embedding space as in-context demonstrations, rather than the full holdout set. Using & 3 as the default, smaller larger values yield no improvement, with win rates below 50% relative to the default: 43.5% for 1, 48.0% for & 5, and 46.0% for & 10. The performance drop for À 3 may be due to additional holdout examples being less relevant. This indicates that a small number of holdout examples sufficient to maintain strong alignment while preserving computational efficiency.
More frequent score updates improve alignment We investigate the effect of the total number of score computations R model performance. In the default setting, scores are computed once at initialization (R = 1). Increasing R to 3, 5, and 9 yields win rates of 50.73%. 52.77%, and 51.6%, respectively. These results suggest that increasing R can further improve alignment, while very frequent updates (e.g., R = 9) appear unnecessary, as the dynamics of ICA weights (see Section 3.5 show that the relative importance of training examples stabilize in later stages of training.
8

---

Published as a conference paper at ICLR 2026
Table 3: Win rates (%) our method against each baseline when applied to simPO across models and datasets (higher values indicate better performance of our method).
UltraFeedDack--hnnnrieedd
StanfordNL.PSHHP
Model
w/o
RHO-Loss
One-Shot
w/o
RHO-Loss
One-Shot
LLaMA 3B
62.20
47.20
55,45
93,90
50.70
55,90
LLaMA 8B
64.70
49.90
57.70
63.30
46.50
51.30
Qwen 4B
64.95
47.65
51.20
66.00
46.80
49.60
Qwen 8B
65.55
50.60
54.35
77.10
45.50
53.60
Filtering is less effective As an alternative to our default reweighting approach, we consider percentile-based filtering using ICA scores, retaining only examples above a specific percentile. Simple filtering is generally less effective, with win rates below 50% relative to reweighting. A threshold of the 75th percentile yields a higher win rate (48.67%) than the 50th (40.80%) and 90th (40.07%), indicating that retaining some lower-scoring examples can be beneficial, while too many degrade performance, so the threshold must be chosen carefully. In contrast, adaptive reweighting eliminates this need for manually selecting filtering threshold by automatically adjusting example importance.
More advanced embeddings improve performance To evaluate the effect of different embed- ding models for selecting the top-k holdout examples, we test BAAl/bge-m3 (Chen et al.) (202)), stronger embedding model. BAAl/bge-m3 yields higher win rates (52%) compared to the default all- all-mpnet- mpnet-base--2,, suggesting that more capable embeddings can further improve the alignment achieved by our method.
5.4
ADDITIONAL RESULTS
We present additional results complement the main evaluations, using the SFT setting rep- resentative example. Specifically, we report absolute performance metrics that do not rely on a model-based judge and further evaluate the effectiveness of our method on reasoning reasoning-intensivee intensive benchmark. We highlight the main findings below, with detailed results deferred to Appendix C.2 and C.4
Consistency across metrics To complement GPT-judged win rates, we report absolute perfor- mance as measured by perplexity (PPL) and SeeeTccoreZZang et al. (2019) for SFT models trained on Yahoo Answers Topics using our reweighting method, compared against each baseline. As shown in Appendix C.2, the performance trends under these absolute metrics are consistent with those observed under GPT-judged win rates.
Mitigating the effect of noise on GSM8k To evaluate our method's 's ability to select valuable training examples from datasets that may contain suboptimal instances, conduct SFT both the original training split and a variant with intentionally corrupted CoT annotations. Details on how the data were corrupted and how the holdout set constructed provided in Appendix B. Pass accuracy under zero-shot CoT prompting is reported in Table 16 in Appendix C.4 with key findings summarized below.
SFT trained on the original high-quality split achieves the best performance. Using our reweighting method on the corrupted split yields the second-bes results, with only small performance gap. These results show that reweighting noisy training data with our method can achieve performance comparable training on the original high-quality dataset, highlighting its potential to mitigate the impact of substantial noise, even on complex reasoning tasks.
5.5 ANALYSIS
Computational complexity Our method introduces two sources of overhead beyond standard fine-tuning: (i) one-time precomputation obtain embeddings, and (ii) periodic score updates.

---

Published as a conference paper at ICLR 2026
The latter is dominated by the in-context approximation term, which involves additional forward passes and accounts for most of the computational cost. We measure runtime on four NVIDIA A6000 GPUs (48 GB each) for LLaMA-3B Instruct fine-tuned on Yahoo Answers Topics. Precom- putation is negligible, on the order of seconds, and the additional time for score computation and reweighting is reported adds only ~ 1..5% overhead, compared to roughly 10% for RHO-Loss and 4% for one-shot learning. Full results are provided in Appendix B.7
percentage relative the runtime of standard fine-tuning. Our method
Score distribution across training examples We analyze SFT on Yahoo ..wrsnnnnswsssa.. Toopic to ex- amine the scores assigned to training examples. Using a holdout set sampled from the Sports topic, we compute average score for examples from each domain (Figure[T). Examples from Sports receive higher scores under both our method and RHO-Loss compared to One-Shot Learning and other top- ics, demonstrating that our method achieves strong alignment while requiring less computational overhead.
Dynamics of ICA weights We observe that ICA weights adjust dynamically during the training process, with most changes occurring early and stabilizing as training progresses. Using SFT results Yahoo Answers. Topics with R. update and subsequent updates are 0.89, 0.75, 0.69, and 0.71. Since the weights stabilize later, very frequent score updates are unnecessary. Additional analyses in Appendix exammin response pat- terns produced by models trained with our method, and Appendix D.3 presents rnstroction-response pairs with high and low ICA scores, offering further insight into our method's effectiveness.
example, correlations between weights from the first
RHO-LoSS
Health
One-shot learning
Business & Fipence
Computers Internet
Science & Mathematics
0.75
1.00
O150
ports
Entertainment
Politics Government
Fouration Referance
Family Relationships
Figure 1: Average scores assigned to training examples from each domain (normalized to [o, 1).. The target domain is highlighted in red, and different colors indicate different scoring methods, with our method in blue. Higher scores indicate stronger alignment with the target domain.
CONCLUSION AND DISCUSSION
We propose principled, resource-efficient framework for data selection in LLM fine-tuning. Our approach leverages an in-context approximation (ICA) to estimate the holdout loss of the model after including each candidate example in training, without requiring additional finetuning or a ref- erence model. The resulting ICA scores are used to dynamically reweight gradient updates during fine-tuning. Empirical results show that ICA score-based reweighting consistently improves model alignment across SFT, DPO, and SimPO over diverse datasets and backbone architectures, with only marginal computational overhead (~ 1.5%).
We also note limitations and directions for future work. We focus settings where a high-quality holdout set is available to estimate the importance of training examples. If the holdout set is noisy or unrepresentative, generalization to unseen data may be impaired, even if alignment appears strong. Additionally, our experiments use off-policy training paradigms. Directly applying ICA to on on-policy methods, such as PPO, would require frequent recomputation of scores for all newly generated data, creating a substantial computational bottleneck. Addressing these challenges is promising direction for future work.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
All authors have read, adhere to, and explicitly acknowledge the ICLR Code of Ethics This work does involve human subjects private user data, and all datasets used in experiments publicly available and appropriately cited, with full details of data processing and holdout/test splits provided in the appendix ensure transparency and reproducibility.
All experimental procedures and analyses adhere to accepted research integrity standards, and we disclose any potential conflicts of interest: none exist. This work does not involve applications that directly pose safety or legal risks beyond standard LLM use.
The assistive use LLMs in preparing this paper, refine grammar and improve the clarity of the text, is documented in Appendix E. and the authors take full responsibility for all content.
REPRODUCIBILITY STATEMENT
All code and materials used in our experiments are available at httts:///gthhii com/ microsoft
To facilitate reproducibility, we provide detailed de- scriptions of all datasets used, including holdout and test set construction and dataset splits, in Ap- pendix B.I The algorithm used in our experiments detailed in Appendix B.4 with full training parameters, LoRA configurations, and evaluation settings provided in Appendix B.3 Prompts used for querying LLMs when computing ICA scores, as well prompts used for GPT to judge win rates, are provided in Appendix D.I
For theoretical results, our method builds on the holdout loss expression introduced in Mindermann et al. (2022), and we reproduce the complete derivation in Appendix A Together, these materials allow others to reproduce our experiments and verify our results.
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Alexander Bukharin and Tuo Zhao. Data diversity matters for robust instruction tuning. arXiv preprint 1rXiv:2311.14733 2023.
Dan A Calian, Gregory Farquhar, lurii Kemaev, Luisa M Zintgraf, Matteo Hessel, Jeremy Shar, Junhyuk Oh, András György, Tom Schaul, Jeffrey Dean, et al. Datarater: Meta-learned dataset curation. arXiv preprint arXiv:2505. 17799 2025.
Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models. arXiv preprint arXiv:2307.06290, 2023.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, nulti-functionnaliyy multi-granularity text embeddings through self-knowledge dis- tillation, 2024. URL
org/ abs/2402 03216
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint N1I1::23370087000 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Utrafeedback: Boosting language models with high-quality feedback. 2023.
11

---

Published as a conference paper at ICLR 2026
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers, 2023. URL httttt://azxiii// org/ abs/ / 221222222..... 10559
Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangman He. Less more: Improving Ilm alignment via preference data selection. arXiv preprint arXiv:2502. 14560, 2025.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00233 2022.
Karel D'Oosterlinck, Winnie Xu, Chris Develder, Thomas Demeester, Amanpreet Singh, Christo- pher Potts, Douwe Kiela, and Shikib Mehri. Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment. Transactions of the Association for Computational Linguistics, 13:442 4600, 2025.
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels, 2024.
/ttttp:://axxx..oro///a
12926
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In International Conference on Machine Learning, pp. 5988-6008. PMLR, 2022.
Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment perfor- mance of generative language models. arXiv preprint arXiv:2404.09824, 2024.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pp. 2242-2251. PMLR, 2019.
David Grangier, Pierre Ablin, and Awni Hannun. Bilevel optimization to learn training distributions for language modeling under domain shift. In NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2023.
Yuxian Gu, Li Dong, Hongming Wang, Yaru Hao, Qingxiu Dong. Furu Wei, and Minlie Huang. Data selection via optimal control for language models, 2025. URL http:ttss////ii...r.t.t.
2410.07000 07064
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
7/arxiv. wttpvv/.rrgaab///1000/ 0166
Chengyu Huang and Tanya Goyal. Derm: A heuristic to measure response pair quality in preference optimization, 2025. URL Intpps: arxiv. org/ abs httpa/bb///55111/ 2506.
Feiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang, Rongxing Du, Anit Kumar Sahu, and Ruoxi Jia. Get more for less: Principled data selection for warming up fine-tuning in lims. arXiv preprint arXiv:2405.08 002744, 2024.
Angelos Katharopoulos and François Fleuret. Not all samples created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525-2534 PMIR, 2018.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464-5474, PMLR, 2021.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting 11m performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308. 12032, 2023a.
Yunshui Li, Binyuan Hui, Xiaobo Xia. Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Ling- Hao Chen, Junhao Liu, Tongliang Liu, et al. One-shot learning as instruction data prospector for large language models. arXiv preprint ar Aiv:::23 10302, 2023b.
12

---

Published as a conference paper at ICLR 2026
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for align- ment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15685, 2023.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. arXiv preprint arXiv:2308.0707 2023.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-122235, 2024.
Sören Mindermann, Jan M Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Win- nie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prior- itized training on points that are learnable, worth learning, and not yet learnt. International Conference on Machine Learning, pp. 15630-15649, PMLR, 2022.
Tetsuro Morimura, Mitsuki Sakamoto. Yuu Jinnai, Kenshi Abe, and Kaito Ariu. Filtered direct preference optimization. arXiv preprint arXiv:2404.1388 13846, 2024.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730 27744, 2022.
Rui Pan, Dylan Zhang, Hanning Zhang, Xingyuan Pan, Minrui Xu, Jipeng Zhang, Renjie Pi, Xiaoyu Wang, and Tong Zhang. Scalebio: Scalable bilevel optimization for 11 data reweighting, 2025.
URL BtRt:sts:
org. abs/ 19976
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 01153728-537411 2023.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks, 2019. URL htl.attss///xx...org////
10084
Han Shen, Pin-Yu Chen, Payel Das, and Tianyi Chen. Seal: Safety-enhanced aligned 11m fine-tuning via bilevel data selection. arXiv preprint arXiv:2410.07471, 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instmaction-oolloinng llama model, 2023.
Jiachen T. Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia. Data shapley in one training run, 2025. URL
Jiachen Tianhao Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. Greats: Online se- lection of high-quality data for I1m training every iteration. Advances in Neural Information Processing Systems, 37:131 191111997-3112, 2024.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint 1nXiv:2109.01655 2021.
Junkang WU, Xue Wang, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Alphadpo: Adaptive reward margin for direct preference optimization. In Forty-second International Conference on Machine Learning.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 3-dpo: Direct preference optimization with dynamic B. Advances Neural Information Processing Systems, 37: 17:9449 129966. 2024.
13

---

Published as a conference paper at ICLR 2026
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming- Aiung- Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508.05629, 2025.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Se- lecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333 2024.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34227, 2023.
Ping Yu, Weizhe Yuan, Olga Golovneva, Tianhao Wu, Sainbayar Sukhbaatar, Jason Weston, and Jing Xu. Rip: Better models by survival of the fittest prompts. arXiv preprint 18857.2501..8578, 2025.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv:1904. 2019.
Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning. arXiv preprint arXiv:2402.04833 0448333 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006-55021, 2023.
A DERIVATION OF EQUATION 3 UNDER A BAYESIAN FRAMEWORK
2022 for completeness. Let O, 0- (D,) denote the model trained on Dr, and define the loss as the negative logliikelhoodd ((y
We reproduce here the derivation from Mindermann
x; 0) log ply x: 0). We use xh and yoo denote the collections of inputs and outputs the holdout examples, respectively.
Under a Bayesian framework, the holdout loss of model trained with particular example can be expressed as
log xx",:2, Dr (x,))) log ply x; xx t De) ply x²" xx; Di)
ply x, xo; xxho;D,) Dz)
log ply x; yho hho D.) p(ylio x(ho;D,)
ply x; D:)
Ely 0.) Ely x; 0 (D, U'o.o)
where the first equality applies Bayes' rule, the second uses a conditional independence assumption, ie., ply; X., xj, D.) ply: x;, Dz), for any i of j. and the last line drops the candidate- independent term.
B DETAILS OF EXPERIMENTS
B. DATASET SPLITS
We provide details on the construction of the training, holdout, and test sets for all datasets used in our experiments.
Alpaca and Alpaca-cleaned TToor al. 2023) Alpaca-cleaned is curated version of Alpaca. The holdout set consists of the first 10,000 examples from Alpaca-cleaned, and the test set consists of the last 10,000 examples. For training, use all Alpaca examples except those whose corresponding examples in Alpaca-cleaned have been reserved for the test resulting in 84,022 training examples.
14

---

Published as a conference paper at ICLR 2026
Table 4: Win rates of our method against each baseline for
on Ya-
hoo Answers Topics, averaged over three GPT-based evaluations from single training run, with corresponding standard deviations reported.
Ours against
w/o
RHO-Loss
One-shot
Win Rate (%) 85.10 0.30
54.03 + 0.20
66.93 + 0.50
Table 5: Win rates (%) of our method against each baseline on Yahoo_ Answers.Topics, averaged over three independent runs with different random seeds, reported for both LLaMA-3-8B-nnttnue and Qwen-3-8B. Standard deviations are also provided for reference.
Model
w/o
RHO-Loss
One-shot
LLaMA 8B
82.86 1 1.96
49.92 1 0.39
62.59 1 0.57
Qwen 8B
81.67 + 1.09
54.23 + 0.18
63.81 1: 0.60
Yahoo Aattor_AnswerssToo Topics The holdout and test sets each contain 3,000 examples from the Sports domain. The training set consists 10,000 examples from each of the remaining domains, together with the holdout examples, totaling 100,300 training examples.
AetraFeedback-binartie binarized (Cui et al. 2023) The dataset is split into a training set (66,282 examples) and test set (2,000 examples). From the training set, a holdout of 5,147 pairs is selected, consisting of examples where the chosen response has quality score 2 and the rejected response has a quality score 2 All examples in the training set are used
for training.
StanfordNI.PSHHP (Hthayarajh et al. 2022) We use the Baking domain as the target domain. From the official test split, we rank Baking examples by (score_A + score_B), selecting the top 1,000 as the holdout set and the next 1,000 as the test set. For training, we use the official train split, selecting 1,000 examples from each domain and including the holdout set.
GSM8K (Cobbe et al. 2021 This dataset contains 8.5K high-quality grade school math word problems requiring multi-step reasoning. From the training split, we set aside 1,000 examples as high-quality holdout. The remaining training examples used the model training set, of which randomly selected 40% are intentionally corrupted (e.g., via CoT dropout, shuffled CoT, or CoT re-generation using weaker model). The original test split is used for evaluation.
In main experiments, the holdout set included in the training data to ensure fair comparison with baselines that do not use reweighting. We additionally conduct experiments where the holdout set is used only for scoring in Appendix
B.2 TRAINING AND EVALUATION VARIABILITY
Each model in the main experiments trained and evaluated only once. To verify evaluation stability, we repeated the GPTT-ased evaluation three times for LLaMA-3-8B-Insstne on Ya- -wt.seswss_oppccs Table 4 reports the average win rates of our method against each baseline, along with standard deviations across these runs. Higher win rates indicate better alignment of our method, and the small standard deviations show that the evaluation results are stable across runs.
To assess training variability, we performed three independent SFT runs with different random seeds and Qwen-3-8B models on Yahoo Answers.Toppe
for each of the MEM-A-3-B--Instte reports the average win rates of our method against each baseline, with standard deviations provided for reference. The mean performance across these training runs aligns closely with the single-run results reported in the main experiments.
Table 5
15

---

Published as a conference paper at ICLR 2026
Table 6: Detailed training parameters for full fine-tuning, LoRA, and evaluation.
Full Fine-tuning Parameters
SFT
DPO
SimPO
toyph
bíloat16 attentott cosine
bfloat16
bfloat16
attn attleemplemnnnniionn
flash aushaattentoon22 cosine
atassaattn
Ir s..cheuullrr. .ype
cosine
caadient.ccuumulaionn
16
learning_re
10-5
1 x 10
1 x 10
max_length
1024
max-prompt.
1024 1024
1024 1024
max.seq..engee
num wwm.train.com troch optim
1
adamw
adamw
adamw_torch
per_device_ httpviieetraii_bbee batch_sstessiie
per_device_eve httpiiie_eeva_bbcchiiiieie seed
42
42
42
warmup ratio loss Jope beta
0.1
0.1
0.1
all
sigmoid
sigmoid
2.5
bata.
0.55
sft_weight
0.0
disable.dropout
Truc
LoRA Fine-tuning Parameters
learning.rate
1 x 10
x 10
x 10
lora_r
Iora lora-alpha
lora_dropout
0.1
0.1
0.1
www.farget_mmdduee
Iq-proj. k.proj. y-proj, up-proj. down-proj. o.proj, gate-proj] CAUSAL_IMM C. CAUSAL_IM
lora_task_typ
Evaluation Parameters
tester
azure GPT
api version
2025-01-01 111-011-1--peee -01-previeww 20244000024-
model
lemperature
top-p
0.95
seed
None
max_tokens
1600
B.3 DETAILS OF TRAINING AND EVALUATION CONFIGURATIONS
We summarize the training and testing configurations for full fine-tuning, LoRA fine-tuning, and evaluation Table
B.4 IMPLEMENTATION DETAILS OF ALGORITHM I
We provide details of the two techniques introduced in the main text for improving the computational efficiency of Algorithm
Selecting in-context demonstrations via kNN Instead using the full holdout set Dno for in- context learning, condition smaller, more relevant subset of Dno selected using embedding similarity.
Specifically, first precompute and store embeddings of the inputs for all training and holdout examples:
Fz) for 1...., | hD = f(x!", (!)) for = 1, 1,..., |D**),
where f denotes the embedding function (by default, we use all-mpnet-base-ye RReimers & Gurevych 2019)).
16

---

Published as a conference paper at ICLR 2026
Algorithm 2 Enhanced Algorithm with Computational Efficiency Techniques
1: Input: Training set D; Holdout set Dnoi Pre-trained model parameters O: Number of training steps T; Total number of score computations Rt kNN hyperparameter Embedding function
f: Batch size ng; Optimizer OPTIMIZER
2: Initialize 0g f- 0
3: D, Dho <- PREPROCESSING(D, Dho J, k)
4: for = 0, , do
5: if t mod [P]
then
// Recompute scores every P steps
ngR
6:
for (x,, Yi, h, Score;) E do
7:
0 <- GETTEENONSTTRATOONSTY;;;; Dro, hoohh k:)
8:
Conditionalloss; + f(y: x, C,
9:
Loss; f (yy; x;
10:
Score; Loss; Conditionalioss.
11:
end for
12:
end if
13:
Sample batch B, C D of size ng
14:
Compute per-example weights within the batch using equation g
15:
Compute the reweighted batch gradient ge B, using equation 9
16:
0z+1 + OPTIMIZER g:)
end for
Return Finetumed model parameters O
18:
function PREPROCESSINGC.. Dno; J,k)
for (x) D do
Compute h2 + y() 21: Initialize Score; +-
20:
// Embedding for training example i
22:
end for
23:
Update training set + y(x,, hú, Scores))
24:
for (xho y in Dno do
25:
Compute ho + f(xth" y!oo
// Embedding for holdout example
26:
end for
27:
Update holdout set Dno < f(xho, yho bh) |
28:
end function
29: function GETDEMONSTRATIONSSTTN Dbo: k)
I1 Select top k holdout examples using
embedding similarity
C {(xho, ho) (xoh",,"ho) e Dho ho is among the l nearest to h}
31: Return C
32: end function
For each candidate (x E 'D with embedding h, we select the top-k holdout examples whose embeddings are closest h to form demonstration subset C* We replace the full Dno with this subset when computing the ICA score (line 6 of Algorithm Then the ICA score can be computed
s(x, y: 0.) Ely x; 0.) f(y x, Ch: (h:0.)) 0,),
where
C : (xx hoo E Dno h is among the & nearest to h).
Although in our experiments we recompute the kNN search at each scoring step, the demonstration subsets be precomputed once and reused across iterations to further amortize the cost.
Periodic score updates In Algorithm scores are computed at every training step. To improve efficiency, instead perform score computation only R times during training. At each recomputa-
17

---

Published as a conference paper at ICLR 2026
Table Effect of Top-k on win rates; higher values indicate better performance for the correspond- ing k.
Top-k
10
Win rate (%)
43.50
48.03
46.07
Table 8: Effect of total number of score computations R on win rates; higher values indicate better performance for the corresponding R.
R
5
Win rate (%
50.73
52.77
51.60
tion point, scores for all training examples are updated and stored; in the intervening steps, the most recent scores are reused to determine weights.
The reweighting algorithm incorporating these two techniques is presented Algorithm - where the score update frequency is determined from the training set size, batch size, and the total number of score computations R.
B.5 IMPLEMENTATION DETAILS OF BASELINES
We provide additional details on how scores are computed for each reweighting baseline. These scores used in place of the ICA score within reweighting framework. For the subset lection methods LESS (Xia et al. 2024) and GREATS (Wang et al. (024), we reimplement their publicly available code and follow their original data selection procedures: selecting the top 5% before training for LESS, and 50% of each batch for gradient updates for GREATS.
RHO-Loss (Mindermann al.) 2022) approximates the holdout loss score in equation by replacing the second term with separate model trained once the holdout set. For each candidate, the resulting score is
SRHO-Lons (x, f(y x; 0,)
E(y x; 0* (Dnn)).
replicate this method, we train the target model on the holdout set Dno to obtaín 0* (Dno).
One-shot learning (Li et al. 2023b) computes a score for each candidate as the difference between the one-shot loss with the candidate included context and the zero-shot loss without it:
Sone shod (x,y) L(Dr.; 0,) L(Dno y);
where, for consistency with our setting, use the pretrained model O, to perform this one-shot evaluation and compute losses on the holdout set Dno instead the predefined subtasks used in the original paper.
B.6 RESULTS OF ABLATION STUDIES
Tables 110 summarize the ablation studies on LLaMA-3B-Instrun trained on Ya-
Answers.Topice Win rates indicate the percentage of responses preferred over the default the embedding model). Each table
setting (i.e.. k 3, R = 1, using all-mppet-base-v2 corresponds to varying one design dimension: Top-k (Table 7) total number of score computations R during training (Table 8) percentile threshold for filtering (Table 9); and embedding model (Table (T). Higher win rates indicate better performance the corresponding design choice; 50% corresponds to parity with the default.
18

---

Published as a conference paper at ICLR 2026
Table Effect of percentile threshold for filtering on win rates; higher values indicate better perfor- mance for the corresponding percentile. Here, indicates retaining examples with scores above the æ-th percentile.
Filtering 2 x
50
75
90
Win rate (%)
40.80
48.67
40.07
Table 10: Win rates under stronger embedding model against the default; higher values indicate better performance for the stronger model.
Embedding Model
Win rate (%)
BAAI/bge-m3
52.13
B.7 COMPUTATIONAL OVERHEAD
We report the computational overhead of our method compared baseline methods. Table TT presents the runtime for embedding precomputation, score computation, and total additional runtime relative to standard fine-tuning (computed as additional time divided by the runtime of standard fine- tuning). We train LLaMA-3B BELLA--B-Insttae Instructtt on - Topics using the default settings of our method, and measure runtime on four NVIDIA A6000 GPUs (48 GB each).
C ADDITIONAL EXPERIMENTS
C.I ADDITIONAL RESULTS USING LORA
To assess the effectiveness of our method across different parameter updating paradigms, we conduct reports win rates LoRA with our reweight-
LoRA training using L.LaMA-3-BB-Insmme ing method relative to regular LoRA training across various alignment methods and datasets. Each value represents the percentage of responses judged closer to the target by EPT-44-2024-08 with higher values indicating better performance. The results demonstrate that method consistently improves alignment, even under the LoRA parameter updating setting.
Table repports
C.2 ABSOLUTE METRICS ON YAHOOLANSWERS.TOREE
Table 13 reports perplexity (PPL) and BERTScore for SFT models trained Ya- hoo Answers. Topics under different reweighting methods. Lower PPL and higher BERTScore in- dicate better model performance.
C.3 COMPARISON WITH LESS AND GREATS
Tables 14 and report win rates of our method against LESS and GREATS under the same data selection paradigm for SFT on Yahoo Answers Topics, together with PPL and BERTScore. Comparison results on GSM8K are reported in the following section, using Pass accuracy. Run- time computing scores for training examples using each data selection method provided in Table 17
C.4 RESULTS ON GSM8K
Table reports @ass@ accuracy for SFT models trained on the original and corrupted splits of GSM8k using reweighting method and baseline approaches.
19

---

Published as a conference paper at ICLR 2026
Table 11: Runtime for embedding precomputation and score computation (seconds), and total ad- ditional runtime relative to standard training (percentage). Higher values indicate greater computa- tional cost.
Metric
Ours
RHO-Loss
One-shot
Precomputation (8)
2
5400
2
Score Computation (s)
800
800
2000
Additional Runtime (%
10
4
Table 12: Win rates of LoRA trained with our method relative to LoRA trained without our method using LLaMA-3-8B-Insmnne Higher values indicate better performance.
Alignment Method
SFT
DPO
SimPO
Alpaca
Yahoo
UltrFeeedback
SHP-2
UltraFeedback
SHP-2
Win rate (% 1)
71.55
68.13
61.23
57.33
56.03
60.13
Table 13: Absolute-metric performance of SFT-trained models on Yahoo www__Anssess_pppp under different reweighting strategies (including reweighting), evaluated using perplexity (PPL) and BERTScore. Lower PPL and higher BERTScore indicate better model performance. Best and second-best results highlighted in bold and underlined, respectively.
LLaMA 8B
Qwen SB
Method
PPL (4)
BERTScore (1)
PPL. (L)
BARTScore (1)
Ours
11.23
0.84
10.30
0.87
w/o
16.71
0.80
13.04
0.82
RHO-Loss
11.49
0.84
10.00
0.88
One-shot
14.92
0.82
12.02
0.82
C.5 EXCLUDING HOLDOUT EXAMPLES FROM TRAINING
In our main experiments, the holdout set is included in training to ensure fair comparison with baselines that do not use reweighting, which otherwise have no access to holdout examples. To examine the potential impact of including holdout examples on performance, Yahoo Answers Topic dataset, keeping the holdout set strictly for scoring. Results are presented in Table 18 Even without training the holdout set, method outperforms the standard baseline and one-shot learning, although it performs slightly below RHO-Loss, which trains a reference model on the holdout set. Nevertheless, our method remains comparable RHO-Loss while using holdout examples only as in-context demonstrations for scoring.
reran SFT on the
C.6 PERFORMANCE WHEN USING THE HOLDOUT SET ALONE
To show that test performance relies the broader training data and the holdout set alone does not suffice, we train the model solely on the holdout set (see results in Table [9). As expected, its performance is lower than our method trained on the full training with TCA reweighting. highlighting the benefit of combining broader training data with data selection guided by the holdout set.
D ADDITIONAL. ANALYSIS
D.1 PROMPTS FOR IN-CONTEXT APPROXIMATION AND MODEL EVALUATION
Table 20 shows the prompts we use. The first section provides standard query, the second includes the query with holdout examples in-context demonstrations used for computing ICA scores via
20

---

Published as a conference paper at ICLR 2026
Table 14: Comparison with additional baselines under the SFT setting on Yasoo__nswens_ TTopics using LLaMA-3-8B. Win rates computed within the same selection paradigm. Lower PPL and higher BERTScore indicate better performance. Best values are shown bold.
Win rate (%)
PPL (4)
BERTScore (1)
Method
(Ours vs Baseline)
Online selection
GREATS (50% per batch) Ours (50% per batch)
52.03
22.07
0.78
22.01
0.79
Pre-training selection LESS (Top 5%)
51.40
20.55
0.80
Ours (Top 5%)
18.91
0.80
Table 15: Comparison with additional baselines under the SFT setting on Yahoo A using Qwen-3-8B. Win rates computed within the same selection paradigm. Lower PPL and higher BERTScore indicate better performance. Best values are shown in bold.
Win rate (%)
PPI.
BERTScore (t)
Method
(Ours vs Baseline)
(1)
Online selection
GREATS (50% per batch) Ours (50% per batch)
50.91
19.70
0.80
19.74
0.80
Pre-training selection LESS (Top 5%)
50.90
17.24
0.77
Ours (Top 5%)
17.03
0.78
Table 16: @ass@1 accuracy (%) GSM8k under zero-shot CoT prompting for LLaMA-3-80 3-B and Qwen-3-8B, trained with SFT using different data selection strategies. Best and second-best results are highlighted in bold and underlined, respectively.
Method
L.LMA-3-8B
Qwen-3-8B
Without reweighting
Pretrained
75.28
80.52
SFT (original)
81.91
83.27
SFT (corrupted)
52,75
62.40
Reweighting based methods
RHO-Loss
79,50
81.16
One-shot
72.42
76.19
Ours (reweighting)
79.95
83.04
Pre-training selection
LESS (Top 5%)
77.63
80.81
Ours (Top 5%)
78.92
80.81
Online selection
GREATS (50% per batch) Ours (50 % per batch)
80.03 80.18
80.50
80.49
ICA approximation, and the third shows prompts used for evaluation, where GPT judges which candidate response is closer to the target output.
D.2 RESPONSE LENGTH AS AN ALIGNMENT INDICATOR
Our method effectively captures the characteristics of the holdout dataset. In Ya- hoo Answers Topics, which consists of daily conversational data collected from internet users, the
21

---

Published as a conference paper at ICLR 2026
Table 17: Score computation time (in seconds) on LLaMA-3-8B for our method, LESS, and GREATS when training Yahoo Answers. Topics with SFT. Times for LESS and GREATS based on our reimplementation. For LESS, the two numbers correspond to influence score compu- tation and periodic model retraining to address the circular dependence between data selection and model updates.
Dataset
Ours
LESS
GREATS
Yahoo Answers Topies
808
1,394 + 1,800
1,004
GSM8K
117
310 900
704
Table 18: Win rates of our method against each baseline reweighting method for models trained petseess.Topiee using Dho as a strict holdout for scoring (not included
with SFT on Yahoo in training). Higher win rates indicate better performance of method. PPL. and BERTScore also reported, with the best and second-best values indicated in bold and underlined, respectively.
LLMMA-3-8B
Qwen-3-8B
Win rate (%)
PPL
BERTScore
Win rate (%)
PPL
BERTScore
Method
(Ours vs Baseline)
(L)
(7)
(Ours vs Baseline)
(L)
(t)
w/o
89.27
26.77
0.72
87.4
22.44
0.80
RHO-Loss
47.29
12.08
0.81
49.21
12.94
0.85
One-shot 71.33
19.45
0.79
67.02
18.73
0.82
Ours
14.05
0.81
13.29
0.86
Table 19: Comparison of our method against the baseline model trained only on the holdout set
Dno for SFT on Yahoo_Answent AnswerssTTpics.. Win rates indicate the percentage of examples where our method is preferred over the baseline; higher values indicate better performance our method. PPL and BERTScore are also reported. with best values indicated in bold.
LLaMA-3-8B
Qwen-3-8B
Win rate
PPL ())
BERTScore (1)
Win rate (%)
PPL (L)
BERTScore
Method
(Ours vs Baseline)
(Ours Baseline)
(t)
Dno-only
58.13
12.40
0.82
56.80
11.92
0.82
Ours
11.23
0.84
10.30
0.87
Sports category contains notably shorter responses, with an average token length of 62.07 compared to 86.01 in other categories. Moreover, the proportion of extremely short answers (token length 5) substantially higher in the Sports category (16%) ) than in others (4%). The model trained using our approach reflects these patterns, producing similarly concise responses to sports-related questions, as shown in Table 21
D.3 EXAMPLES OF HIGH- AND LOW-SCORING RESPONSES
To provide intuition for the effectiveness of our scoring method, we select two example pairs and present them in Table 22 In each pair, the prompt is the same, but the responses differ: one is from Alpaca and the other from Alpaca-cleaned. For each response, we show the score assigned by our method. We observe that responses receiving higher scores tend to exhibit clearer structure, more complete coverage the instruction's 's intention, and more precise or domain-relevan content, while low-scoring responses often display redundancy, incomplete answers, missing key details. These examples illustrate that our method prioritizes clearer structure and precise content, helping explain why it improves model alignment.
22

---

Published as a conference paper at ICLR 2026
Table 20; Prompts for ICA computation and model evaluation
Query
You are an expert assistant.
Answer the following question:
(question)
Query with In-Context Demonstrations
You are an expert assistant. Follow the examples:
Q:
(example.qussionn.]
A:
(example.answer.))
Q:
[oxemplo.question.2
A:
(examplc.answer.2]
Answer the following question: [question]
Evaluation
You expert assistant. Given question, standard answer, and two candidate answers, indicate which candidate closer to the standard. Instructions:
Choose the candidate closer the standard, not necessarily the higher quality.
2. Consider content, length, and style relative to the standard answer.
3. Be concise; output only JSON object with the winner.
Question: (question) Standard answer: (standardLanswer) Candidate 1: {candi date_ansver.
Candidate 2: {candi daze answer 2.
Output format:
If candidate 1 is closer:
("winner": 1]
If candidate 2 is closer: ("winner": 2}
Table 21:
Example responses from the Sports category showing alignment of model outputs with target domain response length.
Response Example
Question: Are there any sports where it's advantage be little?
Answer from dataset: Jockey.
Answer from model after SFT: Yes, would say that wrestling is one of ther.
Answer from model after Curs+SFT: Polo.
Answer from GCaMA-3B-Insttee can be an advantage in some sports, there are indeed some sports where being shorter can be beneficial. Examples include: gymnastics, wrestling, boxing, table tennis, fencing, weightlifting, and cycling, Shorter athletes movement, or easier maneuverability, though technique, strength, and endurance remain crucial.
before fine-tuning: While being tall
benefit from lover
of gravity, faster
E USE OF LARGE LANGUAGE MODELS
In adherence to ICLR 2026 policy, we disclose the assistive use of Large Language Models (LLMs) in the preparation of this paper. The models were employed to refine grammar and improve the
23

---

Published as a conference paper at ICLR 2026
clarity of the text. The authors have reviewed all LLM-generated contributions and take full respon- sibility for the content and integrity of this work.
24

---

Published as a conference paper at ICLR 2026
Table 22: Example pairs from Alpaca and Alpaca-cleaned with scores assigned by our method
Example
Instruction: Analyze the following document and group the topics it talks about.
Response (score 1.99) Uber being leading company in the side-sharing industry since its launch in 2009. has di disrupted the transportation industry, creating a business model in which it acts as a middleman between two parties: the rider and the driver, Uber's main service offering includes mobile app and website, which allows riders to request ride from available driver. This ride can be paid for using Uber's s payment system, which was developed in partnership with a variety of third-party processors. The comрany also offers various add-on services, such as \UberPool", which allows riders money, and \UberRush", which
to share ride with other riders allows riders to schedule ride for future time.
Response (score 7.45): Topic 1: Ride-sharing industry
Tepic Uber's business model
Topic 3: Mobile App/Nebsite
Tepic 4: Payment System
Tepic 5: Add-on services (UberPoo1 & UberRush)
Example 2
Instruction: Describe the types of facial recognition technol ogy used in security systems.
Respense (score 5.56): Facial recognition technology used various security systems to identify and verify individuals based en their facial features. Some comnon types of facial recognition technologies used in security systems include:
2D Facial Recognition: This is the recognition technology, where the system captures a 20 image of person's s face and compares it with the images in its database to find a match.
commonly used facial
2. 3D Facial Recognition: This technology
sensors capture the shape and contours of person's face, providing more accurate and reliable identification than 2D recognition.
3.
Thermal Facial Recognition: This technology uses thermal imaging capture the heat patterns that are unique each individual's face.
This technology is particularly useful in low-light conditions where the subject is wearing items that obscure their face.
4. Skin Texture Analysis: This advanced facial recognition technology captures the unique patterns and lines on person's s skin to identify and verify their identity.
These are some of the common types of facial recognition technologies that are used in security systems, and they offer varying levels of accuracy and reliability in identifying individuals.
Response (score = 15.40): Facial recognition technology used in security systems can come in many forns, such as biometrics-based authentication, 3-D face recognition, detect facial features. Biometrics-based authentication requires users to provide physical attributes such as fingerprints, iris scans, and facial recognition for access. 3-D face recognition is more secure and uses more than txo-dimensional image recognition to identify person. Algorithms detect the shape, size, and position of facial features, allowing for more accurate verification.
the use of algorithms to
25


Reviews:
[{"review_id": "oCMoxyOeL6", "reviewer": "Reviewer_tdGW", "summary": "This paper introduces In-Context Approximation (ICA), a resource-efficient framework for data selection and reweighting when fine-tuning Large Language Models. At its core, the method estimates the value of a training example by measuring how it would reduce the loss on a small, curated holdout set. Crucially, this estimation is performed using the model's in-context learning capabilities. By dynamically reweighting gradient updates based on these ICA scores, the method prioritizes high-value data, leading to improved model alignment with minimal computational overhead.", "questions": "+ In the description of the demonstration set C (line 114), what does the function s(...) represent? It appears to be a formatting function but is not explicitly defined.\n+ Could a direct comparison of the ICA scores and the resulting data rankings be provided against those computed by the RHO-Loss method? This would offer deeper insight into how closely ICA's efficient approximation mirrors the baseline's more expensive calculation.\n+ How sensitive is the method's performance to the quality of the holdout examples used for in-context demonstration? It would be insightful to see an ablation study where demonstrations are drawn from random training examples instead of a curated, high-quality set, especially in scenarios where the training and test data share a similar distribution.\n+ Could this data selection method be effectively applied to fine-tuning tasks that require complex reasoning, such as solving math problems or answering scientific questions?\n+ The derivation for the holdout loss approximation in Equation 3 relies on a conditional independence assumption. This seems to imply that adding a single example (x, y) does not significantly alter the model's posterior. How valid is this assumption in practice, especially since adding more examples to the context is known to change the model's output?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ A major advantage of the ICA method is its ability to approximate the holdout loss (similar to RHO-Loss) without the significant overhead of training a separate reference model on the holdout set. This makes it a more practical and scalable solution.\n\n+ The method demonstrates consistently strong results across various models, datasets, and alignment tasks (SFT, DPO, SimPO). Its performance is shown to be comparable to the more resource-intensive RHO-Loss baseline, validating its effectiveness.", "weaknesses": "- The framework's success is dependent on having a high-quality, demonstration holdout set to act as a proxy for the desired test distribution. The performance could degrade if the holdout set is noisy, small, or unrepresentative of the target domain.\n\n- The experiments focus primarily on instruction-following and preference alignment tasks. The paper does not evaluate the method on more complex reasoning domains like mathematics or science, where data quality and logical consistency are critical.\n\n- The method introduces new hyperparameters that may require careful tuning. The paper shows that k=3 (the number of in-context examples) works well for their experiments. However, the optimal k could be highly dependent on the model's context window size, the complexity of the task, and the nature of the domain. This adds an extra layer of tuning complexity for practitioners looking to adopt the method.\n\n- The entire method is built on the assumption that a model's behavior when conditioned on a few examples in-context (ICL) is a faithful approximation of how its parameters would change after a gradient-based update on those same examples (fine-tuning). However, for large SFT training, I am not sure if ICL can mimic the benefit from SFT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RPZP9ojHAq", "reviewer": "Reviewer_zAys", "summary": "The paper proposes In‑Context Approximation (ICA) to estimate, per training example, how much the holdout loss would decrease if the example were used for training.  It defines an ICA score as the difference between unconditional and conditional losses and uses these scores to reweight per-example gradients. The method targets SFT, DPO, and SimPO and is evaluated on LLaMA-3 (3B/8B) and Qwen-3 (4B/8B) across Alpaca/Alpaca‑cleaned, Yahoo Answers Topics (domain shift), UltraFeedback‑binarized, and SHP‑2. Results are reported as GPT‑4o judged win rates of  vs. standard training (no reweighting),  RHO‑Loss, and One‑Shot Learning.", "questions": "1. Holdout leakage: Why are holdout examples included in the training set (Appendix B.1, Yahoo/SHP‑2/UltraFeedback)? Please re‑run the core experiments with strictly disjoint train/holdout/test splits and report results, especially for Yahoo‑Sports and UltraFeedback. \n\n2. Theory gap: Can you provide the promised derivation (or at least a formal sketch with assumptions) showing the local‑linearization equivalence between ICA and a first‑order update toward the holdout optimum? What conditions on model smoothness / context length / \nk are required? \n\n3. Variance and significance: Please run ≥3 training seeds for at least one representative setting per paradigm (SFT/DPO/SimPO) and report mean ± std and binomial confidence intervals for win rates. Also include absolute scores on standard benchmarks where available. \n\n4. Dynamic vs. static: Since the default is R=1, how much do results change for R>1 under a fixed compute budget (i.e., reducing training steps to keep total GPU hours constant)? \n\n5. Prompt sensitivity: How sensitive is ICA to demonstration formatting and ordering? Can you include a robustness table varying the in‑context template / example order?\n \n## Suggestion:\n\n- Ablate proper splits: Re‑report Tables 1–3 with strictly disjoint holdout (for scoring) and training datasets, and add a variant where you train on the holdout alone to show that ICA’s gains are not due simply to including curated data. \n\n- Absolute metrics: In addition to judge‑based win rates, include task‑specific metrics or reward m- odel scores to contextualize effect sizes. \n\n- Compute‑fair baseline suite: Add at least one influence‑based (or Grad‑Match / importance‑resampling) baseline under matched compute, even on a reduced scale, to strengthen comparative claims. \n\n- Report overhead vs. R: Extend Table 10 to show overhead for R=3/5/9 to back the claim of efficiency when more frequent updates are used (Table 7).  Also see https://openreview.net/pdf?id=p0KTYl2B9T for overhead scaling.\n\n- Clarify claims: Either include the local‑linearization derivation or soften language from “equivalent” to “motivated by,” and discuss limitations (e.g., prompt length constraints and ICA’s off‑policy nature mentioned in Sec. 6)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Simple, general mechanism: Framing per-example value via an in‑context estimate of “training on the holdout set” is elegant and easy to drop into standard training loops (Eq. 6–9; Alg. 1). \n\n- Broad applicability: Demonstrated on SFT, DPO, SimPO and across two model families/sizes. Tables 1–3 show “Ours vs. w/o reweighting” win rates generally >60% and often much higher (e.g., LLaMA‑8B SFT Yahoo: 85.10%), supporting cross‑paradigm usefulness. \n\n- Low overhead claim with ablations: The method adds ~1.5% runtime (Table 10), and ablations on k (few in‑context demos), R (score recomputations), weighting vs. filtering, and embedding choice are included (Sec. 5.3; Tables 6–9), which helps practitioners calibrate cost/performance trade‑offs. \n\n- Clear exposition of pipeline: Pseudocode (Alg. 1; Alg. 2) and prompts (Table 12) make the approach easy to reimplement; training/eval config is summarized (Table 5)", "weaknesses": "1. Apparent holdout leakage in multiple setups: In Yahoo Answers Topics, the paper states: “The holdout and test sets each contain 3,000 Sports examples. The training set consists of 1,000 examples from each of the remaining domains, together with the holdout examples, totaling 12,000 training examples.” (Appendix B.1, emphasis added). This means the holdout set is inside the training set, which undermines the notion of “holdout loss,” confounds the ICA scoring (which conditions on $D_{ho}$), and risks overstating gains. A similar pattern occurs for SHP‑2 and UltraFeedback‑binarized (“From the training set, a holdout set … is selected… All examples in the training set are used for training”). These choices likely cause data leakage between “scorer” (holdout) and “trainer,” conflating reweighting with simply learning from the curated subset directly. \nThis affects both internal validity (does ICA help when the holdout is truly held out?) and comparability to RHO‑Loss/One‑Shot under proper splits.\n\n2. Evaluation design relies on single‑seed training and model‑judge only: The paper explicitly trains and evaluates each model only once due to compute constraints (Sec. 5.1). Variability is assessed only by repeating GPT‑based evaluation (Table 4), not by rerunning training. This leaves uncertainty about training variance and statistical significance of reported win‑rates. Moreover, the main metric is pairwise win rate judged by GPT‑4o (temperature 0, seed None; Table 5/“Evaluation Parameters”), with no human evaluation or confidence intervals. \n\n\n3. Theoretical grounding for ICA is informal The abstract claims: “Under a local linearization, ICA is equivalent to a first‑order update toward the holdout optimum” (p. 1), but the paper does not present a derivation or bounds for this equivalence. Appendix A reproduces the RHO‑Loss Bayesian derivation but does not analyze the error of the ICA approximation or its dependence on 𝑘, context formatting, or model scale. This weakens the “theoretically grounded” claim. \n\n4. Comparisons and reporting: Absolute performance is not reported; only win rates vs. baselines. Without absolute metrics (e.g., accuracy, reward, BLEU, or task‑specific scores), it is hard to gauge practical effect size. Baselines exclude several strong data‑selection methods (e.g., influence‑function variants, Grad‑Match, importance resampling) beyond citations. Given the paper’s emphasis on efficiency, a compute‑matched subset of these would strengthen claims. \n\n5. Dynamic reweighting is emphasized, yet the default setting computes scores once at t=0 (R=1; Sec. 5.1), which effectively turns the method into a static ranking for most experiments; the ablation (increasing R) shows only modest gains (Table 7).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XHl0mu8bRU", "reviewer": "Reviewer_t4k1", "summary": "This paper proposes a new framework for data selection during the fine-tuning of LLMs. The core problem addressed is the negative impact of noisy or off-target data often present in large fine-tuning datasets. The proposed solution aims to identify and prioritize high-value training examples based on their potential to improve the model's performance on a small, high-quality holdout set. The key innovation is the In-Context Approximation (ICA) method, which leverages the LLM's in-context learning (ICL) ability to efficiently estimate the holdout loss change associated with training on a candidate example, without requiring expensive retraining or auxiliary models. These ICA scores are then used to dynamically reweight gradient updates during fine-tuning across various paradigms like SFT, DPO, and SimPO. The authors present experimental results suggesting that this method consistently improves model alignment compared to standard training.", "questions": "1. The ablation study shows that using k=3 for ICL examples performs best, with performance degrading for k=5 and k=10. This seems counter-intuitive, as ICL performance often benefits from more examples. Could the authors provide some intuition or analysis for this observation?\n\n2. How sensitive is the method to the choice of the k holdout examples selected via kNN? Does the variance introduced by different kNN selections significantly affect the stability of the ICA scores and the final performance?\n\n3. There seems to be an inconsistency in model naming: the text mentions using LLaMA-3-3B-Instruct and Qwen-3-4B , but Table 3 refers to \"Qwen 3B\". Please clarify if a 3B Qwen model was used or if this is a typo for 4B.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The core idea of using ICL to approximate the impact of a training sample on holdout loss (the ICA score) is innovative and computationally efficient. It cleverly avoids the high costs associated with traditional data valuation methods that require retraining or reference models, potentially making holdout-loss-based selection more practical.\n\n\n2. The method is motivated by theoretical considerations, building upon the holdout loss approximation framework established in prior work (e.g., RHO-Loss). Linking the ICA score to a first-order update towards the holdout optimum provides a principled justification for its use.\n\n\n\n\n3. The framework demonstrates versatility by showing positive results across multiple fine-tuning techniques (SFT, DPO, SimPO) , different model families and sizes (LLaMA-3, Qwen-3; 3B/8B) , and varied data selection scenarios (quality improvement, domain relevance).", "weaknesses": "1. The theoretical justification for the core In-Context Approximation (ICA) itself appears weak. While the goal (approximating holdout loss impact) is theoretically grounded, the paper does not provide a rigorous analysis or proof showing why or how accurately a single forward pass with k-shot ICL (Eq. 6) approximates the complex model state change ($\\theta^{*}(\\mathcal{D}_{t}\\cup\\mathcal{D}_{ho})$ in Eq. 4) that involves actual gradient updates. The reliance on the general observation that ICL performs \"implicit fine-tuning\" feels insufficient to bridge this gap, weakening the claim of a fully \"theoretically grounded\" method.\n\n2. The empirical evaluation relies heavily on a potentially unreliable and limited metric, hindering convincing validation. The primary evaluation metric is the \"win rate\" determined by GPT-4o judging pairwise comparisons. This methodology is susceptible to the biases and limitations of the judge model, costly, hard to reproduce, and lacks granularity. The absence of standard, objective task-specific metrics (e.g., accuracy on downstream tasks, perplexity, benchmark scores like MT-Bench/AlpacaEval) makes it difficult to assess the actual impact on model capabilities beyond alignment style as perceived by one specific LLM.\n\n3. The experimental scope feels somewhat narrow, potentially limiting the generalizability of the findings. While multiple models and methods are tested, the experiments primarily use only two main dataset pairs (Alpaca/Alpaca-cleaned for quality, Yahoo/SHP for domain). Validation on a wider range of dataset types, sizes, and noise levels would be necessary to confirm the robustness and broad applicability claimed. Furthermore, the comparison is limited to RHO-Loss and One-Shot learning; comparing against simpler heuristics (e.g., loss-based filtering, perplexity) or more recent data selection methods could provide better context.\n\n4. This paper would be strengthened by a comparison of this method to other recent frameworks for data selection and robust optimization during fine-tuning [1, 2, 3].\n\n[1] Provably Robust DPO: Aligning Language Models with Noisy Feedback. ICML 2024.\n\n[2] ROPO: Robust Preference Optimization for Large Language Models. ICML 2025.\n\n[3] LESS: Selecting influential data for targeted instruction tuning. ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MJEQt7kphM", "reviewer": "Reviewer_HKFV", "summary": "This paper presents a framework to select and re-weight training examples during LLM fine-tuning based on In Context Approximation (ICA) of holdout loss. The ICA score measures (by proxy) a training example's contribution to minimizing the holdout loss for one step. The authors improve the algorithm by sampling the top k most similar holdout examples (to a training example) and updating scores at intervals. Their results show improvement in win rate (alignment performance) across SFT, DPO and SimPO fine-tuning settings.", "questions": "1. While the paper focuses on selecting samples for alignment, it would be interesting if the method affects overall LLM performance.\n2. Did you consider comparison with other domain adaptation / alignment baselines?\n3. This paper will be strengthened with a re-weighting sensitivity analysis to determine over multiple runs (at least a couple runs in case of compute restrictions) that the same set of examples are highly valued (generally measured by the standard deviation of data value across runs).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This algorithm uses in-context learning as a data re-weighting tool to improve alignment-based finetuning - which I find very interesting and seems practically efficient (only ~1.5% overhead reported). \n2. Experiments with SFT, DPO and SimPO suggest generalizability across different paradigms. Their LLM-as-a-judge metric shows a clear empirical signal to support the utility of the in-context approximation. Particularly liked the experiment in Figure 1 -  the analysis furthers support for their work and has far-reaching implications in domain adaptation. \n3. Smart optimizations backed by ablation experiments further the case for in-context approximation to select high-valued samples for LLM alignment.", "weaknesses": "1. This paper needs to be compared with other off-policy re-weighting baselines like GREATS [1]. \n2. Reweighting stability across different runs is unclear. Another ablation experiment that could support the paper would be if this method aids in faster convergence. \n\n\n[1] GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)