Task-Aware Data Selection via Proxy-Label Enhanced Distribution Matching for LLM Finetuning
Authors: Hao Cheng
Rui Zhang
Ling Li
Na Di
Jiaheng Wei
Zhaowei Zhu
Bo Han
Venue: ICLR 2026 Poster (2026)
Keywords: Data Selection; Task-specific model fine-tuning
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a proxy-label enhanced distribution matching method for task-specific model finetuning which shows superior performance in multiple benchmarks.

Abstract: Task-specific fine-tuning of foundation models is critically dependent on the quality and relevance of the instruction data. While prevailing data selection methods rely exclusively on instruction instances X to approximate the target distribution, we argue that selection should align with the joint distribution of instructions and task-specific labels (X,Y). However, task-specific labels Y are typically unavailable in practice. To address this, we reformulate the task-specific data selection problem and present a novel pipeline that leverages the reasoning capabilities of large language models (LLMs) to infer proxy labels, thereby facilitating joint distribution alignment. Our approach begins by propagating proxy labels from a small target set to a large, unlabeled source corpus. A two-stage filtering process then removes instances with label noise and refines the subset through distribution alignment. This strategy produces more semantically meaningful and task-aware selections than conventional similarity measures based on $X$ alone. Experimental results show that fine-tuning on a subset of only 10K samples, selected from a pool of 300K, achieves performance competitive or superior to state-of-the-art methods. Code is available at https://github.com/tmlr-group/TADS.

Published as a conference paper at ICLR 2026
TASK-AWARD DATA SELECTION VIA PROXY-LABEL ENHANCED DISTRIBUTION MATCHING FOR LLM FINETUNING
Hao Cheng* Rui Zhang Ling Li", Na JiahJinnngg Wei2.3, Zhaowei Zhu", Bo Han TMLR Group, Department of Computer Science, Hong Kong Baptist University The Hong Kong University of Science and Technology (Guangzhou) DD Data (haocheng, bhanm]] } @comp. hkbu. edu. hk, moeyco233@gma com, (11i297, nd1595 connect. hkust-gz, edu. cn, jiahengwei @hkust gz. edu. cn, zzw@d5data.ai
ABSTRACT
Task-specific fine-tuning of foundation models is critically dependent on the quality and relevance of the instruction data. While prevailing data selection methods rely exclusively on instruction instances X to approximate the target distribution, we argue that selection should align with the joint distribution of instructions and task- specific labels (X, Y). However, task-specific labels Y are typically unavailable in practice. To address this, we reformulate the task-specific data selection problem and present novel pipeline that leverages the reasoning capabilities of large language models (LL.Ms) to infer proxy labels, thereby facilitating joint distribution alignment. Our approach begins by propagating proxy labels from small target set to large, unlabeled source corpus. A two-stage filtering process then removes instances with label noise and refines the subset through distribution alignment. This strategy produces more semantically meaningful and task-aware selections than conventional similarity measures based on alone. Experimental results show that fine-tuning a subset of only 10K samples, selected from pool of 300K, achieves performance competitive or superior to state-of-the-art methods. Code is available at https: //op:::///it gigghihthh.... com tttr:/mlllrrreeeeeem wttp:///moo/goo///I...-----
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing tasks, owing to their extensive pretraining diverse corpora. (Touvron) et al, 2023; Acciia et al., 2023 Guo et al., (025). To adapt these models to specific downstream applications, fine-tuning has become standard practice. However, the success of fine-tuning critically depends on the quality and relevance of the instruction data, as low-quality or irrelevant data can lead to degraded generalization (Chen al. 2023 Bukharin & Zhao 2023). Therefore, selecting high-quality, mask-appropriat instruction data is essential for effective model adaptation.
In the field of data selection for LLMs, two primary settings have emerged. The first is task- unspecific data selection (Liu et al.) 2023 Chen et al. (023). which is motivated by the observation datasets often contain low-quality or false instructions, which can degrade model performance if used directly in fine-tuning. Moreover, fine-tuning on very large datasets is computationally expensive. Hence, efficient data selection strategies are essential. In contrast, task-specific data selection (Xia al. 2024 Liu et al. 2024) leverages small task (target) dataset to retrieve the most relevant instruction samples from a source pool, aiming at maximizing model performance particular downstream tasks.
that instruction-tuniin
Compared to general unspeenseci data selection, task-specific data selection has drawn growing interest, particularly as practitioners seek to specialize LLMs for specific tasks where limited task
"Equal contribution.
Corresponnence Bo Han (bhanml Saaiamm@@coom.hhhu...c

---

Published as a conference paper at ICLR 2026
(target) examples available. Most approaches in this area rely exclusively on aligning input features (X), typically by measuring embedding similarity or gradient similarity between source and target samples (Xie et al.) 2023 Xia al. 2024 Liu et al. 2024). However, such X -olly alignment strategies suffer from inherent limitations. For example, consider a target set from the legal domain containing "marital disputes" and "construction contracts." " A source sample related to "cell phone contract" might be selected based on embedding similarity due to semantic overlap with "construction contract." " Similarly, if gradient similarity is used-where the model assigns high influence to shared concepts like "contract" the same irrelevant sample may still be matched. Nevertheless, its actual domain label ("telecom services") does not align with the target label distribution. This discrepancy underscores that input similarity alone is insufficient to ensure task relevance.
To address this, we propose leveraging the joint distribution P(X, Y) rather than marginal distribution P(X) for alignment. Specifically, we harness the reasoning capability of LLMS to infer proxy-labed (Y) for the target task, enabling more semantically meaningful and task-aware selection process. Nevertheless, aligning two random variables is more challenging than aligning only one, and the use of proxy-labels inevitably introduces label noise. In this work, we reformulate the task-specific data selection problem and introduce novel framework that effectively tackles these issues.
The idea of incorporating auxiliary labels/taxonomy to improve task performance is evident in LLM research. For example, instruction finetuming with the taxonomy of human knowledge has been shown to enhance model generalization (Li et al. 2024). Similarly, in mathematical reasoning, generating intermediate steps (as pseudo-labels) has proven essential for learning complex problem-solving skills (Didolkar et al. 2024). These successes motivate us to leverage proxy-labels to enhance distribution matching for taek-spec data selection. Our contributions are threefold:
We reformulate task-specific data selection joint distribution alignment problem and propose a framework that uses LLM-generated proxy-labels to enable alignment.
We design two-stage filtering process that first removes -ii-o---ditribution instances prone to open-set label noise, and then refines the subset via distribution alignment.
Experiments show that fine-tuning on a subset of only 10K samples selected from 300K in- struction instances achieves competitive performance compared to state-of-the-art methods.
2
RELATED WORKS
Task- Tasp-unseeccfic Data Selection for Instruction Tuning. Existing approaches on task-unspecific data selection can be broadly categorized into two types: LLM-ffee free and free approaches (Cao et al. 2023 He et al. (024)) typically leverage handcrafted heuristics or proxy metrics-suce as K-NN embedding distances in a feature space, sentence or token-level length or Shapley values to approximate the utility or difficulty of individual samples without invoking large models. In contrast, LLM-based strategies (Chen et al. 20233] Liu et al.) 2023 Pang et al. (024)) treat powerful pre-trained language models as automated evaluators to score and filter candidate instruction. Notably, (Lu et al., 2023) propose InsTag, which uses LLM to generate pseudo- labels and refines them for sample selection based on tag statistics. However, InsTag operates in a task- unskeensecc manner and lacks alignment with a target task distribution. In comparison, our framework is more comprehensive and explicitly incorporates task-aware alignment.
methods. LLM-
Task-Specific Data Selection for Instruction Tuning. Task-Specific Data Selection for Instruction Tuning aims to identify and retrieve the most relevant samples from a large, general-purpos corpus to improve performance on a specific target task, given only small subset from that target domain. Some methods rely on feature-space similarity: (Yao et al. 2022p select data by ranking candidates based on their similarity to the target set, while (Xie et al.
2023) estimate importance weights in a reduced feature space for resampling. Other methods leverage gradient information: (Xia et al. 2024) score candidates by their maximum gradient similarity to any example in the target set. A more recent approach by (Liu et al. 2024) formulates the problem through the lens of optimal transport, seeking a minimal-cost mapping to align the distributions of the candidate pool and the target set. This method further incorporates diversity-awand regularization to mitigate sample redundancy. However, a common limitation across these methods is that they perform alignment or ranking using only input features, without incorporating the target labels from the target set. This omission may fail to
2

---

Published as a conference paper at ICLR 2026
fully capture task- task-specific relevance, as the semantic alignment between candidate's 's output and the desired target outputs remains unverified.
Domain Adaptation and Learning with Noisy Labels. Our approach is further motivated by principles from Domain Adaptation (Wang & Deng 2018: Zhang et al., 2013) and Learning with Noisy Labels (Natarajan et al. 2013 Song al. (022). Specifically, the process of propagating proxy-labels from the target set to the source pool inevitably introduces label noise, a challenge central to learning with noisy labels. Furthermore, even after filtering this noise, domain shift may persist between the refined source dataset and the target domain, which is core problem addressed by domain adaptation. Our pipeline is designed tackle these challenges: we explicitly incorporate mechanisms to identify and filter OOD samples with noisy labels, then align the distributions between the source and target domains to ensure task relevance.
Our Position. Our work challenges the prevailing paradigm in task-specific data selection for instruction tuning (e.g., LESS (Xia et al. 2024), TSDS (Liu et al., 2024)), which primarily focuses aligning marginal input distributions. We argue that true task relevance is defined by the joint distribution of data and labels. By reformulating the selection criteria to incorporate target labels, we unlock class of solutions for this problem. The pipeline we introduce is concrete instantiation of this principle, designed to select data that are not just superficially similar, but semantically congruent with the target task.
PROBLEM FORMULATION
Define the following concepts:
Let T denote a target dataset, where each instance is sampled from joint distribution Piarge X, Y2). Here, X represents text instruction instance (typically comprising an instruction, an input, and a response), and Y, denotes the corresponding domain-speciff labels (e.g.. for the legal domain, V/ [family law, contract law... }.
Target Dataset:
Source Dataset: Let D U D& represent multi-source dataset comprising K source domains. Each D is sampled from distribution (X, Y1). We assume that the support of the target distribution is contained within the union of the supports of the source distributions, that is,
K
supp Paarget)
S U
supp(Pk),
k=1
which implies that the source domains are relevant to target domains.
The goal is to select a fixed-size subset S C D where |S| N such that such that the empirical distribution Ps approximates Piarget as closely as possible. The text instances from will then be used to fine-tune large language model (LLM) with the aim of maximizing performance the target task.
Objective:
Challenges: The true Labels Y are unobservable in both T and D.
4 METHODOLOGY
This section is organized as follows. Section 4.1 describes the use of an LLM for target dataset annotation. Section 4.2 covers label clustering and propagation the source data. Section 4.3 details the filtering of samples with open-set label noise. Section 4.4 outlines our sampling approach to mitigate domain shift, and Section 4.5 provides an inoematiiontheoreede interpretation. An overview of the pipeline is presented in Figure]
In general, T may comprise instances from multiple domains Isobe et al. represent the set of tags across all constituent domains.
2021 in which case Y, would
"This assumption aligns with standard domain adaptation theory (Ben-David et al. 2010) and the common premise of data selection for instruction tuning, where a large source corpus is available. However, our pipeline also applicable when the source and target domains only partially overlap, such as when source data is limited. See Appendix C for details.
3

---

Published as a conference paper at ICLR 2026
Step1
Step2
Target Set
LLM + Annotation
Target
Label Clustering
Propagating Anchors to Source
TASK
Stepp3
Scoring Samples Based on Anchors
TOPIC
Prompt
Filtering OOD Samples
Mitigating Domain Shift via Incremental Sampling
AUDIENCE
Step4
SCORE
Task-Oriented
Distribution Matching
Figure I: An overview of our 4-step data selection pipeline. We start by using prompt-based LLM annotation generate domain-specifie labels, which are then clustered and propagated source dataset. Next, an LLM- -LM--enerated quality score filters out OOD samples with open-set label noise. Finally, incremental sampling mitigates domain shifts between the source and target datasets.
4.1 GENERATING PROXY-LABELS BY LLM
We design structured proxy-label space comprising four key fields: Task, Topic, Style, and Audience. These dimensions are designed to capture distinct yet complementary aspects of each instruction instance. Specifically, Task refers to the core functional intent, Topic reflects the subject, Style describes the rhetorical manner, and Audience indicates the intended user group. Task is modeled as single-labe label field since most instructions have a dominant purpose, whereas Topic, Style, and Audience are multi-label fields to account for multifaceted instruction characteristics. We then prompt pre-trained LLM with structured summarization template (see Appendix B) to annotate each instruction with tags in four semantic fields-Taske Task, Topic, Style, and Audience.
This semantic decomposition results in modular and interpretable representations that enable struc- tured downstream processing, including clustering, aligmment, and semantic similarity estimation. The of single-label format for Task provides well-defined functional anchor for each instance, while multi-label modeling for Topic, Style, and Audience captures the multifaceted nature of in- structional content. Free-form, descriptive phrase tags allow the representation space to remain open-ended and continuous, avoiding constraints imposed by fixed label taxonomies and facilitating compatibility with embedding- semantic matching.
is worth noting that our defined four general label domains (Task, Topic, Style, Audience) are used demonstrate our framework's flexibility. practice, general label domains are applied when target knowledge is scarce, for instance, when only set of target samples is available without clear target details. Conversely, when prior target knowledge exists, more specific label domains can be adopted. For example, using domain-speciff label such as "mathematical problem-solving skills" for math reasoning tasks has been shown to improve exemplar quality (Didolkar et al., (2024), indicating that more precise domain definitions generally lead to better data selection. In this paper, use general label domains experiments to emphasize our pipeline's 's efficiency.
4.2 PROXY-LABEL CLUSTERING AND PROPAGATION
To bridge the semantic gap between source domains and our L.L. --M--eerrteddd labels, we employ a robust methodology centered on clustering and semantic propagation. First, encode cach label into a dense vector space using sentence-level embedding model, processing each field (e.g., Topic, Style) independently to preserve their unique semantic characteristics.
Proxy Label clustering We apply k-means clustering (with = 100) to these embeddings, which the robustness to k). This process is fundamental because it accomplishes two key objectives: it abstracts away the significant lexical variation inherent in I.L--iiiii phrases and condenses their broad diversity
effectively groups semantically similar tags into distinct clusters (Figure (Figure/demooosssttttt demonnstrats
4

---

Published as a conference paper at ICLR 2026
into compact set of semantic centroids. These centroids function semantic anchors, providing stable and noise-reduced representation the entire target domain's conceptual landscape.
Propagation We project samples from the source domains into the same embedding space and compute their cosine similarity against all semantic anchors. This allows for soft, interpretable matching process. We then assign the most relevant anchors-seled top-3 for Topic, Style, and Audience, to each source sample. This selective assignment is not a hard label transfer but test-guided semantic projection, enabling a nuanced and measurable integration of the target domain's is characteristics into the source.
the top-1 for Task and the
4.3 FILTERING OOD SAMPLES WITH LABEL NOISE
Topic Distribution Comparison
Distribution Comparison
Percentage (N)
Percontags
These figures visualize the results of our incremental sampling method. The left and right panels show the matching results for the Topic and Style, respectively. The anchors ranked from highest to lowest count of target set (left to right) for clarity, and the top 15 anchors are shown to enhance readability.
Figure 2:
The propagation of target domain anchors to source domains inevitably introduces open-set label noise where irrelevant instructions are assigned to an target label. To ensure data quality, filtering these mismatched samples becomes essential. learning with noisy labels, common detection strategy leverages the memorization effect of deep neural networks (DNNs), wherein DNNs learn clean patterns before memorizing noisy ones (Arpit et al. 2017). By training DNN for a few epochs and ranking samples by loss values, high-loss instances typically identified as noisy
approach
effective for both closed-set and open-set noise (Jiang et al. 2017 Han et al. 2018 Li et al. 2020
Sachdeva al. 202I)
However, this method presents limitations in our multi-field setting: training separate DNNs is computationally expensive and lacks generalization capability. Each new task or label space change would necessitate retraining, creating scalability bottlenecks. To overcome these constraints, we leverage the robust reasoning capabilities of large language models (LLMs). Specifically, employ LLMs to score each sample based on its semantic alignment with assigned anchors, evaluating the quality and relevance of the assigned anchors (see Appendix B for prompt details). Samples falling below predefined threshold (c.g., score < 6) are identified as out-of- out-of-distributtin (OOD) indicating open-set label noise and subsequently removed from the source corpus (Appendix C has further explanations).
4.4 INCREMENTAL SAMPLING FOR MITIGATING DOMAIN SHIFTS
In practice, after removing OOD samples, a simple approach randomly sample a fixed number of instances (c.g., 10,000) from the source domain for fine-tuning. However, after OOD filtering, a distribution shift may persist between the remaining source data and the target set. This section presents method to mitigate such domain shifts more effectively.
Leading works in domain adaptation (Zhang et al. 2013 Zhao et al. 2019) theoretically and empirically demonstrate that target shift (or prior probability shift) is common and critical type of distribution discrepancy in real-word world scenarios and suggest that aligning for target shift is often more advantageous for robustness and effectiveness than aligning for covariate shift. Consequently,
5

---

Published as a conference paper at ICLR 2026
pipeline prioritizes mitigating target shift between the source and target domains. The implicit assumption of target shift that the conditional distribution P(X remains stable across the target and cleaned source domains, while the label distribution P(Y) varies. Note that the label space here corresponds to the anchors (clusters) defined in Section 4.2 not the labels/tags generated by the LLM in Section 4.1
Sampling Objective: Let (Y( be the empirical anchor distribution the target dataset and P(Y) be the empirical anchor distribution of the sampled source dataset. To mitigate label (anchor) shift, our objective is to minimize ||P* (X) P(Y)
Given a sampling budget N, we propose an incremental sampling method outlined in Algorithm [ At each step, we compute the label gap and select training instance associated with the label with the largest positive gap (Lines 4-5, Algorithm [T. Because each instance contains fixed number of labels per field (e.g., one for Task, three for Topic), all candidates contribute equally to the label counts. The process iterates until the budget is exhausted no viable candidates remain, producing subset whose empirical distribution P(Y) closely approximates the target distribution PY()..
Algorithm is applied independently to each field (e.g., Task or Topic). For each selected sample, we update the counts for all associated labels. This introduces natural balancing effect across the label space, accelerating convergence toward P (Y), especially in dense semantic fields where tags frequently co-occur. F produces a source subset whose anchor distribution aligns with the target distribution. Although perfect matching is rarely achievable due to limited sample sizes for certain classes in the source dataset and the sampling budget, our incremental sampling method effectively reduces the distribution gap between the source and target domains. key implication of this method its flexibility, Since alignment is performed per field, we can choose to align by Task, Topic, or other fields based on the specific requirements of the downstream application.
the matching results, demonstrating that incremental sampling
Algorithm 1 Distribution Matching via Incremental Sampling
Require: source pool D, target set T, sampling budget N
1: Compute target label distribution p* (Y) from T
2: Initialize empirical distribution P(Y) + and selection S
3: while [S| < N do
4: Compute label-wise gap: g(v) P* (y) P(y)
5: Identify label y = arg max,, (v) such that unused candidates with tag y exist
6: if no valid y" found then
7: break
8: end if
9: Select candidate x" with tag y²
10: S S {x"); update P(Y)) using labels from x²
end while
12:
return Sample set S
4.5 EPORMATION-THHORRTT EXPLANATION
We explain the benefits of incorporating label information in the dataset selection process from an thomattin-thhoreticc perspective. Let T be random variable representing the target domain, which charactirized by Parget(X) (X, Y,). The outcome of the selection algorithm is a subset S C D. The instances and their associated labels of S are described by the random variables X and Ys.
The mutual information between the task and the selected data can decomposed as:
[T'; (Xs, Ys)) Xs) [(T;; Ys Xxs)
This decomposition reveals two distinct sources of information:
T(T; Xs) quantifies the information about the task T contained in the input features Xs of the selected subset.
[[T; Ys Xs) captures the additional information about T provided by the labels Ys of the subset, given the inputs Xs.

---

Published as a conference paper at ICLR 2026
Conventional task- data selection methods in LLM (Xia et al. 2024; Liu et al. 2024) which rely exclusively on the similarity between imput features (i.e., Xs), primarily aim to maximize the first term, [[T; Xs). However, they inherently ignore the second term, [(;; Ys! Xs). This constitutes a limitation, since instances with semantically similar X may be associated with labels that are irrelevant even contradictory
the target task T. In contrast, joint distribution between the selected data and the target domain, which implicitly maximize the complete objective (T'; (Xs, Ys)).
proposed method matches the
EXPERIMENTS
5.1 EXPERIMENTS SETUP
We utilize three pretrained LLMs across different stages of our pipeline: Qwenn255-BB-Instruet (Yang et al. 2024) is used for annotation and scoring, BGE-M3 (Chen et al., 2024) for embedding and clustering, and L...A-331-88 (Grattafiori et al. 2024) for supervised instruction tuning on the
selected subsets using LoRA (Hu et al. 2022).
Table I:
Instruction tuning data pool and evaluation benchmarks
Instruction Tuning Data Pool
Dataset
Flan V2
Open-Assistant 1
WizardLM
Dolly
Stanford Alpaca
Total
Training Size
100K
33K
100K
15K
52K
300K
Evaluation Benchmarks
Benchmark
MMLU
TruthfulQA
GSM8K
BBH
TyDiQA
Total
Test Size
14,042
790 Truthfulness
1,319
6,511
5,077
27,739
Capability
Factuality
Reasoning
Reasoning
Multilinguality
Our data pool comprises two components: source corpus collected from five widely- used instruction tuning datasets Flan V2 (Longpre al. (023), Open-Assistant 1 (Köpf al.
Data Pool
2023). WizardLM (Xu et al. 2023). Dolly [Databricks] 2023), and Stanford Alpaca (Taori et al.
2023
and a set of evaluation benchmarks. Evaluation conducted on five standard alignment bench- marks, each targeting distinct model capability: MMLU (Hendrycks et al.) 2020). TruthfulQA (Lin)
al. 202T), GSM8K (Cobbe et al. 202T) and BBH (Suzgun et al. 2022). and TyDiQA (Clark et al., 2020 Detailed training set statistics and the sizes of evaluation sets are provided in Table [] These benchmarks also serve as our alignment targets: for each dataset, we sample 20% of the test set to form a target set; all target sets are then combined to guide our subset selection. The remaining 80% of each test set is reserved for final evaluation.
Baselines: We compare our method against a set of representative baselines commonly used in instruction tuning and data selection:
Vanilla Base Model: The base model evaluated in zero-shot setting, without any supervised
fine-tuning. This serves as a reference point to measure the impact of instruction tuning.
Full (300K): The entire 300K training is used for instruction tuning.
Completion Length: Following (Zhao et al. 2024), we rank all samples by the total number of tokens in the prompt-response pair. The top 10K longest samples are selected under the assumption that longer responses carry higher information density.
k-NN: Each sample is scored by its average distance to the & nearest neighbors in sentence embedding space (Reimers & Gurevych] (2019). Higher distances indicate semantic uniqueness, and the 10K most isolated samples are selected.
BM25: A retrieval metric which considers term frequency and inverse document frequency.
RDS+ (Ivison et al.) 2025): representation- based retrieval method which uses weighted mean pooling of pretrained L.M hidden states.
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison across alignment benchmarks. All the methods select 10K samples from the source except the method of Full (300K) and Vanilla Base Model. For our align based method, we use min score filter samples in this table, as described in Section 4.3 For each benchmark, top-performing score is shown in bold, while second-best score is underlined. We also record mean and sid for selection methods.
Methods
MMLU
TruthfulQA
GSM
BBH
TyDiQA
Vanilla Base Model
64.3
32.8
51.0
54.8
22.7
Full (300k)
63.6
44.1
49.0
57.9
66.4
Completion Length
63.3p.04
7.71.1
54.31.43
61.0p.65
62.3p.45
k-NN-10
61.9p.37
41.10.70
52.0n.41
61 61.30.49
61.3p.05
Random
63.70.24
29.13.11
54.01.22
60.20.33
60.50.33
BM25
63.2 63.2p.03
25.22.77
51.3u.20
58. 68.80.28
62.0p.20
RDS+
63.8p.12
3.60.04
53.8u.20
59.4g.21
60.3p.08
LESS
63.3p.04
35.1
56.10.61
61.10.54
64.0p.54
TSDS
63.6p.0s
44.1o.69
50.01.22
62.1o.22
63.5p.54
Align_topic
64.5p.12
46.90.29
57.0n.22
60.2 60.20.29
58.6p.6r
Align_styp style
64.2p.08
47.20.12
57.80.61
58.90.29
59.3p. 15
Align_taie task
64.4g.16
46.50.53
57.50.41
58.90.57
59.30.u 90
Align_audienes
64.4p.21
47. 1.211.22
55.81.13
58.50.73
60.21.05
Table 3: Performance comparison for our align-based method with respect to min score.
Methods
MMLU
TruthfulQA
GSM
BBH
TyDiQA
Align-based selection, min score 2 7
Align_topic
64.5
46.9
57.0
60.2
58.6
Align_style
64.2
47.2
57.8
58.9
59.3
Align task
64.4
46.5
57.5
58.9
59.3
Align audience
64.4
47.1
55.8
58.5
60.2
Align-based selection, min score 2 6
Align_topil _opic
65.0
24.7
55.5
61.9
57.0
Align_style
65.1
32.1
55.0
62.5
58.9
Align_tate hign_task_
65.0
38.0
54.0
59.8
58.1
dign_audience
64.9
42.6
55.0
59.7
59.1
Align-based selection, min score 2 5
Align _opicc
65.1
36.2
52.5
61.4
57.6
Align_st style
64.9
31.2
55.0
60.4
59.4
Align_task
64.9
35.6
54.5
60.0
57.7
Align_audiens 65.1
39.2
55.0
60.6
58.0
Random: A simple baseline that randomly selects 10K examples from the data pool.
2024): A gradient- based data selection framework that estimates the influence of training samples using gradient similarity few-shot target examples.
LESS Xia et al.
TSDS (Liu al. 2024): ; A task-specific data selection framework that aligns the source distribution
with a small representative set from the target task. TSDS formulates data selection as an optimal transport problem with diversity-promoting promoting regularizer based kemnel density estimation. We use the target set to estimate the task distribution and select 10K training samples accordingly, ensuring parity with proposed method.
8

---

Published as a conference paper at ICLR 2026
Table 4: Performance comparison across alignment benchmarks with 1K/5K selection size.
Methods
MMLU
TruthfulQA
GSM
BBH
TyDiQA
Random (1K)
64.0
3.8
49.0
54.2
56.4
Align style (IK)
64.2
6.8
50.5
57.7
54.5
Random (5K)
63.7
3.5
52.0
59.4
58.7
Align Align_style (5K)
63.9
32.4
52.0
61.0
59.7
5.2 OPENLLM LEADERBOARD EVALUATION RESULTS
Following the Lp Leaderboard evaluation protocol, we adopt exact match (EM) as the scoring metric for MMLU, TruthfulQA, GSM8K, and BBH. For TyDiQA, we use the 1-shot F1 score as reported by the leaderboard.
As shown in Table 2 and Table 3 our method achieves competitive superior performance compared to SOTA methods (LESS, TSDS) on task-specific data selections for the MMLU, TruthfulQA, GSM8K and BBH benchmarks. However, our approach underperforms on TyDiQA. We hypothesize that this is because TyDiQA is multilingual dataset, and the label fields we designed may not adequately capture its inherent characteristics. Consequently, samples related to TyDiQA in the source pool were likely filtered out by our selection algorithm.
Results Analyses:
From Table the min score threshold also has impacts on performance. Analogous to the problem of learning with noisy labels (Jiang et al. 2017; Han et al. 2018), higher threshold risks retaining too few clean (relevant) samples, while lower threshold risks involving too many irrelevant samples.
It is important to note that even though intention is not to suggest empirically testing all label domains for each new task. The primary purpose of presenting four domains is to illustrate the inherent flexibility of our pipeline. The results in Table 2 demonstrate that selecting any of these domains yields strong performance on MMLU, TruthfulQA, and GSM benchmarks compared baseline methods. For tasks with more specific information, a custom domain can also be designed. as discussed in Section [ We further evaluate our pipeline with limited data. As shown in Table 4 our method achieves good performance even with smaller selection sizes (IK and 5K), demonstrating its effectiveness.
four label domains for performing experiments, our
5.3 ABLATION STUDIES
Table 5 presents the results of our ablation study on the contributions of OOD sample filtering and incremental sampling. We compare three configurations:
This baseline applies incremental sampling to the entire 300K dataset after label propagation, without any OOD filtering; OOD Filtering only: This variant filters the dataset by retaining only samples with an I.L.Massigned min score 2 6 followed by random sampling: Complete Method: This is our full pipeline, which first applies OOD filtering (min score 6) and then performs incremental sampling are reported under the bert-performing label field. The result clearly indicates that combining both techniques is most effective and leads to superior performance.
Incremental Sampling only:
the filtered subset. For each benchmark, results
Table Evaluating individual contributions OOD sample filtering and incremental sampling.
Methods
MMLU
TruthfulQA
GSM
BBH
TyDiQA
OOD Filtering
Incremental Sampling
64.3
3.5
55.0
49.1
60.5
64.8
42.1
55.0
61.9
56,4
65.1
42.6
55.5
62.5
59.1
To further verify the necessity of joint alignment on P(X, Y) over input-only alignment on P(X), and to demonstrate the advantage of our design, conduct the following experiments:

---

Published as a conference paper at ICLR 2026
MMLU
TruthfuIQA
GSM
60
60
555
13-
55-
150
59-
Porformamce
500
Performan
15-
40-
10-
40-
33
Style Task
-3-
-B- Style Task
---
Style
Tusk
30
30-
30-
20
100
150
50
150
20
150
Kvalue
Figure 3: Performance of our pipeline with respect the number of clusters i in K-means clustering.
Adding semantic information to existing selection pipeline. After LLM annotation, insert the proxy label information into each instruction and apply the standard RDS+ selection pipeline; Applying pipeline directly to inputs X. We remove the LLM-annotation step entirely. Target instructions are clustered into anchors, source samples are propagated according to anchor- source embedding similarity, and we then perform selection and incremental sampling; Replacing LLM- filtering. Instead of using LLM-assigned scores,
based OOD filtering with embedding-similann we compute cosine similarity between source and anchor embeddings, filter out similialattty samples, and then apply our selection and incremental sampling procedure.
As shown in Table 6 (1) The direct integration of semantic information (RDS+ with semantic info) fails to yield consistent improvements over RDS+, indicating that explicit distribution alignment is necessary to utilize semantic-fiee field information effectively; (2) Our pipeline outperforms direct application to X, confirming that the performance gains stem from joint alignment on P(X, Y) rather than from clustering and sampling alone; (3) Compared similarity- OOD filtering, our method demonstrates that LLM-bbsed judgments provide more reliable filtering.
Table 6: Comparison of alternative designs for our joint alignment.
Methods
MMLU
TruthfulQA
GSM
BBH
TyDiQA
RDS+
63.6
3.5
54.0
59.1
60.2
RDS+ (with semantic information)
63.7
5.4
53.5
60.7
59.7
Applying our pipeline directly X Our pipeline with similarity OOD filtering
63.4
40.8
54.0
59.9
58.2
63.0
28.3
51.0
57.4
61.6
Our full pipeline (align topic, min score:7)
64.6
46.4
57.0
60.0
59.3
We also evaluate the impact of the K-means cluster count on our pipeline's performance in Figure 3. The results show consistent performance for &o between 50 and 150 across datasets and label domains. However, a smaller i can lead higher variance, as seen on TruthfulQA. For robustness and reproducibility, we fix k = 100 in all experiments.
The Appendix includes more experiments and analyses including the robustness of our pipeline to variations in label quality, deeper investigation into the connection between our pipeline and the fields of Domain Adaptation (DA) and Learning with Noisy Labels (LNL), etc.
CONCLUSION
In this paper, we revisit task-aware data selection by formulating as a joint alignment both input features (X) and task-specific labels (Y)) This paradigm shift moves beyond input-only alignment to naturally address real-world challenges like label noise and domain shift. By integrating techniques from noisc-robust learning and domain adaptation. we propose pipeline that effectively mitigates these issues. The primary contribution is not merely the specific pipeline but the reconceptualization of the problem itself. Framing it as joint distribution alignment unlocks broader solution space for future research. Our work serves as an initial instantiation of this perspective.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
HC and BH were supported by NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation Nos. 2024A1515012399, and HKBU CSD Departmental Incentive Scheme.
ETHICS STATEMENT.
Our work focus on task-specific data selection for fine-tuning large language models (LLMs), All datasets and models used are publicly available; therefore, no intellectual property or personal privacy concerns are involved.
REPRODUCIBILITY STATEMENT.
We have released our code at wetpttps: http://gtttuu.com///mmgggorrr///
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08721 2023.
Woosuk Kwon anda Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Devansh Arpit, Stanistaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look memorization in deep networks. In International conference on machine learning, pp. 233-242.
PMLR, 2017.
Shai Davidd John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175, 2010.
Alexander Bukharin and Tuo Zhao. Data diversity matters for robust instruction tuning. arXiv preprint 17X1:2231..14336, 2023.
Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models. arXiv preprint trXiv:2307.062900 2023.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: multi-granularig text embeddings through self-knowledge
Multi-lingual, multi-functionnn distillation, 2024.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, al. Alpagasus: Training better alpaca with fewer data. arXiv preprint arxiv:2007.0870) 2023.
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information-sseking question answering in ty pologically di verse languages. Transactions of the Association for Computational Linguistics, 454-470, 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.

---

Published as a conference paper at ICLR 2026
Databricks.
The world's first truly open instruction- 2000://///00ccom/////////2/////0 hol.:p/wss........mmraaaaaaaaaa-
Introducing
dolly:
tuned
Ilm.
2023.
Accessed: 2024-05-15.
Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Jimenez Rezende, Yoshua Bengio, Michael C Mozer, and Sanjeev Arora. Metacognitive capabilities of Ims:: An exploration in mathematical problem solving. Advances in Neural Information Processing Systems, 37:19783-19812 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788. 2024.
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https: /ttps::///tthbb wwt.://wo../gccceeeeee http:/hhggggngcc.oooooooo htt.g/iinaaccc/c/eeeeeetee accelerate. 2022.
www.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprins arXiv:2501. 2025.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018
Yexiao He, Ziyao Wang, Zheyu Shen, Guoheng Sun, Yucong Dai, Yongkai Wu, Hongyi Wang, and Ang Li. Shed: shapley-based automated dataset refinement for instruction fine-tuning, arXiv preprint arXiv:24005.r .000:22405.00705, 2024.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and distrr-distriuution examples in neural networks. arXiv preprint 1111v:1610.02136, 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICIR, 1(2):3, 2022.
Takashi Isobe, Xu Jia, Shuaijun Chen, Jianzhong He, Yongjie Shi, Jianzhuang Liu, Huchuan Lu, and Shengjin Wang. Multi-target domain adaptation with collaborative consistency learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 8187-8196, 2021.
Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scale data selection for instruction tuning. arXiv preprint arXiv:2503.00 0/800, 2025.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks corrupted labels. arXiv preprint 1711v:171..00005, 2017.
Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant couversations-democratiiz large language model alignment. Advances in Neural Information Processing Systems, 66:47669-47681 2023.
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shao- han Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv: 13064, 2024.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. arXiv preprint arXiv:2002.07394 2020.
12

---

Published as a conference paper at ICLR 2026
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021.
Hong Liu, Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Qiang Yang. Separate to adapt: Open domain adaptation via progressive separation. In Proceedings of the IEEE/CVE conference on computer visionMelacognifive Cupabilities of LLMs: An Exploration in Mathematical Problem Solving and paltern recognition, pp. 2927-2936, 2019.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15662 2023.
Zifan Liu, Amin Karbasi, and Theodoros Rekatsinas. Tsds: Data selection for task-specific model finetuning, arXiv preprint arXiv:2410.11303, 2024.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631-22666 PMLR, 2023.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. arXiv preprini MXXx:2338.00077 2023.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013.
Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In Proceedings of the IEEE international conference on computer vision, pp.
2017.
Jinlong Pang, Jiaheng Wei, Ankit Parag Shah, Zhaowei Zhu, Yaxuan Wang. Chen Qian, Yang Liu, Yujia Bao, and Wei Wei. Improving data efficiency via curating llm-driven rating systems. arXiv preprint Arxi::2410. 10877, 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: /0084, 2019.
Ragav Sachdeva, Filipe Cordeiro, Vasileios Belagiannis, Ian Reid, and Gustavo Carneiro. Evi- dentialmix: Learning with combined open-set and closed-set noisy labels. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3607-3615, 2021.
Hwanjun Song, Minscok Kim, Dongmin Park, Yooju Shin, and Jae- Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE transactions on neural networks and learning systems,
2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint 1rXiv:2210.09261, 2022.
Rohun Taori, Ishaan Gulrajani, Ting Zhang, Yann Dubois, Xiaodan Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An Aktructtoo-followiing llama model. https
//github. 2024-05-15.
2023.
GitHub repository. Accessed:
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey, Neurocomputing, 312: 135-153, 2018
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04433 0433333333,,,000 2024.
13

---

Published as a conference paper at ICLR 2026
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. Advances Neural Information Processing Systems, 36: 34201-34227, 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.122 12244, 2023.
An Yang, Baosong Yang, Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. NIp from scratch without large-scale pretraining: A simple and efficient framework. In International Conference on Machine Learning, pp. 25438-25451, PMLR, 2022.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. International conference on machine learning, pp. 819-827, 827. Pmlr, 2013.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. International conference on machine learning, pp. PMLR, 2019.
Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning, arXiv preprint arXiv:2402.04833, 2024.
14

---

Published as a conference paper at ICLR 2026
APPENDIX ARRANGEMENT
The Appendix is arranged as follows:
Section A
Statement of the Use of Large Language Models (LLMs) in writing.
Section Prompt templates employed in our proposed pipeline.
Section Deeper amalysis the connections between our pipeline and the fields of Domain
Adaptation (DA) and Learning with Noisy Labels (LNL).
Section D Additional experimental results and visualizations.
Section Detailed experimental settings.
A STATEMENT OF THE USE OF LARGE LANGUAGE MODELS
In this paper, DeepSeek was used solely to check grammar and polish writing style. No scientific material or core research content was generated by any Large Language Model (LLM).
B LLM PROMPT TEMPLATE
The following is the prompt template used in the task-specific label generation in Section 4.1
Prompt Template for LLM Annotation
<System Prompt>: You are helpful assistant.
<User Prompt>: [Background Information]
Instruction: [Instruction]
Input: [Input]
Response: [Response]
Please strictly output in JSON format and generate rich, high-quality summaries suitable for L.M.. Summarize the entire conversation **as single summary not per message per role.
Requirements:
1. Task: Describe the single most appropriate task type from the background as a short sentence or detailed multi-phrase descriptor (e.g.. "Answering open-domain factual questions based on user imput"). Style: List the top most relevant styles tones detailed phrases brief sentences (c.g., "Uses formal and professional tone suitable for academic writing").
Topic: List key topics covered the background.
4. Audience:
List the top 3 intended audiences.
Only single JSON object as shown below:
"Task": "<short sentence descriptive phrase>", "Style":
"<csscriptive phrase>", *<esscriptvve phrase>" *<aescriptive phrase>*
},
"Topic": [
*<short sentence>",
*<short sentence>", "<short sentence>"
}, "Audience";
*<aescriptive phrase>",
*<aescriptive phrase>" *cdsscriptvve *
15

---

Published as a conference paper at ICLR 2026
The following is the prompt template used in Section 4.3 to score the source sample based on its assigned anchors.
Prompt Template for Scoring Source Samples
<System Prompt>: You are a helpful assistant.
<User Prompt>: You are expert evaluator. Please evaluate the following sample based on these criteria:
Completeness (1-10): How complete is the response?
Information Richness (1-10): How much useful information does it contain?
Rarity (1-10): How unique rare this type of content?
Complexity (I-10): How complex is the task/content?
Sample to Evaluate: [sample_format] format]
Associated Tags:
Task: [Assigned Task anchor]
Style tags: [Assigned Style anchor]
Topic tags: [Assigned Topic anchor] Audience tags: [Assigned Audience anchor]
Use these tags as reference points for your scoring decisions. Consider ALL the tags in each category when evaluating. Please respond with ONLY JSON object in this exact format:
"Competteness" "<1-10>"
"Information Richness"; "<1-10>",
"Rarity": "<1-10>",
"Complexity": "<1-10>"
"Overall Score": "<1-10>"
Note that each anchor may contain many domain speciffc tags, as the anchors are generated by clustering. Therefore, the anchor description in the prompt template above summarized by LLM,, which extracts the 20 most representative keywords or key phrases for each anchor.
C DEEPER ANALYSES OF OUR PIPELINE
In this section, we provide answers and explanations for the following two questions:
1. Why our approach in Section 4.3 can be regarded proxy for filtering label noise?
Can our pipeline be applied to scenarios where the target and source domains only partially overlap?
For the first question, our approach in Section 4.3 can be viewed as a proxy for filtering label noise because mirrors& established technique in learning with noisy labels. In traditional classification, models tend to be less confident about mislabeled samples compared to clean samples, resulting in low maximum softmax probability (the "confidence score"). This score is a common signal for identifying potential label noise (Hendrycks & Gimpel 2016). Instead of training a model to obtain confidence score, we leverage the reasoning capability of an LLM. The LLM assesses sample's relevance to its most similar anchors and assigns score, which functions as a semantics-awee aware confidence score. Thus, our method provides a direct analog to confidence-based filtering without need for model training.
For the second question, we use Figure 4 to demonstrate our pipeline's applicability to scenarios where the target and source domains only partially overlap.
In Figure (a). the support of the target domain is a subset of the source domain. Our OOD filtering step first produces filtered source dataset with support that approximately matches the target. Subsequent incremental sampling then aligns the distribution of the selected dataset with the target distribution.
16

---

Published as a conference paper at ICLR 2026
COD Filtering
Sampling
(a)
OOD Filtering
Incremental Sampling
MARM
Figure 4: Illustration of how our pipeline aligns the target distribution by sampling from the source. The upper figure (a) shows the case where the support of the target distribution is a subset of the source support. The lower figure (a) shows the case of partial overlap between the target and source distributions.
Table 7: Performance comparison of different data selection approaches for fine-tuning the Mistral- 7B-v0.3 model. Each approach selects 10K samples. For each benchmark, top-performing score is shown in bold, while second-best score is underlined.
Methods
MMLU
TruthfulQA
GSM
BBH
TyDiQA
Vanilla Base Model
59.7
30.0
38.0
49.1
54.4
LESS
59.6
32.0
38.5
51.9
59.2
TSDS
59.4
37.0
41.5
57.4
56.4
Ours (Align-topic)
60.3
35.4
44.0
52.8
53,9
Ours (Align-task)
60.4
36.6
46.0
53.0
54.2
Ours (Align-style)
60.0
34.3
40.5
51.0
55.3
Ours (Align-audience)
60.3
35.1
43.5
51.5
55.1
Figure 4 (b), after OOD filtering, support the filtered source dataset becomes subset of the target domain. Incremental sampling then matches the distribution of the selected data to the distribution the partial target (i.e., the shared portion), confirming that pipeline successfully selects task-relevant samples.
Notably, the scenario (b) corresponds to the open-set Domain Adaptation (DA) setting. Similar to traditional open-set DA methods (Panareda Busto & Gall, 2017 Liu et al., (019). which focus aligning shared classes while rejecting unknown ones, our pipeline inherently handles this case by filtering and aligning only the overlapping regions.
D ADDITIONAL EXPERIMENTAL RESULTS AND EXAMPLES.
D.1 FINETUNING ON OTHER FOUNDATION MODELS
In the main paper, we fine-tune the Llama-3.1 model on our selected data. In this section, we instead fine-tune Misvral-7B-v0.3 demonstrate the generalizability of our selection pipeline. The results in Table 7 demonstrate that our method achieves competitive performance on the Mistral model. Specifically, Align-task achieves top-1 or top-2 performance across all benchmarks except TyDiQA.
D.2 EXAMINING THE LLM ANNOTATION QUALITY
We conduct thorough evaluation the quality of proxy labels generated by the LLM, assessing both precision (whether label correctly describes its corresponding instruction) and consistency (whether the same label is applied to semantically similar instructions). Our examining methodology follows the framework established by Lu et al. 2023
17

---

Published as a conference paper at ICLR 2026
For precision evaluation: We randomly sampled 500 instruction-label pairs. GPT-4 was asked to assess if the label adequately describes the instruction. From these, the first 50 pairs were selected for human evaluation by three independent annotators.
For consistency evaluation: We randomly sampled 500 labels, ensuring each label was associated with at least two distinct instructions. GPT-4 was asked determine the instructions for given label were semantically consistent. Similarly, the first 50 sets of instructions were assessed by three human annotators.
We calculated the agreement between raters using Cohen's 's Kappa (for pairwise agreement between human and GPT-4) and Fleiss' Kappa (for inter-annotator agreement among humans). The results are presented in Table
Table 8: Annotation Quality Comparison
Agreement
Metric
GPT-4 Annotation
Human Annotation
Human- Human Human-GPT
Tag Precision
0.958
0.94
0.4823
0.7899
Tag Consistency
0.856
0.9
0.73
0.736
As shown Table g both GPT-4 and human evaluations indicate high precision and consistency for the proxy labels. The Human-GPT agreement scores (exceeding 0.7) indicate solid alignment between human and LLM judgments.
D.3 LLM SCORE CONSISTENCY
To assess the reliability of L.M--based OOD filtering, evaluate consistency in ways:
Intra-consistency: We instruct Alama-3--BB --8B-Instruct score the same set of source samples twice, using different random seeds. The consistency between the two scoring rounds is measured using Spearman's 's rank correlation (range: -1 to 1).
Inter-consistency: We compare scores generated independently by Llama-3 1aama-3-8Bl 3-8B-1888- In-trust and Qwen-2.5-77 2-5-7B-Instruct on the same samples, again using Spearman's correlation to quantify agreement.
After calculation, the Intra-consistency is 0.72, and the Inter-consistency is 0.65, indicating LLM- based OOD scoring is reasonably stable across repeated runs (intra-con caura-consistency) and aligned between different models (inter-consistency).
D.4 ROBUSTNESS TO LABEL NOISE
While Table g confirms the high quality of our proxy labels, we further evaluate the robustness of our pipeline label noise through controlled experiment. Inspired by techniques in learning with noisy labels, we inject synthetic noise by replacing 20% of the original LLM annotated tags in the MMLU, TruthfulQA, BBH, and TyDiQA datasets with randomly selected tags from the GSM dataset. This allows precise control over the noise level while preserving the label distribution's structure.
We focus evaluation MMLU, BBH, and TyDiQA due to their larger test sets, which provide statistically reliable performance measurements on noisy labels. As shown in Table introducing 20%label noise leads to only marginal performance drops, demonstrating the pipeline's robustness. We attribute this resilience to our anchor-based propagation design: rather than propagating labels directly, we first cluster proxy labels form stable anchors. This clustering step effectively averages out the impact of individual noisy labels, making the propagation process inherently more robust.
D.5 IMPACT OF ANNOTATION MODEL CHOICE
To evaluate the sensitivity of our pipeline to the choice of annotation model, we conduct an experiment where we vary the LLM used for generating proxy labels while holding all other hyper- serameparseeters constant. Specifically, we compare Hlama-3-8B-Instruue and Qwen-2-7B IIInttruuct as annotators, finetuning the same target model (I ([lama3--8B) in both cases.
18

---

Published as a conference paper at ICLR 2026
Table 9: Performance of Selected Data Under Label Noise (Averaged Across Label Domains)
Label noise ratio
MMLU
BBH
TyDiQA
0% label noise
64.3
60.9
59.0
20% label noise
64.1
60.7
58.5
As shown in Table 10 the performance across most datasets (MMLU, TruthfulQA, GSM, and BBH) remains stable (< + points difference) when switching annotators. The sole exception is TyDiQA, where performance drops by 6.6 points with Qwen-2.5 annotations. This suggests that: (1) Our pipeline is largely robust to the annotator model family. (2) Language-specinn biases (e.g.. Qwen-2's Chinese pretraining) may impact performance on certain tasks like TyDiQA (multilingual QA).
Table 10: Performance with Different Annotation Models (Finetumed on Llama-3.. 3.1-BB)
Methods
MMLU
TruthfulQA
GSM
BBH
TyDiQA
Annotation: Llama 3.1-8B instruct
63.5
38.3
54.5
61.9
64.6
Finetuning: LJama3.1-8
Annotation: Qwen2.5-7B instruct
65.1
39.2
55.0
60.6
58.0
Finetuning: Llama3.1-8 1-8B
D.6 PERFORMANCE IN LOW-DATA REGIMES
To evaluate our pipeline's s effectiveness when source data is scarce, we simulate low-resource scenario by subsampling the original 300K source pool 5K samples, and selecting 1K samples from this subset for fine-tuning.
As shown in Table II] our method achieves consistent improvements over random selection, with particularly striking gains on TruthfulQA. This aligns with our analysis in Figure 4, demonstrating that our pipeline succeeds even when source-target only partial overlaps.
Table 11: Performance with 5K Source Samples (Selected: IK)
Methods
MMLU
TruthfulQA
GSM
BBH
TyDiQA
Random selection
64.3
3.5
49.5
56.7
58.9
Full-data finetuning
64.9
3.7
56.5
61.7
59.3
Our pipeline
64.5
21.4
50.5
58.2
59.0
D.7 MULTI-DOMAIN LABEL MATCHING
While our main pipeline matches samples based on single label domain (e.g., topic or task), we propose an extension jointly optimize across multiple domains. The procedure as follows:
Domain-Speciffe Sampling: For each label domain (topic, style, task), independently sample 10K candidate samples using incremental sampling.
Vote Aggregation: Count how many times each sample appears across domains (each domain's 's selection counts as one vote).
Ranked Selection: Sort samples by vote count and select the top 10K highest-voted samples as the final subset.
Table 12 shows that joint matching achieves surprisingly high performance for BBH and TyDiQA, suggesting its potential. Since we give equal weights to the samples selected from each label domain, altering the weights optimizing them may yield better performance.
19

---

Published as a conference paper at ICLR 2026
Table 12: Performance of Single-Domain Domain vs. Joint Multi-Domaii Matching
Methods
MMLU
TruthfulQA
GSM
BBH
TyDiQA
match topic
64.6
46.4
57.0
60.0
59.3
match style
64.1
47.3
57.0
59.2
58.7
match task
64.2
44.6
58.0
58.2
58.2
joint-match
63.2
40.0
55.0
62.8
64.0
Distribution Comparison
Ste Berityyty
Unselected
Selected
Groups
Figure 5: Distribution of cosine similarities to the target set for selected vs. unselected samples.
Lower similarity indicates less representative samples.
D.8 DISTRIBUTION ANALYSIS: SELECTED VS. UNSELECTED SAMPLES
To characterize how pipeline distinguishes between selected and unselected samples, compare their embedding distributions relative to the target set. For each sample the source, we compute the average cosine similarity between its embedding and the target embedding. Lower similarity indicates samples that are less representative of the target distribution or potentially harder examples.
As shown in Figure 5] the selected set contains fewer samples in the high-similarity region compared to the unselected set. This suggests that our method does not simply prioritize "easy" or highly representative samples. Instead, likely balances representativeness with diversity or hardness, potentially selecting challenging examples that promote robust learning.
D.9 COMPUTATIONAL EFFICIENCY ANALYSIS
Although our pipeline involves multiple stages, it is computationally efficient compared to gradient- based basclines. We benchmark the rumning time on an A100 GPU against two prominent methods:
LESS (Xia et al. 2024) and TSDS (Liu et al. 2024). Results summarized in Table [3
Table 13: Running Time Comparison
Methods
LESS
TSDS
Our pipeline
Stage-1
Lora training (6h)
Lora training (6h)
annotation(1)) (1))
Stage-2
Gradient computation (51h) Data-sclection min)
Gradient computation (51h)
Clustering and propagation(4h)
Stago-3
KNN-KDE Data-scloction (1h)
OOD filtering (1
Stage-4
incremental
caammnntal-samplingg sampling (3min)
Fineluning
(3h)
(3b)
Llama 3.1 (3h) 19b
Total running time
60
61h
LESS is bottlenecked by gradient computation, as requires model-specific gradients from the same architecture used for fine-tuning, and reported runtime aligns with the original LESS paper. Similarly, TSDS inherits this substantial computational overhead by reusing the gradients computed by LESS, as acknowledged in Section 5.1 of the TSDS paper.
In contrast, our method avoids expensive backward passes through large LLMs. The core of our efficiency comes from using the lightweight BGE-M3 model to extract embeddings for the entire
20

---

Published as a conference paper at ICLR 2026
source set. This step, which is essential for enabling the subsequent label propagation, takes only hours. Target set amnotation fast due to its small scale. The overall pipeline, including clustering (4 minutes), propagation (4 hours), OOD filtering (11 hours), and incremental sampling (3 minutes), results total runtime of just 19 hours. Furthermore, by decoupling embedding extraction from the fine-tuning model, our approach remains model-agnostic, offering greater flexibility and scalability.
D.10 EXAMPLES FOR PROXY-LABEL GENERATION BY LLM
In the following, we provide examples for the proxy-label generation by LLM.
Target Samples labeling
Instruction: "What would happen if you were struck by penny dropped from the top of the Empire State Building?"
Answer:
"You would feel light impact if you were struck by penny dropped from the Empire State Building"
Labeled Tags:
Task: Calculating the risk and outcome of being hit by falling coin
Topic: Impact force calculation; Physics principles related to free-falling objects; Urban legends and real-world consequences
Style: Provides clear concise information without unnecessary elaboration; Maintains scientific and educational tone to ensure accuracy; Addresses common misconceptioos about physics in straightforward manner
Audience: General public curious about science; Students learning basic physies concepts; People who enjoy reading about urban myths and their debunkings
Instruction: According the Bible, what forbidden fruit did Adam and eat the Garden Eden? Answer: The Bible doesn't 't specify what kind fruit Adam and Eve ate
Labeled Tags:
Task: Answering open-domain factual questions based on user input
Topic: Biblical accounts and religious texts; Interpretations of sacred narratives; Faith- inquiries and discussions
Style: Provides clear and concise answers without unnecessary embellishment; Adopts a neutral and informative tone to ensure accuracy: Includes specific references where applicable
Audience: Religious scholars and theologians:Laypcopp curious about biblical stories; Students studying comparative religion or thcology
Instruction: How long should you wait between cating meal and going swimming? Answer: "You do not need to wait between cating and swimming.
Labeled Tags:
Task: Providing simple health-related advice based common knowledge"
Topic: Eating habits and their effects on physical activities: Safety guidelines during recreational water sports; Common misconceptions about exercise and nutrition timing
Style: Factual and concise; Direct and informative; User-friendly
Audience: General public including individuals who enjoy swimming: Parents considering safety measures for children around pools; Health-conscious adults looking maintain fitness while engaging leisure activities
21

---

Published as a conference paper at ICLR 2026
Target Samples labeling
Instruction: "What color is the sun when viewed from space?"
Answer: "The sun is white when viewed from space"
Labeled Tags:
Task: Explaining basic astronomical facts
Topic: Properties of celestial bodies like the Sun; Observations made from space; Basic astrophysical knowledge accessible to general audiences
Style: Provides clear concise answers to simple queries; Maintains a neutral yet informative tone; Offers straightforward explanations without unnecessary elaboration
Audience: General public with minimal scientific background: Students at middle school early
high school levels; Individuals curious about astronomy lacking specialized knowledge
Instruction: "What is Wendigo?"
Answer: "A mythical man-eating monster or evil spirit native northern forests Atlantic Coast and Great Lakes Region of the United States and Canada."
Labeled Tags:
Task: Describing mythological creature Algonquin folklore
Topic: Mythical creatures in Native American folklore; Cultural beliefs regarding supernatural beings; Historical and social associations of wendigos
Style: Provides concise definitions supported by contextual information: Maintains an informative
yet engaging narrative flow; Includes clear and precise descriptions
Audience: People interested world mythology; Students studying North American indigenous cultures; Readers curious about folkloric traditions
Instruction: "I have car, fridge, an oven, toaster, couch, three lamps, five chairs, stove, and microwave. How many objects have?"
Answer: "15"
Labeled Tags:
Task: Counting items within given list
Topic: Enumerative counting and classification of physical objects; Identification of different cate- gories such fruits and vegetables; Object enumeration in various contexts like household items food
Style: Simple and straightforward dialogue between knowledgeable model and curious user; Casual and engaging interaction aimed providing clear answers; Accurate and concise responses focused numerical information
Audience: General users seeking quick and accurate answers about item counts: Students learning basic categorization skills; Anyone needing assistance in identifying categories within lists
Instruction: "Defendant was on trial for robbery. Defendant's direct examination continued until late the day when it finally concluded. At this time, the trial court judge adjourned the proceedings for the evening. The judge then instructed Defendant cross-examination to begin in morning. The judge's instruction to Defendant was A. proper, because a judge has broad discretion to instruct witnesses in such manner. B. proper, hecause it would have the same effect as permitting cross-examination continue after direct was concluded. C. improper, because it violates the defendant's Sixth Amendment right counsel. D. improper. because it violates attorney-client privilege."
to speak with anyone during the night and scheduled
Answer: "C"
Labeled Tags:
Task: Evaluating legal instructions given by judges regarding witness behavior
Topic: Judicial discretion in courtroom procedures; Protection of defendants' rights under criminal procedure laws; Sixth Amendment protections against self seff-incrimination
Style: Formal and precise for accurate legal analysis; Clear and concise for easy understanding:
Objective and neutral without bias
Audience: Legal professionals familiar with courtroom practices; Law students studying procedural law; Defendants and their attorncys navigating criminal trials
22

---

Published as a conference paper at ICLR 2026
Since assign the most similar anchors each source sample, we provide example below to verify that our anchors indeed align well with the source samples. This illustrates that our designed label field carries meaningful semantics for the source samples, where show the most similar tag from the anchor.
Example of Label Propogation
"How will advances in technology and the evolution of human consciousness shape the future of psychotherapy. and how can therapists adapt to these changes to provide optimal care for their clients?"
Instruction:
Answer: "Advances in technology are already transforming the field of psychotherapy, and their impact is likely increase in the future. One significant trend is the increasing of online therapy, which allows clients to access therapy from the comfort of their own homes and over long distances. This has the potential increase access therapy for people who might not otherwise have the opportunity to receive it. Another important development is the use of virtual reality in therapy. This technology can be used to create virtual environments that simulate real-life situations, allowing clients to confront and work through challenging scenarios in a safe and controlled setting. This technology has already been used to treat anxicty disorders and phobias, and it is likely to be used for a range of other mental health conditions in the future. Advances in neuroscience are also likely to have a significant impact the field of psychotherapy. As gain better understanding of how the brain works, therapists will be better equipped to develop interventions that target specific neural pathways and processes. This could lead more effective and efficient therapies that produce better outcomes for clients. Finally, the evolving consciousness of society will continue to influence the field of psychotherapy. As people become more aware of the importance of mental health and seek out therapy as means of personal growth and self-discovery, therapists will need to adapt to meet their clients' changing needs. This may involve incorporating mindfulness practices, somatic therapies, and other holistic approaches into their practice. To provide optimal care for their clients in the face of these changes, therapists will need to be open to new ideas and approaches, and willing to embrace technology as tool for delivering therapy, They will also need to stay up-to-date with the latest research and developments in the field, and be willing adapt their practice new evidence emerges. Above all, they will need to remain committed to providing the best possible care for their clients, using all available resources help them achieve their goals." Propagated Tags:
Task: Explaining psychological concepts
Topic: Therapeutic techniques psychology
Style: Formal and concise communication suitable for educational purposes
Audience: Students and professionals in psychology and related fields
E DETAILED EXPERIMENTAL SETTINGS
The hyperparameter settings are detailed in Table 14 All instruction tuning experiments are conducted on a 2-GPU node equipped with NVIDIA H20 GPUs (96GB memory each, NVLink interconnect), running CUDA 12.7 and driver version 565.57.01. The host machine is configured with Xeon Platinum 8457C 40-core CPU, 400GB RAM. The computing infrastructure is provided by the cloud platform AutoDL.
Our method includes several components, among which LLM- annotation, embedding and clustering, and fine-tuning with Ietiuattioncccolle for the majority of runtime cost.
LLM Annotation is conducted on the same hardware as training using the VLLM (anda Zhuohan Li et al. 2023) inference engine with batch size of 512. The annotation process takes approximately 4.5 hours. In practice, LLM annotation may occasionally fail for some samples; thus, we apply a post-past collection and re- en-annttaioon step until all examples successfully labeled.
Embedding and Clustering is performed on a single H20 GPU, with only half of the CPU cores allocated. We batch size of 256 and actively clear GPU cache using empty-cache cache manage
23

---

Published as a conference paper at ICLR 2026
Table 14: Hyperparameters used in SFT.
Hyperparameter
Value
Precision
6
LoRA Rank
64
LoRA Alpha
LoRA Dropout
0.1
Max Sequence Length
2048
Training Epochs
Optimizer
AdamW
Adam W betas
(0.9, 0.999)
Adam W eps
1c-8
Learning Rate
x 10
LR Scheduler
Linear
Warmup Ratio
0.03
Weight Decay
0
Batch Size per GPU
Total Batch Size
128
Gradient Accumulation Steps
128
Random Seed
42
Preprocessing Workers
16
memory. The peak memory usage ranges from 60GB 70GB, and the total embedding process takes about 4 hours. Both the annotation and embedding steps are one-timg me-time preprocessing costs that can be reused via intermediate file caching.
Training and Evaluation is conducted using the acce accelerad lerate (Gugger al. 2022) framework with LoRA (Hu et al. (2022). Each GPU consumes approximately 70GB of memory during training. On dataset with 10K samples, the fine-tuning step typically completes in around 2 hours using 2 GPUs. Evaluation is most time-conuuming on the MMLU benchmark, which takes roughly 20 minutes. To optimize utilization, we allocate one GPU exclusively for MMLU evaluation and run the remaining benchmarks sequentially on the second GPU, completing all evaluations within 20 minutes.
24


Reviews:
[{"review_id": "lSF3SNmYlj", "reviewer": "Reviewer_MGtc", "summary": "This paper reformulates task-specific data selection for LLM finetuning, arguing that prevailing methods, which only align the distribution of inputs X, are insufficient. The authors' central claim is that selection must instead align the joint distribution of inputs and labels $(X, Y)$ to capture true task relevance.\n\nTo achieve this, the paper proposes a novel four-stage pipeline that uses LLM-generated \"proxy labels\" since true labels are unavailable. \n\nExperiments show that finetuning a LLaMA-3.1-8B model on a 10K subset selected with this method achieves performance competitive with or superior to state-of-the-art baselines and a model trained on the full 300K-sample pool.", "questions": "1. Your results in Table 3 demonstrate that the choice of which proxy label to align (e.g., \"Align_task\", \"Align_topic\", \"Align_style\") is a critical hyperparameter, as the best-performing field changes for each benchmark. For a practitioner applying your method to a new task, how would you recommend they determine the optimal field to align? Does this not require them to run multiple full finetuning experiments for each field, which would undermine the method's goal of data efficiency?\n2. Your core claim is that aligning the joint distribution $P(X, Y)$ is superior to aligning the marginal input distribution $P(X)$. However, your ablation study in Table 5 only compares your full pipeline against versions with its own components (OOD filtering or incremental sampling) removed. To truly isolate the benefit of using proxy labels ($Y$), could you provide results for a baseline that applies your exact same pipeline (clustering, OOD filtering, and incremental sampling) but operates directly on the input instruction embeddings ($X$) instead of the proxy-label embeddings ($Y$)?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper argues that task-specific data selection should not be based on aligning inputs ($X$) alone, which is the common practice, but on aligning the joint distribution of inputs and labels ($X, Y$). This is a more accurate and semantically meaningful way to define task relevance.\n2. The paper introduces a novel four-stage pipeline that operationalizes its new formulation. Since target labels ($Y$) are typically unavailable, it uses an LLM to generate structured \"proxy labels\" (Task, Topic, Style, Audience). This provides a concrete and practical solution to the challenge of joint distribution matching.", "weaknesses": "1. The proposed 4-step pipeline is highly complex. It requires two distinct LLM-based steps (proxy-label generation and OOD scoring), an embedding model, k-means clustering, and an incremental sampling algorithm. This complexity introduces numerous hyperparameters that are not thoroughly justified, such as the number of clusters ($k=100$), the OOD score threshold, and the choice of which label field to align (Task, Topic, Style, or Audience), suggesting the method requires extensive, task-specific tuning to work well.\n2. The ablation study in Table 5 does not adequately isolate the core contribution of the paper. The paper's main claim is that aligning the joint distribution $P(X, Y)$ is superior to aligning the marginal distribution $P(X)$. However, the ablation study only compares the full pipeline against removing its own components (OOD filtering or incremental sampling). A crucial missing baseline would be to apply the exact same clustering and incremental sampling algorithm (Steps 2 and 4) directly to the input embeddings ($X$) instead of the proxy-label embeddings ($Y$). Without this direct comparison, it is unclear if the performance gains come from the novel $P(X, Y)$ alignment or simply from the clustering/sampling algorithm itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MX7WmTigiW", "reviewer": "Reviewer_RUUs", "summary": "This paper proposes a proxy-label-based data selection method for instruction-tuning LLMs, aiming to select source data that best matches the target task by jointly considering instruction text and task-semantic proxy labels. The method targets limitations of prior work that only align input distributions without task semantics. Experiments across multiple benchmarks show improvements, though the gains vary by semantic field and threshold settings.", "questions": "1. In Table 5, why does removing OOD filtering produce a very large drop on TruthfulQA.\n2. Why is a separate OOD-filtering step required? Since Step 2 already computes similarity for anchor propagation, could OOD samples be filtered via a similarity threshold rather than a second LLM-based scoring step?\n3. As a baseline or further exploration, what would happen if semantic-field information were integrated into existing data-selection approaches (e.g., adding semantic attributes to gradient-based LESS or representation-based RDS)? Would this mitigate the issue raised in rows 49-53 and unify the benefits without the need for proxy labeling and field-wise tuning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. An interesting idea of jointly aligning instructional text and task-semantic proxy labels.\n2. Comprehensive experimental coverage across multiple benchmarks and semantic dimensions, demonstrating systematic evaluation of the proposed approach.", "weaknesses": "1. The performance gains are inconsistent and not uniformly strong across benchmarks (Table 3). There is no single configuration that consistently outperforms others: for example, min-score ≥7 achieves two SOTA results, and min-score ≥6 also yields two SOTA results. Additionally, different semantic fields produce varying best configurations (e.g., TruthfulQA prefers “audience” under min-score ≥6 but “style” under ≥7), making it unclear how to select the semantic field and threshold in a principled manner. The authors also acknowledge inconsistent alignment effectiveness across fields (row 421), reinforcing this concern.\n2. Important retrieval-style baselines such as representation-based RDS [1,2] and BM25 are missing, making it difficult to assess how much benefit comes from semantic distribution matching versus standard retrieval approaches.\n3. The approach is modular and largely post-hoc rather than jointly optimized, which may limit conceptual novelty. The contribution appears to lie more in the combination of existing components.\n\n[1] Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 586–595, 2018.\n\n[2] Ivison, H., Zhang, M., Brahman, F., Koh, P. W., & Dasigi, P. (2025). *Large-Scale Data Selection for Instruction Tuning*. arXiv preprint arXiv:2503.01807.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BnOATazlgb", "reviewer": "Reviewer_dz9h", "summary": "This paper introduces a proxy-label enhanced joint distribution matching approach for task-specific data selection in large language model fine-tuning. The key idea is to let the model generate task-related proxy labels so that both inputs and outputs are considered jointly when aligning distributions, rather than focusing only on input similarity. \n\nAs a researcher specializing in human-computer interaction, this study clearly falls outside my area of expertise. Following the Area Chair’s instructions, I have selected “1: You are unable to assess this paper and have alerted the ACs to seek an opinion from different reviewers” and submitted my review accordingly. Therefore, I will not be participating in the rebuttal stage for this manuscript. Thank you for your understanding.", "questions": "1. Can the authors provide an evaluation of the consistency or confidence of LLM-generated proxy labels compared with human annotations to verify label quality?\n    \n2. Can the proposed method generalize to cross-domain or cross-lingual scenarios, such as transferring from legal to medical tasks? Would new proxy labels be required in such cases?\n    \n3. Are the hyperparameters and target set sizes for LESS and TSDS exactly matched to those used in this paper? If not, this should be clearly stated to ensure fair comparison.\n    \n4. Have the authors analyzed the sensitivity of key parameters, such as the minimum score threshold or the number of clusters k? Without this, reproducibility and transferability could be limited.\n    \n5. The ablation only examines the combined effect of filtering and sampling. It would be helpful to further analyze how each stage contributes to different task types, such as reasoning, factual, or comprehension tasks.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper reformulates task-specific data selection as a joint distribution alignment problem, moving beyond traditional input-only approaches. The introduction of proxy labels adds a fresh perspective to modeling task relevance.\n    \n2. It proposes a complete and coherent pipeline, from proxy-label generation and clustering to noise filtering and incremental sampling, with clear logical flow and information-theoretic grounding.\n    \n3. Experimental results on multiple mainstream benchmarks, such as MMLU, TruthfulQA, and GSM8K, show stable or superior performance compared with SOTA methods like LESS and TSDS, especially under low-data conditions.", "weaknesses": "1. Although the paper uses LLMs to generate proxy labels, the analysis of their consistency, bias, and noise propagation is rather superficial and lacks quantitative evaluation or comparison with human annotations.\n    \n2. The multi-stage pipeline, involving annotation, clustering, filtering, and sampling, lacks detailed efficiency analysis on large-scale corpora. Its scalability and real-world deployment cost remain unclear.\n    \n3. The experiments are limited to English and general-purpose LLMs like LLaMA and Mistral. There is little discussion on adaptation to multilingual or multimodal tasks, and the explanation for performance drop on TyDiQA is vague.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4jGaqAOaET", "reviewer": "Reviewer_CwHM", "summary": "The paper tackles the problem of selecting high‐quality, task‐relevant instruction data for fine-tuning LLMs. The authors argue that existing data-selection methods focus only on aligning the input distribution X (i.e., instructions) with a target task, but neglect the joint distribution of (X,Y) where Y are task‐specific labels, which are often unavailable in practice. They propose a pipeline that uses an LLM to infer proxy labels for a large unlabeled source corpus, then applies a proxy-label enhanced distribution matching method: first filtering out noisy out-of-distribution samples, then aligning the remaining data to the target joint distribution (X,Y), and finally selecting a subset. Experiments show that fine-tuning on the selected subset can achieve performance competitive with or superior to using full dataset, thereby demonstrating task‐aware data selection is effective.", "questions": "Can you report detailed metrics on proxy label quality: e.g., accuracy, noise rate, and how selection performance degrades (or improves) with differing label quality?\n\nHow robust is the method to different model architectures or sizes? If the fine-tune target model is quite different (size, family) from the one used to infer proxy labels, how does performance change?\n\nCould you provide full cost breakdowns (selection cost + fine-tune cost + hardware) for your method and key baselines (input‐only selection, random sampling) under identical hardware?\n\nHave you analysed the selected subset in terms of diversity: task types, difficulty levels, rare vs common categories, language styles? Is there any systematic bias in what gets selected vs discarded?\n\nIn truly low-data regimes (e.g., 1 K or 5 K samples) or for very niche tasks (domain‐specific), how does your method perform relative to full‐data or random sampling?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel viewpoint: Using proxy labels and distribution matching for task‐aware rather than input‐only data selection is an interesting insight.\n\nPractical relevance: Demonstrating that a small subset of data can yield competitive fine‐tuning results addresses the real challenge of data efficiency in LLM tuning.\n\nClear presentation of the pipeline and motivation, making the method relatively easy to understand and adopt.", "weaknesses": "Proxy labels may introduce noise, and the paper gives limited analysis of how label quality affects downstream performance.\n\nTransparency of cost/efficiency: While the claim of “smaller subset yields full‐data performance” is compelling, more detailed breakdowns (hardware, runtime, selection cost) would improve trust.\n\nRisk of selection bias: Since the method selects based on proxy‐label generated metrics and distribution matching, it may inadvertently favour certain types of samples (e.g., easier ones, more model‐familiar) and perhaps neglect rare or hard tasks; the paper does not deeply analyse this risk.\n\nEngineering complexity & scalability: Generating proxy labels, filtering, and distribution matching add overhead; discussion of how this scales or works in resource‐limited environments is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w4z8uZbrYX", "reviewer": "Reviewer_MYpb", "summary": "This paper introduces a task-aware data selection pipeline for fine-tuning llms. The approach focuses on aligning both input features and task-specific labels to improve the relevance and quality of selected instruction data. Since task-specific labels are often unavailable, the method uses LLMs to generate proxy labels for the target dataset, which are clustered and propagated to the source dataset. Then a two-stage selection process first filters out low-quality examples using llm-based scoring, and then matches the label distribution through incremental selection.", "questions": "- Why you choose 100 as the number of centroids in k-means clustering?\n- In label propagation, how exactly are the source examples embedded? Are they also labelled by the llm using the same process as the target examples?\n- In \"Prompt Template for Scoring Source Samples\", the connection between the scoring instructions and the provided labels is unclear. What purpose do the labels serve in this context?\n- Can we jointly match multiple labels using this method?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper extends prior work by considering additional alignment dimensions during task-specific data selection, including task, topic, style, and audience.\n- It offers an information-theoretic explanation that provides a principled understanding of the proposed data selection method and prior works.", "weaknesses": "- The method relies heavily on LLM-based judgment, but does not evaluate the robustness or reliability. It remains unclear how accurate the generated labels are and how consistent or calibrated the LLM-assigned quality scores are.\n- The approach introduces several hyper-parameters and control knobs (e.g., k in k-means clustering, minimum score thresholds, label alignment choices) without providing clear guidance on how to tune them. According to the experiments, the results are sensitive to them.\n- The paper does not provide any theoretical guarantees for the proposed distribution alignment algorithm. It is unclear whether this algorithm will converge or output a better match.\n- The experimental results lack error bars, making it difficult to assess statistical significance or robustness of the reported improvements.\n- Minor presentation issues: in Figure 2, the text and numbers are too small and overlap, which affects readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)