Title: T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Instruction tuning

Data Selection

Token-selective Quality Score

Robust Hierarchical Selection
Abstract: Instruction tuning is essential for Large Language Models (LLMs) to effectively follow user instructions. To improve training efficiency and reduce data redundancy, recent works use LLM-based scoring functions, e.g., Instruction-Following Difficulty (IFD), to select high–quality instruction-tuning data with scores above a threshold. While these data selection methods often lead to models that can match or even exceed the performance of models trained on the full datasets, we identify two key limitations: (i) they assess quality at the sample level, ignoring token-level informativeness; and (ii) they overlook the robustness of the scoring method, often selecting a sample due to superficial lexical features instead of its true quality. In this work, we propose Token-Selective HIeRarchical Data Selection for Instruction Tuning (T-SHIRT), a novel data selection framework that introduces a new scoring method to include only informative tokens in quality evaluation and also promote robust and reliable samples whose neighbors also show high quality with less local inconsistencies. We demonstrate that models instruction-tuned on a curated dataset (only 5% of the original size) using T-SHIRT can outperform those trained on the entire large-scale dataset by up to 5.48 points on average across eight benchmarks. Across various LLMs and training set scales, our method consistently surpasses existing state-of-the-art data selection techniques, while also remaining both cost-effective and highly efficient. For instance, by using GPT-2 for score computation, we are able to process a dataset of 52k samples in 40 minutes on a single GPU. Our code is available at https://github.com/Dynamite321/T-SHIRT.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSelection of high quality data is critical for efficient and effective instruction-tuning. Existing methods turn toward sample-level quality scores assigned by humans or proprietary LLMs, which are expensive to compute and overlook token-level features of instruction samples. Further, semantically similar samples can often have varying quality score assignments, which is not accounted for by threshold-based data selection. This paper addresses these challenges through T-Shirt, a reference-free data selection framework that considers only informative tokens when assigning quality scores and prioritizes samples whose neighbors achieve high quality scores to ensure robustness and consistency. Empirical evaluations show T-Shirt is highly scalable and efficient while incurring low monetary and time costs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* The paper introduces a novel proposal to consider token-level information and local quality scores of related neighbor data points to evaluate training data quality.\n* The proposed methodology is well-motivated and clearly described; figures and data presentation are effective.\n* Experiments have good coverage of relevant benchmarks, baselines, and ablations, and results are strong even when using only GPT2 for score computation.\n* The paper provides evidence that using even a weak selector model can get effective samples in short amount of time, consistent with some recent works.\n\nWeaknesses\n* No major weaknesses. While results are convincing, the T-Shirt-selected data is only used to fine-tune two moderately sized base LLMs (<=8B parameters), with efficacy at larger scales unclear.\n\nQUESTIONS:\n* How does random noise injection compares to systematic lexical perturbations when it comes to IFD and S-IFD scores, and selection efficacy? \n* How effective is the pipeline toward fine-tuning LLMs across model scales (e.g., efficacy of the selected data for larger models trained with LoRA, will the observed performance gains still hold)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies two key limitations of prior data selection approaches (e.g., IFD) for instruction tuning: (i) they evaluate sample-level data quality, ignoring token-level information; and (ii) they overlook the robustness of data quality assessment. To address these issues, the authors propose T-SHIRT, a new data selection framework that incorporates fine-grained token-level information and favors samples with consistently high-quality neighbors. They conduct comprehensive experiments demonstrating that datasets curated using T-SHIRT outperform full datasets and those filtered by other data selection methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The authors use a semantics-preserving word substitution example to illustrate that IFD-based scores are highly sensitive to subtle textual changes, suggesting that selecting samples with high-quality neighbors is more effective than relying solely on fixed score thresholds.\n- This study provides fine-grained signals for high-quality data selection by incorporating token-level information, emphasizing that not all tokens contribute equally when assessing example quality.\n- They conduct comprehensive experiments using two instruction-tuning datasets of different scales, evaluate performance on eight benchmarks, and compare results against six baseline methods.\n- The authors perform comprehensive ablation studies to evaluate the contribution of each major component in their proposed method.\n### Weaknesses\n- Since this study emphasizes token-level information when selecting training examples, it would be interesting to investigate whether examples selected by T-SHIRT offer advantages at the token level compared to those selected by other baselines during instruction tuning.\n- I’m not fully convinced that removing all tokens with small delta_t values is justified. Since these values are already quite small, their cumulative contribution to the final score may be negligible. Could you please provide a plot showing the distribution of delta_t values to better understand their overall impact? Additionally, have you considered weighting tokens by their delta_t values instead of removing the less significant ones entirely?\n\nQUESTIONS:\n- Have you considered other scoring models used in score computation? Do you have any insights into how the choice of scoring model influences the quality of data selection?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for selecting high-quality instruction tuning data by addressing two important gaps in the existing literature: (1) overlooking token-level information for selection and (2) failing to incorporate robustness as a selection metric. The authors propose to firstly be selective when computing IFD and only consider the influence of tokens that exhibit large and informative signals, and secondly consider the neighborhood of samples through random perturbation in the embedding space and select data with high score and low variance. Through empirical experiments, the authors show impressive performance and efficiency of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper builds on an interesting observation that not all tokens are contributing equal to the quality of the instruction tuning sample, hence proposing a novel method that carefully eliminates the effects of those less useful (or noisy) tokens when performing data selection.\n2. The robustness perspective in scoring the utility of the data is new and useful.\n3. Throughout the paper, there are good illustrative examples to demonstrate the rationale behind the specific designs of the methodology.\n4. The proposed method exhibits impressive empirical performance, computational cost. At the same time, the paper includes extensive ablation studies.\n\nFor weaknesses, refer to the Questions section below.\n\nQUESTIONS:\nI don’t have much to complain about the paper. I am happy to have read a high-quality write-up with insightful methodology design and comprehensive experiments. \n\nSome minor points: \n\n1. It would be good to add repeated experiments and hence error bars for the experiments. Especially, baselines like RANDOM or DS2 have randomness in the selection, and the proposed T-SHIRT also uses random noises, it is critical to show the stability of the method through repeated, reproducible experiments. I could not be certain about the statistical significance right now.\n2. It seems that the selection ratio $k$ is an essential hyperparameter that affects the performance much according to Table 1 and Figure 6. As your method does not rely on a validation dataset, what would be a recommended method to choose, or potentially optimize for this hyperparameter.\n3. While the authors have included “Model Scale” as a limitation of the paper, the paper should at least show some larger-scale model experiments to show the scalability and generalizability of the proposed method. I suggest the authors to run these experiments.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMany recent papers have studied how to select samples to use for instruction tuning by scoring with an LLM. In particular, an approach called IFD scores samples based on the change in response perplexity when conditioned on the instruction. However, this approach has two issues: 1) the change in response perplexity is computed over all tokens, even though some tokens in the response are very predictable regardless of the instruction (i.e., low change in PPL), and 2) the change in response perplexity may be sensitive to the choice of instruction --- small perturbations may impact this score. To address this, the paper presents T-SHIRT, which scores samples based on the top k% of token-level perplexity changes, rather than the perplexity change over the entire response. This scoring mechanism reduces the noise from uninformative tokens, like \"Einstein\" often coming after \"Albert\". Second, T-shirt more robustly selects samples based on a smoothed score within a neighborhood, where the neighborhood is constructed by adding Gaussian noise to the token embeddings. Experiments are conducted on two instruction tuning datasets and Llama/Qwen as base models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nQuality: \n- T-SHIRT's performance is demonstrated on both Alpaca-GPT-4 and Magpie datasets. This is promising, given that these datasets differ in how old they are and in terms of how heavily curated they are. This suggests that T-SHIRT is robust and may continue to offer improvements on future instruction tuning datasets.\n\nClarity:\n- Paper is well-motivated, with illustrative examples of 1) cases where IFD and S-IFD have different vs similar scores and 2) cases where IFD/S-IFD are sensitive to the wording in the instruction. \n\nSignificance:\n- The ability to score samples for data selection in a cheap and effective way is important. In comparison, many existing data selection methods involve using relatively large, costly LLMs.\n\n**Weaknesses**\n\nQuality:\n- In Table 1/Table 2, there are some tasks (HS, MMLU, BBH, AH) where IFD does better than both versions of T-SHIRT. Is this due to the choice of T-SHIRT's hyperparameter k? That is, is there a setting of k where T-SHIRT consistently outperforms IFD, but this was not identified due to coarse sweeping? The method would be more compelling if you could confirm that there always is some k/other hyperparams such that using top-k% token scoring always outperforms scoring based on all tokens equally.\n- In Figure 6a, it seems like the trends of k% vs performance are not consistent between Llama and Qwen. For Llama, higher k% translates to better performance, with approximately a 1 point difference between k=25 vs 75 - this is quite significant when you look at the range of accuracies in Table 1. For Qwen, performance appears to slightly decrease as k% increases. This inconsistency is surprising given that 28% of tokens have low $\\triangle_t$ for both Llama and Qwen (L152). This raises concerns about whether we fully understand how T-SHIRT interacts with different base models, and about how practitioners using other base models should set k if they want to use this method. \n- A minor suggestion to strengthen the case for T-SHIRT in comparison to methods that use API-based models to score samples: it would be interesting if one could implement DEITA/DS^2 using GPT2-based scores. Perhaps, you could show that GPT-2 fails to produce meaningful scores, and thus T-SHIRT/IFD are the only methods that would work at all when using a smaller LM. That is, can you argue that if a practitioner is budget constrained and must use a weaker LM, then no construction of DEITA or DS² will be viable? This would help clarify a practical niche that T-SHIRT/IFD has.\n\nClarity:\n- What does token embedding perturbation look like in terms of the actual text? Does it look vastly different from the word substitution example provided that swaps \"mean\"/\"average\"?\n\nOriginality: \n- While the proposed modifications to IFD are sensible, they have somewhat limited novelty, given that token-level scoring and neighborhood smoothing for robustness are well-known in many contexts. However, I am not completely familiar with all the works in the instruction tuning data selection space, so this concern is minor.\n\nQUESTIONS:\n1. In Table 1/Table 2, there are some tasks (HS, MMLU, BBH, AH) where IFD does better than both versions of T-SHIRT. Does there exist a k besides 50%/75% that does allow for T-SHIRT to outperform IFD? \n2. Can we better understand the inconsistency in the relationship between k% and the performance across Qwen and Llama? For instance, perhaps a more fine-grained sweep of k or an assessment on another LM would be helpful. \n3. What does token embedding perturbation look like in terms of the actual text? Does it look vastly different from the word substitution example provided that swaps \"mean\"/\"average\"?\n \nOverall, my score is a borderline accept. The proposed modifications are well-motivated and likely to be useful in practical, budget-constrained settings. However, I am not giving a 5 because I believe the experiments could be expanded to more thoroughly assess the limits of T-SHIRT and its behavior across different base models. I also find the novelty somewhat incremental, though I defer to reviewers with deeper familiarity in this area, and I am open to changing my opinion on this.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new instruction tuning method, called T-SHIRT, i.e., Token-Selective Hierarchical Data Selection for Instruction Tuning.  The general idea is to select high–quality instruction-tuning data to improve training efficiency and reduce data redundancy.  Some recent studies use LLM-based scoring functions, e.g., Instruction-Following Difficulty (IFD). While these data selection methods often lead to models that can match or even exceed the performance of models trained on the full datasets, this work identifies two key limitations: (i) they assess quality at the sample level, ignoring token-level informativeness; and (ii) they overlook the robustness of the scoring method, often selecting a sample due to superficial lexical features instead of its true quality. \n\nThis work proposes Token-Selective HIeRarchical Data Selection for Instruction Tuning (T-SHIRT), a novel data selection framework that introduces a new scoring method to include only informative tokens in quality evaluation and also promote robust and reliable samples whose neighbors also show high quality with less local inconsistencies. The paper demonstrates that models instruction-tuned on a curated dataset (only 5% of the original size) using T-SHIRT can outperform those trained on the entire large-scale dataset by up to 5.48 points on average across eight benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n1.  It identifies two limitations of recent instruction tuning works which use LLM-based scoring functions: (i) they assess quality at the sample level, ignoring token-level informativeness; and (ii) they overlook the robustness of the scoring method, often selecting a sample due to superficial lexical features instead of its true quality.  \n\n2. It proposes a new instruction tuning method, T-SHIRT, which selects high–quality instruction-tuning data to improve training efficiency and reduce data redundancy.  The new methodology on instruction tuning: It uses hierarchical selection to choose samples whose neighbors exhibit high average S-IFD and low variance. T heir token-selective hierarchical data selection for instruction tuning method introduces a new scoring method to include only informative tokens in quality evaluation and also promote robust and reliable samples whose neighbors also show high quality with less local inconsistencies\n\n3.  High performance: The new method T-SHIRT can generate new instruction-tuned models on a much small, curated dataset (5% of the whole set) but outperforms those trained on the entire large-scale dataset.\n\n4.  The overall analysis seems to be clear and in high quality.\n\n Weakness:\n\n1. For Table 1 (the major performance comparison between our method, T-SHIRT, and other baseline methods on Alpaca-GPT-4), the performance gain seems not that great.   Especially using LLM as a judge (which is closer to human evaluation), the other method seems to show higher performance on Llama 3.1-8B.   I am not sure how you can explain this result.\n\n2. The ablation study does not show quite impressive differences.  I am not sure how you can explain this performance result (Table 4).\n\n3.  The experiments are all done on relatively smaller models (e.g., Llama 3.1-8B, Qwen 2.5-7B).  You mentioned this in the limitations but it should have some discussions on how the method can be extended to large models and what can be difficulties based on your experiments.\n\nQUESTIONS:\nI raised a few questions in the weakness section.  Please check those questions and have some clear explanation.\n\nLIMITATIONS:\nThe limitation section discussed a few issues related to the current work.  I think the limitation discussion section is clear and informative.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}