Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
Authors: Jinlong Pang
Na Di
Zhaowei Zhu
Jiaheng Wei
Hao Cheng
Chen Qian
Yang Liu
Venue: ICML 2025 poster (2025)
Keywords: LLM
Supervised Fine-tuning
Data Selection
Token Cleaning
Primary area: deep_learning->large_language_models
TL;DR: We propose a fine-grained token cleaning pipeline for LLM instruction tuning to boost overall performance.

Abstract: Recent studies show that in supervised fine-tuning (SFT) of large language models (LLMs), data quality matters more than quantity. 
While most data cleaning methods concentrate on filtering entire samples, the quality of individual tokens within a sample can vary significantly. After pre-training, even in high-quality samples, patterns or phrases that are not task-related can be redundant, uninformative, or even harmful. Continuing to fine-tune on these patterns may offer limited benefit and even degrade downstream task performance.
In this paper, we investigate token quality from a noisy-label perspective and propose a generic token cleaning pipeline for SFT tasks. Our method filters out uninformative tokens while preserving those carrying key task-specific information. Specifically, we first evaluate token quality by examining the influence of model updates on each token, then apply a threshold-based separation. The token influence can be measured in a single pass with a fixed reference model or iteratively with self-evolving reference models. The benefits and limitations of both methods are analyzed theoretically by error upper bounds. Extensive experiments show that our framework consistently improves downstream performance. Code is available at https://github.com/UCSC-REAL/TokenCleaning.

Token Cleaning: Grainer Data Selection for LLM Supervised Fine-Tuning
Jinlong Pang' Na Di Zhaowei Zhu
Jiaheng Wei Hao Cheng Chen Qian Yang Liu'
Abstract
involves fine-tuning pre-trained models using annotated in- structional data (Touvron et al., 2023). Following general data scaling laws (Zhang et al., 2024), significant efforts have been dedicated to collecting large-scale instructional data containing millions of examples (Wang et al., 2022; Chung et al., 2024; Longpre et al., 2023).
Recent studies show that in supervised fine-tuning (SFT) of large language models (LLLMs), data quality matters more than quantity. While most data cleaning methods concentrate on filtering entire samples, the quality of individual tokens within a sample can vary significantly. After pre- training, even in high-quality samples, patterns or phrases that are not task-related can be redun- dant, uninformative, or even harmful. Continuing to fine-tune these patterss may offer limited benefit and even degrade downstream task per- formance. In this paper, we investigate token quality from a noisy-label perspective and pro- pose a generic token cleaning pipeline for SFT tasks. Our method filters out uninformative tokens while preserving those carrying key task-specific information. Specifically, we first evaluate to- ken quality by examining the influence of model updates on each token, then apply a threshold- based separation. The token influence can be mea- sured in a single pass with a fixed reference model or iteratively with self-evolving reference mod- els. The benefits and limitations of both methods are analyzed theoretically by error upper bounds. Extensive experiments show that our framework consistently improves downstream performance. Code is available at https: github. com/ UCSC- Crsc-RRaa//ToeenCleent
Recent studies on the SFT have widely agreed that data quality matters far more than quantity (Zhou et al., 2024; Chen al., 2023; Pang et al., 2024b; Liu et al., 2023). That is, small, well-curated dataset can often deliver effective or even superior performance on downstream tasks, highlight- ing the critical role of data cleaning or selection. Existing data cleaning approaches primarily emphasize identifying high-quality samples in large dataset pools via some metrics, including perplexity (Cao al., 2024), completion length (Zhao et al., 2024), confidence scores (Chen & Mueller, 2024), LLM-ggnnrraed quality ratings (Chen et al., 2023; Pang et al., 2024b; Liu et al., 2023) or even costly human an- notations (Zhou et al., 2024). Although these methods have proven effective, focusing solely on sample-level cleaning may overlook complexities within each sample.
In practice, each sample typically contains hundreds of to- kens, some of which occur frequently regardless of the sample's quality. These common tokens/patterns can over- shadow task-specific words that are crucial for model perfor- mance during training. Moreover, during interference, if the model continually outputs these frequent tokens, it may ne- glect more informative ones, producing outputs that appear correct yet fail address specific tasks. Thus, even well- curated samples can contain token-level noise that dilutes es- sential signals. For convenience, we refer to tokens that are uninformative, low- www-impact, or even harmful as uninforma- live tokens, and the rest as informative tokens. Addressing these token-level issues by removing or down- weightigg the uninformative tokens can help the model prioritize impor- tant and informative tokens and then improve downstream task performance (Lin al., 2024).
1. Introduction
Supervised fine-tuning (SFT) has served as widely adopted approach and a fundamental step in aligning large language models (LLMs) with human expectations. This process ensures that LLMs can accurately understand human in- structions and produce relevant responses. In practice, SFT
UUniversity of California, Santa Cruz NNortheastern Univer- sity DDocta.ai Hong Kong University of Science and Technology (Guangzhou) 'Hong Kong Baptist University. Correspondence to: Yang Liu <yangliu @ ucsc.edu>.
In this paper, we go beyond traditional sample-level data cleaning by proposing generic token cleaning pipeline and an analytical framework for LLM SFT tasks. Specifically, we filter out uninformative tokens while retaining those with meaningful speciffe information. This achieved by first assessing token quality using an influence-guided
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
scoring mechanism, followed by fireshold-based separation.
completion length (Zhao et al., 2024), perplexity (Cao et al., 2024), reward scores (Gou & Nguyen, 2024), discrete confi- dence scores (Chen & Mueller, 2024), the loss disparities when certain examples are included or excluded (Li et al., 2023), gradient matching (Zhou et al., 2023) and influence function scores (Xia et al., 2024). Another line of work uses advanced LLMs directly to filter out low-quality sam- ples according to different metrics, such as quality-based rating scores (Chen et al., 2023; Liu et al., 2023; Pang et al., 2024b) and fine-grained tags (Lu et al., 2023). Diversity- aware scoring has also been integrated into the overall qual- ity assessment, highlighting its importance. Although ex- tensive data selection methods have shown promise, fine- grained token-level selection remains underexplored. Re- cent studies (Lin et al., 2024) have highlighted the signifi- cant benefits of token selection during the pre-training phase, yet its application in the SFT has received limited attention.
Figure/ illustrates two scoring strategies supported by our token cleaning pipeline. The high-level idea is to evaluate the influence of model updates on each token, which can be calculated by the loss disparity between a base model and a reference model. We introduce two implementations:
Fixed-Model Cleaning. In this strategy, both the base model and the reference model remain fixed, and a one- shot token cleaning is applied to the entire SFT dataset. The base model is then fine-tuned the cleaned tokens, producing the final model output. This strategy is similar to the latest token selection method for pre-trained data (Lin et al., 2024). We defer detailed comparisons to Section 4.3.1 (analytical) and Section 6 (experimental). Self-Evolvin Cleaning. In this strategy, the base model remains fixed while the reference model is updated it- eratively. The data is divided into multiple parts, with each iteration cleaning one part. The reference model is then updated sequentially using the cleaned results from each part. Unlike the fix-model cleaning, the final model output is the reference model obtained in the last iteration.
Noisy Data Cleaning The learning with noisy labels has been extensively studied (Vahdat, 2017; Veit et al., 2017: Li et al., 2017; Liu & Wang. 2021; Yuan et al., 2024). Vari- ous approaches have been proposed mitigate label errors, including developing noise-tolerant loss functions (Natara- jan et al., 2013; Reed et al., 2014; Zhu al., 2021) and identifying clean samples while re-labeling corrupted ones (Northcutt et al., 2021; 2017; Cheng et al., 2021; Zhu et al., 2022). Recently, the issue of noisy labels in LLM alignment has gained increasing attention, driven by the observation that data quality is far more critical than quantity (Zhou et al., 2024). Recent work (Chong et al., 2022) investi- gated the effectiveness of leveraging pre-trained models to identify inherent label errors in natural language datasets. Additionally, efforts have been made to mitigate label errors in LLM alignment datasets (Zhu et al., 2024), particularly in the context of binary harmlessness classification. Fur- thermore, Pang et al. (2024b) systematically analyzed error patterns in LLM- LLM generated quality rating scores to reduce score errors. Another line of research has focused on devel- oping noise-tolerant DPO-based loss functions, including cDPO (Mitchell), robust-DPO (Chowdhury et al., 2024), and PerpCorrect (Kong et al.). The above studies primarily focus on noisy labels at the sample level. In contrast, our work explores fine-grained, token-level noisy labels to iden- tify and filter out uninformative tokens, thereby boosting downstream task performance.
Our main contributions can be summarized as follows.
Generic Token Cleaning Pipeline. We formulate the prob- lem from the perspective of noisy labels and present a novel influence-guides token cleaning pipeline that scores and filters out uninformative tokens, enhancing task per- formance by focusing model training on the most relevant tokens. The pipeline not only encompasses existing ap- proaches but also inspires new implementations.
Self-Envolving Cleaning. Beyond merely calculating the influence scores with a pair of fixed models, we propose to update the reference model iteratively, which could progressively enhance the quality of supervision signals, leading to better downstream performance.
Analytical Framework. We provide rigorous analyses to show when and why SFT with the cleaned token outper- forms the full tokens. Specifically, we establish upper bound on the error incurred when learning with full to- kens (Theorem 5.1), offering theoretical insights into the trade-offs of different cleaning strategies. Our analysis explains why fixed-model cleaning yields stable but lim- ited improvements, while self-evolving cleaning shows greater potential but requires careful implementation.
Comprehensive Experiments. We conduct extensive ex- periments across multiple tasks, demonstrating that our token cleaning pipeline consistently boosts performance over baselines and validates its practical merits.
3. Preliminary
3.1. Next-Token Prediction
2. Related Work
Consider a data pool comprising N samples, denoted as {a;)/ Each sample x; represents a sequence of to- kens (including the prompt and the response) defined as x; a= ( where I, denotes the token length for the
LLM Data Selection In the LLM SFT phase, various met- rics have been introduced to assess data quality including
2

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Pine-Tuning
Dataset
Dataset
Split
Base Model
Do
Dr
Token Scores
Reference Model
Base Model
1-st Reference
2-nd Reference Model
3-rd Reference Model
(Warmup) Model
Model
Self-Evolving Reference Models
Token Scores =
Base Loss Reference Loss
Token Scores for D: Base Loss t-th Reference Loss
Fixed-Model Cleaning
Self-Evolving Cleaning
Figure 1: Overview of the token cleaning pipeline. Fixed-Model Cleaning applies one-shot cleaning process to the entire dataset D. In contrast, Self-Evolving Cleaning follows an iterative approach. It begins with warm-up phase, where a model fine-tuned on the full tokens of split-0, denoted as Do, and then used to clean the next data split, transforming Di into Dj. The reference model subsequently updated by fine-tuning the warm-up model (i.c., the first reference model) on Dj. This iterative process continues, progressively refining the reference model with each newly cleaned data split.
i-th sample. The training LLMs can be framed as mini- mizing the negative log- ligeliill of the observed tokens in the dataset. The model predicts the conditional proba- bility P(zrylæt.yj 0) for each token 2ú, given its preced- ing context, where 9 represents the model parameters, and Ti.y denotes the first j - tokens, i.e. {Ii,1, .Zis-1)- Denote by D = (Jraj Frsh IK.)). v(,,)) E $}, where S = {(i,))li E [N].] E [L]]] [N] (1,2, .N.. The loss function for the dataset be expressed as:
corresponding to the prompt part are ignored, they do represent the model's predictions. Therefore, for prompt tokens, Di.j 0, and for response tokens, Dt. 1.
4. Token Cleaning: A Noisy Label Perspective
4.1. Intuition
In the phase of SFT, some tokens are deemed uninformative or noisy since most of the knowledge has been obtained in the pretraining phase, e-g., common patterns and structures, sigh-freuency phrases. In practice, the SFT phase assigns label of 1 to every token in the response, resulting in noisy token labels, where irrelevant tokens are incorrectly labeled as important (ūt.; = 1). Such noise can hinder the model's optimization process by introducing misleading gradients, reducing the signal-to-noise ratio by hiding informative tokens, and potentially leading suboptimal performance.
LD(() Eisees Vi. (JJ)S
Σ
Visl(Fis (ai,s;)) (1)
where f( (i.g: == log P(,;,) (..gi 0), and Vi.j E (0, 1) is binary (ground-truth) label indicating whether the token æ4, is valid target or not. By iteratively updating 6, the model learns to assign higher probabilities to the correct tokens while disregarding irrelevant ones.
To address this issue, it is essential perform fine- finiinn token label cleaning. This involves identifying and filter- ing out uninformative tokens while preserving the tokens that carry valuable task-specific information. As suggested by the noisy label literature (Zhu et al., 2022; 2024), the cleaning process typically involves two key components: a scoring function assess the quality of each token and threshold to distinguish between informative and uninfor- mative tokens, which will be detailed in the next subsection.
3.2. Token-Level Labels
Token-level labels Yi.j E {0,1) }} play a crucial role in de- termining which tokens contribute to the loss calculation. However, its ground-truth value often unknown. Denote Ji.y by the (noisy) token label that we in practice, which may or may not be identical to Yi.- During different training phases, the criteria for setting Vi.j may vary:
Model Pretraining: When training on general text data without explicit distinction between prompts and responses, all tokens are typically considered valid targets (ūa, = 1), unless specific tokens are identified as irrelevant or redundant (Lin et al., 2024).
4.2. Token Cleaning Pipeline
In this section, we will introduce the main components of the token-cleaning pipeline, including the scoring function and a simple yet effective threshold.
Supervised Fine-tuning (SFT): In this phase, the tokens
3

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
4.2.1. SCORE FUNCTIONS: AN INFLUENCE-GUIDED APPROACH
labels is unclear, which is beyond the scope of our paper and left for future explorations. In this paper, use fixed ratio (i.e., selected token proportion) k% to separate between informative and uninformative tokens. Denote by ūhs the token label after cleaning. We have
We deliver our intuition when designing score functions using the following example. Suppose that the model is improved from 0 to 0' by fine-tuning it on some data. Ac- cording to Koh & Liang (2017); Pang et al. (2024a), the model update influences the prediction accuracy of each token, which can be written as
Mi.i = { if (atole ranks top 1%, Vi, j; otherwise.
(4)
Infl (ri p 0,00 8²) == E(zi,) Dr 0) ((zi, ( 0).
4.3. Selection of l and #'
(2) Intuitively, a more negative Infl (mtyy / 0, 0*) indicates a higher confidence improvement on predicting zi given Fi.z- The equation can be explained from two perspectives:
We discuss two feasible strategies for selecting 0 and 0'.
4.3.1. FIXED-MODER CLEANING
Following the analyses in Section 4.2.1, we can assume the access to model I' that outperforms 6. For example, a moderately performing Llama model can be considered O, while well-performing Llama model can be considered as O' (Mindermann et al., 2022: Lin et al., 2024). Specifically, given the warm-up model 6' and the base model 6, we compute the token scores for the entire dataset D according to Eq. (3) and use a fixed threshold kixsed to assign token labels jhey according to Eq. (4). Note that token cleaning is performed globally, meaning that some samples may be entirely removed if they contain no positive tokens. This differs from Lin et al. (2024), where each sample retains fixed proportion of positive tokens. The benefits and limitations of this strategy will be discussed theoretically in Section 5.2.
Assume Token Quality: As demonstrated by Pang et al. (2024a), if we believe that token Zi,j is the best choice given context Ta,y, the above influence can be used evaluate the quality of data that brings the model from I to 8' on this specific task, i.e, more negative Infl (Rig)ais)) 0, 0*) indicates a higher data quality.
From another perspective, if we believe the model 6' performs better on I on this specific task, the above influence can be used to evaluate the quality of token ziij since a good and underfitted choice of Fi tends to have negative Infl (I% ( 0, ).).
Assume Model Quality:
In this paper, we assume the model quality and use the negative of the influence defined in Eq. (2) to evaluate the quality of tokens, i.e.,
4.3.2. EVOLVIVG CLEANING
Score(z;,, ( 8') = IInfl (Ia.s (...y 0, 8),
(3)
Inspired by the success of semi-supervised learning (SSL), we propose do token cleaning iteratively. Specifically, in the 1-th iteration, we fix the base model 0 and adopt 0' = O., then fine-tune 6 with the selected tokens after cleaning. See Algorithm for more details.
where a higher score indicates a higher token quality.
Extending from the current use of influences (Koh & Liang, 2017; Pang et al., 2024a), we notice that 0 and 0' are not necessarily to be the same model structure, as long as they share the same tokenizer. We will discuss potential choices of 8 and 6' in Section 4.3.
The overall procedure is outlined in Algorithm 1. First, we evenly partition the dataset D into series of subsets, denoted as (Do, Di, ..... Dr}. Next, the base model is fine-tuned on the initial subset Do to produce a warm-up model fg. which serves as the initial reference model. Rather than relying solely on 0g as fixed reference model, self-evolving mechanism is introduced in Lines 4-8. Specifically, for each subsequent subset D,, we keep the base model fixed and utilize the latest updated model as the reference model, i.e., 0 = On and 0' = O., to compute token scores. By applying a threshold Aistf--ooll to these scores, we obtain the cleaned labels Ør.3- The up- dated model 0, is then fine-tuned on the cleaned subset D., producing the reference model for the next iteration. This process continues iteratively, and the final reference model is used as the output of the algorithm.
Algorithm Details
4.2.2. THRESHOLD
After computing token scores, a threshold is directly applied to filter out uninformative tokens. The threshold separates the tokens that significantly improve the model performance from those that do not. An ideal approach is to use algo- rithms to estimate the ratio of the corrupted and then select the informative tokens according to the ratio. However, although there are lots of trials in the literature on noisy labels, most works focus on cleaning the image labels (Lad & Mueller, 2023) or sample-level text labels (Zhu et al., 2024; Pang et al., 2024b). To the best of our knowledge, a feasible algorithm for estimating the noise ratio of token
4

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
Algorithm 1 Token Cleaning Pipeline
Theorem 5.1 (Error of learning with full tokens). With probability at least 1 --, the generalization error of learning with full tokens is upper-bounded by
1: Input: Entire dataset D, base model fo. threshold
Rvelf-evol
2: Split dataset D into series of subset (Do.... .D)) Denote their indices by .... ,
2 2log(4/6) M
LD) S
m(D)
(5)
3: Warmup Model 0;: Finetune from base model 0o on
Datt quality
Do subset with all tokens.
Data quanaitt
4: for in 21. . do
where M Ei= Lj denotes the number of tokens.
5: Compute scores for subset D.'s tokens via Sc. ( Vo.,, E Dr.
Theorem 5.1 shows that the error of learning with full tokens depends on two factors:
6:
Assign token labels ūi, with a threshold Kaelf-cvol.
7: Obtain 0(+1 by finetuning 0g on cleaned subset D
Data Quality: ryD) denotes the noise rates of learning with full tokens, where a higher noise rate leads to a larger eτror, i.e., a worse performance. The negative impact of wrong token labels may not be canceled by increasing the number of tokens.
((ra.) Bras (..), (( E S.}.
8: end for
9: Output: 0T+1
When the number of tokens M increases, the generalization could be smaller, showing that the token cleaning result cannot merely return a small set of high-quality tokens, i.e., the precision and recall of the token cleaning algorithm are both important.
Data Quantity:
5. Theoretical Analyses
Let [.}} be the indicator function taking value when the specified condition is satisfied and otherwise. Define the 0-1 loss as ((0((X(ox)) X := [{0((prev)) # Xnea). where X Xexx is the random variable for the next token, X prev is the random variable for tokens before the next token, and 0( X prev) stands for the prediction of next token for model 0 given Xprev as input. Without loss of generality, we consider the ideal case where all the training instances for next-token prediction are i.i.d. and minimize 0-1 loss in the following analyses. The loss can be generalized to bounded loss E()) and finite function space FF following the generalization bounds that can be introduced using Rademacher complex- ity (Bartlett& & Mendelson, 2002).
Denote by y the random variable of token labels after clean- ing. We show the superiority of token cleaning compared to the full-token case in the following corollary.
Corollary 5.2. With probability as least 1 - 26, token clean- ing performs better than full-tokens in terms of the error upper bound when
I VI (V--1). (6)
n(D) n 22log(4/8)
where 9) : P(Ý + Y) denotes the noise rates of cleaned labels and == P(Ý 1) denotes the ratio of positive lokens after token cleaning.
5.1. Exceed the Performance of Full Tokens
Denote by D == {(raj) Li.j; (.,), Ji the full-token dataset. By minimizing the noisy loss
Corollary 5.2 shows token cleaning is preferred when the positive impact of reducing noise rate outweighs the nega- tive impact of reducing the number of feasible tokens. For example, when M is larger larger dataset), the inequality in Corollary 5.2 is more likely to hold since the right-hand side is smaller.
1 N
(0) N Eラ= D2s5
Σ
Σ Wis (0(a),))) j=1
we can get model 05 = arg ming (0). Denote by Y. Y the random variables for Ii.j and the corresponding ground-truth label yi.j- The expected loss of training with full tokens can be denoted by
5.2. Fixed-Model Cleaning: Stable But Limited Improvement
(0) Y 1(((Xpre)) prev) Xmeui]
We now analyze the benefits and limitations based The- orem 5.1 and Corollary 5.2. By selecting an appropriate model 6', we can take a one-shot token cleaning on all the tokens in the candidate data pool. In this case:
where D is the distribution of D. Denote by
n(D) = P( Y)
Data Quality: The noise rate of cleaned tokens is fixed, i.e., the data quality term in Eq. (5) is fixed. By carefully selecting the threshold kixed, there exists a token cleaning result whose noise rate i) is less than 7y (D).
the noise rate of full tokens. Theorem 5.1 shows the error upper bound of learning with full tokens. See Appendix C for the proof.
5

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine- Fine-Tuning
Data Quantity:
With more tokens being cleaned, M is increasing. Then the total generalization error be consistently reduced.
suffers from instability due to the limited number of ef- fective tokens. According to Corollary 5.2, the model's error upper bound may not decrease consistently, leading to unstable convergence.
Therefore, under this strategy, as long the reference model O' is sufficiently good to reduce the noise rate from ry( Dy) to a lower rate i). the model's performance can be improved by fine-tuning with additional i..d. cleaned tokens, demonstrat- ing the advantage on stability. However, even as MM - oo, the total error does not go to zero due to imperfect data quality, showing the limitations on final performance.
From the above analyses, we know that self-evolving mod- els are more adaptive and aggressive compared to fixed models. The theoretical insights further highlight strategies for achieving better performance.
6. Experiments
5.3. Self-Evolving Cleaning: Potential Matthew Effect
6.1. Experiments Setup
For ease of presentation, we divide the data into three groups according to their task difficulty and number of ii.d clean tokens in the training dataset:
Data Pool We utilize a high-quality data pool with 50k sample size from five popular SFT datasets (300k in total): Flan v2 (Longpre et al., 2023), Open Assistant 1 (Köpf et al., 2024), Stanford Alpaca (Taori et al., 2023), Dolly (Databricks, 2023), and WizardLM (Xu et al., 2023). The data pool is constructed based on a new powerful data cu- ration pipeline proposed by (Pang et al., 2024b), which involves selecting data samples using quality rating scores generated by LLMs. More dataset statistical information including token length can be found in Appendix D. For the self-evolving cleaning strategy, we heuristically divide the data pool into five equally sized subsets (10k samples).
G1 (Rich Group): Characterized by lower noise rates after token cleaning and higher proportion of effective tokens. This group typically experiences significant performance gains during warmup (Line 3, Algorithm 1) and has a great number of relevant tokens.
G2 (Poor Group): Marked by higher noise rates after to- ken cleaning and fewer effective tokens. This group often exhibits limited or even degraded performance during warmup and has a scarce number of relevant tokens.
Ga (Intermediate Group): Falling between the rich and poor groups in terms of data quality and quantity. While it generally sees reasonable performance improvement during warmup, its convergence tends to be unstable due to limited number of effective tokens.
Base Models In this paper, we select three popular open- source LLMSs as our base models, including I.L.MA-3.2-3B. LLaMA-3.1-8D (Dubey et al., 2024) and Misvral-7B-v0.3 (Jiang et al., 2023). These base models will be fine-tuned using samples from our data pool.
Note that the definition of groups only applies to the theo- retical analyses, which does not mean need to explicitly know the group attribute of each data. In fact, it is challeng- ing to know this information. Theoretically, there are three observations during SFT.
Baselines There are several baselines for performance comparisons: 1) BASE denotes the used base model; 2) DS² (Pang et al., 2024b) fine-tunes base model on 10k selected high-quality samples (with full tokens) from the entire data pool (50k); 3) FULII TOKENS utilizes all tokens to fine-tune the base model; 4) UNIFORM RANDOM randomly selects 1% tokens from the 50k data pool without replacement; 5) Rho (Mindermann et al., 2022; Lin et al., 2024) directly computes the excess loss for all tokens between the base and reference model and then selects top-k% tokens. Recall that k is the pre-defined threshold for token cleaning.
Observation I: The rich get richer (G When n (D) and MM is sufficiently large, according to Corollary 5.2, fine-tuning on D2 benefits from lower token noise rates and a higher number of effective tokens, thereby reducing the error upper bound and resulting in a better model f2. With better reference model and similar number of effective tokens, the model in the next iteration can be further improved, i.e., the rich get richer.
Warmup We warmup by tnn-ttunig the base model on subset Do with full tokens, and make it the (initial) reference model for Rho, our FIXED-MODEL CLEANING, and SELF- EVOLVING CLEANING. The warmup model is equivalent to the DS² baseline (Pang et al., 2024b).
Observation 2: The poor get poorer (G2). When the model 0, underperforms compared to 0o. 7)1 increases significantly and may even exceed 0.3. According to Corollary 5.2, continued fine-tuning on tokens with such high noise rates can further degrade performance, exem- plifying a "the poor get poorer" effect.
Evaluation
To comprehensively evaluate the efficacy of token cleaning methods, we adopt seven OpenL.M Leader- board tasks, including MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2021), TydiQA (Clark et al., 2020), HellaSwag (Zellers et al., 2019), ARC-Challenge (Clark et al., 2018), BoolQ (Clark et al., 2019) and LogiQA. These
Observation 3: Unstable convergence (Ga). f is com- parable to 77(D) and f- M is only marginally sufficient, fine-tuning on D3 may yield moderate improvements but
6

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
Table 1:
Performance (original score x 100) comparison of different baselines on various benchmarks. We highlight the best
result in boldface and the second-best with underline. By default, the selected token proportion (i.e., threshold) is 0.6.
Model
TruthfulQA
TydiQA
LoqiQA
MMLU
HellaSwag
ARC-C
BoolQ
AVG
Base model:
LLaMA-3.2-3B
BASE
39.39
21.10
22.17
56.29
55.24
42.20
72.95
44.19
DS (10K)
43.35
41.20
24.96
56.93
55.64
44.62
74.80
48.79
FULL TOKENS (50K)
43.32
49.60
24.34
56.87
55.57
44.44
74.98
49.87
UNIFORM RANDOM (50Kx0.6)
43.79
47.00
23.41
56.96
55.37
44.44
75.05
49.43
RHO
45.57
53.60
26.05
57.10
55.16
45.39
77.36
51.46
MREEDMMODEL CLEANING
48.96
52.60
25.89
57.09
56.43
45,39
77.52
51.98
SELF-EVOLVING CLEANING
51.07
56.38
28.22
56.18
55.81
45.99
77.33
53.00
Base model:
LLaMA-3.1-8B
BASE
45.10
22.80
26.51
65.29
59.92
50.82
82.18
50.37
DS (10K)
49.57
45.80
27.44
65.77
60.37
53.49
83.26
55.10
FULL TOKENS (50K)
47.51
58.10
28.53
65.78
60.42
54.01
82.49
56.70
UNIFORM RANDOM (50Kx0.6)
48.68
56.60
27.29
65.81
60.40
54.09
83.11
56.57
RHO
54.63
61.90
28.99
65.74
62.14
54.78
81.66
58.55
SRED-MMDEEL CLEANING
56.02
62.38
28.22
65.71
61.92
55.12
82.67
58.90
SELV-EVOLVING CLEANING
59.58
63.58
26.05
65.07
62.67
54.87
82.49
59.20
Base model: Mistral-7B-vil
BASE
42.56
54.70
25.74
62.41
60.77
48.92
82.30
52.88
DS (10K)
44.24
55.70
25.27
62.50
61.10
50.39
83.45
53.85
FULL TOKENS (50K)
43.67
55.60
25.27
62.41
61.14
50.56
83.85
54.12
UNIFORM RANDOM (50Kx0.6)
43.82
55.70
24.81
62.47
61.20
50.04
83.76
53.64
RHO
43.92
54.50
25.43
62.12
61.35
$1.08
83.29
53.60
TIXEDMMDDE CLEANING
44.52
59.03
26.05
61.45
61.47
51.68
82.03
55.20
SELF-EVOLVING CLEANING
45.41
56.17
27.44
62.30
61.40
50.65
81.28
55,00
datasets are sufficiently diverse to thoroughly assess the fine-tuned model across various aspects, including factual accuracy, reasoning, and multilingual capability. The task performances are evaluated on the Im-eval-hareness, reposi- tory. More evaluation and training details can found in Appendix D.2.
the local-ranking method (RhO) yields lower average per- formance than the global-ranking method TIXED-MODEE CLEANING). e.g., 51.46 vs. 51.98 for LLaMA-3.2-- and 53.60 vs. 55.20 for Mistral-7B-vO.. demonstrating that global ranking leads to more stable performance improve- ments. One possible explanation that local ranking is constrained by the quality of individual samples. For exam- ple, in SFT, low-quality sample may not contain any useful tokens, while almost all tokens in a high-quality sample may be useful. Since local ranking removes the same proportion of tokens from both samples, it inevitably retains uninfor- mative tokens from low-quality samples while discarding informative ones from high-quality samples. This limitation can be mitigated through global token ranking, as employed in fixed-model cleaning approach.
6.2. Main Empirical Results
As shown Table 1, our proposed strategies consistently outperform baselines across three base models on seven evaluation benchmarks. Notably, compared to using full tokens, self-evolving cleaning has achieved the average performance improvement of 6.3% on the 3B model and 2.0 %4.4% % on the 7B/8B models.
Local Ranking vs. Global Ranking Compared to RHO (Lin et al., 2024), which ranks token scores locally within in- dividual samples and removes the same proportion of tokens per sample, our fixed- model cleaning method globally ranks token scores across the entire dataset. As shown in Table I,
Self- Self-Evolving Cleaning Follows the Matthew Effeet Ta- ble 2 presents the model's performance across different training iterations (checkpoints), illustrating three phenom- ena arising from the Matthew effect, as discussed in Sec- tion 5.3. Specifically, performance on TruthfulQA, TydiQA, and LogiQA steadily improves over iterations, representing Observation I: the rich get richer. In contrast, MMLJU
Phtpps: : / / wegithhbb... com EleutherAT/ emevall i on-ionnhha harnaas

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
Table 2: Performance results of self-evolving cleaning strategy over iterations (checkpoints) on seven benchmarks. Base
model: LLaMA-3.. 3 These performance results align with three observations arising from the Matthew effect.
Model
TruthfulQA
TydiQA
LoqiQA
MMLU
HellaSwag
ARC-C
BoolQ
REFERENCE- 1
45,46
50.05
27.44
57.31
56.10
45.91
76.87
REFERENCE-2
46.67
53.18
27.44
56.89
56.25
46.51
77.15
REFERENCE-3
48.91
54.36
28.22
56,43
56.13
46.43
77.36
REFERENCE-4
51.07
56.38
28.22
56.18
55.81
45.99
77.33
Table 3:
Performance comparison with a new reference model: LLaMA-3.1-80 1A-MA--.1-8B-Instruue IB-Intruct. Blue: A better reference model brings a higher performance improvement. Red: The counterpart.
52.9
52.9
53.0
52.6
52
5200
51.9
52.0
51
51
510
Performance
50-
Models
MMLU
BoolQ
TydiQA
ARC-C
48
LLAMA
51.77
RHO
$7.04
75.94
39.37
46.08
46
Fixed-Model Self-Evolving
FIXED MODEL CLEANING
56.96
76.37
39.17
46.08
REFERENCE: WARMUP
56.93
74.80
41.20
44.62
44
0.5
0.6
0.7
0.8
RiO
57.10
77.36
53.60
45.39
Selected taken proportion
FIXED BINED-MODEL CLEANING
57.09
77.52
52.60
45.39
Figure 2: Average performance results of two cleaning pipelines under different token proportions. Base model: LLaMA-3.2-3B.
in Table 3, more powerful reference model generally leads to greater performance improvements in datasets such Ty- diQA and ARC-C. However, some counterintuitive results emerge: in MMLU and BoolQ, despite the 8B reference model significantly outperforming the warmup model, it fails to yield further improvements through token cleaning, A possible explanation for this phenomenon the distribu- tion shift. we divide the evaluation task distribution into two parts: an in-distribution segment aligned with our data pool and an -iitr--diitiitttionn segment, LLaMA-3.1 1-BB- Instruct, while achieving high overall performance, may not necessarily surpass the warmup model in the in-distribution subset. Investigating this hypothesis and validating this as- sumption are promising directions for future research. More detailed results can be found in Appendix E.
which focuses on factual knowledge, exhibits slight per- formance decline, illustrating Observation 2: the poor gel poorer. Meanwhile, for the remaining tasks (HellaSwag, ARC-C, and BoolQ), performance improvements are ob- served but exhibit fluctuations, aligning with Observation 3: unstable convergence.
6.3. Ablation Study
Impact of Selected Token Proportion Here, we inves- tigate the impact of selected token proportion for our pipeline using a series token proportion values includ- ing {0.3, 0.4, 0.9). As shown Figure 2, the best results are achieved when the selected token proportion is approximately 50% to 70%. Beyond this range, the overall performance declines, which may be attributed to uninfor- mative tokens. One valuable empirical finding is that the performance gains in SFT tasks largely rely on a small num- ber of highly informative and clean tokens. This observation supports the prevailing view that data quality is more cru- cial than mere volume. Full results can be referred to in Appendix E (Table 7).
7. Conclusion
This work has demonstrated the effectiveness and impor- tance of token cleaning, introducing a generic token cleaning pipeline that removes uninformative tokens while preserv- ing task-relevant information. Our theoretical analysis has revealed strengths and limitations of two scoring strate- gies: fixed-model cleaning, which provides stability but limited improvements, and self-evolving cleaning, which has shown the potential for greater performance gains but requires more careful implementation. Empirically, we have found that filtering out approximately 30%-40% of tokens consistently enhances performance across both strategies, achieving an average 6.3% improvement over the full-token baseline the 3B model scale.
Impact of Reference Model reference model on performance, we run RHO and our fixed- model cleaning approach using LLaMA-3.1 1.-MA---.--B-IIsttuee as the new reference model and compare the results with our previous reference model 2DS2 as the warmup). As shown
To assess the impact of the
8

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
Acknowledgment
Clark, C.,, Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXh::190S.1004, 2019.
Pang and Qian were partially supported by NSF Grants 2322919, 2420632, 2426031, and 2426940.
Impact Statement
Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolacv, V., and Palomaki, J. Tydi qu A benchmark for peoomatiion-sekiiiee question answering in ty pologically di verse languages. Transactions the Association for Computational Linguistics, 8:454-470, 2020.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint /803v:/80.0054: 2018.
References
Bartlett, I.. and Mendelson, Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482 2002.
Databricks. Free dolly: Introducing the world's first truly open instruction-tuned 1lm, 2023.
Cao, Y., Kang. Y., Wang, C., and Sun, L. Instruction min- ing: Instruction data selection for tuning large language models. In First Conference on Language Modeling, 2024. URL https: pettps://ope.rree http:///ppeeeeeeeeeeeeee wwwpeeeweeee/rruueeeeeee id-wF 6k0aWjAu.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXhv:2407.21783, 2024.
Chen, J. and Mueller, J. Automated data curation for robust language model fine-tuning. arXiv preprint arXiv:2403.12776 2024.
Gou, Q. and Nguyen, C.-T. Mixed preference optimization: Reinforcement learning with data selection and better reference model. arXiv preprint arXiv:2403. 19443, 2024.
Chen, L., Li, S., Yan, J., Wang, H., Gunaratna, K., Yadav, V. Tang, Z., Srinivasan, V., Zhou, T., Huang, H., et al. Alpagasus: Training better alpaca with fewer data, arXiv preprint ar Xiv:2307.00801. 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint ND0:20099033000 2020.
Cheng, H., Zhu, Z., Li, X., Gong, Y., Sun, X., and Liu, Y. Learning with instance-dependent label noise: A sample sieve approach. In International Conference on Learning Representations, 2021. URL https: http////eeeeeeeeeeeeeeeeeeeeee.e net/ cest/forumiiie i 2
Hu, E. J., Shen, Y., Wallis, P., Allen-ll--ii Z., Li, Y., Wang, S., Wang, L.,, Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., al. Mistral 7b, arXiv preprint 10..0223 2023.
Chong, D., Hong, J.. and Manning, C. Detecting label errors by using pre- preitrained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9074-9091, 1, Abu Dhabi, United Arab Emirates, December 2022. Association for Compu- tational Linguistics. doi: 10. 10.18653w//20022emmppp 618. URL https: htt///aaaanhooooooo http:///aaaaaatooo.ooroo//////// emn momll-mminn in. 61.8.
Koh, P. W. and Liang. P. Understanding black-box predic- tions via influence functions. In International conference on machine learning, pp. 1885-1894. PMLR, 2017.
Kong, K., Xu, X., Wang, D., Zhang, J., and Kankanhalli, M. Perplexity- awarre correction for robust alignment with noisy preferences. In The Thirty-eighth Annual Confer- ence Neural Information Processing Systems.
Chowdhury, S. R., Kini, A., and Natarajan, N. Provably ro- bust dpo: Aligning language models with noisy feedback. arXiv preprint 24v:24033000000 2024.
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D., Stan- ley, O., Nagyfi, R., et al. Openassistant conversations- democratizing large language model aligmment. Advances Neural Information Processing Systems, 36, 2024.
Chung. H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction- cestruction-fieeuueed language models. Journal of Machine Learning Research, 25(70):1-53, 2024.

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tunine
Lad, V. and Mueller, J. Estimating label quality and errors in semantic segmentation data via any model. arXiv preprint vrXii:2307.050800 2023.
are learnable, worth learning, and not yet learnt. In In- ternational Conference on Machine Learning, pp. 15630- 15649, PMLR, 2022.
Li, M., Zhang. Y., Li, Z., Chen, J., Chen, L., Cheng, N., Wang, J., Zhou, T., and Xiao, J. From quantity to quality: Boosting Ilm performance with self- self-guided data selection for instruction tuning. arXiv preprint 2tti:2308.1222222. 2023.
Mitchell, E. A note dpo with noisy preferences and rela- tionship to ipo, 2023. URI. https://ericmitchell. ai/cdpo. pdf.
Natarajan, N., Dhillon, I. S., Ravikumar, P. K., and Tewari, A. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1204, 2013.
Li, Y., Yang, J., Song, Y.. Cao, L., Luo, J., and Li, L.-J. Learning from noisy labels with distillation. In Proceed- ings of the IEEE International Conference on Computer Vision, pp. 1910-1918, 2017.
Northcutt, C.,, Jiang, L., and Chuang, I. Confident learn- ing: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 10:1373-1411, 2021.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 2021.
Northcutt, C. G., Wu, T., and Chuang, I. L. Learning with confident examples: Rank pruning for robust classifi- cation with noisy labels. In Proceedings of the Thirty- Third Conference on Uncertainty in Artificial Intelligence, 177 AUAI Press, 2017. URL http: /uuai. hrg//aiz011/pprceeeee http://eeddiinspppppe////// pddf..
Lin, Z., Gou, Z., Gong. Y., Liu, X., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., Chen, W., et al. Not all tokens are what you need for pretraining. In The Thirty-eights Annual Conference on Neurul Information Processing Systems, 2024.
Pang, J., Wang, J., Zhu, Z., Yao, Y., Qian, C., and Liu, Y. Faimesss without harm: An influence-guided ac- tive sampling approach. In The Thiriy-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https: / net/ forum? id=YYJojVBCed.
Liu, W. Zeng, W., He, K., Jiang, Y., and He, What makes good data for alignment? comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312. 1...:2222..1.85.... 2023.
Liu, Y. and Wang, J. Can less be more? when wawpassiin----bblaaiinnnn label noise rates consid- ered beneficial. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 17467-17479. Curran Associates, Inc., 2021. URL https: /ptppeeeeeeegg... neurips.cc/paaee hctpsssecc/papeep2/////1/ 1Le00felessaf2eee333eeeeeeeee Paper.. pdf.
Pang. J., Wei, J., Shah, A. P., Zhu, Z., Wang, Y., Qian, C., Liu, Y., Bao, Y., and Wei, W. Improving data efficiency via curating llm-driven rating systems. arXiv preprint 2024b.
Reed, S., Lee, H., Anguelov, D.,. Szegedy, C., Erhan, D., and Rabinovich, A. Training deep neural networks on noisy la- bels with bootstrapping. arXiv preprini arXiv: 1412.6596 2014.
Taori, R. Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-followingg Ilama model. https: ///thub.ttt/u/// com/tatsu-lab) stanford_alpad 2023. GitHub repository.
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y. Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effec- tive instruction tuning. In International Conference on Machine Learning, pp. 22631 -26488. PMLR, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Lu, K., Yuan, H., Yuan, Z., Lin, R., Lin, J., Tan, C., Zhou, C., and Zhou, J. # instag: Instruction tagging for ana- lyzing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations, 2023.
Vahdat, A. Toward robustness against label noise in training deep discriminative neural networks. Advances in Neural Information Processing Systems, pp. 5596-5605, 2017.
Mindermann, S., Brauner, J. M., Razzak, M. T., Sharma, M., Kirsch, A., Xu, W., Höltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., et al. Prioritized training on points that
10

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
Veit, A., Alldrin, N., Chechik, G., Krasin, I., Gupta, A., and Belongie, S. Learning from noisy large-scale datasets with minimal supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 839-847 2017.
Zhu, Z., Dong, Z.., and Liu, Y. Detecting corrupted labels without training model to predict. In International Con- ference Machine Learning, pp. 27412-27427. PMLR, 2022.
Zhu, Z., Wang. J., Cheng, H., and Liu, Y. Unmasking and improving data credibility: study with datasets for training harmless language models. In The Twelfth International Conference on Learning Representations, 2024. URL / /oppnrevieeeee./////// net forum? Ad=6bcADg6688.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint ar Xiv:2221 10560, 2022.
Wang. Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K. R., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., and Hajishirzi, H. How far can camels go? exploring the state of instruction tuning open resources, 2023.
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. Less: Selecting influential data for targeted instruction tuning. arXiv preprint 20t.:2402.04333 2024.
Xu, C., Sun, Q.. Zheng. K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.. 12244, 2023.
Yuan, S., Feng. L., and Liu, T. Early stopping against label noise without validation data. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpp:://ppenreeeeeeeee id=CMFF2 http:///eeeeeeeeeee///eeeeee wwwpeeewwwwwwweffoooooeoeeeeeeoe
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: /905.0783.. 2019.
Zhang, B., Liu, Z., Cherry, C.. and Firat, O. When scal- ing meets Ilm finetuning: The effect of data, model and finetuning method. arXiv preprint 17Xiv:2402.17193, 2024.
Zhao, H., Andriushchenko, M., Croce, F., and Flammarion, N. Long is more for aligmment: A simple but tough-to- beat baseline for instruction fine-tuning. arXiv preprint arXiv:2402.0m 048333 2024.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Zhou, D.,. Wang. K., Gu, J., Peng, X., Lian, D.,, Zhang, Y., You, Y., and Feng, J. Dataset quantization. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 17205-17216, 2023.
Zhu, Z., Liu, T., and Liu, Y. A second-order approach to learning with destance-dependees label noise. In Proceed- ings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10113-10123, 2021.

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine- Tine-Tuning
Appendix
Organization of the Appendix
The Appendix is organized as follows.
Section discusses the potential limitations of this work.
Section B discusses the additional computational cost of our proposed piplines.
Section C provides full proof for Theorems shown in Section 5.
Section D illustrates the experimental details including data pool, evaluation benchmarks, and training details.
Section E demonstrates detailed omitted performance results and the visualization of token influence scores.
Section F provides several samples with tokens selected by the self-evolving cleaning pipeline.
A. Limitations
While the proposed token cleaning pipelines demonstrate competitive performance compared to other baselines, we acknowledge that there are still potential limitations:
Base Model Scale Our experiments are primarily conducted on a base model with a 3B-8B scale. It remains uncertain how well the pipeline would perform on larger-scale base models.
Data Pool Scale Due to cost considerations, our data pool is limited to 50k samples. The performance of the proposed pipeline on larger-scale data pool remains uncertain.
B. Computational Time and Cost Analysis
Computational Cost The computational costs associated with the token cleaning pipelines primarily consist of the training costs, akin to those of standard SFT, along with two types of additional inference costs. These additional costs stem from one base model and another reference model, both of which are used to calculate token-level influence scores. Compared to the Rho baseline (Lin et al., 2024), our two proposed piplines do not incur any additional inference costs since the total data size for inference remains unchanged. For the Naive Fixed-Model Cleaning, we perform a one-shot inference on all samples simultancously, mirroring the process used in Rho. For the Self-Evolving Cleaning pipeline, we simply segment the data pool into several partitions for independent inference using different reference models, i.e., 50k samples -> (10k, 10k, .... 10k) samples. Consequently, the inference cost for the Self-Evolving Cleaning pipeline is also equivalent that of Rho baseline.
Computational Time It's important to note that compared to standard SFT, token cleaning pipelines does not achieve computational speedup because the next-token prediction Σıy still relies on the context formed by previous tokens X,,; The current implementation simply masks out uninformative tokens to ignore their token loss, which is simple and compatible with all training paradigms. Existing token-level approaches, including RHO and our approaches, mainly focus on performance efficiency rather than data (token) training efficiency. In practice, the GPU memory usage of these methods remains the same as when using full tokens. Therefore, investigating and enhancing token training efficiency represents promising direction for future research, such as exploring ways to skip uninformative token memory occupation altogether.
C. Proof for Theorem 5.1
We first reproduce the definitions as follows.
Denote by D {(r, Di Dr.)), Vi the full-token dataset. By minimizing the noisy loss
(0) Eiiyes Wios (i.j)s
Σ
0r, ((0(a).)))
where S ((i,))li [N].j E [T.]]. [N]: (1,2, NN. we can get model 07 == arg ming (0). When train with
12

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine- Tine-Tuning
full tokens, have Vi. 1, Vi,j. The corresponding expected loss can be denoted by
= = ( 10001(0Xppp))
((((Xpor)) pow)
Xnex) X E [1 poev) Xnexa)
where D the distribution D. Denote by Y the random variable for noisy token label Øt.j. and Y the random variable for ground-truth token label Vi.g- Accordingly, with ground-truth token labels, the expected loss
(D( E
E ((0 X peev), X
Decomposition With the above definitions, the generalization of model Op the clean distribution could be decomposed
CD(0g) = -Col0p)-(o(00))) + (0)
Term-
Term-2
where Term-1 transforms the evaluation of 0 from clean distribution D to the noisy distribution D. Term-2 is the generalization error but the model is trained and evaluated on noisy distribution D. Both terms are analyzed as follows.
C.1. Term-1 Upper Bound
For certain model 0, there always exist a random variable Neex such that when Y = 1, X = Xnext and when Y 0,
E Ex ((((Xpee)) Xmex)) E ExIr= ((( Xmexi]
Then we have
E Y- 1(0(X prrcx), Xnext)
P(YY =1) -x (1 (0 prev), Xnex)]
+ P(Y =0)- Exxy [(( ((o0))))) Xnex))
-P(Y = Ex (1 Xprew) X Xmex))] + P(Y = 0) Exy= [1((((pee))) X Xmeai)
=Exiy=1 [1((0(Xpov)) Xnex))
Y 1(0( X ppew)) Xnex)]
E|
13

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine- Tine-Tuning
(0) (8(
- -/x BDM prov) o Xnex -Y PPO((X ) + Xmeo/ P(X) dx
/ Y (-P(((XXpe)) prer) + x -P((O( prev) Xnex| PPX) dx
= / (P(O(X)LV) o Xnexl X) - PPO((X pocv) # X P(X) dx
= J, (P(O(Xpre)) = Neea x) - (0 X poes) Xneu(X)) P(X) ax
/ (PrOCX xx P(O(( Xnoxl [x]
p(r( # XX PP(O(Xprev) + Xmoulx3| P(X) ax
(0)
previ TD (D(Xnex Xprevi f)ev; (X prev: (X) dx
X
(b)
1
PP(xnext k|X)- PPX x/x) P(X) dx
J kE[K]
=P(Y- = 1) 1) / Σ
P(X = K|X = PeN = k|X, P(XYx = =1) dx
kE[R]
+ +P(Y 0). 2 /
1
Σ
PP = k|X, 0) = P(X 0) dx
kE(K]
<P(Y 0)
=P(( E
Dn)
where in equality (a), given model 0 and previous tokens, we can treat X (X Xrevvi 0) as Bernoulli random variable such that
P(xnexul (Xrevi 0) = +) = (P(O((prev)) prev) Xnexc X) and P( (X pxovvi 0) = -) (P(((Xpro) # Xnexa X).
Then according the definition of total variation of two distributions, i.e.,
1 d.P. dQ ldu, f TD(PIIQ) == 2 du du
we can summarize the integrand as the total variation between Xnex X mevi 0) and X (X previ 0).
Inequality (b) holds due to the data processing inequality since the probabilities [IP [P((xnext = 0(X )), P( P(Xnex + 0(X prev generated by (Px next i), Vi [K]] and the probabilities [PP(Xnext next ((X prxee))) (P # 0 X prex))] are generated by [P(Xnext k). Vk [K]].
The results also hold for an arbitrary Y. To prove it, we need to split the loss into two part: 1) and 2) Y 0. The first part follows the previous proof, yields upper bound of PPY 1,Y 0). The second part leads to an upper bound of PPY = 0, Y = 1). By summing up two bounds, we still have the same result, i.e., an upper bound of P(Y + Y).
14

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine- Tine-Tnning
C.2. Term-2 Upper Bound
Recall that
(0) Esssess Vis (KJ)ES
E ( Passs: 0).
Let 0 denote the model trained by minimizing 0-1 loss with noisy dataset D, i.e.,
0 = arg min (0).
Recall that the expected error of model 0 distribution is is
(5 - 1(0((Xppa)))) Xmex)
The optimal classifier is denoted by
05 arg min (0).
Denote by M Em= L the number of tockns. With probability at least - 6, we have:
()(0p) ()((p))
= -p((0)- (0())) +
+ (C2(0)))Cx((0)))
2 50 (2(()) c=(00)++1 (0(0))
2 (49()) n/
M
where inequality (a) holds since 1) (0p) achieves the minimum empirical risk according to its definition, thus (0,) (0p)) < 0; 2) each of the following two terms will be no greater than the corresponding gap (00(0))-5(0)) Specifically, inequality (b) holds due the Hoeffding's inequality, i.e., given any classifier Öz. and 0zg, with probability at least 8/2. we have the following bounds independently:
- -c)())) log(4/8)
log(4/8)
(0 (5 S
2M
2.M
By the union bound, have inequality (b) with probability at least 1 - 5.
Supposing unique next token for given previous tokens in distribution D, we have (-((7) O.
D. Experimental Details
D.1. Data Pool
The data pool used this work primarily composed of five widely used SFT datasets, derived either from human annotations or generated by advanced LLMs. Further details on these datasets can be found in Table 4. Notably, these datasets differ in format, quality, prompt length, and target tasks, highlighting the diversity of our foundational data pool. For consistency, adopt the "Tulu" template format proposed by Wang et al. (2023) to standardize these datasets. The "Tulu" template includes two primary tags, < User and < Ass ittant >> which denote the roles of the user and the assistant, respectively.
D.2. Evaluation Benchmarks
For MMLU, TruthfulQA, LogiQA, ARC-C, BoolQ, and HellaSwag datasets, we directly use accuracy as the performance metric. In particular, for the TruthfulQA dataset (a multiple-choice question benchmark), we utilize the MC2 metric, which evaluates only the answer assigned the highest probability by the model (i.e., the model's most confident choice). For the TydiQA dataset, we consider using the 1-shot Fl score. One can double-check these performance metrics in the LM-RVALUATION HARNESS repository. By default, we use all benchmark samples to conduct evaluation.
15

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
Table 4: Statistical summary of our 50k data pool. The average prompi length and the average response length are measured
based on LLaMA-3.. 2-3B.
Datasets
Data source
Data quality
# Data size
Prompt_Len
Completion Overall
OPEN-ASSISTANE 1
generatte
Both
2418
45.4
364.1
409.5
STANFORD ALPACA
Generated w/ Davinci-003
Normal
4598
22.5
84.8
107.2
WIZARDLM
ChalGPT-generatee
High
34772
126.1
419.23
545.4
DOLLY
Human- generated
Human-generated
Normal
1567
119.1
133.7
252.9
FLAN V2
numan-generatte
Normal
6645
475.1
38.6
513.7
D.3. Training Details
Following the experimental setup (Wang et al., 2023), we apply the LoRA technique (Hu et al., 2022) with a rank-size of 64 and scaling factor of 16. The overall batch size is 48, with the learning at le-4 well as training epoch. By default, the maximum input length is 2048. All experiments are conducted eight NVIDIA L40S GPUs. Fine-tuning 7B 8B models on dataset of 50k samples typically takes approximately hours.
E. Additional Experimental Results
E.1. Empirical Results Still Follows the Observation of the Matthew Effort
Here, we also provide the empirical results of the self-evolving cleaning strategy on the other two base models. As shown in Table the performance results over iterations also align with the theoretical observation, discussed in Section 5.3.
Table 5: Performance results of self-evolving cleaning pipeline over iterations (checkpoints) on seven benchmarks. Base models: LLaMA-3.. 1-BB and Mistral-7B--a- 3. These performance results still align with three observations arising from the Matthew effect.
Model
TruthfulQA
TydiQA
LoqiQA
MMLU
HellaSwag
ARC-C
BoolQ
Base model: LLaMA-3.1-8B
REFERENCE-1
53.45
58.62
28.68
65.65
61.58
55.56
82.98
REFERENCE-2
56.77
63.23
27.91
65.40
62.29
56.16
82.64
REFERENCE-3
59.01
65.27
26.05
65.18
62.56
55.81
82.74
REFERENCE-4
59.58
63.58
26.05
65.07
62.67
54.87
82.49
Base model: Mistral-7B-v0.33
REFERENCE-1
44.36
55.79
26.20
62.31
61.32
51.34
84.03
REFERENCE-2
44.78
56.04
27.13
62,41
61.37
51.25
83.82
REFERENCE-3
44.93
55.69
27.44
62.32
61.30
50.82
80.36
REFERENCE-4
45.41
56.17
27.44
62.30
61.40
50.65
81.28
E.2. Impact of Selected Token Proportion
This section provides experimental results to explicitly demonstrate the impact of the selected token proportion across iterations. Here, Table and Table 7 show the full performance results of the fixed-model cleaning and self-evolving cleaning pipeline, respectively.
E.3. Impact of Reference Model
Here, we provide the full performance results under different reference models, as shown in Table 8. Here, we select LLaMA-3, Pttpps: htttps://huggingeee coee...o/ettt ama /.le//laane amlam 3. -88B Inst.rct.
our reference model.
16

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
Table 6: Performance comparison of self-evolving cleaning pipeline over different iterations on various benchmarks under different selected token proportions.
Model
Token Prop
TruthfulQA
TydiQA
LoqiQA
MMLU
HellaSwag
ARC-C
BoolQ
Base Model:
LLaMA-3.2-3B
REFERENCE-1
45.68
49.36
27.60
57.23
56.06
45.56
77.08
REFERENCE-2
46.84
50.10
28.22
56.53
56.01
47.11
77.18
0.3
REFERENCE-3
49.15
50.69
27.44
56.11
55.83
46.51
77.15
REFERENCE-4
51.57
52.96
26.51
55.42
55.55
46.51
77.24
REFERENCE-1
45.56
46.88
26.98
57.32
56.29
45.99
77.15
REFERENCE-2
46.95
51.73
29.30
56.82
56.34
46.94
77.18
0.4
REFERENCE-3
48.72
54.56
28.84
56.52
56.05
46.68
77.52
REFERENCE-4
50.69
55,20
27.91
56.27
55.77
46.86
77.36
REFERENCE-1
45.47
47.49
27.13
57.18
56.22
45.82
76.99
REFERENCE-2
46.93
53.02
27.44
57.04
56.45
46.68
77.30
0.5
REFERENCE-3
48.73
54.65
28.37
56.55
56.10
46.94
77.33
REFERENCE-4
50.42
55.19
28.37
56.25
55.83
46.77
77.12
REFERENCE-1
45.46
50.05
27.44
57.31
56.10
45.91
76.87
REFERENCE-2
46.67
53.18
27.44
56.89
56.25
46.51
77.15
0.6
REFERENCE-3
48.91
54.36
28.22
56.43
56.13
46.43
77.36
REFERENCE-4
51.07
56.38
28.22
56.18
55.81
45.99
77.33
REFERENCE-1
45.34
50.45
27.60
57.42
56.09
45.65
76.90
REFERENCE-2
46.82
52.02
26.98
56.86
56.34
46.43
77.30
0.7
REFERENCE-3
48.48
55.27
27.91
56.38
56.18
46.77
77.21
REFERENCE-4
50.66
55.47
27.91
56.10
55.93
46.43
77.36
REFERENCE-1
45.09
56.21
26.67
57.50
45,65
32.73
76.87
REFERENCE-2
46.25
56,39
26.67
57.21
46.08
32.93
77.08
0.8
REFERENCE-3
47.30
56.34
26.67
57.05
45.99
33.33
77.33
REFERENCE-4
48.40
56.23
26.82
56.89
45.91
33.33
77.18
REFERENCE-1
44.75
46.88
26.20
57.52
56.10
45.39
76.68
REFERENCE-2
45.06
50.47
26.20
57.27
56.42
46.08
76.96
0.9
REFERENCE-3
45.23
50.36
25.43
57.37
56.45
46.08
77.02
REFERENCE-4
46.36
51.68
26.98
57.25
56.34
45.65
76.90
E.4. Token Influence Score Visualization
We visualize the token influence scores from our Self-Evolving Cleaning approach using scatter plots, as shown in Figure 3. As training progresses across iterations, we observe that token points gradually spread out from the diagonal line y = where most tokens were initially concentrated. This trend indicates that the influence scores hecome increasingly diverse. Figures 3a through 3b reveal similar token influence score patterns, as both base models originate from the LLaMA series, despite differing in scale. Specifically, more informative tokens (highlighted in red) shift into upper-left region of the plot, where the reference model assigns low loss but the base model assigns higher loss. This pattern suggests that the Self-Evolving Cleaning strategy successfully amplifies the contrast in token-level influence scores across iterations, thereby enabling more effective separation between informative and uninformative tokens.
F. Illustrative Examples with Selected Tokens
Intuitively, those common tokens are less likely to selected by our proposed strategies. Here, we provide several samples with tokens selected by the proposed self-evolving cleaning strategy. The selected (i.e., clean) tokens are highlighted blue.
17

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine- Tine-Tuning
Iteration-1
Iteration-2
Ieeraionn3
Iteration-4
20.
Base model loee
31.5
17
300
12.
304
300
23
12
13-
75-
14
54
50
Uninformativa tokens
Informative
25
th5 be 3/s Reference model loss
on
Mis Reference model loss
(6a 1/5
202
160
19.9
115
Reference model loss
Reference model loss
(a) LLaMA-3.2-3B
Iteration-1
Iteration-2
Iteration-3
Iteration-4
175
37.
17.
Base model lase
157
13.5
125
324
125
120
384
100
254
12
75-
75-
54-
52
50-
Uninformative tokens
25
23
tokens
ni s/s Reference model loss
aia
25 1/5
be 12
Reference model loss
Reference model
Reference model loss
(b) LLaMA-3.1-8B
Iteration-1
Iteration-2
Iteration-3
Iteration-1
202
20.0
201
2000
-75
175
37.
275
Basee model inss
150
15.0
35.
13.5
125
325
125
120
324
25
15
25
75
52-
Uninformative tokens
25
tokens
da it Reference model loss
aiz
35 sis
nio 22 ula Reference model loss
Reference model
Reference
(c) Mistral-7B--0.3
Figure 3: Visualization of token loss (i.e., influence scores) across training iterations of the Self-Evolving Cleaning pipeline. We plot the cases for three base models. The minimum influence score value of informative tokens is approximately in the range [0.0003, 0.001]. As the training progresses, token points diverge from the y diagonal, indicating greater variability in influence scores. This allows Self-Evolving Cleaning to better differentiate between informative and uninformative tokens.
18

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
Table 7: Performance comparison of two token cleaning pipelines and RHO on various benchmarks under different selected token proportions. Token Prop denotes the selected token proportion.
Model
Token Prop
TruthfulQA
TydiQA
LoqiQA
MMLU
HellaSwag
ARC-C
BoolQ
AVG
Base Model: LLaMA-3.2-3B
FIXED MODEL CLEANNING
0.3
48.46
51.08
26.36
56.93
56.22
46.25
77.36
51.80
FIXED MODEL CLEANNING
0.4
47.97
51.77
26.36
57.12
56.39
46.25
77.24
51.90
MODEL CLEANNING
0.5
47.65
52.61
26.51
77.15
$1.90
FIXED MODEL CLEANNING
0.6
48.96
52.60
25.89
$7.09
56,43
45.39
77.52
51.98
FIXED MODEL CLEANNING
0.7
47.53
51.96
25.74
$7.06
56.55
45,99
77.15
51.70
FIXED MODEL CLEANNING
0.8
47.33
50.04
25.74
57.10
56.56
46.34
76.87
$1.40
FIXED CLEANNING
0.9
46.26
49.75
24.81
57.18
56.39
46.34
76.03
51.00
FIXED MODEL CLEANNING
1.0
43.32
49.60
24.34
56.87
55.57
44.44
74.98
49.87
SELF-EVOLVING CLEANING
0.3
$1.57
$2.96
26.51
55.42
55.55
46.51
77.24
$2.25
SELF-EVOLVING CLEANING
0.4
50.69
55.20
27.91
56.27
55.77
46.86
77.36
$2.87
SELF-BVOLVING CLEANING
0.5
50.42
55.19
28.37
56.25
55.83
46.77
77.12
52.85
SELF-EVOLVING CLEANING
0.6
51.07
56.38
28.22
56.18
55,81
45.99
77.33
53.0
SELN CLEANING
0.7
50.66
55.47
27.91
56.10
55.93
46.43
77.36
52.83
SELF-EVOLVING CLEANING
0.8
48.40
56.23
26.82
$6.89
45,91
33.33
77.18
49.25
SELF-EVOLVING CLEANING
0.9
46.36
51.68
26.98
$7.25
56.34
45.65
76.90
51.59
SELF-EVOLVING CLEANING
1.0
43.32
49.60
24.34
56.87
55.57
44.44
74.98
49.87
RнO
0.3
49.68
49.54
26.98
56,94
55.97
46.51
76.65
51.80
0.4
48.17
51.09
26.20
57.01
46.25
76.96
$1.70
RHO
0.5
48.04
53.31
26.67
$7.00
56,39
46.17
77.12
52.10
RHO
0.6
45.57
53.60
26.05
$7.10
55.16
45,39
77.36
$1.46
RHO
0.7
47.56
52.92
25.58
$7.08
56,42
46.08
76.28
$1.70
0.8
46.97
49.66
26.36
56.45
76.00
$1.30
RHO
0.9
46.19
50.16
25.12
$7.22
56.06
45.91
75.26
50.80
RHO
1.0
43.32
49.60
24.34
56.87
55.57
44.44
74.98
49.87
Sample
Write response that appropriately completes the request. ### Instruction: Does Palmeiras have Fifa club world cup?
User:
### Response:
Assistant: Despite many claims that Palmeiras
club world cup in 1951, the official list of the Fifa World Club winners starts in the year 2000 with Corinthians being the first champion of the tourna- ment and Palmeiras doesn't 't have won any Fifa World Cup tournament since.
Sample 2
User: Seven Wonders of the Underwater World. Answer:
Assistant: Here's a list of the Seven Wonders of the Underwater World: Palau- Belize Barrier Reef, Belize- Great Barrier Reef, Australia- Deep-sea hydrothermal vents (worldwide) Galápagos Islands, Ecuador- Lake Baikal, Russia- Northern Red Sea, bordered by Saudi Arabia and Yemen on the eastern shore, and Egypt, Sudan, Eritrea, and Djibouti on the western shore
19

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
Table 8: Performance results under different reference models. Base model: LLaMA-3.1 --.BBB... Additional reference model: 1-8--33.-8B--sttrrcee
Model
TruthfulQA
TydiQA
LoqiQA
MMLU
HellaSwag
ARC-C
BoolQ
AVG
BASE MODEL
39.39
21.10
22.17
56.29
55.24
42.20
72.95
44.19
LLAMA-SB-ING
54.12
21.63
31.47
68.18
59.0
$1.77
84.03
52.89
RHO (LLAMA)
44.06
39.37
25.58
57.04
55.35
46.08
75.94
49.1
FIXED-MODEL CLEANING (LLAMA)
44.27
39.17
24.96
56.96
55.22
46.08
76.37
49.0
WARMUP REFERENCE MODEL
43.35
41.20
24.96
56.93
55.64
44.62
74.80
48.79
RHO (WARMUP)
45.57
53.60
26.05
$7.10
55.16
45.39
77.36
51.46
FIXED-MODES CLEANING (WARMUP)
48.96
52.60
25.89
57.09
56.43
45.39
77.52
51.98
Sample 3
User: In restaurant kitchen, chef uses food processor to prepare sauce. The recipe calls for 2 cups of chopped tomatoes, cup of diced onions, and 1/2 cup of minced garlic. The food processor has maximum capacity of cups. If the chef wants to make enough sauce for 6 servings, how many times must the ingredients be processed in the food processor? Hint: Assume that the food processor can process the entire recipe at once.
The recipe requires total of 2 1 + 0.5 3.5 cups of ingredients. To make enough sauce for 6 serv- ings, the chef needs to prepare 6 3.5 21 cups of ingredients in total. Since the food processor can process up to cups of ingredients at a time, the chef needs to process the ingredients 21 / 5.25 times. However, since the chef cannot process ingredients fraction of time, they will need to process the ingredients times to make enough sauce for 6 servings.
Assistant:
Sample 4
User: What is the meaning of "real life"? How does one distinguish the real world from the imaginary? Can virtual reality experiences be considered real life? How do our senses interpret reality?
Assistant: The meaning of "real life" can be subjective and varies from person to person. Generally, it refers to experiences and interactions that occur in the physical world, opposed to in one's imagination or virtual real- ity. Distinguishing the real world from the imaginary can be difficult, as imagination and perceptions can be influ- enced by personal beliefs, emotions, and past experiences. However, in general, the real world is considered to be physical environment we inhabit
experience through our senses, whereas the imaginary world is one that ex- ists solely in our minds and not physically present. Virtual reality experiences can provide sense of im- mersion that feel very real, but they still created and controlled by technology, and not experienced in the physical world, As such, they are not typically considered "real life." Our senses play critical role in interpreting and experiencing reality. Our brains receive information from our eyes, ears, nose, skin, and other sensory receptors, and then process and interpret that information to create our experience of the world. How- ever, our senses are not perfect and can be influenced by factors such as fatigue, illness, or external stimuli, which can alter our perceptions of reality.
20

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tune Pine-Tuning
Sample 5
User: Develop a comprehensive metric that takes into consideration various factors such as user engagement, user feedback, user retention, user conversion rate, and user satisfaction to evaluate the effectiveness of a given phone application designed to connect people with mental health professionals. The metric should also take into account the different types of mental health conditions that the application caters to, such as depression, anxiety, bipolar disorder, and schizophrenia, and should provide separate scores for each condition. Additionally, the metric should consider the geographical location of the users and the availability of mental health professionals in their area. The evaluation should be done over period of at least six months to account for any seasonal variations. Finally, the metric should be presented in a format that is easily understandable by both technical and non-technical stakeholders, such as a dashboard with charts and graphs, and should also include recommendations for improving the application based on the evaluation results.
Assistant: The following is a comprehensive metric that takes into consideration various factors to evaluate the effec- tiveness of given phone application designed to connect people with mental health professionals: User Engage- ment: This factor measures the level of user interaction with the application. The following metrics can be used to evaluate user engagement: Number of downloads: The total number of downloads of the application over pe- weekly monthly basis. - Session Duration: The average time spent by users on the application during each session. Frequency of Use: The number of times users access the application in a week or month. 2. User Feedback: This factor measures the level of satisfaction of users with the application. The following metrics can be used to evaluate user feedback: App Rating: The average rating of the application on app stores such as Google Play Store and Apple App Store. User Reviews: The number and content of user reviews on app stores and other forums.- Net Promoter Score (NPS): metric that measures the likelihood of users recommend the application others. 3. User Retention: This fac- tor measures the ability of the application to retain users over period of time. The following metries can be used to evaluate user retention: Churn Rate: The percentage of users who stop using the application over a period of time. Retention Rate: The percentage of users who continue to use the application over period of time. 4. User Con- version Rate: This factor measures the ability of the application to convert users into paying customers. The fol- lowing metrics can be used to evaluate user conversion rate: Conversion Rate: The percentage of users who upgrade to a paid subscription or purchase additional services. - Average Revenue Per User (ARPU): The aver- age amount of revenue generated per user. 5. User Satisfaction: This factor measures the level of satisfaction of users with the mental health professionals they connect with through the application. The following met- rics can be used to evaluate user satisfaction: Patient Satisfaction Score (PSS); A metric that measures the satisfac- tion of patients with the mental health professionals they connect with through the application. - Mental Health Professional Satisfaction Score (MHPSS): A metric that measures the satisfaction of mental health profession- als with the application and the quality of patients they connect with. 6. Mental Health Condition: This factor takes into account the different types of mental health conditions that the application caters to, such as depression, anxiety, bipolar disorder, and schizophrenia. The following metries can be used to evaluate the effectiveness of the appli- cation for each condition: -oneition-specifie Engagement: The level of engagement of users with the applica- tion for each mental health condition. conndition-specifie Feedback: The level of satisfaction of users with the application for each mental health condition. Condition-specti
riod of six months. Active Users: The number of users who actively use the application
Retention: The ability of the application to re- tain users with each mental health condition over a period of time. -ondition-specific Conversion Rate: The abil- ity of the application to convert users with each mental health condition into paying customers. - Condition- specific Satisfaction: The level of satisfaction of users with the mental health professionals they connect with through the application for each mental health condition. 7. Geographical Location: This factor takes into account the geographical location of the users and the availability of mental health professionals in their area. The following met- rics can be used to evaluate the effectiveness of the application for each geographical location: Location Location-specifii En- gagement: The level of engagement of users with the application for each geographical location. - Location-specie Feedback: The level of satisfaction of users with the application for each geographical location. - Location-specifie Retention: The ability of the application to retain users in each geographical location over a period of time. - Loca- tion-specific Conversion Rate: The ability of the application to convert users in each geographical location into pay- ing customers. Location- specific Satisfaction: The level of satisfaction of users with the mental health professionals they connect with through the application in each geographical location. The evaluation should be done over a period of at least six months to account for any seasonal variations.
21

---

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuie Pine-Tuning
Sample 6
User: What's the furthest back we can prove human existence? Is there good reason to believe it goes further then this point?
Assistant: Human is an ever-evolving species. we starts from Homo sapiens the species, there are traces of human Africa since 300,000 years ago. For Homogenus, earliest fossil record discovered so far dated around 2.8 mil- lion years ago. The above dates are based on fossil records. It is completely possible for new fos- sil emerges with an earlier date. It is also possible that carbon dating, technique used determine fossil age, may carbon- 14 than we thought there would be in the past, would
be accurate. For example if there are underestimate the age of existing fossils, pushing human history further back.
User: How reliable is carbon dating exactly?
Assistant: There is no exact answer to how reliable carbon dating is. depends on many factors, such as the quality of the sample, the calibration of the equipment, and the assumptions made about the past. Generally speaking, car- bon dating can be accurate up about 50,000 years ago, but beyond that it becomes less reliable. The margin of er- ror can range from a few years to several thousand years, depending the situation.
22


Reviews:
[{"review_id": "b7PtDEcsy2", "reviewer": "Reviewer_W8kk", "summary": "Recent large language models are trained in multiple stages with vast amounts of data to achieve stellar performance. However such a training regime also brings in several challenges, including the challenge of observing similar high frequency phrases over and over again. This work aims to improve the supervised fine-tuning of large language models by performing token cleaning, masking out uninformative tokens while preserving the important ones. Through theoretical and experimental analyses, the work further aims to highlight the strengths of the proposed token cleaning pipeline.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank reviewer W8kk for the positive feedback! We will address individual comments below.\n> While the theoretical analyses is not fully precise, e.g in Sections 5.1 & 5.2 it makes a few trivial observations such as the challenges caused by the noise in tokens...\n\n**Response**: We agree that LLM learning dynamics and convergence are much more complicated than our analytical framework, but our analyses in Section 5, form a complete theoretical framework based on error upper bounds that provides valuable support for our claims.\nOur main claim is that the two proposed token cleaning methods can effectively improve LLM SFT performance. Our theoretical analyses support this claim in two important ways:\n\n **Explaining why the proposed methods work**: \n- Theorem 5.1 and Corollary 5.2 establish an analytical framework that precisely describes when token cleaning outperforms using all tokens through the trade-off between data quality (η) and data quantity (M);\n- Section 5.2 analyzes the characteristics of Fixed-Model Cleaning, explaining why this method provides stable but bounded improvements\n- Section 5.3 analyzes the dynamic behavior of Self-Evolving Cleaning through the Matthew Effect, explaining why this method can produce more significant improvements in certain scenarios\n\n **Provide practical guidance for method selection and application**:\n - Clearly identifying when Self-Evolving methods might lead to performance degradation (G2 group data)\n- Providing theoretical basis for selecting appropriate cleaning methods in different scenarios\n- These alerts and guidance are validated in our experiments, as seen in Table 2 where performance changes across different tasks align well with theoretical predictions\n\nWe agree that while some analyses provide foundational observations, our theoretical framework offers deep insights into method behaviors, supporting our claims and guiding practical applications. In the revised paper, we will clarify the links between theory and experiments to explicitly highlight the importance of the theoretical aspects.\n\n\n> However, I would like to note that the theoretical claims do not explicitly support the proposed approach...\n\n **Response**: We agree that our theoretical analyses primarily serve as intuitive explanations rather than explicit proofs for our proposed approaches, and they are indeed based on certain simplifying assumptions.\nWe acknowledge that the experimental results constitute the primary evidence supporting our claims about the effectiveness of our token cleaning methods. The theoretical analyses were intended to complement these empirical findings by:\n- Providing a conceptual framework to understand when and why token cleaning can be beneficial\n- Offering explanations for the observed performance patterns across different methods\n- Helping practitioners make informed decisions about method selection in various scenarios\n\nWe appreciate the reviewer's perspective that this does not diminish the overall support for our claims. In the revised manuscript, we will clarify the role of the theoretical analyses as providing intuitive explanations and insights, rather than positioning them as rigorous proofs of our methods' effectiveness. \n\n> How do you think token cleaning effects the supervised fine-tuning efficiency, both with respect to the data requirements and number of training iterations?\n\n**Response**: Token cleaning affects supervised fine-tuning efficiency in several important ways:\n- Regarding data efficiency, our approach selects only 60% of the most informative tokens rather than utilizing all tokens. This effectively reduces the data requirement by focusing the model's learning on tokens that contribute most significantly to performance. For our self-evolving cleaning method, we divide the total data samples into several chunks, each representing one iteration phase, which allows for progressive refinement of token selection.\n- Despite this data partitioning for self-evolving cleaning, the number of training iterations remains consistent with standard SFT. Therefore, token cleaning does not impact the efficiency of SFT in terms of training iterations.\n\nHowever, it's important to note that compared to standard SFT, token cleaning does not achieve computational speedup because the next-token prediction $x_{ij}$ still relies on the context formed by previous tokens $x_{i,:j}$. The current implementation simply masks out uninformative tokens to ignore their token loss, which is simple and compatible with all training paradigms.\nExisting token-level approaches, including RHO and our methods, mainly focus on performance efficiency rather than data (token) training efficiency. In practice, the GPU memory usage of these methods remains the same as when using full tokens. Investigating and enhancing token training efficiency represents a promising direction for future research, such as exploring ways to skip uninformative token memory occupation altogether.", "claims_and_evidence": "The paper is built on the intuition that during SFT, the model observe a reiteration of the previously learned concepts in addition to the new information that the SFT aims to distill. The authors formulate this challenge analogously with a noisy label scenario and take inspiration from that literature. Specifically, they utilize influence functions and a hard thresholding mechanism on top for _cleaning the tokens_ that are not very informative. The authors then claim that this approach should _improve the optimization process by preventing misleading gradients_.\n\nWith respect to this claim, the authors present theoretical analyses in addition to experimental results on several well-established large language models. While the theoretical analyses is not fully precise, e.g in Sections 5.1 & 5.2 it makes a few trivial observations such as the challenges caused by the noise in tokens, I believe that that the overall discussion and the examples used (especially the discussion in Section 5.3) are valuable for highlighting the motivations behind the work and support the claims.\n\nIn addition, as shown in Table 1, the proposed approach(es) mostly improves the performance of different well-established models on several benchmarks. Thus, the overall claims of the paper are well-supported.", "methods_and_evaluation_criteria": "- Benchmarks are well-known datasets in the field and are sufficient to support the claims and highlight the contributions of the work.\n- Utilized baseline models (namely two exemplars from the LLaMA-3 Herd and Mistral) are also reasonable and sufficient", "theoretical_claims": "I checked the theoretical claims and could not find any obvious issues. However, I would like to note that the theoretical claims do not explicitly support the proposed approach: They mainly provide a more intuitive explanation as to why the proposed approaches could ultimately work under several strong assumptions. I do not believe that this casts a shadow with respect to the support of the claims, but rather that the theoretical part is a more minor factor in doing so.", "experimental_designs_or_analyses": "The general experimental design follows the established standards and the analyses are sound based on the quantitative evidence.", "relation_to_broader_scientific_literature": "Supervised fine-tuning is a widely adopted approach in large language model training. The provided issue in the work, namely the reiteration of the same knowledge, is also a very real problem, especially considering the scale of the pretraining data these models go through. Based on this, the work proposes an interesting and well-founded approach to addressing this issue and providing a more efficient training. I believe that the work could be interesting to the broader community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Overall, the presented figures, tables and the algorithm are neatly explained. The narrative of the work is also fluent and was not very challenging to read through. I believe that this work overall is a strong and meticulous submission with interesting and well-supported ideas presented in it.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- How do you think token cleaning effects the supervised fine-tuning efficiency, both with respect to the data requirements and number of training iterations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v6BQf6PsTr", "reviewer": "Reviewer_dD3A", "summary": "This paper enhances the SFT process by scoring and filtering out uninformative tokens, treating them as noisy labels. They introduce a novel influence-guided token cleaning pipeline to address this issue. Furthermore, they offer rigorous analyses to demonstrate when and why SFT with cleaned tokens outperforms the use of the complete set of tokens.\n\n\n## update after rebuttal\n\nI still don't believe the performance improvement is due to masking out uninformative tokens. Even in the example provided by the authors, they use the phrase \"negatively impactful tokens\" to explain how it works. I don't think such negatively impactful tokens can be considered merely uninformative. I believe the author should distinguish between negatively impactful tokens and uninformative tokens, rather than treating them merely as generic error labels.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer dD3A for the time and effort. We will address individual comments below.\n\n> **W1: evaluation benchmarks**\n\n**Response**: The benchmarks we used are standard in SFT studies (see Related Work) and assess a broad spectrum of capabilities, including question answering, reasoning, and multilingual comprehension, not just classification. While some benchmarks use multiple-choice formats, they test the model's generative abilities and knowledge without converting it into a classification model.\n\nFor your concern, we use two popular LLM-Judge benchmarks, MT-Bench and Vicuna-bench, to evaluate our model generations (the higher numbers indicate the better results). These results highlight the superiority of our methods.\n| Method (llama-3b)  | MT-Bench | Vicuna-Bench |\n|-|-|-|\n| DS2| 5.62 | 4.46|\n| Full_token| 5.40 | 4.46|\n| Random| 5.32 | 4.80|\n| Rho|6.33| 5.65|\n| Fixed-model cleaning| 6.27| 5.74|\n| Self-evolving cleaning| **6.63**| **5.76** |\n\n> **W2: Additional baseline STAFF**\n\n**Response**: We replicated the STAFF baseline with pruning rates of 40%/60%. Due to the absence of a comparable smaller model for Mistral-7B, we assessed STAFF using two LLaMA base models, using LLaMA-3.2-1B to compute speculative scores. \nWhile STAFF remains a strong baseline, yet Self-Evolving Cleaning shows competitive results. For instance, with LLaMA-3.1-8B, our method surpasses STAFF in average performance (59.20 vs. 57.0/56.4). Although STAFF performs better on BoolQ, Self-Evolving Cleaning consistently improves across various tasks. In our revised manuscript, we will cite this reference and discuss. \n\n| Model | TruthfulQA| TydiQA | LogiQA | MMLU | HellaSwag | ARC-C | BoolQ | Average|\n|-|-|-|-|-|-|-|-|-|\n| STAFF (llama-3b, 20k)|44.1| 45.92| 23.72| 56.86| 55.6|43.5|76.13| 49.4|\n| STAFF (llama-3b, 30k) | 43.15| 48.31| 24.03| 56.78|55.43| 44.19|73.78| 49.4|\n|Self-Evolving Cleaning (llama-3b) | 51.07| 56.38|28.22| 56.18| 55.81|45.99 |77.33 |**53.0**|\n| STAFF（llama-8b, 20k) | 46.3| 61.34| 26.82 | 65.07| 60.35| 54.61| 84.57| 57.0|\n| STAFF（llama-8b, 30k) | 46.06| 61.98 |24.03| 63.57| 60.29 | 55.12| 83.57| 56.4|\n|Self-Evolving Cleaning (llama-8b) |59.58| 63.58| 26.05 |65.07 |62.67|54.87|82.49 |**59.2**|\n\n> **W3: the necessity of manually selecting uninformative tokens**\n\n**Response**:\nWe clarify the distinction between uninformative and noisy tokens as follows. In the SFT phase, every response token is labeled as 1, making it difficult to identify which tokens should be included for task-specific knowledge and which excluded for higher knowledge density. As a result, traditional labels are considered noisy, consisting of informative tokens (correctly labeled as 1) and uninformative tokens (incorrectly labeled as 1 but should be 0), with the latter corresponding to label errors.\n\nWe acknowledge that while uninformative tokens have low loss, their impact on model convergence cannot be ignored. These tokens may lead the model to make trivial, non-task-specific predictions. Our findings, supported by curriculum learning literature [1], show that while \"easy\" patterns are learned first, \"difficult\" patterns require more focus. Therefore, adjusting the weights of these tokens is crucial for optimizing downstream task performance.\n\n[1] A Survey on Curriculum Learning, TPAMI'21.\n\n> **Q1: Computational cost**\n\n**Response**: There are two types of additional inference costs in our method: one associated with the base model and the other with the reference model. Besides, there is also training cost for the reference model on a small subset (10k).  Compared to the baseline Rho-1, our approach does not incur any extra inference costs. This is because the Self-Evolving Cleaning pipeline merely divides the data pool into several partitions for independent inference, i.e. (50k samples -> 10k, 10k, …, 10k samples).\n\n> **Q2: Influence function**\n\n**Response**:\nThe standard influence form is defined as loss($\\theta$) - loss($\\theta’$), given that the $\\theta$ model is the current model and $\\theta’$ is the model by removing a token.\nHowever, in our scenario, we do not have to counter-factually remove a token to calculate the influence since the training procedures are naturally adding tokens, which also have a form of loss($\\theta$) - loss($\\theta’$) but $\\theta’$ can be seen as the current model (for example) and $\\theta$ can be seen as the model after adding one token. Specifically, Self-Evolving Cleaning iteratively use the influence function to identify informative tokens for the next-round training. Therefore, we can obtain those two models naturally with the original base model as $\\theta$ and the current model as $\\theta’$.\nTherefore, the only difference is we are adding tokens iteratively [1] while the standard approach is deleting tokens (re-training) [2].\n\n[1] Estimating Training Data Influence by Tracing Gradient Descent, NeurIPS’20.\n\n[2] Understanding black-box predictions via influence functions, ICML’17.", "claims_and_evidence": "I question the claim to regard the selection of tokens as a noisy-label problem.", "methods_and_evaluation_criteria": "The evaluation is relatively too easy. As far as I know, the datasets mentioned are all classification tasks instead of generation tasks. I hope the authors can correct me if my understanding is wrong.", "theoretical_claims": "yes", "experimental_designs_or_analyses": "The evaluation seems overly simplistic. Based on my understanding, the datasets referenced pertain solely to classification tasks rather than generation tasks. If my interpretation is incorrect, I would appreciate clarification from the authors.\n\nI suggest you compare with more baselines such as the sample-level data selection to show why the token-level one is necessary. For example, you can compare with the method in \"Speculative Coreset Selection for Task-Specific Fine-tuning.\"", "relation_to_broader_scientific_literature": "token-level understanding in SFT", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The main issue is that it's not immediately clear why uninformative tokens should be considered noisy. I recommend providing a demonstration to clarify why these tokens can be categorized as such. \nFrom my perspective, the loss associated with uninformative tokens tends to be minimal during training. In the Next-Token Prediction paradigm, the loss of uninformative tokens is naturally down-weighted due to its small value. Therefore, I don't understand the necessity of manually removing them.", "other_comments_or_suggestions": "I want to summarize my main concerns here:\n\n1. I'm unclear on the necessity of manually selecting uninformative tokens, as this process can be naturally handled by the loss function. The loss associated with uninformative tokens can decrease naturally, serving as a form of automatic reweighting. One possible explanation is that if the loss for these tokens becomes too low, it could negatively impact generalization. Therefore, it might be necessary to prevent further reduction in their loss by adjusting their weight.\n\n2. I find the experiments somewhat lacking, especially when considering the benchmarks and baselines used.", "questions_for_authors": "What is the cost of this method? I believe the cost is quite significant due to the additional inference required.\n\nAdditionally, I find the definition of the influence function somewhat unusual, though still acceptable. The primary goal should be to assess the influence of a token. I think a token's influence should be measured by the change in loss across the entire dataset before and after the token is removed. This approach differs significantly from your current definition.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zPaOEpfcXj", "reviewer": "Reviewer_wKJy", "summary": "The paper introduces a token cleaning method for supervised fine-tuning (SFT) of large language models that operates at a fine-grained, token level rather than discarding whole samples. It employs an influence-guided scoring function to assess each token’s contribution by comparing loss differences between a base model and a reference model. Tokens deemed uninformative or redundant are filtered out using a threshold, thereby preserving key task-relevant information. Two cleaning strategies are proposed: a fixed-model approach that performs one-shot cleaning and a self-evolving approach that iteratively updates the reference model. The method is supported by theoretical analysis and extensive experiments across various downstream tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank Reviewer wKJy for their positive feedback and comments. We will address individual comments below.\n\n> **Weakness 1: how accurate is using influence function to assess the quality of token itself**\n\n**Response**: Thank you for raising this important question. We'd like to clarify with the following points:\n- The mentioned work made the claim because of the violations of required theoretical assumptions (e.g., positive-definite Hessian matrix) for language models. These assumptions are proposed for approximation. However, in this work, the method involves directly working with the difference in loss values rather than making approximations or Taylor expansion. Thus, the accuracy of the influence function would not be affected.\n\t\nHere is a more detailed comparison between our implementation of the influence function and the standard one who needs approximations. The standard influence form is defined as  loss($\\theta$) - loss($\\theta’$), given that the $\\theta$ model is the current model and $\\theta’$ is the model by removing a token, which requires some approximations to avoid model re-training. However, in our scenario, we do not have to counter-factually remove a token to calculate the influence since the training procedures are naturally adding tokens, which also have a form of loss($\\theta$) - loss($\\theta’$) but $\\theta’$ can be seen as the current model (for example) and $\\theta$ can be seen as the model after adding one token. Specifically, in the self-evolving clearing pipeline, we are iteratively using the influence function to identify informative tokens for the next-round training. Therefore, we can obtain those two models naturally with the original base model as $\\theta$ and the current model as $\\theta’$.\nTherefore, the only difference is we are adding tokens actively [1-2] while the standard approach is deleting tokens (i.e., re-training) [3].\n\n- Besides, our theoretical analysis is based on a unified and analytical framework, which mainly explains the when and why the SFT with cleaned tokens outperforms with full tokens, regardless of the specific token quality evaluation metric used. As such, our theoretical insights are independent of the precise accuracy of the influence function, focusing instead on the overall reduction in token noise rates.\n\nFinally, to address the reviewer's concern, there are more insights here.\nIntuitively, we hope that the base model, denoted as $\\theta$, will perform similarly to the reference model, $ \\theta' $, on downstream tasks after fine-tuning. To approximate this criterion, we use the token loss calculated by the reference model, expressed as $\\\\ell(x_{i,j} | x_{i,:j}, \\theta') $. If the token loss calculated by the base model, $ \\\\ell(x_{i,j} | x_{i,:j}, \\theta) $, is significantly higher than that calculated by the reference model, it indicates that this particular token requires fine-tuning to better align the base model with the reference model. Relying solely on $ \\ell(x_{i,j} | \\boldsymbol{x}_{i,:j}, \\theta) $ as a metric for token quality—by ranking and selecting tokens based on this loss—would only consider token-level information and neglect the task-specific insights provided by the reference model.\n\n[1] Estimating Training Data Influence by Tracing Gradient Descent, NeurIPS’20.\n\n[2] Fairness without harm: an influence-guided active sampling approach, NeurIPS’24.\n\n[3] Understanding black-box predictions via influence functions, ICML’17.\n\n> **Weakness 2: larger-scale model performance evaluation**\n\n**Response**:\nThank you for your valuable feedback. Given our constraints with GPU resources and time, we utilized LLaMA-2-13B-hf as the base model to assess the efficacy of our proposed methods. All experimental settings are consistent with the original settings. The results demonstrate the superiority of our methods.\n| Method                    | truthfulqa | tydiqa | logiqa | mmlu | hellaswag | arc_c | boolq | Average |\n|---------------------------|----------------|--------|--------|------|-----------|---------------|-------|---------|\n| base (llama-2-13b-hf)     | 36.73          | 33.79  | 26.05  | 55.14| 60.11     | 47.80         | 80.64 | 48.6    |\n| ds2                       | 37.48          | 35.47  | 27.91  | 55.19| 60.37     | 48.49         | 81.16 | 49.4    |\n| full_token                | 41.73          | 38.29  | 27.60  | 55.33| 60.33     | 49.70         | 82.40 | 50.8    |\n| random                    | 41.67          | 37.44  | 27.60  | 55.30| 60.35     | 49.61         | 82.33 | 50.6    |\n| rho                       | 44.41          | 42.12  | 28.06  | 55.35| 61.33     | 50.99         | 81.87 | 52.0    |\n| fixed_model_cleaning      | 44.03          | 40.07  | 27.91  | 55.63| 61.32     | 50.90         | 81.62 | 51.6    |\n| self_evolving_cleaning    | 49.13          | 44.65  | 26.67  | 55.42| 62.23     | 51.16         | 82.49 | **53.1**    |", "claims_and_evidence": "The authors support their claims with theoretical analysis in proving training with cleaned token is lower bounded by training with full tokens under the assumption that influence function can be used to evaluate the quality of token. The proof seems sound under the assumption of model quality. However, I do have some concerns for this assumption, which is detailed in \"Other Strengths And Weaknesses.\" Empirical results in demonstrates self-evolving method is performing better than fixed model on Llama-3.2 and Llama 3.1 models. In general, I think the claims made in this paper are well justified.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria makes sense. The idea of token cleaning flows naturally from pretraining (Rho-1) to fine-tuning, and is an intuitive extension from sample-level data cleaning to token-level.", "theoretical_claims": "I did not check the proof step-by-step, but it looks promising.", "experimental_designs_or_analyses": "In general the proposed experiments are sound in terms of design and baseline choices.", "relation_to_broader_scientific_literature": "This work is a natural extension of Rho-1.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n1. This paper has strong theoretical analysis supported in token selection during SFT, which is a novel contribution.\n2. The paper is well written and easy to follow. \n\nWeakness:\n1. One concern I have is how accurate is using influence function to assess the quality of token itself. Previous work [1] indicates that influence functions might not work well on LMs, and \"can lead to strong biases in this estimator.\" If the influence function itself is biased, the soundness of the theoretical finding in this paper might be affected. While I understand that there are differences between theoretical assumptions and practical settings, I hope the authors can provide more insights and intuitive explanations in justifying using influence functions to score the token.\n2. The size of the model considered in the experiment section are 3B and 7B. I'm wondering how effective this method is when presenting to a larger and more powerful model. Will larger model more robust to the SFT noise or will it still benefit from token cleaning?\n\n[1]Causal estimation of memorisation profiles,", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Refers to weakness section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HOnjfZd3Ac", "reviewer": "Reviewer_fypD", "summary": "This paper introduces two token cleaning methods for LLM fine-tuning, Fixed-model cleaning and Self-evolving cleaning. The token cleaning method ignores the loss from \"unimportant\" tokens, thereby improving task performance. The authors also provide theoretical analyses and extensive experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank Reviewer fypD  for their positive feedback and comments. We will address individual comments below.\n\n\n> **Weakness 1: the effectiveness of this approach**\n\n**Response**: \nThank you for your insightful suggestions！We present the average results from three independent trials (using three different random seeds) across two base model configurations. Clearly, our proposed methods still outperforms all baselines. Additionally, the corresponding t-tests results confirm that the performance improvements are significant.\n\n| Method(LLaMA-3.2-3B)| truthfulqa | tydiqa | logiqa | mmlu | hellaswag | arc_c | boolq | Average |\n|--|--|-|-|-|--|--|--|--|\n| ds2  | 41.51 ± 1.08   | 41.83 ± 2.13 | 24.6 ± 0.39  | 56.97 ± 0.14 | 55.7 ± 0.09   | 44.39 ± 0.87 | 76.99 ± 0.62 | 48.87 ± 0.12 |\n| full_token | 43.92 ± 0.52   | 43.83 ± 2.32 | 24.81 ± 0.47 | 56.95 ± 0.11 | 55.59 ± 0.04  | 44.99 ± 0.49 | 74.24 ± 0.38 | 49.17 ± 0.21 |\n| random   | 43.88 ± 0.09   | 42.77 ± 3.08 | 24.24 ± 0.74 | 57.09 ± 0.13 | 55.48 ± 0.1   | 44.96 ± 0.45 | 74.44 ± 0.53 | 48.97 ± 0.31 |\n| rho     | 47.97 ± 0.77   | 50.93 ± 0.85 | 26.15 ± 0.09 | 57.16 ± 0.01 | 56.48 ± 0.04  | 46.03 ± 0.62 | 77.04 ± 0.1  | 51.67 ± 0.15 |\n| fixed_model_cleaning| 48.24 ± 0.61   | 51.8 ± 0.47  | 26.05 ± 0.16 | 57.03 ± 0.11 | 56.46 ± 0.04  | 45.97 ± 0.51 | 77.27 ± 0.27 | 51.83 ± 0.06 |\n| self_evolving_cleaning| 52.01 ± 0.93   | 54.38 ± 0.33 | 28.43 ± 0.36 | 56.31 ± 0.12 | 55.75 ± 0.05  | 46.68 ± 0.6  | 77.25 ± 0.07 | **53.0** ± 0.2   |\n\n| Method (LLaMA-3.1-8B)  | truthfulqa  | tydiqa | logiqa | mmlu | hellaswag | arc_c | boolq  | Average|\n|--|-|--|-|-|-|-|--|-|\n| ds2     | 48.12 ± 1.3    | 53.02 ± 2.05  | 27.34 ± 0.32  | 65.81 ± 0.03   | 60.47 ± 0.1 | 53.37 ± 0.18 | 83.38 ± 0.0  | 55.93 ± 0.12 |\n| full_token   | 49.08 ± 1.36| 52.5 ± 2.83| 28.06 ± 0.47| 65.77 ± 0.02| 60.33 ± 0.08  | 54.12 ± 0.2 | 82.92 ± 0.38 | 56.1 ± 0.2   |\n| random   | 49.84 ± 1.01| 54.09 ± 1.39  | 27.7 ± 0.59   | 65.83 ± 0.06 | 60.32 ± 0.08  | 54.41 ± 0.3 | 83.24 ± 0.14 | 56.47 ± 0.12 |\n| rho | 55.76 ± 0.99| 58.49 ± 3.46  | 28.11 ± 0.93  | 65.75 ± 0.01   | 62.0 ± 0.12   | 54.95 ± 0.23  | 82.16 ± 0.45 | 58.2 ± 0.61  |\n| fixed_model_cleaning| 56.04 ± 0.11| 61.85 ± 0.72| 28.06 ± 0.27| 65.62 ± 0.08 | 61.96 ± 0.04| 55.04 ± 0.14| 82.74 ± 0.08 | 58.77 ± 0.15 |\n| self_evolving_cleaning| 59.75 ± 0.16 | 64.1 ± 0.85 | 26.56 ± 0.54| 65.17 ± 0.1| 62.65 ± 0.03  | 54.75 ± 0.2 | 82.54 ± 0.09 | **59.33** ± 0.23 |\n\nThe t-test results indicate that Self-Evolving Cleaning significantly outperforms RHO, showing statistically significant improvements with a p-value less than 0.05. \n\n| Method Comparison  (T-test value, p value) | LLaMA-3.2-3B | LLaMA-3.1-8B  |\n|---|--|-|\n| Fixed-model cleaning vs. Rho| (1.77, 0.152)   | (1.56, 0.193)   |\n| Self-Evolving Cleaning vs. Rho     | (9.18, 0.001)   | (3.02, 0.039)   |\n\n\n> **Weakness 2: Computational cost**\n\n**Response**:\nThank you for highlighting this important issue. The computational costs associated with the token cleaning pipeline primarily consist of the training costs, akin to those of standard supervised fine-tuning (SFT), along with two types of additional inference costs. These additional costs stem from one base model and another reference model, both of which are used to calculate token-level influence scores.\n\nCompared to the Rho-1 baseline, our two proposed methods do not incur any additional inference costs since the total data size for inference remains unchanged. For the Naive Fixed-Model Cleaning, we perform a one-shot inference on all samples simultaneously, mirroring the process used in Rho-1. For the Self-Evolving Cleaning pipeline, we simply segment the data pool into several partitions for independent inference using different reference models,  i.e., (50k samples -> 10k (reference model 1), 10k (reference model 2), …, 10k samples).  Consequently, the inference cost for the Self-Evolving Cleaning pipeline is also equivalent to that of Rho-1.\n\n> **Weakness 3: technical contribution clarification**\n\n**Response**:\nThank you for raising this concern. We would like to clarify that although the concept of masking unimportant tokens, initially introduced by Rho-1, is not new but remains very timely. Based on this, our work makes several novel contributions:\n- Self-Evolving Cleaning: Our iterative approach that progressively refines token selection is entirely new and shows superior performance.\n- Theoretical Framework: We provide a rigorous analysis of when and why token cleaning outperforms full-token training, with error upper bounds that explain the observed Matthew effect.\n- Noisy-Label Perspective: We reframe token selection as a noisy-label problem, providing a new theoretical lens that connects token cleaning to a rich body of work on learning with noisy labels.\nThese contributions extend beyond prior work (Rho-1) on token selection, offering both theoretical insights and practical improvements.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The work is very closely related to previous token selective fine-tuning methods. They differ from previous works by providing a semi-supervised style algorithm, self-evolving cleaning", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths\n- The paper is well written and easy to follow\n- Theoretical analysis is interesting and adequately supports the findings\n- Experiments are done thoroughly'\n\n## Weaknesses\n- My main concern is on the effectiveness of this approach. In Table 1, I am not sure if the author's approach is significantly better than previous work (e.g. RHO). I would like to see the average result from multiple runs and corresponding t-tests to see if the authors' approaches are consistently better.\n- Another question is on the computational cost. It seems that multiple computations are needed in the token cleaning pipeline. How does the computation cost, e.g. in terms of latency, compare with vanilla SFT and other baselines?\n- Although minor, I have some concerns regarding the technical contributions. The idea of removing unneeded tokens for fine-tuning is not new, and may somewhat limit the novelty of this work.\n\nIf all or part of my concerns are resolved, I am willing to re-evaluate my score.", "other_comments_or_suggestions": "See Weaknesses", "questions_for_authors": "See Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: A summary of the strengths and weaknesses based on the reviews and rebuttal (including the follow-up discussion among the reviewers and authors) is provided below. 


**STRENGTHS**

1\ The authors introduced a new influence-guided token cleaning pipeline in supervised fine-tuning of large language models (instead of discarding whole samples). 

2\ The proposed approach generally performs better empirically under different LLMs and benchmarks. This is further demonstrated with the additional results in the rebuttal.

3\ The theoretical analyses provide intuitive explanations and insights.


**WEAKNESSES**

1\ *Clarity*: A major concern is that the authors describe uninformative tokens as either noisy/negatively impactful or minimally impactful. I agree with Reviewer dD3A that this can be extremely confusing to a reader. The community would generally consider uninformative to mean not containing information. Does negative impact then imply no information? 

To eliminate ambiguity and confusion, I would strongly recommend that the authors avoid using the words "informative" or "uninformative", and simply revert to using positively impactful, negatively impactful or minimally impactful. We trust that the authors will do the necessary to resolve this major yet rectifiable issue of clarity. You can also consider using words/definitions like "helpful" vs. "harmful", like in the following missing reference which the authors can consider citing:

Helpful or Harmful Data? Fine-tuning-free Shapley Attribution for Explaining Language Model Predictions. ICML 2024.

Secondly, among the tokens that are filtered out, what proportion of the 40% corresponds to negatively impactful and what proportion of it corresponds to minimally impactful? The authors should show these empirical results in their revised paper.


2\ *Time efficiency* of the proposed method should be analyzed rigorously (Reviewers dD3A, fypD).


From the above, the pros outweigh the cons. 

The authors are strongly encouraged to revise their paper based on the above feedback and that of the reviewers.
Decision: Accept (poster)