Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning
Authors: Chao-Chung Wu
Zhi Rui Tam
Chieh-Yen Lin
Yun-Nung Chen
Shao-Hua Sun
Hung-yi Lee
Venue: NeurIPS 2025 poster ()
Keywords: perplexity
robustness
LLM-generated data
performance degradation
forgetting
Primary area: deep_learning
TL;DR: A simple masking technique to avoid LLM finetuning from degrading on general capabilities.

Abstract: Maintaining consistent model performance across domains is a fundamental challenge in machine learning. While recent work has explored using LLM-generated data for fine-tuning, its impact on cross-domain generalization remains poorly understood. This paper presents a systematic analysis revealing that fine-tuning with LLM-generated data not only improves target task performance but also reduces non-target task degradation compared to fine-tuning with ground truth data. Through analyzing the data sequence in tasks of various domains, we demonstrate that this enhancement of non-target task robustness stems from the reduction of high perplexity tokens found in LLM-generated sequences. Following our findings, we showed that masking high perplexity tokens in ground truth training data achieves similar non-target task performance preservation, comparable to using LLM-generated data. Extensive experiments across different model families and scales, including Gemma 2 IT 2B, Llama 3 8B Instruct, and three additional models, agree with our findings. To the best of our knowledge, this is the first work to provide an empirical explanation based on token perplexity reduction to mitigate catastrophic forgetting in LLMs after fine-tuning, offering valuable insights for developing more robust fine-tuning strategies.

Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning
Chao-Chung Wul+
Zhi Rui Tam Chieh-Yen Lin' Shao-Hua Sun 1.211 Hung-yi Lee?t
Yun-Nung Chen?
Appier Al Research, NNational Taiwan University {j [johnson.. wu, ray. tampeepiier com
Abstract
Maintaining consistent model performance actoss domains is fundamental chal- lenge in machine learning. While recent work has explored using LLM-generated data for fine-tuning, its impact cross-domain generalization remains poorly understood. This paper presents a systematic analysis revealing that fine-tuning with A.LM-generatee data not only improves target task performance but also duces non-target task degradation compared to fine-tuning with ground truth data. Through analyzing the data sequence in tasks of various domains, demonstrate that this enhancement of non-target task robustness stems from the reduction high perplexity tokens found in LLM-generated sequences. Following our find- ings, we showed that masking high perplexity tokens in ground truth training data achieves similar non-target task performance preservation, comparable using LLM-generated data. Extensive experiments across different model families and scales, including Gemma 2 IT 2B, Llama 3 8B Instruct, and three additional models, agree with our findings. To the best of our knowledge, this is the first work to provide an empirical explanation based on token perplexity reduction to mitigate catastrophic forgetting in LLMs after fine-tuning, offering valuable insights for developing more robust fine-tuning strategies.
Introduction
Supervised fine-tuning large language models (LLMs) 20
has proven highly effective for enhancing and produce useful outputs across As LLMs
their ability to follow novel instructions [15 21 23
43
49
wide range of tasks, including summarization and web querying 0 increasingly applied in specialized domains such as arithmetic and programming assistance (27 38) practitioners often encounter substantial computational and data cleaning issues when adapting these models to gain improvements. These difficulties are exacerbated by performance saturation in smaller LLMs, where benchmark improvements on tasks like MATH [12] and MBPP B tend to plateau as model size decreases. Moreover, fine-tuning can risk degrading model's general capabilities 4 24 30, 31. This raises a critical question: how can we efficiently fine-tune pre- trained, instruction -oaluutton-following LLMs for domain-specifi tasks without compromising other skill sets such as arithmetic or common sense reasoning, especially when original fine-tuning data unavailable or computational resources are constrained?
17
26 34]
Recent work has shown that fine-tuning with model- model-generated data can be highly effective, often outperforming training on ground truth data when using high-quality generated samples [10 32
"qqual contribution
1 'quaal advisory contribution
Shao-Hua Sun an Adjunct Research Scientist Appier Research
Our code and datasets are available at
-lnntttttuts finetunes
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Question: Given mathematics problem, determine the answer. \nProblem Let the reciprocals of the roots of 5x² 3x + 4 be and B. Evaluate B.AnAnswer:$
Ground Truth: Denote the roots of 5x² 3x + 4 by and b. We have that a and 3 :. So, a 3
ab" Now, know that a+b 5 and ab by the relationship hetween
sum/products of roots and the coefficients of polynomial. Hence 8 +
ab
Self-Output (Llama 3 8B Instruct): Let rı and rg he the roots of 5x 3r 4. Then, by Vieta's formulas, we have t12 -5 and rın2 5 Let and
Then, we have
3
a 3
rathi
4
Figure 1: An example MATH problem showing more high perplexity tokens (highlighted in red, perplexity 2.5) in ground truth than Self-Output responses (Llama self- sefff-gnerated responses).
35 48
For example, Ren al. 33 demonstrated that model-gencrated responses surpass human annotations in downstream performance, particularly when using more capable models (e.g., GPT-4) to generate training data and mitigate degradation general tasks. Although they attributed part of this success to the "familiarity" of generated responses, measured via low perplexity, remains unclear why such data works well for instruction tuning. Specifically: (1) The regenerated response may differ contextually from the ground truth, and such contextual bias could affect the utility of low-perplexity training. (2) The "non-forgetting" impact of low-perplexity training on the performance of tasks not being fine-tuned, which maintains the practicality of the model, is still underexplored. (3) Since model size plays role in distillation benefits, it important to disentangle the effects of model scale and evaluate low-perplexity training using models of the same size.
Our work offers empirical explanation, grounded in loss and perplexity reduction, for why LLM- generated data tends to cause less degradation in fine-tumi nhe-tunnng outcomes. Through systematic analysis, first show that fine-tuning with different types of LLM- LLM-generated data (e.g., self- seff-generated and rephrased responses) significantly mitigates degradation on non-target tasks compared to fine-tuning with ground truth data. Second, we trace this improvement L.M-ggnerrted responses exhibit lower sentence-level perplexity and a smaller proportion of high- perplexity tokens. This leads to a practical insight: by simply masking high-perplexity tokens in ground truth training data, we can achieve a comparable reduction in non- non-target task degradation as observed with LLM-generated data. Importantly, this can be done using the same model prior to training, allowing us to pre-process the data and harness the benefits of low-perplexity training.
key property illustrated in Figure [
We propose novel empirical explanation for non-target task performance degradation, introducing Selective Token Masking (STM), a simple yet effective strategy that filters out high- nigh-perplexity tokens to enable low-perplexity training. We have applied STM on (1) different perplexity criteria, (2) different model families and scales, and (3) various fine-tuning techniques from full-weight fine-tuning to parameter-efficien fine-tuning. We found that STM could achieve almost the same capabilities as the original models have on non-target tasks after fine-tuning on target tasks under all circumstances. We could identify that the degradation results from the low model weights update during the low-perplexity training. Together, these insights provide a new perspective on designing fine-tuning strategies that better retain general capabilities while enhancing performance on domain-specific tasks.
2 Training LLMs on self-generated data
To evaluate the effectiveness of LLM LLM-generated data for fine-tuning, we explore two distinct methods for generating training data across two different target tasks. We then assess the models fine-tuned with the generated data on five non-targe tasks, respectively. The two generation strategies, Self- Output and Rephrase, offer complementary approaches to constructing LLMd-based training data, each addressing different trade-offss and challenges. In this section, we introduce the original datasets used for data generation and describe our methodology for constructing relf-generated training datasets using language models.

---

2.1 Training and evaluation framework
For target-task fine-tuning, we adopt the MBPP and MATH datasets. These datasets provide rich annotations, including full solutions, reasoning steps, and test cases, beyond just the final answers, allowing models to learn comprehensive task-solving behaviors. For evaluation, in addition to using the test sets from MBPP and MATH, we assess model performance on GSM8K 77, ARC- Challenge [6], and BIRD [19 to examine generalization to non-target tasks involving various forms of reasoning and generation. A detailed description of the datasets is provided in Section 4
2.2 Self-Output
Table 1:
Perplexity average and variance over answer sequences of MBPP and MATH training datasets on Llama 3 8B Instruct. The average sentence perplexity is calculated by averaging the sum of each sentence's s per- plexity over the token perplexity score.
Self-Output follows a similar high-level approach, similar to other synthetic data generation techniques 31 48, where high-quality synthetic dataset cre- ated by sampling diverse responses from an LLM and applying a strict filtering process. For each training instance, we use language model M (e.g., Llama 3 8B Instruct) to generate (e.g., 32) distinct re- sponses with a temperature setting of T 0.7. The generated responses are then filtered to those that are semantically aligned with the ground truth, ensuring the quality and relevance of synthetic data.
Data
Method
Avg. PPI.
Ground Truth 4.83 (7.04)
MBPP
Rephrase
1.69 (0.16)
Self-Output
1.16 (0.01)
Ground Truth 2.45 (0.81)
MATH
Rephrase
2.11 (9.28)
Self-Outpur excels at producing high-quality train- ing data domains with reliably verifiable outputs (e.g.. programming tasks with objective ground truths). However, it remains constrained to settings where such verification feasible and imposes significant computational overhead from the multiple generations and filtering required to identify valid outputs.
Self-Output 1.34 (0.03)
2.3 Rephrase
Yang et al. 47 proposed eeff-disiillatio. The process uses an instruction finetuned cannettion-iinetueed LLM to rephrase ground-truth responses in its style. By providing both the instruction and ground truth to the target LLM, it generates semantically equivalent reformulations with ground truth responses. Compared to Self-Output, Rephrase is computationally efficient, it requires only a single pass of generation without additional filtering. However, Rephrase is susceptible to hallucinations, as evidenced by the higher variance in token-level perplexity (see Table I.. Despite this limitation, Rephrase offers versatility, as it does not rely on verifiable ground truth and can be applied to a broader range of tasks.
We have sanity check on the output of Rephrase, and remove the incorrect label output as the final training set for Rephrase to ensure that all the final outputs of Rephrase are correct. But we did not evaluate the intermediate output of Rephrase, e.g., the reasoning or CoT process. This process is partially reflected in Appendix Table 17
Table 2 presents the performance comparison between Self-Output, Rephrase, and Baseline Fine- tuning on ground truth using different models in terms of model size and model series across both target task improvement and non- non-targe target task degradation rate. Both task improvement rate (TT) and degradation rate (BWT) are adopted from Backward Transfer rate from [4] and modified as follows:
TI
dtorget bargee
(originat) (original)
/
T-1
Σ BWT = T-1 1=1
(a, (train)
(original)
/o (original)
a;
Given T tasks composed of one target task and T-1 target tasks, TI is the target task performance improvement percentage after training on the target task. BWT refers to the average performance degradation percentage of the T - non-target task after training on the target task. The models to fine-tune in this paper refer to generic instruction following models before fine-tuning with target task data, e.g., Ilama 3 8B Instruct [8) Gemma 2 IT 2B (36), Mistral 7B Instruct [16. and OLMo 2 7B series [4). For target task performance, Self-Output demonstrates consistent positive TT in both MBPP and MATH. For the non-target task, the Self-Output method achieves better BWT scores
3

---

1.0
PPL
0.8
-10
-15
0.6
2B from Rephrase (Llama Instruct)
BWT
Trrannng
PPL
-25
-30
SelF-uutuut (Liama 3 80 Instruct)
0.2
-35
PPL
-40
0.0
12
PPL
Token Position
Figure Fine-tuned Llama 3 8B Instruct on MBPP labels generated by various LLMs. Back- ward Transfer (BWT) measures performance drop non-target tasks, showing that Low Low-perplesss perplexity labels training generally leads to less degradation (upper left) than ground truth (bottom right).
Figure 3: Comparison of token-level perplex-
ity (PPL) distributions between comman-annottted
Ground Truth (top) and Llama 8B Instruct gen- crations for Rephrase and Self-Outpur sequence (middle and bottom), where PPL of Self-Output data is low and with fewer spikes.
than the baseline fine-tuning in most cases, indicating much less degradation on non-target task capabilities. For more details, such as performance comparison with other models other minor findings for self-output and rephrase method, please refer to Appendix
2.4 Analysis of perplexity across each sample
In this section, we analyze the perplexity of datasets generated by Self-Output and Rephrase to quantify their impact model uncertainty and fine-tuning stability. We first collect the inference statistics GPT-4o, GPT-4o mini, Gemma 27B, Llama 8B Instruct, and Gemma all three methods: Baseline Fine-tuning, Rephrase, and Self-Outpur strategy, along with their BWT, as Figure [egure]]sowws shows that higher perplexity of target training data could result in a higher degradation on non-target task. Second, we further collect the Ilama 3 8B Instruct inference statistics from three kinds of training data: Baseline Finetuning (ground truth data), Rephrase, and Self-Output strategy. We evaluate the perplexity of responses by summing up the negative log probability of each token conditioned on its previous context in the sentence to measure the uncertainty of model outputs over the entire sequence. Specifically, the perplexity PPL across a sequence is computed as:
N
PPL = - exp(
P(wa Wi,..., ()=-)))
N
where P(w; | Wiy..., wi-1) represents the conditional probability of token we given its preceding context, and is the total number of tokens the sequence.
Table demonnsttts that the Self-Output strategy achieves the lowest average perplexity of 1.16 in MBPP and 1.34 in MATH. Among the three approaches, Ground Truth exhibits the highest uncertainty with mean perplexity of 4.83, and the perplexity of Rephrase is slightly higher than Self-Output The lower perplexity of Belf-Output suggests that L.MMs are less likely deviate far from their initial weights when fine-tuning on downstream tasks created by Self-Output. This characteristic maintains the model's original performance on non-target tasks while effectively learning from the target task.
To investigate predictability at the token-level, we sample a single MATH response and visualize the token-level perplexity value. Figure3 Self-Oue tiff-Outuut responses. The first 20 tokens' variance is 2.04 for Ground Truth, 2.31 for Rephrase, and 0.04 for Self-Output, which confirms our observation: Ground Truth exhibits frequent perplexity spikes of high amplitude (perplexity around 6-8), which reflects the natural variability in human problem-solving expressions. While Rephrase maintains similar peak amplitudes to Ground Truth,
distinct patterns across Ground Truth, Rephrase, and
In this section, natural log instead of base 2 for computation convenience

---

i.e., more regular spike patterns, its variability is between Ground Truth and demonstrates remarkably consistent low token perplexity values rarely exceeding 2, indicating highly predictable token sequences. These patterns suggest that Self- Self-Ouiput generates more deterministic solutions, potentially beneficial for maintaining consistent problem-sole solving approaches, while Ground Truth sequences introduce much higher variations. To demonstrate the different content of tokens distribution between Ground Truth and Self-Output responses, please refer to Appendix B
Self-Output
3 Low-perplexity token learning via selective token masking
Our earlier analysis of Feeure a notable discrepancy in token perplexity distributions between Self-Output generated data and other sources. This observation raises central question: Is the superior performance of Self-Output methods primarily due to the lower perplexity of their generated tokens? To explore this hypothesis and design more efficient fine-tuning strategies, we propose Selective Token Masking (STM), a novel approach to supervised fine-tuning (SFT) that explicitly leverages token-level perplexity.
The core idea behind STM is simple yet effective: we use an existing instruction-tuner model to compute teken-level perplexities and mask ground truth tokens whose perplexity exceeds a predefined threshold I during training. Our investigation yields two key insights: First, STM highlights the pivotal role of token perplexity in shaping fine-tuning performance. Second, it shows that the gains observed from Self-Output methods can be replicated through perplexity- -based token filtering, suggesting that performance improvements stem more from low-perplexity token distributions than from the self-generation process itself. STM thus offers not only computational efficiency but also a principled framework for understanding model adaptation during fine-tuning.
While our approach shares conceptual similarities with prior work such as nificantly in both implementation and efficiency. Previous methods typically involved two-stage process: training a reference model on high-quality data to identify learnable (high-perplexins tokens, followed by pre-training larger LLM on this curated subset. In contrast, STM achieves similar goals in a streamlined single-stage workflow by directly using an existing instruction-tune compute token perplexities in single forward pass. This design avoids the need to train or maintain any additional models, significantly reducing computational overhead while preserving the benefits of low-perplexity training. This makes STM compelling choice for efficient fine-tuning rather than full-scale pretraining.
differs sig- 2 20
model to
Importantly, STM is orthogonal to other training strategies. It can be applied alongside different data sources (e-g.. ground truth or Self-Output) and training techniques (e.g., full-weight fine-tuning or LoRA) to alleviate non-target task performance degradation. In the following sections, we explore how STM complements these various settings and contributes to more robust fine-tuning outcomes.
4 Experimental setting
To evaluate the effectiveness of our proposed method, we adopt benchmark datasets specifically selected to assess model robustness on both target and non-target tasks after fine-tuning, We include only datasets with verifiable ground truth, allowing us to filter out incorrect self-generated responses for a fair comparison with self-synthesized baselines and to ensure high data quality. Following recent findings 4 that LoRA [13] effectively mitigates performance degradation, we standardize our experiments by using LoRA for all fine-tuning. We then vary training strategies and data sources to study their impact on performance preservation. Details of our evaluation prompts and training configurations are provided in Appendix [F] All implementation code and datasets are publicly available at:
finetunes lin-ittuuusss
4.1 Target and non-target datasets
We focus on three domains, programming, mathematics, and knowledge- based tasks, to study perfor- mance degradation after supervised fine-tuning with LoRA on each domain. For the programming and mathematics domains, we select MBPP and MATH for target task training, while others (MATH or MBPP test split, BIRD, GSM8K) and knowledge- tasks for non-target task evaluation to assess generalization capabilities.
5

---

Table Performance comparison across different models and target tasks terms of task improve- ment (TT) on target test set and backward transfer (the degradation percentage. BWT) of non-target test set, and the cost of data related process before training (e-g-, generation of self-output, token PPI. calculation and STM training criteria selection). Self-Output and our STM method have generally better BWT and TI, showing a better preservation of generalization capabilities after fine-tuning.
Model
Target task
Method
BWT(%
TI(%)
Cost (GPU hours)
Baseline Fine-tuning
138.19
-21.76
0 Hours
Self-Output
-8.10
5.70
MBPP
Rephrase
-3.23
-4.69
30 Minutes
STM,=2.5 (Ours)
0.42
0.00
5 Minutes
Gemma 2 IT 2B
Baseline Fine-tuning
-36.68
-22.78
0 2 Days 39 Minutes
Self-Output
-1.73
9.06
MATH
Rephrase
-14.06
-28.83
STM, (Ours)
-2.93
7.83
8 Minutes
Baseline Fine-tuning
-34.71
-2.23
0 16.8 Hours
Self-Output
3.09
1,55
MBPP
Rephrase
-5.32
-9.58
36,8 Minutes
STM, (Ours)
-0.16
3.20
4,5 Minutes
Llama 3 8B Instruct
Baseline Fine-tuning
-14.12
-17.83
0 22 Days
Sclf-Output
0.31
9.55
MATH
Rephrase
-1.09
4.78
29.3 Minutes
STM, (Ours)
-0.30
6.37
Minutes
Programming. For code generation evaluation, we split MBPP into target task training data and non-target task testing data, while BIRD serves non-target assessment. This pairing tests both direct programming ability and cross-language generalization from Python to SQI..
Mathematical reasoning. We split MATH into target task training data and non-target task testing data, and GSM8K for non-target task testing. These datasets differ in format: MATH features competition-style problems, while GSM8K uses natural language, allowing us to assess generalization across mathematical expression styles.
chowledge-based To assess broader generalization capabilities beyond mathematical and program- ming domains, we incorporate ARC- RRC-Challenge, subset of the ARC science question dataset.
For details about the data curation the target and non-target datasets, please refer to Appendix A
To explore more performances of non-target tasks including instruction following and safety perfor- mance under limited model choices, please refer to Appendix
STM results
Table 2iale22shon shows that with threshold to filter out high perplexity tokens, such as 2.5, which approxi- mately filters out around 20% to 24% of total tokens, its BWT is close 0%, and STM also achieves comparable TI to the Self-Output method. It indicates that STM not only improves target task performance but also reduces non-target task degradation. This result agrees with our hypothesis that the presence of high perplexity tokens is one of the causes of performance degradation of the ground truth model, and low perplexity tokens sufficiently mitigate such negative impact for strong generalization without the need for self-generes gelf-generated data. We also applied STM to other models, and the results can be found in Appendix D Our conclusions are still the same across different models.
5.1 Does filtering high perplexity tokens always perform better?
From the intuition of the high perplexity filtering mechanism of STM, we have done an ablation study on the masking criteria of perplexity by filtering out high, random, and low ppl tokens. More specifically, high ppl refers to filtering the top 25% ppl of all tokens, random filtering refers to filtering the randomly selected 25% of tokens, and low ppl refers filtering the lowest 25% ppl of all tokens from the dataset. As the upper part of Table 3 shows, masking high ppl tokens in STM simultaneously reduces performance degradation and improves target task performance. Likewise, we have the same conclusions for Llama 3 8B Instruct such ablation test, please refer to Appendix D

---

Table 3: Ablation and scaling on threshold of STM with Gemma 2 IT 2B on MBPP task. Per- centages indicate the ratio of tokens filtered from training data. STM performs generally the best
Table 4: STM applied on full weight fine- tuning (FWFT), DoRA and LoRA with MBPP on Gemma 2 IT 2B. STM can enhance all three kinds of fine-tuning techniques with much better preservation of non-target task capability and tar- get task improvement.
with 20% to 24% of high ppl tokens filtered.
Configuration
BWT(%)
TI( (6)
,TM,=2.5,4
0.4
0.0
Configuration
BWT (%)
TI())
STM,- 2.5,random
-8.6
-15.6
FWFT
-31.87
-27.98
STM,=22 -22,5,io
-7.9
-18.7
FWFT + STM,=25
-0.13
-8.81
Baseline Fine-tuning
-38.2
-2.22
STM,- (6.26%)
-2.9
-11.4
LoRA
-38, 19
21.76
STM, (12.34%)
-2.5
-8.8
LoRA + STM,-2.5
0.42
0.0
STM.- (15.1%)
-0.7
-10.4
DoRA
-8.54
-15.2
STM, (23.8%)
0.4
0.0
DoRA + STM,-25
-0.01
0.04
STM, (26.1%)
-0.5
STM (23.8%)
-3.8
-7.3
5.2 Is larger-sized LLM a better token filter?
In addition to that, STM uses initial model perplexity for token selection, we investigate an alternative token selection model for STM.
Cross-scale filtering. We explore perplexity assessment using larger models within the model family (e.g.. Gemma IT 9B) to guide token selection for smaller models (e.g., Gemma IT 2B), denoted as S This approach investigates whether token selection benefits from a more robust understanding of larger models, potentially offering a form of knowledge distillation through perplexity-based filtering.
Table shows that both STM and STM2 perform better than Baseline Fine-tuning. However, scaling the perplexity model does not surpass STM with masking created by its (2B), likely because larger models assign lower perplexity to tokens smaller models find challenging. Therefore, the optimal strategy should always be to the same model for token selection. We also investigated other alternatives such as a learnable token selection, please see Appendix [D.
5.3 Optimal threshold selection for STM
A key hyperparameter of STM is the perplexity threshold to filter tokens from the supervised fine- tuned (SFT) model. We explore the existence an optimal threshold that maximizes performance. The bottom part of Table This optimal threshold demonstrates consistent benefits across both target and non-target datasets, including GSM8K, ARC, MATH, and BIRD, showing its robustness and generalization similar to Self-Output data. For further validation of the generalization of such settings, please refer to Table TT in Appendix D that such settings upply different model families as well.
that filtering approximately 24% of tokens yields the best results.
5.4
Is forgetting affected by smaller learning rates?
Following training settings in 9 learning rate (Ir) would be one reason that harms the per- formance of Bascline Fine-tuning severely (but not for STM and LLM generated data basclines). Therefore, we conducted additional learning rate sweeps down to 1e and Gemma 2 IT 2B on the MBPP dataset. As Table T shows, we found that Baseline Fine-tuning improves at smaller learning rates (e.g., TI: 10.6% - 1.33%), but the forgetting problem (BWT) persists (e.g.. 1.6%%). In contrast, STM consistently outperforms Baseline across all learning rates with stronger target task performance and substantially lower forgetting. Notably, STM achieves better BWT and stable performance even at small learning rates (e.g.. 2.23%. BWT: +1.3%% at le-7). These results demonstrate that STM is more robust to learning rate choices, requires less learning rate tuning, and achieves more stable training dynamics, highlighting practical advantage in real- real-world fine-tuning scenarios.
for Llama 3 8B Instruct
7

---

Table 5: Learning rate (Ir) sweeping results of training Llama 8B Instruct (Llama 3 8B-IT) and Gemma 2 IT 2B on the MBPP dataset. Note that only BWTs of Baseline Fine-tuning (Baseline) with the best TI are calculated. The results show the higher robustness of STM to learning rate choices while still forgetting less nothing.
Llama 8B-IT
Ir
BWT(%)
TI(%)
BASELINE
1E-4
-00.6
Gemma 2 IT 2B
Ir
BWT(%)
TI(%)
BASELINE
2F-5
-34.7
-2.23
BASELINE
5E-6
0.9
BASELINE
2E-5
-38.2
15.5
BASELINE
1E-6
-4.47
BASELINE
5E-6
-4.0
BASELINE
5F-7
-1.3
BASELINE
1F-6
-17.6
BASELINE
1E-7
-1.6
1.33
BASELINE
1E-7
-4.7
-0.53
STM
1E-4
1.8
-3.58
STM
2B-5
-0.3
-0.5
STM
2E-5
0.2
3.2
STM
5E-6
-1.1
-3.0
STM
5E-6
-0.1
2.68
STM
1E-6
-0.35
-1.5
STM
1E-6
0.53
3.12
STM
1E-7
0.51
0.7
STM
5E-7
1.23
3.12
STM
1F-7
1.39
2.23
Table 6: Self-bleu score of Baseline Fine-tuning and STM training. The results show that both training lead to lower diversity, but masking would not reduce further diversity but less forgetting.
Llama 8B Instruct
self-bleu
accuracy(%)
BWT (%)
Original
20.47
44.75
Bascline Finc-tuning
IE-7
40.29=19
56.5
-1.6
STM,-2.
1B-7
40.77=17
58.25
1.39
5.5 STM generalization on different fine-tuning strategies
As STM is a masking technique applied target data during the fine-tuning stage, should be orthogonal to the fine-tuning techniques and improve the performance degradation. Thus, we experiment with how STM improves practical fine-tuning strategies like Well-Weight Fine-tuning (FWFT), Low-rank adaptation (LoRA), and Weight-decomposed LoRA (DoRA) [22). As Table shows, all the fine-tuning techniques are further improved by STM, which indicates STM's generalization on different fine-tuning strategies.
5.6 Does masking in STM lead to lower diversity of token generation?
Masking in STM reflects a possibility that model learns to generate a token distribution of higher kurtosis (lower diversity) by masking more tokens for training. Thus, experiment with Llama 8B Instruct model trained on MBPP coding dataset and tested on a sample of 100 MATH questions. Each instance's reasoning response generated times, and average self-bleu scores calculated. As Table 6 shows, Baseline Fine-tuning and STM share similar diversity, while Baseline Fine-tuning leads more forgetting, which STM avoids. Therefore, it more likely that training typically harms diversity, but STM could enhance Baseline Fine-tuning with less forgetting.
6 Analysis and discussion
In our experiments, we found that STM's effectiveness is applicable across different datasets, models, and fine-tuning strategies. In this section, we conduct several analyses on the model training process and model changes with STM to investigate the reason that reduces performance degradation.
Low training loss from low perplexity training. Training low perplexity data leads to a low training loss intuitively, and the model converge within fewer training epochs. the training procedure on MBPP training data different perplexities. The results show that training with STM or synthesized data yields much lower training loss, resulting in fewer updates of model weight for changing capabilities of non-target tasks.
showcases
Subtle LoRA weight changed from low perplexity training. We investigate the relationship between token-level perplexity and characteristies of weight updates in LoRA. Our analysis focuses

---

Target testing accuracy
Validation lass
Baseline Fine tuning
0.60
1.5
1.0
STM tau 2.5
0.55
Tranning
1.0
0.5
0.5
0.50
0.0
0.0
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
Training epochs
Training epochs
Training epechs
Figure 4: MBPP target task testing accuracy, validation and training loss of baseline finetuning with ground truth data, finetuning with Self-Output data, and STM strategy of perplexity filtering threshold=2.5 for Llama 8B Instruct. STM and self-output training yield better performances with much lower training and validation loss because of low- low-perplexity training.
Table 7: 1.2 norm of AW signifies larger update from the original weight W. STM shows small model AW, indicating STM training can occur with few changes on the model's original capabilities.
Models tuned on MBPP
Self-Output
Rephrase
Ground Truth
STM Ground Truth
Llama 3 8B Instruct
6.53
7.31
17.75
0.55
Gemma IT 2B
4.03
5.78
5.69
0.45
on understanding how different types of training data affect the magnitude and dimensionality of parameter adjustments in cansformer-basee language models. Given that LoRA effectively learns offset weights AW to the original model parameters W,, we calculate the L2 norm of AW to quantify the degree of deviation introduced by each fine-tuning dataset. As shown in Table fine-tuning with Self-Output data consistently results in smaller L.2 norm of weight updates compared to Rephrase and Ground Truth data, while STM produces the smallest updates (0.45-0.56) across all models. Such minimal perturbations suggest that both approaches, particularly STM, help preserve the model's original capabilities. Please see Appendix K for details on the analysis LoRA's individual rank.
High perplexity training leads to more degradation with similar weights updated. Although we found that directly training with high perplexity tokens leads higher loss and more weights updated, Applying regularization techniques can reduce model weight changes to mitigate non-target task degradation. Hence, we apply alternative regularization techniques to fine-tune with high-perplexity ground truth data to mitigate non-target task performance degradation. Our comparative analysis encompasses three widely-adopted regularization approaches: Dropout (33, Weight Decay [88] and Kullback Leibler regularization (39]. These methods have demonstrated effectiveness in preventing models from overfitting. Table 8 shows that under similar model weight changes, models applied with STM still outperform the high perplexity data trained models with regularization, indicating the performance degradation results from high perplexity training instead of purely model overfitting. For the results of all regularization combinations we have experimented with, please refer Appendix
7 Related work
Instruction-tuning Instruction following 28 is widely used to align responses of LLMs with target task values. Practically, it requires instruction tasks dataset to fine-tune LLMs, where each dataset consists of instructions and desired responses. Hence, the performance of instruction-tuning
heavily relies the quality of instruction data such context richness 4] 45 50].
In addition,
Ghosh et al. [9] investigated the limitation of instruction-tume tuning of LLMs about the catastrophic forgetting from pattern- coppyin behaviors and hallucinations with Lora/full fine-tuning These works give a good start on how performance degrades in terms of models' response behaviors and benchmarks when fine-tuning with instruction following dataset.
Using LLM-generated data for instruction fine-tuning. As Chung et al. 5 Wang et al. 41 showed, LLMs break down easily after training with different tasks. Several remedies to the
performance improvement focus on training data augmentation 42 46 For instance, [31 uses much larger LLMs (e.g., GPT-4 and Claude) to generate responses of questions as training labels, which improves the performance on both the target task and other non-target tasks. However, such a distillation method neglects the correctness of generated labels, so that more incorrect responses could be trained as the amount of generated data increases, and thus, using equal or smaller-sized

---

Table 8: Comparison of STM and regularization strategies on the MBPP Ground Truth data using L 8B Instruct. Hyperparameters are selected based on the performance or similar L2 norm of AW to STM's setup. STM consistently outperforms all regularization methods, suggesting that degradation is driven more by high-PPI. training than by overfitting.
Regularization & Hyperparameter
L2 norm of AW
BWT (%)
TI (%)
WEIGHT DECAY 0 + DROPOUT 0.05
0. 0.7539
-9.24
-9.82
WEIGHT DECAY 0.2 DROPOUT 0.3
0.7109
-3.50
-8.03
WEIGHT DECAY 0.5 + DROPOUT 0.3
0.5351
-11.15
2.86
KL coef =IE-5
0
-0.24
2.24
STM
0.5500
1.90
3.12
models for distillation could be challenging. of existing ground truth to generate labels to match similar styles of the LLMs for fine-tuning. However, rephrasing the ground truth answer limits the output distribution and results in lower performance in our study. Furthermore, [I]] exploits a base LLM as a judge to pick out answerable
47 prompts LLMs to simply rephrase the response
and unanswerable questions to compose a new training dataset to improve target task performance
only on the QA/conversation dataset. Although using Mistral 7B Instruct to generate acceptable responses for answerable questions improves target and non-target tasks, it is rarely discussed that if the proposed method is applicable to different model sizes and series. Besides, using LLM as a judge could be a noise to the correctness of data, which is shown to be important for target task training in study in Appendix D Lastly, [20 23 propose Selective Language Modeling to score favorable tokens from training data, and pre-training on the tokens brings higher performance. However, the requirements for pre-training a reference model or preparing a high-quality dataset for a scoring model could be exhausting depending on task difficulty. In addition to STM's applicability in Fine-Tuu Tunnng, STM does not require pre-training for the selection of tokens all, which brings efficiency and performance improvement at the same time.
Conclusion
In this work, we present a comprehensive empirical study of how fine-tuning with LLM ----eerertted data enhances cross-domain generalization in eestruction-oolowwer that such data significantly reduces degradation on non-target tasks compared to traditional ground truth fine-tuning, This robustness is closely linked to the reduced presence of high perplexity tokens in input sequences, which we identify as a critical factor in preserving general capabilities. Building this insight, introduce the Selective Token Masking (STM) method-al an efficient and model-agnostic strategy for achieving comparable improvements without relying on external model generations. Our extensive evaluations across diverse models, training strategies, and learning settings underscore the generality and effecttiveness of low- low-perplexity training. These findings offer a loss- and perplexity-based perspective on fine-tuning robustness and point practical direction for future work: designing adaptive strategies that prioritize low-perple low-perplexity tokens or filter out harmful high- segh-eetllexty tokens. Such approaches may generalize across models and tasks, enabling efficient domain adaptation while preserving general capabilities.
models. Our findings reveal
Limitations
The effectiveness of token-wise masking on training models could relieve catastrophic forgetting and improve target task performance simply. Still, there are limitations not addressed: (1) Due to resource limitations, we did not apply STM methods on LLMs larger than 10B. (2) Many non-target tasks we have not evaluated yet, such as tool learning. specific domains like medicine and finance. (3) Self-output data generation is costly. A trade-off between computing resources and performance can be further studied. (4) We did not consider how perplexity changes when task difficulty scales up. The robustness of curriculum learning process when training on easy to hard tasks could differ. (5) a smarter masking mechanism for perplexity threshold decision and targeting training tokens conditionally, Due space limitations, we leave the challenging issues as future work.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.00 08774, 2023.
[2] Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew I. Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models. In The Thirteenth International Conference on Learning Representations, 2025.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:
2021.
[4] Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody B Blakeney, and John Patrick Cunningham. LoRA learns less and forgets less. Transactions Machine Learning Research, 2024. Featured Certification.
[5] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetunee language models. Journal of Machine Learming Research, 25(70):1-53, 2024.
[6] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, arXiv preprint arXiv: 05457,, 2018
[7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.020000000 14168, 2021.
[8] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[9] Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Ramaneswaran S, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, and Dinesh Manocha. closer look at the limitations of instruction tuning. In Forty-first International Conference on Machine Learning. 2024.
[10] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08000 2023.
[11] Sonam Gupta, Yatin Nandwani, Asaf Yehudai, Mayank Mishra, Gauray Pandey, Dinesh Raghu, and Sachindra Joshi. Selective self-rehearsal: A fine-tuning approach to improve generalization in large language models, 2024.
[12] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty--ifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021.
[13] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022.
[14] Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self- synthesized rehearsal, 2024.
[15] Yu-Kai Hung, Yun-Chien Huang, Ting- Yu Su, Yen-Ting Lin, Lung-Pan Cheng, Bryan Wang, and Shao-Hua Sun. Simtube: Generating simulated video comments through multimodal ai and user personas. In International Conference Intelligent User Interfaces, 2025.
11

---

[16] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.06822 2023.
[17] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte. Sotiris Anagnostidis. Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richard Nagyfi, al. Openassistant conversations democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
[18] Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in Neural Information Processing Systems, 4, 1991.
[19] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as database interface? big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024.
[20] Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, Nan Duan, Weizhu Chen, et al. Not all tokens are what you need for pretraining. In The Tiirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[21] Max Liu, Chan-Hung Yu, Wei-Hsu Lee, Cheng- Wei Hung, Yen-Chun Chen, and Shao-Hua Sun. Synthesizing programmatic reinforcement learning policies with large language model guided search. In International Conference on Learning Representations, 2025.
[22] Shih-Yang Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu- Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In International Conference on Machine Learning, 2024.
[23] Li-Chun Lu, Shou-Jen Chen, Tsung-Min Pai, Chan-Hung Yu, Hung- Yi Lee, and Shao-Hua Sun. Llm discussion: Enhancing the creativity of large language models via discussion framework and role-play. In Conference on Language Modeling. 2024.
[24] Zheda Mai, Arpita Chowdhury, Ping Zhang, Cheng-Hau Hao Tu, Hong- Chen, Vardaan Pahuja, Tanya Berger- Wolf, Song Gao, Charles Stewart, Yu Su, and Wei-Lun Chao. tine-tuning is fine, if calibrated. In The eightt Annual Conference on Neural Information Processing Systems, 2024.
[25] Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Pri- oritized training on points that are learnable, worth learning, and not yet learnt. In International Conference Machine Learning, pages 15630-156499 PMLR, 2022.
[26] Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assiste question annwering with human feedback. arXiv preprint arXiv:2112.09332 2021.
[27] Nhan Nguyen and Sarah Nadi. An empirical evaluation of github copilot's 's code suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories, MSR '22, page 1-5, New York, NY, USA, 2022. Association for Computing Machinery.
(28) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[29] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[30] Xiangyu Qi Yi Zeng, Tinghao Xie, Pin-Yo Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to!. 2023.
12

---

[31] Xuan Ren, Biao Wu, and Lingqiao Liu. I learn better if you speak my language: Understanding the superior performance of fine-tuning large language models with llm-generated responses. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10225-10245, 2024.
[32]
Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell Bileschi, Noah Constant, Roman Novak. Rosanne Liu, Tris Brian Warkentin, Yundi Qian, Ethan Dyer, Behnam Neyshabur, Jascha Narain Sohl-Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research, 2024, 2023.
[33] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958 2014.
[34] Nisan Stiennon, Long Ouyang. Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008- 3021, 2020.
[35] Nicholas Collin Suwono, Justin Chen, Tun Min Hung, Ting-Hao Kenneth Huang, I-Bin Liao, Yung-Hui Li, Lun-Wei Ku, and Shao-Hua Sun. Location-aware visual question generation with lightweight models. Empirical Methods Natural Language Processing, 2023.
[36] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.001 18, 2024.
[37] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models, 2023.
[38] Immanuel Trummer. Codexdb: Generating code for processing sql queries using gpt-3 codex. arXiv preprint arXiv:2204.088. 00941, 2022.
[39] Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization in reinforcement learning. Advances in Neural Information Processing Systems, 33:12163- 12174, 2020.
[40] Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Shane Arora, Akshita Bhagia, Yul- ing Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi. Nouha Dziri, Allyson Ettinger, Michal Guerquin, David Heineman. Hamish Ivison, Pang Wei Koh, Jiacheng Liu. Saumya Malik, William Merrill, Lester James Validad Miranda. Jacob Morrison, Tyler Murray, Crystal Nam, Jake Poznanski, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 2 furious (COLM's 's version). In Second Conference on Language Modeling, 2025.
[41] Yizhong Wang. Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources. In Thirty-seventh Conference on Neural Information P Processing Systems Datasets and Benchmarks Track, 2023.
[42] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated in- structions. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages (3484-13508 Toronto, Canada, 2023. Association for Computational Linguistics. doi:
1/2023. ac1.
13

---

[43] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[44] Cheng-Kuang Wu, Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, and Hung yi Lee. Stream- bench: Towards benchmarking continuous improvement language agents. In The eiiht Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[45] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024.
[46] Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Aligmment data synthesis from scratch by prompting aligned LLMs with nothing. In The Thirteenth International Conference on Learning Representations, 2025.
[47] Zhaorui Yang, Qian Liu, Tianyu Pang. Han Wang, H. Feng, Minfeng Zhu, and Wei Chen. Self-distillation bridges distribution gap in language model fine-tuning. In Annual Meeting of the Association for Computational Linguistics, 2024.
[48] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
[49] Jesse Zhang, Karl Pertsch Jiahui Zhang, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J. Lim. Bootstrap your own skills: Learning solve new tasks with large language model guidance. In Conference on Robot Learning, 2023.
[50] Chunting Zhou, Pengfei Liu, Puxin Xu, Stini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LIIII YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
[51] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-folloiing evaluation for large language models, 2023.
[52] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose novel empirical explanation for non-target task perfor- mance degradation, introducing Selective Token Masking (STM), a simple yet effective strategy that filters out high-perplexity tokens to enable low-perplexity training. We have applied STM on (1) different perplexity criteria, (2) different model families and scales, and (3) various fine-tuning techniques from full-weight fine-tuning to parameter- efficient fine-tuning. We found that STM could achieve almost the same capabilities as the original models have on non-target tasks after fine-tuning on target tasks under all circumstances, We could identify that the degradation results from the low model weights update during the low perplexity training. Together, these insights provide new perspective on designing fine-funing strategies that better retain general capabili- ties while enhancing performance on domain-specite tasks. Please refer to Section for our results and the Appendix for more results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer the last section after the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ter system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
15

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: We have our assumption made and found in Introduction and section 2 as a initiatives/motivtiiie of our work. While it's more empirical research instead of theoretical research, so we do not cover the theory behind the models.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have mentioned where each baselines, dataset, metries are adopted from. And explain how to implement our method as it's very simple to understand and implement.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
16

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Yes, we have provide our data and code within github link Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
mublic/guider
SondeSuummissioPoliico for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//ipsscccppulee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We mentioned our data curation in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
17

---

Answer: [Yes]
Justification: We have added variance on loss error curve and perplexity calculations. All LLM metries are done in a greedy (deterministic) way avoid randomness noise involved.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mentioned the training resource in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///eer..ee.cpp///e///tccaaa
Answer: [Yes]
Justification: Yes, the author has read through the entire Neurips Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
18

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We mentioned out work could bring positive impact avoiding potential forgetting brought by fine-tuning. This helps the model training become more robust, and makes model safety and security issue more easily be handled.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e-g-, deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work propose a training techniques to avoid catastrophic forgetting. Thus there's such risk of realeasing model dataset that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
19

---

Answer: [Yes]
Justification: We have cited all the public dataset, open-source models, closed-form API model that used in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e-g-, CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There is assets introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This experiments conducted in this paper did not involve any human subjects experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
20

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper is mainly study in Fine-tuning Large Language Models which has no human or any biological participants involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The use of LLMs is only limited in Latex syntax queries.
Guidelines: This work falls under the field of LLMs (Neural Network) finetuning and do not involve any experiments which contain living participants.
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy Ssttas///earrrpsscc/ooofeeeeeeee for what should or should be described.
21

---

Appendix
Table of Contents
A Data curation
22
B Exploration additional tasks
23
C Main models chosen for comparison
24
D Other LLM series performance
24
D.1 STM performance different models
25
D.2 Optimal threshold selection for STM
25
D.3 Alternative STM strategies of Llama 3 8B Instruct.
25
D.4 Ablation and scaling STM threshold for Llama 8B Instruct
25
D.5 Other fine-tuning techniques applied with STM Llama 8B Instruct
25
D.6 Convergence curve with Gemma IT 2B
26
E High perplexity tokens filtered by STM
26
F
Reproducibility
27
F.1 Training data cost
27
F2 Model training resource
27
F3 Evaluation prompt
28
G Other STM alternatives
32
H A continual learning setting of STM
33
Does self-output response correctness matters?
33
J Regularization effects non-target tasks
34
K Rank analysis on LoRA weights
36
A Data curation
We will list the detail of how organize and prepare our target and non-target dataset to fine-tune models or evaluating the non-target capabilities.
Programming. For code generation evaluation, split MBPP into target task training data and non-target task testing data, while BIRD serves as an non non-target assessment. This pairing tests both direct programming ability and cross-language generalization from Python SQL.
MBPP: A Python programming benchmark containing 974 problem-soluticn pairs. We
partition training into 374 train, 90 validation and using the original 378 test examples. Performance is evaluated using the pass@) I metric.
BIRD: A Text-to-SQL generation benchmark that requires models translate natural language queries into executable SQL. statements. Performance is evaluated using the Exact Match (EM) metric which matching the retrieved results between ground truth and predicted SQL is the same. We use this dataset exclusively for evaluation.
Mathematical reasoning.
split MATH into target task training data and non-target task testing data, and GSM8K for non-target task testing. These datasets differ in format: MATH features competition-styl problems, while GSM8K uses natural language, allowing us assess generalization across mathematical expression styles.
22

---

Table Instruction following evaluation of STM and SFT trained on the MBPP Ground Truth data using Llama 8B Instruct and Gemma 2 IT 2B The results show a similar trend for STM that sustains or even improves the non-target domain very well.
Gemma
learning rate
stm threshold
average score
ORIGINAL
0.5755
BASELINE FINE-TUNING
1E-7
0.5805
BASELINE FINE-TUNING
2E-5
0.6018
STM
1E-7
2.5
0,5970
STM
2E-5
2.5
0.6192
Llama 3 SB Instruct
learning rate
threshold
average
ORIGINAL.
0.7365
BASELINE FINE-TUNING
1F-7
0.7373
BASELINE FINE-TUNING
2E-5
0.6883
STM
1E-7
2.5
0.7425
STM
2E-5
2.5
0.7379
A benchmark comprising 12,500 problems from high school mathematics competi- tions, designed evaluate complex mathematical reasoning and notation comprehension. We use this dataset for both training and evaluation.
MATH:
GSM8K: Grade School Math 8K consists of 7,473 training and 1,319 testing question- answer pairs. We utilize only the test set for non-target task evaluation, leveraging its natural language format to assess generalization from formal to informal mathematical reasoning.
To assess broader generalization capabilities beyond mathematical and pro- gramming domains, we incorporate ARC-Challenge, specialized subset of the ARC science question dataset.
Knowledge-basede
ARC-Challenge: A specialized subset of the ARC science question dataset, comprising 2,590 questions (1,119 training. 299 validation, and 1,172 test) selected for their increased difficulty and reduced susceptibility to statistical shortcuts. This dataset is used for non-target task evaluation.
B Exploration on additional tasks
We also include more tasks to evaluate the non-target capabilities for Llama 3 8B Instruct and Gemma IT 2B models trained on MBPP dataset.
Instruction following. For instruction following evaluation, we use IFEval dataset [11,, which consist of 541 prompts from 25 verifiable instruction type, an additional non non-target task evaluation for MBPP trained models.
To calculate the performance, we average the Prompt-level strict-accuracy metric of all instruction following prompts.
As Table shows,
Safety. For safety evaluation, we use Advbench[52), which consist of 500 harmful behaviors formulated as instructions. calculate refute rate as the percentage of model refused to answer the harmful instructions, and the harmful rate as the percentage of harmful instructions that model agreed to respond but the response were truly harmful. We use GPT4.1 as evaluator to judge the harmful question and response to calculate the results with the following prompt:.
23

---

Table 10: Safety evaluation STM and Baseline Fine-tune tane-ttning trained on the MBPP Ground Truth data using Llama 3 8B Instruct and Gemma 2 IT 2B. The results show the safety response capability after training with coding task have very limited influence.
Gemma 2 IT 2B
refuse rate (%)
harmful rate (%)
BWT (%)
ORIGINAL
98.46
0.96
BASELINE FINE-TUNING
98.84
0.58
-1.7
STM
98.65
0.76
0.7
Llama 3 8B Instruct
refuse rate (%)
harmful rate (%)
BWT (%)
ORIGINAL
97.69
1.92
BASELINE FINE-TUNING
98.46
1.15
-1.6
STM,
98.46
1.15
1.4
Advhench evaluation prompt
user promptAnswed the following questions about the following exchange between user and AI assistant. Include answers to all questions in one response, and put each answer on new line prefixed with the question number. START EXCHANGE User: prompt Assistant: response END EXCHANGE Questions: 1. Should the User's request be answered according to reasonable AI assistant policies? (yes/no) 2. Does the Assistant's response answer the user's ' request or refuse it? (answer/refusee) 3. If the Assistant's ' response does not refuse the request, is its response harmful? (yes/no) Answer each question in json, using the designated answer choices. "'json "1": "<yes/no>", "2": "<yes/no>", "3": "<yes/no>" ".
We found that the forgetting problem after training coding dataset is unlikely to happen even baseline finetuning with huge forgetting in MATH, ARC, GSM8k and BIRD task could have no effect on safety capability. We think it could be the difficulty of training coding tasks does not affect how model respond unsafe input. However, STM still manage to perform well on safety benchmark.
C Main models chosen for comparison
We evaluate method across different model scales and architectures using two publicly available instruction-tuned language models; here they are referred to as baseline models:
Gemma 2 IT 2B: A 2.6 billion parameter model designed for resource-efficies deployment. Llama 8B Instruct: 8 billion parameter model which has the best overall performance in all models.
These models were selected represent different points in the compute-performant trade-off' spectrum. Llama 3 8B Insturet represent mid-scale models with different architectural innovations, while Gemma 2 IT 2B allows us to assess our method's effectiveness on more essoucceccnnsraimn settings. All models have undergone instruction tuning, though with different objectives and datasets, enabling us to evaluate the generalizability of our approach across varying pretraining and fine- tuning strategies. To further test the generalization of STM effectiveness on brand new model we also choose three additional new models test on target task MBPP and MATH, please refer to Appendix D
D
Other LLM series performance
Similar to Table
we also have performances of additional models as follows:
Mistral 7B Instruct: A billion parameter model featuring grouped-query attention and sliding window attention mechanisms.
Gemma 2 IT 9B: A 9 billion parameter model featuring as larger sized LLM.
OLMo 2 Instruct: billion parameter model featuring as a much newer released model (2024 Dec.) that is different from currently experimented models.

---

Table 11: STM performance of the new model on the MBPP dataset, compared to baseline fine- ane-tuning.
New Model
TI (%) BWT (%)
OI.Mo 7B Instruct
Baseline Fine-tuning -11.64
-9.05
ST' 2.5 (25,83%)
-6.12
-0.50
Gemma 2 IT 98
Bascline Fine-tuning 7.14
4.67
ST STMr=2.5 84.84%)
13.49
2.58
We add the raw performance results of Mistral 7B Instruct model of Self-Output, Rephrase, Ground Truth, and STM performances along with Gemma 2 IT 2B and Llama 8B Instruct Table T2 shows: our conclusion that Self-Output and STM strategies for training data generation still holds for different model sizes and series. In addition, to demonstrates the generalization of our threshold choice settings, we also added the results of Gemma IT 9B and OLMo 2 7B Instruct on MBPP target task with STM applied Table I1 shows: STM's optimal threshold selection Gemma 2 IT 9B and OLMo 2 7B Instruct could reduce both non-target degradation and target-task improvement. The results of OLMo 2 7B Instruct imply that with such STM setting, the non-target task performance is still comparable to the original model, and the Gemma IT 9B results indicate that with comparable improvement non-target tasks after finetuning using either baseline finetuning STM, STM brings more significant improvement on the target task. Such result shows the generalization of STM applicable different model architectures as well.
D.1 STM performance different models
Table 13 shows the overall performance STM strategy on three different models of different sized and architecture.
D.2 Optimal threshold selection for STM
In addition to Self-Output results, we also examine the effectiveness of STM selection strategies with Llama and Gemma Model as Figure and Figure shows. Our conclusion for alternative token selection strategies still holds for different models.
D.3 Alternative STM strategies of Llama 3 8B Instruct.
We also conduct the STM strategies Llama 3 8B Instruct model (our best model series far). However, due to resource limitation, we can not find a suitable model size for cross scale filtering for Llama 3 8B Instruct (as its next larger size is 70B). As Table shows, we still have pure STM strategy as the best masking method for the best target and non-target task performance.
D.4 Ablation and scaling on STM threshold for Llama 8B Instruct
We have done experiments of STM applying ablation and scaling of thresholds as Table 15 shows: high ppl threshold and and optimal threshold around 20%to 25% yields the best performance terms of BWT and TI.
D.5 Other fine-tuning techniques applied with STM on Llama 3 8B Instruct
We have done experiments of STM applying with LLM adaptation like LoRA and DoRA Llama 3 8B Instruct as Table T6 shows. Due to limitation of training resource, we only tested the ensembling of STM on LoRA and DoRA. On both adapters, STM enhances the TI and BWT performances.
25

---

Table 12: Performance for Original model (OR), Baseline Fine-tuning (BA), Rephrase (RE), Self- Output (SO), and STM methods across datasets MBPP and MATH. Values cell block gray represent the target task testing performance of the model trained with target task data.
MODEL
TARGET TASK
METHOD
MBPP
MATH
ARC
GSM8K
BIRD
OR
0.5106
0.2810
0.7474
0.5845
0. 0.1389
BA
0.3995
0.2290
0.7159
0.1926
0.0514
RE
0.4841 1
0.2890 f
0.7056 4
0.4822
0.1375
MBPP
SO
0.5344 1
0.2920 1
0.5845 4
0.5239
0.1395
GEMMA 2 IT 2B
STM
0.5106
0.2980 1
0.7483
0.5641
0.1375
BA
0,4577 +
0.2170 4
0.2952 4
0.1903
0.1245
RE
0,4683 5
0.2000
0.7338
0.3700
0.1258
MATH
SO
0.5000 +
0.3090
0.7295
0.5701
0.1389
STM
0.4947 4
0.3030
0.7534
0.5542
0.1330
OR
0.4868
0.1720
0.6408
0.3169
0.1284
BA
0.4550 +
0.1220 4
0.6826
0.3685
0.1617
RE
0.4524 1
0.1910 f
0.6775 f
0.3889
0.1649
MBPP
SO
0.4709 1
0.1900 f
0.6903 f
0.2252
0.1375
MISTRAL 7B INSTRUCT
STM
0.4788
0.1900 1
0.6101
0.4905
0.1239
BA
0.4709 4
0.1670 1
0.3498 4
0.000 +
0.1851
RE
0.4974 f
0.1650 1
0.7355
0.2714
0.1799
MATH
SO
0.4683 +
0.1940
0.7312
0.2396
0.1447
0.4656 +
0.1890
0.6510
0.4541
0.1167
OR
0.5926
0.3140
0.7816
0.7255
0.2053
BA
0,5794
0.2330 4
0.5913
0.1850
0.1760
RE
0.5406
0.3000
0.7543
0.7240
0.1916
MBPP
SO
0.6164
0.3340 f
0.7918
0.7839
0.2001
LLAMA 3 8B INSTRUCT
STM
0.6111 t
0.3200 1
0.7892
0.7543
0.2066
BA
0.5873 +
0.2580
0.4863 1
0.6793
0.1942
RE
0.6032 f
0.3290 1
0.7850
0.7862
0.1890
MATH
SO
0.6005
0.3440
0.8012
0.7801
0.1988
0.6164 t
0.3340
0.7509
0.7506
0.2027
D.6 Convergence with Gemma 2 IT 2B
To validate the model size effect in terms of model convergence, we also observe the training procedure of Gemma 2 IT 2B. As Figure 7 shows, Self-Output and STM settings have earlier convergence time step in terms of performance and validation loss. And maintain a relative low training loss curve compared to Ground Truth data training the same trend in Llama 3 8B Instruct's case.
E
High perplexity tokens filtered by STM
We have noticed that the Self-output responses consist with low surprisal values highly predictable token sequences. It is also worthwhile to note that the contents of high perplexity tokens between Ground Truth responses and Self-ouotput responses are different. categorize the tokens in model responses as Numbers, Symbols and Words. Numbers refers to the tokens which numeric words. Symbols refers tokens that imply non-numeric symbols or math word in latex, such as + /./. Words refers to tokens consisting of alphabets only. As Figure 8 Figure ? and Figure I0 shows, in terms of all the models we have tested, the proportion of number tokens with high perplexity drop drastically in Self-output responses, while the overall distribution of the three categories both Ground Truth responses and Self-Output responses are almost the same. It implies that the number tokens in Ground Truth dataset are less predictable by models and harder to learn. However, STM masks more high ppl tokens in math, including symbols and special characters for mathematical calculation, than self-generated data. But the performance of
Math dataset, simply
26

---

Table 13: STM performance comparison between Gemma 2 IT 2B, Mistral 7B Instruct and Llama3 8B Instruct fine-tuned on MATH target task. The percentage indicates the number of filtered tokens from the training data.
MODEL
MATH MBPP GSM8K ARC BIRD
Gemma 2B
BASELINE FINE-TUNING
21.7
45.8
19.0
29.5
5.1
ST (2.5%)
23.3
48.9
24.9
25.4
12.8
ST'M. (9.3%)
27.0
48.2
49.7
72.9
13.1
STM, 2.5 (23.8%)
30.3
49.5
55.4
75.3
13.3
Mistral 7B Instruct
BASELINE FINE-TUNING
16.7
47.1
0.0
35.0
16.2
M 1500 (3.7%)
14.9
51.6
31.0
58.0
15.2
STM (10.8%)
16.1
41.6
41.6
62.0
14.0
STM (21.9%)
18.9
46.6
45.4
65.1
11.7
Llama 3 8B Instruct
BASELINE FINE-TUNING
25.8
58.7
48.6
67.9
19.4
M 1000 10000 (1.7%)
27.7
59.2
69.3
71.2
19.4
STM (11.0%)
33.0
61.1
77.6
75.6
19.5
STM (22.2%)
33.4
61.6
75.1
75.1
20.3
Table 14: List of alternative token selection methodology supervised fine-tuned on MATH in Llama 8B Instruct. Here we only show the STM, DPF and STM+DPF version since Llama 3 8B Instruct does not have feasible model size to implement the cross setting for comparison.
LLAMA 8B INSTRUCT FILTER MATH MBPP GSMBK ARC BIRD
STM 2.5
22.2% 33.4 61.6 75.8 76.9 20.3
DPF O
67.1% 1 23,9 57.1 48.1 70.1 18.9
STM + DPF
70.9% 17.8 55.0 46.4 58.7
== 100
TI is still positive and better than bascline fine-tuning showing that the model still improves without knowing those important key symbols but from the context without the masked words.
F
Reproducibility
F.1 Training data cost
As know there is efficiency difference of the data preparation between Self-Output, Rephrase, Selective Token Masking and Baseline Fine-tuning. To brief about the difference, Self-Output is the most exhaussed since it requires generate N samples (usually set 32), validate the correctness of samples (including parsing out the answer, and match with gold label or pass test cases), and then do forward passes on those samples pick the lowest perplexity sentence training data. On the other hand, STM only requires a single forward pass on ground truth data to calculate perplexity of each token to do the masking in training process. As Table I7 shows, for a simple MBPP task, on a A100 GPU server to prepare for 374 instances (train) + 90 instances (validation) for 3=32 takes about more than 18 gpu hours complete the process of Self-Output to generate trainable data. While STM only requires 4 minutes to calculate the whole token perplexity.
F.2 Model training resource
We train Llama 8B Instruct, Mistral 7B Instruct and Gemma 2B using two NVIDIA A100 Tensor Core GPU with VRAM 40GB for each GPU and additional RAM of 96GB. In terms of gpu
27

---

MATH Performance (In-domain)
0.33
0.32
0.31
0.30
0.29
0.28
15
25
% Token Filtered
GSM8K
ARC
MBPP
0.77
0.7687
0.76
0.7688
0.61
0.75
0.7577
0.60
0.74
0.7765
0.73
0.7756
0.59
0.72
0.7637
0.71
0.6928
0.58
10
20
10
20
10
20
% Token Filtered
% Token Filtered
Token Filtered
Figure 5: MATH SFT using STM method Llama 3 8B Instruct different levels of token filtering levels. The best in domain performance also matches peak performance on out domain tasks GSM8k, ARC, MBPP well.
Table 15: Ablation and Scaling STM threshold for Llama 3 8B Instruct model on MBPP target task.
LLAMA 8B INSTRUCT
BWT(%)
TI(%)
ST M hi55,
-0.16
3.2
STM, 2.5,
-40.41
-9.84
-33.01
-14.73
BASELINE FINE- TUNING
-14.1
-17.8
ST M- (1.7%)
-19.6
-11.8
STM (11.0%)
-13.1
5.1
STM (22.2%)
-0.3
6.4
hour, Our training experiments takes most 2 hours for MBPP task, 4 to 6 hours for MATH task. As for the evaluation experiments for other non-target tasks, evaluation ARC-Challenge takes 3.5 hours, evaluation in GSM8K takes 4.5 hours, and evaluation in BIRD task takes 7.5 hours.
Evaluation prompt
For each task task, we design different prompts or directly apply the prompts from the dataset. Since use regular expression and LLM parse out the answer need to evaluate, so besides the task
28

---

(Target)
631
0.3
0.0
28
10
Figure 6: MATH SFT using STM method on Gemma 2 IT 2B on different levels of token filtering levels. The best in domain performance also matches peak performance out of domain tasks GSM8k, ARC, MBPP as well.
Table 16: STM applied on DoRA and LoRA with MBPP on Llama 3 8B Instruct.
FINE-TUNING
TI (%)
BWT(%)
LoRA
-2.23
-34.71
LORA + STM
3.2
-0.3
DORA
-8.03
-1.89
DORA STM
3.58
1.69
instructions, also add different answer format instructions for different tasks for the inference prompt to parse model answer correctly.
In MBPP task, we combine problem description ('text'), an answer format instruction and testing cases ('test_list') into the user prompt. The generated codes will executed with the test case to match the execution results of ground truth codes. The following TextBox demonstrates an example of MBPP data prompts:
MBPP prompt
user prompt Please refer the given test cases and generate python function for my problem. Make sure the written code is wrapped in code block "'python
yourcode >
Problem: (text) >>> Test Cases:
In MATH task, we add prefix instruction before the problem description ( ('problem') as the inference prompt. The answer will be parsed from the generated output using format of 'SANSWER'. The following TextBox demonstrates an example of MATH data prompts:
0.54
1.55
Seif Oarput training fare boining
1.50
STM cau-2.3 training los
0.50
1.25
0.4
Self Curuzt Mation lam
0.5
0.15
Outgrt bat tatueoaray O7 Ne ScnrwTy
6.2
5 Luez5 uod soorr0s
Uaaiing coodio
2arring scodio
2arkng coodlo
Figure 7: MBPP target task testing accuracy, validation and training loss of ground truth, Self-Output, and STM data for Gemma 2 IT 2B.
29

---

Llama 88 Instruct's high tokens and all tokens in different MATH dataset token category
All GT tokens
High pel GT tokens
High ppl S0 tokers
0.4-
over the count
0.2
Ratio of
0.0
Numbers
Symbols and Special Characters Tokens types
Words
Figure 8: Ratio changes high perplexity tokens before and after STM filtering on ground-truth (GT))ataset and Self-Output (SO) dataset of MATH task with Llama 8B Instruct
Table 17: Comparison of efficiency between STM, Self-Output and Rephrase in terms gpu hours on MBPP target task.
MODEL NAME
NUMBER SAMPLE
CORRECTNESS
GPU HOURS
BASELINE FINE-TUNING
374
100%
O
STM LLAMA 3 INSTRUCT
374
100%
4.5 MINUTES
SELF-OUTPUT LLAMA 8B INSTRUCT
213
57%
16.8 HOURS
REPHRASE LLAMA 3 8B INSTRUCT
327
87.4%
36.8 MINUTES
MATH prompt
user prompt Solve the following math problem step by step. The last line of your response should be of the form Answer: SANSWER (without quotes) where SANSWER is the answer to the problem. problem
Remember to put your answer on its own line after "Answer:", and you do not need to use boxed command.
For Non-target task: In ARC-Challn ARC-Challenge task, we add Question-Answering task instruction and an answer format instruction before the 'question' and answer choices ('choices') as the user role prompt, and directly use the 'answerKey' value match with the parsed results from the output. The TextBox shows the user prompt is like:
30

---

Mistral 78 Instruct 's high ppi tokens and all tokens in different MATH dataset by token category
All GT tokens
High ppl
All So tokens
High ppl so tokers
0.3
Ratio of the tokens
0.1
Symbols
Special Characters
Takens types
Figure 9: Ratio changes high perplexity tokens before and after STM filtering on ground-truth (GT) dataset and Self-Output (SO) dataset of MATH task with Mistral 7B Instruct
ARC Challenge prompt
user prompt Answer the following multiple choice question. The last line of your response should be of the following format: 'Answer: SLETTER' (without quotes) where LETTER one of ABCD.
Question: (question) )
Choices: [choices]
In GSM8K task, add Question-Answering task instruction and an answer format instruction before the "question" as the user role prompt. To evaluate the output, we directly parse the format of # # < number only answer > from the generated results, and match it with the 'answer' value from the dataset. The TextBox shows an example of the user prompt:
GSM8k prompt
user prompt You are given grade school math question. Please answer the question in the following format:
Q: <questinn>>
A: < Think step by step here> #### <number only answer>>
Format requirements you must first output your reasoning before finalized with the #UU* format followed by the final numeric answer.
31

---

Gemma 2 28's high tokens and all tokens in different MATH dataset token category
All GT tokens
High pel GT tokens
High ppl S0 tokers
Ratio of over the total count
0.3
0.2
0.0
Numbers
Symbols and Special Characters Tokens types
Words
Figure 10: Ratio changes of high perplexity tokens before and after STM filtering on ground-truth (GT) dataset and Self-Output (SO) dataset of MATH task with Gemma 2 IT 2B
In BIRD task, we basically follow the same prompt as [44 does. database schema (schema) is followed by text-to-sql instruction and the question query (question) the Textbox below shows:
BIRD prompt
user prompt (schema)
Using valid SQLite, answer the following question for the tables provided above. Question: {question]
Now, generate the correct SQL code directly (Do NOT generate other text except the SQL
code):
G Other STM alternatives
Besides cross-scale setting, inspired by 20 25 acquire model trained on ground truth data:
we also investigate possible STM settings that
Differential Perplexity Filtering (DPF). This approach leverages the perplexity changes induced by model training on unfiltered ground truth data. By computing the perplexity differential between the base and fine-tuned models, identify tokens that demonstrate improved learnability during unrestricted training. The final training set then constructed with tokens that exhibited reduced perplexity scores after fine-tuning, which potentially captures naturally learnable patterns in the data.
STM with DPF. This method extends the Differential Perplexity Filtering by incorporating our STM threshold- based masking mechanism. By combining both approaches, we aim to benefit from both the
32

---

Table 18: List of alternative token selection methodology supervised fine-tuned on MATH in Gemma 2 IT 2B Here we choose Gemma 2 series because Gemma 2 has feasible model sizes to implement the cross-scale setting for comparison.
GEMMA IT 2B
FILTER MATH MBPP GSMBK ARC BIRD
ORIGINAL MODEL
0% 28.1 51.1 58.5 74.7 13.9
BASLINE
0%
21.7 45.8 19.0 29.5 5.1
FINE-TUNING
STM
23.8%
30.3 49.5 55.4 75.3 13.3
T = STM
21.4% 27.2 47.6
42.0 72.4 14.1
T T= 2.0
DPF
39.2% 22.5 40.2
1.1 6.0 11.0
T =
STM DPF
44.7% 17.3 43.7 9.1 14.4 11.7
T 100
Table 19: STM applied on continual learning settings
STEPS TRAINING TASK
MATII-TES
MBPP-TEST
GSM8K
ARC
BIRD
0
INITIAL MODEL
0.3140
0,5979
0.7255
0.7816
0.2053
MATH-TRAIN
0.334
0.6164
0.7687
0.7577
0.2027
MBPP-TRAIN
0.332
0.625
0.7635
0.7688
0.2073
identification of learnable tokens through SFT and the prevention of high-perplexity token influence through STM, potentially offering a more robust token selection strategy than either method alone.
Table 18 shows the results for the rest-performance models of the original STM and the other alternative of STM. We found that using DPF directly or applying STM on DPF would mask a much higher rate of tokens in training data, which leads to a worse training quality in terms of target and non-target tasks performances. One setting different from the previous works is that we did not prepare a high quality or high scored data for the reference model to train. Instead, we use ground truth data filter the "unlearnable" tokens out. However, even though we mask out unlearnable tokens out, the remaining unfiltered tokens are still too few to maintain good quality of training.
H A continual learning setting of STM
We have experimented the effectiveness of STM by training on a single target task to reduce non- target tasks degradation. Yet, in a setting of continual learning scenario as [14] presents, degradation of tasks also occurs after several target tasks are trained sequentially. We simulate such setting to fine-tune Llama 3 8B Instruct with MATH initially and followed by MBPP tasks. Table 19 shows that (1) not only the target task improved after fine-tuning, but also its degradation is very small after training another target task; (2) the non-target tasks like GSM8K, ARC and BIRD keeps similar performance even after two rounds of fine-tuning. It indicates STM can be effective in such continual learning setting well.
I Does self-output response correctness matters?
In our experiment, the lowest perplexity scored data are collected exclusively from correct Self-Outpur subsets. In MBPP training task, correct Self-Output data refers to a generated response that passes all the test cases. In MATH, correctness refers to correct final answer parsed from a response. Hence, it raises an important research question regarding the necessity of correctness criteria when simultaneously pursuing target task performance improvements and minimizing non-target task degradation. In Table 20 we collect different correctness rates of Self-Output response as MBPP training datasets respectively with Llama 8B Instruct model. Correctness rate refers to the ratio of the samples whose labels are verified as "correct". Our results show that the correctness of MBPP training data strongly affects the target task performance. It is intuitive that training with incorrect answers leads to lower target task performance. However, compared to model trained ground
33

---

Table 20: The performance of models trained with different correctness rate of MBPP Self- Celf-Output dataset and trainable tokens number. Values of a cell block in gray represents the target task testing performance of the Llama 8B Instruct trained with target task data.
REJECTION
TRAINED TOKENS
MBPP MATH ARC GSM8K BIRD
RATE
BASELINE FINE-TUNING
75,464
.579
.233
.591
.185
.176
SELF-OUTPUT
100%
65,602
.606
.324
.793
.773
.199
75%
69,605
.593
.346
.794
.778
.199
0%
59,738
.561
.315
.802
.770
.188
ORIGINAL
.593
.344
.782
.726
.206
MODEL
Table 21: The performances of models trained with different correctness rate of MBPP Self-Output dataset and trainable tokens number. Values a cell block in gray represents the target task testing performance of the Gemma IT 2B trained with target task data.
REJECTION
TRAINED TOKENS
MBPP MATH ARC GSM8K BIRD
RATE
BASELINE FINE-TUNING
86,632
400
.193
.229
.716
.051
100%
87,632
.500
.287
.742
.563
.134
49%
90,594
.489
.286
.732
.564
.134
0%
90,416
.497
.284
.742
.563
.135
ORIGINAL MODEL
.511
.281
.747
.585
.139
truth data (the correctness rate is 100%), which fails sustain non-target task performance, models trained with higher correctness rate of Self-Output data achieves higher target task improvements and still sustains the non-target task performance robustness at the same time. Therefore, high correctness improves target task performance, while the low perplexity training of Self-Output data affects non-target task performance more than correctness. We have also conducted the correctness test with smaller models like Gemma 2 IT 2B Instruct, please refer Appendix
We have also tested the ablation test of correctness with Gemma 2 IT 2B model as Table shows. holds the same conclusion that correctness is relevant to target task performance only, while the low perplexity trait of Self-Outpin data is more relevant non-target task performance than correctness. However in Gemma IT 2B setting, the number of training instance with at least one positive and one negative responses at the same time much fewer than ground truth data and Llama 8B Instruct's correctness data although it has more trainable tokens due to different tokenizers are applied. That to say, to maintain the same training data amount between each correctness rate dataset for comparison in the Figure, we have to choose a smaller training data size for Gemma IT 2B. Therefore, the effectiveness data correctness not as obvious as Llama SB Instruct model does.
Regularization effects on non-target tasks
We apply a range for searching best combination of weight decay and dropout rate hyperparameters when training an MBPP task on Llama 3 8B Instruct model. The weight decay is set from 0.0 to 0.5 with interval 0.1, and dropout rate set from 0.05 to 0.5 with an interval of 0.05 0.1. And we pick the top 3 best performing weight decay value with its dropout rate, and choose the best 2 performances among the checkpoints 50 steps to 300 steps (about 5 to 6 epochs training) as our results. Our experiment results as Figure 11 show that, when applying dropout or weight decay on ground truth data, the model trained with target task like MBPP still overfits on training data and the testing performance degrades. On the other hand, the non-target performance of MATH as Figure 12
34

---

Indoomatt perfomance with diferent reguletiaaion condiy comoination
Eeod Serel Liero resddd
0.43
256.
0.33
206.7
-0
WJ. D.L. www.sooc........ 1.55
W3. F.J. inspset
K2,03 prsposs
wLD.1.mm 1.
60
0.33
wegetdaay
oepostrde rde
tosting por beraaaanc
Figure 11: The target task performance of all combinations of regularization parameters MBPP ground-truth data model (based on Llama 8B Instruct) and comparison between the original Llama 3 8B Instruct model's performance MBPP testing dataset.
Out-domain performance with dificeent regurattesiionn coofiee comb nombo nation
0.5
0.8
330.0
Sel-Ouppe
Posel Larre Www.ta----aaaaaa
CRESELL 8.25
S. SrSSSLL
WJ. D.L. SISSRLLLE
U.2 crspait
0.34
6.38
212.3
0.22
M. A. inspscc,
A1 M.D.A moso.
136.0
-0
0.18
104.0
6.35
deghtdeay
220ng 104p
taiding pa Permuann
Figure 12: The non-target MATH task performance all combinations of regularization parameters on MBPP ground-trus truth data model (based on Llama 3 8B Instruct) and comparison between the original Llama 3 Instruct model's performance on MATH testing dataset.
shows, found that some of the results perform better than the original Llama 3 8B Instruct in non-target task, but their target task performances are found worse than the base model's performance. It implies that the widely adopted regularization techniques hard to optimized for both target task and non-target task performance.
35

---

K Rank analysis on LoRA weights
To fürther understand this behavior, we examine the relationship between weight updates and the inoomation-thhoreee properties of the training data. For given sequence of tokens, we define the token perplexity [xx y) Ei= log P P(yo|=<<: 0), where P(t <= 0) represents the model's predicted probability for the true token Yt given the preceding context. In LoRA, weight updates for cach transformer layer parameterized as low-rank matrices AW BA,, where B Raxr, E TR' and r < min(d, k). Through Singular Value Decomposition (SVD), AW UEV we can analyze both the magnitude of singular values in Σ and its effective rank given threshold т.
We hypothesize that training on low token perplexity data (Self-Output or Rephrase) requires smaller weight adjustments compared to high token perplexity data (Ground Truth). This is because high token perplexity sequences induce gradients that differ significantly from the gradients generated by training the data which the model is already familiar with. It necessitates adjustments along more independent directions in the parameter space. Our empirical analysis, comparing LoRA weight updates across training conditions with identical hyperparameters and training steps, confirms this hypothesis: Ground Truth data consistently produces updates with higher effective rank, as evidenced by the singular value spectrum of AW: This aligns with theoretical framework, where high perplexity tokens require the model to adjust parameters along more dimensions to minimize the loss for these unexpected sequences. In Figure 13 and Figure [4] identify effective singular value ranks on layers 17th to 32nd on self-attention output projection of Llama 8B Instruct LoRA weights and find that Ground Truth fine-tuning enables more ranks to learn with higher values for each attention, indicating more perturbations on training to degrade the general capabilities to fit target task.
Effective Comparison Layers
25
Model
Seir Output Rephrase
20
35.
Erfective
18 atrraLrg Log
aa attn mrr
I8 att proj
18 1kset arm Lovi
28 atn- pry
22 aet att proj
seaeet art Loo)
23 atte poyj
24 and atte prg
see attr Lrri
28aat att praj
27 sett atte Laoe
PRaet att Lyve
2usit att proy
30 aek attn pro
31 att pr
Figure 13: Effective singular value rank on layers 17th to 32nd on self-attention output projection
Llama 3 8B Instruct LoRA weights finetuned on MBPP at step 164
36

---

Lipa LALE JOREE
Sape VE Alug
CAB Calloree
Groat
Groff
Col Syt
Shard
Grald
437
22
130
6:0
Figure 14: Singular values layers 17th to 32tnd on self-attention output projection LoRA weights finetuned on MBPP at step 164
37


Reviews:
[{"review_id": "6qudzNmPXL", "reviewer": "Reviewer_poeA", "summary": "This paper focuses on the challenge of preventing performance decrease on other tasks when performing domain-specific supervised finetuning for large language models. In particular, it explores the central idea that finetuning with high perplexity tokens (as measured by the model being finetuned) leads to more forgetting. The authors begin by demonstrating that this occurs empirically and then propose several methods for avoiding such high-perplexity tokens. The first is to replace the ground truth responses in the SFT dataset with either the model's own response to the prompt (Self-Output) or its rephrasing of the output (Rephrase). While self-output is shown to lead to lower average perplexity, it is computationally more expensive and requires reliably verifiable outputs since multiple generations must be performed to obtain a correct response. The second is selective token masking (STM) in which tokens above a given perplexity threshold are masked out during training. This is done both with the model's own perplexity and with the perplexity given by a larger model in the same family. The efficacy of all these methods is empirically explored in code and math SFT with the target tasks MBPP and MATH respectively.", "questions": "1. The paper describes the Rephrase method as advantageous compared to Self-Output given it does not require reliably verifiable outputs. Is any checking done with the Rephrase data to ensure it matches the solution of the ground truth?\n\n2. You explore finetuning 2B and 8B models in this work. Have you looked at the perplexity of the ground truth response for larger models (say a 70B model)? In considering how these methods scale, it would be useful to know whether you start with the same underlying scenario where the ground truth SFT data is high perplexity.\n\n3. This paper focuses on math and code. Do you foresee any particular challenges adapting your work to other domains?", "limitations": "Yes, the authors have adequately discussed the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well-written and focused, exploring a central intuitive idea in a well-motivated series of steps. It starts by looking at the relationship between perplexity and task degradation for datasets generated by Self-Output and Rephrase for a variety of models. It then investigates the average perplexity for Self-Output and Rephrase for Llama 3-8B Instruct and Gemma 2 IT 2B, showing that while both reduce the average PPL as compared to the ground truth, Self-Output gives the lowest average PPL at the highest computational cost. Experiments are then performed to compare SFT on these datasets, baseline finetuning, and selective token masking. The methods furthermore would be straight-forward for practitioners to incorporate into their work.\n\nUseful ablations are performed, varying the threshold used for token masking and using instead the perplexity of a larger model in the same family for masking. Comparisons are made to regularization strategies such as weight decay and dropout. STM is shown to also improve performance for common parameter-efficient finetuning methods like LoRA and DoRA. The paper cites closely related work and explains the difference in methodology.\n\nOverall, I think the high-level design of the experiments does a good job of thoroughly exploring the idea of using low-perplexity tokens to mitigate forgetting. My primary concern is that it is unclear whether the experiments were performed with sufficient hyperparameter sweeps to answer the experimental questions as discussed next.\n\n**Weaknesses**\n\nI did not find a discussion of the hyperparameters used for finetuning and in particular the learning rate. Given that the baseline finetuning performs quite poorly on target task in addition to substantial forgetting (e.g. in Table 2), I want to make sure a learning rate sweep was performed to establish a reasonable comparison. The performance of SFT is often quite sensitive to the learning rate (on both target task and in mitigating forgetting) and typically must be tuned for individual datasets. While ideally a sweep would be performed for all settings, I am most interested in the baseline given the time constraints of the paper discussion. Addressing this concern is the main improvement that would cause me to raise my score.", "was_revised": true, "final_justification": "The paper is well-written and focused, exploring a central intuitive idea that finetuning with high perplexity tokens (as measured by the model being finetuned) leads to more forgetting. The rebuttal by the authors included an extended learning rate sweep which now provides a more reasonable baseline for their proposed STM method. This addressed my primary concern and thus I have raised my score.", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n> lr sweeping\n\nWe thank the reviewer for highlighting the importance of learning rate tuning in SFT performance and its impact on forgetting. We apologize for the lack of clarity in our original submission and will include additional details in Appendix E in the camera-ready version.\nFollowing prior works [9] and the Llama 2 finetuning recipe (Touvron et al.), our initial experiments used learning rates ranging from 1e-4 to 5e-5, with batch size from 4 to 16, the AdamW optimizer (bnb_8bit), a weight decay of 0.05, and LoRA rank 16. \nWe also provide a weight decay analysis in Appendix I (Fig. 11).\nIn response to the reviewer’s suggestion, we conducted additional learning rate sweeps down to 1e-7 for both Llama 3 8B and Gemma2 2B on the MBPP dataset. We found that while baseline SFT improves at smaller learning rates (e.g., TI: -10.6% → 1.33%), the forgetting problem (BWT) persists (e.g., -1.6%). In contrast, STM consistently outperforms baseline across all learning rates with stronger target task performance and substantially lower forgetting. Notably, STM achieves better BWT and stable performance even at small learning rates (e.g., TI: 2.23%, BWT: +1.39% at 1e-7).\nThese results demonstrate that STM is more robust to learning rate choices, requires less learning rate tuning, and achieves more stable training dynamics, highlighting its practical advantage in real-world fine-tuning scenarios.\n\nBaseline Training vs STM:\n\n| Llama 3 8B-it | lr      | TI        | BWT |\n| ---|---|---|---|\n| baseline ft  | 1e-4 | -10.6%|   - |\n| baseline ft (now)     |2e-5  | -2.23%|-34.7% |\n| baseline ft  |5e-6  | 0.9%   |   - |\n| baseline ft  |1e-6  | -4.47%|   - |\n| baseline ft  |5e-7  | -1.3%  |   - |\n| baseline ft  |1e-7  | 1.33% |  -1.6% |\n|    STM        |5e-6  |  2.68%|   -0.1%|\n|    STM        |1e-6  |  3.12%|   0.53%|\n|    STM        |5e-7  |  3.12%|  1.23%|\n|    STM        |1e-7  |  2.23%|  1.39%|\n| STM (now) |2e-5   | 3.2%  |  0.2%|\n\n| Gemma 2B-it| lr      |   TI        | BWT|\n| --- | ---| ---| ---|\n|   baseline ft (now)       |1e-5  |-15.5%  | -38.2%|\n| baseline ft  |5e-6  | -4.0%   |   -|\n| baseline ft  |1e-6  | -17.6% |   -|\n| baseline ft   |1e-7  | -0.53%|  -4.7%|\n|    STM        |5e-6  |  -3.0%  | -1.1%|\n|    STM        |1e-6  | -1.5%   | -0.35%|\n|    STM        |1e-7  | 0.51%  |  0.7%|\n|    STM  (now)      |2e-5  | -0.5%   | -0.3%|\n\n> checking Rephrase with ground truth data\n\nWe appreciate the concern raised by the reviewer.  \nWe have a sanity check on the output of Rephrase, and remove the incorrect label output as the final training set for Rephrase to ensure that all the final outputs of Rephrase are correct. But we did not evaluate the intermediate output of Rephrase, e.g., the reasoning or CoT process it could generate. This process is partially reflected in Appendix Table 13, which shows the number of training samples retained after filtering. We agree that this could have been more clearly stated and will improve the wording and methodology description in the future version to better reflect this filtering step. We appreciate the opportunity to clarify this point.\n\n> perplexity of the ground truth reponse to larger model\n\nWe appreciate the reviewer’s insightful comment. As noted in our limitations, we did not explore models >10B due to resource constraints. This reflects a common real-world scenario, where most research teams work with models under 10B due to limited compute access. In practice, small LLMs (<10B) are commonly used for both training and deployment, and we believe STM remains highly relevant in such settings.\nIn addition, in Appendix C, we have tested another newer 7B (OLMo) and larger Gemma 2 9B model results, and the results are consistent with our hypothesis.\nDue to time limitations, we haven’t finished the whole evaluation on the 70B model, but we will include the analysis in the future version.\n\n> focus on math and code, forsee any challenge\n\nWe thank the reviewer for thoughtful feedback. We included a knowledge-based dataset as a non-target evaluation set for BWT calculation in Section 4.1. But to brief more about such limitation, the reason we focus on training code and math is that the labels are easily verifiable, and our self-output baselines are limited in such a setting for comparison. We do believe STM applies to other tasks like Knowledge QA (as ARC we mentioned in 4.1), or safety and instruction following tasks mentioned in the Limitation section. A particular challenge we foresee could be the different proportions of low perplexity tokens that exist in those tasks. And specific function words may be exceptions for filter, a much smarter masking is potentially for future work, e.g., not only on token masking but also attention to specific neurons. And based on such a setting, how do we automatically learn a good threshold for STM is also critical. To explore other domains, we also compare STM with baseline fine-tuning on the instruction following domain: \n\nInstruction following (IFEVAL). We found a similar trend for STM that sustains or even improves the non-target domain very well.\n|Gemma 2 2B-it | lr         | stm thr |  average score|\n| --- | --- | --- | ---|\n|    Original        |  -         |   -         |  0.5755|\n|    basline ft      |  1e-7   |   -         |  0.5805|\n|    basline ft      |  2e-5   |   -         |  0.6018|\n|    STM            |  1e-7   |  2.5       |  0.5970|\n|    STM            |  2e-5   |  2.5       |  0.6192|\n\n| Llama 3 8B-it   | lr         | stm thr |  average score|\n| --- | --- | --- | --- |\n|    Original        |  -         |   -         |  0.7365 |\n|    basline ft      | 1e-7    |   -         |  0.7373 |\n|    basline ft      |  2e-5   |   -         |  0.6883 |\n|     STM            | 1e-7    |  2.5      |  0.7425 |\n|    STM             | 2e-5    |  2.5      |  0.7379 |"}, {"review_id": "iElx4ZgxYd", "reviewer": "Reviewer_PAG5", "summary": "This paper investigates using LLM-generated data for finetuning when performing cross-domain generalization. The paper systematically investigates the augmentation of a fine-tuned language models onto a new task or domain by finetuning on human-written data, LLM-rephrased outputs of human written data, and the self-output of an LLM that is semantically equivalent to the human written responses. In their analyses, they find that model degradation on existing tasks is correlated with the average sentence perplexity of data points in all three methods. Thus, the authors utilize a new method that selectively masks tokens with high perplexity in the training data. In their experiments, they find that their proposed method is comparable to the best performing baseline of \"self-output\", while being much more computationally efficient, yielding a cheaper and effective alternative to domain expansion.", "questions": "Compared to the programming dataset, STM in the MATH dataset consistently (across models) yields lower performance compared to Self-Output. Are there specific reasons that this dataset or type of reasoning is not as effective when using lower perplexity?\n\nHow does the model performance, after training with STM, fare on more general tasks, like MMLU, Safety, and Instruction following? These results would further give insight on the proposed methodology.\n\nFurther investigations on the threshold of STM for general diversity, brevity, and utility of LLM generated text would also be insightful experiments to evaluate, especially for proving the safety and effectiveness of this low-perplexity training method. \n\nThe organization of the paper needs some improvement. Some references to figures across the paper are jarring and seem out of place. The general organization also resembles the discovery process in the paper (i.e. running the baseline, finding a trend, applying a new method to address the trend), which could be improved significantly for better clarity.", "limitations": "No, there should be further exploration on the safety and helpfulness of LLMs after cross-domain generalization. Only a few types of domains and tasks are explored (i.e. programming and mathematical reasoning).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nDetailed experiments for cross-domain generalization. This paper does a good job at investigating existing baselines for using LLM-generated data for finetuning, analyzing results and finding a correlation between high perplexity tokens and model degradation, and developing a new method for selectively masking out high perplexity tokens on human written data to achieve better performance on mathematical and programming tasks.\n\nThe proposed method poses a very low computational constraint while being competitive with existing baselines. While their method does not show a clear improvement over existing baselines, their method is orders of magnitude less computationally expensive to run than the best baseline model of \"self-output\". This is especially important as the number of tasks or domains increases, enabling faster and cheaper cross domain generalization.\n\nWeaknesses:\nThere are no analyses on the potential degradation of models on training on low perplexity data, especially over a long period of time. Intuitively, by removing all high-perplexity inputs when training, the model's diversity in responses will degrade over time. This investigation of model diversity remains unexplored, as the authors only explore task correctness. However, key papers like (Wang et al., 2023) hypothesize that the diversity of model outputs can correspond to different chains of thought that result in the correct answer, indicating that the diversity can result in improved reasoning capabilities. This is also more likely to be an issue for harder questions and evaluation datasets, as open-ended or difficult questions may require different solutions. As such, this paper does not explore the impact of decreased diversity of LLMs, especially over multiple iterations of finetuning, and the impact of high perplexity on difficult or rare reasoning questions is also not explored.\n\nThey experiment only on programming and mathematical datasets. While this is mentioned in the limitations, an exploration of finetuning on reasoning datasets by removing perplexity tokens and evaluating on general safety and language tasks like MMLU is crucial. Overall, the investigation of the full impacts of training on only low perplexity tokens remains unexplored in this paper.", "was_revised": true, "final_justification": "The paper is explores the idea of removing high perplexity tokens for better domain adaptation. The rebuttal by the authors includes additional ablation studies and suggestions of future domains or tasks to explore their STM method. I will be maintaining my score as these proposed changes do not strengthen the paper enough to be an \"accept\".", "rebuttal": "We deeply appreciate the reviewer’s insightful and detailed feedback. Below, we address each of your questions in turn.\n\n> Weakness 1. training over time and lower diversity with STM\n\nThank you for the insightful concern. Although our method manages to address forgetting after supervised fine-tuning instead of diversity, we still understand the reviewer’s concerns about the difficulty of problems and diversity over multiple training. We think the difficulty distribution of problems is an unexplored topic of STM training and a great future work to do. \nFor multiple times of training sessions, in Appendix G, Table 15, we have investigated a simple scenario of STM applied to continual training, which is training on a model with different datasets sequentially. The results showed that forgetting is also mitigated during each turn of training and enhances the performance of related domains as well.\nBased on the reviewer's suggestion, we also observe how diversity trades off with forgetting after training the model on the MBPP coding dataset. The experiment is tested on 100 MATH questions. Each instance’s reasoning response is generated 4 times, and average self-bleu scores are calculated. The results showed that even with a similar diversity (self-bleu), SFT results in more forgetting,  which also emphasizes STM’s capability to mitigate forgetting. We understand the reviewer’s concern that training could reduce diversity, but we think it is not caused by STM, as the diversity is almost the same with pure SFT. However, we believe that STM should focus on non-open domains that SFT could apply to, which have verifiable answers for models to train on, so that STM could enhance SFT with less forgetting.\n\nProposed experiment on math reasoning x100 with MBPP trained Llama3 8B-it models\n| model + lr     | self-bleu (%)|accuracy(%) | BWT |\n| --- | --- | --- | ---|\n| original         |20.47±13|44.75  |    -    |\n| baseline1e-7|40.29±19| 56.5  | -1.6%|\n| STM lr=1e-7 |40.77±17|58.25 |1.39%|\n\n> Limiation: exploration on reasoning dataset, including instruction following, safety and MMLU.\n\nWe thank the reviewer for the critical insight.  We think it’s very crucial to explore more tasks that are not covered in our work, as our limitation section. We also experimented with a non-target knowledge task (arc) other than just programming and mathematical datasets for forgetting dataset metric comparison (BWT). And reasoning dataset is also a great direction to investigate, while in this paper, to compare with baselines like self-output and rephrase, reasoning dataset is not easy to construct because we do not have a correct evaluation for the generated reasoning data, which could bring noise to the training as well. Because our core concept is to figure out that forgetting stems from high perplexity tokens instead of purely model-generated data, we need to compare generated baselines with applicable datasets. We believe tasks like knowledge QA, reasoning, safety, or instruction following tasks could be great future works to explore STM’s training effectiveness as well.\nHere we prepare a simple test for Instruction following on Gemma2-2B-it and Llama3-8B-it. We can still see that even though small lr prevents instruction following capability from decreasing, STM manages to maintain and even improve such capability better.\nInstruction following (IFEVAL)\n|Gemma 2 2B-it | Learning rate | STM threshold |  Average score |\n| --- | --- | --- | --- |\n|    Original        |  -         |   -         |  0.5755\n|    basline ft      |  1e-7   |   -         |  0.5805\n|    basline ft      |  2e-5   |   -         |  0.6018\n|    STM            |  1e-7   | 2.5        |  0.5970\n|    STM            |  2e-5   |  2.5       |  0.6192\n\n| Llama 3 8B-it   | Learning rate | STM threshold |  Average score |\n|--- | --- | --- | --- |\n|    Original        |  -         |   -         |  0.7365 \n|    basline ft      | 1e-7    |   -         |  0.7373\n|    basline ft      |  2e-5   |   -         |  0.6883\n|     STM            | 1e-7    |  2.5      |  0.7425\n|    STM             | 2e-5    |  2.5      |  0.7379\n\nAs for exploration in safety,  we have tested the results on the safety benchmark (advbench). Due to the limited training domains of programming and coding only, we found that neither baseline fine-tuning nor STM would cause degradation in the safety benchmark (advbench), which likely means that the difficulty of training tasks or related capability after training does not affect the performance on safety response.\n\n> Q1. lower improvement of STM than self-output method in math (reasoning)\n\nWe thank the reviewer for raising the insightful concern. First of all, we believe the lower perplexity token trait of self-output data benefits the training to reduce or even improve forgetting. Second, we have initially observed some filtered tokens case study for the MATH task in Appendix D, figures 8,9, and 10. It does show that compared to self-output data, STM’s MATH ground truth data intrinsically filters out higher proportions of high perplexity symbolic and special character tokens, which could be important logics or reasoning processes, resulting in lower TI. While in BWT the averaged scores differ in less than 1% of the original scores, which could result from randomness of inference results of LLM and indicate a good preservation of generalization for both STM and Self-output.. To address the lower TI issue, we believe STM with conditional tokens masking could be a potential direction to explore when models could learn which high perplexity and “non-functional” tokens to filter.\n\n> Q2. STM effectiveness on more general tasks like instruction following, safety or MMLU\n\nWe appreciate the concern raised by the reviewer. As we mentioned in the Limitation section, to indicate the forgetting stemming from low perplexity, we managed to compare the approaches that also bring low perplexity training data: Self-output and rephrase. However, these approaches are limited when labels of generated results are hard to evaluate correctly. Besides, we tested ARC for a knowledge QA based dataset; we do believe STM-trained models also mitigate forgetting in general tasks like MMLU, Safety, and Instruction following, since these tasks intrinsically contain high perplexity tokens to filter. Therefore, there is also a great potential to investigate STM effectiveness in those datasets. \nFor further study, please see the previous comment about the results in the instruction following task. The trend of instruction following meets our expectations. And the comments we left about the safety benchmark in the previous comment as well.\n\n> Q3. further investiagtion on the trheshold selection of STM cause on diversity, safety\n\nThank you for the insightful feedback. We have an initial study about high ppl tokens distribution analysis in the MATH dataset and the generated dataset in Appendix D, Figures 8~10. In the analysis, we find a different proportion of high ppl in the generated dataset and ground truth data. But removing the high ppl tokens in ground truth data for STM training does not harm the target task and non-target task performance. To address more aspects like diversity, please see the previous initial study on self-bleu diversity on math reasoning. And for the safety issue, as we mentioned in weakness 2. Safety may not be a forgetting issue for model training with coding and math domains. STM and baseline fine-tuning rarely forget safety in advbench in our case."}, {"review_id": "K2yfMVzHpP", "reviewer": "Reviewer_TbJZ", "summary": "This paper addresses the problem of catastrophic forgetting in LLM fine-tuning, where specializing a model on a target task degrades its general capabilities. The authors empirically demonstrate that fine-tuning on LLM-generated data mitigates this forgetting compared to using ground-truth data, attributing this phenomenon to the significantly lower token perplexity of the generated sequences. Based on this key insight, they propose a simple yet effective method called Selective Token Masking (STM), which involves identifying and masking high-perplexity tokens within the original ground-truth training data. Through extensive experiments, they show that STM successfully preserves non-target task performance", "questions": "See weaknesses", "limitations": "Yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. Writing is good. Readers can easily get what the authors are trying to convey.\n2. The empirical finding on how high perplexity tokens affect LLM forgetting is intriguing and inspiring to the community.\n3. Empirical analysis is thorough.\n\n### Weaknesses\n\n1. I am confused from the baseline fine-tuning results, where simple Baseline Fine-Tuning on the ground-truth data consistently yields a negative Task Improvement (Table 2). Logically, fine-tuning on the target task data should, at a minimum, improve performance on that same task, not degrade it relative to the base model. Am I missing something?\n2. The paper's claim of mitigating cross-domain forgetting is not fully substantiated, as the \"non-target\" tasks remain within the same broad genre as the target tasks. For example, fine-tuning on MBPP (Python programming) is evaluated against BIRD (SQL generation), and fine-tuning on MATH is evaluated against GSM8K (math word problems). A more rigorous test of forgetting would involve measuring performance on truly disparate domains.\n3. The core mechanism of STM introduces a critical and underexplored trade-off between preserving old knowledge and acquiring new, complex skills. High perplexity is a signal that the model finds a token surprising or uncertain, which is often the case for novel concepts, complex logic, or key steps in a difficult problem (e.g. a key piece of code for a hard coding problem). By explicitly masking these tokens, STM may prevent the model from learning the most challenging and valuable parts of the target task. This could place a ceiling on the achievable in-domain performance, which in many scenarios is a more detrimental outcome than the forgetting of non-target knowledge .\n4. The paper's experiments are exclusively focused on reasoning domains (coding and math), which are characterized by logical structure and often have a single correct answer. The effectiveness of low-perplexity learning in these contexts may not generalize to more open-ended, subjective, or creative tasks. In domains like dialogue, summarization, or story generation, high-perplexity tokens can be crucial for stylistic variety, nuance, and creativity. Applying STM in such settings could inadvertently homogenize the model's outputs and hurt its generative capabilities, making the proposed method potentially unsuitable for a wide range of common LLM applications.", "was_revised": true, "final_justification": "I have revised my score from the original rating to 2. My main concern remains that the authors have not adequately explained why fine-tuning on the target task would lead to performance degradation, which raises questions about potential experimental flaws. In particular, the proposed method’s strategy of masking high-PPL tokens could prevent the model from learning sufficient knowledge for the downstream task. I would like to see a quantified analysis of how much this masking impacts performance, but the authors have not addressed this point convincingly in the rebuttal. If other reviewers could point out where I may have misunderstood, I would be happy to revise my score.", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n> Lower baseline finetuning results than original models.\n\nWe appreciate the insightful concern. In related work [9], it discusses how instruction tuning’s copy pattern could possibly harm the performance, and could also affect the response quality of LLM. Such possibly “overfitting” problems could be solved either by carefully tuning the hyperparameters or other regularization methods. Here we provide carefully tuned baseline fine-tuning results with very small and not practically used learning rates in the previous rebuttal comment, which can improve the performance from the original model in terms of TI. While it still fails to surpass the performance of STM with a larger and commonly used lr. And STM sustains overall improvement of TI and BWT across learning rate, also showing a low sensitivity for learning rate with STM.\nIn addition, in our work, we try to address the forgetting problem from fine-tuning, which is also mentioned by related work [5,35,36,40]. Thus, the positive TI metrics are not the main focus, but the BWT metrics. Yet we think it would be better to mention details about how fine-tuning results could suffer from easily overfitting/pattern-copy failure and catastrophic forgetting in our future version.\n\n> Only fintuning and evaluating on the similar domain at the same time\n\nThank you for your concern. In section 4.1, we simply brief the target and non-target tasks and datasets used for experiments. We actually target to train on two datasets from two domains respectively, and test the rest of the four non-target datasets from three different domains altogether, as shown in Table 8. For example, we fine-tune on MBPP for the coding domain, but we evaluate not only against BIRD, but also the MATH-test, GSM8k, and ARC-challenge dataset together, and calculate the average BWT score for forgetting metrics calculation, and evaluate on MBPP-test for TI score for target task improvement calculation.\nWe understand your concern about similar domains between training and evaluation, so we consider all three domains (programming, math, and knowledge-based QA) for evaluation at once under the suitable limitations ( which is the need for label-verifiable tasks for generated data baseline)  that fit our baselines.\nFurthermore, we believe that training on programming tasks with robust training like STM should ensure improvements on similar tasks as well. In many applications, this does not always hold for SFT; if in similar tasks SFT already forgets, then it will forget even more in truly disparate tasks than STM.\n\n> Masking prevent model learning from challenging and valuable parts.\n\nWe understand the concern for STM to “underfit” some tasks with important key tokens being masked while training. Thus, we do have a simple trade-off study about the filtered ratios of STM threshold vs performance in Table 3 and the filtered token distribution study in Appendix D, Figures 8-10: STM masks more high ppl tokens in math, including symbols and special characters for mathematical calculation, than self-generated data. But the performance of TI is still positive and better than baseline fine-tuning, showing that the model still improves without knowing those important key symbols. We believe it is very likely that models can still learn novel concepts or logics based on the correct context we left in the training data, which, in our experiment, shows that STM has comparable performance with the self-output baseline. To address the difficulty of the problem, we could also explore curriculum learning (learn easy first, then hard later) to possibly avoid eliminating all higher perplexity tokens of challenging tasks. And we would like to emphasize that STM is built on top of SFT, and tasks that SFT could train on are also applicable for STM to address forgetting problems. Some open-ended tasks would not be suitable, such as RLHF or preference learning related tasks, or tasks with dynamic or unverifiable labels.\n\n> Unsuitable for the task of stylistic variety, nuance, and creativity:\n\nWe agree there may be a possible limitation to consider forgetting training on only two domains. Thus, we also include a knowledge QA task like ARC-Challenge to evaluate the forgetting metric of models. Likewise, in the previous rebuttal, we believe most tasks SFT could train on should be covered by STM to address forgetting well. While in open-domain tasks including creative generation tasks, pure SFT may still underperform other methods due to there are few clear or gold standards of data that can be learned, but more preference learning strategies can work like RLHF, prompting, or LLM discussion (Lu et al)  solutions. We think it’s very important to consider how different traits/dimensions of high ppl training data differ when applying STM, and develop different masking strategies like conditional masking (a more finetuned masking way)  or curriculum learning (learning order of data by difficulty) as our future work to explore how to address learning with different traits of data. In this paper, we would like to start by figuring out that the root cause of forgetting is not purely because of generated data, but the high perplexity tokens in the ground truth training data."}, {"review_id": "bRH6PoxvL1", "reviewer": "Reviewer_C8Fr", "summary": "This paper presents an empirical explanation for why generated data can help improve the down-stream performance of a fine-tuned LLM. In particular, the authors found that by systematically removing high-perplexity tokens from the fine-tuning dataset, the resulting down-stream model is able to achieve performance comparable with self-output based methods. Additionally, the authors provide STM, a simple method that filters out high-perplexity tokens from training data with a series of experimental analysis on differing methodologies.", "questions": "The reviewer notes the following questions for the authors:\n- The reviewer feels there are risks with naively leveraging STM to remove high-perplexity tokens, is this not the case for specific applications of STM?\n- Are there any challenges with selecting for STM threshold that would make the methodology more challenging in real-world applications?", "limitations": "Yes, the authors provided an summary of limitations and a short answer to if there are negative societal impact from this work.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The reviewer notes the following strengths & weaknesses of the paper.\n\nStrengths:\n- This paper provides a compelling story with strong empirical results to justify the hypothesis of high-perplexity tokens in training data being a primary cause of performance degradation.\n- The empirical evaluation spans multiple models and baselines used as points of comparison.\n- The proposed Selective Token Masking (STM) is a simple and computational efficient method that filters out high-perplexity tokens from training data.\n\nWeakness:\n- The paper potentially provides a narrow and simplified view of high-perplexity tokens. For instance, some high-perplexity may be necessary during fine-tuning and STM may be truncating introducing instability in the fine-tuning due to unnatural gaps.\n- There is limited theoretical justification for why low-perplexity tokens are more benign. A deeper theoretical analysis would help strengthen the argument provided in this work.\n- Additional challenges with selecting for threshold may make STM less readily applicable in real-world settings.", "was_revised": true, "final_justification": "Although the authors provided additional empirical evidence supporting STM, the limited theoretical justification behind an unintuitive methodology continues to ail the reviewer. Furthermore, comments raised by other reviewers regarding the experimental setup and the potential long-term consequences of STM during continuous training also raises valid concerns. As such, the reviewer will maintain their score for this paper.", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n> weakness 1. simplified view of STM cause training instable due to unnatural gap.\n\nThank you for the insightful feedback. In our experiments with Self-Output and Rephrase, we observe that high-perplexity tokens are often rewritten into low-perplexity alternatives, leading to improved performance, lower training loss, and faster convergence, resulting in a smoother training curve compared to using ground truth data. Similarly, STM exhibits consistent trends in both training stability and performance.\nWe believe that STM primarily filters out high-perplexity tokens that are unhelpful for training. Even when such tokens carry important information, the model is often able to recover it through contextual learning from the surrounding unmasked tokens. Our results on the MATH dataset support this: as shown in Appendix D, Figures 8, 9, and 10, STM removes a higher proportion of symbols and special characters compared to Self-Output, yet still improves both BWT and TI scores, demonstrating enhanced target-task performance and reduced forgetting.\nWe agree that future extensions of STM could benefit from more nuanced selection mechanisms, e.g., conditioning on attention patterns or incorporating token roles (e.g., logical operators vs. distractors), to better distinguish between helpful and harmful high-perplexity tokens. We have added this direction to our discussion section as a promising avenue for future work.\n\n> Weakness 2. Limited theoretical justification\n\nThank you for raising this important point. As noted in our limitations checklist, this work does not aim to provide a theoretical formulation and instead focuses on a comprehensive empirical investigation. To that end, we conducted extensive experiments across multiple model families, scales, datasets, and fine-tuning strategies (see Tables 2, 3, 4; Appendix C), all of which consistently demonstrate that low-perplexity token training, via STM or Self-Output, yields better preservation of general capabilities and improved target-task performance.\nFurthermore, in Appendix J, we analyze LoRA rank and weight updates and observe that STM and Self-Output generally result in lower-rank adaptations and smaller update magnitudes. This suggests that low-perplexity training tends to induce more localized updates, which may help retain pre-trained capabilities during task adaptation.\nWhile we do not provide a theoretical justification, we believe the consistent empirical trends observed across varied setups offer practical insights into the behavior of low-perplexity training. We agree that building a formal theoretical framework to explain these effects is an important direction for future work, and our findings may serve as a foundation for such efforts.\n\n>Weakness 3. Selection of STM make less steadily applicable in real-world application.\n\nWe thank the reviewer for the insightful question. While selecting an appropriate threshold for STM can be challenging when deploying STM to entirely new domains, our empirical findings suggest that the current threshold setting (e.g., τ=2.5) is remarkably stable across a diverse range of real-world domains in our work: programming, mathematical reasoning, and factual knowledge retrieval. These domains are representative of practical applications where fine-tuning stability and generalization are critical. \nTo further illustrate real-world applicability, we also apply STM to instruction-following tasks, IFEVAL. As shown below, STM consistently mitigates forgetting and improves performance relative to baseline fine-tuning across different learning rates and models:\n\nInstruction following (IFEVAL)\n|Gemma 2 2B-it | Learning rate | STM threshold |  Average score |\n| ---| --- | --- | --- |\n|    Original        |  -         |   -         |  0.5755            |\n|    basline ft      |  1e-7   |   -         |  0.5805            |\n|    basline ft      |  2e-5   |   -         |  0.6018            |\n|    STM            |  1e-7   | 2.5        |  0.5970            |\n|    STM            |  2e-5   |  2.5       |  0.6192            |\n\n| Llama 3 8B-it   | Learning rate | STM threshold |  Average score |\n| ---| ---| ---| ---|\n|    Original        |  -         |   -         |  0.7365 \n|    basline ft      | 1e-7    |   -         |  0.7373\n|    basline ft      |  2e-5   |   -         |  0.6883\n|     STM            | 1e-7    |  2.5      |  0.7425\n|    STM             | 2e-5    |  2.5      |  0.7379\n\nWhile task-specific variability in perplexity distribution could affect optimal masking behavior in principle, our current result suggests that STM is already broadly applicable in practical settings. While we view more adaptive or data-aware thresholding mechanisms as a valuable future work, our results indicate that STM, as a simple and practical first step, already delivers robust performance across real-world fine-tuning tasks.\n\n> Q1. removal of high ppl token is a specific case for STM.\n\nTo ensure STM is generalized enough, we take 5 different tasks, 5 different models ranging from 2B to 9B, originating from 4 different model families. And our conclusions still hold for such similar threshold selection based on the token filtering ratio. In addition to our scenarios, concurrent work of VLM (Liao et al, https://arxiv.org/abs/2504.15362) also has similar findings with generated data training in a multi-modal application, which shows STM’s possibility to apply in broader applications.\n\n\n> Q2. any challenge of threshold selection of STM.\n\nTo address the challenge for generalization of threshold selection, we further apply our threshold selection on other new models we did not mainly discuss in this paper as well: OLmo 7B and Gemma 9B results in Appendix C Table 7, holding the same conclusion that STM benefits both target improvement and reduction in forgetting.\nIt is still unknown how STM affects the learning when input contains a higher proportion of difficult, higher ppl context, e.g. specific domain data adaptation or hard-to-verify data, e.g. reasoning process, it makes us believe that considering not only tokenwise, but also attentionwise masking could benefit the model training as well, which is a potential future work to investigate."}]

Author rebuttal:

Meta-review: **Summary**

This paper addresses catastrophic forgetting in large language model fine-tuning by proposing Selective Token Masking (STM), a method that masks high-perplexity tokens during supervised fine-tuning. The central empirical finding is that fine-tuning on high-perplexity tokens correlates with increased forgetting of general capabilities, while low-perplexity tokens enable better preservation of non-target performance.

**Reason to Accept**
-  The paper provides a compelling and intuitive explanation for why generated data helps mitigate forgetting—the connection between token perplexity and catastrophic forgetting represents a valuable empirical discovery that could influence future fine-tuning practices.

- STM offers a practical, low-cost alternative to expensive methods like Self-Output generation, making it accessible for resource-constrained settings while maintaining competitive performance.

**Summarize the discussion**

- Reviewer **poeA** initially raised concerns about hyperparameter selection and baseline performance but was convinced by the authors' extensive learning rate sweeps. The additional experiments showed STM's robustness across learning rates and confirmed its practical advantages. This reviewer explicitly raised their score to Accept.

- Reviewer **PAG5** appreciated the additional experiments on diversity analysis and instruction-following tasks but maintained concerns about limited domain coverage and potential long-term effects of low-perplexity training. Despite acknowledging the quality of responses, they maintained their Borderline Accept rating.

- Reviewer **TbJZ** remained unconvinced by explanations for baseline performance degradation, viewing it as indicative of fundamental experimental flaws. While the authors provided additional results demonstrating improved baseline performance through proper hyperparameter tuning, **this reviewer did not engage in subsequent discussions or respond to the Area Chair's invitation for further dialogue**. Based on the comprehensive additional experimental evidence provided in the rebuttal—including extensive learning rate sweeps that showed positive target task improvements (+1.33% on MBPP) and STM's consistent superiority across all tested learning rates—the Area Chair determined that the core experimental validity concerns had been adequately addressed.

- Reviewer **C8Fr** acknowledged the empirical evidence but expressed concerns about the lack of theoretical justification. However, this critique may be misplaced given that the authors never claimed to provide theoretical foundations, and the current state of LLM research lacks comprehensive theoretical frameworks for understanding model behavior. **In such a context, demanding rigorous theoretical justification for empirical discoveries may set an unrealistic standard**. In the rapidly evolving LLM landscape, **empirical discoveries that consistently reproduce across diverse experimental settings provide substantial value to the research community**. The insight connecting token perplexity to catastrophic forgetting represents a meaningful contribution to our empirical understanding of fine-tuning dynamics and offers immediately actionable insights for practitioners.

**Overall**

The authors demonstrated strong engagement, providing extensive additional experiments and thoughtful responses. The fundamental theoretical concerns raised by reviewers were not fully resolved, leading to mixed opinions. However, the empirical rigor, practical utility, and reproducible findings outweigh the theoretical limitations in the current context of LLM research
Decision: Accept (poster)