Not All Tokens Are What You Need for Pretraining
Authors: Zhenghao Lin
Zhibin Gou
Yeyun Gong
Xiao Liu
yelong shen
Ruochen Xu
Chen Lin
Yujiu Yang
Jian Jiao
Nan Duan
Weizhu Chen
Venue: NeurIPS 2024 oral (2024)
Keywords: pre-training
next token prediction
data optimization
data selection
Primary area: natural_language_processing
TL;DR: We introduce Selective Language Modeling (SLM), a method for token-level pretraining data selection.

Abstract: Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that ''Not all tokens in a corpus are equally important for language model training''. Our initial analysis examines token-level training dynamics of language model, revealing distinct loss patterns for different tokens. Leveraging these insights, we introduce a new language model called Rho-1. Unlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution. This approach involves scoring training tokens using a reference model, and then training the language model with a focused loss on tokens with higher scores. When continual continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks. After fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and 51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the pretraining tokens. Furthermore, when continual pretraining on 80B general tokens, Rho-1 achieves 6.8% average enhancement across 15 diverse tasks, increasing both data efficiency and performance of the language model pre-training.

Not All Tokens Are What You Need for Pretraining
Zhenghao -in
Zhibin Gou ad
Yeyun Gong
Xiao Liu
Yelong Shen
Ruochen Xu Chen Lin
Yujiu Yang
Jian Jiao
Nan Duan
Weizhu Chen
Xiaamen University
"Tsinghua University PShanghai Al Laboratory
Microsoft
https : //aaa//aaaaaaaa ms/rho
Abstract
Previous language model pre- pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that "Not all tokens in corpus equally important for language model training". Our initial analysis examines token-level training dynamics of language model, revealing distinet loss patterns for different tokens. Leveraging these insights, we introduce new language model called RI10-1. Unlike traditional LMs that learn to predict every next token in corpus, RHO-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution. This approach involves scoring tokens using reference model, and then training the language model with focused loss on tokens with higher scores. When continual pretraining on 15B Open WebtWhhhttth corpus, RHO-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks. After fine-tuning, RHO-1-1B and 7B achieved state-of-the-arr results of 40.6% and 51.8% on MATH dataset, respectively matching DeepSeekMath with only 3% of the pretraining tokens. Furthermore, when continual pretraining on 80B general tokens, RHO-1 achieves 6.8% average enhancement across 15 diverse tasks, increasing both data efficiency and performance of language model pre-training.
Avg Few-shot Acc on 18 LMs
Avg Few-shot Acc on 78 LMs
DeepSeeedAath (1508 Tocens)
50
20
Baseline
Rho 1:18
SeepSesiMtth (5008 Tokens)
Rho-1-78
Math Acc 0%)
15
40
16. 4% better
16.3% better
35
5x faster
10
30
10x
25
5
5
10
15
0
5
10
15
Tokens (B)
Tokens
We continual pretrain 1B and 7B LMs with 15B Open WetMaath tokens. RHO-1 is trained with our proposed Selective Language Modeling (SLM), while baselines are trained using causal language modeling. SLM improves average few-shot accuracy on GSM8k and MATH by over 16% achieving the baseline performance 5-10x faster.
Figure 1:
"Equal contribution. See author contributions for details. Work done during their internships at Microsoft Research Asia. Eз: zhenghao in@etu. xmu, edu, en; zebgou@gmail.co com "Correspondence authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Desired Tokens
Undesired Tokens
Keep loss
Remove loss
Noisy Pretraining Corpus
The farm has 35 hens <Apr12 12 1:24> and 12 pigs. ##ldavidjl 1 22 says totaling 47 animals.
X
X1
X2
X3
X4
X5
X
X7
X
X1
X2
X3
X4
X5
x
X7
Causal Language Modeling
Selective Language Modeling
X1
X-2
X3
X4
X5s
A
X-y
X1
X2
X3
X4
X5
X-7
Figure 2: Upper: Even an extensively filtered pretraining corpus contains token-level noise. Left: Previous Causal Language Modeling (CLM) trains on all tokens. Right: Our proposed Selective
Language Modeling (SLM) selectively applies loss on those useful and clean tokens.
Introduction
Scaling up model parameters and dataset size has consistently elevated the next-token prediction accuracy in large language models, yielding significant advancements in artificial intelligence [Kaplan al., 2020, Brown al., 2020, OpenAI, 2023, Team et al., 2023]. However, training all available data is not always optimal or feasible. As a result, the practice of data filtering has become crucial, using various heuristics and classifiers [Brown al.. 2020. Wenzek et al., 2019] to select training documents. These techniques significantly improve data quality and boost model performance.
However, despite thorough document-level filtering, high-quality datasets still contain many noisy tokens that can negatively affect training, as illustrated in Figure 2 (Upper). Removing such tokens might alter the text's meaning. while overly strict filtering could exclude useful data [Welbl et al., 2021, Muennighoff et al., 2024] and lead to biases [Dodge et al., 2021, Longpre et al., 2023]. Furthermore, research indicates that the distribution of web data does inherently align with the ideal distribution for downstream applications [Tay et al., 2022, Wettig et al., 2023]. For example, common corpus at the token level may include undesirable content like hallucinations highly ambiguous tokens that are hard to predict. Applying the same loss to all tokens can lead to inefficient computation on non-essential tokens, potentially restricting LLMs from achieving more advanced levels of intelligence.
To explore how language models learn at the token level, initially examined training dynamics, particularly how the token-level loss evolves during usual pretraining. In $2.1, we evaluated the model's 's token perplexity at different checkpoints and categorized tokens into different types. Our findings reveal that significant loss reduction is limited to select group of tokens. Many tokens are "easy tokens" that are already learned, and some "hard tokens" that exhibit variable losses and resist convergence. These tokens can lead to numerous ineffective gradient updates.
Based these analyses, we introduce RHO-1 1 models trained with novel Selective Language Modeling (SLM) objective 3, As shown in Figure (Right), this approach inputs the full sequence into the model and selectively removes the loss of undesired tokens. The detailed pipeline is depicted in Figure 4: First, SLM trains a reference language model on high-quality corpora. This model establishes utility metrics to score tokens according to the desired distribution, naturally filtering out unclean and irrelevant tokens. Second, SLM uses the reference model to score each token in corpus using its loss ($2.2). Finally, we train language model only on those tokens that exhibit high excess loss between the reference and the training model, selectively learning the tokens that best benefit downstream applications ($2.2).
We show through comprehensive experiments that SLM significantly enhances token efficiency during training and improves performance downstream tasks. Furthermore, findings indicate that SLM effectively identifies tokens relevant to the target distribution, resulting in improved perplexity scores
"'hoo" denotes selective modeling of tokens with higher information "density (p)".

---

(a) Loss for different token types
(b) Example L-+L tokens
(c) Example H-H tokens
0.3
LoL. laken
3.5
H-H Tiken
h-lt
3
L-d. Token
h-lft Token 3
0.2
3.0
Loss
Loss
H-th (112))
1
2.0
0.0
1.5
Trained (Bkens(B)
Trained TBkens(B)
Trained Tokens(B)
Figure 3: The loss of four categories of tokens during pretraining. (a) shows the loss of H-H, L-H, H-+L, and L- tokens during pretraining. (b) and (c) show three cases of fluctuating tokens' loss in L and H-+H during pretraining, respectively.
on benchmarks for models trained with the selected tokens. $3.2 shows the effectiveness of SLM on math continual pretraining: both 1B and 7B RHO-1 outperform CLM-trained baselines by over 16% on the GSM8K and MATH datasets. SLM reaches baseline accuracy up to 10x faster, as shown in Figure 1. Remarkably, RHO-1-7B matches the state-of-the-ar performance of DeepSeekMath-7B using only 15B tokens, compared to the 500B tokens required by DeepSeekMath. Upon fine-tuning, RHO-1 1-10 and 7B achieve 40.6% and 51.8% on MATH, respectively. Notably, RHO-1-IB the first IB LM to exceed 40% accuracy, nearing the early GPT-4's CoT performance of 42.5%. $3.3 confirms the efficacy of SLM in general continual pretraining: Training Tinyllama- IB on 80B tokens with SLM improves 6.8% on average across 15 benchmarks, with gains over 10% in code and math tasks. In $3.4, we demonstrate that in settings without high-qualite reference data, we can use SLM for self-referencing, leading average improvement of up to 3.3% in downstream tasks.
2 Selective Language Modeling
2.1 Not All Tokens Are Equal: Training Dynamics of Token Loss
Our investigation begins with critical look how individual tokens' losses evolve during standard pre-training. We continue pre-training Tinyllama- 1B with 15B tokens from Open WebMath, saving checkpoints after every 1B tokens. We then evaluate token-level loss these intervals using the validation set of approximately 320,000 tokens. Figure 3(a) reveals a striking pattern: tokens fall into four categories based on their loss trajectory-persisteer loss (L-HH), decreasing loss (H-L), and consistent low loss (L-L). For further details on these categories, see I. Our analysis uncovers that a mere 26% of tokens show a notable loss reduction (H-+),, while the majority 11%) remain in the L-L category, indicating they have already been learned. Interestingly. 11% of the tokens are persistently challenging (H (H-+), likely due to high alcatoric uncertainty [Hüllermeier and Waegeman, 2021]. Additionally, 12% of tokens experience an unexpected loss increase (L.+)) during training.
high loss (H-H), increasing
Our second observation is that a significant number of token losses exhibit persistent fluctuations, and resist convergence. The loss of many L-L and H-HH tokens, depicted in Figure 3 (b) and (c), show high variance during training. In $D.2, we visualize and analyze the content of these tokens and find that many of them noisy, which is consistent with our hypothesis.
Consequently, we learn that the loss associated with each token during training does not decrease smoothly like the overall loss; instead, there is a complex training dynamic among different tokens. If we can select the appropriate tokens for the model to focus on during training, we may be able to stabilize the trajectory of the model's training and enhance its data efficiency.
2.2 Selective Language Modeling
Overview Inspired by the practice of reference model in document-level filtering, we propose a simple pipeline of token-level data selection, termed "Selective Language Modeling (SLM)". Our method comprises three steps, as depicted Figure 4. We begin by training a reference model on a curated, high-quality dataset. This model then assesses the loss of each token within the pretraining
3

---

Step
Step
Step 3
High-quality Corpus
Train reference model on high-quality text.
Calculate each token's ppl the pretraining corpus.
Pretraining Corpus
Train LLM with loss focused high-score tokens.
Reference Model
Language Model
Figure 4: The pipeline of Selective Language Modeling (SLM). SLM optimizes language model performance by concentrating on valuable, clean tokens during pre-training It involves three steps: (Step 1) Initially, train reference model on high-quality data. (Step 2) Then, score each token's loss a corpus using the reference model. (Step 3) Finally, selectively train the language model on tokens that have higher scores.
corpus. In the final phase, we train the language model selectively, focusing on tokens with high excess loss between the training and reference model. The intuition is that tokens with high excess loss are more learnable and better aligned with the desired distribution, naturally excluding tokens that are either irrelevant or of low quality. Below, we provide a detailed description each step.
Reference Modeling We begin by curating high-quality dataset that reflects the desired data distribution. We train reference model (RM) using standard cross-entrope loss on the curated data. The resulting RM then used assess the token loss within larger pretraining corpus. We compute the reference loss (CRM) of a token æ; based on the probability that the RM assigns to this token. The calculation formalized follows:
(RM(F;) - log P(x)(res)
By evaluating LRM for each token, we establish the reference loss for selective pretraining, allowing us to focus on the most influential tokens in language modeling.
Selective Pretraining Note that Causal Language Modeling (CLM) employs the cross-entropy loss:
N log ((z:(res)))
1 N
(C
(2)
Here, (CLM(O) represents the loss function parameterized by model 0. N is the length of the sequence, x; is the i-th token in the sequence, and øęi represents all tokens before the i-th token. contrast, Selective Language Modeling (SLM) trains the language model with focus on tokens that exhibit a high excess loss when compared to the reference model. The excess loss (Ca) for a token x; is defined as the difference between the current training model loss (Co) and the reference loss:
(a(x,) Co(x,) (x,(
We introduce token selection ratio 1%, which determines the proportion of tokens be included based on their excess loss. The cross-entropy loss for the selected tokens is computed as follows:
N
(O
Σ
(x((x;)) log P(x,(r=s;))
1%
Here, N R% defines the number of tokens that fall within the top 1% of excess loss. The indicator function L 4.% (x+) is defined as:
[1 if x; ranks in the top 4% by S(z:)
T1%
otherwise

---

Table I: Few-shot Co reasoning results of math pretraining. All models are tested with few-shot prompting. Previous best results are highlighted in blue, while our best results are purple. "Only unique math-related tokens are caleulated. For RHO-1, calculate only the selected tokens that are used for training. We use OpenAl's MATH subset [Lightman et al., 2023] for evaluation, since some original test samples have been used in public training such as PRM800K. "The SAT only has 32 four-choice problems, so we average our results over the last three checkpoints, available.
Uniq.
Train Toks
MMLU STEM
Model
0 Data
GSMSK MATHI SVAMP ASDiv MAWPS TAB MQA
SAT!
Toks
1-28
Base Models
Tinyllama
1.1B
2.9
3.2
11.0
18.1
20.4
12.5
14.6
16.1
21.9
13.4
Phi-1.5
1.3B
32.4
4.2
43.4
53.1
66.2
24.4
14.3
21.8
18.8
Qwenl.5
1.8B
36.1
6.8
48.5
63.6
79.0
29.2
25.1
31.3
40.6
40.0
Gemma
2.0B
18.8
11.4
38.0
56,6
72.5
36,9
26.8
34.4
50.0
38.4
Le
1.3B OWM
14B
150B
11.5
8.9
29.6
31.3
DeepSeekMath
1.3B
1208
1508
23.8
13.6
Continual Pretraining on Timyllama-Ih
Tinyllama-CT
6.4
36.7
47.7
17.9
RH0-1-Math
1.IB OWM
14B
9B
29.8
14.0
49.2
61.4
79.8
25.8
30.4
24.7
28.1
38.1
40%
+23.4
+11.6
+27.5
+24.7
+32.1
+7.9
+16.5
+1.7
+3.1
+16.5
RHO- 1 M-thhh
LIB OWM
14B
30B
36.2
15.6
52.1
67.0
83,9
29.0
32.5
23.3
28.1
40.9
7B Base Models
LLaMA-2
7B
14.0
3.6
39.5
51.7
63.5
30.9
12.4
32.7
34,4
31.4
Mistral
7B
41.2
11.6
64.7
68.5
87.5
52.9
33.0
49.5
59.4
52.0
Minerva
164B
16.2
14.1
35.6
Minerva
62B
39B
109B
52.4
27.6
53.9
Minerva
540B
39B
26B
58.8
33.6
63.99
LLemma
PPile
2008
38.8
17.2
56.1
69.1
82.4
48.7
41.0
45.4
59.4
50.9
LLemma
34B
PPile
55B
50B
54.2
23.0
67.9
75.7
90.1
57.0
49.8
54.7
68.8
60.1
Intern-Maii
7B
31B
125B
41.8
14.4
61.6
66.8
83.7
50.0
57.3
24.8
37,5
48.7
Intern-Math
20B
31B
125B
65.4
30.0
75.7
79.3
94.0
50.9
38.5
53.1
71.9
62.1
DeepSeekMath
500B
64.1
34.2
83.9
92.4
63.4
Continual Pretraining
Mistral-78
Mistral-CT
42.9
22.2
71.0
86.1
45.1
RHo---Mathh
7 OWM
14B 10.5B
66.9
31.0
77.8
79.0
93.9
49.9
58.7
54.6
84.4
66.2
-30%
+24.0
+8.8
+9.2
+8.0
+7.8
+4.8
+11.0
+2.0
+18.8
+10.4
By default, we use LA as the score function S. This ensures that the loss is applied only to tokens that are deemed most beneficial for the language model to learn from. In practice, token selection can be implemented by ranking the tokens batch according to their excess loss and using only the top 4% of tokens for training. This process eliminates the loss for undesired tokens without incurring additional costs during pretraining, making our approach both efficient and casily integrated.
Experiments
We continually pretrained models in both mathematical and general domain and designed ablation and analysis experiments to understand the effectiveness of SLM.
3.1 Experimental Setup
Reference Model Training To train our mathematical reference model, we gathered dataset of 0.5B high-quality, math-related tokens. This dataset blend of synthetic data from GPT [Yu et al., 2024, Huang et al., 2024] and manually curated data (Yue et al., 2024, Ni et al., 2024]. For the general reference model, we compiled corpus of 1.9B tokens from open-source datasets, such as Tulu-v2 [Ivison et al., 2023] and OpenHermes-25. [Teknium,, 2023]. We trained the reference models for epochs. The maximum learning rate was set 5e-5 for 1B models and le-5 for 7B models, applying a cosine decay schedule. We set the maximum sequence lengths to 2048 for 1B models and 4096 for 7B models, packing multiple samples into these lengths for model input. all main experiments, we initialized the continual pretraining model and the reference model with the same base model.
5

---

Table 2: Tool-integrated reasoning results of math pretraining.
Model
Size
Tools
SFT Data
GSM8K
MATH
SVAMP
ASDív
MAWPS
TAB
GSM-H
AVG
Used SFT?
Previous Models
GPT4-0314
92.0
42.5
64.7
78.3
GPT4-0314 (PAL)
94.2
51.8
94.8
92.6
97.7
95.9
77.6
86,4
MAmmoTH
70B
MI-260K
76.9
41.8
82.4
TORA
ToRAA69K
68.8
73,9
62.4
TORA
70B
ToRA-69k
84.3
49.7
82.7
86.8
93.8
74.0
67.2
76,9
DeepSaekMath
7B
ToRA-69k
79.8
52.0
80.1
87.1
93.8
85.8
63.1
77.4
Dur Pretrained Models
Tinyl Ja
18
ToRA-69K ToRA-69k
$1.4
38.4
53.4
66.7
81.7
20.5
42.8
50.7
RHO-I-MaR
1B
59.4
40.6
60.7
74.2
88.6
26.7
48.1
56.9
+8.0
+2.2
+7.3
+7.5
+6.9
+5.3
+6.2
MistralCTT
7B
ToRA-69K
77.5
48.4
76.9
83.8
93.4
67.5
60.4
72.6
7B
ToRA-69k
81.3
51.8
80.8
85.5
94.5
70.1
63.1
75.3
+3.8
+3.4
+3.9
+1.7
+1.1
+2.6
+2.7
+2.7
Pretraining Corpus For mathematical reasoning, we utilize the Open WebMath (OWM) dataset [Paster et al., 2023], which comprises approximately 14B tokens sourced from math-related web pages the Common Crawl. the general domain, we combine the Slim SlimPajama [Daria al., 2023] and StarCoderData [Li et al., 2023a] (both part of the Tinyllama corpus) with MatnWehMatt,, training on a total 80 billion tokens with a mix ratio of 6:3:1.
Pretraining Setting For math pretraining, continue pretraining the Tinyllama-I. [Zhang et al., 2024] and the Mistral-7B model [Jiang et al., 2023] with learning rates of 8e-5 and 2e-5, respectively. For the 1.1B model, conducted configuration allowed us to train approximately 15 billion tokens in around 3.5 hours and 50 billion tokens in about 12 hours. In the of the 7B model, training the same 15 billion tokens took approximately 18 hours under similar hardware conditions. For general domain, we set the learning rate for Tinyllama-1. 1.11 1.1 model to le-4 and train 80B tokens under the same hardware conditions, which takes approximately 19 hours. The batch size is uniformly set to 1M tokens for both domains. Regarding the token selection ratio, we use 60% for the Tinyllama-I.I model and 70% for the Mistral-7B model.
1.IB 1.1 model
training on 32 H100 80G GPUs. This
Baseline Setting We models that have been continually pretrained (Tinyllama-CT and Mistral- CT) through regular causal language modeling as baselines. Moreover, we compare RHO-1 with well-known and tww-toofooneeee perfoffrrming baselines, including Gemma [Team et al., 2024], Qwen1.5 [Bai et al.. 2023], Phi-1.5 [Li et al., 2023b], DeepSeckLLM [DeepSeck-Aae -A, 2024], DeepSeekMath [Shao et al., 2024], CodeLlama [Roziere al., 2023], Mistral [Jiang et al., 2023], Minerva [Lewkowycz et al., 2022], Tinyllama IZhang et al., 2024], LLemma [Azerbayev et al., 2023]. and MiternLM2-Mann [Ying
al., 2024]. For fine-tume tne-ttning results, we also compare with previous best models MAmmoTH[[ et al., 2024] and ToRA[Gou et al., 2024].
Evaluation Setup To comprehensively evaluate pretrained models, we compare their few-shot capabilities and fine-tuning performance across a variety of tasks. We adopt the Im-eval-harness" [Gao al., 2023] for general tasks, and develop math evaluation suite for math tasks. We use vllm (v0.3.2) [Kwon et al., 2023] to speed up inference. Further details on our evaluation can be found in Appendix E.
3.2 Math Pre-training Results
Few-shot CoT Reasoning Results We evalute base models prompting with few-shot chain-of- thought (CoT) [Wei et al., 2022a] examples following previous works [Lewkowyez et al., 2022, Azerbayev et al., 2023, Shao et al., 2024]. As results shown in Table 1, in comparison to continue pretraining directly, RHO-1 Math has achieved the average few-shot accuracy improvement of
"nttpe://githhb.ccmm/ httppp///tttaaccc//eeeeeeeeeeeea httpp://ttt...cceee///aaaoattttt

---

16.5% on IB models and 10.4% on 7B models. Furthermore, after training for multiple epochs on Open WebMath, find that RHo-1 could further increase the average few-shot accuracy 40.9%. Compared to DeepSeekMath-711 which pretrained on 500 billion math-related tokens, RHO-1-7B pretrained on only 15 billion tokens (selecting 10.5 billion tokens) achieved comparable results, demonstrating the efficiency of our approach.
Tool-Integrated Reasoning Results We fine-tune RHO-1 and baseline models on 69k ToRA corpus [Gou al., 2024], consisting of 16k GPT-4-generated trajectories in tool-integrated reasoning format, and 53k answer-augmented samples using LLaMA. As presented in Table 2, R1OO-1-IB and RHo-1- 1-7B achieved offee-of-the-ar 40.6% and 51.8% MATH dataset, respectively. On some unseen tasks (e.g., TabMWP and GSM-Hard), RHO-1 I also demonstrates a certain degree of generalizability, with average few-shot accuracy improvement of 6.2% on the RHO- 1-to-- and 2.7% on RHO- 1-Math-7B..
Performance of General Pretrained Base Model
+11.3
+399
+7.8
Metrics
20-
+6.5
+6.9
+5.0
MMLU
BBH
MATH
GSMBK
MBPP(p@1)
MBPP(p@10)
HumEval(p@11
HumEval(p@10)
+0.9
Tinyllama
+8.6
Tinyllama-CT
Metrics (%)
+0.2
Rho-1-18
+3.4
+5.0
20
AGIEval
ARC-C
ARC-E
BooIQ
PIQA
HellaSwag
WinoGrande
OBQA
TydiQA
Figure 5: General pretraining results. We continual pretraining Tinyllama- on 80G general
tokens. Tinyllama-CT Ca is etrained with CLM, while RHO-1 is trained with our proposed SLM.
3.3 General Pre-training Results
We confirm the efficacy of the SLM in general pretraining by continual training Tinyllama-I. 1.11 on 80 billion tokens. The results depicted Figure 5 indicate that although Tinyllama has already undergone extensive training on the majority of these tokens, the application of SLM yields an average enhancement 6.8% across 15 benchmarks compared direct continual pretraining. The improvements were especially pronounced in code and math tasks, exceeding 10%.
3.4 Self-Reference Results
In this section, we demonstrate that SLM can enhance the effectivvness of model pre-training using only pre-training corpora, without the need for additional high-quality data. Specifically, we initially trained the reference model on the Open WebMath (OWM) corpus, subset of Proof-Pile-2 (PPile). We evaluated OWM and PPile using the trained reference model and selected tokens for training. In this scenario, we assume the absence of downstream task-related data, common situation in real-world applications. We hypothesize that key factor not scoring the desired distribution but filtering out noisy tokens. Therefore, reference model loss, LRM. and the information entropy of the next token, HRM, which measures the uncertainty of the next token. Details are provided in Appendix H.
employed two different scoring functions based on the

---

Table 3: Self-Reference results. We use OpenWebMath (OWM) train the reference model.
Score Function
Uniq. Toks
Train Tokk
Model
Data
GSM8K MATH SVAMP
ASDiv MAWPS
MQA
AVG
Tinyllama- CT (RM)
OWM
14B
15B
6.3
2.6
21.7
36.7
47.7
13.9
21.5
Tinyllama-Si SLM
CRM
OWM
14B
6.7
4.6
23.3
54.5
14.3
23.9
Tinyllama- SLM
HRM
OWM
14B
10.5B
7.0
4.8
23.0
50.5
13.5
23.0
Tinyllama-Se SLM
CRM
HRM
OWM
14B
7.1
5.0
23.5
41.2
18.0
Tinyllama-Ch CT
PPile
55B
52B
8.0
6.6
23.8
41.0
54.7
14.2
24.7
Tinyllama-SLM
LRM
HRM
PPile
55B
36B
8.6
8.4
24.4
43.6
57.9
16.1
26.5
(a) Selected Token Loss
(b) Downstream Token Loss
(c) Unselected Token Loss
Baseline Rho-1
Baseline Rho-1
4.1
Baseline
1.05
Rho-1
1.06
mmm Anamy
1.00
Loss
Loss
Mmw
0.90
2.9
0.38
0.85
2.5
Tokens (B)
Tokens (B)
Tokens (B)
Figure 6: The dynamics of pretraining loss and downstream loss. (a) and (c) represent the loss of tokens selected/unselected by SLM during pretraining in both SLM and CLM methods, while (b)
represents the loss of the SLM and CLM methods on MetaMath [Yu et al., 2024]. We tested the above results through the process of pretraining with total of 4 billion tokens.
The experimental results, shown in Table 3, indicate that using only the OWM-trained reference model can effectively guide the model in pre-training on the same corpus, improving average downstream performance by +2.4%. Using only the information entropy as the score function brought about similar improvement. Additionally, we considered training on the intersection of tokens selected by the two scoring functions and found better performance. with a 40% reduction in tokens and +3.3% performance. Furthermore, training the SLM on the PPile, despite only using the OWM subset to train the reference model, still achieved 1.8% improvement with 30% fewer tokens used. For more details, please refer to Appendix H.
3.5 Ablation Study and Analysis
Selected Token Loss Aligns Better with Downstream Performance We utilized the reference model to filter tokens and assess their impact on validation and downstream losses after training. As depicted in Figure 6, pretrained 4B tokens and tracked loss variations across methods and validation sets. The RHo-1 showed greater loss reduction on selected tokens than regular pretraining. Cross-referencing figures (a). (b), and (c) reveals that selected-token pretraining substantially lowers downstream loss, while traditional pretraining's 's effect on downstream loss is less pronounced despite initial loss reductions. Therefore, we expect that selecting tokens for pretraining
efficient.
In Figure 7, we demonstrate that the loss of selected tokens correlates with downstream task perfor- mance, following power law similar recent findings [Gadre et al., 2024]. Our analysis shows that tokens selected by SLM positively impact performance, while those not selected have a negative impact. Thus, reducing loss across all tokens is not imperative for improved model performance. Refer to Appendix F for further details.
What Tokens are Selected with SLM? We aim to analyze the tokens selected by the SLM method in pretraining to further explore its working mechanism. To this end, we visualize the token selection process during the training RHO-1 using the Open WebMath. $G.1, we have highlighted in blue the tokens that were retained during actual pretraining. We observe that the majority of tokens chosen

---

(a) Accuracy vs. Selected Tokens' Loss
(b) Accuracy vs. Unselected Tokens' Loss
22.5
22.5
Selected Bouns 5:
20.0
seacted kiens 8:
Selected Bisnnc 138
Accuracy(98)
17.5
Selected Bisera a: 140
Accuracy(%)
17.5
15.0
15.0
10.0
10.0
Urselcted Talons at
Uraelecis Tbkana at
7.5
7.5
Uraelecoss Takaru at 20
lireddeted Thlmes
Ureelected Thlens
Loss
Loss
Figure 7: The relationship between the selected tokens / unselected tokens loss in SLM and downstream task performance. The y-axis represents the average few-shot accuracy on GSM8k and MATH. The x-axis represents the average loss selected tokens /nsselcced tokens corresponding checkpoint (2B, 5B, 8B, 11B, and 14B).
PPL of Tokens Selected by Different CKPT
Accuracy vs. Select Ratio
2.65
Selected Tolors Selected lokers 36
20
Selected Tokans
Math
Selected Thbers 11B
Accuracy (%)
2.55
PPL
2.45
2.35
11
14
Tokens(B)
50
100
Token Select Ratio (%)
Figure 8: The PPL of tokens selected by differ- ent checkpoint. We test the PPL of the tokens selected at 2B, 5B, 8B, 11B, and 14B.
Figure 9: Effect of token select ratio. We train IB LM with SLM objective 5B tokens.
by the SL.M method are closely related to mathematics, effectively training the model on the parts of the original corpus that are pertinent to mathematical content.
Furthermore, we investigated the differences in token filtering across various checkpoints during the training process and tested the perplexity of these tokens on different checkpoints. As illustrated in Figure 8, we found that the tokens selected by later checkpoints tend to have higher perplexity towards the later stages of training and lower perplexity the earlier stages. This may suggest that the model first optimizes tokens with larger Icarnable space, thereby increasing learning efficiency, Moreover, we noticed sample-wise "double descent" [Nakkiran al., 2021] on the loss of selected tokens, where the select token's 's perplexity initially increases before decreases. This might be an effect of selecting tokens based on excess loss, targeting those most need each checkpoint.
Effect of Token Select Ratio We investigate the impact of token selecting ratios of the SLM. Generally, the selecting ratio is defined by heuristic rules, similar the approach previously employed in the training of Masked Language Models (MLMs) [Devlin et al., 2019, Liu et al., 2019]. As shown Figure 9, the selected tokens suitable for accounting for about 60% of the original tokens.
Conclusion
In this paper, we propose using Selective Language Modeling(SLMM) to train RHO-1, which select more suitable tokens for current pretraining stage. We conducted the detailed analysis of the loss of tokens during the pretraining process and found that not all tokens are equal during pretraining. Our

---

experiments and analysis in the fields of mathematics and general have demonstrated the effectiveness of the SLM method, emphasizing the importance of token level in the LLM pretraining process. In the future, how to improve pretraining of L.LMs from the perspective of token level worthy of in-depth research.
Acknowledgments
Zhenghao Lin and Chen Lin were supported by National Key R&D Program of China (No. 20222D0160501] the Natural Science Foundation of China (No.62372390,62452011 Gou and Yujiu Yang were supported by the Shenzhen Science and Technology Program (CCJJ20200188100 1801000000)) and the "Graph Neural Network Project" of Ping An Technology (Shen- zhen) Co., Ltd.
Zhibin
References
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836. 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advonces in neural information processing systems, 33: 13:1877-1901, 2020.
OpenAL. Gpt-4 technical report, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- can--Batistt Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11880) 2023.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave Cenet: Extracting high quality monolingual datasets from web crawl data, arXiv preprint arXiv:1911. 100.:199,.0039 2019.
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguisticx: EMNLP 2021, pages 2447-2469, 2021.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
Jesse Dodge, Maarten Sap, Ana Marasovié, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286-1305, 2021.
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's guide to training data: Mcasuring the effeets of data age, domain coverage, quality, & toxicity. arXiv preprint (3XXi:3305.13666 2023.
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207. 10551, 2022.
Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling? In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguisties, pages 30000 Dubrovnik, Croatia, May 2023, Association for Computational Linguistics. doi: 10. 10.1165531112222.cccceee attpe////ccaathooogg...o org/2023.......ooooo. eacl-main. 217.
URL
Eyke Hüllermeier and Willem Wacgeman. Alcatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 1 1103))457-506, 2021.
Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang. James Kwok, Zhenguo Li. Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In ICLR, 2024.
10

---

Yiming Huang, Xiao Liu. Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point- driven data synthesis with its enhancement mathematical reasoning. arXiv preprint urXiv:2403.02333. 2024.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. ICLR, 2024.
Xinzhe Ni, Yeyun Gong, Zhibin Gou, Yelong Shen, Yujiu Yang. Nan Duan, and Weizhu Chen. Exploring the mystery of influential data for mathematical reasoning. 2024.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters. Pradeep Dasigi, Joel Jang. David Wadden, Noah A Smith, Iz Beltagy, et al. Camels a changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:231 1../2/111.07000 10702, 2023.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist Ulm assistants, 2023. URL attps://huggingtte /t hoo/daaasess///eee teetttstteeiiiiiit im htn/ppeeeeemees---e-ee 2.5.
Hunter Lightman, Vineet Kosaraju. Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.200553 2023.
Keiran Paster, Marco Dos Santos, Zhangir Azerhayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.
Soboleva Daria, Al-Khateeb Faisal, Myers Robert Steeves Jacob R, Hestness Joel, and Dey Nolan. SlimPajama: A 627B token eleaned and deduplicated version of RedPajama. https://uwe cerebras. net/blog/ slimpajama-a- toweeetee--------eeoo- tww.etoooeeeeeeeeeeeee -ndeee--eeeadddddddddddeeeeeeeee wwwp-aadaddddddddiaaaaaaeeeerera -rdd--jaaaaaaaaaa..a.aaaaaaaaa-a 2023. URL
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltomozhskii, Terry Yue Zhuo Thomas Wang. Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang. Rudra Murthy V, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Moustafa-Fahmi
Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh. Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao. Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jerniee, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! CoRR, has/23055061661 2023a.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.
Albert Q Jiang, Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint
2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre. Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0829 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16660) 2023.
Yuanzhi Lii, Séhastien Bubeck, Ronen Eldan, Allic Del Giomo, Suriya Gunasekar, and Yin Tat Lee. Texthooks are all you need i:: phi-1.5 technical report, 2023b.
DeepSeck-Al. Deepseek ll:: Scaling open-source source language models with longtermism. arXiv preprint arXiv:2401.0299 2024. URL https::////// //github. cww
LLM.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Mingchuan Zhang, Li. Y Wu, Daya Guo. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402. 033000 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308.1 /2950, 2023.
11

---

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Zhangir Azerbayev, Hailey Schoelkopf. Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang. Jia Deng. Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310. 10631, 2023.
Huaiyuan Ying, Shuo Zhang, Linyang Li. Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma. Jiawei Hong. Kuikun Liu, Ziyi Wang, et al. Internlm- math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.0633 06332, 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang. Nan Duan, Weizhu Chen, et al. Tora: gool-integrated reasoning agent for mathematical problem solving. In ICLR, 2024.
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoclkopf, Aviya Skowron, Lintang Sutawika, Eric Tang. Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL htw.h:ttp///eett //zenodo wwgg/reoords///5ee0 10256836.
Wonsuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operoting Systems Principles, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Lc, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. In NIPS, volume 35, pages 24824-24837, 2022a.
Samir Yitzhak Gadre, Georgios Smymis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Kch, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song. Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel,, Niklas Muennighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on downstream tasks. Preprint, 2024.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Iya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12)122003, 2021.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirec- tional transformers for language understanding. In NAACL-HLT (I), pages 4171-4186. Association for Computational Linguistics, 2019.
Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11666 1692,, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1 2020.
Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. arXiv preprint (1XXi:2009.0339/ 2020.
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javahcripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need, arXiv preprint arXiv:2306. 1144,, 2023.
Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini, Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06-499 2021.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697-10707. PMLR, 2022.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving I1m pretraining via document de-duplication and diversification. In NIPS, volume 36, 2023.
12

---

Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36, 2024a.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighotf, Bairu Hou, Liangming Pan, Haewon Jeong. Colin Raffel, Shiyu Chang. Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language models, 2024.
Sang Michael Xie, Hicu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoe V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances Neural Information Processing Systems, 36, 2024b.
Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang. Ce Zhang, Frederic Sala, and Christopher Ré, Skill-it! data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024.
Yingwei Ma. Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help LLMs reasoning? In The Twelfth International Conference on Learning Representations, 2024. URL https:/ /ttp///ppeeeeeeeeeeee/ee/eeeeeee net net/forum? ceoorummid-Kia Md-KIIIKKT443
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang. Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting Ilm performance with self-guided data selection for instruction tuning. arXiv preprint 2.308 12032, 2023c.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In ICLR, 2024.
Yunshui Li. Binyuan Hui, Xiaobo Xia, Jiaxi Yang. Min Yang. Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, et al. One shot learning as instruction data prospector for large language models. arXiv preprint arXiv:2312. 10302, 2023d.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Dangi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2440..04333, 2024.
Feiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang, Rongxing Du, Anit Kumar Sahu, and Ruoxi Jia. Get more for less: Principled data selection for warming up fine-tuming in LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL httpa: /tttppa://opnneeveeee frrum?dd=QmYNBVVV
Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng. Zhaopan Xu, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, and Yang You. Infobatch: Lossless training speed up by unbiased dynamic data pruning. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview. wwt.eee//foooo net/tfruum 2idc66ss51esX
Together Computer. Redpajama: an open dataset for training large language models, 10 2023. URL https: //gi //github.
ama-Date Data.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach, arXiv preprint 1eXXvv:708.0044040 2017.
Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning. Advances in neural information processing systems, 34:14488-14501, 2021.
Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint /511::55/1.0.44 2015.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv: /5X0055511.05552 2015.
Haw-Shiuan Chang, Erik Learned- Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advances in Neural Information Processing Systems, 30, 2017.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pages
PMLR, 2018
Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton. et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint 19000000. 2019.
13

---

Hwanjun Song. Minseok Kim, Sundong Kim, and Jae-Gil Lee. Carpe diem, seize the samples uncertain' at the moment" for adaptive batch selection. In Proceedings of the 29th ACM International Conference on Information Knowledge Management, pages 1385-1394, 2020.
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors. Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 166391 PMLR, 17-23 Jul 2022. URL https:// httpp::/pprccedingg ngs.inggn..n www............................. .mlr. pww.prresssaeeee 1e111//mminemmammni .ttml.
Simin Fan and Martin Jaggi. Irreducible curriculum for language model pretraining. arXiv preprint arXiv:2310. 2023.
Jean Kaddour, Oscar Key. Piotr Nawrot, Pasquale Minervini, and Matt Kusner. No train no gain: Revisiting efficient training algorithms for transformer-based language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL ftttp:://openrevie...eee http://www.....m..m.dobbbboggg/.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy I Liang, Jure Leskovec, and Matci Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv: 1829, 2019.
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXtv:240).1292. 2024.
Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust language modeling. arXiv preprint arXiv: (1.999902000, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, $6(11):2278 2324, 1998.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher sentiment treebank. In Proceedings of the
Potts. Recursive deep models for semantic compositionality 2013 conference on empirical methods natural language processing, pages 1631-1642, 2013.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018
Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang. Zhiyuan Liu, and Maosong Sun. Train no evil: Selective masking for task-guided pre-training. Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6966-6974, Online, Novemher 2020. Association for Computational Linguistics. doi: URL https::/acclaantholoog org/2020. emnlp-main. main.566.
Tanish Lad, Himanshu Maheshwari, Shreyas Kottukkal, and Radhika Mamidi. Using selective masking as bridge between pre-training and fine-tuning. arXiv preprint 0rX1v:2211.38115, 2022.
Qihuang Zhong, I Liang Ding, Juhua Liu, Xuebo Liu, Min Zhang, Bo Du, and Dacheng Tao. Revisiting token dropping strategy in efficient BERT pretraining. In Anna Rogers, Jordan Graher, and Naoaki Okazaki, editors, Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages
Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10. 18.18853/v lon. URL https:t/aaaaaaaeeeee /aaclanthooogy org/2023. acl-long. 579.
Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. Token dropping for efficient BERT pretraining. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio. editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers). pages 3774-3784, Dublin. Ireland, May 2022. Association for Computational Linguistics. doi: URL haap:::///laaaooooo/o/ org/2022. 2022. acl-long 262.
10.188553v1/20222 2oo2
Tianjian Li, Haoran Xu, Philipp Koehn, Daniel Khashabi, and Kenton Murray. Error norm truncation: Robust training in the presence of data noise for text generation models. arXiv preprint 00X0v:2310.00840, 2023e.
Jessica Rumbelow and Matthew Watkins.
Solidgoldmagikarp (plus, prompt generation).
LessWrong, 2023.
URL https://www. lesssrong. cct.///pstt//aaaaaaassttrrraraa.
sol solidgoldmagi tittarrppluuppee wwwpppptgeeeeatttai...
Sander Land and Max Bartolo. Fishing for magikarp: Automatically detecting mnder-trained tokens in large language models. arXiv preprint 055:424 2024.
14

---

Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with sveca. arXiv preprint a.X0v: 2018.
Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. The Grammar-learning trajectories of neural language models. arXiv preprint arXiv:2109.060980 2021.
Leo Liu, Yizhong Wang, Jungo Kasai, Hannanch Hajishirzi, and Noah Smith. Probing across time: What does roberta know and when? arXiv preprint arXiv:2104.00 07885, 2021.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2001.02177, 2022.
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. Training trajectories of language models across scales. arXiv preprint arXiv:2212.09803 2022.
Danny Hemandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Sealing laws for transfer. arXiv preprint arXiv:2102.01293, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15555 2022.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Schastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206. 2766.0682, 2022b.
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models, arXiv preprint arXiv:2402.04177, 2024.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274-38290, 2022.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXtv:220..07646, 2022.
Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 2023.
Stella Biderman, USVSN PRASHANTH, Lintang Sutawika. Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 36, 2024.
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability learning from repeated data. arXiv preprint arXiv:2205. 10487, 2022.
Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling Ilm under token-crisis. Advances in Neural Information Processing Systems, 36, 2024.
Charles AE Goodhart and CAE Goodhart. Problems of monetary management: the UK experience. Springer, 1984.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730-27744 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñchiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word prohlems, 2021. URL htapp:://axxivv org/ abs/ abbb/2110 .211/111111..........11 14168.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora. Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NIPS, 2021.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-ate aided language models. arXiv preprint arXiv:2211.10455 2022.
15

---

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies. pages 2080-2094, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18853/1122222.mma waw.oll...acallnnn maacl-main. https://aclanthool hwwpchhooogo.oogg/aeeo org/2021 wwacl-aallaaaaa manel-mmann... 168.
URL
Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975-984, Online, July 2020. Association for Computational I. Linguistics. doi: 10.18653/v1/ 1.200.cl-main URL https: /acla//hoolm org/2020 acl- main. .22.
Rik
Subhro Roy. Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, pages 1152- 1152-1157, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/N16 1136. URL https: /aclsttoolll
Frgg/N1-110 1136.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structurrd mathematical reasoning. In The Eleventh International Conference on Learning Representations, 2023. URL. httpps///peereviieet net/ net/forumiice DittidddBD.HH..TT....
Aida Amini, Saadia Gabriel, Peter Lin, Rik Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv: 13319, 2019.
Yejin Choi, and Hannaneh Hajishirzi. Mathqa:
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, arXiv preprint arXiv:2009.03300 2020.
Mirac Suzgun, Nathan Scales, Nathanacl Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quac V Le, Ed H Chi, Denny Zhou, et al. Challenging big-beneh tasks and whether chain-of- thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint wxX1::2304.06364. 2023b.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal. Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1.XXvv/800.0555. 2018.
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty natural yes/no questions. arXiv preprint arXiv:1905. 10044, 2019.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 7432- 7439, 2020.
Rowan Zellers, Ari Holtzman. Yonatan Bisk, Ali Farhadi, and Yejin Choi, Hellaswag: Can a machine really finish your sentenec? arXiv preprint arXiv: 078300 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications the ACM, 64(9):99-1)) 1066, 2021.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sahharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv: 009999789, 2018
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufci Xue, Lei Shen, Zihan Wang, Andi Wang. Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedingy of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5673-5684, 2023.
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information question answering in ty pologically di verse languages. Transactions the Association for Computational Linguistics, 8:454 470, 2020.
16

---

Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108. 07732, 2021.
Daya Guo. Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK al. Deepseek-coder: When large language model meets programming the rise code intelligence. arXiv preprint urXiv:2401.141990 2024.
Xiang Lisa Ari Holtzman, Daniel Fried, Perey Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer. and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In ACL (1), pages 12286 12312. Association for Computational Linguistics, 2023f.
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreviee httppperrrriee.ntttotrtrreeee jissii122222
Yao Fu, Hao Peng. Litu Ou Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. In International Conference Machine Learning, pages 10421-10430. PMLR, 2023.
17

---

Appendix
Contents
A Author Contributions
19
B Related Works
19
Pretraining Data Optimization
Data Selection
B.3 Language Model Training Dynamics
20
B.4 Scaling Laws
20
C Limitations and Future Work
20
Analysis and Visualization of Tokens Pretraining
21
More Details Four Categories Tokens
21
D.2
Non-Converging Tokens Pretrainig
22
E Evalution Details
22
E.1 Math Evalution
22
E.2 General Evalution
22
F Relate the Selected Tokens' Loss Downstream Task Performance
G Examples of Tokens Selected by SLM
23
G.I Token Selected Examples
23
G.2 Dynamic Token Selected
23
Self-Reference Setting
23
Weak-to-Strong Generalization
18

---

A Author Contributions
Zhenghao Lin designed and implemented detailed token selection process, conducted extensive preliminary experiments, developed the pre-training and evaluation pipeline, conducted most of the pre-training experiments and analysis, implemented baselines, and significantly contributed to the writing. Zhibin Gou presented a preliminary proposal, introduced the method of using excess loss for reweighting tokens, compiled high-quality corpora, trained reference models, set up the fine-tuning and evaluation pipelines, designed the experimental analysis, and significantly contributed to the writing. Yeyun Gong proposed the initial project and co-led the project with Weizhu Chen, they offered extensive advice and guidance on experiments and writing, and oversaw team collaboration and resource management. Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, and Nan Duan offered research mentorship, coordinated the project, and contributed to the writing.
B Related Works
B.1 Pretraining Data Optimization
The objective of optimizing pre-training corpora is to maximize the performance and efficiency of language model training by improving the quality and scale of the pretrain data mixture. This includes data collecting through crawling [Raffel et al., 2020] or synthesis [Polu and Sutskever, 2020, Gunasekar et al., 2023]. de-duplication [Lee al., 2021, Kandpal et al., 2022, Tirumala et al.. 2023], filtering and selection [Xie et al., 2024a, Albalak et al.. 2024], as well as data composition [Xie et al., 2024b] and curriculum [Chen et al., 2024, Ma et al., 2024].
B.2 Data Selection
Data selection for fine-tuning has been extensively studied, focusing on improving quality [Li et al., 2023c], diversity [Liu et al., 2024], and distribution matching [Li et al., 2023d, Xia et al., 2024, Ni al., 2024, Kang et al., 2024]. For pretraining, various lightweight filters are utilized [Albalak et al., 2024], including heuristic-basee based (e.g.. language and item count filtering), classifier- [Brown et al., 2020], and perplexity or loss-based approaches [Wenzek et al.. 2019, Qin et al., 2024]. The dataset [Computer, 2023], for example. leverages over 40 quality
massive public CadPajama-Data--a indicators for data filtering and reweighting. Nevertheless, strict filtering like blocklist [Raffel et al., 2020] and Safety API filtering [Welbl et al., 2021], have been found to hurt evaluation loss or induce bias [Dodge et al., 2021].
Sample-level selection has been extensively studied in previous research [Sener and Savarese, 2017, Killamsetty al., 2021], particularly through online batch selection [Loshchilov and Hutter, 2015, Schaul et al., 2015, Chang et al., 2017, Katharopoulos and Fleuret, 2018, Jiang et al., 2019]. These approaches have been applied various classification tasks [Song et al., 2020, Mindermann et al.. 2022] and language modeling [Fan and Jaggi. 2023]. However, Kaddour et al. [2023] find that batch selection is not computationally efficient.
Many previous works have employed the general idea of using a reference model as a proxy for data selection. For instance, Selection Via Proxy trains proxy model to select samples with high uncertainty [Coleman et al., 2019]. Xie et al. [2024a] and Engstrom et al. [2024] utilize n n-gram models or datamodels with a target dataset to estimate importance weights. Additionally, Xie et al. [2024b] optimize the worst-case excess loss [Oren et al., 2019] relative to reference model to determine domain weights. One of SLM's scoring functions is excess loss, and the most relevant work related to excess loss is RHO-LOSS [Mindermann et al., 2022], which trains a small model on a holdout set and uses the difference between training loss and holdout loss select in-batch samples. Although excess loss is mathematically identical to RHO-LOSS. SLM differs in three important ways: 1) The focus is distinct. Motivated by the training dynamics of token loss, the core idea of SLM is to select useful tokens for pre-training. Its score functions are highly flexible and not limited to excess loss (see Appendix H for other functions). In contrast, RHO-LOSS aims to mathematically derive a reducible holdout loss to minimize generalization loss. 2) The meaning and training procedure of the proxy model are different. SLM trains a reference model on high-quality data to reflect the desired data distribution, whereas RHO-LOSS trains small model on random holdout set. 3) The selection scale and granularity vary. RHO-LOSS selects sample-level data on a small scale (typically 1K-1M samples) for task-specific fine-tuning tasks such MNIST [LeCun et al., 1998] and SST-2 [Socher
19

---

et al., 2013]. In contrast. SLM conducts fine aieeggrined token-level selection on large-scall scale language model pre-training, involving up 80B tokens.
Token-level training strategies have also been explored, especially for the pre-training of BERT-like models using Masked Language Modeling (MLM) [Devlin et al., 2018]. Specifically; "selective masking" involves masking important tokens in the input to focus on learning tokens that are more relevant to downstream tasks [Gu et al., 2020, Lad et al., 2022], whereas "token dropping" aims to reduce training costs by omitting less important tokens [Zhong et al., 2023a, Hou et al., 2022]. [Li et al., 2023e] assesses the quality of each token based on the skewness of its predicted distribution and truncates the noisy tokens during training. Additionally, some research has approached the analysis and detection of under-trained trained tokens from a tokenization perspective [Rumbelow and Watkins, 2023, Land and Bartolo, 2024]. To our knowledge, we are the first to explore token-leve data selection for large language model training, aimed at enhancing data quality and information density at the most fundamental granularity.
B.3 Language Model Training Dynamics
Investigating the training dynamics of language models is essential for understanding their behavior throughout the training process. This research includes studying internal representations [Saphra and Lopez, 2018], the acquisition of linguistic knowledge [Choshen et al., 2021, Liu al., 2021], and the phenomenon of grokking [Power et al., 2022]. The analysis by Xia et al. [2022] is the most related ours, which examines token-level training trajectories in models of varying sizes. Our findings, however, diverge from those of Xia et al. [2022], who posit that tokens with little change in perplexity are "already learned"'. We identify spectrum of token patterns, including "easy tokens" and "hard tokens" that resist convergence. Recognizing this, we propose method of selective language modeling that targets the influential tokens, optimizing the learning process.
B.4 Scaling Laws
Scaling laws guide us in discovering the impact of factors such as parameter count, data size, and compute on language model performance and behavior. These studies usually foeus on predicable scaling though power law [Kaplan et al., 2020, Hernandez et al., 2021], optimal resource allocation [Hoffmann et al., 2022], downstream tasks [Wei et al., 2022b, Isik et al., 2024, Gadre al., 2024], architectures [Tay et al., 2022], memorization [Tirumala et al., 2022, Carlini et al., 2022, Henighan al., 2023, Biderman et al., 2024], and repeating data [Hernandez et al., 2022, Muennighoff et al., 2024, Xue et al., 2024]. Most scaling laws on model performance study cross-entory loss on all training tokens, while we focus on the tokens loss of desired distributions.
C Limitations and Future Work
Generalizability In math continual pretraining, as depicted in Figure 6, training exclusively with SLM leads quickly convergence the domain focused by the reference model, accompanied by significant rise in the loss of unselected tokens. Although no adverse effects, like biases, have been observed from the increased loss yet, a general pretraining loss on text and code may prevent overfitting [Goodhart and Goodhart, 1984], as suggested by Ouyang et al. [2022] and Azerbayev et al. [2023]. Furthermore, future efforts could broaden the corpus scope of the reference model, and enlarge the pretraining data size, exemplified by DeepSpeedMath [Shao et al., 2024].
Scalability Due to budget constraints, we have only verified the effectiveness of our method on smaller models (<=7B parameters) and smaller datasets (<100B tokens). Smaller models benefit significantly from removing the loss of irrelevant tokens and focusing on important ones. However, it's possible that very large models trained on extensive corpora may naturally develop this inductive bias to compress useful data (i.e., compressing everything), although may sounds inefficient for now. Therefore, future works should study whether this selective language modeling technique can scale to very large models and data [Kaplan et al., 2020].
Is training reference model necessary? To score tokens, we need high-quality reference model. This could be a base model trained with small amount high-quality data, or a performant open-source model. In fact, since only need input logprobs or perplexity from reference model,
20

---

[a] Loss for different token types
(b) Example L-4. tokens
[c) Example H- tokens
0.15
4.5
4.0
3
0.10
3.5
3.0
(3156)
0.05
1
(12%))
2.5
Hush Toxon 2
Hish Toxen
27-41
0.00
2.0
6
15
10
15
Trained Takens(B)
Trained Tokers(B)
Trained Tokens(B)
Figure 10: The loss of four categories of tokens during Mistral-7B pretraining on MetnWebMath. (a) shows the loss of H-*H, L-+,, H-L, and L-L tokens during pretraining. (b) and (c) show three cases of fluctuating tokens' loss in L- and H-+H during pretraining, respectively.
we could even utilize more powerful proprietary model APIs. We can input tokens and use the log probabilities of the input returned by the API as reference scores. We leave this for future works.
How to improve upon SLM? There are many natural extensions of SLM, e.g., reweighting tokens instead of selecting may improve robustness; using a reference model as reward model to guide pretraining with reinforcement learning; adopting multiple reference models to reduce overfitting; designing token-level curriculum learning and iterative strategies for continuous improvements, etc.
Expanding the use of SLM SLM may be extended supervised fine-tuning to address the noise and distribution mismatches in many SFT datasets. Another potential application is alignment, e.g., by training reference model to emphasize helpfulness, truthfulness, and harmlessness, we may obtain base model that is natively aligned during the pretraining stage. Meanwhile, we believe that the idea SLM may find broader applications in multimodal data such as images, videos, and speech, which have a high noise-to-informatio ratio than text.
D Analysis and Visualization of Tokens in Pretraining
D.1 More Details of Four Categories Tokens
We categorize tokens into four categories: H-H, L-H H-,HHHHH-LL H--,, L-+.. During the training process, we collected the loss of each token after training on each 1 billion tokens training data. We then used linear fitting and took the difference in loss between the first and last points as evidence of whether the loss decreased during the training process.
Specifically, suppose we have sequence of token's loss (lo, (,,,,.,,))) Our goal is to minimize the sum of the squares of the differences between each data point and its linear predictive value:
f(a, b) = minimize Σ (44 (ax; + b)))
(6)
1=0
where .op = 0 is the initial checkpoint and = n is the final checkpoint. Substituting these into the fitted equation, we can obtain the Loss values at the start and end after fitting: Lstant = b and Lond an + b. The change in loss can then be expressed as: AL Land Cstant Meanwhile, represent the average Loss of the last checkpoint as Lmaan-
Next, we can classify the tokens based on AL and the Luean- We categorize tokens with AL < - -0.2 as H-L (loss decreases from high to low) category tokens, and tokens with AL > 0.2 as L-H (loss increases from low to high) category tokens. If -02 < AL < 0.2 and Ln < Emean- then tokens are classified as L-L (loss remains low); if I, > Lmean- they are classified as H-+H (loss remains high). In Figure 10, we have added the tokens" loss curves of the 7B model which is consistent with the other experimental settings in 12.1, for readers to refer to whether similar phenomena exist on larger models. In Figure 11, we visualize examples of the four categories of tokens in actual text.
21

---

D.2 Non-Converging Tokens in Pretrainig
In $2.1, we mentioned that during the training process, only a minority of tokens belong to the H-L category. Among the remaining categories H-H and L-L tokens, there are tokens that exhibit significant fluctuations during training. Furthermore, there are instances where H-L tokens are not effectively learned. Therefore, in analysis, we specifically select those tokens from these categories that demonstrate considerable variability and distinct loss. We visualize these tokens that exhibit abnormal behavior during the training process. As illustrated in Figure 12, we find that the majority of these tokens originate from rather chaotic corpora. For instance, the corpora may include mix of custom symbols, unintelligible gibberish, and information such as timetables and bibliographic references. Within a segment of normal text, there may also be fluctuations in the usage of common conjunctions, word suffxxes, and punctuation marks. The latter may not necessarily be disastrous for training; in fact, it could represent a normal occurrence. However, if we can effectively mitigate the losses caused by the former, might lead to more stable and efficient model training.
E
Evalution Details
E.1
Math Evalution
We conducted a comprehensive evaluation of the model across various math reasoning benchmarks, encompassing a range of difficulties from elementary to university level, multiple mathematical domains, and diverse question types including multiple-choice and open-ended questions. Our benchmarks include GSM8K [Cobbe et al., 2021], MATH (Hendrycks et al., 2021], GSM-Hard [Gao et al., 2022], SVAMP [Patel et al.. 2021], ASDIV [Miao et al., 2020]. MAWPS [Koncel-Kedziorg Kedziorski al., 2016], TabMWP (TAB) [Lu et al., 2023], MathQA (MQA) [Amini et al., 2019], MMLU-STEM [Hendrycks et al., 2020], and SAT [Azerbayev et al., 2023].
E,2 General Evalution
In the evaluation of general domain, we followed the Im-evaluation-harness [Gao et al., 2023] and evalute model on MMLU [Hendrycks et al., 2020], BBH [Suzgun et al., 2022], AGIEval [Zhong et al., 2023b]. ARC-EaR Easy and ARC-Challenge [Clark et al., 2018], BoolQ [Clark et al., 2019], PIQA [Bisk et al., 2020], Hellaswag [Zellers et al., 2019], WinoGrande [Sakaguchi et al., 2021 1] Open [Mihaylov al., 2018). On HumanEval [Zheng et al., 2023] and TydiQA [Clark et al., 2020), we follow the evaluation pipeline of open-instreut [Ivison et al., 2023] and report Pass@1 and Pass @ 10 for HumanEval and F1 for TydiQA. For MBPP [Austin et al., 2021] benchmark, follow the evaluation pipeline of DeepSeek-Coder [Guo et al., 2024], and report Pass and Pass 10.
F Relate the Selected Tokens' Loss to Downstream Task Performance
In this section, we declare the details about correlating the loss of selected tokens with the performance of downstream tasks. Concurrent study has explored similar methods to study the impact of scaling laws with the performance of models in downstream tasks [Gadre et al., 2024]. Our analysis here differs in that it aims to elucidate the relationship between the decrease/increase in loss for selected/unselected tokens and the model's ss performance on downstream tasks.
We use the average accuracy of MATH and GSM8K as the standard for measuring downstream tasks performance of model. Based on the trend of data points in Figure 7, we propose the relationship between the average accuracy of downstream tasks and selected/unselected tokens' loss,
Acc( = log (a * L cc
(7)
The parameters and are fitted from the data. If the loss of selected tokens C, is used for fitting, then O. Conversely, the loss of unselected tokens Lus is used for fitting, then a < O. Therefore, we believe that training the model on selected tokens can effectively improve its performance on downstream tasks, while unselected tokens may have a detrimental effect on the model's 's performance in downstream tasks.
22

---

Table 4: Full Self-Reference results on Tinyllama-I.IB.
Score Function
Select Ratio
GSM8K MATH SVAMP ASDiv
MAWPS MQA
AVG
100%
6.3
2.6
21.7
36.7
47.7
13.9
21.5
90%
7.4
4.4
23.4
38.7
51.9
14.4
23.4
80%
6.4
4.6
23.1
39.7
52.0
14.3
23.4
(xs)
70%
6.7
4.6
23.3
40.0
54.5
14.3
23,9
60%
7.0
4.6
22.2
38.5
52.2
13.7
23.0
50%
5.7
4.2
20.7
36.7
46.7
10.3
20.7
90%
6.7
3.0
23.7
40.3
52.3
13.1
23.2
80%
6.8
3.6
22.5
40.6
52.9
13.6
23.3
(x4)
70%
7.0
4.8
23.0
39.3
50.5
13.5
23.0
60%
6.5
4.8
26.5
37.3
49.7
15.6
23.4
50%
4.7
5.8
20.9
33.8
42.5
11.1
19.8
50% U 70% (80%)
6,4
3.6
22.7
38.4
52.6
15.3
23.2
70% U 60% (77%)
6.3
4.6
24.4
39.6
51.4
16.3
23.8
(x:) U CRM (2,)
70% U (75%))%)
6.9
5.6
23.2
39.9
52.9
12.6
23.5
60% U (7%%)
6.7
5.2
24.7
39.2
50.6
14.6
23.5
60% 50% (68%))
5.8
21.7
37.3
49.6
15.3
22.8
60% U 40% (65%)
7.3
6.0
23.6
36.9
48.6
13.1
22.6
80% O (76%))
6.0
4.4
23.7
38.5
51.2
13.3
22.8
75% n 75%(67%)
5.2
24.2
39.4
54.9
14.7
24.4
(x)( O LRM
70% n 90% (68%)
6.8
4.6
22.2
40.3
53.0
14.8
23.6
80% 80%
8.2
6.4
21.2
39.1
53.4
15.0
23.9
70% n 70%
7.1
5.0
23.5
41.2
53.8
18.0
24.8
G Examples of Tokens Selected by SLM
G.1 Token Selected Examples
In Figure 13, present several examples of tokens selected by the SLM method, with content marked in blue indicating the tokens actually chosen during the pretraining process.
G.2 Dynamic Token Selected
In Figure 14, we display the dynamic changes in token selection tendencies throughout the SLM training process. We chose four checkpoints during the training process (0%, 33%, 66%, and 100%) to analyze the current tendencies in token selection. The preferences for token selection are indicated by different colors, ranging from high low preference, typically represented deep blue, blue, black, orange, and dark orange, respectively.
H Self-Reference Setting
In this section, will provide detailed introduction the reference loss score function and information entropy score function SLM. Reference loss score function to directly use the loss of the reference model the basis for selecting tokens. The higher the token's loss of the reference model, the lower the expectation that the token will be selected. The score (RM(#;) can be directly obtained by referring to Equation Information entropy score function is to select the corresponding token based on the information entropy of the reference model in each token. The information entropy of token æ; can be expressed as:
V
HaM():)
P(ta|=<e) log P(tx(aee),
(8)
A=1
23

---

Table 5: tana--t result math benchmark.
MMLU STEM
Model
Train Toks GSM8K MATH SVAMP ASDiv MAWPS
TAB MQA
SAT
AVG
15B
28.4
13.6
50.3
62.8
79.5
37.6
34.1
41.6
43.5
Llama-2-788 2-7---2---CTT----- IB RM
10.5B
29.8
16.0
55.5
63.7
80.4
37.9
34.3
38.2
43.8
44.4
where 4 represents the i-th token in the vocabulary, and represents the size of the vocabulary. The intuition of this strategy is that the higher the information entropy, the higher the uncertainty of the token in the context. Therefore, we consider that the language model still uncertain for certain tokens after pretraining,
do not expect that the language model will Iearn during pretraining. In Table 4, we provide more SLM results, including different select ratios and combinations two score functions, for the convenience of the readers to refer to.
I Weak-to-Strong Generalization
Apart from the main experiments where we use the same base model for the reference and continual pretraining, we also investigate smaller reference model can effectively guide the pretraining of a larger model. We use Tinyllama-1.. 1.1aaa1..BBBB as reference model and continual pretraining Llama-2-7B on 15B Open WebMath tokens. Results presented in Table indicate that, despite the considerable gap between the small and large models [Li et al., 2023f], employing the small reference model to token selection can still yield benefits to the pre- pre-training the larger model. If reference and training models have different vocabularies, one can consider performing token alignment [Wan et al., 2024, Fu et al., 2023], which we leave for future work.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and $1, we clearly demonstrate the contribution and scope of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Appendix C, we have thoroughly discussed the limitations of our article, hoping to guide more future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [Yes]
Justification: $2.1 and $2.2, we elaborated on the motivation and theoretical derivation of method, with complete proof process in place.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided detailed descriptions of the experimental setup in $3.1 and methods in $2.2 to ensure that our experiment can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: This may be temporary, and we working hard promote the process of open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
wwweemissssssilliiiioeoom (iPolllyy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //ippscccpppiiiiiio wwtppcccccc///cccccccccccccccclt wwwp///g////ee/eee/eeeeceec.seet https//oooSSuumissisee.e
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In $3.1 and Appendix E, we clearly demonstrated various experimental settings, including hyperparameters, model settings, training settings, evaluation settings, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the high cost of pre- pre-training and the significant results obtained across various settings, we do not repeat the same experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In 13.1, have provided sufficient information the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /entps://neuriii wwweurrip..cppubllii cc/public//niccccccccccccceceeee /Ettii///iiieeeee htte/EEhhiiissieeeeeeeeeeeeeeeee csGuidelines"
Answer: [Yes]
Justification: We guarantee that the research conducted in the paper complies with NeurIPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The purpose this paper is to improve the training process large language models, without any negative societal impacts.
Guidelines:
28

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The creators or original owners of the assets used in the paper, such code, data, and models, have been appropriately recognized, and the licenses and terms of use have been clearly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
29

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31

---

Examples of Four Categories of Tokens
GMAT 1: 670 Q49 V31 in GMAT 710 Q50 V35 'n Followers: 175 In 'In Kudos [?]: 890 [0], given: 235 In In Re: Mr. and Mrs Wiley, VIC[4permalink] 13 Feb 2010. 01:03 la Ans A in in their first child was born after J years.. An An thus I child ->j j years in ln => thus after another J years his age J ln In thus his age is J J>> after 2J years and 2j after 3j years in \n his present age is T which is after T years. \n \n thus total time after 2years will be T+2 In since after every J year they have a child after T+2 they will have Vfrac (Trac((T+5)) (J) + +1 ( (+1 is for the oldest) In An thus A 'n
In In Fight for your dreams :For all those who fear from Verbal- lets give it a fight 'n In Money Saved is the Money Farned 'n in Jo Bole So Nihaal, , Sat Shri Akaal In \nn Gmat test review: 141 (4411442.hmm in 'n Intern ln Joined: 06 Apr 2012 An Posts: 28 In Followers: 0 In in Kudos [?]: 4 [0]. given: 37 In \n Re: Mr. and Mrs Wiley, VIC[4permmalink] 21 Nov 2012, 07:46 In jecteshsingh wrote: In Need the solution using Algebra... \n in Mr. & Mrs Wiley have child every years. Their oldest child now years old. If they have child years from now, how many children will they have in total? In in (A) (frac +1 \n In (B) JT + 1 \n (C) (J
In 670-1o-7100
Vfrac [[T] In (I) TJ- An In (E) ( [J] in [Reveal| Spoiler: OA: In (A) In Source: Manhattan Guide \n Bunuel- would really uppreciate you providing your bit on solving the original problem above algebraically. The problem and various explanations remain confusing. Should we think of as a progression or some other way? Please share your take. Thank you. in Veritas Prep GMAT Instructor In Joined: 16 Oct 2010 In Posts: 4566 \n Location: Pune, India Followers: 1029 in Kudos [2]: 4460 [1], . given: 162 'n in Re: Mr. and Mrs Wiley, VIC[#permalink] 21 Nov 2012, 09:45 In In KUDOS in Expert's 8 post in jeeteshsingh wrote: in Need the solution using Algebra.... In 'n Mr. & Mrs Wiley have a child every years. Their oldest child is now T years old. If they have child 2 years from now, how many children will they have in total? in An (A) \frac + 1 An \n (B) JT + 1 An An (C) Vfrac (T) + Afrac(] I) (T) In in (D) TJ- In in (E) Mfrac [+J) (J) In in [Reveal] Spoiler: OA: In (A) An An Source: Manhattan Guide in An Think of it as an Arithmetic Progression where every subsequent term (child) has a difference of J yrs from the previous term (child). \n In 1t child, 2nd child, 3rd child, ...... nth child (to be horn after 2 yrs) In In What is the difference between first and last terms (children)? (age difference between two consecutive kids)? J yrs in In What is the number of terms (children)? (T 2y/J + 1 An (Number of terms of an AP is n = (Last term First termo/Common Difference + 1. 1.) 'n
+ 2) yrs An An What is the common difference
An In Karishma in Veritas Prep GMAT Instructor in My Blog in In Save $100 on Veritas Prep GMAT Courses And Admissions Consulting Enroll now. Pay later. Take advantage of Veritas Prep's 's flexible payment plan options. Veritas Prep Reviews Re: Mr. and Mrs Wiley, VIC [*permalin]] 21 Nov 2012, 099:45 Similar topics Replies Last post Similar Topies: Mr. and Mrs. O'Leary (SC) 5 08 Jul 2012, 07:15 Mr. INVESTOR invested total of$12,000 for one-n--ee year 30 Mar 2007, 09:24 \n 2 Mr. and Mrs. Wiley have a child every J years. Their oldest 7 19 Feb 2007, 11:40 \n Mr. kevincan 6 16 Aug 2006, 12:26 ln PS: Mr. & Mrs. Smith 2 06 Dec 2005, 00:03 An Display posts from previous: Sort by Sciencemadness Discussion Board > Fundamentals Reagents and Apparatus Acquisition Sulphuric Acid Australia Select Forum Fundamentals x Chemistry in General Organic Chemistry Reagents and Apparatus Acquisition Reginnings w Responsihle Practices Miscellancous The Wiki Special topics > Technochemistry > Energetic Materials Biochemistry > Radiochemistry > Computational Models and Techniques > Prepublication Non-chemistry > Forum Matters > Legal and Societal Issues in Pages: 1 2 Author: Subject: Sulphuric Acid in Australia hissingnoise International Hazard Posts: 3939 In Registered; 26- 12-2002 In Member Is Offline in Mood: Pulverulescent! \n I've stated several times on various threads, that SO<sub>3</subb produces a practically incondensable acid mist when led to water and, BTW, at 700 the decomposition rate of SO<sub>3</suh>> is '87% ... In Cracking Macsubs2</uubs/esuubbeee www.ea/wwwssbbssssuuuseeeeeee proceeds at "466" and the issuing gasses are readily absorbed by conc, Saaaulsse//uub--ssuuubbbeee
to form oleum! In \n Phthalic Acid In Harmless in in Posts: 19 In Registered: 7-8-2011 I \n Location: Australia in Member Is Offline in In Mood: No Mood In An That's a good idea Neil, I'll be sure to try that next time (probably for H202). Just went to Tradelink and asked if they sold Moflo drain cleaner. The guy said yeah and I asked for a liter of it. No problems whatsoever, he just said "he careful with it". It was $45 hut liter will last me a while and making it myself would ve been vastly more expensive I imagine. Success! Synhh Hazard to Others Posts: 107 Registered: 29-7-2011 1 Member Is Offline Mood: http://ww. http://wwwyuutubb.oom www.ube.com/wwtteee. htmm/wacch/-SSltgg VuDlo Sulfuric acid can be produced in the laboratory by burning sulfur in air and dissolving the gas produced in hydrogen peroxide solution. SO2+ + H2O2 + H2S04 this was found on wikipedia... did you not look through the sullfuric acid wiki before boiling down batery acid? anyways... There are some good videos on youtube that demonstrate how to synthesize sulfuric acid using different methods. The drain cleaner you get from the store will he impure and may contain organic matter that discolors the acid.
Figure 11: Sample text containing four categories of tokens. Among them, blue represents tokens of categorie H-I., green indicates tokens of categorie I.-,I, yellow signifies tokens of categorie H-+ and red denotes tokens of categorie I.-H.
32

---

Examples of Tokens that Exhibit Abnormal Behavior during Training
as in in lbegin ( a A Vin V'pm lbegin (bmmarrix/cossttheetttt Neosttheta & Asinntheta W Winnthhtt & lcosttheta W Vend [bmatrix] ). pm \begin( ( bmatrix |cosktheta & Asinhthet W ssinhtheta & "coostheeaa W Vend (bmattrx), 'pmm \begin (egiin(bmn (bmatrix sinhhthela & Acooshthheta W Scosh\theta & Asinh'theta W. (bmatrix)) \pm Vbegin (bmmatrix Ji ksinhkhheta (quadkuad(quad((5)) In 'n I suspect this class transformations has name in the grand group classification scheme, but don't tt know what is. #iH Mathematics Class 'n 'n Unit-I: Sets and Functions 'n Chapter 1: Sets \n Unit II: Algebra \n Chapter 5: Binomial Theorem 'n Chapter 6: Sequence and Series 'n Unit-lll: Coordinate Geometry 'n Chapter I: Straight Lines in Chapter 2: Conic Sections \n Unit-IV: Calculus In Unit-V V: Mathematical Reasoning 'n Unit- VI: Statistics and \n Graphs of Trigonometric Functions In (i) Geometry in any held. Queries are case independent. Funct Wildeard queries are specified by (e.g. functions, functorial, etc.). Otherwise the search is exact. "Topologgcal group" Phrases (multi-words) should be set in "straight quotation marks". au: Bourb aki & ti: Algebra Search for author and title. The and-operator & is default and can he omitted. Cheb yshev ITsscchyschefff The or oproppetoor I allows to search for Cheb yshev or Tschehyscheff.. "Quasi" map*" py: 1989 The resulting documents have publication year 1989, so: Eur* J Mat Soc" cc: 14 Search for publications in particular source with Mathematics Subject Classification code (cc) in 14, "Partial diff eqt") elliptic The not-operator! ! eliminates all results containing the word elliptic. dt: b & au: Hilbert The document type is set to books; alternatively: j for journal articles, for book articles. py: 2015 cc: (94A 11T) Number ranges accepted. Terms can he grouped within (parentheses). la: chinese Find documents in a given language. ISO 639-1 language codes can also be used.
Noosh W leosh\theta & - Isinhttheta W (bmatrix (Veendaaligeee
Probability in Chupter I: Statistics in Chapter 2: Probability
Code: Select all in in 64.y y 86, rule B3/S23 in 2tbos5hooooooo52222664444444aaa3 20 412b2:44h2o33olIbbss
$13620399
6bo2bo$bu506 An o3b2u82bo49bo
2a6b oboS5lbob
7bo497bb4
13c4763o$10bo50
\n 42bo59b204b20
42b 2o$17bo7S13boS36
obo6bo54b
206b
30$4bo$54bo554
obb$13b 20 39b2oS 22:2444h20$3ll 2o
206b
2o6b30$4bo$54ho$
$4bbbbo
33b2039220$ 222os1222244
12b2244420$$3 2bo40
Nou2bobbu0b22otnn 666bbu55 bobobbbbbbbboooooo
(bbb657bobba39007 13bbbs3bb obo6bo54b
206b $o$4b5554b oboS13b 2039b 20$ In 12b 2044b
2oS3o1 lbo43bo3b
206b obo$5 lb obo7ho$5bo49ho 3o47b 5o$10bo5b 2042bo87b
2obo4b 2040b 20bo57b obo7bb39b obo! The 16 bittirter thus goes down to 9 gliders. It does not reduce any further
17-bitters, though. Princess of Science. Parely Taxel Kazyan Posts: 867 Joined: February 6th, 2014, 11:02 pm #MH
Re: 17 in 17: Elficient 17-bil synthesis project Good catch, Sokwe. #61 in ISG:
Ground Penetrating Radar for Archucology in In Workshop Decemberl 1-5 p.m. 1012251 College (Archacological Research Facility) in In Scott Byram, Research Associate, Archaeological Research Facility, UC Berkeley in In Archaeological Research Facility in in At Ipm the workshop will begin at the UC Faculty Club lawn where subsurface features are being mapped. in \n ### Student Probability/PDE: Seminar: Seminaa Large Deviation Principle for random graphs II In in Seminar December! 2: 10:30 p.m. 8ynEs Hall \n \n Fraydoun Rezakhanlou. UC Berkeley \n \n Department of Mathematics \n \n ### BLC Fellows Forum In 'n Presentation Decemberl I 3-5 p.m. Dwinelle Hall, B-4 (Classroom side) In FAll 2017 BLC BLCllloos, UC Berkeley In In Berkeley Languuge Center in In Teaching French Listening Comprehension and Cultural Awareness through Regional Variation In Elyse Ritchey, GSR, French 'n At the university level, French language instruction in the US traditionally includes course on phonetics and pronunciation. While the major aim of such courses
improve students' speaking and listening competence, they also emphasize speaking 'correctly" using... More > In In ### MENA Salon In in Workshop Decemberl 3-4 p.m. 340Stephens Hall In Every Friday in the semester, the CMES hosts an informal week
Figure 12: An example of an abnormal state of token perplexity during pretrainig process. The tokens highlighted in orange represent tokens that were significant abnormalities during the pretraining process.
33

---

Token Selected Examples
Process the student answer as a Math Object Formula, and break down its parse tree by its top-level operators. The idea is to create an array of the student's s primitive factors, so say 3xx+1 gives (3xx+1xx+2). Because we may want factoring over Z, checking the ged of coefficients within each factor. Pass each of these things to SAGE and ask if the nonconstant factors are reducible over Z or Q. Also ask if they are monic. These things at least we learned how to do at the Vancouver code camp. The end goal is to count the following forms as correct, possibly controlled by flags: n M prod (factor) power, where each factor is irreducible in Z[X], n in Z r M) prod (factor) power, where each factor is irreducible and monic in Q[X]. r in Q I suppose on the last one the monic requirement could be dropped with a flag. have no plans to check that the form is fully condensed, e.g. forcing (x+1)"2 and rejecting (x+1 (1+xx)
The equation of the path traversed by projectile is called equation of trajectory. 'n in Suppose, the body reaches the point P after time (1) () in In Horizontal motion has no acceleration. Thus, using kinematic equation, horizontal distance covered will be in An x = u Aoos Wheta t (n in Or, lquad t= ( (Mrrac [ x ] loos lthet }) \n in Vertical motion has constant acceleration (g) ). Thus, distance covered will be - Am An = ( Asin lhhea ) t- 'lftt /frac
(1) bright) ) g r22 In ln (u Isin \theta) Meftt ( (frac [x] (u leos \hhea) \right) )- Veft (Mrcc (1) Wright ) Meft C Mrue (x) [u lcos [hhet] bright y"2 In in= Ueft ( Man ltheta bright) x Meft ( (frac (g) (2 u²2 lcos \theta) bright ) x'2 In In In this equation, Mthtaa, Nu \ Mext [and] \g) are constants. Thus, in in 1. Term Meft ( Man khhta bright ) is a constant, let it is (p In 2. Term Meft [ lleft ( /Mrac [g [2 2 loos"2 22 \theta] \right) ) bright/ is also constant, let is (g) (g)w in in So, 'quad = q x in in Therefore, y 'propto x²2) ) which is required condition of parabola. The trajectory the projectile a parabola. in in MWN Time of Maximum height
in As the body is projected it goes up. Vertical component of velocity (u Isin \heta) gradually diminishes and becomes zero the maximum height of flight. After that, body starts moving downwards. In in Let, ( (Lmm) ) is the time to reach maximum height ( (h_m) of flight. in in Therefore, from kinematic equation, we have In An 0 isin Wheta-- -g f_m 'n Or, lquad t_m= Meft Miaa [u Isin uheta| [g] bright) ) \n ### Time of Flight in In Total time taken by the projectile hetween the instant is projected and till reaches a point the horizontal plane of its projection called Time of flight. in Let, the body reaches point on ground after time (T_f) of projection. Then in In Net vertical displacement covered during the time flight is zero. Using kinematic equation of motion, we get \n
U
Asin "theta) ) T_f- Meft Mfaac (1) \right) ) g (T2 In In Or, Vquad T_f= left ( (Mrrac {2 u isin \theta] [g] \right >= Neft Afaac [u Isin ltheta) (g) bright) 1n= Lm in Thus, lquad ttext (Total time of flight] = Mext (Time ascent) Mext (Time descent) in 2 stimes Mext (Time of maximum height.) An In MND Maximum height of Flight In It is the maximum height reached by a projectile. It denoted by h m In In the highest point of flight, the vertical component of velocity becomes zero. in In From kinematic equation motion, we have In In v²2 = u'2 2 a in in Therefore, lquad 0'2 - (u Isin theta )22 = 2 g h m In Or, kquad h m Meft Mfrrc (u'2 Asin 2 \theta] (2 g) bright)
We identify two equations having the same solution with the equivalence relation: An In S(a.b) Isim (c.d) Imbox if and only if } ad be$ in in To show that this is an equivalence relation: \n An 1. Reflexivity: $S(a.b) Isim (a.b)$S if and only if SSab = buS$ which is true. Hence it is reflexive. An 2. Symmetry: $S(a,b) \sim (c,d)S$ if and only if SSad = bc$$ if and only if $Sbc = adS$ if and only if $S(c.d) Isim (a.b)S$. Hence it is symmetric. An 3. Transitivity: bS(a.)) Isim (c.d)SS and $S(c.d) Isim (s,SSS if and only if SSad bcSS and Stcf = deSS. Multiplying these equations together, we get $Sadef = bcdeS$. We can cancel SSdSS and S$cS$ from both sides to get $Saf = be$S. Hence bS(a,)) Isim (e.S) In in Hence, we have successfully formed the set of rational numhers when we factor out the equivalence classes! In In Slmathhb Q) = \frac| |\mathbb/ /Z] trimes \mathbb 77.Ahccsslssh/// } [SsimSS In An Let's s now take a look at what members of S$mmtthbb Q)S$ look like, say for the equation $52x = 3S$. This equation is represented by the ordered pair
Il the light moves in purely radial direction, we can describe its path by the coordinate functions $5: and SSr(aambaa)SS. The equation of motion Sidds"2 =0$$ then takes the form SSg_[t]) [tt] (Ifrc (dr) ] ddllmmdda /rright)2 {m() (ertt (dr (d'lamhda| Saaaaa/rii /rrghh) = 0,5$ which we can rewrite
(frre (dt) (dr (drr/rihh))2 Mfrac| ((r) (ggrt] J.SS In In The length of the rod is then SSL= = e \int_ (r___rr___________ [ Vfrac dt] | [dr] Vext( d]r= c ]} \text( dSs where have taken the positive square root because 2$ in 'n Notice that the length is independent of the signature of the metric, whether you work with the (+++) or (+-) metric is purely conventional and will not change the physics. \n An For the Schwarzschild metrie, we obtain explicitly S&L= = -r_ + s Mulleft(Mfrac (r 2 -r.s [r_
lint (r_1""r__ 'sqrtt-Afieee (frrac (rr) [
(tt) [g
2 In An Now what happens if you mugically, instantaneously increase the mass of the black hole? I think the length SSLS$ of the
rod stays the same (I ' here assuming that the rod infinitely stiff). but that it would now "appear shorter" to the distant observer- - i.c. it would no longer occupy the entire space between $$r 1S$ and $Sr 28$.
Figure 13: Specific examples of selecting tokens during the selective pretraining process of the RHO-1. The tokens marked in blue represent the actual tokens trained during the training process, while the remaining black tokens are not trained during the training process.
34

---

After Training 0% Checkpoint
Item Type: Journal Article Copyright of this article belongs to Elsevier. Division of Mechanical Sciences >
Mechanical Engineering 28 May 2007 19 Sep 2010 04:36
Question
#8de97 In In Dec 10. 2016 'n In That is not un identity. \n \n HMM Explanation: In An Recall that In \n S("cot] [2]
x I= [tesc] xS. \m in So. we can write in \n SMruec - [[)ssc))((2 [2] x) [esc) ) 2 x = =Mrrac (1 - Meeft (V [2co x
[hrigh)) (tesc] 2 x8 In \n S= [)col]((2) Miracx]]] [(sss) 2 x5 In in Recall also that $1oot x = \cos Mircc|]] [)sin]
xS and Slese x = \frac|] [ [(sin] xS. \n In This allows us to continue 'n \n S= Virac (rrac((ccos) ['cos][[2] Vfrac [x] [|sin] 2
x [ (1] [ 2 x in in $= [ (2] frrac xx] [ssin] > 2 x tedot ['sin [2 [2 [xrac[ [1] In In $= (1cos]"]
xS In In Which is not identically Sicos x$. \n An (S[ [[o))][2]] x = lcos xS only when $1oos x = 1$ or S0S) In \n Dec
10, 2016 In \n No. It is equal to S( [/)ssnn]]]]] (ssiin[[2])) [[] X 1S. In In ### Explanation: \n An If we have Sirrac [1 x [x]S, we
can write it as Strac [1rac[1]] (1) - V [x] (S. \n \n The same way, Sifrac - (Besc] x] [Vesc] > 28 can be written
as MMrra [1] [besc]
x- -Vircc (Dese) ) [2] X (Yese] [1] xJS. 'n 'n This equal to S (()sin]"(2] x- 1S, SMS scnews
item created by Hannah Bryant at Wed 25 May 2022 1227 \n Type: Seminar in Distribution: World 'n Expiry: 31
May 2022 In Calendar1: 31 May 2022 1 1500-1 1600 In Call Io: Quad $224 & via Zoom In CalTitlel: SMRI 'hhat is
virtual knot?' Hans Boden (MeMaster University) \n Auth: hannahb @wwdw/w2w2222cc.c wwwldd/622ssaaffsyydnee. stt.sssyy...yyye..edd.. edu.a. (hbrys(83) (3)
After Training 33% Checkpoint
Item Type: Journal Article Copyright of this article belongs to Flsevier. Division of Mechanical Sciences
Mechanical Engineering 28 May 2007 19 Sep 2010 04:36
# Question
#8de97 In In Dec 10, 2016 in In That is not an identity. \n \n #i# Explanation: An \n Recall that in \n (2)
x+ 1= (2ese]((2 xS. \n in So, we can write In \n Strree {1 - [csce][(]] (Yese] 22 x2
\frrc - -Meee(( (2)
( 0 2 x \n \n S= [)cor][(2) Mfrac [x] ( (eesc) 2 xS In In Recall also that $1oot \cos \frac ((sin]
xS and Slese x Mfrac (1) ('sin) xS. \n in This allows us to continue In In S= Virac ( [2cos][[2] Mfrac [x] [[sin] 2
[Nircc [1) [1sin] 2 235 in in $= [)cos]((2) fruc| [x] [1sin] 2 ledot ['sin] [2] [x [1]S in \n $= [[cos][[2]
x$ In Which not identically Stoos x$. \n (S (1c))) x = Icos xS only when $1oos x = 1$ or SOS) \n Dec
10, 2016 in No. It equal to (ftsinn"(2] x- IS. In ###n Explanation: \n we have Strrae|| [1 (x]S, we
can write as Strac [1] - \frrcc [x] [x]S.
\n The same way, Sifrac - ()esc][(2) x] [lesc] 2$ can written
as Sifrac [1] [eesc] 2 x- ( [(ese] x] [Yese]"[22 x]S. 'n This equal to S("sin]]((2 x- 1S, SMS scnews
item created by Hannah Bryant at Wed May 2022 1227 'n Type: Seminar \n Distribution: World In Expiry: 31
May 2022 'n Calendarl;
May 2022 1500-1600) 'n CalLoci: Quad S224 via Zoom In CalTitlel: SMRI 'What is
...a virtual knot?' Hans Boden (McMaster University) in Auth: hannahb wwwll 11446222 slaff.yydney.cduu... (hbry8683)
After Training 66% Checkpoint
Item Type: Journal Article Copyright of this article belongs to Elsevier. Division of Mechanical Sciences > Mechanical Engineering 28 May 2007 19 Sep 2010 04:36
wpt.rintsiisc.enet...i.. infid//prinn/11277
Question
#8de97 In in Dec 10. 2016 in In That is not an identity. In \n #### Explanation In Recall that in An S (()c
x + 1 = [1csc]"(2) xS. \n In So, we can write In In SMrra ( (1- [Ncse] (2) (Vose ) x = Vfrrc - Meft([Ncot]](P x
[rrigh))
2 x$ An An S= [Ncor] [2) Mfacc [x] (Nese) 2 xS In In Recall also that $1oot x= \cos Mfrac [x] [
x$ and Stese x = Mfrac [ ('sin] xS. \n in This allows us to continue In \n S= Vrrac ( [ccos] [2] Mfrac [x] [[sin] 2
x) (Mrac 1) [ssin] 2 23$ in in $= [)cos]"(2] frruc [x] [[sin] 2 x lodot ['sin] [2] Mrac [1]] In An $= [Ycos] [[]
x$ in In Which is not identically Shoos x$. \n \n (S[ (("ccos)""" [2] x = lous xS only when $1cos x = 1$ or $05) ln \n Dec
10. 2016 \n In No. It is equal (o (('sin] [2] x 1$. In ln ###A Explanation: In An If we have $trac [1 1-xx [x)SS we
can write it as Strac (1) 103--free -ffrac(x [x][xS. In In The same way, S |rac| - [ecsc]"(2) x] (Nscc) * 2$ can be written
as $Mraac ] [lasc]" 2 -xrrrrc/ ( [\osc][(2] x H-HHHHcss (12) x1S. An in This is equal to S [[s) x 1S. SMS scnews
item created by Hannah Bryant at Wed 25 May 2022 1227 \n Type: Seminar in Distribution: World in Expiry: 31
May 2022 \n Calendarl: 31 May 2022 1 1500-1600 In CalLocl: Quad $224 & via Zoom In CalTitlel: SMRI 'What is
...a virtual knot?' Hans Boden (McMaster University) \n Auth: hannahb @w w 14n622 sta.2.stafffffff... staff..yddee.edd.. (hbry8 83)
After Training 100% Checkpoint
Item Type: Journal Article Copyright of this article belongs to Elsevier. Division of Mechanical Sciences > Mechanical iacaaaiialll Engineering 28
May 2007 19 Sep 2010 04:36 http://eprints..icceee.... int.///p////////////////// 1rin7/10227// # Question
#8de97 In In Dec 10. 2016 In In That is not an identity. In \n #ill Explanation In An Recall that in \n S (1)
+ = [lesc]"(2] xS. In In So. we can write in \n Srraac 1 [lcsc) (Yes) ) 22x = Mrrac (1 - Meeft (2))
[ 2 x$5 In An S= (Ncot)" [[] Mirac(x) lesc) 2 In In Recall also that $1oot = \cos Vfrac [x] (1sin]
x$ and Stese x= = Vfrra (1 ('sin) xS. \n \n This allows us to continue In \n S= Virac ( (Acos] [2(2) Mfrac [x] [Isin] 2
x x)rrcc 11 [ssin] 22 x155 In \n $= (22) |frac (x) [(sin] 2 lodot [>sin] \frac (1)]{1]]] 1.5 In In $= (lcos)
xS in In Which is not identically Shcos x$. \n An (5( [(cos)(((] x = lcos xS only when $ooos x 18 or $0S) \n An Dec
10. 2016 An \n No. It equal to (['sin]"[2] x 18. \n \n ###H Explanation: An An If we have Straac| [1 (x)S, we
can write as Strra [X] Vfrrc (x]]S. \n \n The same way. S Stace[ [lese]((2 (Yese) " 2$ can be written
as $frrac ] (lese) "2x x /frrcc (lesc)" x (\cscc"(2] x}S. \n This equal
x- 1S. SMS senews
item created by Hannah Bryant Wed 25 May 2022 1227 \n Type: Seminar 'nn Distribution: World in Expiry: 31 May 2022 \n Calendarl: 31 May 2022 1500-1 \n CalLocl: Quad $224 & via Zoom in CalTitlel: SMRI 'What
...a virtual knot?' Hans Boden (McMaster University) An Auth: hannahb @w] 144622 www.//22staf..ssdnnlooooo staff..ydney.eduu au (hbry8
Figure 14: An example of dynamic token selection changes during the training process, which illustrated with five different score levels represented by deep blue, light blue, black, light orange, and dark orange. The bluer the color indicates higher tendency for the token to be selected, while the more orange the color suggests a lower tendency for the token to be selected.
35


Reviews:
[{"review_id": "MhQWANJqdF", "reviewer": "Reviewer_QeQV", "summary": "The authors propose a method to train LLMs on the most influential tokens selectively. They suggest training a reference model on a small high-quality corpus using the standard CLM loss. They then compute the excess loss of each token in the training corpus as a difference in losses of the reference model and target model on that token. Finally, the target model is trained of the k% subset of the training corpus with the highest excess loss. The paper describes continued pre-training experiments for 1b and 7b models to demonstrate the effectiveness of this method. The experiments show improvements compared to standard continually pre-trained baselines and some open models in terms of performance on popular benchmarks and training efficiency (number of training tokens required to match the performance of open models).", "questions": "The end of section 2 talks about how tokens are selected for training in practice, it says that token selection can be implemented by ranking the tokens by their excess losses and only using the top k% for training. This seems like a crucial detail to ensure that the efficiency gains translate to training wall-clock time. How can this be done while maintaining token sequencing within samples?", "limitations": "While the authors have discussed the limitations of their work, an additional direction for the future could be to study the use of multiple domain specialist reference models to select influential training tokens.", "rating": 9, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The Selective Language Modelling method proposed in the paper is a novel approach to pre-training LLMs. The authors' experiments demonstrate significant improvements in training efficiency which is an important problem in LLM pre-training. The paper also describes a study of LLM training dynamics which could provide useful insights to other researchers working in the field for further exploring efficient token selection strategies for LLM pre-training.", "weaknesses": "The experiments in the paper are performed in the continued pre-training setting and the impact of the original pertaining performance is not discussed in the paper. It is possible that the method might not work well if the base model is undertrained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer QeQV, \n\nThank you for your comprehensive review and positive remarks!\n\n### **Pre-training from scratch**\n\n> The experiments in the paper are performed in the continued pre-training setting and the impact of the original pertaining performance is not discussed in the paper. It is possible that the method might not work well if the base model is undertrained.\n\nDue to budget constraint, we conducted experiments on a continued pre-training setting to verify the effectiveness of the SLM. We will rephrase the description in the paper to make it more rigorous and leave the original pre-training setting as future work.\n\n### **Detailed implementation of SLM**\n\n> The end of section 2 talks about how tokens are selected for training in practice, it says that token selection can be implemented by ranking the tokens by their excess losses and only using the top k% for training. This seems like a crucial detail to ensure that the efficiency gains translate to training wall-clock time. How can this be done while maintaining token sequencing within samples?\n\nIn the implementation, we maintained the token sequencing in each sample and simply removed the losses of tokens with lower scores in the output part (therefore, the forward computational cost of token loss was not reduced). Due to the removal of these low-score tokens, the SLM achieved better performance compared to training with all tokens when using the same amount of input tokens, thereby improving data efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0TkBRH4ZJl", "reviewer": "Reviewer_3eTg", "summary": "The authors explore how loss for specific tokens changes in continued pre-training and note that they fall into four categories (high->high, high->low, low->high, low->low) with each category having at least 10%. They run continued pre-training on tokens that are learnable and domain-useful (judged by reference model) and find that this leads to higher accuracy with less tokens used. The main results are in the math domain, but there are also a variety of other results (tool use generalization, general domain, etc.).", "questions": "Please see weaknesses above for some explicit and implicit questions.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "For originality, I'm not deeply acquainted with related work, but it seems that the authors are (based on Related Work section in appendix). This works seems novel and well-contextualized with respect to related work. The experiments are of high quality and explore a few domains/problems. The paper is generally clear and easy to read. I think this work seems significant in that future research/application could use it (especially with a particularly low-quality dataset).", "weaknesses": "* The greatest weakness (in my opinion) is that \"tokens\" in many cases could refer to \"number of tokens after % filtering\" and \"total number of tokens before filtering.\" This may be making some results misleading. This ambiguity is present throughout the paper. Just one example is in 3.3 - is 80B the total before filtering or after filtering?\n    * Relatedly, in the case when it's after the % filtering, the \"x-axis\" should be total number of tokens before filtering in my opinion, because the % filtering isn't making training cheaper. I believe the results will still look good after these changes (the numbers in Table 1, for example, are great). But I think Figure 1, for example, should use total tokens (not after % filtering) if it's not already.\n* It seems like OpenWebMath is very messy. How would this method work on a clean dataset (like the small, high-quality reference dataset). Is the benefit of the method mostly in \"cleaning\" the data, or in selecting useful tokens?\n* How was the % for filtering chosen for the experiments?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3eTg,\n\nThank you for your thoughtful review and for recognizing the novelty, effectiveness, and significance of our work!\n\n### **Definition of “number of tokens”**\n\n> The greatest weakness (in my opinion) is that \"tokens\" in many cases could refer to \"number of tokens after % filtering\" and \"total number of tokens before filtering.\" This may be making some results misleading. This ambiguity is present throughout the paper. Just one example is in 3.3 - is 80B the total before filtering or after filtering?\n\n> Relatedly, in the case when it's after the % filtering, the \"x-axis\" should be total number of tokens before filtering in my opinion, because the % filtering isn't making training cheaper. I believe the results will still look good after these changes (the numbers in Table 1, for example, are great). But I think Figure 1, for example, should use total tokens (not after % filtering) if it's not already.\n\nFor a fair comparison, we default to using “number of tokens **before** filtering” when we say “tokens” throughout the paper, just as you suggested.\n\nOn this basis, when referring to the actual selected tokens used for training **after** filtering, we usually make a special note, such as in L165:\n\n> \"15 billion tokens (selecting 10.5 billion tokens)\".\n\nIt is worth noting that in Tables 1, 3, and 5, the total number of input tokens (i.e., before filtering) for all comparative experiments is the same. Therefore, to facilitate the presentation of the actual training tokens, we use “Uniq. Toks” to denote unique tokens and “Train Toks” to denote selected tokens. This is additionally clarified in the caption of Table 1.\n\nWe appreciate your suggestion to define \"number of tokens\" more clearly, and we will specifically clarify this point in the paper and table captions to make it more explicit.\n\n### **OpenWebMath**\n\n> It seems like OpenWebMath is very messy. How would this method work on a clean dataset (like the small, high-quality reference dataset). Is the benefit of the method mostly in \"cleaning\" the data, or in selecting useful tokens?\n\nAs we know, OpenWebMath is currently a relatively high-quality open-source pre-trained dataset for mathematics. It has undergone meticulous cleaning and is widely adopted by various models [1]. Nevertheless, scoring with the reference model can still identify \"messy\" and dirty data, as shown in Appendix C, particularly the noisy tokens within a single document that cannot be filtered by traditional document-level methods.\n\nSecondly, we believe that the concepts of \"cleaning\" the data and \"selecting useful tokens\" overlap. SLM achieves the effect of filtering the data by selecting useful tokens for the relevant field from the pre-trained data. Therefore, we assert that both \"cleaning\" and \"selecting\" are present in our settings.\n\nMoreover, our work focuses on pre-training. Whether the SLM method can still be effective with high-quality data, such as in SFT, remains a question for future work. We believe this is a good scenario to validate whether SLM can bring benefits solely through \"selecting”.\n\n###  **How to choose k% for filtering**\n\n> How was the % for filtering chosen for the experiments?\n\nIn the pre-experiment phase, we trained on small-scale data subsets with different percentages and observed the model's accuracy to guide our selection, as shown in **Figure 9**. We found that within a certain percentage range (e.g., from 50% to 70%), there was no significant difference in performance. Therefore, we directly chose 60% and 70% for the larger-scale experiments.\n\n---\n[1] Azerbayev, Zhangir, et al. \"Llemma: An open language model for mathematics.\" *arXiv preprint arXiv:2310.10631* (2023). https://arxiv.org/pdf/2310.10631", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lH0lVFIg2y", "reviewer": "Reviewer_pLtQ", "summary": "The paper analyzes token-level training dynamics in continued pretraining, identifying four loss patterns: persistent low loss, persistent high loss, increasing loss, and decreasing loss. Motivated by these patterns, the paper proposes a modification to language modeling called Selective Language Modeling (SLM), which only trains on a subset of the input tokens. This subset is selected by training a high-quality reference model and computing the \"excess loss\" of the target model -- i.e., the token-level difference between the target model and reference model loss. The model trained using SLM, Rho, achieves strong performance on math and other benchmarks relative to a model using normal continual pretraining.", "questions": "Q1a. The contents of each token category seem quite important to the paper’s stated motivation of removing noisy tokens from pretraining. Can you provide example sets of tokens / themes?\n\nQ1b. It's not really clear from the few examples provided in Figure 11-14 how to interpret these four loss categories. Are there specific tokens which are generally in one category regardless of the document they occur in (e.g. very rare tokens, or numbers in math equations)? \n\nQ1c. Figure 14: What conclusion should we draw from the differences in tokens selected over time? It's hard to interpret this figure. \n\nQ2. What artifacts do you plan to release? In particular, do you plan to release the model Rho? The 0.5 B and 1.9B datasets you compiled? Checkpoints trained on increasing selection percentages? \n\nQ3. I understand if this is not possible to address in the rebuttal period, but I'm curious if using this method has any impact on the downstream memorization of pretraining data.\n\nOther suggestions/line comments (no need to address in rebuttal):\n* Line 33: \"limiting LLM's potential to merely mediocre intelligence\" is a pretty meaningless phrase -- what does \"mediocre intelligence\" mean? I suggest revising to be more specific about the claim here (e.g. \"limiting the model's capabilities\"). \n* Figure 8 is hard to understand\n* Figure 11 is not colorblind-friendly\n* Line 764: typo in spelling of Tinyllama", "limitations": "Limitations listed look reasonable.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "S1. The four categories of token-level loss are interesting and (to the best of my knowledge) not a previously noted phenomenon. The authors provide an interesting analysis of this phenomenon, and use it to motivate their method.\n\nS2. The idea of selecting a subset of tokens to train on is clever and appears effective. The results, particularly on math benchmarks, after continual pretraining with SLM are quite strong and compared to sensible baselines. \n\nS3. The analysis is relatively comprehensive and contains several interesting points, especially the section comparing the correlation between token losses and downstream performance for selected/unselected tokens.", "weaknesses": "W1.  *Concerns about training time/cost*. The \"10x faster\"/\"5x faster\" claims in Figure 1 don't factor in the cost of pre-scoring each token by both the reference and training model. Can you measure and report this cost? It seems like what the figure actually shows horizontally is *data efficiency*, not speed of training. More generally, I think the claims about efficiency need to be more clearly explained-- the *data* efficiency claim is well-supported, but \"efficient\" used more broadly (e.g. lines 83, 114-115, 205) generally suggests a time or space efficiency claim, which I don't think the paper supports (or even really intends to claim). \n\nW2. *Scope of claims in title/abstract*. The title and start of the abstract suggest that the method is meant to be applied throughout pretraining, but the paper focuses on continued pretraining. Additionally, the eval focuses predominately on math datasets, which involve many tokens which may be relatively infrequent in pretraining corpora but frequent in-domain. This seems like the ideal domain for this kind of strategy--and, as Figure 5 shows, the gains are much more modest for other tasks. It seems the main finding is that \"SLM is a strong method for continual pretraining for math tasks (and slightly beneficial for general domain tasks)\", but the title/first 10 lines seem to suggest \"SLM should be used instead of CLM for pretraining from scratch,\" which isn't supported or claimed elsewhere in the paper. \n\nW3. The reference model should be included in the results tables as well. Does Rho outperform the reference model used for token selection?\n\nW4. Doing hard selection cutoffs seems a bit heavy-handed; it's possible that weighting examples according to their \"excess loss\" might lead to higher performance. The authors do mention this as a future direction in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer pLtQ,\n\nThank you for your detailed review and thoughtful feedback, especially for finding our token-level loss research novel, our SLM idea clever and effective, and our analysis comprehensive!\n\n### **W1. Data efficiency vs. Training time**\n\nThank you for your suggestion! We believe that the term \"data efficiency\" better captures the essence of the SLM method we aim to present. Therefore, in the revised version, we will highlight the advantages of the method from the perspective of data efficiency.\n\nAdditionally, it is important to clarify that we only need to perform a single forward pass for scoring, which is significantly faster than the training process. During the training stage, the loss score is automatically calculated in the forward pass of the training model, so there are no additional costs during training.\n\n### **W2. Scope of claims**\n\nThank you for your feedback. Our method is initially designed for pre-training. However, due to budget constraints, we were unable to conduct large-scale pre-training from scratch. We will leave the exploration of SLM in pre-training from scratch for future work.\n\nIt is worth noting that SLM not only achieved excellent results in continual pre-training for math but also showed significant improvements in general knowledge, such as MMLU (+11.3%), and in coding tasks, such as MBPP (p@10, +7.8%) and HumanEval (p@10, +10.6%), on TinyLlama, which has been pre-trained with 3T tokens, as shown in **Figure 5**.\n\nWe will revise the relevant claims and sections according to your comments and change the term \"pre-training\" to \"continual pre-training\" where appropriate.\n\n### **W3. Reference model performance**\n\nWe will refine the table of experimental results. The performance of the model trained by SLM is can outperform the reference model.  As the results shown in Table 3, Tinyllama-CT (RM) is the reference model used in all experiments of this table.\n\n### **W4. Improving SLM by weighting tokens**\n\nThank you for your suggestions. To guarantee the simplicity and convenience of the experiment, only the selection was carried out in this paper, and rich analysis experiments were conducted for better understanding. Just as you mentioned, we have reserved the weighting method for future work in Appendix B.\n\n### **Q1a. The contents of each token category**\n\nIn Figures 11-14, we have provided examples including four categories of tokens and several examples of selected tokens. Due to the excessive length of the examples, we have placed a small number of visualization results in the appendix for interested readers to observe the actual situation of tokens in different contexts. According to your opinion, we will increase the number of related examples in revision.\n\n### **Q1b. Token category statistic**\n\nThis sounds like an interesting statistic and we consider supplementing it in Appendix G. According to our observation, in most cases, each token will belong to different categories in different contexts, but there will be a tendency of the categories. For example, some token with word suffixes frequently appear in the L→L category.\n\n### **Q1c. Figure 14**\n\nThe intention of placing Figure 14 in the appendix was to demonstrate that, even with the same context during the training process, the scores obtained for each token at different training stages were not completely consistent. This was done to help better understand the training dynamics of SLM.\n\n### **Q2. Release plan**\n\nWe will release the pre-trained and fine-tuned Rho-1B and Rho-7B. Meanwhile, the data and code will be open-sourced after review process.\n\n### **Q3. Impact of SLM on memorization of pretraining data**\n\nThis is a very interesting question! Methods like Selective Language Modeling (SLM), which selectively include or exclude token losses during training, show great potential in addressing issues such as memorization and repetition in LLMs. And we have also discovered that recent work using similar methods has yielded promising results. [1] We believe this is an area worthy of further exploration!\n\n### **Other suggestions**\n\nFinally, thank you very much for your meticulous suggestions! They are very helpful for us in improving the paper!\n\n---\n\n[1] Hans, Abhimanyu, et al. \"Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs.\" *arXiv preprint arXiv:2406.10209* (2024). https://arxiv.org/pdf/2406.10209", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a very interesting method for filtering pre-training data for LLM training. The idea is basically first to create a golden data set for which we are certain of the quality of the tokens (no noisy tokens etc.), train a language model on that, and then use that language model to assign a score for each token in a large target pre-training corpus which we want to train the final LLM on. Then the scores per token are used to predict whether or not the loss from that token should be used in training the target language model. This ensures that we are only using the highest quality data aligned with the golden corpus in training the target LLM.

The paper is clearly written with a solid set of experiments and analysis.
Decision: Accept (oral)