Title: Not All Tokens Are What You Need for Pretraining
Venue: NeurIPS 2024
Accepted: yes
Keywords: pre-training

next token prediction

data optimization

data selection
Abstract: Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that ''Not all tokens in a corpus are equally important for language model training''. Our initial analysis examines token-level training dynamics of language model, revealing distinct loss patterns for different tokens. Leveraging these insights, we introduce a new language model called Rho-1. Unlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution. This approach involves scoring training tokens using a reference model, and then training the language model with a focused loss on tokens with higher scores. When continual continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks. After fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and 51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the pretraining tokens. Furthermore, when continual pretraining on 80B general tokens, Rho-1 achieves 6.8% average enhancement across 15 diverse tasks, increasing both data efficiency and performance of the language model pre-training.

Not All Tokens Are What You Need for Pretraining

Abstract

Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that 'Not all tokens in a corpus are equally important for language model training' . Our initial analysis examines token-level training dynamics of language model, revealing distinct loss patterns for different tokens. Leveraging these insights, we introduce a new language model called RHO-1. Unlike traditional LMs that learn to predict every next token in a corpus, RHO-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution. This approach involves scoring tokens using a reference model, and then training the language model with a focused loss on tokens with higher scores. When continual pretraining on 15B OpenWebMath corpus, RHO-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks. After fine-tuning, RHO-1-1B and 7B achieved state-of-the-art results of 40.6% and 51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the pretraining tokens. Furthermore, when continual pretraining on 80B general tokens, RHO-1 achieves 6.8% average enhancement across 15 diverse tasks, increasing both data efficiency and performance of the language model pre-training.

0

5

10

15

Tokens (B)

5

10

15

20

Math Acc (%)

16.3% better

10x faster

DeepSeekMath-1B (150B Tokens)

Rho-1-1B

Baseline

0

5

10

15

Tokens (B)

20

25

30

35

40

45

50

16.4% better

5x faster

DeepSeekMath-7B (500B Tokens)

Rho-1-7B

Baseline

The farm has 35 hens <Apr12 1:24> and 12 pigs. ##davidjl123 says totaling 47 animals.

𝑥

!

𝑥

"

𝑥

#

𝑥

$

𝑥

%

𝑥

&

𝑥

'

EOS

Undesired Tokens

Desired Tokens

✘

Remove loss

✓

Keep loss

𝑥

!

𝑥

"

𝑥

$

𝑥

%

𝑥

#

𝑥

&

𝑥

'

𝑥

(

𝑥

!

𝑥

"

𝑥

$

𝑥

%

𝑥

#

𝑥

&

𝑥

'

𝑥

(

✘

✘

✘

✓

✓

✓

✓

Noisy Pretraining Corpus

𝑥

!

𝑥

"

𝑥

#

𝑥

$

𝑥

%

𝑥

&

𝑥

'

EOS

✓

1 Introduction

Scaling up model parameters and dataset size has consistently elevated the next-token prediction accuracy in large language models, yielding significant advancements in artificial intelligence [Kaplan et al., 2020, Brown et al., 2020, OpenAI, 2023, Team et al., 2023]. However, training on all available data is not always optimal or feasible. As a result, the practice of data filtering has become crucial, using various heuristics and classifiers [Brown et al., 2020, Wenzek et al., 2019] to select training documents. These techniques significantly improve data quality and boost model performance.

However, despite thorough document-level filtering, high-quality datasets still contain many noisy tokens that can negatively affect training, as illustrated in Figure 2 (Upper). Removing such tokens might alter the text's meaning, while overly strict filtering could exclude useful data [Welbl et al., 2021, Muennighoff et al., 2024] and lead to biases [Dodge et al., 2021, Longpre et al., 2023]. Furthermore, research indicates that the distribution of web data does not inherently align with the ideal distribution for downstream applications [Tay et al., 2022, Wettig et al., 2023]. For example, common corpus at the token level may include undesirable content like hallucinations or highly ambiguous tokens that are hard to predict. Applying the same loss to all tokens can lead to inefficient computation on non-essential tokens, potentially restricting LLMs from achieving more advanced levels of intelligence.

To explore how language models learn at the token level, we initially examined training dynamics, particularly how the token-level loss evolves during usual pretraining. In §2.1, we evaluated the model's token perplexity at different checkpoints and categorized tokens into different types. Our findings reveal that significant loss reduction is limited to a select group of tokens. Many tokens are 'easy tokens' that are already learned, and some are 'hard tokens' that exhibit variable losses and resist convergence. These tokens can lead to numerous ineffective gradient updates.

Based on these analyses, we introduce RHO-1 models trained with a novel Selective Language Modeling (SLM) objective 3 . As shown in Figure 2 (Right), this approach inputs the full sequence into the model and selectively removes the loss of undesired tokens. The detailed pipeline is depicted in Figure 4: First, SLM trains a reference language model on high-quality corpora. This model establishes utility metrics to score tokens according to the desired distribution, naturally filtering out unclean and irrelevant tokens. Second, SLM uses the reference model to score each token in a corpus using its loss (§2.2). Finally, we train a language model only on those tokens that exhibit a high excess loss between the reference and the training model, selectively learning the tokens that best benefit downstream applications (§2.2).

Weshow through comprehensive experiments that SLM significantly enhances token efficiency during training and improves performance on downstream tasks. Furthermore, our findings indicate that SLM effectively identifies tokens relevant to the target distribution, resulting in improved perplexity scores

0

5

10

15

Trained T okens(B)

0

1

2

3

4

Loss

(a) Loss for different token types

H

H (11%)

L

H (12%)

H

L (26%)

L

L (51%)

0

5

10

15

Trained T okens(B)

0.0

0.1

0.2

0.3

Loss

(b) Example L

L tokens

L

L T oken 1

L

L T oken 2

L

L T oken 3

0

5

10

15

Trained T okens(B)

1.5

2.0

2.5

3.0

3.5

Loss

(c) Example H

H tokens

H

H Token 1

H

H Token 2

H

H Token 3

on benchmarks for models trained with the selected tokens. §3.2 shows the effectiveness of SLM on math continual pretraining: both 1B and 7B RHO-1 outperform CLM-trained baselines by over 16% on the GSM8k and MATH datasets. SLM reaches baseline accuracy up to 10x faster, as shown in Figure 1. Remarkably, RHO-1-7B matches the state-of-the-art performance of DeepSeekMath-7B using only 15B tokens, compared to the 500B tokens required by DeepSeekMath. Upon fine-tuning, RHO-1-1B and 7B achieve 40.6% and 51.8% on MATH, respectively. Notably, RHO-1-1B is the first 1B LM to exceed 40% accuracy, nearing the early GPT-4's CoT performance of 42.5%. §3.3 confirms the efficacy of SLM in general continual pretraining: Training Tinyllama-1B on 80B tokens with SLM improves 6.8% on average across 15 benchmarks, with gains over 10% in code and math tasks. In §3.4, we demonstrate that in settings without high-quality reference data, we can use SLM for self-referencing, leading to an average improvement of up to 3.3% in downstream tasks.

2 Selective Language Modeling

2.1 Not All Tokens Are Equal: Training Dynamics of Token Loss

Our investigation begins with a critical look at how individual tokens' losses evolve during standard pre-training. We continue pre-training Tinyllama-1B with 15B tokens from OpenWebMath, saving checkpoints after every 1B tokens. We then evaluate token-level loss at these intervals using the validation set of approximately 320,000 tokens. Figure 3(a) reveals a striking pattern: tokens fall into four categories based on their loss trajectory-persistent high loss (H → H), increasing loss (L → H), decreasing loss (H → L), and consistent low loss (L → L). For further details on these categories, see §D.1. Our analysis uncovers that a mere 26% of tokens show a notable loss reduction (H → L), while the majority (51%) remain in the L → L category, indicating they have already been learned. Interestingly, 11% of the tokens are persistently challenging (H → H), likely due to high aleatoric uncertainty [Hüllermeier and Waegeman, 2021]. Additionally, 12% of tokens experience an unexpected loss increase (L → H) during training.

Our second observation is that a significant number of token losses exhibit persistent fluctuations, and resist convergence. The loss of many L → L and H → H tokens, as depicted in Figure 3 (b) and (c), show high variance during training. In §D.2, we visualize and analyze the content of these tokens and find that many of them are noisy, which is consistent with our hypothesis.

Consequently, we learn that the loss associated with each token during training does not decrease smoothly like the overall loss; instead, there is a complex training dynamic among different tokens. If we can select the appropriate tokens for the model to focus on during training, we may be able to stabilize the trajectory of the model's training and enhance its data efficiency.

2.2 Selective Language Modeling

Overview Inspired by the practice of reference model in document-level filtering, we propose a simple pipeline of token-level data selection, termed 'Selective Language Modeling (SLM)'. Our method comprises three steps, as depicted in Figure 4. We begin by training a reference model on a curated, high-quality dataset. This model then assesses the loss of each token within the pretraining

Step 1

Train a

reference

model on

high-quality

text.

Reference

Model

High-quality

Corpus

Pretraining

Corpus

Step 2

Calculate

each token's

ppl in the

pretraining

corpus.

Language

Model

Step 3

Train an

LLM with

loss focused

on high-score

tokens.

corpus. In the final phase, we train the language model selectively, focusing on tokens with high excess loss between the training and reference model. The intuition is that tokens with high excess loss are more learnable and better aligned with the desired distribution, naturally excluding tokens that are either irrelevant or of low quality. Below, we provide a detailed description of each step.

Reference Modeling We begin by curating a high-quality dataset that reflects the desired data distribution. We train a reference model (RM) using standard cross-entropy loss on the curated data. The resulting RM is then used to assess the token loss within a larger pretraining corpus. We compute the reference loss ( L RM) of a token x i based on the probability that the RM assigns to this token. The calculation is formalized as follows:

By evaluating L RM for each token, we establish the reference loss for selective pretraining, allowing us to focus on the most influential tokens in language modeling.

Selective Pretraining Note that Causal Language Modeling (CLM) employs the cross-entropy loss:

Here, L CLM ( θ ) represents the loss function parameterized by model θ . N is the length of the sequence, x i is the i -th token in the sequence, and x <i represents all tokens before the i -th token. In contrast, Selective Language Modeling (SLM) trains the language model with a focus on tokens that exhibit a high excess loss when compared to the reference model. The excess loss ( L ∆ ) for a token x i is defined as the difference between the current training model loss ( L θ ) and the reference loss:

We introduce a token selection ratio k % , which determines the proportion of tokens to be included based on their excess loss. The cross-entropy loss for the selected tokens is computed as follows:

Here, N ∗ k % defines the number of tokens that fall within the top k % of excess loss. The indicator function I k % ( x i ) is defined as:

By default, we use L ∆ as the score function S . This ensures that the loss is applied only to the tokens that are deemed most beneficial for the language model to learn from. In practice, token selection can be implemented by ranking the tokens in a batch according to their excess loss and using only the top k % of tokens for training. This process eliminates the loss for undesired tokens without incurring additional costs during pretraining, making our approach both efficient and easily integrated.

3 Experiments

We continually pretrained models in both mathematical and general domain and designed ablation and analysis experiments to understand the effectiveness of SLM.

3.1 Experimental Setup

Reference Model Training To train our mathematical reference model, we gathered a dataset of 0.5B high-quality, math-related tokens. This dataset is a blend of synthetic data from GPT [Yu et al., 2024, Huang et al., 2024] and manually curated data [Yue et al., 2024, Ni et al., 2024]. For the general reference model, we compiled a corpus of 1.9B tokens from open-source datasets, such as Tulu-v2 [Ivison et al., 2023] and OpenHermes-2.5 [Teknium, 2023]. We trained the reference models for 3 epochs. The maximum learning rate was set at 5e-5 for 1B models and 1e-5 for 7B models, applying a cosine decay schedule. We set the maximum sequence lengths to 2048 for 1B models and 4096 for 7B models, packing multiple samples into these lengths for model input. In all main experiments, we initialized the continual pretraining model and the reference model with the same base model.

Pretraining Corpus For mathematical reasoning, we utilize the OpenWebMath (OWM) dataset [Paster et al., 2023], which comprises approximately 14B tokens sourced from math-related web pages in the Common Crawl. In the general domain, we combine the SlimPajama [Daria et al., 2023] and StarCoderData [Li et al., 2023a] (both part of the Tinyllama corpus) with OpenWebMath, training on a total of 80 billion tokens with a mix ratio of 6:3:1.

Pretraining Setting For math pretraining, we continue pretraining on the Tinyllama-1.1B model [Zhang et al., 2024] and the Mistral-7B model [Jiang et al., 2023] with learning rates of 8e-5 and 2e-5, respectively. For the 1.1B model, we conducted our training on 32 × H100 80G GPUs. This configuration allowed us to train approximately 15 billion tokens in around 3.5 hours and 50 billion tokens in about 12 hours. In the case of the 7B model, training the same 15 billion tokens took approximately 18 hours under similar hardware conditions. For general domain, we set the learning rate for Tinyllama-1.1B model to 1e-4 and train 80B tokens under the same hardware conditions, which takes approximately 19 hours. The batch size is uniformly set to 1M tokens for both domains. Regarding the token selection ratio, we use 60% for the Tinyllama-1.1B model and 70% for the Mistral-7B model.

Baseline Setting We use models that have been continually pretrained (Tinyllama-CT and MistralCT) through regular causal language modeling as baselines. Moreover, we compare RHO-1 with well-known and top-performing baselines, including Gemma [Team et al., 2024], Qwen1.5 [Bai et al., 2023], Phi-1.5 [Li et al., 2023b], DeepSeekLLM [DeepSeek-AI, 2024], DeepSeekMath [Shao et al., 2024], CodeLlama [Roziere et al., 2023], Mistral [Jiang et al., 2023], Minerva [Lewkowycz et al., 2022], Tinyllama [Zhang et al., 2024], LLemma [Azerbayev et al., 2023], and InternLM2-Math [Ying et al., 2024]. For fine-tuning results, we also compare with previous best models MAmmoTH[Yue et al., 2024] and ToRA[Gou et al., 2024].

Evaluation Setup To comprehensively evaluate pretrained models, we compare their few-shot capabilities and fine-tuning performance across a variety of tasks. We adopt the lm-eval-harness 4 [Gao et al., 2023] for general tasks, and develop math evaluation suite 5 for math tasks. We use vllm (v0.3.2) [Kwon et al., 2023] to speed up inference. Further details on our evaluation can be found in Appendix E.

3.2 Math Pre-training Results

Few-shot CoT Reasoning Results We evalute base models prompting with few-shot chain-ofthought (CoT) [Wei et al., 2022a] examples following previous works [Lewkowycz et al., 2022, Azerbayev et al., 2023, Shao et al., 2024]. As results shown in Table 1, in comparison to continue pretraining directly, RHO-1-Math has achieved the average few-shot accuracy improvement of

16.5% on 1B models and 10.4% on 7B models. Furthermore, after training for multiple epochs on OpenWebMath, we find that RHO-1 could further increase the average few-shot accuracy to 40.9%. Compared to DeepSeekMath-7B, which pretrained on 500 billion math-related tokens, RHO-1-7B pretrained on only 15 billion tokens (selecting 10.5 billion tokens) achieved comparable results, demonstrating the efficiency of our approach.

Tool-Integrated Reasoning Results Wefine-tune RHO-1 and baseline models on 69k ToRA corpus [Gou et al., 2024], consisting of 16k GPT-4-generated trajectories in a tool-integrated reasoning format, and 53k answer-augmented samples using LLaMA. As presented in Table 2, RHO-1-1B and RHO-1-7B achieved a state-of-the-art 40.6% and 51.8% on MATH dataset, respectively. On some unseen tasks ( e.g., TabMWP and GSM-Hard), RHO-1 also demonstrates a certain degree of generalizability, with an average few-shot accuracy improvement of 6.2% on the RHO-1-Math-1B and 2.7% on RHO-1-Math-7B.

MMLU

BBH

MATH

GSM8k

MBPP(p@1)

MBPP(p@10)

HumEval(p@1)

0

10

20

30

40

Metrics (%)

+11.3

+3.9

+5.0

+28.2

+6.5

+7.8

+6.9

+10.6

Performance of General Pretrained Base Model

AGIEval

ARC-C

ARC-E

BoolQ

PIQA

WinoGrande

OBQA

TydiQA

20

30

40

50

60

70

80

Metrics (%)

+1.1

+5.0

+8.6

+11.3

+0.9

+1.4

+0.2

+3.4

+8.9

Tinyllama

Tinyllama-CT

Rho-1-1B

3.3 General Pre-training Results

We confirm the efficacy of the SLM in general pretraining by continual training Tinyllama-1.1B on 80 billion tokens. The results depicted in Figure 5 indicate that although Tinyllama has already undergone extensive training on the majority of these tokens, the application of SLM yields an average enhancement of 6.8% across 15 benchmarks compared to direct continual pretraining. The improvements were especially pronounced in code and math tasks, exceeding 10%.

3.4 Self-Reference Results

In this section, we demonstrate that SLM can enhance the effectiveness of model pre-training using only pre-training corpora, without the need for additional high-quality data. Specifically, we initially trained the reference model on the OpenWebMath (OWM) corpus, a subset of Proof-Pile-2 (PPile). We evaluated OWM and PPile using the trained reference model and selected tokens for training. In this scenario, we assume the absence of downstream task-related data, a common situation in real-world applications. We hypothesize that the key factor is not scoring the desired distribution but filtering out noisy tokens. Therefore, we employed two different scoring functions based on the reference model loss, L RM, and the information entropy of the next token, H RM, which measures the uncertainty of the next token. Details are provided in Appendix H.

0

1

2

3

4

Tokens (B)

0.98

1.02

1.06

1.10

Loss

(a) Selected T oken Loss

Baseline

Rho-1

0

1

2

3

4

Tokens (B)

0.85

0.90

0.95

1.00

1.05

1.10

Loss

(b) Downstream Token Loss

Baseline

Rho-1

0

1

2

3

4

Tokens (B)

2.5

2.9

3.3

3.7

4.1

Loss

(c) Unselected T oken Loss

Baseline

Rho-1

The experimental results, as shown in Table 3, indicate that using only the OWM-trained reference model can effectively guide the model in pre-training on the same corpus, improving average downstream performance by +2.4%. Using only the information entropy as the score function brought about a similar improvement. Additionally, we considered training on the intersection of tokens selected by the two scoring functions and found better performance, with a 40% reduction in tokens and +3.3% performance. Furthermore, training the SLM on the PPile, despite only using the OWMsubset to train the reference model, still achieved a 1.8% improvement with 30% fewer tokens used. For more details, please refer to Appendix H.

3.5 Ablation Study and Analysis

Selected Token Loss Aligns Better with Downstream Performance We utilized the reference model to filter tokens and assess their impact on validation and downstream losses after training. As depicted in Figure 6, we pretrained on 4B tokens and tracked loss variations across methods and validation sets. The RHO-1 showed greater loss reduction on selected tokens than regular pretraining. Cross-referencing figures (a), (b), and (c) reveals that selected-token pretraining substantially lowers downstream loss, while traditional pretraining's effect on downstream loss is less pronounced despite initial loss reductions. Therefore, we expect that selecting tokens for pretraining is more efficient.

In Figure 7, we demonstrate that the loss of selected tokens correlates with downstream task performance, following a power law similar to recent findings [Gadre et al., 2024]. Our analysis shows that tokens selected by SLM positively impact performance, while those not selected have a negative impact. Thus, reducing loss across all tokens is not imperative for improved model performance. Refer to Appendix F for further details.

What Tokens are Selected with SLM? We aim to analyze the tokens selected by the SLM method in pretraining to further explore its working mechanism. To this end, we visualize the token selection process during the training of RHO-1 using the OpenWebMath. In §G.1, we have highlighted in blue the tokens that were retained during actual pretraining. We observe that the majority of tokens chosen

0.86

0.89

0.92

0.95

0.98

5.0

7.5

10.0

12.5

15.0

17.5

20.0

22.5

Accuracy(%)

(a) Accuracy vs. Selected T okens' Loss

Selected Tokens at 2B

Selected Tokens at 5B

Selected Tokens at 8B

Selected Tokens at 11B

Selected Tokens at 14B

3.50

3.54

3.58

3.62

3.66

5.0

7.5

10.0

12.5

15.0

17.5

20.0

22.5

Accuracy(%)

(b) Accuracy vs. Unselected T okens' Loss

Unselected Tokens at 2B

Unselected Tokens at 5B

Unselected Tokens at 8B

Unselected Tokens at 11B

Unselected Tokens at 14B

Loss

Loss

PPL of T okens Selected by Different CKPT

Selected Tokens at 2B

Selected Tokens at 5B

Selected Tokens at 8B

Selected Tokens at 11B

Selected Tokens at 14B

11

14

40

50

60

70

80

90

100

Token Select Ratio (%)

0

5

10

15

20

Accuracy (%)

Math

8

Tokens(B)

by the SLM method are closely related to mathematics, effectively training the model on the parts of the original corpus that are pertinent to mathematical content.

Furthermore, we investigated the differences in token filtering across various checkpoints during the training process and tested the perplexity of these tokens on different checkpoints. As illustrated in Figure 8, we found that the tokens selected by later checkpoints tend to have higher perplexity towards the later stages of training and lower perplexity in the earlier stages. This may suggest that the model first optimizes tokens with a larger learnable space, thereby increasing learning efficiency. Moreover, we noticed a sample-wise 'double descent' [Nakkiran et al., 2021] on the loss of selected tokens, where the select token's perplexity initially increases before decreases. This might be an effect of selecting tokens based on excess loss, targeting those most in need at each checkpoint.

Effect of Token Select Ratio We investigate the impact of token selecting ratios of the SLM. Generally, the selecting ratio is defined by heuristic rules, similar to the approach previously employed in the training of Masked Language Models (MLMs) [Devlin et al., 2019, Liu et al., 2019]. As shown in Figure 9, the selected tokens is suitable for accounting for about 60% of the original tokens.

4 Conclusion

In this paper, we propose using Selective Language Modeling(SLM) to train RHO-1, which select more suitable tokens for current pretraining stage. We conducted the detailed analysis of the loss of tokens during the pretraining process and found that not all tokens are equal during pretraining. Our

PPL

2.65

2.55

2.45

2.35

2

5

experiments and analysis in the fields of mathematics and general have demonstrated the effectiveness of the SLM method, emphasizing the importance of token level in the LLM pretraining process. In the future, how to improve pretraining of LLMs from the perspective of token level worthy of in-depth research.

Acknowledgments

Zhenghao Lin and Chen Lin were supported by National Key R&D Program of China (No. 2022ZD0160501), the Natural Science Foundation of China (No.62372390,62432011). Zhibin Gou and Yujiu Yang were supported by the Shenzhen Science and Technology Program (JCYJ20220818101001004) and the 'Graph Neural Network Project' of Ping An Technology (Shenzhen) Co., Ltd.

References

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

OpenAI. Gpt-4 technical report, 2023.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359 , 2019.

Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 2447-2469, 2021.

Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems , 36, 2024.

Jesse Dodge, Maarten Sap, Ana Marasovi´ c, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 1286-1305, 2021.

Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint arXiv:2305.13169 , 2023.

Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551 , 2022.

Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling? In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 2985-3000, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.eacl-main.217. URL https://aclanthology.org/2023.eacl-main.217 .

Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning , 110(3):457-506, 2021.

Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In ICLR , 2024.

Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-pointdriven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333 , 2024.

Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. In ICLR , 2024.

Xinzhe Ni, Yeyun Gong, Zhibin Gou, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Exploring the mystery of influential data for mathematical reasoning, 2024.

Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702 , 2023.

Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL https://huggingface.co/datasets/teknium/OpenHermes-2.5 .

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050 , 2023.

Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.

Soboleva Daria, Al-Khateeb Faisal, Myers Robert Steeves Jacob R, Hestness Joel, and Dey Nolan. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https://www.cerebras.net/blog/ slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama , 2023. URL https://huggingface.co/datasets/cerebras/SlimPajama-627B .

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Moustafa-Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! CoRR , abs/2305.06161, 2023a.

Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 , 2024.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023b.

DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 , 2024. URL https://github.com/deepseek-ai/DeepSeek-LLM .

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024.

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 , 2023.

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems , 35:3843-3857, 2022.

Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.10631 , 2023.

Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332 , 2024.

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. In ICLR , 2024.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo.org/records/10256836 .

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NIPS , volume 35, pages 24824-24837, 2022a.

Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on downstream tasks. Preprint , 2024.

Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124003, 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1) , pages 4171-4186. Association for Computational Linguistics, 2019.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR , abs/1907.11692, 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020.

Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.03393 , 2020.

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644 , 2023.

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499 , 2021.

Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning , pages 10697-10707. PMLR, 2022.

Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving llm pretraining via document de-duplication and diversification. In NIPS , volume 36, 2023.

Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems , 36, 2024a.

Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language models, 2024.

Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems , 36, 2024b.

Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems , 36, 2024.

Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help LLMs reasoning? In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=KIPJKST4gw .

Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308.12032 , 2023c.

Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In ICLR , 2024.

Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, et al. One shot learning as instruction data prospector for large language models. arXiv preprint arXiv:2312.10302 , 2023d.

Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333 , 2024.

Feiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang, Rongxing Du, Anit Kumar Sahu, and Ruoxi Jia. Get more for less: Principled data selection for warming up fine-tuning in LLMs. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/ forum?id=QmYNBVukex .

Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng, Zhaopan Xu, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, and Yang You. Infobatch: Lossless training speed up by unbiased dynamic data pruning. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=C61sk5LsK6 .

Together Computer. Redpajama: an open dataset for training large language models, 10 2023. URL https: //github.com/togethercomputer/RedPajama-Data .

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 , 2017.

Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning. Advances in neural information processing systems , 34:14488-14501, 2021.

Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint arXiv:1511.06343 , 2015.

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952 , 2015.

Haw-Shiuan Chang, Erik Learned-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advances in Neural Information Processing Systems , 30, 2017.

Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning , pages 2525-2534. PMLR, 2018.

Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton, et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.00762 , 2019.

Hwanjun Song, Minseok Kim, Sundong Kim, and Jae-Gil Lee. Carpe diem, seize the samples uncertain" at the moment" for adaptive batch selection. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 1385-1394, 2020.

Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 15630-15649. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/mindermann22a.html .

Simin Fan and Martin Jaggi. Irreducible curriculum for language model pretraining. arXiv preprint arXiv:2310.15389 , 2023.

Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt Kusner. No train no gain: Revisiting efficient training algorithms for transformer-based language models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=thbXgJ8gNK .

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv:1906.11829 , 2019.

Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12926 , 2024.

Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust language modeling. arXiv preprint arXiv:1909.02060 , 2019.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing , pages 1631-1642, 2013.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. Train no evil: Selective masking for task-guided pre-training. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6966-6974, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.566. URL https://aclanthology.org/2020.emnlp-main.566 .

Tanish Lad, Himanshu Maheshwari, Shreyas Kottukkal, and Radhika Mamidi. Using selective masking as a bridge between pre-training and fine-tuning. arXiv preprint arXiv:2211.13815 , 2022.

Qihuang Zhong, Liang Ding, Juhua Liu, Xuebo Liu, Min Zhang, Bo Du, and Dacheng Tao. Revisiting token dropping strategy in efficient BERT pretraining. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 10391-10405, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.579. URL https://aclanthology.org/2023.acl-long.579 .

Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. Token dropping for efficient BERT pretraining. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3774-3784, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.262. URL https://aclanthology.org/2022.acl-long.262 .

Tianjian Li, Haoran Xu, Philipp Koehn, Daniel Khashabi, and Kenton Murray. Error norm truncation: Robust training in the presence of data noise for text generation models. arXiv preprint arXiv:2310.00840 , 2023e.

Jessica Rumbelow and Matthew Watkins. Solidgoldmagikarp (plus, prompt generation). LessWrong, 2023. URL https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/ solidgoldmagikarp-plus-prompt-generation .

Sander Land and Max Bartolo. Fishing for magikarp: Automatically detecting under-trained tokens in large language models. arXiv preprint arXiv:2405.05417 , 2024.

Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with svcca. arXiv preprint arXiv:1811.00225 , 2018.

Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. The grammar-learning trajectories of neural language models. arXiv preprint arXiv:2109.06096 , 2021.

Leo Z Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A Smith. Probing across time: What does roberta know and when? arXiv preprint arXiv:2104.07885 , 2021.

Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 , 2022.

Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. Training trajectories of language models across scales. arXiv preprint arXiv:2212.09803 , 2022.

Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293 , 2021.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022b.

Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. arXiv preprint arXiv:2402.04177 , 2024.

Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems , 35:38274-38290, 2022.

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646 , 2022.

Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread , 2023.

Stella Biderman, USVSN PRASHANTH, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems , 36, 2024.

Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.10487 , 2022.

Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling llm under token-crisis. Advances in Neural Information Processing Systems , 36, 2024.

Charles AE Goodhart and CAE Goodhart. Problems of monetary management: the UK experience . Springer, 1984.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168 .

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NIPS , 2021.

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435 , 2022.

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2080-2094, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.168. URL https://aclanthology.org/2021.naacl-main.168 .

Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 975-984, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.acl-main.92. URL https://aclanthology.org/2020.acl-main.92 .

Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1152-1157, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/N16-1136. URL https://aclanthology.org/N16-1136 .

Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview. net/forum?id=DHyHRBwJUTN .

Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319 , 2019.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-ofthought can solve them. arXiv preprint arXiv:2210.09261 , 2022.

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364 , 2023b.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 , 2019.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 7432-7439, 2020.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789 , 2018.

Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 5673-5684, 2023.

Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information-seeking question answering in ty pologically di verse languages. Transactions of the Association for Computational Linguistics , 8:454-470, 2020.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401.14196 , 2024.

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In ACL (1) , pages 12286-12312. Association for Computational Linguistics, 2023f.

Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=jiDsk12qcz .

Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning , pages 10421-10430. PMLR, 2023.

Contents

Appendix

A Author Contributions

Zhenghao Lin designed and implemented detailed token selection process, conducted extensive preliminary experiments, developed the pre-training and evaluation pipeline, conducted most of the pre-training experiments and analysis, implemented baselines, and significantly contributed to the writing. Zhibin Gou presented a preliminary proposal, introduced the method of using excess loss for reweighting tokens, compiled high-quality corpora, trained reference models, set up the fine-tuning and evaluation pipelines, designed the experimental analysis, and significantly contributed to the writing. Yeyun Gong proposed the initial project and co-led the project with Weizhu Chen, they offered extensive advice and guidance on experiments and writing, and oversaw team collaboration and resource management. Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, and Nan Duan offered research mentorship, coordinated the project, and contributed to the writing.

B Related Works

B.1 Pretraining Data Optimization

The objective of optimizing pre-training corpora is to maximize the performance and efficiency of language model training by improving the quality and scale of the pretrain data mixture. This includes data collecting through crawling [Raffel et al., 2020] or synthesis [Polu and Sutskever, 2020, Gunasekar et al., 2023], de-duplication [Lee et al., 2021, Kandpal et al., 2022, Tirumala et al., 2023], filtering and selection [Xie et al., 2024a, Albalak et al., 2024], as well as data composition [Xie et al., 2024b] and curriculum [Chen et al., 2024, Ma et al., 2024].

B.2 Data Selection

Data selection for fine-tuning has been extensively studied, focusing on improving quality [Li et al., 2023c], diversity [Liu et al., 2024], and distribution matching [Li et al., 2023d, Xia et al., 2024, Ni et al., 2024, Kang et al., 2024]. For pretraining, various lightweight filters are utilized [Albalak et al., 2024], including heuristic-based ( e.g., language and item count filtering), classifier-based [Brown et al., 2020], and perplexity or loss-based approaches [Wenzek et al., 2019, Qin et al., 2024]. The massive public RedPajama-Data-v2 dataset [Computer, 2023], for example, leverages over 40 quality indicators for data filtering and reweighting. Nevertheless, strict filtering like blocklist [Raffel et al., 2020] and Safety API filtering [Welbl et al., 2021], have been found to hurt evaluation loss or induce bias [Dodge et al., 2021].

Sample-level selection has been extensively studied in previous research [Sener and Savarese, 2017, Killamsetty et al., 2021], particularly through online batch selection [Loshchilov and Hutter, 2015, Schaul et al., 2015, Chang et al., 2017, Katharopoulos and Fleuret, 2018, Jiang et al., 2019]. These approaches have been applied to various classification tasks [Song et al., 2020, Mindermann et al., 2022] and language modeling [Fan and Jaggi, 2023]. However, Kaddour et al. [2023] find that batch selection is not computationally efficient.

Many previous works have employed the general idea of using a reference model as a proxy for data selection. For instance, Selection Via Proxy trains a proxy model to select samples with high uncertainty [Coleman et al., 2019]. Xie et al. [2024a] and Engstrom et al. [2024] utilize n-gram models or datamodels with a target dataset to estimate importance weights. Additionally, Xie et al. [2024b] optimize the worst-case excess loss [Oren et al., 2019] relative to a reference model to determine domain weights. One of SLM's scoring functions is excess loss, and the most relevant work related to excess loss is RHO-LOSS [Mindermann et al., 2022], which trains a small model on a holdout set and uses the difference between training loss and holdout loss to select in-batch samples. Although excess loss is mathematically identical to RHO-LOSS, SLM differs in three important ways: 1) The focus is distinct. Motivated by the training dynamics of token loss, the core idea of SLM is to select useful tokens for pre-training. Its score functions are highly flexible and not limited to excess loss (see Appendix H for other functions). In contrast, RHO-LOSS aims to mathematically derive a reducible holdout loss to minimize generalization loss. 2) The meaning and training procedure of the proxy model are different. SLM trains a reference model on high-quality data to reflect the desired data distribution, whereas RHO-LOSS trains a small model on a random holdout set. 3) The selection scale and granularity vary. RHO-LOSS selects sample-level data on a small scale (typically 1K-1M samples) for task-specific fine-tuning tasks such as MNIST [LeCun et al., 1998] and SST-2 [Socher

et al., 2013]. In contrast, SLM conducts fine-grained token-level selection on large-scale language model pre-training, involving up to 80B tokens.

Token-level training strategies have also been explored, especially for the pre-training of BERT-like models using Masked Language Modeling (MLM) [Devlin et al., 2018]. Specifically, 'selective masking' involves masking important tokens in the input to focus on learning tokens that are more relevant to downstream tasks [Gu et al., 2020, Lad et al., 2022], whereas 'token dropping' aims to reduce training costs by omitting less important tokens [Zhong et al., 2023a, Hou et al., 2022]. [Li et al., 2023e] assesses the quality of each token based on the skewness of its predicted distribution and truncates the noisy tokens during training. Additionally, some research has approached the analysis and detection of under-trained tokens from a tokenization perspective [Rumbelow and Watkins, 2023, Land and Bartolo, 2024]. To our knowledge, we are the first to explore token-level data selection for large language model training, aimed at enhancing data quality and information density at the most fundamental granularity.

B.3 Language Model Training Dynamics

Investigating the training dynamics of language models is essential for understanding their behavior throughout the training process. This research includes studying internal representations [Saphra and Lopez, 2018], the acquisition of linguistic knowledge [Choshen et al., 2021, Liu et al., 2021], and the phenomenon of grokking [Power et al., 2022]. The analysis by Xia et al. [2022] is the most related to ours, which examines token-level training trajectories in models of varying sizes. Our findings, however, diverge from those of Xia et al. [2022], who posit that tokens with little change in perplexity are 'already learned'. We identify a spectrum of token patterns, including 'easy tokens' and 'hard tokens' that resist convergence. Recognizing this, we propose a method of selective language modeling that targets the influential tokens, optimizing the learning process.

B.4 Scaling Laws

Scaling laws guide us in discovering the impact of factors such as parameter count, data size, and compute on language model performance and behavior. These studies usually focus on predicable scaling though power law [Kaplan et al., 2020, Hernandez et al., 2021], optimal resource allocation [Hoffmann et al., 2022], downstream tasks [Wei et al., 2022b, Isik et al., 2024, Gadre et al., 2024], architectures [Tay et al., 2022], memorization [Tirumala et al., 2022, Carlini et al., 2022, Henighan et al., 2023, Biderman et al., 2024], and repeating data [Hernandez et al., 2022, Muennighoff et al., 2024, Xue et al., 2024]. Most scaling laws on model performance study cross-entory loss on all training tokens, while we focus on the tokens loss of desired distributions.

C Limitations and Future Work

Generalizability In math continual pretraining, as depicted in Figure 6, training exclusively with SLM leads to quickly convergence to the domain focused by the reference model, accompanied by a significant rise in the loss of unselected tokens. Although no adverse effects, like biases, have been observed from the increased loss yet, a general pretraining loss on text and code may prevent overfitting [Goodhart and Goodhart, 1984], as suggested by Ouyang et al. [2022] and Azerbayev et al. [2023]. Furthermore, future efforts could broaden the corpus scope of the reference model, and enlarge the pretraining data size, as exemplified by DeepSpeedMath [Shao et al., 2024].

Scalability Due to budget constraints, we have only verified the effectiveness of our method on smaller models (<=7B parameters) and smaller datasets (<100B tokens). Smaller models benefit significantly from removing the loss of irrelevant tokens and focusing on important ones. However, it's possible that very large models trained on extensive corpora may naturally develop this inductive bias to compress useful data ( i.e., compressing everything), although it may sounds inefficient for now. Therefore, future works should study whether this selective language modeling technique can scale to very large models and data [Kaplan et al., 2020].

Is training a reference model necessary? To score tokens, we need a high-quality reference model. This could be a base model trained with a small amount of high-quality data, or a performant open-source model. In fact, since we only need input logprobs or perplexity from reference model,





















































































































we could even utilize more powerful proprietary model APIs. We can input tokens and use the log probabilities of the input returned by the API as reference scores. We leave this for future works.

How to improve upon SLM? There are many natural extensions of SLM, e.g., reweighting tokens instead of selecting may improve robustness; using a reference model as a reward model to guide pretraining with reinforcement learning; adopting multiple reference models to reduce overfitting; designing token-level curriculum learning and iterative strategies for continuous improvements, etc .

Expanding the use of SLM SLM may be extended to supervised fine-tuning to address the noise and distribution mismatches in many SFT datasets. Another potential application is alignment, e.g., by training a reference model to emphasize helpfulness, truthfulness, and harmlessness, we may obtain a base model that is natively aligned during the pretraining stage. Meanwhile, we believe that the idea of SLM may find broader applications in multimodal data such as images, videos, and speech, which have a high noise-to-information ratio than text.

D Analysis and Visualization of Tokens in Pretraining

D.1 More Details of Four Categories Tokens

We categorize tokens into four categories: H → H, L → H, H → L, L → L. During the training process, we collected the loss of each token after training on each 1 billion tokens training data. We then used linear fitting and took the difference in loss between the first and last points as evidence of whether the loss decreased during the training process.

Specifically, suppose we have a sequence of token's loss ( l 0 , l 1 , ..., l n ) . Our goal is to minimize the sum of the squares of the differences between each data point and its linear predictive value:

where x 0 = 0 is the initial checkpoint and x n = n is the final checkpoint. Substituting these into the fitted equation, we can obtain the Loss values at the start and end after fitting: L start = b and L end = an + b . The change in loss can then be expressed as: ∆ L = L end -L start. Meanwhile, we represent the average Loss of the last checkpoint as L mean .

Next, we can classify the tokens based on ∆ L and the L mean . We categorize tokens with ∆ L < -0 . 2 as H → L (loss decreases from high to low) category tokens, and tokens with ∆ L > 0 . 2 as L → H (loss increases from low to high) category tokens. If -0 . 2 ≤ ∆ L ≤ 0 . 2 and l n ≤ L mean , then tokens are classified as L → L (loss remains low); if l n > L mean , they are classified as H → H (loss remains high). In Figure 10, we have added the tokens' loss curves of the 7B model which is consistent with the other experimental settings in §2.1, for readers to refer to whether similar phenomena exist on larger models. In Figure 11, we visualize examples of the four categories of tokens in actual text.

D.2 Non-Converging Tokens in Pretrainig

In §2.1, we mentioned that during the training process, only a minority of tokens belong to the H → L category. Among the remaining categories of H → H and L → L tokens, there are tokens that exhibit significant fluctuations during training. Furthermore, there are instances where H → L tokens are not effectively learned. Therefore, in our analysis, we specifically select those tokens from these categories that demonstrate considerable variability and distinct loss. We visualize these tokens that exhibit abnormal behavior during the training process. As illustrated in Figure 12, we find that the majority of these tokens originate from rather chaotic corpora. For instance, the corpora may include a mix of custom symbols, unintelligible gibberish, and information such as timetables and bibliographic references. Within a segment of normal text, there may also be fluctuations in the usage of common conjunctions, word suffixes, and punctuation marks. The latter may not necessarily be disastrous for training; in fact, it could represent a normal occurrence. However, if we can effectively mitigate the losses caused by the former, it might lead to more stable and efficient model training.

E Evalution Details

E.1 Math Evalution

We conducted a comprehensive evaluation of the model across various math reasoning benchmarks, encompassing a range of difficulties from elementary to university level, multiple mathematical domains, and diverse question types including multiple-choice and open-ended questions. Our benchmarks include GSM8k [Cobbe et al., 2021], MATH [Hendrycks et al., 2021], GSM-Hard [Gao et al., 2022], SV AMP [Patel et al., 2021], ASDIV [Miao et al., 2020], MAWPS [Koncel-Kedziorski et al., 2016], TabMWP (TAB) [Lu et al., 2023], MathQA (MQA) [Amini et al., 2019], MMLU-STEM [Hendrycks et al., 2020], and SAT [Azerbayev et al., 2023].

E.2 General Evalution

In the evaluation of general domain, we followed the lm-evaluation-harness [Gao et al., 2023] and evalute model on MMLU [Hendrycks et al., 2020], BBH [Suzgun et al., 2022], AGIEval [Zhong et al., 2023b], ARC-Easy and ARC-Challenge [Clark et al., 2018], BoolQ [Clark et al., 2019], PIQA [Bisk et al., 2020], Hellaswag [Zellers et al., 2019], WinoGrande [Sakaguchi et al., 2021], OpenBookQA [Mihaylov et al., 2018]. On HumanEval [Zheng et al., 2023] and TydiQA [Clark et al., 2020], we follow the evaluation pipeline of open-instrcut [Ivison et al., 2023] and report Pass@1 and Pass@10 for HumanEval and F1 for TydiQA. For MBPP [Austin et al., 2021] benchmark, we follow the evaluation pipeline of DeepSeek-Coder [Guo et al., 2024], and report Pass@1 and Pass@10.

F Relate the Selected Tokens' Loss to Downstream Task Performance

In this section, we declare the details about correlating the loss of selected tokens with the performance of downstream tasks. Concurrent study has explored similar methods to study the impact of scaling laws with the performance of models in downstream tasks [Gadre et al., 2024]. Our analysis here differs in that it aims to elucidate the relationship between the decrease/increase in loss for selected/unselected tokens and the model's performance on downstream tasks.

We use the average accuracy of MATH and GSM8K as the standard for measuring downstream tasks performance of model. Based on the trend of data points in Figure 7, we propose the relationship between the average accuracy of downstream tasks and selected/unselected tokens' loss,

The parameters a and c are fitted from the data. If the loss of selected tokens L s is used for fitting, then a > 0 . Conversely, if the loss of unselected tokens L us is used for fitting, then a < 0 . Therefore, we believe that training the model on selected tokens can effectively improve its performance on downstream tasks, while unselected tokens may have a detrimental effect on the model's performance in downstream tasks.

G Examples of Tokens Selected by SLM

G.1 Token Selected Examples

In Figure 13, we present several examples of tokens selected by the SLM method, with content marked in blue indicating the tokens actually chosen during the pretraining process.

G.2 Dynamic Token Selected

In Figure 14, we display the dynamic changes in token selection tendencies throughout the SLM training process. We chose four checkpoints during the training process (0%, 33%, 66%, and 100%) to analyze the current tendencies in token selection. The preferences for token selection are indicated by different colors, ranging from high to low preference, typically represented as deep blue, blue, black, orange, and dark orange, respectively.

H Self-Reference Setting

In this section, we will provide a detailed introduction to the reference loss score function and information entropy score function in SLM. Reference loss score function is to directly use the loss of the reference model as the basis for selecting tokens. The higher the token's loss of the reference model, the lower the expectation that the token will be selected. The score L RM ( x i ) can be directly obtained by referring to Equation 1. Information entropy score function is to select the corresponding token based on the information entropy of the reference model in each token. The information entropy of token x i can be expressed as:

where t k represents the i-th token in the vocabulary, and V represents the size of the vocabulary. The intuition of this strategy is that the higher the information entropy, the higher the uncertainty of the token in the context. Therefore, we consider that if the language model is still uncertain for certain tokens after pretraining, we do not expect that the language model will learn it during pretraining. In Table 4, we provide more SLM results, including different select ratios and combinations of two score functions, for the convenience of the readers to refer to.

I Weak-to-Strong Generalization

Apart from the main experiments where we use the same base model for the reference and continual pretraining, we also investigate if a smaller reference model can effectively guide the pretraining of a larger model. We use Tinyllama-1.1B as reference model and continual pretraining Llama-2-7B on 15B OpenWebMath tokens. Results presented in Table 5 indicate that, despite the considerable gap between the small and large models [Li et al., 2023f], employing the small reference model to token selection can still yield benefits to the pre-training of the larger model. If reference and training models have different vocabularies, one can consider performing token alignment [Wan et al., 2024, Fu et al., 2023], which we leave for future work.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and §1, we clearly demonstrate the contribution and scope of this paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Appendix C, we have thoroughly discussed the limitations of our article, hoping to guide more future work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In §2.1 and §2.2, we elaborated on the motivation and theoretical derivation of our method, with a complete proof process in place.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided detailed descriptions of the experimental setup in §3.1 and methods in §2.2 to ensure that our experiment can be reproduced.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: This may be temporary, and we are working hard to promote the process of open source.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In §3.1 and Appendix E, we clearly demonstrated various experimental settings, including hyperparameters, model settings, training settings, evaluation settings, etc.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Due to the high cost of pre-training and the significant results obtained across various settings, we do not repeat the same experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In §3.1, we have provided sufficient information on the computer resources needed to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We guarantee that the research conducted in the paper complies with NeurIPS Code of Ethics in all aspects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The purpose of this paper is to improve the training process of large language models, without any negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of the assets used in the paper, such as code, data, and models, have been appropriately recognized, and the licenses and terms of use have been clearly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Examples of Four Categories of Tokens

GMAT 1: 670 Q49 V31 \n GMAT 2: 710 Q50 V35 \n Followers: 175 \n

\n Kudos [?]: 890 [0], given: 235 \n

\n Re:

Mr. and Mrs Wiley, VIC[#permalink] 13 Feb 2010, 01:03 \n Ans A \n

\n their first child was born after J years... \n

\n thus 1 child -> j years \n

\n => thus after another J years his age = J \n

\n thus his age is J -> after 2J years and

2j after 3j years \n

\n his present age is T which is after T years. \n

\n thus total time after 2years will be T+2 \n

since after every J year they have a child after T+2 they will have \frac{(T+2)}{J} + 1 ( +1 is for the oldest) \n

\n thus A \n _________________ \n

\n Fight for your dreams :For all those who fear from Verbal- lets give it a

fight \n

\n Money Saved is the Money Earned \n

\n Jo Bole So Nihaal , Sat Shri Akaal \n

\n Gmat test review :

\n 670-to-710-a-long-journey-without-destination-still-happy-141642.html \n

\n Intern \n Joined: 06 Apr 2012 \n

Posts: 28 \n Followers: 0 \n

\n Kudos [?]: 4 [0], given: 37 \n

\n Re: Mr. and Mrs Wiley, VIC[#permalink] 21 Nov

2012, 07:46 \n jeeteshsingh wrote: \n Need the solution using Algebra.... \n

\n Mr. & Mrs Wiley have a child every

J years. Their oldest child is now T years old. If they have a child 2 years from now, how many children will they

have in total? \n

\n (A) \frac{T+2}{J} + 1 \n

\n (B) JT + 1 \n

\n (C) \frac{J}{T} + \frac{1}{T} \n

\n (D) TJ - 1 \n

\n (E) \frac{T+J}{J} \n

\n [Reveal] Spoiler: OA: \n (A) \n

\n Source: Manhattan Guide \n

\n Bunuel - would really

appreciate you providing your bit on solving the original problem above algebraically. The problem and various

explanations remain confusing. Should we think of it as a progression or some other way? Please share your take.

Thank you. \n Veritas Prep GMAT Instructor \n Joined: 16 Oct 2010 \n Posts: 4566 \n Location: Pune, India \n

Followers: 1029 \n

\n Kudos [?]: 4460 [1] , given: 162 \n

\n Re: Mr. and Mrs Wiley, VIC[#permalink] 21 Nov

2012, 09:45 \n 1 \n KUDOS \n Expert's post \n jeeteshsingh wrote: \n Need the solution using Algebra.... \n

\n Mr.

&Mrs Wiley have a child every J years. Their oldest child is now T years old. If they have a child 2 years from

now, how many children will they have in total? \n

\n (A) \frac{T+2}{J} + 1 \n

\n (B) JT + 1 \n

\n (C) \frac{J}{T}

+ \frac{1}{T} \n

\n (D) TJ - 1 \n

\n (E) \frac{T+J}{J} \n

\n [Reveal] Spoiler: OA: \n (A) \n

\n Source: Manhattan

Guide \n

\n Think of it as an Arithmetic Progression where every subsequent term (child) has a difference of J yrs

from the previous term (child). \n

\n 1st child, 2nd child, 3rd child, ....... nth child (to be born after 2 yrs) \n

\n

What is the difference between first and last terms (children)? (T + 2) yrs \n

\n What is the common difference

(age difference between two consecutive kids)? J yrs \n

\n What is the number of terms (children)? (T + 2)/J + 1

\n (Number of terms of an AP is n = (Last term - First term)/Common Difference + 1. ) \n _________________

\n

\n Karishma \n Veritas Prep | GMAT Instructor \n My Blog \n

\n Save $100 on Veritas Prep GMAT Courses

And Admissions Consulting Enroll now. Pay later. Take advantage of Veritas Prep's flexible payment plan options.

Veritas Prep Reviews Re: Mr. and Mrs Wiley, VIC [#permalink] 21 Nov 2012, 09:45 Similar topics Replies Last

post Similar Topics: 1 Mr. and Mrs. O'Leary (SC) 5 08 Jul 2012, 07:15 Mr. INVESTOR invested a total of$12,000

for a one-year 4 30 Mar 2007, 09:24 \n 2 Mr. and Mrs. Wiley have a child every J years. Their oldest 7 19 Feb

2007, 11:40 \n Mr.kevincan 6 16 Aug 2006, 12:26 \n PS: Mr. & Mrs. Smith 2 06 Dec 2005, 00:03 \n Display posts

from previous: Sort by Sciencemadness Discussion Board » Fundamentals » Reagents and Apparatus Acquisition

» Sulphuric Acid in Australia Select A Forum Fundamentals

» Chemistry in General

» Organic Chemistry

»

Reagents and Apparatus Acquisition

» Beginnings

» Responsible Practices

» Miscellaneous

» The Wiki Special

topics

» Technochemistry

» Energetic Materials

» Biochemistry

» Radiochemistry

» Computational Models

and Techniques

» Prepublication Non-chemistry

» Forum Matters

» Legal and Societal Issues \n

\n Pages: 1 2 \n

Author: Subject: Sulphuric Acid in Australia \n hissingnoise \n International Hazard \n

\n Posts: 3939 \n Registered:

26-12-2002 \n Member Is Offline \n

\n Mood: Pulverulescent! \n

\n I've stated several times on various threads,

that SO<sub>3</sub> produces a practically incondensable acid mist when led to water and, BTW, at 700°C the

decomposition rate of SO<sub>3</sub> is ˜87% . . . \n Cracking Na<sub>2</sub>S<sub>2</sub>O<sub>7</sub>

proceeds at ˜466°C and the issuing gasses are readily absorbed by conc. H<sub>2</sub>SO<sub>4</sub> to form

oleum! \n

\n Phthalic Acid \n Harmless \n

\n Posts: 19 \n Registered: 7-8-2011 \n Location: Australia \n Member

Is Offline \n

\n Mood: No Mood \n

\n That's a good idea Neil, I'll be sure to try that next time (probably for

H2O2). Just went to Tradelink and asked if they sold Moflo drain cleaner. The guy said yeah and I asked for a

liter of it. No problems whatsoever, he just said 'be careful with it'. It was $45 but a liter will last me a while and

making it myself would've been vastly more expensive I imagine. Success! MeSynth Hazard to Others Posts: 107

Registered: 29-7-2011 Member Is Offline Mood: http://www.youtube.com/watch?v=5ZltqlVuDIo Sulfuric acid

can be produced in the laboratory by burning sulfur in air and dissolving the gas produced in a hydrogen peroxide

solution. SO2 + H2O2

→

H2SO4 this was found on wikipedia... did you not look through the sullfuric acid wiki

before boiling down batery acid? anyways... There are some good videos on youtube that demonstrate how to

synthesize sulfuric acid using different methods. The drain cleaner you get from the store will be impure and may

contain organic matter that discolors the acid.

16

Examples of Tokens that Exhibit Abnormal Behavior during Training

as \n \n \begin{aligned}A \in \{\pm \begin{bmatrix}\cos\theta & - \sin\theta \\ \sin\theta & \cos\theta \\ \end{bmatrix}, \pm \begin{bmatrix}\cos\theta & \sin\theta \\ \sin\theta & - \cos\theta \\ \end{bmatrix}, \pm \begin{bmatrix}i \sinh\theta & -\cosh\theta \\ \cosh\theta & i \sinh\theta \\ \end{bmatrix}, \pm \begin{bmatrix}i \sinh\theta & \cosh\theta \\ \cosh\theta & - i \sinh\theta \\ \end{bmatrix}\}\end{aligned} \quad\quad\quad(25) \n \n I suspect this class of transformations has a name in the grand group classification scheme, but I don't know what it is. ### Mathematics Class XI \n \n Unit-I: Sets and Functions \n Chapter 1: Sets \n Unit-II: Algebra \n Chapter 5: Binomial Theorem \n Chapter 6: Sequence and Series \n Unit-III: Coordinate Geometry \n Chapter 1: Straight Lines \n Chapter 2: Conic Sections \n Unit-IV: Calculus \n Unit-V: Mathematical Reasoning \n Unit-VI: Statistics and Probability \n Chapter 1: Statistics \n Chapter 2: Probability \n \n # Graphs of Trigonometric Functions \n \n (i) Geometry in any field. Queries are case-independent. Funct* Wildcard queries are specified by * (e.g. functions, functorial, etc.). Otherwise the search is exact. 'Topological group' Phrases (multi-words) should be set in 'straight quotation marks'. au: Bourb aki & ti: Algebra Search for author and title. The and-operator & is default and can be omitted. Cheb yshev | Tschebyscheff The or-operator | allows to search for Cheb yshev or Tschebyscheff. 'Quasi* map*' py: 1989 The resulting documents have publication year 1989. so: Eur* J* Mat* Soc* cc: 14 Search for publications in a particular source with a Mathematics Subject Classification code (cc) in 14. 'Partial diff* eq*' ! elliptic The not-operator ! eliminates all results containing the word elliptic. dt: b & au: Hilbert The document type is set to books; alternatively: j for journal articles, a for book articles. py: 2000-2015 cc: (94A | 11T) Number ranges are accepted. Terms can be grouped within (parentheses). la: chinese Find documents in a given language. ISO 639-1 language codes can also be used. Code: Select all \n \n x = 64, y = 86, rule = B3/S23 \n 13bo$3bobo6bo$4b2o6b3o$4bo$54bo$54bobo$13b2o39b 2o $12b2o44b2o$3o11bo43b \n o3b2o$2bo49bo 6bo2bo$bo50b 2o6b obo$51bob o7bo$7bo49bo$7 b3o47b3o$10bo5b2o \n 42bo$9b2o4b2o 42b 2o$17bo7$13bo$3b obo6bo$4b 2o6b 3o$4bo$54bo$54b obo$13b 2o 39b2o$12b2o44b2o$3o11bo43bo3b2o$2bo49bo6bo2bo$bo50b 2o6b obo$51bobo7bo$7bo49bo$7b3o47b3o$10bo5b2o42bo$9bo5b2o42bo$9b2o6bo41b2o7$13bo$3bobo6bo$4b 2o6b3o$4bo$54bo$54bobo$13b2o39b2o$12b2o44b2o$3o11bo43bo3b2o$2bo49bo 6bo2bo$bo50b2o6bobo$51bobo7bo$7bo49bo$7b3o47b3o$10bo5b2o42bo$7b3o5b2o 40b3o$7bo9bo39bo7$13bo$3b obo6bo$4b 2o6b 3o$4bo$54bo$54b obo$13b 2o39b 2o$ \n 12b 2o44b 2o$3o11bo43bo3b 2o$2bo49bo6bo2bo$bo50b 2o6b obo$51b obo7bo$7bo49bo$7b 3o47b 3o$10bo5b 2o42bo$7b 2obo4b 2o40b 2obo$7b obo7bo39b obo! The 16-bitter thus goes down to 9 gliders. It does not reduce any further 17-bitters, though. Princess of Science, Parcly Taxel Kazyan Posts: 867 Joined: February 6th, 2014, 11:02 pm ### Re: 17 in 17: Efficient 17-bit synthesis project Good catch, Sokwe. #61 in 15G: Ground Penetrating Radar for Archaeology \n \n Workshop | December1 | 1-5 p.m. | 1012251 College (Archaeological Research Facility) \n \n Scott Byram, Research Associate, Archaeological Research Facility, UC Berkeley \n \n Archaeological Research Facility \n \n At 1pm the workshop will begin at the UC Faculty Club lawn where subsurface features are being mapped. \n \n ### Student Probability/PDE Seminar: Large Deviation Principle for random graphs II \n \n Seminar | December1 | 2:10-3:30 p.m. | 891Evans Hall \n \n Fraydoun Rezakhanlou, UC Berkeley \n \n Department of Mathematics \n \n ### BLC Fellows Forum \n \n Presentation | December1 | 3-5 p.m. | Dwinelle Hall, B-4 (Classroom side) \n \n FAll 2017 BLC Fellows, UC Berkeley \n \n Berkeley Language Center \n \n Teaching French Listening Comprehension and Cultural Awareness through Regional Variation \n Elyse Ritchey, GSR, French \n At the university level, French language instruction in the US traditionally includes a course on phonetics and pronunciation. While the major aim of such courses is to improve students' speaking and listening competence, they also emphasize speaking 'correctly' using... More > \n \n ### MENA Salon \n \n Workshop | December1 | 3-4 p.m. | 340Stephens Hall \n \n Every Friday in the semester, the CMES hosts an informal week

Figure 12: An example of an abnormal state of token perplexity during pretrainig process. The tokens highlighted in orange represent tokens that were significant abnormalities during the pretraining process.

17

Token Selected Examples

We identify two equations having the same solution with the equivalence relation: \n \n $(a,b) \sim (c,d) \mbox{ if and only if } ad = bc$ \n \n To show that this is an equivalence relation: \n \n 1. Reflexivity: $$(a,b) \sim (a,b)$$ if and only if $$ab = ba$$ which is true. Hence it is reflexive. \n 2. Symmetry: $$(a,b) \sim (c,d)$$ if and only if $$ad = bc$$ if and only if $$bc = ad$$ if and only if $$(c,d) \sim (a,b)$$. Hence it is symmetric. \n 3. Transitivity: $$(a,b) \sim (c,d)$$ and $$(c,d) \sim (e,f)$$ if and only if $$ad = bc$$ and $$cf = de$$. Multiplying these equations together, we get $$adcf = bcde$$. We can cancel $$d$$ and $$c$$ from both sides to get $$af = be$$. Hence $$(a,b) \sim (e,f)$$. \n \n Hence, we have successfully formed the set of rational numbers when we factor out the equivalence classes! \n \n $\mathbb{Q} = \frac{\mathbb{Z} \times \mathbb{Z}\backslash\{0\}}{\sim}$ \n \n Let's now take a look at what members of $$\mathbb{Q}$$ look like, say for the equation $$2x = 3$$. This equation is represented by the ordered pair

The equation of the path traversed by a projectile is called equation of trajectory. \n \n Suppose, the body reaches the point P after time ( t ) . \n \n Horizontal motion has no acceleration. Thus, using kinematic equation, horizontal distance covered will be - \n \n x = u \cos \theta t \n \n Or, \quad t = ( \frac { x }{ u \cos \theta } ) \n \n Vertical motion has constant acceleration ( g ) . Thus, distance covered will be - \n \n y = ( u \sin \theta ) t - \left ( \frac {1}{2} \right ) g tˆ2 \n \n = ( u \sin \theta ) \left ( \frac {x}{u \cos \theta} \right ) - \left ( \frac {1}{2} \right ) g \left ( \frac {x}{u \cos \theta} \right )ˆ2 \n \n = \left ( \tan \theta \right ) x - \left ( \frac {g}{2 uˆ2 \cosˆ2 \theta} \right ) xˆ2 \n \n In this equation, ( \theta, \ u \ \text {and} \ g ) are constants. Thus, \n \n 1. Term \left ( \tan \theta \right ) is a constant, let it is ( p ) \n 2. Term \left [ \left ( \frac {g}{2 uˆ2 \cosˆ2 \theta} \right ) \right ] is also a constant, let it is ( q ) \n \n So, \quad y = p x - q xˆ2 \n \n Therefore, ( y \propto xˆ2 ) , which is a required condition of a parabola. The trajectory of the projectile is a parabola. \n \n ### Time of Maximum height \n \n As the body is projected it goes up. Vertical component of velocity ( u \sin \theta ) gradually diminishes and becomes zero at the maximum height of flight. After that, body starts moving downwards. \n \n Let, ( t_m ) is the time to reach at maximum height ( h_m ) of flight. \n \n Therefore, from kinematic equation, we have - \n \n 0 = u \sin \theta - g t_m \n \n Or, \quad t_m = \left ( \frac {u \sin \theta}{g} \right ) \n \n ### Time of Flight \n \n Total time taken by the projectile between the instant it is projected and till it reaches at a point in the horizontal plane of its projection is called Time of flight. \n \n Let, the body reaches at point B on ground after time ( T_f ) of projection. Then - \n \n Net vertical displacement covered during the time of flight is zero. Using kinematic equation of motion, we get - \n \n 0 = ( u \sin \theta ) T_f - \left ( \frac {1}{2} \right ) g \ ( T_f )ˆ2 \n \n Or, \quad T_f = \left ( \frac {2 u \sin \theta}{g} \right ) = 2 \left ( \frac {u \sin \theta}{g} \right ) \n \n = 2 t_m \n \n Thus, \quad \text {Total time of flight} = \text {Time of ascent} + \text {Time of descent} \n \n = 2 \times \text {Time of maximum height.} \n \n ### Maximum height of Flight \n \n It is the maximum height reached by a projectile. It is denoted by ( h_m ) \n \n At the highest point of flight, the vertical component of velocity becomes zero. \n \n From kinematic equation of motion, we have - \n \n vˆ2 = uˆ2 + 2 a s \n \n Therefore, \quad 0ˆ2 - ( u \sin \theta )ˆ2 = 2 ( - g ) h_m \n \n Or, \quad h_m = \left ( \frac {uˆ2 \sinˆ2 \theta}{2 g} \right )

If the light moves in a purely radial direction, we can describe its path by the coordinate functions $$t(\lambda)$$ and $$r(\lambda)$$. The equation of motion $$dsˆ2 =0$$ then takes the form $$g_{tt} \left(\frac{dt}{d\lambda}\right)ˆ2 + g_{rr} \left(\frac{dr}{d\lambda}\right)ˆ2 = 0,$$ which we can rewrite as $$\left(\frac{dt}{dr}\right)ˆ2 = \frac{g_{rr}}{g_{tt}}.$$ \n \n The length of the rod is then $$L = c \int_{r_1}ˆ{r_2} \frac{dt}{dr} \text{ d}r = c \int_{r_1}ˆ{r_2} \sqrt{-\frac{g_{rr}}{g_{tt}}} \text{ d}r,$$ where I have taken the positive square root because $$r_2 > r_1$$. \n \n Notice that the length is independent of the signature of the metric, so whether you work with the (-+++) or (+-) metric is purely conventional and will not change the physics. \n \n For the Schwarzschild metric, we obtain explicitly $$L = r_2 - r_1 + r_s \ln\left(\frac{r_2 - r_s}{r_1 - r_s}\right) > r_2 - r_1.$$ \n \n Now what happens if you magically, instantaneously increase the mass of the black hole? I think the length $$L$$ of the rod stays the same (I'm here assuming that the rod is infinitely stiff), but that it would now 'appear shorter' to the distant observer - i.e. it would no longer occupy the entire space between $$r_1$$ and $$r_2$$.

Figure 13: Specific examples of selecting tokens during the selective pretraining process of the RHO-1. The tokens marked in blue represent the actual tokens trained during the training process, while the remaining black tokens are not trained during the training process.

18

After Training 0% Checkpoint

Item Type: Journal Article Copyright of this article belongs to Elsevier. Division of Mechanical Sciences > Mechanical Engineering 28 May 2007 19 Sep 2010 04:36 http://eprints.iisc.ernet.in/id/eprint/10277 # Question #8de97 \n \n Dec 10, 2016 \n \n That is not an identity. \n \n #### Explanation: \n \n Recall that \n \n ${\cot}ˆ{2} x + 1 = {\csc}ˆ{2} x$. \n \n So, we can write \n \n $\frac{1 - {\csc}ˆ{2} x}{\csc} ˆ 2 x = \frac{1 - \left({\cot}ˆ{2} x + 1\right)}{\csc} ˆ 2 x$ \n \n $= {\cot}ˆ{2} \frac{x}{\csc} ˆ 2 x$ \n \n Recall also that $\cot x = \cos \frac{x}{\sin} x$ and $\csc x = \frac{1}{\sin} x$. \n \n This allows us to continue \n \n $= \frac{{\cos}ˆ{2} \frac{x}{\sin} ˆ 2 x}{\frac{1}{\sin} ˆ 2 x}$ \n \n $= {\cos}ˆ{2} \frac{x}{\sin} ˆ 2 x \cdot {\sin}ˆ{2} \frac{x}{1}$ \n \n $= {\cos}ˆ{2} x$ \n \n Which is not identically $\cos x$. \n \n (${\cos}ˆ{2} x = \cos x$ only when $\cos x = 1$ or $0$) \n \n Dec 10, 2016 \n \n No. It is equal to ${\sin}ˆ{2} x - 1$. \n \n #### Explanation: \n \n If we have $\frac{1 - x}{x}$, we can write it as $\frac{1}{x} - \frac{x}{x}$. \n \n The same way, $\frac{1 - {\csc}ˆ{2} x}{\csc} ˆ 2$ can be written as $\frac{1}{\csc} ˆ 2 x - \frac{{\csc}ˆ{2} x}{{\csc}ˆ{2} x}$. \n \n This is equal to ${\sin}ˆ{2} x - 1$. SMS scnews item created by Hannah Bryant at Wed 25 May 2022 1227 \n Type: Seminar \n Distribution: World \n Expiry: 31 May 2022 \n Calendar1: 31 May 2022 1500-1600 \n CalLoc1: Quad S224 & via Zoom \n CalTitle1: SMRI 'What is ...a virtual knot?' Hans Boden (McMaster University) \n Auth: hannahb@w1d4n6z2.staff.sydney.edu.au (hbry8683)

After Training 33% Checkpoint

Item Type: Journal Article Copyright of this article belongs to Elsevier. Division of Mechanical Sciences > Mechanical Engineering 28 May 2007 19 Sep 2010 04:36 http://eprints.iisc.ernet.in/id/eprint/10277 # Question #8de97 \n \n Dec 10, 2016 \n \n That is not an identity. \n \n #### Explanation: \n \n Recall that \n \n ${\cot}ˆ{2} x + 1 = {\csc}ˆ{2} x$. \n \n So, we can write \n \n $\frac{1 - {\csc}ˆ{2} x}{\csc} ˆ 2 x = \frac{1 - \left({\cot}ˆ{2} x + 1\right)}{\csc} ˆ 2 x$ \n \n $= {\cot}ˆ{2} \frac{x}{\csc} ˆ 2 x$ \n \n Recall also that $\cot x = \cos \frac{x}{\sin} x$ and $\csc x = \frac{1}{\sin} x$. \n \n This allows us to continue \n \n $= \frac{{\cos}ˆ{2} \frac{x}{\sin} ˆ 2 x}{\frac{1}{\sin} ˆ 2 x}$ \n \n $= {\cos}ˆ{2} \frac{x}{\sin} ˆ 2 x \cdot {\sin}ˆ{2} \frac{x}{1}$ \n \n $= {\cos}ˆ{2} x$ \n \n Which is not identically $\cos x$. \n \n (${\cos}ˆ{2} x = \cos x$ only when $\cos x = 1$ or $0$) \n \n Dec 10, 2016 \n \n No. It is equal to ${\sin}ˆ{2} x - 1$. \n \n #### Explanation: \n \n If we have $\frac{1 - x}{x}$, we can write it as $\frac{1}{x} - \frac{x}{x}$. \n \n The same way, $\frac{1 - {\csc}ˆ{2} x}{\csc} ˆ 2$ can be written as $\frac{1}{\csc} ˆ 2 x - \frac{{\csc}ˆ{2} x}{{\csc}ˆ{2} x}$. \n \n This is equal to ${\sin}ˆ{2} x - 1$. SMS scnews item created by Hannah Bryant at Wed 25 May 2022 1227 \n Type: Seminar \n Distribution: World \n Expiry: 31 May 2022 \n Calendar1: 31 May 2022 1500-1600 \n CalLoc1: Quad S224 & via Zoom \n CalTitle1: SMRI 'What is ...a virtual knot?' Hans Boden (McMaster University) \n Auth: hannahb@w1d4n6z2.staff.sydney.edu.au (hbry8683)

After Training 66% Checkpoint

Item Type: Journal Article Copyright of this article belongs to Elsevier. Division of Mechanical Sciences > Mechanical Engineering 28 May 2007 19 Sep 2010 04:36 http://eprints.iisc.ernet.in/id/eprint/10277 # Question #8de97 \n \n Dec 10, 2016 \n \n That is not an identity. \n \n #### Explanation: \n \n Recall that \n \n ${\cot}ˆ{2} x + 1 = {\csc}ˆ{2} x$. \n \n So, we can write \n \n $\frac{1 - {\csc}ˆ{2} x}{\csc} ˆ 2 x = \frac{1 - \left({\cot}ˆ{2} x + 1\right)}{\csc} ˆ 2 x$ \n \n $= {\cot}ˆ{2} \frac{x}{\csc} ˆ 2 x$ \n \n Recall also that $\cot x = \cos \frac{x}{\sin} x$ and $\csc x = \frac{1}{\sin} x$. \n \n This allows us to continue \n \n $= \frac{{\cos}ˆ{2} \frac{x}{\sin} ˆ 2 x}{\frac{1}{\sin} ˆ 2 x}$ \n \n $= {\cos}ˆ{2} \frac{x}{\sin} ˆ 2 x \cdot {\sin}ˆ{2} \frac{x}{1}$ \n \n $= {\cos}ˆ{2} x$ \n \n Which is not identically $\cos x$. \n \n (${\cos}ˆ{2} x = \cos x$ only when $\cos x = 1$ or $0$) \n \n Dec 10, 2016 \n \n No. It is equal to ${\sin}ˆ{2} x - 1$. \n \n #### Explanation: \n \n If we have $\frac{1 - x}{x}$, we can write it as $\frac{1}{x} - \frac{x}{x}$. \n \n The same way, $\frac{1 - {\csc}ˆ{2} x}{\csc} ˆ 2$ can be written as $\frac{1}{\csc} ˆ 2 x - \frac{{\csc}ˆ{2} x}{{\csc}ˆ{2} x}$. \n \n This is equal to ${\sin}ˆ{2} x - 1$. SMS scnews item created by Hannah Bryant at Wed 25 May 2022 1227 \n Type: Seminar \n Distribution: World \n Expiry: 31 May 2022 \n Calendar1: 31 May 2022 1500-1600 \n CalLoc1: Quad S224 & via Zoom \n CalTitle1: SMRI 'What is ...a virtual knot?' Hans Boden (McMaster University) \n Auth: hannahb@w1d4n6z2.staff.sydney.edu.au (hbry8683)

After Training 100% Checkpoint

Item Type: Journal Article Copyright of this article belongs to Elsevier. Division of Mechanical Sciences > Mechanical Engineering 28 May 2007 19 Sep 2010 04:36 http://eprints.iisc.ernet.in/id/eprint/10277 # Question #8de97 \n \n Dec 10, 2016 \n \n That is not an identity. \n \n #### Explanation: \n \n Recall that \n \n ${\cot}ˆ{2} x + 1 = {\csc}ˆ{2} x$. \n \n So, we can write \n \n $\frac{1 - {\csc}ˆ{2} x}{\csc} ˆ 2 x = \frac{1 - \left({\cot}ˆ{2} x + 1\right)}{\csc} ˆ 2 x$ \n \n $= {\cot}ˆ{2} \frac{x}{\csc} ˆ 2 x$ \n \n Recall also that $\cot x = \cos \frac{x}{\sin} x$ and $\csc x = \frac{1}{\sin} x$. \n \n This allows us to continue \n \n $= \frac{{\cos}ˆ{2} \frac{x}{\sin} ˆ 2 x}{\frac{1}{\sin} ˆ 2 x}$ \n \n $= {\cos}ˆ{2} \frac{x}{\sin} ˆ 2 x \cdot {\sin}ˆ{2} \frac{x}{1}$ \n \n $= {\cos}ˆ{2} x$ \n \n Which is not identically $\cos x$. \n \n (${\cos}ˆ{2} x = \cos x$ only when $\cos x = 1$ or $0$) \n \n Dec 10, 2016 \n \n No. It is equal to ${\sin}ˆ{2} x - 1$. \n \n #### Explanation: \n \n If we have $\frac{1 - x}{x}$, we can write it as $\frac{1}{x} - \frac{x}{x}$. \n \n The same way, $\frac{1 - {\csc}ˆ{2} x}{\csc} ˆ 2$ can be written as $\frac{1}{\csc} ˆ 2 x - \frac{{\csc}ˆ{2} x}{{\csc}ˆ{2} x}$. \n \n This is equal to ${\sin}ˆ{2} x - 1$. SMS scnews item created by Hannah Bryant at Wed 25 May 2022 1227 \n Type: Seminar \n Distribution: World \n Expiry: 31 May 2022 \n Calendar1: 31 May 2022 1500-1600 \n CalLoc1: Quad S224 & via Zoom \n CalTitle1: SMRI 'What is ...a virtual knot?' Hans Boden (McMaster University) \n Auth: hannahb@w1d4n6z2.staff.sydney.edu.au (hbry8683)

19 Figure 14: An example of dynamic token selection changes during the training process , which illustrated with five different score levels represented by deep blue, light blue, black, light orange, and dark orange. The bluer the color indicates a higher tendency for the token to be selected, while the more orange the color suggests a lower tendency for the token to be selected.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes token-level training dynamics in continued pretraining, identifying four loss patterns: persistent low loss, persistent high loss, increasing loss, and decreasing loss. Motivated by these patterns, the paper proposes a modification to language modeling called Selective Language Modeling (SLM), which only trains on a subset of the input tokens. This subset is selected by training a high-quality reference model and computing the \"excess loss\" of the target model -- i.e., the token-level difference between the target model and reference model loss. The model trained using SLM, Rho, achieves strong performance on math and other benchmarks relative to a model using normal continual pretraining.\n\nSTRENGTHS:\nS1. The four categories of token-level loss are interesting and (to the best of my knowledge) not a previously noted phenomenon. The authors provide an interesting analysis of this phenomenon, and use it to motivate their method.\n\nS2. The idea of selecting a subset of tokens to train on is clever and appears effective. The results, particularly on math benchmarks, after continual pretraining with SLM are quite strong and compared to sensible baselines. \n\nS3. The analysis is relatively comprehensive and contains several interesting points, especially the section comparing the correlation between token losses and downstream performance for selected/unselected tokens.\n\nWEAKNESSES:\nW1.  *Concerns about training time/cost*. The \"10x faster\"/\"5x faster\" claims in Figure 1 don't factor in the cost of pre-scoring each token by both the reference and training model. Can you measure and report this cost? It seems like what the figure actually shows horizontally is *data efficiency*, not speed of training. More generally, I think the claims about efficiency need to be more clearly explained-- the *data* efficiency claim is well-supported, but \"efficient\" used more broadly (e.g. lines 83, 114-115, 205) generally suggests a time or space efficiency claim, which I don't think the paper supports (or even really intends to claim). \n\nW2. *Scope of claims in title/abstract*. The title and start of the abstract suggest that the method is meant to be applied throughout pretraining, but the paper focuses on continued pretraining. Additionally, the eval focuses predominately on math datasets, which involve many tokens which may be relatively infrequent in pretraining corpora but frequent in-domain. This seems like the ideal domain for this kind of strategy--and, as Figure 5 shows, the gains are much more modest for other tasks. It seems the main finding is that \"SLM is a strong method for continual pretraining for math tasks (and slightly beneficial for general domain tasks)\", but the title/first 10 lines seem to suggest \"SLM should be used instead of CLM for pretraining from scratch,\" which isn't supported or claimed elsewhere in the paper. \n\nW3. The reference model should be included in the results tables as well. Does Rho outperform the reference model used for token selection?\n\nW4. Doing hard selection cutoffs seems a bit heavy-handed; it's possible that weighting examples according to their \"excess loss\" might lead to higher performance. The authors do mention this as a future direction in the appendix.\n\nQUESTIONS:\nQ1a. The contents of each token category seem quite important to the paper’s stated motivation of removing noisy tokens from pretraining. Can you provide example sets of tokens / themes?\n\nQ1b. It's not really clear from the few examples provided in Figure 11-14 how to interpret these four loss categories. Are there specific tokens which are generally in one category regardless of the document they occur in (e.g. very rare tokens, or numbers in math equations)? \n\nQ1c. Figure 14: What conclusion should we draw from the differences in tokens selected over time? It's hard to interpret this figure. \n\nQ2. What artifacts do you plan to release? In particular, do you plan to release the model Rho? The 0.5 B and 1.9B datasets you compiled? Checkpoints trained on increasing selection percentages? \n\nQ3. I understand if this is not possible to address in the rebuttal period, but I'm curious if using this method has any impact on the downstream memorization of pretraining data.\n\nOther suggestions/line comments (no need to address in rebuttal):\n* Line 33: \"limiting LLM's potential to merely mediocre intelligence\" is a pretty meaningless phrase -- what does \"mediocre intelligence\" mean? I suggest revising to be more specific about the claim here (e.g. \"limiting the model's capabilities\"). \n* Figure 8 is hard to understand\n* Figure 11 is not colorblind-friendly\n* Line 764: typo in spelling of Tinyllama\n\nLIMITATIONS:\nLimitations listed look reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors explore how loss for specific tokens changes in continued pre-training and note that they fall into four categories (high->high, high->low, low->high, low->low) with each category having at least 10%. They run continued pre-training on tokens that are learnable and domain-useful (judged by reference model) and find that this leads to higher accuracy with less tokens used. The main results are in the math domain, but there are also a variety of other results (tool use generalization, general domain, etc.).\n\nSTRENGTHS:\nFor originality, I'm not deeply acquainted with related work, but it seems that the authors are (based on Related Work section in appendix). This works seems novel and well-contextualized with respect to related work. The experiments are of high quality and explore a few domains/problems. The paper is generally clear and easy to read. I think this work seems significant in that future research/application could use it (especially with a particularly low-quality dataset).\n\nWEAKNESSES:\n* The greatest weakness (in my opinion) is that \"tokens\" in many cases could refer to \"number of tokens after % filtering\" and \"total number of tokens before filtering.\" This may be making some results misleading. This ambiguity is present throughout the paper. Just one example is in 3.3 - is 80B the total before filtering or after filtering?\n    * Relatedly, in the case when it's after the % filtering, the \"x-axis\" should be total number of tokens before filtering in my opinion, because the % filtering isn't making training cheaper. I believe the results will still look good after these changes (the numbers in Table 1, for example, are great). But I think Figure 1, for example, should use total tokens (not after % filtering) if it's not already.\n* It seems like OpenWebMath is very messy. How would this method work on a clean dataset (like the small, high-quality reference dataset). Is the benefit of the method mostly in \"cleaning\" the data, or in selecting useful tokens?\n* How was the % for filtering chosen for the experiments?\n\nQUESTIONS:\nPlease see weaknesses above for some explicit and implicit questions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to train LLMs on the most influential tokens selectively. They suggest training a reference model on a small high-quality corpus using the standard CLM loss. They then compute the excess loss of each token in the training corpus as a difference in losses of the reference model and target model on that token. Finally, the target model is trained of the k% subset of the training corpus with the highest excess loss. The paper describes continued pre-training experiments for 1b and 7b models to demonstrate the effectiveness of this method. The experiments show improvements compared to standard continually pre-trained baselines and some open models in terms of performance on popular benchmarks and training efficiency (number of training tokens required to match the performance of open models).\n\nSTRENGTHS:\nThe Selective Language Modelling method proposed in the paper is a novel approach to pre-training LLMs. The authors' experiments demonstrate significant improvements in training efficiency which is an important problem in LLM pre-training. The paper also describes a study of LLM training dynamics which could provide useful insights to other researchers working in the field for further exploring efficient token selection strategies for LLM pre-training.\n\nWEAKNESSES:\nThe experiments in the paper are performed in the continued pre-training setting and the impact of the original pertaining performance is not discussed in the paper. It is possible that the method might not work well if the base model is undertrained.\n\nQUESTIONS:\nThe end of section 2 talks about how tokens are selected for training in practice, it says that token selection can be implemented by ranking the tokens by their excess losses and only using the top k% for training. This seems like a crucial detail to ensure that the efficiency gains translate to training wall-clock time. How can this be done while maintaining token sequencing within samples?\n\nLIMITATIONS:\nWhile the authors have discussed the limitations of their work, an additional direction for the future could be to study the use of multiple domain specialist reference models to select influential training tokens.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}