Does your data spark joy? Performance gains from domain upsampling at the end of training
Authors: Cody Blakeney
Mansheej Paul
Brett W. Larsen
Sean Owen
Jonathan Frankle
Venue: COLM (2024)
Keywords: pretraining data, data mix, data interventions
TL;DR: We show that changing your data mix to upsample domains at the end of training can lead to performance gains and provides a tool for characterizing the utility of individual datasets.

Abstract: Pretraining datasets for large language models (LLMs) have grown to trillions of tokens composed of large amounts of CommonCrawl (CC) web scrape along with smaller, domain-specific datasets. It is expensive to understand the impact of these domain-specific datasets on model capabilities as training at large FLOP scales is required to reveal significant changes to difficult and emergent benchmarks. Given the increasing cost of experimenting with pretraining data, how does one determine the optimal balance between the diversity in general web scrapes and the information density of domain specific data? In this work, we show how to leverage the smaller domain specific datasets by upsampling them relative to CC at the end of training to drive performance improvements on difficult benchmarks. This simple technique allows us to improve up to 6.90 pp on MMLU, 8.26 pp on GSM8K, and 6.17 pp on HumanEval relative to the base data mix for a 7B model trained for 1 trillion (T) tokens, thus rivaling Llama-2 (7B)—a model trained for twice as long. We experiment with ablating the duration of domain upsampling from 5% to 30% of training and find that 10% to 20% percent is optimal for navigating the tradeoff between general language modeling capabilities and targeted benchmarks. We also use domain upsampling to characterize at scale the utility of individual datasets for improving various benchmarks by removing them during this final phase of training. This tool opens up the ability to experiment with the impact of different pretraining datasets at scale, but at an order of magnitude lower cost compared to full pretraining runs.

Published as a conference paper at COLM 2024
Does your data spark joy? Performance gains from domain upsampling at the end of training
Cody Blakeney, Mansheej Paul," Brett W. Larsen; Sean Owen, & Jonathan Frankle
Databricks Mosaic Research
New York, NY 11215, USA (cody. bl akeney, mansheej paul, brett. bare hareet..arrrendddaaaiiiikk.c...
Abstract
Pretraining datasets for large language models (LLMs) have grown to tril- lions of tokens composed of large amounts of CommonCrawl (CC) web scrape along with smaller, domain-specific datasets. It is expensive to un- derstand the impact of these domain-specific datasets on model capabilities training at large FLOP scales is required to reveal significant changes to difficult and emergent benchmarks. Given the increasing cost of ex- perimenting with pretraining data, how does one determine the optimal balance between the diversity in general web scrapes and the information density of domain specific data? In this work, we show how to leverage the smaller domain specific datasets by upsampling them relative to CC at the end of training to drive performance improvements on difficult bench- marks. This simple technique allows us to improve up to 6.90 pp MMLU, 8.26 pp on GSM8K, and 6.17 pp on HumanEval relative to the base data mix for 7B model trained for 1 trillion (T) tokens, thus rivaling Llama-2 (7B) model trained for twice as long. We experiment with ablating the duration of domain upsampling from 5% to 30% of training and find that 10% 20% percent is optimal for navigating the tradeoff between general lan- guage modeling capabilities and targeted benchmarks. We also use domain upsampling to characterize at scale the utility of individual datasets for improving various benchmarks by removing them during this final phase of training. This tool opens up the ability to experiment with the impact of different pretraining datasets at scale, but at an order of magnitude lower cost compared to full pretraining runs.
1 Introduction
Pretraining datasets for large language models (LLMs), such as Dolma (Soldaini et al. 2023) have grown trillions of tokens. To accommodate such large scales, they are typically composed of two types of data sources. First, they contain large amounts of web scraped data processed from CommonCrawl (CC) dumps. These are typically hundreds of billions to trillions of tokens in size and contain a diverse distribution of information. However, because of their size, they are necessarily less information dense and are not as filtered. Second, LLM pretraining mixes contain datasets that either target certain domains or come from single high quality sources. These are much smaller (often less than hundred billion tokens tokens). They are also more carefully processed and are dense with information from domains we want LLMs to be good at; however, since their sources are limited, they are often less diverse [Computer 2023).
One of the biggest challenges to pretraining LLMs is determining the optimal strategy for mixing datasets that come from CC and smaller domain specific sources. Some previous works have opted to pretrain entirely on heavily processed CC data (Penedo let al. 2023). Others have used different heuristics to balance between CC and more domain specific datasets (Computer 2023), However, most recent language models trained scale
Related Works:
'Equal contribution.

---

Published as a conference paper at COLM 2024
disclose limited information on the contents of their pretraining data (Tourron et al. 2023 Jiang et al. 2023: 2024 Team et al. 2024). At smaller scales, there have been attempts to algorithmically optimize the data mix proportions, but these methods have not been openly validated at the scale most modern language models are trained (Xie et al. 2024). Given the sheer cost of validating data mixing strategies at this scale, there a paucity of open research on pretraining data for LLMs.
Ideally one would conduct data mix experiments at smaller scales to identify what is good data mix. However, this is often ineffective because large FLOP scales are required to reveal significant changes in difficult and emergent benchmarks. In fact, most LLMs trained at smaller scales register random accuracy on many important benchmarks such as MMLU (Wei et al. 2022). As a result, experiments smaller scales can often be misleading; the variation between different data mixes on important benchmarks is often due to noise rather then dataset quality this scale. On the other hand, is prohibitively expensive and impractical to exhaustively characterize datasets by doing multiple training runs at the scale needed to measure above random performance on these metrics.
In this work, our goal is to characterize the utility of an alternative approach to conduct pretraining data experiments at a reasonable scale. Our strategy is to modify the data mixture af the end of training after we have already trained for enough FLOPs to measure meaningful signal on difficult benchmarks. We show that this is an effective strategy for improving LLM pretraining data mixes with experiments that are order of magnitude cheaper than full training runs.
Contributions:
We begin with a baseline mix of publicly available datasets that achieves the same scaling of performance with FLOPs as the Llama-2 model family for a 7B model trained for 1 trillion tokens.
We introduce domain upsampling data intervention which upsamples domain specific datasets relative to Common Crawl at the end of training- demonstrate that it can boost challenging metrics. In particular, we observe improvements of up to 6.90 pp on MMLU, 8.26 pp on GSM8K, and 6.17 pp on HumanEval relative to the base data mix our training setup. This makes our performance comparable to Ilama-2 (7B) but at approximately half the training FLOOPs
We ablate the percentage of training that utilizes domain upsampling and show 10%-20% is optimal for navigating the tradeoff between general language modeling capabilities and targeted benchmarks.
We show how domaín upsampling can be used as a FLOP-efffcciet tool to charac- terize how individual datasets impact model capabilities. By removing a subset of math-heavy pretraining data from the datasets we upsampled at the end of training, we quantified the impact these datasets have on specific benchmarks.
2 Training Details
We studied domain upsampling on 7 bil- lion parameter models trained for 1 tril- lion tokens. This FLOP scale was chosen so that the model performed above the noise floor on key metrics like MMLU enabling us to see the effects of data in- terventions on the model.
Parameter
Value
Optimizer
LionW Chen et al. 2024
Learning Rate
0.00012
Betas
0.9, 0.95
Weight Decay Max Sequence Length
0.00012
4096
The 7B models trained for this work are decoder-only transformers using the MPT architecture in LLM Foundry (Mo- saicML et al.) 2023). To evaluate our models we use the latest version of the Eval Gauntlet vO.3. MM.oaMMM et al.
Batch Size
960
Tokenizer Positional Embedding
Tiktoken (GPT-4)
ALiBi (Press et al. 2022
Table 1: Training Hyperparameters.
2023), an evaluation framework consist-
2

---

Published as a conference paper at COLM 2024
1.0
Lama
Batwo
0.7
(E tol
0.9
HumenEvil Errar
Core Average Eror
Emor
0.6
2 (7))
0.5
CSMBK
0.7
m amao (130)
U6 07%,
Lanoo7
0.8 171 6.57
0.4
0.6
3
0.3
0.5
Uama 217081
30
10
10"
10
10th
10a4
10th
107a
FLOPS
FLOPS
FLOPS
FLOPS
Figure 1: On key benchmarks, our 7B models trained with the data mix presented in Table 2 have errors at or below the error vs. FLOP scaling line of the Llama-2 family models. We first evaluated the performance of the 7B, 13B, and 70B variants of the Llama-2 models on MMLU, GSM8K, HumanEval, and the Gauntlet v0.3 Core Average. We then performed linear regression on the log of the error these metrics vs. the log of the FLOPS used to train these models. This scaling relationship is plotted as a dashed line in the log-log plots shown above; one can observe that the models in the Llama-2 family lie close to this scaling line. For MMLU, our models (square markers) lie on the Llama-2 scaling line. For the other metrics, our models are significantly below the scaling line.
ing of 35 popular in context learning evaluation tasks used to evaluate LLM base models. The Gauntlet v0.3 aggregates scores on benchmarks across categories. is described in Appendix A
We use inverse square root learning schedule similar to (Zhai et al. 2022
Results
Here we present the experiments demonstrating the performance boost achieved by domain upsampling well as its utility in characterizing how datasets affect challenging, emergent metrics.
3.1 Baseline data mix achieves Llama-2 scaling
To construct a baseline data mix, we grouped a set of publicly-available datasets into 4 broad categories:
Large-Scale Common Crawl: Datasets derived from Common Crawl that empha- size scale. These datasets trade off thorough quality filtering in favor of curating large and diverse set of tokens.
Small-Scale Common Crawl: Datasets derived from Common Crawl with more extensive filtering but are smaller than large-scale Common Crawl.
Domain Specific data: Small datasets that target certain domains or are from individual sources and are of high quality (e.g. Wikipedia).
Code: Code data across a variety of programming languages.
We set the proportions for mixing these datasets based on a rough heuristic for the number of epochs each of these groups would be seen during the 1 trillion token training duration. Specifically, we choose 0.5 epochs for the Small-Scale Common Crawl and Domain Specific data and 1 epoch for Code. The remainder of the 1 trillion tokens are filled with Large-Scale Common Crawl. The exact proportions are in Table 2
The rationale behind choosing these proportions is as follows: we expect the Small-Scale Common Crawl and Domain Specific data to be of high quality and we wanted them to be well represented on our trillion token budget. Also, we wanted to emphasize coding ability and so we decided to sample code data at a high percentage- -initaa experiments indicated that high percentage of code around 20% boosted programming and reasoning ability without negatively impacting language abilities. We then treat the Large-Scale CC as filler tokens that increase the diversity of our dataset and allow to fill our token budget.
Importantly, since the goal of our experimental setup is to demonstrate the utility of domain upsampling at the end of training (discussed in section 3.2), we opt for choosing a reasonable
3

---

Published as a conference paper at COLM 2024
Dataset Category
Percentage
Tokens
Epochs (1T)
Large-Scale Common Crawl
34.35%
343.5B
0.148
Small-Scale Common Crawl
36.70%
367.0B
0.5
Domain Specific
7.17%
71.7B
0.5
Code
21.78%
217.8B
Table 2: Proportions for pretraining data mix in terms of the 4 dataset groups. Code data was included at twice the proportion other domain specific datasets to focus on boosting coding capabilities. Large-scale Common Crawl was used to fill the remainder of the tokens once the other proportions were chosen.
heuristic for picking initial data mix proportions without too much optimization. Table and Figure show the performance of this initial pretraining data mix for two 7B models trained for 0.5T and 1T tokens. This heuristic has indeed been validated by our empirical results; plotting eтror vs. FLOPs shows that our models lie on or below the Llama-2 scaling line the Gauntlet v0.3 Core Average, MMLU, GSM8K, and HumanEval. Interestingly, though the overall performance scaling (as measured by Gaunlet v0.3 Core Average) is very similar, our particular data choices and mixing coefficients have led slightly different tradeoffs. The model trained for 1T tokens outperforms the Llama-2 7B model trained for tokens on GSM8K and HumanEval. This indicates that our models have better mathematical and programming ability despite being trained for half the number of tokens.
We also provide a comparison to OpenLlama 7Bv2 (Geng & Liu 2023 a 7B model that provides some open details about their data mix.
Us (7B)
Llama-2
OpenLlama
Benchmark
0.5T tok
1T tok
7B (2T tok)
7Bv2 (1T tok)
MMLU (5-shot)
24.70
35.69
45.51
40.38
GSMSK (8-shot)
10.16
14.71
14.25
7.05
HumanEval (pass@1)
18.02
17.23
13.55
15.20
Gauntlet v0.3
Core Average
32.13
35.37
37.05
32.96
World Knowledge
39.29
41.77
50.94
43.79
Commonsense Reasoning
30.52
38.38
35.48
34.91
Language Understanding
61.47
61.52
65.02
61.00
Symbolic Problem Solving
14.10
16.28
22.23
19.09
Reading Comprehension Programming (HE)
29.36
37.02
35.05
23.82
18.02
17.23
13.55
15.20
Table 3; Full evaluation results for the models presented Figure 1 We note that a 7B model trained with data mix for 1T tokens outperforms Llama2-7B-a model trained for 2T tokens-on GSM8K, HumanEval, and the Commonsense Reasoning and Reading Comprehension subsets of the Gauntlet v0.3. We also compare to OpenLlama 7Bv2, similar model with a publicly available data mix trained for IT tokens. Note that HumanEval (HE) the sole component of the programming section of the Gauntlet.

---

Published as a conference paper at COLM 2024
6.7
1.0
Dana
Lombo
Ulwb
178. 110
Core Auarogga Emor
0.6
0.9
2 (710
U7
HumannFva
Unmed
0.5
0.8
170.
Lima22 (136)
0.6
20%
Lama (333)
Jame2 12001
DU
10"
10
10"
30"h
FLOPS
FLOPS
FLOPS
FLOPS
Figure 2: Domain upsampling (DU) using the proportions presented in Table 4 provides a significant performance boost on challenging metrics for no additional FLOP cost. The dashed lines represent the same scaling for the Llama-2 family of models as described in Figure [] The square markers are the performance of our 7B model trained for 1T tokens with the data mix described in Section 3.1 the diamond markers are the resulting models when domain upsampling is performed with the proportions specified in Table 4 for the final 20% or 200B tokens of training. The light blue arrow emphasizes the improvement we observe from DU: 6.90 pp on MMLU, 8.26 pp on GSM8K, 6.17 pp on HumanEval, and 3.95 PP on the Gauntlet v0.3
3.2 Domain upsampling significantly boosts performance on challenging metrics
Next, we introduce domain up- sampling during the last 20% of training for our 1T token train- ing run. For this, we start with checkpoint at 0.8T tokens of training, change the mixing pro- portions of our pretraining data mix, and continue training for the remaining 0.2T tokens. The ex- act mixing proportions of our do- main upsampled pretraining mix are in Table These percentages were chosen based on the follow- ing heuristic: we hypothesize that though the Large-Scale CC adds lot of diversity to the pretraining data mix, it is advantageous to emphasize Domain Specific data at the end of training to bias our model towards token distributions that have high care about. Thus, we remove Large-Scale Common
Dataset Category
Percentage
Tokens
Large-Scale Common Craw! Small-Scale Scale Common Craw!
0%
0
30%
60B
Domain Specific Code
35%
70B
35%
70B
Table 4: Domain upsampling (DU) is a data intervention in which datasets are removed from the data mix at the end of training in order to scale up or upsample the reamining data. We consider results for removing Large- Scale Common Crawl and scaling up the remaining datasets as specified in this table.
information density in domains Crawl from our data mix while upsampling both Domain Specific and Code subsets. We also maintain Small-Scale Common Crawl at high percentage to prevent large distribution shift in our pretraining data.
The results of this end-of-training data intervention are shown in Table and Figure Domain upsampling was incredibly effective in boosting model performance relative to the initial pretraining data mix on all challenging benchmarks. Given the large amount of code and math related data in the domain upsampled data mix, it is perhaps unsurprising that this intervention led GSMSK and HumanEval scores that are approximately 10pp higher than Llama-2 (7B) despite the model being trained for half the total number of tokens. Additionally, this did not come at a cost to general language modeling capabilities; led to an overall model performance improvement In fact, it improved world knowledge-as measured by MMLU and the Gauntlet v0.3 cebssttrelative to the base data mix, bringing closer to Llama-2 (7B) performance these metrics. There was only a small 1pp tradeoff in the Language Understanding subset.
measured by Gauntlet v0.3 Core Average.
Overall, this across the board improvement challenging benchmarks establishes the efficacy of domain upsampling as a pretraining data intervention for improving model performance. Importantly, even using simple heuristics for choosing the new data mix
5

---

Published as a conference paper at COLM 2024
Us (7B, 1T tok)
Llama-2
OpenLlama
Benchmark
No DU
20% DU
7B (2T tok)
7Bv2 (IT tok)
MMLU (5-shot)
35.69
42.59
45.51
40.38
GSM8K (8-shot)
14.71
22.97
14.25
7.05
HumanEval (pass@1)
17.23
23.40
13.55
15.20
Gauntlet v0.3
Core Average
35.37
39.32
37.05
32.96
World Knowledge
41.77
44.19
50.94
43.79
Commonsense Reasoning
38.38
42.59
35.48
34.91
Language Understanding
61.52
60.08
65.02
61.00
Symbolic Problem Solving
16.28
20.23
22.23
19.09
Reading Compred Comprehension
37.02
45.45
35.05
23.82
Programming (HE)
17.23
23.40
13.55
15.20
Table 5: Full evaluation reults for the models presented in Figure with a comparison to OpenLlama 7Bv2. Overall, our model with 20% domain upsampling outperforms Llama2 (7B) on the Gauntlet v0.3 despite being trained for fewer tokens. Our model particularly excels at GSM8K and HvmanEval but still trails Llama-2 (7B) on MMLU.
proportions has strong positive effects, leaving opportunity for further improvement with better tuned mixing proportions.
3.3 Changing the duration of domain upsampling enables us to navigate the trade-off between targeting specific domains and general purpose language models
24
24
fuerage
39
22
HumanEual
20
28
38
Ulma-2 178, tol)
16
2 tox
16
Uama (78, 27 tok)
Lama 178, 27 tol)
Us (78. tak)
14 Us (o
14
Us (78, 17 tokd
Us (78. 17 toko
100
200
200 Tokens Domain Upsampling
Tokens of Domain Upsampling
Tolonns of Domain Upsampling (8)
Tokens of Domain Upsamplin (6)
Figure Ablating the duration of Domain Upsampling (DU). Here we consider performing DU for the final 5%, 10,, 20%, and 30% (50B, 100B, 200B, and 300B tokens) of training for 7B model for total duration 1T tokens. We observe that GSM8K and HumanEval performance continue to improve with increased DU while MMLU and the Gauntlet Core Average peak 10% and 20% respectively. Looking across the metrics presented in Table 5 we conclude that (1) DU for the final 10%-20% of training provides the best trade-off for this set up and (2) the mix used for DU should not be used for the entire duration of training.
The success of domain upsampling for the last 20% of training raises the question: are the improvements from end-of-training data intervention are they from overall better data mix proportions? Phrased another way, are the data mix proportions in Table Table better than our initial data mix and would training model for 1T tokens with this data mix lead to better performance? In this section, we provide evidence that this is not the case and in fact, treating domain upsampling as an ofd-of-training data intervention helps us better tradeoff domain specific improvements and general language modeling capabilities.
To identify when in training this intervention should be applied, we ablate our previous experiment by performing domain upsampling for the last 5%, 10%, 20%, and 30% of training. The results of this experiment are shown in Figure 3 and Table Note, while the math and programming related benchmarks, such as HumanEval, GSMBK and related Gauntlet v0.3 subscores, continue to improve as we increase the fraction of training that uses domain upsampling, other benchmarks reach optimal performance at 20% less. For

---

Published as a conference paper at COLM 2024
Us (7B, 1T tok)
Benchmark
0%
5%
10%
20%
30%
MMLU (5-shot)
35.69
40.20
43.19
42.59
41.78
GSM8K (8-shot)
14.71
16.98
20.47
22.97
24.56
HumanEval (pass@1)
17.23
18.50
20.39
23.40
24.17
Gauntlet v0.3
Core Average
35.37
37.63
38.46
39.32
38.89
World Knowledge
41.77
43.52
44.72
44.19
43.71
Commonsense Reasoning
38.38
42.97
42.33
42.59
42.19
Language Understanding
61.52
61.05
60.41
60.08
60.35
Symbolic Problem Solving
16.28
18.50
19.55
20.23
20.44
Reading Comprehension
37.02
41.23
43.35
45.45
42.50
Programming (HE)
17.23
18.50
20.39
23.40
24.17
Table 6: Full evaluation results for the models presented in Figure 3
example, MMLU peaks at 10% and Gauntlet v0.3 Core Average peaks at 20%. Thus, as we increase the fraction training with domain upsampling beyond 20%, improvements math and coding benchmarks come at the cost of performance general language modeling abilities.
This apparent trade-off indicates that the domain upsampling data mix proportions are not incontrovertibly better than the initial data mix, and training with it for the full token duration would not lead to better general purpose language model. We do not rule out that P there is an alternate mix that achieves similar performance as the 20% domain upsampling experiment when trained for the full training duration. However, finding such mix is expensive to iterate on for the full training run. Thus, the strength of domain upsampling that gives us a tool to navigate this tradeoff between targeted domains and general language modeling abilities with experiments that are an order of magnitude cheaper.
3.4 Domain upsampling is a FLOP-efficient tool characterize how individual datasets impact model capabilities
Having observed that upsampling code and our domain specific datasets for small per- centage of training leads to significant improvements on difficult and emergent tasks, we explore the question: how does one attribute improvements to specific subsets of these data? Notably, as can be seen in Figure 3 and Table GSM8K scores -a task measuring duration of domain upsampling
math and reasoning abilities abilities-improvee improves monotonically is increased. We hypothesize, given the quantity of math related data in our high-quality datasets that these may be responsible for some all of this improvement. To quantify the impact of these datasets we repeat our experiment, applying domain upsampling for the last 10% of the training duration. We keep our dataset proportions identical to those in Table but remove the math related subsets. We present the results in Table
We observe that not only do the the mathematical knowledge and reasoning skills, as measured by MMLU (which contains STEM subsets) & GSMSK, not reach the same level of performance the model trained using domain upsampling that included them, but in fact performance is worse then the baseline model with no domain upsampling. Moreover, every Gauntlet v0.3 subcategory score for the domain upsampling sans-math with the exception of programming is lower than the baseline model. From this can draw the conclusion that these specific datasets are responsible for the majority of the mathematical knowledge and reasoning capabilities in both the base model and the domain upsampled variant.
With this observation we have successfully done something which generally would be considerably more expensive. That is, we have measured the impact of pretraining datasets at a scale where difficult and emergent tasks can be reliably measured, but at an order of magnitude fewer training FLOPS. We believe application of domain upsampling opens up
7

---

Published as a conference paper at COLM 2024
10% DU
Benchmark
No DU
With Math
Sans Math
MMLU (5-shot)
35.69
43.19
29.71
GSM8K (8-shot)
14.71
20.47
11.37
HumanEval (pass@1)
17.23
20.39
21.15
Gauntlet v0.3
Core Average
35.37
38.46
32.54
World Knowledge
41.77
44.72
39.08
Commonsense Reasoning
38.38
42.33
31.76
Language Understanding
61.52
60.41
59.97
Symbolic Problem Solving
16.28
19.55
16.80
Reading Comprehension
37.02
43.35
26.48
Programming
17.23
20.39
21.15
Table Removing the math-specific datasets during domain upsampling results in sig- nificantly worse performance all metrics except programming vs. performing domain upsampling with these datasets. Experiments such as this provide a significantly cheaper method to characterize datasets compared to full pretraining runs with different data mixes. Furthermore, unlike chepaer experiment with smaller models, we get signal on the effects of the datasets challenging benchmarks like MMLU, GSM8K and HumanEval.
the ability for researchers to experiment with their pretraining datasets in a tractable way as compared to full pretraining runs.
Discussion
Pretraining LLMs has become an increasingly costly and clandestine endeavor given the scale of compute required for each experiment. This problem is exacerbated by the multi- faceted decision space presented to practitioners, especially in the selection of pretraining data. Since many important model capabilities emerge with scale, trying to explore this design space at small compute budgets is often ineffective: observations made about the effects of the pretraining data mix typically do not transfer to larger models training budgets.
In this work, consider baseline data mix of publicly available datasets that achieves exceeds the scaling of the Llama-2 family of models on key benchmarks, Next, we take crucial first step towards making experimentation with pretraining datasets cheaper. We introduce domain upsampling, method that can strongly impact the performance of the model by making targeted changes the data mix the end of training. This enables us to achieve the performance of Llama-2 (7B) but with half the training budget. By varying the duration of domain upsampling, we demonstrate how to navigate the tradeoff between targeting specific domains and making general purpose language models.
Finally, we show how making changes to the data mix only during the domain upsampling period enabled us to cheaply characterize the impact of several math-focused datasets, and we see many opportunities to use this method as a general tool for studying pretraining data in FLOP-efficient manner. It also creates platform to test data interventions at scale: instead of testing possible dataset optimization algorithms at small scales and hoping they test them at the end of training to effectively measure their impact at scale. By bringing down the cost of experimentation we have made pretraining data experiments more accessible.
will generalize,
8

---

Published as a conference paper at COLM 2024
References
[1] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
[2] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint ar arii:2217.003772 2021.
[3] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et Symbolic discovery of optimization algorithms. Advances in Neural Information Processing Systems, 36, 2024.
[4] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- fational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, Asso-
MN, USA, June 2-7, 2019, Volume 7 (Long and Short Papers), pp. ciation for Computational Linguistics, 2019. doi: 10.18653/V /V/N11-1300. URL https://doi. 1t.g//...0 1ww111166vvie http/////nnn11////ee 1300
[5] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ar Xrv preprint ar 1xXx::1803.0544 2018.
[6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ar Xiv preprint ar 1en:21100.14166 2021.
[7] Together Computer. Redpajama: an open dataset for training large language models, 2023.
[8] Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, May 2023. URL https:://iithb. ott.://err////aaaaa...c/./////ar
11ama
[9] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint ar Xix:2009.00300, 2020.
[10] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. ar Xiv preprint ar arXio:2310.06825, 2023.
[11] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint or arXiv:2401.04083 2024.
[12] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics, ACI. 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 1601 1601-1611. Association for Computational Linguistics, 2017. doi: (0.18653/V1/P11-1177 URL http::t///oii.rrg///... 1tt.:/111..833/11/11111
[13] Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema chal- lenge. In Thirleenth international conference on the principles of knowledge representation and reasoning, 2012.
[14] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. ar preprint ar arXit:1889.002889 2018.

---

Published as a conference paper at COLM 2024
[15] MosaicML et al. Introducing mpt-7b: A new standard for open-source, commercially usable Iims, 2023. URL www. mosaicml. com/blog/mpt-7b. Accessed, pp. 05-05, 2023.
[16] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Compulational Linguistics, ACI. 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: (0.18655/V1/P16-1144 URL htttss/////doi org/ 10. 10.18653/v1/p16-1144
[17] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 2080-2094. Association for URL
Computational Linguistics, 2021. doi: 10.18653/ w1tp://////....11/11//////
maaac-mmain...... 168
[18] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay, The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only, arviv preprint arXiv:2306.011166 2023.
[19] Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with lin- ear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL /tRLaattgs://oppeeeeee...v www. www.m dierrmmmdddd88@PPP@0000
[20] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 2392-2392. The Association for Computational Linguistics, 2016. doi: D..1853/VVIDD1-122 URL https:///bi.orrg/00 180../18633/// 1264
[21] Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge. Trans. Assoc. Comput. Linguistics, 7:249-266, 2019. doi: 10.1162/TACL) -A L00266. URL htthhttt: org/10 1162/tacl__000266
[22] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
[23] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9): 99-106, 2021.
[24] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arviv preprint arXie:1904.09728, 2019.
[25] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Pete Walsh, Hannaneh Hajishirzi, Noah A. Smith, Luke Zettlemoyer, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arviv preprint, 2023.
10

---

Published as a conference paper at COLM 2024
[26] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga- Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabili- ties of language models. arXiv preprint ar Aiv::22006.044 04615, 2022.
[27] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa:
A question answering challenge targeting commonsense knowledge. arXiv preprint ar #rXiv:1811.00937, 2018.
[28] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupati- raju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint #rXio:2403.08225, 2024.
[29] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arviv preprint arXiv:2307.09288, 2023.
[30] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abili- ties of large language models. Transactions on Machine Learning Research, 2022. ISSN Survey Certifica-
2835-8856. URL https:::: paattps://ppenrrriee tion.
net/foorum2ddyy2kS555222
[31] Thom Wolfe, Lewis Tunstall, and Patrick von Platen. Jeopardy dataset on hugging face hub. htttps:/bnugrinnfface co http:///n.....ooo/ottttt////rrrr
2022.
[32] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc Le, Tengyu Ma, and Ádams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024.
[33] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluis Marquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 4791-4800. Association for Computational Linguistics, 2019. doi: 10.18653, V1/ P19-1472. /tt.tttss://ool.rrr/// 1ttp:/00001655331111111--1
[34] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 12104-12113, 2022.
[35] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evalu- ating foundation models. arXiv preprint ar 2rX4::2304.06364, 2023.
11

---

Published as a conference paper at COLM 2024
A
Gauntlet v0.3
The Gauntlet v0.3 is a aggregation of benchmark developed by Mosaic Research. Rather than reporting monolithic metric in which all scores are aggregated together, the indi- vidual benchmarks were grouped into six broad competencies corresponding to different capabilities we want our LLMs to have:
1. World Knowledge: Measures the model's factual knowledge across a range of subjects.
2. Commonsense Reasoning: Evaluates the model's ability to do basic reasoning tasks that require commonsense knowledge of objects, their properties, and their behaviors.
3. Language Understanding: Assesses the model's ability to understand structure and properties of language.
4. Symbolic Problem Solving:
Tests the model's ability to solve diverse range of symbolic tasks including arithmetic, logical reasoning, algorithms, and algebra.
5. Reading Comprehension: Measures model's ability to answer questions based on information in passage of text.
6. Programming: Quantifies the ability to generate code from docstring descriptions.
These divisions allow for more fine-grained comparison between models and is especially useful for understanding how datasets affect different capabilities of the model. The random baseline of each metric was subtracted out before aggregating, For example, if the metric is 4-option multiple choice questions giving a random baseline of 25% and the model achieves 30% then this would be aggregated (0.3 - 0.25) - 0.25) = 0.0667, essentially rescaling accuracy above change to be between and If the random baseline is approximately 0, then the metric is reported is. Table x list the benchmarks in each category.
12

---

Published as a conference paper at COLM 2024
Benchmark
Citation
World Knowlledge
Jeopardy (3-shot)
(Wolfe et al. 2022
MMLU (5-shot)
(endryykk et al. 2020
BIG-bench Wikidata (3-shot)
Srivastava al.
2022
ARC-easy (3-shot)
Clark et al.
2018
ARC-challenge (3-shot)
al.
2018
(3-shot)
(Joshi et al.
2017)
Commonsense Reasoning
BIG-bench Strategy QA
Srivastava et al.
2022
BIG-bench Strange Stories
(Srivastava al.
2022
COPA (0-shot)
(Roemmele et al.
2011
PIQA (10-shot)
(Bisk 2020
SIQA (3-shot)
Sap et al. 2019
Openbook QA (10-shot)
Mihaylov al.
2018
Commonsense QA (0-shot)
Talmor et al.
2018)
Language Understanding
LAMBADA
Paperno et al. 2016
HellaSwag
ZZellers et al. 20199
Winograd (3-shot)
Levesque et al. 2012
Winogrande (5-shot)
Sakaguchi al.
202
Symbolic Problem Solving
BIG-bench Elementary Math QA (1-shot)
Srivastava et al.
2022)
BIG-bench Dyck Languages (5-shot)
Srivastava al.
2022
BIG-bench Operators (3-shot)
Srivastava et al.
2022
Simple Arithmetic (with spaces, 5-shot)
MosaicML et al.
2023
Simple Arithmetic (no spaces, 5-shot)
MosaicML et al.
2023
GSMSK (8-shot)
Cobbe et al. [202
SVAMP (5-shot)
Patel et al.
202T
AGI Eval LSAT AR (5-shot)
Zhong et al.
2023)
Reading Comprehension
SQuAD (3-shot)
Rajpurkar et
2016
BoolQ
Clark et al. 2019
CoQA
Reddy et al. 2019
AGI Eval LSAT RC (5-shot)
Zhong et al.
2023
AGI Eval LSAT LR (5-shot)
Zhong et al.
2023
AGI Eval SAT En (5-shot)
Zhong et al.
2023)
Programmiing HumanEval (pass@1)
Chen et al.
2021
Table Metrics included in the Gauntlet v0.3. Evaluation metrics are 0-shot unless otherwise denoted.
13


Reviews:
[{"review_id": "FuSuxD54eZ", "reviewer": "Reviewer_nKUf", "summary": "Paper explores the domain continual pre-training for LLMs. While the concept of domain continual pre-training for LLMs is not new, authors explore real LLMs versus most of empirical papers that have only explored small language models like BERT whose generalization to the LLMs is questionable. Authors explore boosting the domain performance of a LLM by using the domain continual pre-training at the end of the pre-training. Unlike existing continual pre-training works, the continual pre-training’s domain corpus is a mix of domain and non-domain tokens, termed as \"domain upsampling\" - oversampling the domain-specific datasets relative to the web data during the final phase of pre-training.\n\nThey claim the optimal upsampling is 10-20% of total training budget based on the results on a variety of benchmarks beating open source models like LLaMA. Another interesting side result from their experiments is using a predefined mix during general pre-training which performs almost at par with LLaMA with half the training tokens.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "> The choice of datasets and mixing proportions relies on heuristics…which use more principled approach by training a much smaller proxy model though not at LLM scale.\n\nOur primary goal in this work was to get signal on challenging, emergent benchmarks (like MMLU, GSM8K, and HumanEval) which require experiments at scale to get above random results.  Current principled methods like DoReMi have only been explored at smaller scales and have not demonstrated predictive power on these benchmarks.  Performing experiments to understand how DoReMi performs at scale would be another experiment that would cost on the order of hundreds of thousands of dollars (i.e. training a model to 5e22 FLOPS).  Because of this expense, we have primarily focused on proving it is possible to see significant improvements in these benchmarks by changing the data mix during training rather than extensively exploring alternative methods that require you to do full pretraining runs.  In essence, while these principled methods are cheap to implement, validating their efficacy at scale is expensive. \n\nHaving said this, making our results publicly available is valuable for the community because it opens up opportunities for others to explore more principled directions in a way that’s relatively inexpensive.  New proposals for data mixes can be evaluated for efficacy by changing the data mix during the final part of training, and we ourselves have done several science experiments in the paper trying to understand some of the knobs in the data proportions that impact benchmark performance.  This includes the impact of removing math datasets and changing the percentage of training dedicated to domain upsampling. (We are also exploring more principled methods for future work.) The important takeaway from this paper is the new regime in which we can do experiments for pretraining language models at the end of training and still get signal on the benchmarks we care about.\n\n> Previous continual pre-training works like [2, 3] have exclusively used domain data while the authors use a mix of domain + non-domain data with 35% of domain data during this step. Is there any justification for this proportion?\n\nThis was done in order to avoid large domain shifts in the data mix.  Early on, we tried to completely remove common crawl, and this change resulted in a 17 percentage point drop in the language understanding category of the Gauntlet.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper explores an important problem which has not been explored much in the LLM literature while the results from traditional LM models might not generalize on LLMs. Experiments are generally well done and well motivated.\n- They demonstrate how domain upsampling can be used as a cost-effective tool to make the LLM learn a specialized capability.\n- The paper is well-written, with clear motivation, methodology and extensive analysis of results across different experimental settings across different domain tasks like code and math.", "reasons_to_reject": "* The choice of datasets and mixing proportions relies on heuristics. More principled approaches could potentially yield better results. Though it is understandable given that these experiments are quite expensive to perform. \n* There are no baselines in this work, like [1] which use more principled approach by training a much smaller proxy model though not at LLM scale. \n\nPlease see comments for specific points on the weakness.", "questions_to_authors": "Can authors justify why no comparison was made with [1] (cited in the paper) which uses a smaller proxy model, and hence not much of a cost unlike other methods in the space.\nPrevious continual pre-training works like [2, 3] have exclusively used domain data while the authors use  a mix of domain + non-domain data with 35% of domain data during this step. Is there any justification for this proportion? While I understand that these experiments are expensive, given the other works, it’s not clear what’s the main message from the paper. \n\n\n[1] Xie, Sang Michael, et al. \"DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining.\" Thirty-seventh Conference on Neural Information Processing Systems. 2023.\n\n[2] Xie, Yong, Karan Aggarwal, and Aitzaz Ahmad. \"Efficient continual pre-training for building domain specific large language models.\" arXiv preprint arXiv:2311.08545 (2023).\n\n[3] Wu, Chaoyi, et al. \"PMC-LLaMA: toward building open-source language models for medicine.\" Journal of the American Medical Informatics Association (2024).", "extra_scores": {}}, {"review_id": "bWapKgG0Vv", "reviewer": "Reviewer_Rb8m", "summary": "This paper investigates the impact of data recipes during the LLM pretraining stage. The study yields two key findings: (1) By upsampling domain-specific data, the authors demonstrate that their model achieves better performance on several benchmarks while using fewer training tokens compared to the Llama-2 model of the same scale. (2) Conducting domain upsampling near the end of pretraining (at 80% completion) produces promising results and offers an efficient approach to examining the relationships between specific pretraining data and downstream tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The findings of the paper…understand the significance of the conclusions.\n\nOur results show that our technique improves not only the domains related to the datasets upsampled but almost all domains measured across a broad spectrum of 30+ evaluations (list in appendix). \n\nThe fundamental challenge of pretraining data is choosing dataset proportions to get a desired balance of capabilities. This typically is an extremely large space to search over and experiments with each mix at scale are prohibitively expensive. Previous work has tried to deal with this problem by performing experiments at smaller scales and hoping the results translate; however, this approach has not been publicly shown to work for challenging emergent benchmarks.\n\nThe primary significance of our work is thus to provide an alternative to characterize the impact of data on emergent benchmarks that is less expensive than pretraining the full model, i.e. changing the data mix durning the final 10-20% of training. Our result provides the community with a new, less expensive paradigm for characterizing their data.\n\n> Comparing MPT and Llama-2…upsampling and data processing techniques.\n\nThis is in fact what we did: our domain upsampling experiments compared to a baseline model of the same implementation, etc. with and without the proposed upsampling technique (see Table 6, first two columns). The comparison to Llama-2 in the plots was simply to show that our baseline data mix was reasonable.\n\n> Acknowledgement of existing works\n\nThe difference between our work and [1] is that rather than simply training with multiple phases to the data mix, we do the experiments to compare how such a mix performs relative to the baseline and how settings like the amount of training in the domain upsampled phase affects performance. The work in [2] is focused on context-length extension which is a confounding factor when comparing experiments as well as the fact that they are performed on an annealed model. We will add a citation to both works and discussion about these differences.\n\n> It remains unclear whether the observed performance…in the pretraining process.\n\nFLAN was included in both the baseline and domain upsampled models to control for this.\n\n> It is unclear whether such a model…similar level to many other models.\n\nThis is an interesting question, and we will work on running the experiment to check if there is a significant difference finetuning using OpenOrca before and after domain upsampling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper aims to address an important but often overlooked problem: the training data mixture of Large Language Models (LLMs). \n- The paper is well-structured and clearly presented, with a strong motivation behind the research. The experiments and results are carefully designed and executed, providing valuable insights. The authors also demonstrate transparency by offering a closer look at commonly used pretraining data.\n- Positive empirical results.", "reasons_to_reject": "- Significance: The findings of the paper seem to be limited to the observation that \"the models improve on the upsampled domains,\" which is not particularly surprising. Maybe the authors can help me better understand the significance of the conclusions.\n- Scientific value: Comparing MPT and Llama-2 (which is not trained on FLAN, etc.) may not be the most appropriate approach for a controlled experiment. I suggest comparing two models of the same implementation, data processing etc., with and without the proposed upsampling and data processing techniques.\n- Acknowledgement of existing works: [1] proposed a similar approach to the one presented in the submission (section 4.2) and was available on arXiv about 4 months before the submission deadline. The authors should cite and discuss this work to provide a more comprehensive context. Additionally, [2] appears to reach a different conclusion, suggesting that upsampling certain domains can negatively impact general capabilities. It would be great if the authors can include a discussion about it.\n- It remains unclear whether the observed performance gain can be attributed solely to domain upsampling, given that the FLAN dataset, which is often used in finetuning, is included in the pretraining process. \n- Pretraining typically aims to provide a strong foundation for various capabilities that can be further refined through finetuning, rather than primarily focusing on achieving strong performance on downstream tasks. The proposed approach seems to blur the line between pretraining and finetuning by including FLAN in pretraining. It is unclear whether such a model can still benefit from instruction tuning using datasets like FLAN or OpenOrca, at a similar level to many other models. An important comparison would be to evaluate the instruction-tuned versions of the proposed method against other LLM baselines.\n\n\n[1] https://arxiv.org/abs/2312.06550 \n[2] https://arxiv.org/abs/2402.10171", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qLkAYoH8vB", "reviewer": "Reviewer_WNa7", "summary": "In this paper, the authors introduce a novel, yet simple, technique for improving the training efficiency of models by upsampling high quality data samples at the end of training. While they also investigate finding more optimal mixtures of existing pre-training corpora  (they identify this as being “one of the biggest challenges to pertaining LLMs”), one can think of their new strategy as a cheaper alternative to simply trying to find a one-size-fits-all ideal (and scaled) pre-training mixture. \n\nThe results are impressive. They show that 20% domain upsampling (the way domains are selected is still somewhat subjective, and seems in their case to be tailored towards mathematical reasoning) outperforms Llama2 at the 7B skill while being trained for 1 trillion less tokens. They are careful to point out and check several possible confounding factors. For example, checking that their improved data mixture alone doesn’t suffice to see the same improvement (it doesn’t based on ablations these performed on this).  At first, I worried that too much focus was being put on specific tasks such as MMLU and GSM8K, although the results are quite compelling on the more comprehensive Gauntlet set. \n\nIn general, I found the paper easy to follow and read. Most of my criticisms are of a minor nature, which I detail below.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The results are somewhat narrow by only comparing against Llama-2 and OpenLlama. I would like to see this motivated more and I’m left wondering if comparing again more recent models would make more sense (e.g., models with public data mixtures, such as Olmo).\n\nThe comparison to Llama-2 and OpenLlama is primarily to show that our original data mix is a reasonable baseline.  The important comparison for domain upsampling is to the identical model without the domain upsampling intervention.  This was done so that the architecture, data processing, etc. are identical to isolate the effects of the upsampling procedure.  The main motivation for including the Llama-2 comparison is at the time of submission this was the state-of-the-art open, dense model trained at different scales.\n\nWe will add comparisons to Olmo to this table for the camera ready.\n\n> Did you look at language modeling performance, as opposed to just end task performance after applying your intervention method (e.g., does this have any known impact on perplexity). I'm still expecting that somewhere something must be going wrong, and do worry about all of this being too overly tailored to MMLU and GSM8K (which are both datasets known to have serious issues).\n\nGiven that perplexity is a function of your data mix proportion, examining the impact on perplexity before and after a change in your data mix is tricky because it is unclear which data mix you should use to evaluate perplexity after the change.  In such a situation, downstream evaluations provide a consistent framework to compare model quality regardless of data mix.\n\nWe agree that data interventions could be overly tailored to specific benchmarks like MMLU and GSM8K which is why we included evaluations across a broad range of 30+ downstream evaluations contained in the Gauntlet.  We observed that the domain upsampling intervention improved almost all of these domains.\n\n> One alternative to your approach, which has become more popular recently (e.g., in the most recent version of Olmo) is incorporating instruction data into the pre-training regime.\n\nWe have incorporated instruction data by including FLAN in the pretraining data mix (it is one of the domain specific datasets).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- a new “intervention” technique for improve training efficiency and model quality through up-sampling high quality data at the end of training. Given the simplicity of the technique and the promising empirical results, I expect this to inspire other work in this area. \n\n- several interesting auxiliary results that arise from their experiments, such as new insights into new compute friendly data mixtures on C4 (ones that improves empirically over Llama-2).", "reasons_to_reject": "- the results are somewhat narrow by only comparing against llama-2 and OpenLlama. I would like to see this motivated more and I’m left wondering if comparing again more recent models would make more more sense (e.g., models with public data mixtures, such as Olmo).", "questions_to_authors": "- Did you look at language modeling performance, as opposed to just end task performance after applying your intervention method (e.g., does this have any known impact on perplexity). I'm still expecting that somewhere something must be going wrong, and do worry about all of this being too overly tailored to MMLU and GSM8K (which are both datasets known to have serious issues). \n\n- One alternative to your approach, which has become more popular recently (e.g., in the most recent version of Olmo) is incorporating instruction data into the pre-training regime. I'd be curious to hear how you would think about this in relation to your method.", "extra_scores": {}}, {"review_id": "QJHdU0K1kG", "reviewer": "Reviewer_1SzL", "summary": "The paper addresses the important research question of how to build and evaluate the unlabeled dataset (mix) for LLM pretraining at scale.\nThe authors propose a technique, upsampling data closer to the target domain towards the end of pretraining, and show that even with a smaller pretraining corpus similar or better results on common benchmarks can be achieved.\nAdditionally, with meaningful ablation experiments, they show a cheap(er) way to conduct experiments about dataset building for LLM pretraining.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> There is not enough discussion about the relation between domain upsampling and domain-specific fine-tuning. It is known that adapting an LM to the domain of the target task, even with unlabeled data, is beneficial [1]. It seems that there are similar effects at work when upsampling unlabeled data of the target domain towards the end of pretraining.\n\nThank you for pointing out this connection; we will add discussion about the relationship of our results to [1].  There are important differences: (1) Our work does domain upsampling during pretraining while [1] does domain adaptation with an already annealed model.  We think that a very interesting direction of future work is to understand if there are any advantages to doing this with a non-annealed model. (2) While their work aims at domain adaptation, our work specifically targets a general boost in model pretraining performance.\n\n> Table 4: Your model trained on 0.5T tokens outperforms your model trained on 1T tokens for HumanEval and Programming, but not the others. What could be the reason for that?\n\nWe are investigating what is going on with this scaling of HumanEval.  Our current hypothesis is that at this FLOP scale with this percentage of code, HumanEval flatlines and you need to put in many more code FLOPS in order to see more emergent code performance.  The performance boost observed with domain upsampling on the model trained for 1T tokens also fits into this hypothesis (i.e. the performance eventually increases if you add more code FLOPS).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Sound and interesting experimental results, showing both how to improve performance without additional pretraining cost (measured in FLOPs) and a new, cheaper way to experimentally assess the influence of different subsets of the pretraining data.", "reasons_to_reject": "There is not enough discussion about the relation between domain upsampling and domain-specific fine-tuning. It is known that adapting an LM to the domain of the target task, even with unlabeled data, is beneficial [1]. It seems that there are similar effects at work when upsampling unlabeled data of the target domain towards the end of pretraining.\n\n\n[1] Gururangan et al., Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks, ACL 2020 (https://aclanthology.org/2020.acl-main.740)", "questions_to_authors": "Table 4: Your model trained on 0.5T tokens outperforms your model trained on 1T tokens for HumanEval and Programming, but not the others. What could be the reason for that?", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers agree that this paper addresses an important and often overlooked problem, presenting a novel and simple technique for improving the training efficiency of LLMs with impactful results. The results are solid, and the paper is well-written with strong motivation. Therefore, we would like to accept this paper at COLM. Please follow the reviewers' comments and suggestions from both the reviews and follow-up discussions. This includes providing a more in-depth discussion of existing work and comparisons to Olmo, etc.
Decision: Accept