Metadata Conditioning Accelerates Language Model Pre-training
Authors: Tianyu Gao
Alexander Wettig
Luxi He
Yihe Dong
Sadhika Malladi
Danqi Chen
Venue: ICML 2025 poster (2025)
Keywords: language models
pre-training
metadata conditioning
acceleration
URLs
Primary area: deep_learning->large_language_models
TL;DR: We propose to prepend metadata information (such as URLs) to pre-training documents. Our method, metadata conditioning then cooldown (MeCo), significantly accelerates pre-training.

Abstract: The vast diversity of styles, domains, and quality levels present in language model pre-training corpora is essential in developing general model capabilities, but efficiently learning and deploying the correct behaviors exemplified in each of these heterogeneous data sources is challenging. To address this, we propose a new method, termed Metadata Conditioning then Cooldown (MeCo), to incorporate additional learning cues during pre-training. MeCo first provides metadata (e.g., URLs like en.wikipedia.org) alongside the text during training and later uses a cooldown phase with only the standard text, thereby enabling the model to function normally even without metadata. MeCo significantly accelerates pre-training across different model scales (600M to 8B parameters) and training sources (C4, RefinedWeb, and DCLM). For instance, a 1.6B language model trained with MeCo matches the downstream task performance of standard pre-training while using 33% less data. Additionally, MeCo enables us to steer language models by conditioning the inference prompt on either real or fabricated metadata that encodes the desired properties of the output: for example, prepending wikipedia.org to reduce harmful generations or factquizmaster.com (fabricated) to improve common knowledge task performance. We also demonstrate that MeCo is compatible with different types of metadata, such as model-generated topics. MeCo is remarkably simple, adds no computational overhead, and demonstrates promise in producing more capable and steerable language models.

Metadata Conditioning Accelerates Language Model Pre-training
Tianyu Gao Alexander Wettig' Luxi He
Yihe Dong'
Sadhika Malladi
Danqi Chen'
Abstract
1. Introduction
Language models (LMs) achieve remarkable general- purpose capabilities by training on vast web-soureed cor- pora. For instance, Internet documents about Apple CEO Tim Cook range from memes ("Tim doesn's I cook anymore") to biographies ("Tim Cook is the CEO of Apple"). Treating data from these heterogeneous sources identically causes two issues: (1) it overlooks crucial contextual signals that aid comprehension, and (2) can impede models from reliably surfacing appropriate behaviors (e.g., humor or fac- tuality) for downstream tasks.
The vast diversity of styles, domains, and qual- ity levels present in language model pre-training corpora is essential in developing general model capabilities, but efficiently learning and deploy- ing the correct behaviors exemplified in each of these heterogeneous data sources is challeng- ing. To address this, propose new method, termed Metadata Conditioning then Cooldown (MeCo), to incorporate additional learning cues during pre-training. MeCo first provides meta- data (c.g., URLs like en. wikipedia org) alongside the text during training, then transi- tions to a cooldown phase using only standard eexb-enabling the model to perform well without metadata. MeCo significantly acceler- ates pre-training across different model scales (600M to 8B parameters) and training corpora (C4, RefinedWeb, and DCLM). Notably, a 1.6B language model trained with MeCo matches the downstream task performance of standard pre- training while using 33% less data. Addition- ally, MeCo allows us to steer language models by conditioning the inference prompt either real or fabricated metadata that encodes the de- sired output properties for example, prepend- ing wik wikipedia org to reduce harmful gener- ations or factqui zmaster. com (fabricated) to improve performance on common knowledge tasks. We further demonstrate that MeCo is com- patible with various types of metadata, such as model-generated topics. MeCo is remarkably sim- ple, adds no computational overhead, and shows promise for producing more capable and steer- able language models. Our models, data, and code available https:/ //tp//gggttt/t//////eeeeht com/ siinceton-p11 /MMo.
To provide additional information about each document's source, we propose conditioning documents with their cor- responding metadata during pre-training by prepending the widely available source URLs to each document. For in- stance, as shown in Figure adding the source URLs to Tim Cook documents helps the model distinguish among meme, biography, an interview article, and financial report. To ensure the model operates effectively with with- out metadata during inference, we implement a cooldown phase for the final 10% of training, during which we train on standard data without metadata. We call this pre-training method Metadata Conditioning then Cooldown (MeCo).
Metadata conditioning has been investigated in various contexts, such steering model generations (Keskar et al., 2019), improving model robustness against malicious prompts (Korbak et al., 2023a), and enhancing knowledge memorization in synthetic settings (Allen-Zhu & Li, 2024). Distinct from prior explorations, our work establishes the general-purposs utility of this method in two crucial ways. First, we demonstrate that this paradigm can directly ac- celerate realistic language model pre-training and improve downstream performance. Second, the cooldown phase in MeCo ensures the model can perform inference without metadata-a a key advantage over previous methods. We outline the contributions this work below.
1.
MeCo substantially accelerates pre-training ($ 3). We demonstrate that MeCo enables a 1.6B model to achieve the same average downstream performance as a standard pre-trained model using 33% less training data. MeCo exhibits consistent gains across model scales (600M, 1.6B, 3B, and 8B) and pre-training cor-
'Princcton Language and Intelligence, Princeton University. Correspondence to: Tianyu Gao <tianyug @ princeton.edu>..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Metadata Conditioning Accelerates Language Model Pre-training
Standard Pre-training
Metadata Conditioning then Cooldown (MeCo)
Standard
- MeCo
Train on raw documents directly
First 90% prepend metadata (URLs) to documents
10-Task Avg.
58
Tim doesn't Cook any more.
URL: reddit.com
doesn't Cook any more.
Tim Cook is the CEO of Apple.
URL: wikipediaaoog
Tim Cook is the CEO of Apple.
Accuracy (%)
Tim travels with his iPad.
URL: wsj.com
Tim travels with his iPad.
33% less
Apple Q3 did not look good.
URL: bloomberg.com
Apple Q3 did not look good.
Huang's not worth at $122.48.
Last 10%: standard pre pre-training as "cooldown"
48
Nvidia skyrocketing because of Al.
Nvidia skyrocketing because Al.
80
160
240
#Training Tokens (B)
Figure I: A comparison between data used by standard pre-training and MeCo. The figure on the right demonstrates 5-shot downstream task performance averaged across 10 tasks (1.6B models; details in $3).
pora (C4, RefinedWeb, and DCLM).
Gates, then c is en www.wwdiiiii..... org (please refer to $5.2 for ablations other URL variants). This URL information is readily available in many pre-training corpora, since most of them are derived from CommonCrawl', an open repository of web-crawled data.
//en wikipedia . otg/
2. MeCo unlocks a new way to steer language mod- els ($ 4). Prepending appropriate real or fabri- cated URLs the prompt during inference can in- duce desired model behaviors. For example, using fact amaster.st...... com (fabricated) can enhance performance on common knowledge tasks (e.g.. a 6% absolute improvement on zero-shot commonsense question answering), and using wiki wikipedia. org re- duces the likelihood of toxic generations several-fold compared the standard unconditional inference.
Our method consists of two training stages (Figure 1):
1. Pre-training with metadata conditioning (first 90%): The model trained on a concatenation of the meta- data and the document, following this template: URL: wi k kiikiii org'nn' [document]. When using other types of metadata, URL should be replaced with the corresponding metadata name. We only calcu- late the cross entropy loss over the document tokens, disregarding those from the template or the metadata, as we found in our preliminary experiments that train- ing on tokens from the template or the metadata slightly hurts downstream performance.
3. We ablate the design choices for MeCo ($5.1) and demonstrate that MeCo is compatible with various types of metadata ($ 5.2). Ablations using hashed URLs and model-generated topics show that metadata primarily serves to group documents by source. As such, MeCo can effectively incorporate different types of metadata, including more fine-grained ones, even when URLs are not available.
2. Cooldown with standard data (last 10%): Models trained solely on metadata-auggment
data exhibit degraded performance when used without metadata (please refer to results in Table 4). To ensure general usage, we train the model on standard pre-training doc- uments without any metadata during a cooldown stage, which covers the final 10% of steps in the pre-training process. The cooldown stage inherits the learning rate schedule and optimizer states from the metadata con- ditioning stage-i.e., it initializes the learning rate, model parameters, and optimizer states from the last checkpoint of the previous stage and continues adjust- ing the learning rate according to the schedule during the cooldown stage. Please refer to $ A.3 for more details.
Our findings demonstrate that MeCo can significantly im- prove the data efficiency of language models while adding minimal computational overhead and complexity to the pre- training procedure. Moreover, the enhanced steerability afforded by MeCo holds promise in creating more control- lable language models, and its general compatibility with more fin--grained and creative metadata invites further ex- ploration. A Altogether, MeCo is a simple, flexible, and effec- tive training paradigm that simultaneously enhances both the utility and steerability of language models.
2. Metadata Conditioning then Cooldown
In this section, we describe our pre-training approach in detail. We assume each document in the pre-training dataset is associated with some metadata c. In our main experi- ments, we use the document URL's absolute domain name as c. For example, if the document's URL is https:
We also employ the following techniques in all our exper- iments, as they enhance the baseline pre-trained models' performance based on our preliminary experiments: (1) we
hnttpa://comonrravi1 .org/
2

---

Metadata Conditioning Accelerates Language Model Pre-training
Table 1: Our main experimental results of pre-training a 1.6B language model on 160B tokens from DCLM. MeCo significantly outperforms standard pre-training and achieves equivalent average performance to the 240B-token baseline while using 33% less data. Interestingly, validation perplexity (PPI.) does not correlate with downstream performance.
Model
PPL
MMLU
ARC-c
ARC-c
CSQA
HSwag
OBQA
PIQA
SIQA
WG
TruQA
Avg.
Standard
13.2
36.1
75.1
42.7
64.8
66.7
46.0
74.3
54.2
62.0
35.2
55.7
+
Data sel.
37.2
74.6
44.3
62.9
65.5
46.8
74.3
52.4
64.3
37.8
56.0
+ 80B tokens
12.9
37.1
75.2
43.2
64.1
67.7
49.8
74.7
54.9
62.8
37.8
56.7
McCo
13.3
36.3
75.7
44.1
63.8
67.3
51.2
73.4
52.6
64.2
38.5
56.7
10.2
10.6
11.4
41.0
10.6
15.2
10.9
U1.6
12.2
13.3
11.0
Standard
MeCo OpenBookQA
ARC-Easy
ARC-Challenge
HellaSwag
WinoGrande
Average
Accuracy (%)
77
62
47
43
48
80
160
80
160
80
160
80
160
80
160
80
160
# Training Tokens (B)
Figure 2: MeCo downstream task performance throughout training (1.6B model on DCLM). Each checkpoint of MeCo includes a 16B-token cooldown in the end. The total number of tokens used by the baseline and the corresponding MeCo checkpoints are the same for fair comparison. The reported average numbers are over all 10 tasks. Full results in Table 16.
disable cross-document attention (Dubey et al., 2024; Ding et al., 2024), which both speeds up the training (25% faster for 1.6B model) and improves the downstream perfor- mance ($ (BB.I): ) (2) when packing multiple documents into one sequence, ensure each sequence starts with document rather than in the middle of one-this may result in some data being discarded when packing documents to a fixed length, but it proves beneficial for improving down- stream performance.
we perform only one run for each experiment; however, we demonstrate in $ B.2 that the variance of our experiments should be low. $A.5 outlines the computational resources required for our experiments.
Pre-training data. We use the best-performing open-source pre-training corpus, DCLM- -aaelllieel (Li et al., 2024), for our main experiments. Additionally, we conduct experi- ments with two other data sources: a reproduction of Re- finedWeb (Penedo al., 2023) from Li al. (2024) and the C4 dataset (Raffel et al., 2020). In the paper, re- fer to these data sources as DCLM, RefinedWeb, and C4, respectively. Notably, DCLM is a subset of RefinedWeb, acquired by using a fastText classifier (Joulin et al., 2017) for selecting high-quality data (Li et al., 2024). Please refer to $ A.4 for more details.
3. MeCo Accelerates LM Pre-training
In this section, we demonstrate that MeCo can significantly accelerate language model pre-training ($ 3.2). We also show that MeCo leads to consistent gains across different model scales ($3.3) and training data ($3.4).
3.1. Experiment setup
Evaluation. We adopt the OLMES suite (Gu et al., 2024) for evaluation, which includes the following tasks: MMLU (Hendrycks et al., 2021), ARC-Easy (ARC-e; Clark et al., 2018), ARC-Challenge (ARC-c; Clark et al., 2018), CommonsenseQA (CSQA; Talmor et al., 2019), HellaSwag (HSwag; Zellers al., 2019). OpenBookQA- (OBQA: Mi- haylov et al., 2018), PIQA (Bisk al., 2020), Social IQA (SIQA; Sap al., 2019), and WinoGrande (WG; Sakaguchi et al., 2021). We also add the popular TruthfulQA dataset (TruQA; Lin et al., 2022). Throughout the paper, we report the average performance across all 10 tasks as "Avg.". Un- less specified, we always report 5-shot in in-context learning
We utilize the Llama (Touvron et al., 2023a;b; Dubey et al., 2024) version of the Transformer architecture (Vaswani al.. 2017) and the Llama-3 tokenizer for all our experi- ments. We conduct experiments with four different model sizes: 600M, 1.6B, 3B, and 8B. The architecture details are in A.2. We employ standard optimization settings for lan- guage models, i.c., AdamW W optimizer and cosine learning rate schedule. We follow Li et al. (2024) for hyperparame- ters and the details can be found in $ A. Due to the high cost associated with pre-training and our limited resources,
3

---

Metadata Conditioning Accelerates Language Model Pre-training
Standard
MeCo
ARC-Easy
OpenBookQA
Average over 10 Tasks
55
62
+0.6
Accuracy (%)
+3.8
+2.8
+0.6
+0.5
2.2
+0.2
+0.1
64
42
48
600M
1.68
3B
8B
600M
1.68
3B
8B*
600M
1.6B
3B
BB*
#Parameters
Figure 3: MeCo results across different model scales (160B tokens from DCLM except for the 8B model, which is trained on 80B tokens due to resource constraints). Full results Table 17. We report the average numbers across all 10 tasks. MeCo improves models across scales and leads more gains for billion-parameter models compared to smaller models.
Standard
MeCo
ARC-Easy
OpenBookQA
Average Tasks
53
59
Accuracy (%)
+0.6
+5.2
+1.0
+2.2
+0.8
RefinedWeb
DCLM
C4
RefinedWeb
DCLM
RefinedWeb
DCLM
Figure 4: Results of applying MeCo over different pre-training corpora (1.6B models, 160B tokens). Full results in Table 18. We report the average numbers across all 10 tasks. MeCo provides consistent gains across different pre-training sources.
results. OLMES enhances evaluation reliability by offer- ing three key features: (1) provides manually-curated in-context examples for each task; (2) it evaluates with both multiple-choice format and cloze format, and takes the best of two; (3) it applies ablated calibration method (Brown et al., 2020; Holtzman et al., 2021) to each individual task. During evaluation, we sample 1,000 examples for each task, which improves efficiency while providing the same reliable results as full evaluation.
the classifier must be applied over the whole corpus.
Training with more data (+ 80B tokens): We train a standard model with 240B tokens, with the same optimization hyperparameters.
We first observe that MeCo achieves significantly better performance than standard pre-training across most tasks. Additionally. MeCo surpasses the data selection baseline?; unlike data selection, our approach does not incur any com- putational overhead, as it leverages readily available URL information from the pre-training data. More importantly, MeCo achieves performance comparable standard pre- training while using 33% less data and compute, represent- ing substantial gain in data efficiency.
3.2. MeCo achieves comparable performance to standard pre-training with 33% less data
Table shows our main results pre-training 1.6B lan- guage model 160B tokens from DCLM. Besides standard also feature other experi-
pre-training (Standard), ments, both of which use more resources and only serve as references instead of fair comparisons:
We also illustrate the changes in downstream task perfor- mance throughout the pre-training process in Figure 2. For MeCo, each checkpoint in the figure includes a cooldown phase on 16B tokens (10% of the total training tokens). For instance, the 8OB checkpoint consists of 64B tokens of conditional training followed by 16B tokens of cooldown.
Data selection (+ Data sel.): We employ the fastText data selection classifier from Li et al. (2024) choose the top 70% documents from a 250B-to Boken pool of DCLM data-tth this is similar to the high-quality data used in Section 5 of Li et al. (2024). According to the Table 4 from et al. (2024), this fastText classifier achieves state-of-ther -hrt---rr data selection performance. This method incurs additional computational cost since
"It important to note that DCLM data already a subset of the RefinedWeb data selected by this classifier. We do not claim that McCo consistently outperforms data selection; rather, demonstrate that MeCo can be integrated with data selection achieve further improvements, while data selection alone tends yield diminishing returns.

---

Metadata Conditioning Accelerates Language Model Pre-training
Table 2: Conditional inference further improves MeCo per- formance (full results in Table 19).
4. Conditional Inference Steers Language Model Generations
Pre-training
McCo not only improves the general quality of pre-trained language models (evaluated by standard few-shot down- stream task performance), but also unlocks the possibility of steering the model's s generations during inference by con- ditioning it on particular URLs. We term this paradigm conditional inference, as illustrated in Figure 5.
Inference
Standard
MeCo
Unconditional
55.7
56.7
Conditional
55.8 n0.1
57.2 10.5
Unconditional inference prompt
Q: What the circulatory system? A:
We observe that MeCo consistently surpasses the baseline model, particularly in the later stage of training.
Conditional inference prompt
URL: www.factquizzaaterr.com
Q: What is the circulatory system? A:
Discussion of perplexity. Table reveals that validation perplexity does not correlate with downstream performance in our experiments. Notably, when comparing the 240B baseline to the 160B MeCo model, the baseline exhibits much lower perplexity due to the larger data size, yet the two models achieve similar average downstream performance. This observation aligns with previous studies (Tay et al., 2022; Liu et al., 2023; Wettig et al., 2024) indicating that perplexity is not always reliable indicator of downstream performance; the final task performance can be impacted by other critical factors, such as inductive bias.
(does not have to be a real URL)
Illustration of conditional inference: We can con- dition the model by prepending a URL to the prompt. The URL does not need to real one.
Figure 5:
Steering language model generations by conditioning the model on a "control sequence" has been explored in the past, either for style control (Keskar et al., 2019) or for avoiding harmful content (Korbak et al., 2023a). In this section, we study how combining conditional inference and MeCo (even with cooldown) can both improve the downstream task per- formance and reduce the likelihood of harmful generations.
3.3. MeCo improves performance across model scales
Figure 3 demonstrates the results across different model scales (600M, 1.6B, 3B, and 8B). We train all the models with the same optimization hyperparameters and the same amount of data (160B on DCLM) except for the 8B model, which is trained on 80B tokens with a lower learning rate due to resource constraints and training instability (details in A..)).
4.1. Conditional inference improves MeCo's downstream task performance
In this section, we demonstrate how prepending ap- propriate URLs to the inputs improves MeCo's down- stream performance. We first design URI for each downstream task used in our evaluation, for exam- ple, www. www.facttuizmaatte. com for OpenBookQA and www www.socian a sk 1 sassessment. com for So- cial IQA. You can find all the customized URLs in Table 11. We note that (I) the URLs do not need to be real; (2) we did not use trial-and rial-and-error when choosing the URLs to avoid overfitting the test set.
We first observe that MeCo improves model performance across all scales. Although the trend is somewhat noisy, MeCo appears to yield greater improvements for larger mod- els, with billion-parameter models showing more significant gains compared to the 600M model. Note that this is a qualitative observation, as downstream task performance is known to scale less smoothly compared to pre-training loss.
We apply the same set of customized URLs to both the standard model and MeCo (1.6B, 160B DCLM tokens) and the results are shown in Table 2. We see that applying con- ditional inference leads to little difference on the standard model but a significant improvement on MeCo. Overall, MeCo with conditional inference achieves 1.5% absolute improvement compared to standard pre-training with uncon- ditional inference.
3.4. MeCo improves performance across different training corpora
We train 1.6B models on 160B tokens from three different data sources: C4, RefinedWeb, and DCLM. We present the results in Figure 4. If we use the average downstream performance as an indicator for data quality, we can rank the three data sources as DCLM > RefinedWeb > C4. We observe that MeCo provides consistent and significant gains across different data sources, both on the average accuracies and individual tasks.
We also explore the impact of different URLs on perfor- mance, as shown in Table 3. In this experiment, we use two real URLs: boards. 4chan. org, an anonymous im- ageboard known for association with offensive content,
5

---

Metadata Conditioning Accelerates Language Model Pre-training
Table 3: Zero-shot evaluation of MeCo (I.6B, 160B DCLM tokens) with different URLs. We show the delta between unconditional inference and using URLss.
Standard
MeCo
Toxicity
Standard inference
Inference URLs
ARC-e
ARC-c
CSQA
OBQA
Conditional
2.1
inference
Unconditional
69.6
43.2
54.7
48.4
4chan. org
66.7 129 41.1
(2.1
53.6
41.1 47.8 10.6
Obscene
factmonster..
70.7 11.1 45.7
12.5
60.9
16.2 52.4 14.0
Standard inference
Conditional inference
0.3
and www. factmonster.......m com, trivia website. Unlike our main experiment, we employ zero-shot prompting to highlight the effects of different URLs. Our findings in- dicate that selecting appropriate URL can significantly enhance zero-shot results compared to using more ad- versarial one: for example, using factmonster. com outperforms 4chan. org by 7.3% on CommonsenseQQ..
0.2
Insult
Standard inference
Conditional inference
0.5
Toxicity Score (10-²)
Figure 6: McCo with conditional inference (using en wiki wikipedia. org) significantly reduces harmful gen- erations from the model.
4.2. MeCo with conditional inference reduces harmful generations from the model
In addition to improving downstream task performance, MeCo with conditional inference also reduces harmful gen- erations. To evaluate the toxicity of model generations, we follow (Korbak al., 2023b) to sample 4096 text sequences from the models, with temperature T 0.7 and top-p=0.9. The generated sequences have lengths between 10 and 128 tokens. For unconditional inference, the model is only con- ditioned on the BOS token. For conditional inference, the model conditioned on en. wikipedia..o. org.
four different strategies: only standard data, only metadata- conditioned data, directly mixing the two sources of data throughout training (90% URL + 10% standard) and two- stage training (i.e., first 90% with metadata conditioning and then 10% standard data)-the the last one is MeCo.
Table 4 demonstrates the results of the different mixing strategies. First, see that only training on metadata- conditioned data leads to performance degradation, empha- sizing the importance of cooldown. While both directly mixing the two types of data and two-stage training improve the performance compared to the standard pre-training base- line, first training on metadata conaadaata-codiiooneed data and then cooldown with standard data leads to better and more con- sistent gains. We also perform additional ablations on the length of cooldown in B.3, which show that 10%-20% cooldown achieves the best performance (and we use 10% in our experiments).
To obtain toxicity scores, we follow the setup in (Korbak al., 2023b) and the toxic comment classifier Detoxify (Hanu & Unitary team, 2020). We use the unbiased model from Detoxify, which is based on RoBERTa (Liu et al., 2019) and trained on a human-labeled dataset of nearly 2 million comments, created for the task of evaluating unin- tended bias (Borkan et al., 2019). The classifier provides both general toxicity scores and more granular scores (e.g.. obscene, insult).
We show the averaged toxicity scores over all sam- pled generations in Figure 6. We observe that using en wikipedia org for conditional inference reduces the toxicity scores of generations from both the standard pre- training model and MeCo. Conditional inference is more effective on MeCo, leading significantly lower toxicity score compared to the baseline.
5.2. Understanding the role of metadata
To better understand how MeCo works, we experiment with various types of metadata and present the results in Ta- ble 5. Below, we describe these metadata types and their outcomes.
URL variants. We URL variants that provide more information (full URLs) and less information (URL suf- fixes). While full URLs perform similarly to MeCo, using URL suffixes results in significant performance degradation, suggesting that absolute domain names (e.g., en. skiipeiia..... org) provide the appropriate granular- ity as metadata.
5. Ablation Studies
5.1. Different strategies for mixing metadata-condiitooned and standard data
In this section, we study the best strategy to mix metadata- augmented data and standard data. We experiment with
6

---

Metadata Conditioning Accelerates Language Model Pre-training
Table 4: Different strategies of mixing metadatta-nugmeented and standard data. Full results in Table 20.
Model
ARC-e
ARC-c
HSwag
OBQA
10-Task Avg.
100% standard
75.1
42.7
66.7
46.0
55.7
100% URL
72.4 L2.7
28.8 413.
61.5 45.2
42.6 L3.4
50.3 45.4
90% URL + 10% standard
72.5 L2.6
43.1 10.4
66.9
10.2
50.0 14.0
56.4 10.7
McCo
75.7 n0.6
44.1
11.4
67.3
10.6
51.2 15.2
56.7 11.0
Table Ablations on using different metadata for MeCo. The average results are over all 10 tasks. Full results in Table 21.
Metadata
Examples
Avg.
URLs (MeCo)
en. en.wikipede kiikkaeedaa... org
56.7
Full URLs
en en.wikipedia org/wiki/Bil1 Gates
56.8 -0.1
suffixes
org
56.2 .0.6
Top 0.2% URLs (covering 42% texts)
en wm.wikipedia..o org or unknown
56.4 .0.3
Top 2% URLs (covering 65% texts)
en en.wikipedia, org or unknown
56.3 .0.4
Hashed URLs
56.7 -0.0
Model-generated topics
Technology leader biography
56.6 .0.1
Top URLs. We retain only the most frequently appearing URLs from the DCLM data and mark others as "unknown". We experiment with two tiers: top 0.2% URLs (each URL corresponds to roughly more than 1,000 documents, cover- ing 41.6% of the DCLM data) and top 2% URLs (each URL corresponds to more than 100 documents, covering 65. 1% of the DCLM data). The URL distribution in DCLM is highly skewed, with few top URLs covering a large portion of the data. Examples of top URLs are shown in Table 15. This experiment aims determine whether MeCo primar- ily benefits from modeling infrequent URLs. We find that using only top URLs does not match MeCo's performance, indicating that MeCo also benefits from low-frequency URLs.
pensive, taking roughly 1,500 GPU hours, similar to what is required to pre-train the 1.6B model. Hence, it is not a practical method but included for analysis purposes. We observe that using model-generated topics leads to similar results to our main MeCo model, suggesting that metadata based on document contents instead of sources is equally useful, prompting future explorations on more creative ways of generating metadata.
Our ablations suggest that metadata conditioning improves pre-training data efficiency by grouping documents together by source or topic. We propose two preliminary hypothe- ses as to how metadata conditioning affects model training: First, the model may automatically learn to prioritize doc- uments from useful sources or topics, thereby internally optimizing the mixture of training domains, which has been shown to be useful during pre-training (Xie et al., 2024; Jiang et al., 2024). Indeed, Allen-Zhu & Li (2024) also suggested that language models may autonomously identify domains rich in knowledge. Second, the model may use the additional metadata supervision to simply learn more structured representations of these large corpora, with no knowledge of the quality of each of the groups. We believe that the precise mechanism by which MeCo accelerates pre- training and improves model steerability warrants further thcoretical and empirical study.
high-frequency
We map each unique URL into random string investigate whether MeCo needs to learn the seman- ties of URLs simply recognizes that certain documents belong to the same groups. Surprisingly, using hashed URLs achieves performance on par with comantically-mennige URLs, indicating that the semantic meaning of the metadata not necessary for better pre-trained models instead,, sim- ply providing signals that group certain documents together sufficient for improving pre-training data efficiency.
Hashed URLs.
Model-generated topics. We explore ways of generating metadata in case readily available metadata is absent or in- model to generate two-word or three-word topic for each document, such "technology leader biography" or "gaming forum" (more details in $ A.6).. This is more fine-grained metadata compared to domains (c.g., "Wikipedia" or "Books"). Note that prompting models to generate topics is extremely ex-
sufficient. We prompt a --tee...1------setiet
6. Related Work
Metadata conditioning. CTRL (Keskar et al., 2019) first proposed "conditional language models" for controlled gen- eration: the method prepended the pre-training documents with "control codes" such as source domains, which allowed

---

Metadata Conditioning Accelerates Language Model Pre-training
for steering the generation during inference by prompting the model with different control codes. Dhingra et al. (2022)
language models. Consequently, there has been a huge amount of effort invested into improving pre-training data, starting from heuristic-based filtering (Raffel et al., 2020;
used timestamps as the metadata to train time-aware lan- guage models and Liu et al. (2020) adopted document lan- guages as the metadata for a multilingual pre-trained model. Aghajanyan et al. (2022) pre-trained language models on hyper text, which provided extra metadata such as class s and id, which allowed for conditional inference as well. Kyrylov & Chaplynskyi (2023) pre-trained language mod- els on Ukrainian text conditioned on metadata. Weller et al. (2024) demonstrated that prompting models with text like "according to Wikipedia" improves their performance. Con- ditional training was also explored in alignment and prefer- ence optimization: Korbak et al. (2023a) pre-trained models with reward model scores as the prefix and Lu et al. (2022); Liu et al. (2024) conditioned the text on their quality mea- surements in post-training- bott allowed prompting the model with high quality score during inference to output more human-preferred text. Besides, Khalifa et al. (2024) used a similar idea to inject "document IDs" into the pre- training corpus to enable training data attribution, though the "IDs" were appended, instead of prepended.
Rae et al., 2021; Laurençon et al., 2022; Penedo et al., 2023;
Soldaini et al., 2024) and deduplication (Lee et al., 2022;
Anil et al., 2023; Touvron et al., 2023a; Abbas et al., 2023). Recently, model-based data filtering or data selection has emerged: many works sought to use simple ngram models to select those that resemble high-quality domains such as Wikipedia (Brown et al., 2020; Xie et al., 2023; Li et al., 2024) or to use an existing language model for perplexity filtering (Wenzek et al., 2020; Muennighoff et al., 2023; Marion et al., 2023). Gunasekar et al. (2023); Wettig et al. (2024); Penedo et al. (2024); Dubey et al. (2024) instead used a large language model score instances based on abstract values such as whether they are "educational"-buu these methods introduce considerable overheads rumning these language models over the whole pre-training corpus is costly and whether they can lead to better performance under the same computational budget is unclear (Goyal et al., 2024; Kaddour et al., 2024).
Another line of works aimed to adjust the domain mix- ture for more data-efficient training (Xie al., 2024; Xia et al., 2024; Jiang et al., 2024). However, these models require an existing domain taxonomy (which is usually very coarse-grained) and a target loss to optimize for-which has been shown to not always correlate with downstream performance (Tay et al., 2022; Liu et al., 2023).
Recently, Allen-Zhu & Li (2024) investigated language mod- els' ability to memorize knowledge by using synthetically generated biographical data. They trained models on a mix- ture of such data and unrelated data, and tested models on recalling the biographical information. They found that prepending a special token to the biographical data enhanced the model's memorization capacity. The authors argued that this technique helped models recognize high-quality sources and was analogous to adding URLs to pre- training docu- ments. However, the controlled setting in Allen-Zhu & Li (2024) was limited to two synthetic data sources and did not incorporate real URLs, making it fundamentally different from our experimental setup and contributions.
Recently, Wettig et al. (2025) introduced a method for con- structing domain taxonomies and automatically annotating pre-training data-with the domain annotations, they fur- ther explored optimizing domain mixtures for better down- stream performance. This approach also highlighted the link between two data selection approaches mentioned be- fore: applying quality filtering implicitly changes the data domain mixture. We find a connection our work as well: while Wettig et al. (2025) focus annotating data with coarse-grained domains, we utilize URLs to provide more fine-grained domain information.
We also highlight two concurrent works: Zhu et al. (2025) and Wang al. (2025). The former uses synthetic periments and theoretical analysis to show that context- enhanced learning-suce such as prepending metadata- can im- prove sample efficiency. The latter demonstrates the benefits of conditional generative modeling when source distribu- tions share certain similarities.
7. Conclusion
We introduce metadata conditioning then cooldown (MeCo), an extremely simple method that consistently outperforms standard pre-training while incurring negligible computa- tional overhead. MeCo leverages commonly available meta- data, such as source URLs, by prepending them to pre- training documents. At the end of training, MeCo removes the URLs from the data to enable inference without meta- data. Through comprehensive experiments across various model scales and training corpora, we demonstrate MeCo's effectiveness, achieving up 33% speedup in pre-training. Additionally, we show that prompting McCo models with
While our idea and findings echo previous and concurrent literature, our paper is the first to explore the use of metadata conditioning in modern-scale LM pre-training and its effect on downstream task performance. Compared to other types of metadata explored by prior work, we use URLs as they can be acquired with no additional cost and they are more informative than source domains or reward scores.
Selecting pre-training data. The quality of pre- pre-training corpora is essential for the performance of the resulting
8

---

Metadata Conditioning Accelerates Language Model Pre-training
suitable metadata can further enhance their downstream performance and mitigate harmful outputs. Our findings underscore the potential of metadata conditioning to en- hance data efficiency in pre- pre-training and to develop more controllable and steerable language models.
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z. et al. PaLM 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Bisk, Y., Zellers, R., Le bras, R., Gao, and Choi, Y. PIQA: Reasoning about physical commonsense in natural lan- guage. Proceedings of the AAAI Conference on Artificial 2020.
Impact Statement
Intelligence, 34(05):7432-
MeCo is a simple, general method that can significantly ac- celerate language model pre- pre-training, making it highly appli- cable in both industrial applications and academic research. MeCo also enables steering language models to output less harmful content, potentially supporting LM safety research and the safe deployment of LMs in real-world applications.
Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasser- man, L. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, WWW 19, pp. 491-500, New York, NY, USA, 2019. Association for Computing Machinery.
Limitations
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Due to limited resources and the costly nature of pre- training, we do not perform multi-run experiments; however, we show in $B.2 that the variance of our experiments should be low and our results are significant. All our investigations are limited to English corpora. We do not study the inter- play between metadata conditioning and post- post-training pro- cedures. We also do not have a mechanistic understanding of how conditioning metadata helps improve the down- stream performance. We hope our results can shed light on these interesting questions and motivate further research on metadata conditioning.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? Try ARC, the A12 reasoning chal- 2018.
lenge. CoRR,
Dhingra, B., Cole, J. R., Eisenschlos, M., Gillick, D., Eisenstein, J., and Cohen, W. W. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257-273. 2022.
Acknowledgments
We acknowledge Angelica Chen, Sanjeev Arora, Kyunghyun Cho, Yisong Yue, Luca Soldaini, and members of Princeton Language and Intelligence for their helpful feedback and discussion. Tianyu Gao is supported by an IBM PhD Fellowship. This research funded by the National Science Foundation (IIS-221177) and a Sloan Research Fellowship.
Ding, H., Wang, Z., Paolini, G., Kumar, V., Deoras, A., Roth, D., and Soatto, S. Fewer truncations improve language modeling. In Forty-first International Conference on Machine Learning, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 herd of models. arXiv preprint 2024.
arXiv:2407.211
References
Goyal, S., Maini, P., Lipton, 7. C., Raghunathan, A., and Kolter, J. Z. Scaling laws for data filtering- data cura- tion cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22702-22711, June 2024.
Abbas, A., Tirumala, K., Simig, D., Ganguli, S., and Mor- cos, A. SemDeDup: Data-efficient learning at web- scale through semantic deduplication. arXiv preprint arXiv:2303.09550, 2023.
Gu, Y., Tafjord, O., Kuehl, B., Haddad, D., Dodge, J., and Hajishirzi, H. Olmes: A standard for language model evaluations. arXiv preprint arXiv:2406.084444 2024.
Aghajanyan, A., Okhonko, D., Lewis, M., Joshi, M., Xu, H., Ghosh, G., and Zettlemoyer, L. HTLM: Hyper-text pre-training and prompting of language models. In Inter- national Conference on Learning Representations, 2022.
Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T. Giorno, A. D., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang. X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y. Textbooks are all you need, 2023.
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint arXiv:2404.00 05405, 2024.
9

---

Metadata Conditioning Accelerates Language Model Pre-training
Hanu, L. and Unitary team. Detoxify. http://giihab..com//naiaaaaa//at 2020.
Github.
Kyrylov, V. and Chaplynskyi, D. GPT-2 metadata pre- training towards instruction finetuning for Ukrainian. In Proceedings of the Second Ukrainian Natural Language Processing Workshop ( (UNLP), pp. 32-39, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Laurençon. H.,, Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Werra, L. V., Mou, C., Ponferrada, E. G., Nguyen, H., Frohberg, J., Saško, M., Lhoest, Q., McMillan-Major, A., Dupont, G., Biderman, S., Rogers, A., allal, L. B., Toni, F. D., Pistilli, G., Nguyen, O.. Nikpoor, S., Masoud, M., Colombo, P., de la Rosa, J., Villegas, P., Thrush, T., Longpre, S., Nagel, S., Weber, L., Muñoz, M. R., Zhu, J., Strien, D. V., Alyafeai, Z., Almubarak, K., Chien, V. M., Gonzalez-Dios, I., Soroa, A., Lo, K. Dey. M., Suarez, P. O., Gokaslan, A., Bose, S., Adelani, D. I., Phan, L.., Tran, H., Yu, I., Pai, S., Chim, J., Lepereq. V., Ilie, S., Mitchell, M., Luccioni, S., and Jernite, The bigscience ROOTS corpus: A 1.6TB com- posite multilingual dataset. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmurks Track, 2022.
Holtzman, A., West, P., Shwartz, V., Choi, Y., and Zettle- moyer, L. Surface form competition: Why the highest probability answer isn't t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7038-7051, 2021.
Jinng Y., Zhou, A., Feng. Z., Malladi, S., and Kolter, 7. Adaptive data optimization: Dynamic sample selection with scaling laws. arXiv preprint arXiv:2410.11820, 2024.
Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 427-431, 2017.
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison- Burch,, C.,, and Carlini, N. Deduplicating train- ing data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 8424- 8445, 2022.
Kaddour, J., Key, O., Nawrot, P., Minervini, P., and Kusner, M. J. No train no gain: Revisiting efficient training algo- rithms for transformer-basee language models. Advances in Neural Information Processing Systems, 36, 2024.
Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha E., Keh, S., Arora, K., Garg, S., Xin, R., Muennighoff, N., Heckel, R., Mercat, J., Chen, M., Gururangan, S., Wortsman, M., Albalak, A., Bitton, Y., Nezhurina, M., Abbas, A., Hsieh, C.-Y., Ghosh, D.. Gardner, J., Kilian, M., Zhang, H., Shao, R., Pratt, S., Sanyal, S., Ilharco, G., Daras, G., Marathe, K., Gokaslan, A., Zhang, J.. Chandu, K., Nguyen, T.. Vasiljevic. I., Kakade, S., Song, S., Sanghavi, S., Faghri, F., Oh, S.. Zettlemoyer, L.,, Lo, K., El-Nouby, A., Pouransari, H., Toshev, A., Wang, S., Groeneveld, D., Soldaini, I.., Koh, P. W., Jitsev, J., Kollar, T., Dimakis, A. G., Carmon, Y.. Dave, A., Schmidt, L., and Shankar, V. Datacomp- In search of the next generation of training sets for language models. arXiv preprint arXiv:2406. / 2024.
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C.,, and Socher, R. Ctrl: A conditional transformer lan- guage model for controllable generation. arXiv preprint arXiv:1909.05855. 2019.
Khalifa, M., Wadden, D.,. Strubell, E., Lee, H., Wang, L., Beltagy, I., and Peng, H. Source-aware training enables knowledge attribution in language models. In First Con- ference Language Modeling, 2024.
Korbak, T., Shi, K., Chen, A., Bhalerao, R. V., Buckley, C., Phang, J., Bowman, S. R., and Perez, E. Pretrain- ing language models with human preferences. In Inter- national Conference on Machine Learning, pp. 17506- 17533, 2023a.
Korbak, T., Shi, K., Chen, A., Bhalerao, R. V., Buck- ley, C., Phang, J., Bowman, S. R., and Perez, E. Pretraining language models with human prefer- ences. In Krause, A., Brunskill, E., Cho, K., En- gelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Ma- chine Learning Research, pp. 17506-17533. PMLR, 23- 29 Jul 2023b. URL https: /ptp/peeeongnssseeeeeeppppe...ee mlr. treas//202//orbakke html.
Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measur- ing how models mimic human falsehoods. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3214-3252, 2022.
Liu, H., Xie, S. M., Li, Z., and Ma. T. Same pre-training loss, better downstream: Implicit bias matters for lan- guage models. In International Conference on Machine Learning, pp. 22188 22188-22214 PMLR, 2023.
10

---

Metadata Conditioning Accelerates Language Model Pre-training
Liu, H., Sferrazza, C., and Abbeel, P. Chain of hindsight aligns language models with feedback. In The Twelfth International Conference on Learning Representations, 2024.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21 21(1):5485-5511 2020.
Liu, Y. Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer. L., and Stoyanov, V. RoBERTa: A robustly optimized BERT pretraining ap- proach. arXiv preprini
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
16922 2019.
Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., and Zettlemoyer, L. Multilingual denois- ing pre-training for neural machine translation. Transac- tions of the Association for Computational Linguistics, 8: 726 2020.
Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLL)) pp. 1463-4473, 2019.
Lu, X., Welleck, S., Hessel, J., Jiang, L.,, Qin, L.,, West, P., Ammanabrolu, P., and Choi, Y. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022.
Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, 2023.
Marion, M., Üstün, A., Pozzobon, L., Wang, A., Fadaee, M., and Hooker, S. When less is more: Investigating data pruning for pretraining LLMs at scale, 2023.
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkin- son, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muen- nighoff, N., Naik, A., Nam, C., Peters, M., Ravichander, A., Richardson, K., Shen, Z., Strubell, E., Subramani, N., Tafjord, O., Walsh, E., Zettlemoyer, L., Smith, N., Hajishirzi, H., Beltagy, I., Groeneveld, D., Dodge. J., and Lo,, K. Dolma: open corpus of three trillion tokens for language model pretraining research. In Ku, L.-W., Mar- tins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp- 15725-15788, 2024.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 238 12391, 2018
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. Scal- ing data-constrained language models. In shirth-seventh Conference on Neural Information Processing Systems, 2023.
Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Alobei- dli, H., Cappelli, A., Pannier, B., Almazrouei, E., and Launay, J. The refinedweb dataset for falcon Ilm: Outper- forming curated corpora with web data only. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 79155-79172. Curran Associates, Inc., 2023.
Talmor, A., Herzig, J., Lourie. N., and Berant, J. Com- monsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 41 4149-47 4158, 2019.
Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., and Met- zler, D. Scale efficiently: Insights from pretraining and finetuning transformers. In International Conference on Learning Representations, 2022.
Penedo, G., Kydličck, H., Lozhkov, A.. Mitchell, M., Raffel, C., Von Werra, L.,, Wolf, T., et al. The fineweb datasets: Decanting the web for the finest text data at scale, arXiv preprint arXiv:2406. 17557, 2024.
Rae, J. W. Borgeaud, S., Cai, T., Millican, K., Hoffmann, J. Song, F,, Aslanides, J., Henderson, S., Ring, R., Young, S., et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2/ /2./ 1446, 2021.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint 2rX 2023a.

---

Metadata Conditioning Accelerates Language Model Pre-training
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C.,, Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V. Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A.. Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog. I, Nie, Y. Poulton, A., Reizenstein, J., Rungta, R., Saladi, K. Schelten, A., Silva, R., Smith, F. M., Subramanian, R., Tan, X. E., Tang. B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S.,
Xie, S. M., Santurkar, S., Ma. T., and Liang, P. Data selection for language models via importance resam- pling, Advances in Neural Information Processing Sys- tems (NeurIPS), 2023.
Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, S., Le, Q V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pre- training. Advances in Neural Information Processing Systems, 36, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4791-4800, 2019.
and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Zhu, X., Panigrahi, A., and Arora, S. On the power of context-enhanced learning in llms. arXiv preprint urXiv:2503.08821 2025.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t.., and Polosukhin, I. Atten- tion is all you need. Advances in Neural Information Processing Systems (NIPS), 30, 2017.
Wang, R., Zhang, Y., Zheng, C., Li, C., and Wu, G. A theory for conditional generative modeling on multiple data sources. arXiv preprint arXiv:2502.14455
2025.
Weller, O., Marone, M., Weir, N., Lawrie, D., Khashabi, D., and Van Durme, B. "according to .o..." ".. Prompting language models improves quoting from pre-training data. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Pupers), 2024.
Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4003-4012, 2020.
Wettig, A., Gupta, A., Malik, S., and Chen, D. QuRating: Selecting high-quality data for training language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Ma- chine Learning Research, pp. 52915 21-27 Jul 2024.
Wettig, A., Lo, K., Min, S., Hajishirzi, H., Chen, D., and Soldaini, L. Organize the web: Constructing domains enhances pre-training data curation. In International Conference on Machine Learning (ICML), 2025.
Xia, M., Gao, T,, Zeng, Z., and Chen, D. Sheared LLaMA: Accelerating language model pre-training via structured pruning. The Twelfth International Conference on Learning Representations, 2024.
12

---

Metadata Conditioning Accelerates Language Model Pre-training
A. Experiment Details
A.1. Hyperparameters
Table 6 shows the hyperparameter settings used in our experiments. We follow Li al. (2024) for the high learning rate and weight decay except for the model, which requires lower learning rate for numerical stability.
Hyperparameters
Values
Optimizer
AdamW (3) 0.9. 32 = 0.95)
Learning rate
3e - (Sc 4 for the 8B model)
Weight decay
0.033 (0.1 for the model)
Batch size
4M tokens
Warmup
5% lincar warmup
Schedulc
Cosine decay 10% of the peak learning rate
Seq length
Pack 8192 tokens
Table 6: Hyperparameter settings for our experiments.
A.2. Model configurations
We the Llama variant (Touvron et al., 2023a) of Transformers (Vaswani et al., 2017) for experiments. All models the Llama-3 tokenizer (Dubey et al., 2024). We add BOS and an EOS token at the beginning and end of every document. The detailed configurations are specified in Table 7.
#Param
#Layers
Hidden
Intermediate
#Heads
Head Dim
600M
1024
4096
16
1.6B
2048
5504
16
128
3B
28
3072
8192
24
128
8B
32
4096
14336
32
128
Table 7: Model configurations for our experiments.
A.3. Cooldown details
The metadata conditioning stage (90%) and the cooldown stage (10%) share the same learning rate schedule- -i. the metadata conditioning stage will end at the 90% of the learning rate schedule and the cooldown stage will resume from that same point on the schedule and continue the learning rate decay. It also inherits all the optimizer states. To ensure the cooldown stage does not see repeated data as the conditional training stage, we use different subset of data for cooldown for all our DCLM experiments.
For our 8B experiments (80B tokens), due to the checkpoint saving configuration, performed 10B-token cooldown (12.5% instead of 10% of the total training).
A.4. Dataset details
Table 8 shows the dataset details for our pre-training experiments.
Dataset
Description
The SlimPajama (Soboleva al., 2023) subset
RefinedWeb
CeeM-dudodducddd (Li et al., 2024) RefinedWeb
DCLM
M-M.M-al,iin,, which is a filtered version of DCLM-reproduced Refined
Table 8: Pre-training dataset details.
13

---

Metadata Conditioning Accelerates Language Model Pre-training
A.5. Experimental resource
Table 9 shows the resources required train the models in our experiments. Our main models (1.6B, 160B tokens) take roughly 2 days to train on 32 H100 GPUs.
#Params
600M
1.6B
1.6B
3B
8B
#Tokens
160B
160B
240B
160B
80B
#GPU hours
776
1536
2304
3085
3905
Table 9: Resources required to train the models in experiments (H100 GPU hours).
A.6. Prompts for model-generated topics
Table 10 shows the prompt used for generating topics. We prompt 1-mma-3.-8B-IIntruucc model to generate topics. We only use the first 1024 tokens from the document as the snippet. We use greedy decoding.
Based en the given sampled snippet from document (could be a webpage, book, a codebase, paper, anything else), write domain keyphrase (within 4 words; for example, code, international news, food blog, biography, science fiction, polities essay, gaming forum, algebra quiz, physics textbook, restaurant advertisement, religous story, etc.) for the document. The "demain keyphrase" should consider both the topics and the genre/scurce of the document.
Start the snippet
[[snippet]]
End of the snippet
Now output the domain (do not output other things):
Table 10: The prompt for generating topics.
A.7. Customized URLs for conditional inference
Table 11 shows the customized URLs for conditional inference.
Tasks
Customized URLs
MMLU
wstppepporttall.. .com
ARC-Easy
www s www.scieennnn
www.sciienneesuuuyyuppa... com
ARC-Challenge
www. www.scierccetuuyyuu... com
CommonsenseQA
www www. www.gqaata.t.commm
HellaSwag
www.wwwwwoogiz.cco.
OpenBookQA
www www.quuiiasttrr.rcmmmmuuu .om
PIQA
www www.bbaacckkkolleeeeeii...cooo
Social IQA
www www. www.sccccalkkllllaeeeeeeeenn.nt com
WinoGrande
www.tessppepprrccccc..oom www.faacccheckuuu.com
TruthfulQA
Table 11: Customized URLs for conditional inference.
B. Additional Experiments
B.I. Crose-document attention ablation
Table 12 shows a comparison between enabling and disabling cross-document attention. Disabling cross-document attention leads to significant speedups for our training (for a 1.6B model, is 25% faster). We also see that brings considerable
14

---

Metadata Conditioning Accelerates Language Model Pre-training
performance improvement on the vanilla model. Interestingly, the average performance does differ much between two different attention patterns for MeCo, suggesting that prepending the URLs to the document helps the model learn the noisy cross-document attention. Based on these results, all other experiments in this paper disable cross-document attention.
Model
MMLU ARC-
ARC-c CSQA
HSwag OBQA
PIQA
SIQA
WG
TruQA
Avg.
Standard
36.1
75.1
42.7
64.8
66.7
46.0
74.3
54.2
62.0
35.2
55.7
+Cross-doc attn
36.3
73.4
41,6
63.2
65.5
46.0
73.6
52.4
61.3
36.7
55.0
MeCo
36.3
75.7
44.1
63.8
67.3
51.2
73.4
52.6
64.2
38.5
56.7
+Cross-doc attn
35.5
72.7
45.4
66.3
66.1
51.8
74.4
52.8
62.4
38.2
56.6
Table 12: Cross-document attention ablation (160B tokens, 1.6B parameters).
B.2. Experiment variance
Due to the nature of pre-training experiments and the high cost associated with it, we perform single runs for all our experiments and do not report their standard deviations. However, we provide reference point here for estimating the variance of our experiments. We take the 90% checkpoint of the 1.6B- -arammeee,, 160B-token standard pre-training model, and continue the rest 10% of the training with three disjoint sets of data. Table 13 shows their performance. We see that while some individual tasks show performance differences, the standard deviation of the average performance is very low (0.1 1%), demonstrating that the average performance across our selected tasks is an indicative and stable metric.
Model
MMLU ARC-c ARC-c CSQA HSwag OBQA
PIQA
SIQA
WG
TruQA
Avg.
Standard run 1
36.1
75.1
42.7
64.8
66.7
46.0
74.3
54.2
62.0
35.2
55.7
Standard run 2
36.2
73.9
43.4
63.1
67.5
46.2
74.2
53.2
62.0
35.5
55,5
Standard run 3
36.3
73.8
43.2
63.4
67.5
45.8
74.5
54.2
62.8
34.7
55.6
Avg.
36.2
74.3
43.1
63.8
67.2
46.0
74.3
53.9
62.3
35.1
55.6
Std.
+0.1
+0.7
+00.4
+0.9
+0.5
+.02
+0.2
+0.6
+0.5
+0.4
+0.1
Table 13: Multiple runs of the baseline model (1.6B parameters, 160B tokens from DCLM). The average performance across runs shows low variance.
B.3. Cooldown length ablation
Table 14 shows the performance of different cooldown lengths. We see that performing 10% and 20% cooldown achieves similar results, while further increasing the length hurts the performance. For simplicity, we use 10% cooldown for all our experiments. We note that the best cooldown length can vary across different numbers of parameters, total numbers of training tokens, and the pre-training corpora; however, performing such fine-grained search across all different settings is intractable.
Model
MMLU ARC-e ARC-c CSQA
HSwag OBQA
PIQA
SIQA
WG
TriQA
Avg.
10% cooldown
36.3
75.7
44.1
63.8
67.3
51.2
73.4
52.6
64.2
38.5
56.7
20% cooldown
36.5
74.7
46.0
64.2
67.1
49.4
73.6
53.3
64.3
39.0
56.8
30% cooldown
36.7
74.8
45.0
60.9
67.5
49.0
74.2
51.6
62.8
39.2
56.2
Table 14: Ablations on different cooldown lengths (1.6B parameters, 160B tokens).
C. DCLM URL Distributions
Table 15 shows the top 50 URLs from DCLM and the corresponding document ratios.
D. Full Results
Table 16, Table 18, Table 17, Table 20, Table 21, and Table 19 show the detailed results of experiments reported in our main paper.
15

---

Metadata Conditioning Accelerates Language Model Pre-training
URLs
Document ratios
en en wikipedia..og stackoverilow.com
0.256%
0.240%
www. www.cheguardian.com
0.207%
www. www.urbandictionaryyccoo
0. 0.149%
www www.ffafniiiooooo
0. 148%
www www.bssnnessinsider.
0. 1.9%
gizmodo.com
0.12%
everything2.com
0.119%
www.
www.physicsforums.co com www
0.100%
www.refoeence.com
0.090%
www.thcttaatic.com www
0.087%
www.numsnct.com
0.086%
superuser.cco
0.086%
choxhound,ccow..om
0.085%
www.huffingtoopost..co
0.082%
serveriault.com
0.082%
www.engaddeet.com
0.079%
math.staackexchaag.ccom
0.078%
www.nytimes.com
0.075%
news.bbc.co.uk
0.073%
gawker.com
0.071 %
tvtropes.org
0.069%
www.insrructalles.com
0.069%
www.foll.com
0.068%
www.cnotcs.com
0.067%
townhall.com
0.067%
slashdot.org
0.066%
www.
www.foxnewa.co
0.066%
kotaku.com
0.066%
0.064%
www.reddit.com
0.063%
www.complex.com
0.063%
jezebel.com
0.062%
www.gamefags.com
0.061 %
www.al.jazeera.com
0.061 %
askubuntu.com
0.061%
abcnews.go.coo
0.060%
mathove:flow.nee
0.058%
www.csmonitor.com
0.058%
attic.es.lat.mmss.com
0.058%
www.bookzagss.cmm
0.057%
lifehacker.com
0.057%
www.sfgate.com
0.057%
ja
laloonik.com
0.057%
www.ancestry.com
0.057%
www.nifty.org
0.057%
www. www.theregister.co. co.
0.057%
www.omeess.com
0.056%
www.cnet.com
0.055%
www.ign.co
0.055%
Table 15: Top 50 URLs from DCLMM
16

---

Metadata Conditioning Accelerates Language Model Pre-training
#Tokens MMLU ARC-e ARC-c CSQA HSwag OBQA
PIQA
SIQA
WG
TruQA Avg.
Standard
16B
30.4
62.8
34.2
56.0
48.7
43.8
69.9
47.2
55.2
39.1
48.7
32B
32.1
66.8
37.3
60.0
55.9
45.2
70.3
46.7
56.5
38.6
50.9
48B
34.1
67.4
40.0
60.9
58.0
50.2
71.8
52.5
57.3
38.3
53.1
64B
34.0
69.2
39.8
61.6
59.8
46.8
72.7
50.2
59.2
36.3
53.0
80B
72.5
41.4
58.6
62.8
72.8
52.7
35.5
54.0
96B
34.9
71.2
40.2
62.1
63.5
45.8
72.4
53.5
60.4
36.4
54.0
112B
35.6
72.1
42.2
62.9
64.9
44.6
73.3
52.6
60.1
34.6
54.3
128B
35.9
73.5
42.5
62.8
64.5
44.2
73.1
53.9
61.0
35.3
54.7
144B
36.1
73.9
41.1
60.6
66.6
46.6
73.5
53.9
61.6
35.5
55.0
160B
36.1
75.1
42.7
64.8
66.7
46.0
74.3
54.2
62.0
35.2
55.7
McCo
16B
30.4
62.8
34.2
56.0
48.7
43.8
69.9
47.2
55.2
39.1
48.7
32B
66.0
38.7
58.2
53.9
70.6
41.8
51.2
48B
34.0
68.9
43.0
59.2
57.8
48.2
71.6
50.4
57.9
41.2
53.2
64B
34.2
70.6
41.9
62.6
60.4
46.0
72.1
50.5
59.1
40.1
53.8
80B
34.3
72.4
44.0
61.7
61.9
46.6
72.6
49.4
60.7
39.1
54.3
96B
34.9
72.5
44.3
63.1
64.1
48.2
72.9
49.5
61.7
38.7
55.0
112B
73.6
44.4
63.6
64.4
72.4
51.4
63.2
37.8
55.4
128B
35.7
74.6
44.5
64.9
66.9
49.4
73.0
51.5
63.0
37.5
56.1
144B
36.1
75.6
44.8
63.6
67.3
50.0
73.8
52.1
63.7
38.0
56.5
160B
36.3
75.7
44.1
63.8
67.3
51.2
73.4
52.6
64.2
38.5
56.7
Table 16: Intermediate checkpoint results for the 1.6B-parameter, 160B-token runs. For all MeCo checkpoints, we perform a 16B-token cooldown (i.e., the 64B checkpoint is 48B metadata conditioning training + 16B cooldown).
Model
MMLU ARC-c ARC-c CSQA HSwag OBQA
PIQA SIQA
WG TriQA Avg.
600M model, 160B tokens from DCLM
Standard MeCo
32.7
67.5
38.2
58.8
56.4
45.0
71.2
47.9
57.6
39.2
51.5
32.8
67.6
62.0
54.2
47.2
71.0
49.6
37.9
51.7
1.6B model, 160B tokens from DCLM
Standard MeCo
36.1
75.1
42.7
64.8
66.7
46.0
74.3
54.2
62.0
35.2
55.7
36.3
75.7
44.1
63.8
67.3
51.2
73.4
52.6
64.2
38.5
56.7
3B model, 160B tokens from DCLM
Standard MeCo
39.8 39.7
76.8
48.3
66.0
74.1
49.0
76.9
56.0
66.5
38.1
59.2
78.6
48.5
71.0
73.6
51.8
77.0
55.5
65.9
36.4
59.8
model, 80B tokens from DCLM
Standard
39.2
73.3
46.0
66.0
72.8
48.8
76.1
54.8
66.2
35.2
57.8
MeCo
39.5
77.1
71.2
52.6
75.8
53.8
65.2
35.0
58.4
Table 17: Results with different numbers of parameters. All experiments the same hyperparameters except for the 8B modeli, which uses smaller learning rate and fewer tokens due to training instability and limited compute resources.
17

---

Metadata Conditioning Accelerates Language Model Pre-training
Model
MMLU
ARC-e ARC-c CSQA HSwag OBQA
PIQA
SIQA
WG
TruQA Avg.
1.6B model, 160B tokens from C4
Standard McCo
31.0 31.9
59.8
36.1
55,8
64.9
42.8
72.5
49.7
60.0
32.0
50.5
62.0
37.8
54.3
63.6
43.6
74.0
50.0
58.9
39.5
51.6
1.6B model. 160B tokens from RefinedWeb
Standard MeCo
32.4 32.5
68.6
37.1
61.2
63.9
73.9
51.2
59.7
36.7
53.2
69.4
38.0
61.4
64.3
48.2
73.6
53.6
60.6
38.9
54.0
1.6B model, 160B tokens from DCLM
Standard McCo
36.1 36.3
75.1
42.7
64.8
66.7
46.0
74.3
54.2
62.0
35.2
55.7
75.7
44.1
63.8
67.3
51.2
73.4
52.6
64.2
38.5
56.7
Table 18: Detailed results different pre-training corpora.
Model
MMLU
ARC-e ARC-e CSQA HSwag OBQA PIQA SIQA
WG TruQA Avg.
Conditional Inference
Standard MeCo
36.1
73.8
42.4
66.1
66.6
46.2
73.4
53.5
62.6
37.1
55.8
36.3
74.2
44.6
65.2
67.6
51.6
73.4
53.2
66.0
40.1
57.2
Table 19: Full results of using conditional inference (1,6B parameters, 160B tokens).
Model
MMLU ARC-e ARC-c CSQA HSwag OBQA PIQA
SIQA
WG
TruQA Avg.
Standard
36.1
75.1
42.7
64.8
66.7
46.0
74.3
54.2
62.0
35.2
55.7
100% URL
33,9
72.4
28.8
37.2
61.5
42.6
72.9
52.1
60.5
41.0
50.3
90% URL + 10% Standard
36.4
72.5
43.1
63.7
66.9
50.0
75.7
53.1
62.8
39.9
56.4
MeCo
36.3
75.7
44.1
63.8
67.3
$1.2
73.4
52.6
64.2
38.5
56.7
Table 20: Different strategies of mixing metadata-augmeented and standard data.
Model
MMLU
ARC-e ARC-c CSQA
HSwag
OBQA
PIQA
SIQA
WG
TruQA
Avg.
URLs (MeCo)
36.3
75.7
44.1
63.8
67.3
51.2
73.4
$2.6
64.2
38.5
56.7
Full URLs
36.7
75.4
43.9
68.3
66.5
51.2
74.0
52.9
63.2
35.6
56.8
URL suffix
36.2
73.9
42.7
65.2
67.7
49.0
73.1
53.6
62.1
38.1
56.2
Top 0.2% URLs
36.2
76.6
44.1
66.9
66.3
47.6
74.5
53.7
63.1
35.3
56.4
Top 2% URLs
36.5
73.5
44.8
65.4
65.8
48.2
74.3
53.4
64.3
36.9
56.3
Hashed URLs
36.4
73.7
44.2
64.6
67.2
74.3
62.5
37.9
56.7
Topics
36.3
74.5
45.3
64.5
67.4
48.2
74.2
53.5
63.1
38.6
56.6
Table 21: Experiment results on using different types of metadata.
18


Reviews:
[{"review_id": "7kjpMWqphr", "reviewer": "Reviewer_TsUA", "summary": "This paper proposes Metadata Conditioning then Cooldown (MeCo) to accelerate LM pre-training. MeCo starts with pre-training LMs with metadata (URL's absolute domain) prepended in front of the text in the first 90% of pre-training and uses text only (no metadata) for pre-training in the last 10% of pre-training. They conduct experiments with decoder-only LMs of four scales on three pre-training data and show that MeCo improves the performance on most downstream tasks and obtains an average performance gain of 1.0. They conduct extensive ablations to justify the design choice in MeCo and provides partial explanation for its success.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive review! We are glad that you found our method interesting and our experiment design sound. To answer your questions:\n\n**Q1: Chiang and Lee, 2024 show that providing different source information often does not affect RAG results. How to interpret this?**\n\nA1: Thanks for providing the reference and we’ll include a discussion of this work. We believe the influence of metadata depends on both the metadata type and the model’s training. As noted by Chiang and Lee, publication time significantly impacts RAG performance, while source does not—indicating higher sensitivity to temporal information. However, under MeCo training, models also become sensitive to source metadata as it’s explicitly conditioned on such information—this is evidenced by our conditional inference result.\n\n\n**Q2: Why does MeCo work? Can you release intermediate checkpoints?**\n\nA2: Thanks for raising this point! We included empirical ablations and hypotheses to shed light on the possible inner-workings of MeCo. In Sec 5.2 and Table 5, we showed that using hashed URLs can achieve similar performance as natural URLs, suggesting that the semantic meaning of the URLs is not necessary for better pre-trained models, and MeCo mostly provides signals to group similar documents together. We agree with the reviewer that with the grouping information, the models can either learn to upweight certain “higher-quality” domains (such as Wikipedia) or learn an implicit curriculum that helps accelerate training—but it is still unclear to us how exactly MeCo changes the training, and it warrants further investigation.\n\nWe will also make sure to release all intermediate checkpoints to facilitate future research.", "claims_and_evidence": "The main claim is \"MeCo accelerates LM pre-training\". The paper provides sufficient and clear evidence for this, with LM pre-trained on three pre-training datasets and on four model scales. The average performance improvement is convincing.", "methods_and_evaluation_criteria": "The evaluation is good. Using OLMES, an evaluation suite from Ai2, for evaluation, makes the evaluation convincing and reproducible.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiment design is very sound", "relation_to_broader_scientific_literature": "This paper shows that conditioning on metadata improves pre-training efficiency. While prior works do use metadata or some tags to steer LM generations, using metadata to speed up pre-training has never been explored. This is a novel and important contribution to the community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n===\n- The paper is very well-written. I enjoy how the authors arrange the contents and cross-reference results in the latter part of the paper to support their claims in Section 1 and Section 2.\n- The method, MeCo, is simple, elegant, and effective.\n- The evaluation is reproducible.\n- The related works are properly discussed\n\nWeaknesses\n===\nNone", "other_comments_or_suggestions": "I like this paper a lot", "questions_for_authors": "- Q1. There is a paper [1], not strictly relevant but somewhat related, that shows that LLM's answer does not seem to be affected when changing the metadata of the retrieved documents in RAG. This somewhat contradicts the results shown in this paper, showing that adding proper metadata helps, and adversarial metadata deteriorates the results. It would be interesting to hear the author's comments on this and how MeCo may or may not be better at distinguishing more reliable sources in a retrieval-augmented generation. It will also be beneficial to include this discussion in the paper. However, this is not a requirement for the paper.\n\n- Q2. I am personally very interested in understanding why MeCo is more efficient, and I believe that the research community will also be interested in this. While the paper does provide some explanations, I think they are far from complete, as acknowledged by the authors in Section 5.2. I think studying pre-training dynamics may reveal some interesting observations about MeCo. For example, does MeCo prioritize learning the data from certain domains? Does MeCo's training automatically enable some curriculum for the LM? It would be beneficial if the authors could also release the intermediate checkpoints of these models, as pre-training is computationally infeasible for most research groups.\n\n\n\n- [1] [Do Metadata and Appearance of the Retrieved Webpages Affect LLM’s Reasoning in Retrieval-Augmented Generation?](https://aclanthology.org/2024.blackboxnlp-1.24/) (Chiang & Lee, BlackboxNLP 2024)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kLXkWicz9J", "reviewer": "Reviewer_kHX6", "summary": "The paper presents a novel method named Metadata Conditioning then Cooldown (MeCo), which appends metadata (primarily URLs) to pretraining documents and significantly accelerates pre-training. The authors also show how MeCo can be used for model steering by conditioning prompts on metadata, enhancing both downstream task performance and reducing harmful outputs. This approach is working probably because the model can perform data grouping based on data sources.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions and questions! We appreciate that you recognize the paper’s contributions and strengths. To address your concerns:\n\n**W1: Lack of theoretical explanation—how does MeCo change the training dynamics?**\n\nA1: Thank you for raising this point! We acknowledge that this paper did not provide a theoretical justification for MeCo’s effectiveness or a rigorous analysis of how MeCo changes the training dynamics. However, the main contribution of this paper lies in proposing the method and uncovering this interesting phenomenon (metadata conditioning accelerates pre-training), which is both novel and possesses significant empirical impact. Theoretical understanding of such a method is challenging due to the nature of pre-training, and is also beyond the scope of this empirical study.\n\nThat said, we included empirical ablations and hypotheses to shed light on the possible inner-workings of MeCo. In Sec 5.2 and Table 5, we showed that using hashed URLs can achieve similar performance as natural URLs, suggesting that the semantic meaning of the URLs is not necessary for better pre-trained models, and MeCo mostly provides signals to group similar documents together. We agree with reviewer TsUA that with the grouping information, the models can either learn to upweight certain “higher-quality” domains (such as Wikipedia) or learn an implicit curriculum that helps accelerate training—but it is still unclear to us how exactly MeCo changes the training, and it warrants further investigation.\n\nWe also highlight a recent preprint, [Zhu et al., On the Power of Context-Enhanced Learning in LLMs](https://arxiv.org/pdf/2503.01821). Though their setting is synthetic and differs from ours, their theoretical analysis shows that context-enhanced learning—such as providing metadata at the beginning of the sequence without actually optimizing cross entropy loss on these tokens—can improve sample efficiency. We found the result insightful and encourage the reviewers to check it out as well.\n\n\n**W2: Arbitrary duration choice for cooldown**\n\nA2: We provide an ablation study on the duration of cooldown in Table 14 (Appendix B.3), which demonstrates that 10% cooldown leads to competitive performance.\n\n**W3: Though the authors conducted evaluation on some standard benchmarks, there lacks (1) non-English tasks and (2) evaluation on robustness across more diverse tasks.**\n\nA3: Thanks for raising this point! \nWe evaluate our models by using OLMES ([Gu et al., 2024](https://arxiv.org/abs/2406.08446v1)), the industry-standard evaluation behind AI2’s OLMo models ([Groeneveld et al., 2024](https://arxiv.org/abs/2402.00838v3)). Similar tasks are also used in [Llama](https://arxiv.org/abs/2302.13971) and [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard). While our evaluation does not include some of the latest popular LLM benchmarks on math (GSM8K, MATH), coding (HumanEval), graduate-level QA (GPQA), or instruction following (AlpacaEval, IFEval), we emphasize that at our experiment scale (1B model, 160B tokens), models do not achieve any meaningful performance on math, coding, and graduate-level QA—making these benchmarks uninformative. Additionally, since our study focuses on pre-training instead of post-training, the models are not expected to perform well on instruction-following tasks. Other studies that similarly investigate pre-training (e.g., [Wettig et al., 2024](https://arxiv.org/pdf/2402.09739); [Yu et al., 2024](https://arxiv.org/pdf/2406.06046)) adopt comparable evaluation setups.\n\nThat said, we acknowledge that the evaluation suite we use focuses on English tasks, which is a common limitation in language model pre-training studies.", "claims_and_evidence": "The claims are well supported by experiments. The main claims include:\n1. Accelerated Pre-training.\n2. Improved Downstream.\n3. Model Steerability.\n4. Metadata can have different types.\n5. Minimal extra computational cost.", "methods_and_evaluation_criteria": "The Methods And Evaluation are properly performed, including comparing PPL and downstream tasks scores.", "theoretical_claims": "There are not too much theoretical claims. Most of the claims are emprical and based on experiment results, but I think they make sense intuitively.", "experimental_designs_or_analyses": "The Experimental Designs are straightforward and intuitive. They can support the claims of the paper.", "relation_to_broader_scientific_literature": "The method can greatly accelerates pre-training and enhances model steerability without introducing computational overhead or limited applicability. It can be very helpful to the pretraining domain.", "essential_references_not_discussed": "I don't have comments on this section.", "other_strengths_and_weaknesses": "Stengths:\n1. Extensive evaluation across multiple tasks, datasets, and model sizes.\n2. Insightful ablation studies.\n3. Clear, practical and easy method.\n4. The most interesting thing to me is that using hashed URLs can also do the trick.\n\nWeaknesses:\n1. Lack of Theoretical Explanation: Empirical evidence is provided, but I am not really sure how it changes the trianing dynamics.\n2. The choice of Cooldown Strategy seems effective, but arbitrary. The paper does not thoroughly justify the chosen duration.\n3. Although MeCo was tested across several standard benchmarks and datasets, the paper did not extensively evaluate its generalization to languages beyond English or explicitly measure robustness across more diverse downstream tasks.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z9XWjKzbXa", "reviewer": "Reviewer_HiPa", "summary": "This paper proposes to include metadata (source links) in the pre-training of language models to boost learning efficiency. The proposed method, MeCo, pre-trains language models with text augmented with metadata in the first 90% of data and the metadata are removed in the last 10% of data for “cooldown”. MeCo is benchmarked on various commonsense reasoning datasets to show better performance and is also more steerable to different generation styles conditioned on different metadata.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We address your concerns here:\n\n**W1: Evaluation only includes commonsense reasoning and only reflects a few sources such as Wikipedia.**\n\nA1: **We evaluate our models by using OLMES ([Gu et al., 2024](https://arxiv.org/abs/2406.08446v1)), the industry-standard evaluation behind AI2’s OLMo models ([Groeneveld et al., 2024](https://arxiv.org/abs/2402.00838v3))**. Similar tasks are also used in [Llama](https://arxiv.org/abs/2302.13971) and [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard). While our evaluation does not include some of the latest popular LLM benchmarks on math (GSM8K, MATH), coding (HumanEval), graduate-level QA (GPQA), or instruction following (AlpacaEval, IFEval), we emphasize that at our experiment scale (1B model, 160B tokens), models do not achieve any meaningful performance on math, coding, and graduate-level QA—making these benchmarks uninformative. Additionally, since our study focuses on pre-training instead of post-training, the models are not expected to perform well on instruction-following tasks. Other studies that similarly investigate pre-training (e.g., [Wettig et al., 2024](https://arxiv.org/pdf/2402.09739); [Yu et al., 2024](https://arxiv.org/pdf/2406.06046)) adopt comparable evaluation setups.\n\n**We also stress that OLMES covers more than just “commonsense”**. For example, MMLU benchmarks model knowledge on a diverse set of subjects, including math, medicine, and economics; OpenbookQA and TruthfulQA test models’ factual correctness. This suite of tasks has been used by numerous pre-training studies and industry LLMs, and are considered to cover a diverse range of capabilities and to picture a holistic picture of the model performance. **We respectfully disagree with the claim that the evaluation reflects only a narrow set of sources such as Wikipedia**—for example, websites like personal blog-posts often benefit models’ commonsense performance.\n\n\n**W2:  MeCo only discussed the case where all metadata is available. Most curated raw texts do not contain metadata.**\n\nA2: We respectfully disagree with the reviewer on “most curated raw texts do not contain metadata”. Most pre-training sources, such as CommonCrawl, C4, FineWeb, RefinedWeb, and DCLM, provide at least the URL information. For companies that perform their own data crawling, retaining metadata like URLs is standard practice.\n\n\n**W3: Explanation of the pre-training efficiency**\n\nA3: Thanks for raising this point! We included empirical ablations and hypotheses to shed light on the possible inner-workings of MeCo. In Sec 5.2 and Table 5, we showed that using hashed URLs can achieve similar performance as natural URLs, suggesting that the semantic meaning of the URLs is not necessary for better pre-trained models, and MeCo mostly provides signals to group similar documents together. We agree with reviewer TsUA that with the grouping information, the models can either learn to upweight certain “higher-quality” domains (such as Wikipedia) or learn an implicit curriculum that helps accelerate training—but it is still unclear to us how exactly MeCo changes the training, and it warrants further investigation.\n\nWe also highlight a recent preprint, [Zhu et al., On the Power of Context-Enhanced Learning in LLMs](https://arxiv.org/pdf/2503.01821). Though their setting is synthetic and differs from ours, their theoretical analysis shows that context-enhanced learning—such as providing metadata at the beginning of the sequence without actually optimizing cross entropy loss on these tokens—can improve sample efficiency. We found the result insightful and encourage the reviewers to check it out as well.", "claims_and_evidence": "1. MeCo is claimed to be a more efficient pre-training paradigm. However, the included benchmarks only involve commonsense reasoning, which fails to comprehend all kinds of abilities in LMs.\n2. MeCo is claimed to make LM more steerable, which is well-validated by various ablation studies and harmfulness reduction experiments.", "methods_and_evaluation_criteria": "The benchmarks are mostly commonsense reasoning, which is a bit narrow to support the claim that MeCo is universally more efficient. The biggest concern is there to be various sources for pre-training as shown in Table 15. However, commonsense reasoning can only benchmark a few datasets (e.g. Wikipedia) in the corpus.", "theoretical_claims": "N/A, The paper is more about an empirical claim.", "experimental_designs_or_analyses": "The experiment design is reasonable.", "relation_to_broader_scientific_literature": "This paper is connected to the source for pre-training language models, this paper incorporates metadata into the pre-training to enable better generation steering.", "essential_references_not_discussed": "The related literature is well discussed.", "other_strengths_and_weaknesses": "Another weakness of MeCo is the requirement for metadata to be appended to the raw text. The setup in this paper discusses only the case that metadata are available to all pre-training data. However, most curated raw texts do not contain their metadata. Then, will the steerability observed from pre-training with 90% w/ metadata + 10% w/o metadata still appear in the case that most data are without metadata? I feel the conclusion in this paper might not be applicable to larger-scale pre-training, which requires further discussion.", "other_comments_or_suggestions": "This paper proposes an interesting idea, which is worth further discussion. However, it's not ready for publication because its narrow scope of benchmarking, explanation of the pre-training efficiency, and limitation in the case that all metadata are available.", "questions_for_authors": "It is intuitive that metadata enables more steerable language, and may also make sense for better pre-training efficiency. But it's not intuitive that the resulting model will perform better. The improvement is also only shown in commonsense reasoning. Can you provide some intuition as to why metadata can improve LM's performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7W9WNK1asT", "reviewer": "Reviewer_mTck", "summary": "The paper proposes a metadata-enhanced training strategy for LLMs across various model sizes, ranging from 600M to 8B parameters. Specifically, during the first 90% of training, metadata is prepended to the training documents, enabling comparable performance while reducing data usage by 33%. The authors conduct experiments on three training corpora to validate the effectiveness of their approach.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We address your concerns below:\n\n**Q1: Randomness of the experiment results**\n\nA1: Thank you for raising this point. We acknowledge that a limitation of our study is the lack of multiple runs with different random seeds for most experiments, primarily due to the high cost of pre-training (our main 160B run required 1,536 H100 GPU hours). That said, we would like to emphasize the following:\n\n(1) Single-run experiments are standard in LM pre-training studies, given the resource constraints. Our setup follows established practice, consistent with prior work ([Xie et al., 2023](https://arxiv.org/pdf/2305.10429); [Li et al., 2024](https://arxiv.org/pdf/2406.11794); [Wettig et al., 2024](https://arxiv.org/pdf/2402.09739)). It is also generally accepted that pre-training exhibits less variance than fine-tuning.\n\n(2) As you noted, Table 13 shows low variance of our pre-training experiments (3 different seeds and different subsets of data), particularly when averaging across the full evaluation suite. We believe that the result we present is significant and is not due to randomness. \n\n(3) Figures 3–4 demonstrate consistent gains from MeCo across a range of model sizes and datasets, further supporting that the improvements are not artifacts of randomness but reflect meaningful trends.\n\n\n**Q2: Lack of zero-shot results.**\n\nA2: In this paper, we followed [OLMES](https://arxiv.org/pdf/2406.08446)’s setting, which adopts curated 5-shot examples to reduce evaluation variance. But we agree that adding zero-shot results offers a better picture of model performance. We add a zero-shot evaluation for our main 1.6B, 160B, DCLM experiment as following:\n\n|          | MMLU | ARC-e | ARC-c | CSQA | HSwag | OBQA | PIQA | SIQA | WG   | TruQA | Avg. |\n|----------|------|-------|-------|------|-------|------|------|------|------|--------|------|\n| Standard | 35.1 | 70.7  | 41.4  | 59.5 | 65.3  | 46.6 | 72.9 | 48.9 | 63.8 | 35.6   | 54.0 |\n| MeCo     | 35.5 | 71.0  | 45.4  | 60.6 | 66.2  | 52.4 | 73.0 | 47.3 | 64.9 | 35.8   | 55.2 |\n\nAs we can see, the MeCo model still achieves significantly better performance on zero-shot.\n\n\n**Q3: Lack of theoretical contributions**\n\nA3: We acknowledge that this paper did not provide a theoretical justification for MeCo’s effectiveness or a rigorous analysis of how MeCo changes the training dynamics. However, the main contribution of this paper lies in proposing the method and uncovering this interesting phenomenon (metadata conditioning accelerates pre-training), which is both novel and possesses significant empirical impact. Theoretical understanding of such a method is challenging due to the nature of pre-training, and is also beyond the scope of this empirical study.\n\nThat said, we included empirical ablations and hypotheses to shed light on the possible inner-workings of MeCo. In Sec 5.2 and Table 5, we showed that using hashed URLs can achieve similar performance as natural URLs, suggesting that the semantic meaning of the URLs is not necessary for better pre-trained models, and MeCo mostly provides signals to group similar documents together. We agree with reviewer TsUA that with the grouping information, the models can either learn to upweight certain “higher-quality” domains (such as Wikipedia) or learn an implicit curriculum that helps accelerate training—but it is still unclear to us how exactly MeCo changes the training, and it warrants further investigation.\n\nWe also highlight a recent preprint, [Zhu et al., On the Power of Context-Enhanced Learning in LLMs](https://arxiv.org/pdf/2503.01821). Though their setting is synthetic and differs from ours, their theoretical analysis shows that context-enhanced learning—such as providing metadata at the beginning of the sequence without actually optimizing cross entropy loss on these tokens—can improve sample efficiency. We found the result insightful and encourage the reviewers to check it out as well.", "claims_and_evidence": "1. One of the key claims in this work is that prepending metadata, i.e., URLs, to training documents obtains comparable performance with 33% less training data. As shown on the right-hand side of Figure 1, as the number of training tokens increases from 0 to 80B, 160B, and 240B, the proposed method, Meco, consistently outperforms the baseline. However, this improvement may be influenced by randomness. In other words, even with the same metadata-enhanced training data, different random seeds could impact the average performance curve. Some seeds may lead to rapid performance gains, while others may result in slower improvements. Unfortunately, since this work conducts experiments using only a single seed, the extent to which randomness affects the shape of the average performance curve remains unclear.\n\n2. Additionally, Table 13 only shows the performance of three runs at 160B tokens, it is still unclear how performance change across different training token counts.", "methods_and_evaluation_criteria": "1. The primary evaluation in this work is 5-shot, though a very few tasks are evaluated in a zero-shot manner (In Table 3). However, zero-shot evaluation better assesses a model’s ability to generalize, which could provide a stronger indication of the effectiveness of prepending metadata.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. Experiments are conducted based on a single run, which may weaken the strength of the main claim.\n2. Zero-shot evaluation should be included to better assess the effectiveness of the proposed method.", "relation_to_broader_scientific_literature": "Previous studies have explored metadata such as source domains and document IDs, whereas this work leverages URLs, which may be new to the community.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths: the paper is well-structured and easy to follow.\nWeaknesses: the paper lacks a theoretical contribution and primarily presents an empirical study on the effect of prepending URLs in LLM training.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Could you provide results from runs with different random seeds when gradually increasing the number of training tokens?\n2. Could you include zero-shot performance for your main experiment results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission got one Strong Accept, one Accept, one Weak Accept, and one Weak Reject. After reading the reviews and discussion, most reviewers maintained their scores and sound positive about the submission. The paper proposes MeCo, a method to accelerate LM pre-training. Overall, reviewers think the method is well executed, but lacks theoretical analysis. Nevertheless, I think this method is of interest to the community and I think it can have practical benefits and trigger interest in the community. I would encourage the authors to include a discussion about limitations and additional experiments (especially the zero-shot results) in a final revision.For these reasons, I am recommending acceptance.
Decision: Accept (poster)