Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining
Authors: Dongyang Fan
Diba Hashemi
Sai Praneeth Karimireddy
Martin Jaggi
Venue: ICLR 2026 Poster (2026)
Keywords: LLM pretraining
efficient LLMs
metadata
Primary area: foundation or frontier models, including LLMs

Abstract: Incorporating metadata in Large Language Models (LLMs) pretraining has recently emerged as a promising approach to accelerate training. However prior work highlighted only one useful signal—URLs, leaving open the question of whether other forms of metadata could yield greater benefits. In this study, we investigate a wider range of metadata types and find other types of metadata, such as fine-grained indicators of document quality that can also accelerate pretraining when prepended. We identify a common feature among effective metadata: they encode information at a finer granularity. We further introduce metadata appending as a means of improving training efficiency, where predicting an appropriate metadata as auxiliary task can help speed up pretraining. In addition, learnable meta-tokens trained with masked loss can recover part of the speedup by inducing quality-aware latent structure. Using probing, we analyze latent representations to understand how metadata shapes learning. Together, these results yield practical guidelines for integrating metadata to improve both the efficiency and effectiveness of LLM pretraining.

Published as a conference paper at ICLR 2026
BEYOND URLs: METADATA DIVERSITY AND POSITION FOR EFFICIENT LLM PRETRAINING
Dongyang Fan*", Diba Hashemi* Sai Praneeth Karimireddy MMattt Jaggi" o EPFL, & University of Southern California
Correspondence: (dongyang. fan, diba hhshhmi) @epp.. denotes equal contribution
ABSTRACT
Incorporating metadata in Large Language Models (LLMs) pretraining has recently emerged as a promising approach to accelerate training. However prior work highlighted only useful signal-URLa leaving open the question of whether other forms of metadata could yield greater benefits. In this study, we investigate wider range of metadata types and find other types of metadata, such as fine- grained indicators of document quality that can also accelerate pretraining when prepended. We identify common feature among effective metadata: they encode information at a finer granularity. We further introduce metadata appending as a means of improving training efficiency, where predicting an appropriate metadata as auxiliary task can help speed up pretraining. In addition, learnable meta-tokens trained with masked loss can recover part of the speedup by inducing quality-awarc analyze latent representations to understand
latent structure. Using probing, how metadata shapes learning. Together, these results yield practical guidelines for integrating metadata to improve both the efficiency and effectiveness of LLM pretraining.
INTRODUCTION
Large language models (LLMs) are typically pretrained on web-scale corpora sourced from Common Crawl-style snapshots and related aggregates, then aggressively filtered and deduplicated to improve quality and efficiency. Landmark datasets such as C4 (Raffel et al., 2020) and RefinedWeh (Penedo et al. 2023) exemplify this trend: both begin from massive web crawls and rely on heuristic and statistical filtering to remove boilerplate, low-quality, and near-duplicate content before tokenization; more recent open corpora like Dolma (Soldaini et al. 2024a) extend this paradigm to trillions of tokens spanning diverse sources. In parallel, growing literature studies data filterin/sellcctonn, e.g., perplexity- or importance- importance-based based selection, to allocate pretraining compute toward the most useful subsets, for which we refer the readers to survey (Albalak et al. 2024). Together, these efforts frame pretraining efficiency largely
problem of "which web data to keep and how much of it to use."
complementary axis for improving pretraining efficiency is to enrich the input with document-level metadata, such as information about a text's source, domain, time, or other attributes, so the model can condition its representation learning accordingly. Recent work formalizes this idea as "metadata conditioning". showing that prepending simple, readily available indicators (e.g., source URLs or domain tags) during pretraining can accelerate learning and improve downstream controllability, with "cool-down" phase remove the dependency inference (MeCo, Gao et al. (2025). This direction builds on the broader intuition that structural signals beyond raw tokens (such as links, domains, registers) can shape what is learned during pretraining.
However, the landscape of effective metadata types and positions remains underexplored. To date, empirical evidence most consistently supports prepending the URL (or source identifier) before the document as practical and robust way to accelerate LLM pretraining; systematic comparisons further report that other readily available metadata (e.g.. coarse topics or quality indicators) have not yet shown comparable acceleration under similar budgets and setups (Fan et al., (2025.. Whether richer metadata schemas (beyond URLs) or alternative integration strategies (c.g., suffixing, special-token segment headers, side-channels) yield additional efficiency gains is still largely open question.

---

Published as a conference paper at ICLR 2026
Finally, despite encouraging empirical gains, we have limited mechanistic understanding of how metadata shapes latent representations during pretraining, e.g., whether conditioning induces domain- specific subspaces, affects token- and document-level mutual information, alters cross-domain interference and transfer. Preliminary analyses from metadata-condiiioning hint that such signals reorganize representation geometry and influence downstream behavior, but principled account connecting metadata types, injection positions, and emergent representation structure remains to be developed.
We summarize our contributions as follows:
We identify additional useful metadata signals in accelerating LLM pretraining when prepended, beyond URL. We show that the aene-rrrnllartty of the metadata is the key in bringing the acceleration effect (..c)
We introduce and evaluate metadata appending method for accelerating pretraining, and highlight metadata types that are particularly suitable as auxiliary signals in this setup (Section
We show that learnable meta tokens can partially recover the speedup with metadata, where attention patterns to these tokens encode quality- information. (Section (.4).
We conduct layer-wise probing of latent representations for topic, quality, and authorship. providing mechanistic insight into how these factors are better encoded in the latent space (Section 4.5
2
RELATED WORK
Pretraining efficiency of LLMs. Pretraining efficiency is enhanced through three axes: 1) Better pretraining data. RefinedWeb (Penedo et al. 2023) argues that carefully filtered web-only data can outperform mixed curated corpora, while Web (Penedo et al. (024) scales this idea to -15T tokens with stronger decanting and deduplication at crawl-snapshot seale. Both report improved downstream performance per token, i.c., better data efficiency. In parallel, Dolma (Soldaini et al.) 20245) and RedPajama (Weber et al.) 2024) foreground transparent releases with quality signals, dedup IDs, and fine-gran une-grained metadata enable learned filtering and weighting centtgies-aiminne to squeeze more performance out of each pretraining FLOP. 2) More efficient architectural design. Sparse cetture-ff-Experte architectures (e.g.. Switch Transformer, Fedus et al. (2022) activate only a small subset of parameters per token, yielding large effective capacity at near-constant compute and demonstrating multi-x pre-training speedups scale. Multi-Query Attention (MQA, Shazeer (2019) and Grouped- srouped-Query Attention (GQA, Ainslie et al. (2023)) share or group KV heads to reduce KV cache and bandwidth costs; although often discussed for inference, these changes also lower training-time memory traffic and can improve throughput for long contexts. 3) Additional signals from metadata. Recently, [ao et al. (2025): Fan et al. (2025) have both demonstrated an accelcration effect from URL prepending, saving up to 30-40% tokens in pretraining time. This acceleration considerable, even on top data filtering. In this work, we aim to further advance this axis.
Metadata in LLM pretraining. A growing body of work shows that exposing explicit metadata, e.g., URL/domain, document IDs, and timestamps, can improve efficiency, steerability, and attribution. On the source/domain axis, CTRL conditions on control codes derived from data structure (domain, dates, etc.) to steer generation (Keskar et al. 2019); similarly. Fan et al. (2025) find that prepending topic and format tokens yield stronger controllability than raw URL domains. For provenance and attribution, Source-Awar
Training injects document IDs in a light post-pretraining stage to enable intrinsic citation of pretraining sources (Khalifa et al. 2024). Along the temporal dimension, Zhao et al. (2024) align pretrained LMs to target year using timestamp metadata, and Faro et al. (2025) pretrain experts on time-sliced corpora with routing by query time, boosting time awareness while preserving downstream performance. Finally, line of metadata-as-conter work shows theoretically and empirically that prepending metadata tokens (excluded from the loss) can substantially accelerate pretraining (Allen-Zhr & Li 2024 Gao et al.) 2025) Zhu et al. 2025 Fan et al.) (222). However, the benefits of metadata conditioning is not uniform: effectiveness depends on prompt length and
setting (Higuchi et al. 2025 Fan et al. 2025). We extend this line by examining efficiency gains
2

---

Published as a conference paper at ICLR 2026
<boc>rneccdoto<eeo>>
text
Figure 1: Diagram of our tokenization. Each document begins with a default seatoniinnoo--quueeeee (
) token. For each sequence, metadata wrapped between beginning- offriinn-ff-cooetttttt (<boc>) and end-of-context (<eoc>). Depending the metadata position, the metadata is prepended (illustrated on the left) or appended (illustrated on the right) to the document. a long document split into multiple sequences, metadata is attached to each one. A 10% dropout of metadata is always performed.
from metadata appending and by introducing learnable metatokens flexible alternative to fixed metadata strings.
EXPERIMENTS
We follow the experimental setup of Fan et al. (2025). and extend their investigation to more metadata types. The model is adopted 1.5B Llama model with 16 layers in total. We use AdamW optimizer with regularization strength 0.1 (Loshchilov & Hutter 2019). For each batch, the model processes 2.06 million tokens, corresponding cosine scheduler with a maximum learning rate of 3e-4, and in the first 5% steps we perform the learning rate warm-up. To train the models, we use the Megatron- framework (Narayanan et 2021
sequence length of 4,096 and a batch size of 504. We use
We use FineWeb-Edu (Lozhkov et al., 2024) for comparability with Fan al. (2025). In the metadata- preeettd-preendiiggggg setup, we mask the metadata tokens from the loss both for reporting and during backpropagation. the peetadata-appenddee we retain its loss for backpropagation so the model learns predict the metadata. The tokenization approach for metadata prepending and appending is depicted in Figure
setup, we mask the metadata only for loss reporting:
Evaluation Benchmarks. As standard practice, evaluate models general knowledge un- derstanding using LMM-Evl-Harness developed by Gao et al. (2024). The benchmarks used are Arc-Easy (Clark et al. 2018), Are-Challea Co-CChallenge (Clark et al. 2018). CommonSense QA (CSQA, Talmor et al. 2018), MMLU Hendrycks et al. 2020). PIQA (Bisk et all 2020), Social IQA (SIQA, Sap et al.) Paperno al. 2016) and Winogrande
2019), HellaSwag (HS, Zellers et al. 2019). Lambada (LBD,
(WG, Sakaguchi et al. 2021).
Metadata Types. We examine the following types of metadata. Building on Fan et al. 2025 we extend the quality score and domain information to a finer level granularity. addition, for each metadata type, we explore two positional variants: prepending and appending.
Full URL (URL): Provided by the FineWeb-Edu dataset url, from which the document was crawled.
Quality Score (QS-coarse): Provided by the FineWeb-Edt dataset as These scores are generated by a linear regressor trained on 410,000 web samples, each annotated by Llama-3-70BB Instruct scale from 0 (not educational) to (highly educational), reflecting the sample's 's educational value. The scores are then rounded the closest integers, and only scores greater than equal to 3 are kept. There are categories of coarse-grained quality levels.
Sine-Grained Quality Score (QS-fine): Similar to QS-coarse, take the raw score (provided as score) from the regressor instead of the rounded one. The score is presented as lscore 10]. Thus, is least 10 times finer than QS-coarse.
Coarse-Grainte Domain Information (DI-coarse): Following Fan et al. 2025). ).. we annotate each document with WebOrganizer (Wettig al. 2025), where topic and format domains

---

Published as a conference paper at ICLR 2026
returned by pretrained classifiers. There are 24 categories per taxonomy. In total, have 576 different Domain Information types.
Fine-Grained Domain Information (DI-fine): Each document is labeled with topic and format domains generated by Llama3. 1-88.1--B--B 1-8--Instruutt Instruct model (Grattationi et al. (024)) The generation prompt can seen in Appendix C The generation is open-ended, and there are unlimited number of categories.
Meta Tokens: Five empty different meta tokens newly added to the vocabulary. The usage is detailed in Section 4.4
Probing Experiments. In addition to downstream evaluation benchmarks, we employ probing to analyze the information encoded in layerwise representations. While benchmarks reflect the model's overall knowledge and reasoning ability, they provide limited insight into how metadata affects the internal learning dynamics of different layers. Probing allows us examine these differences more directly and shed light the representational role of metadata in accelerating pretraining.
Specifically, we train probing classifiers on the following tasks:
Document quality prediction. We sample 15,000 documents from FineWeb-Edu. annotated with QS-coarse and balanced across three quality levels (5,000 per score). Using 90/10 train-test split, the objective
predict the quality score from the document representation.
Document topic prediction. We sample 20,000 documents from Fib annotated with DI-coarse topics and balanced across 20 categories (1,000 per topic). Using the same 90/10 train-test split, the goal is to predict the topic label from the document representation.
Authorship prediction. We use the dataset from Stamatatos (017), consisting of 25157 Guardian articles written by 13 authors. We adopt 70/30 train test split. This task is relatively straightforward: later layers achieve 100% test accuracy, while earlier layers still retain some distinguishable signal.
For these tasks, train layer- wise three-layer layer MLP classifiers using representations from each layer. The probe takes the last hidden state of each document as input (truncated at the 100th token); for shorter documents, the final token representation is used instead.
4 HOW CAN METADATA ENHANCE PRETRAINING?
In this long section, present results from thorough investigation of metadata types and positions. Throughout, we explain some unique phenomena by probing through the latent representations.
4.1 PREPENDING: TRAINING SPEED-UP UP WITH FINE-GRAINED METADATA CONDITIONING
On top of the metadata types examined by Fan al. (2025) extend the investigation two other fine-grained metadata types. The left panel of Figure shows how downstream performances progress with prepending different metadata. The final average performance across downstream tasks is reported in Table Among the five context types tested, both URL and fine-grained quality scores yield comparable acceleration, reaching the performance of a 100B-token baseline after only 60B tokens. Fine- -eee-gga domain metadata also provides a boost, surpassing the 100B token baseline with 20B fewer tokens. seems that quality score and domain information can helpful, but fine granularity is the key.
Observation 1. Only fine-grained metadata conditioning has positive effect in speeding up pretraining; conditioning on coarse- -raiined meta- meta--nforration yields no noticeable change.
No additive effect from metadata. We further examine whether the model benefits from prepend- ing two types helpful metadata. Specifically, we provide both the URL and the fine-grained quality score metadata and compare the results in Figure 3 The model shows faster learning during the early stages of training, indicating that the combined metadata helps it acquire information more quickly. However, training progresses and more tokens seen, this advantage diminishes, and
4

---

Published as a conference paper at ICLR 2026
Prepending
Appending
0.48
Nig Pertoreance
Performance
Mandard
-9K-
Di-lie fineine
URL
-en standard
-M- coarse
URL
Diffine
QS-line
208
408
66B
808
1038
208
40B
609
806
1008
Figure 2: Pretraining acceleration is measured using downstream evaluation performance. The dashed horizontal line marks the downstream performance achieved after 100B tokens of standard pretraining. DI denotes domain information, and QS denotes quality score; the "-fine" and "-coarse" suffixes indicate fine-grained and coarse garse-grained variants, respectively. Left: Prepending fine-grained metadata matches the 100B-token standard- standard-pretraining performance using only 60B tokens. Right: Appending fine- mne-grineed domain information and coarse-graines quality score matches the 100B- token baseline using 80B tokens.
Table 1: Evaluation results on the average of downstream tasks. When evaluating prepending
models, we add the prefix <boc> <eoc> each evaluation text. When evaluating appending models and the standard model, we add no prefix texts. For every configuration that outperformed standard training on average, we report results as the mean over three random seeds reduce variance. DI denotes domain information, and QS denotes quality score.
Are-C
Are-E
CSQA
MMLU
PIQA
SIQA
HS
LBD
WG
Avg
standard
33,9
68.6
45.1
3188
71.7
41.5
41.9
30.6
54.9
46.7
<boc><<ec>>
34.0
69.2
43.5
31.9
71.9
41.5
42.5
30.8
55.2
46.7
prepending
URL
34.8
71.7
46.9
32.5
72.3
41.5
42.5
32.4
$5.7
47.7
QS (Coarse)
32.7
69.6
41.0
31.8
72.1
41.6
43.0
32.4
55.2
46.6
QS (Fine)
35.3
70.0
45.4
32.1
72.4
41.6
42.3
32.3
54.5
47.3
DI (Cnarse)
34.0
69.8
42.6
32.1
71.9
40.3
42.0
31.8
55.5
46.7
DI (Fine)
34.7
46.0
32.6
72.8
40.8
42.1
32.1
54.5
47.3
Meta Tokens
35.1
45,0
32.1
72.1
40.1
32.1
55.3
47.1
appending
URL
34.3
70.1
44.6
32.0
72.0
40.7
41.9
31.2
54.9
46.8
QS (Coarse)
34.4
69.0
46.0
32.1
72.6
40.6
42.3
31.1
56.2
47.1
QS (Finc)
34.8
69.8
44.9
32.2
71.8
36.5
42.2
33.0
55.6
46.8
(Coarse)
44.6
31.7
71.9
40.5
41.9
32.3
54.9
46.7
DI (Fine)
34.7
45.3
32.1
71.2
34.4
53.2
47.3
the final performance is comparable to that of a model trained without metadata. Notably, the model does not appear leverage either the URL or the fine- ine-grained quality score consistently.
standard
prepensing
prepending QS-fiee
prepandin QS-fra
challenge/ace
mmlu continuation/aee
pigalace
winogrande/ace
average
0.56
0.48
0.34
0.72
0.46
0.31
0.71
6.30
0.30
0.70
0.52
0.66
208
408
608
508
1008
208
408
60B
808
1008
208
400
608
808
1006
208
408
608
808
1008
208
408
608
808
1008
Figure 3: Comparison of downstream performance when prepending URL fine-grained quality score (QS-Fine individually versus in combination. Both metadata types are effective on their own, but combining them yields no additive effect on metadata acceleration.
Fine-grained metadata prepending performs better than coarse-grained. In Figure and T we observe that the fine-grained metadata types lead to stronger model performance compared to their grained counterparts. We hypothesize that more fine-grained metadata distinctions
5

---

Published as a conference paper at ICLR 2026
improve the model'ss ability capture and represent salient information in the data. To test this hypothesis, we probe document-topic prediction using representations from two models trained with either fine- yoarse-graines metadata prepended, described in Section E As shown in Figure 4 prepending either fine- or coarse-grand caaseegrrined metadata improves performance over the standard model. The fine-gre ene-ggaieed variant yields modest but consistent gain, suggesting that fine-graines metadata prepending helps the model encode topic information more effectively in its latent representations. To verify that our probing procedure is sound, we provide further evidence in Appendix B.4
Prubing Accuricy of OS presended Modsh Decumnnt Pazskten
Poliin Sacuracy of properddc Modes Docament Topic hod
15-
14-
13-
23-
15-
10-
Layer'
Lipe''
Layer
Layarac
Laye1
LEye-3
Figure 4: Probing results on document topic prediction for models prepended with quality scores (QS) and domain information (DI), with fine- and coarse-grained variants. Representations from models trained with finer-grained metadata encode topic information more effectively. When doing inference, only empty metadata is prepended, i.e. <boc><eoc>.
Which parts of URL matter for pretraining? Unlike the other metadata types, URLs con- //en. sist of different information. Taken nttps://enn.ikkipddi.. wwkipedia. wttg//wik//Mteeee example, we categorize https: // URL prefix, en www.iieiiia.... org as URL domain and everything after URL domain as URL suffix. Each component encodes different information, for example, URL domain usually corresponds to format and quality information, and URL suffix usually corresponds to topic information.
as an
To investigate which parts of the URL contribute to the success of URL prepending, we sample 100 documents from Fineweb-Edu conditioned on their URLs and analyze attention patterns across layers. The absolute attention weights given to each part of the prepended components are plotted out in Figure 5 where it is observed that a significant portion of attention is directed toward the URL prefix the part of the URL that carries no content information from the document. This observation highlights common attention behavior: the model often focuses consistent initial tokens, which can act as an "attention sink" without providing a meaningful signal for the task (Gu et al. 2025).
Since the majority of attention is given to the URL prefix part, does the attention pattern indi- cate that prepending the URL prefix should re- cover most of the URL-prepended performance? It turns out to be the case. We conducted three ablation runs with different URL compo- nents prepended, as shown Table Adding only the prefix fails to surpass the standard base- line, even though its training loss trajectory closely follows that of the full-URL. condition (Figure 13). We interpret this faster loss reduc- tion as a copying effect: the suffix tends to act as a brief synopsis of the page and reveals part of the upcoming text, making next-token predic- tion easier. URL domain and suffix, even though given relatively little attention to, are vital for the improvement in downstream tasks. Moreover, neither of them catch the performance of full URL prepended run, suggesting that the domain and suffix encode complementary information. We offer more evidence and analysis in Appendix B.2
0.014
uf srafie ur dommm suffi
boc token
0.02
0.030
0.005
0.006
0.044
10
11
15
14
15
16
Figure 5: Averaged attention weights across all attention heads in different layers. A finer- finer-grained per-layer per-attention head attention pattern is pro- vided Figure 6
URL-prepended model shows a pronounced attention sink on URL prefix, but this does not contribute to the improved performance. In contrast, the URL domain and suffix provide complementary contributions.
Observation 2.

---

Published as a conference paper at ICLR 2026
www Hadd
UPE Mate.Lie Head
Figure 6: Entropy over the attention weight distribution to the different parts in the prepended context, plotted per layer and per attention head. Higher entropy denotes attention more uniformly distributed, while low entropy indicates an attention sink.
Table Downstream evaluation results for prepending various components of URL. Prepending URL domains provides a benefit, but it does not match the gains from prepending full URLs.
Are-C
Arc-E
CSQA
MMLU
PIQA
SIQA
HS
LBD
Avg
34.8
71.7
46.9
32.5
72.3
41.5
42.5
32.4
55.7
47.7
URL prefix
34.0
68.9
43.0
32.1
72.0
41.7
42.0
32.8
53.1
46.6
URL domain
35.3
69.3
46.4
32.3
72.9
39.9
42.3
32.1
54.2
47.2
URL. suffix
34.1
68.1
44.7
31.9
72.1
41.0
42.4
32.0
55.8
46,9
4.2 APPENDING: PREDICTING AUXILIARY INFORMATION MAY HELP
Beyond prepending, metadata can also be inserted at various positions during training. Khalifa et al. (2024) explored placing metadata at the beginning or end of document, as well as repeating it multiple times within the text, for the purpose of data attribution. natural question for improving pretraining performance is whether appending metadata could be beneficial.
Appending turns metadata prediction into an auxiliary task: after processing the entire sequence, the LM is asked to predict information such as the quality score topic of the sequence. This setup may encourage the model to build an internal representation of the input that is sufficiently informative to recover the metadata at the end, potentially incentivizing it to compress salient aspects of the sequence into its hidden states. this way, the auxiliary objective could serve as a form of soft regularization, providing another learning signal for the model weights.
We report the downstream performance for the five metadata types in the right panel Figure 2 with the corresponding averages summarized in Table Among these runs, appending fine- fine-grained domain information helps the most. Additionally, coarse-grained quality score and URL appending also help improve downstream performance. In contrast, inn-gnained quality score does not improve

---

Published as a conference paper at ICLR 2026
standard pretraining. The acceleration effect of helpful metadata is, average, not great as prepending: however, we are still able to train on 20% fewer tokens to achieve the same performance standard pretraining.
Observation 3. Auxiliary tasks such as coarse-grained quality score and fine-grained domain information prediction can accelerate training.
Fine-grained vs. coarse-grained quality score. is noteworthy that appending coarse-grained quality scores (single-digit integers in 3, 4,5) leads better downstream performance than appending fine-grained quality scores (two-digit integers ranging from 25 to 50). In principle, a model trained with fine-grained scores could simply learn predict the first digit and thereby achieve performance comparable to the coarse-grained case. However, it does not appear to do so. To verify this, we evaluated the fine-grained modells s ability to predict the two-digit quality score by prompting it to continue generation after <boc>, and found that its predictions were highly accurate. We hypothesize that the model becomes overly focused on solving this auxiliary prediction task, which detracts from its ability to develop other skills that would improve downstream performance.
To evaluate this hypothesis, we run the probing experiments described in Section Results for predicting document topic and quality appear in Figure 7 In the left panel where the task is unrelated to the included metadata type (quality), the QS-coarse appended model outperforms the QS-find appended model, especially in the early layers. In contrast, in the right panel where the task is to predict quality. the QS-fine model performs slightly better. This pattern suggests the QS- fine model specializzng the quality-predictioo auxiliary task, with corresponding trade-off in other capabilities.
Prob ng Accuracy of 05 aspended Modea en Decument Took Prockt
Prosing Acorray of OS upoendoc Nodak on Docament Qual Trediciio
40
Liger
Lia 22
Logi
L4 1aper
Layer
L4yer
Laye
Figure 7: Probing results of two models that differ only in the granularity of the appended quality- score labels for predicting document topic and quality. The model with fine- ane-ggained quality scores performs better par with the coarse caars--griined model for document quality prediction, but performs worse on document topic prediction.
4.3 DOES ACCELERATION ALSO APPLY TO TRAINING LOSS?
How is the training acceleration reflected in the training curves? In the left panel Figure g we show the training perplexity curves for all runs that outperform the standard pretraining. Overall, there appears to be no clear correlation between downstream performance and training loss. The only notable exception is URL prepending, which leads visibly faster decrease in loss, consistent with the findings of Fan et al. (2025). We attribute the faster loss reduction to a copying effect: URLs s offer shortcut for next-token prediction by enabling the model simply copy tokens from the URL
On top of this, we plotted out the gradient norm change throughout the whole training. Compared to the other runs that improve downstream performance, the standard pretraining run exhibits higher loss spikes. This suggests that including metadata may help stabilize LLM pretraining.
4.4 CAN LMS LEARN TO ACQUIRE META INFORMATION?
It is unsurprising that LMs leverage additional metadata infer latent clusters. A natural question whether LMs can also infer such metadata on their own. To investigate this, we introduce five new meta tokens, <s1> through <s5>, These tokens do not exist in the original vocabulary and
8

---

Published as a conference paper at ICLR 2026
standard
standard
Di-fiee prepend
14
prepend
URL prepend
fine
QS-fine prepend
Di-ine append
Di-fine preperd
Porpleatty
Di-ine append
12-
10.0
7.5-
5.0
2.5
0.0
0.9
1.
0.0
1.0
1433
Jetl
Figure 8: Training loss and gradient norm throughout the whole training.
prepended to each sequence with probability 0.9, enclosed by <boc> and <eoc>, As all the prepending runs, we mask out the meta token loss for backpropagation.
With the five learnable meta tokens, we observe acceleration effect as well, shown in the left panel Figure 2 What information do the meta tokens encode? As the five meta tokens are always the same, hypothesize that is the attention to the five tokens that encodes useful information. To study this, we gathered synthetic documents of three different quality levels, and plotted out the average attention weights given the five tokens for each quality level in Figure 9 Compared the other low- and medium medium-quality documents, quality documents attend significantly less to <s4>,
Average attention meta tokens
Moan
0.0150
0.0125
medium
low
high
0,07
408
0.0100
0.0075
0.05
0.0050
0.0025
0.3
0.0000
meeduum
<34>
Mota
Figure 9: Attention pattern to the five prepended meta tokens (last layer). Left: average attention weights to each the meta tokens; Right: inter- and intra-category attention pattern distance. Each category is quality level and the distance is calculated via euclidean distance of stacked attention weights from the first 100 tokens of a document to meta tokens.
We further flatten the attention weights from the first 100 tokens of each document to meta tokens and compute the average Euclidean distances both within and across quality clusters. The results in Figure [9] show that inter- inter-cluster distances are consistently larger than intra-cluster distances, indicating that documents of different quality levels exhibit distinct attention patterns. In contrast, similar experiments based topic and format did not reveal such a clear separation in attention patterns across clusters, see Appendix B.I
Observation 4. LLMs can learn to encode quality-aware latent cluster information in Icarnable tokens that do not inherently carry any semantic meaning.
4.5 LATENT REPRESENTATION SHAPING
To better understand the differences in learned latent representations, we probe over three different tasks: writing style (approximated by authorship), document topic, and document quality. We present the results from an intermediate layer, as intermediate layers strike a balance between preserving signal and avoiding over-specialization (Skean al. 2025)

---

Published as a conference paper at ICLR 2026
Probing Accuracy Layer for Various Tosks
10-
20
15
Sandard
$0.0
stepens
57.5
sppane 05-aarse-pordd
25.0
52.5
56
Figure Probing accuracy across different model checkpoints on three tasks. each configuration, we conducted 3 seed runs, and the standard deviation is reflected in the red bars.
For authorship, models enhanced with URL and DI-Fine metadata achieve the highest performance, suggesting that both types of metadata contribute to capturing writing style. Across all three tasks, the standard pretrained model shows the lowest probing accuracy. indicating a limited latent understand- ing of these higher-level concepts, shown in Figure TO For quality, the most effective metadata are URL and QS-Fine, reinforcing the idea that URL encodes information about document quality. In the case of document topics, the top-performing models QS-Fine prepend, URL-append, and DI-Fine append, though no consistent pattern emerges across these results. We argue that the task is very challenging and the signal may be harder to observe.
Observation 5. Including the URL as metadata (whether prepended or appended), enhances the model's latent grasp of writing style and overall document quality.
CONCLUSION
In this work, show that LLM pretraining can be accelerated by conditioning wider range of metadata, extending beyond the commonly used URL. Our experiments indicate that une-ggrined metadata, such as fine-grained quality scores and domain information, consistently yields greater improvements than coarse-graines alternatives when prepended. Appending metadata as auxiliary prediction tasks can accelerate training as well. We witness the biggest acceleration when fine-grained domain information is appended.
Overall, our findings highlight metadata versatile and underexplored lever for improving the effi- ciency and quality of LLM pretraining. An open question remains whether metadata can also enhance post-training. While we made initial attempts to explore how metadata shapes representations and gained some mechanistic insights into what aspects are improved, we still lack a clear understanding of why metadata is effective. hope this work motivates the community to investigate these directions further.
Acknowledgment. This work was supported as part of the Swiss Al Initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID a06 on Alps. We acknowledge funding from SNSF Grant number 10005248.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Joshua Ainslie, James Lee-Thoreeee Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://apenreview net/ forum? 1d=hhm0wOW222
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language models, 2024. /tRt::ts:://aat wttpv//aavvvvv//a///a//e.. www/abs///aooooooooo 2402.16882
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws, 2024. h wwwppxwwwwwoga//////////
05405
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 2439, 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1eXXiv:8033.500 880.00557, 2018.
Dongyang Fan, Vinko Sabolčec, and Martin Jaggi. URLs help, topics guide: Understanding metadata utility in LLM training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL URL wwt.tttt/a///.m wwwppppwwwwwwwwweeeeeeeeeeeeeeee net
Ncx
Robin Faro, Dongyang Fan, Tamar Alphaidze, and Martin Jaggi. Timoe: Time-aware mixture of language experts, 2025. URL htttts:///axxii///
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 120):1199, 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman. Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. A framework for few-shot language model evaluation, 07 2024. Utt/htts::/ /ttp:/::////ooo..oo o wwwgg//eoorddeeeeeeeeee
Tianyu Gao, Alexander Wettig, Luxi He, Yihe Dong, Sadhika Malladi, and Danqi Chen. Metadata conditioning accelerates language model pre-training, 2025. URL. https: stttps://arxiv.org
01956
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino. Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee. Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe. Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley
11

---

Published as a conference paper at ICLR 2026
Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang. Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishman, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong. Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou. Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li. Yuming Mao. Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpemn Grant Herman, Grigory Sizov, Guangyi. Zhang, Guna Lakshminarayanan. Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li. Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina. Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu,
12

---

Published as a conference paper at ICLR 2026
Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyighe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk. Suraj Subramanian, Sy Choudhury. Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kochler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Ilama 3 herd of models, 2024. WRL.https:// http://twwxw.p.gaoooo//ooooo
21783
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view, 2025. URL
https://arxiy: org/ abs 2410. 10781
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint wrXiv:2009.03300, 2020.
Rei Higuchi, Ryotaro Kawata, Naoki Nishikawa, Kazusato Oko, Shoichiro Yamaguchi, Sosuke Kobayashi, Seiya Tokui, Kohei Hayashi, Daisuke Okanohara, and Taiji Suzuki. When does metadata conditioning (not) work for language model pre- training? study with context-free grammars, 2025.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation, 2019. URL https // wttp://.w.o//bbb/99.coo 05858
Muhammad Khalifa, David Wadden, Emma Strubell, Honglak Lee, Lu Wang, Iz Beltagy, and Hao Peng. Source-aware training enables knowledge attribution in language models, 2024. URL
https://arxiv: org/ aacp///b//aaa 2404. 01019
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https 05101
//axxiv.rrg/abs//
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Finewel-edu: the finest collection of educational content, 2024. URL htttps::/hugginggaaatt..o caaaaaaaaaaeeeeeeeeeoaeae
MeiogingFaceEVee http://iiwwwwbbieeeeee edu.
Deepak Narayanan, Mohammad Shocybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vi- jay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-Im, 2021. /tt.httss:/axxivvooom
aww.///s....///o/////../aobatt.a 04473
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint (00i::660.0000 2016.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only, 2023.
wttp:o/g/abbb/aoooeee 01116
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data scale, 2024. URL https: :ttppps///rrae http://xwi...or/////////000 17557
13

---

Published as a conference paper at ICLR 2026
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei 7hang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: arxiv. org/ abs 412.151115
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
h
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Roman LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:/904..00 09728, 2019.
Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019. URL https wtwpv/..rg//abs/111.002110
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. In Forry- second International Conference on Machine Learning, 2025, URL lattps://opprreeiise
feet/forumidd-Pe
TX
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillioon tokens for language model pretraining research, 2024a. URL /ttts:ttt://// arxiv. org/ abs/
/ 2 02.02.2 00159
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Sehwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz. Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024b. BtLhttt:: / wrw.a.iiiivv...vi org/ abs/ 2402 00159
Efstathios Stamatatos. Authorship attribution using text distortion. In Proc, of the 15th Conf. of the European Chapter of the Association for Computational Linguistics, volume 1, pp. 1138-1149, 2017.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937 2018.
Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, al. Redpajama: an open dataset for training large language models, 2024. URL https://arxiv. org/abs/2411.12277 25, 2024.
Alexander Wettig, Kyle Lo, Sewon Min, Hannanch Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation, 2025. URL https
/ a tariiv.... aa/aabbs/222.. 10341
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 1905.07800 2019.
14

---

Published as a conference paper at ICLR 2026
Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannanch Hajishirzi, and Noah A. Smith. Set the clock: Temporal alignment of pretrained language models, 2024.
Xingyu Zhu, Abhishek Panigrahi, and Sanjeev Arora. On the power of context-enhanced learning in Ilms, 2025. URL https: / //wxxvv.avvv//////////eaa/aaaavv org/ abs abaaabbs//23/ 01821
15

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
We used LLMs to polish writing well as modify plotting scripts, Furthermore, we utilized LLMs for synthetic document generation Section 4.4
B MORE EXPERIMENTAL RESULTS
B. ATTENTION PATTERN TOPIC AND FORMAT CLUSTERS
Additional experiments to Section 4.4 The attention pattern to meta tokens does encode topic/for-
mat information, as we do not observe more similar attention patterns within one cluster in Figure and Figure 12 respectively.
Average atvention meta tokens
0.0150
Academic Wriiin
Mean palnwise
Knowledge Article
0.0125
3308
Structured Data
3300
User Review
Studsind
0.0050
Ioey Reddm
208
0.0025
Assdermoc Wininng
Fremdeade Artil
Spee AM
Sencttonal Dica
0.0000
<S1>
<52>
< Meta tokens
<
<55
Figure 11: Attention pattern to the five prepended meta tokens (last layer). The documents are clustered by different topics. Left: average attention weights to each of the meta tokens; right: inter- and intra-category attention pattern distance.
Average attention tekens
Mean painwise distances
0.014
Games
Gamees
631
651
History
0.16
0.010
Utterature
Health
0.008
0.004
0.002
Ltordtuee
623
023
517
0.12
Ganbel
Healt
Hauey
<
Figure 12: Attention pattern the five prepended meta tokens (last layer). The documents clustered by 4 different formats. Left: average attention weights to each of the meta tokens; right: inter- and nntroccaegorry attention pattern distance.
B.2 FURTHER EXPERIMENTS ON UNDERSTANDING DIFFERENT URL PARTS
We extend the analysis from Figure 5 to all heads by aggregating attention into five categories: URL prefix, domain, suffix, boc, and eoc. We then normalize these values to obtain probability distribution and compute the entropy. Low entropy values indicate concentrated distribution that approaches one-hot vector, signaling significant attention sink for the URL, prefix. These patterns are illustrated in Figure
We also show the training loss curves for models prepending different portions of the URL in Figure [3 Notably, adding only the URL suffix yields a drop in training loss nearly as fast as when the full URL is prepended. We attribute this to a copying effect: the model can easily repeat tokens from the URL suffix, which often functions as a concise summary. However, relying solely on the suffix does not achieve the same strong downstream performance using the full URL (as shown in Table 2). indicating that the domain and other upstream components of the URL provide additional useful signals.
16

---

Published as a conference paper at ICLR 2026
15
standard
14.
Full URL
URL prefix
13.
URL domain
URL suffix
12
1.0
Trained Tokens
Figure 13: Training loss curves across different models with URL parts pre-conditioning. Both full URL and URL suffix enable faster loss decrease.
B.3 No ADDITIVE EFFECT FROM PREPENDING AND APPENDING
Given that multiple prepended metadata types offer cumulative benefit, we tested hybrid approach: prepending and appending two different and helpful metadata types. We experimented with this by prepending URL and appending QS- QS-coarse, which both outperform standard pretraining individually. However, as shown in Figure 14 pairing both these methods fails to surpass the performance of using prepended URL by itself.
standard <do
prepensing URL
+dn propending URL appending Q5-cearse
aspensing QS
arc mrc_challeng//ace
piga/ace
winagrande/acc
average
0.32
0.72
0.46
0.54
0.30
0.52
208 100
606
926
1006
206
<cB воB
1008
208 408 608
808
1008
20B
408
626
808
1908
20B
408
50B
208
1828
Figure 14: There is no additive effect for URL prepending and QS-coarse appending.
B.4 PROBING ACCURACY FOR OTHER MODELS
To verify that our probing procedure is sound, the Qwen2.5 model family (Qwen et al. 2025). The results are presented Figure [5] For a model with comparable size but greater capability than ours, the probes achieve higher accuracy. Increasing the model scale further, from 1.5B to 7B, yields an additional boost in probing performance.
sanity check latent representations from
Probing Accurady on Documen: Top Prediction
Prob Accurocy on Decument Prodiction
Figure 15: Comparison of probing accuracy for our standard, and URL-prepended model with Qwen 2.5 models for document topic and quality prediction models.
C PROMPT FOR DI-FINE GENERATION
We used the Llama3. 1--8B--nnsssrctt model to annotate the documents for DI-fine models pretraining. The following prompt used for the generation.
17

---

Published as a conference paper at ICLR 2026
Based the given sampled snippet from document (it could be webpage, book, codebase, paper, or anything else), write two concise keyphrases that together capture the document' domain:
TOPIC (less than 3 words) the main subject matter.
FORMAT (less than words) the document's s genre or source type.
Examples of valid outputs include: quantum physics, research paper
healthy cocking, personal blog
video games, forum thread
*** Start the snippet
Msnippet\/
End of the snippet
Now output only the two keyphrases in the exact form:
<TOPIC>, <FORMAT>
18


Reviews:
[{"review_id": "7DvLMUyEXz", "reviewer": "Reviewer_QQyK", "summary": "This paper extends the prior work on using URLs to accelerate pre-training to add additional metadata information. The experiments are with a 1.5B LLaMA model on FineWeb-Edu (which I think the authors should amend the abstract to mention upfront). They also study whether it's better to append or prepend the metadata. Both provide acceleration but it seems like appending might be better for building a more rich representation space. They abstract away concrete metadata entirely by providing learnable meta-tokens that have no semantic meaning but encode quality-related structures in the attention patterns.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The space of data augmentation via metadata is underexplored and quite promising for accelerating pre-training with negligible extra computational cost. \n2. It is interesting to see more interpretability analyses on what metadata does in the model. The connection to attention sink is especially interesting.\n3. Experiments and ablations are run well and conducted thoroughly. The paper is written well and easy to understand.", "weaknesses": "There is a lot of speculation around what the metadata does and it does not have clear grounding in empirical results. First, the optimization speedup is hard to understand and isn't described quantitatively. Figure 7 provides little insight into it -- I am especially unsure what to take away from the gradient norm, given the lack of other information (gradient moment estimates, update norms, etc). \n\nThere are also not enough hyperparameter ablations (I know they are expensive) to draw clean conclusions about the benefit of metadata wrt optimization. For example, Section 4.2 speculates about a soft regularization but it is hard to understand what that truly corresponds to -- for example, is it an optimization or generalization benefit? Both? The text is too vague on this point and I think there are not enough experiments to make claims of this type.\n\nThe interpretability studies are interesting but done too coarsely (eg averaged across all heads and layers). The authors may want to expand the appendix to detail more information for this to be a useful interpretability study to others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DfqSwz0yQo", "reviewer": "Reviewer_1P2e", "summary": "This paper is a thorough study on the effects of conditioning *or* predicting metadata for pre-training documents. By means a number of controlled pre-training runs, the authors study the effects of different types of metadata and contrast conditioning and predicting. The main finding is that more granular data leads to more benefits. The paper proceeds to consider attention sinks and probing of internal states to understand the representational benefits of metadata learning.", "questions": "Do you think the attention sink effect of metadata conditioning is an important benefit of the technique?\n\nCan metadata conditioning and metadata prediction be combined to yield complementary benefits (for two different types of metadata information)?\n\nWhat would be the recommendation of the paper with regards to best practices for metadata conditioned pre-training?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper presents a comprehensive analysis around a relatively understudied type of pre-training technique, metadata conditioning.\n* The paper introduces a new technique, metadata prediction, and finds that it also provides benefits.\n* The pre-training scale of the paper (1.5B runs with up to 100B tokens) is quite extensive.\n* The paper takes a first step to build a more mechanistic understanding of the benefits of metadata conditioning by probing hidden representations.", "weaknesses": "* While the paper adds more evidence to Observation 1 (need for fine-grained granularity), the hypothesis was already formed and supported by some ablations by Gao et al., Metadata Conditioning Accelerates Language Model Pre-training.\n* The paper should attempt to quantify the variance in the pre-training results and evaluations. I am little skeptical that the takeaways are all statistically significant. Specifically, the results in Figure 3 are worrying, since information-theoretically, prepending two types of information should yield similar benefits. (Unless it leads to substantially fewer \"predicted\" tokens during training)\n* The insights in Observations 2, 3, and 4 are interesting, but rather specific and anecdotal, such that the wider relevance and applicability of these insights is not clear to me. The paper also makes limited progress in my opinion towards a foundational explanation of why metadata conditioning leads to improved pre-training results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ePjhNBqjF4", "reviewer": "Reviewer_9nY6", "summary": "This paper presents a systematic empirical study of how different metadata types (URL, quality, domain) influence LLM pretraining.\n\n\nThrough controlled pretraining runs and a suite of probing and attention analyses, the paper finds that fine-grained metadata consistently accelerates learning and shapes latent representations, while coarse metadata contributes little.\n\nNotably, URL metadata enhances stylistic and quality-related features but also introduces attention-sink behavior.\n\nThe work further explores “learnable meta tokens” as a self-organizing latent conditioning mechanism.\n\nThe study is systematic, clearly written, and practically relevant to LLM data curation pipelines.", "questions": "1. Lack of quantitative definition of metadata informativeness\n    - The paper repeatedly claims that fine-grained metadata is more beneficial than coarse-grained metadata, yet this distinction is treated qualitatively. There is no quantitative measure of metadata informativeness (e.g., entropy, number of distinct buckets, mutual information with the text, or token-level perplexity gain).\nWithout defining an “information budget,” it is difficult to generalize the conclusion or to predict when a given metadata type will be helpful. A systematic analysis relating metadata information content to training acceleration would make the claims much stronger.\n\n2. Over-smoothed attention analysis\n    - The attention analysis reports only average attention weights aggregated across all heads and layers. Such averaging may mask the presence of a few specialized heads that truly focus on useful metadata components (e.g., URL domain), while most others attend to superficial tokens like the prefix.\nA head-wise or layer-wise breakdown would clarify whether meaningful metadata utilization emerges in specific submodules rather than being uniformly weak across the network. As it stands, the conclusion that “prefix attention is a sink and unhelpful” may be an artifact of excessive averaging.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides a broad and well-controlled comparison across five metadata types and multiple usage paradigms (prepending, appending, prediction).\n\n2. Demonstrates that fine-grained metadata yields measurable pretraining efficiency gains, informing real-world LLM data curation.\n\n3.  Includes diverse diagnostic views: loss curves, gradient stability, attention visualization, and probing of latent representations.\n\n4. Writing and figures are clear; each section presents concrete “observations” that summarize key findings.", "weaknesses": "1. Limited robustness under real web-scale noise.\n\n    - The paper’s findings rely on moderately curated corpora (e.g., FineWeb-Edu), where metadata fields are clean and semantically aligned with the text.\n    - On truly raw web data, where a large fraction of pages contain boilerplate, encoding errors, or misaligned metadata, the assumed correlation between metadata and content weakens.\n\n2. The learnable meta-token experiment is under-specified and likely reflects optimization or statistical effects rather than true semantic abstraction.\n    - Since the tokens are inserted randomly and unsupervised, any observed clustering by “quality” could simply result from correlations between quality scores and superficial statistics such as document length or domain frequency, rather than genuine latent metadata inference.\n\n3. Potential artifact in attention-sink analysis.\n    - The paper attributes performance differences to an “attention sink” on URL prefixes, but the phenomenon is likely positional rather than semantic.\nBecause all metadata are prepended, prefix tokens naturally dominate early-layer attention regardless of content.\nAn append or randomized-position control would likely remove this effect. Without such controls, the claim remains unsubstantiated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ot4Ee842zJ", "reviewer": "Reviewer_7BEe", "summary": "- It is widely believed that prepending metadata helps LLMs learn latent cluster structure during pre-training; however, evidence beyond URL tags has been limited.\n- The authors systematically investigate which metadata works best and find that fine granularity is key to accelerating LLM pre-training.\n- Specifically, they compare several kinds of metadata (URL; coarse- vs. fine-grained quality scores and domain labels) and contrast prepending with appending.\n- They show that fine-grained metadata yields larger gains than coarse-grained metadata, and that appending can also accelerate training.\n- To probe the mechanism, they further analyze URL metadata: they observe an “attention sink” toward the URL prefix, yet the literal prefix content itself is not important.\n\nNote: I used ChatGPT for minor language editing and phrasing assistance; all technical assessments are my own.", "questions": "- Isn’t it trivial to prefer fine-grained metadata? Did you consider hypotheses under which (i) granularity does not matter, or (ii) coarse metadata might be preferable for some reason?\n- It appears that [1] (which you cite in line 106) also compares metadata at different granularities—they state, “We compare the results by varying the depth of prepended metadata…” in Section 3. Do you see substantive similarities between your results and theirs?\n- Can we expect that using both \"prepending\" and \"appending\" in the same training sequence would further boost pre-training?\n\n[1] Higuchi, R., Kawata, R., Nishikawa, N., Oko, K., Yamaguchi, S., Kobayashi, S., ... & Suzuki, T. (2025). *When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars.* arXiv:2504.17562.\n\n\nNote: I used ChatGPT for minor language editing and phrasing assistance; all technical assessments are my own.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- They compare metadata across five configurations (varying granularity and including non-URL metadata) and evaluate both prepending and appending strategies.\n- Their experimental analysis is multifaceted:\n  - (i) They evaluate a broad suite of benchmarks;\n  - (ii) They explore combinations of different metadata types;\n  - (iii) They measure probing accuracy for latent cluster prediction;\n  - (iv) They analyze attention scores and distances between attention patterns;\n  - (v) They report perplexity and gradient norms.\n- It is also interesting that QS-coarse model outperforms QS-fine model when the task is irrelevant to metadata. \n\n\nNote: I used ChatGPT for minor language editing and phrasing assistance; all technical assessments are my own.", "weaknesses": "- The motivation for studying metadata granularity could be clarified further; otherwise, it risks seeming trivial to prefer fine-grained metadata whenever available. (See the question section.)\n- The discussion in lines 347–353 may already address this concern; if so, please make this explicit—e.g., by emphasizing the key argument—and, if possible, provide any additional supporting rationale.\n\nNote: I used ChatGPT for minor language editing and phrasing assistance; all technical assessments are my own.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)