URLs Help, Topics Guide: Understanding Metadata Utility in LLM Training
Authors: Dongyang Fan
Vinko Sabolčec
Martin Jaggi
Venue: NeurIPS 2025 poster ()
Keywords: Context-aware pretraining
LLM-pretraining
context guidance
controllable generation
Primary area: deep_learning

Abstract: Large Language Models (LLMs) are commonly pretrained on vast corpora of text without utilizing contextual metadata such as source, quality, or topic, leading to a context-free learning paradigm. While recent studies suggest that adding metadata like URL information as context (i.e., auxiliary inputs not used in the loss calculation) can improve training efficiency and downstream performance, they offer limited understanding of which types of metadata are truly effective and under what conditions. In this work, we conduct a systematic evaluation and  find that not all metadata types contribute equally. Only URL context speeds up training, whereas quality scores and topic/format domain information offer no clear benefit. Furthermore, the improved downstream performances of URL conditioning emerge only when longer prompts are used at inference time. In addition, we demonstrate that context-aware pretraining enables more controllable generation than context-free pretraining, in a classifier-free guidance fashion. Although topic and format metadata do not accelerate training, they are effective for steering outputs, offering human-interpretable control over generation.

URLs Help, Topics Guide: Understanding Metadata Utility in LLM Training
Dongyang Fan
Vinko Sabolčec
Martin Jaggi
EPFL, Switzerland firstname www.sllaeeatlal..ommmmm ch
Abstract
Large Language Models (LLMs) are commonly pretrained on vast corpora of text without utilizzig contextual metadata such as source, quality, or topic, leading to context-free learning paradigm. While recent studies suggest that adding metadata like URL information context (i.e., auxiliary inputs not used in the loss calculation) can improve training efficiency and downstream performance, they offer limited understanding of which types of metadata are truly effective and under what conditions. In this work, we conduct systematic evaluation and find that not all metadata types contribute equally. Only URL context speeds up training, whereas quality scores and topic/format domain information offer no clear benefit. Furthermore, the improved downstream performances of URL conditioning emerge only when longer prompts are used at inference time. In addition, we demonstrate that context-aware pretraining enables free pretraining, in a classifier-free guidance fashion. Although topic and format metadata do not accelerate training. they are effective for steering outputs, offering human-ine crimann-nterrretable control over generation.
controllable generation than context-
Introduction
Large Language Models (LLMs) have become increasingly integrated into a wide range of real-worls applications, prompting both researchers and practitioners to explore optimal strategies for their development. A critical component of this process is the pretraining phase, which serves as the foundational stage which models acquire general-purpose linguistic and world knowledge from vast corpora of text. Even with simple next-token prediction loss, the scale of data and model size enables emergent generalization of knowledge I
Conventional LLM pretraining is typically context-free, in the sense that training data consists solely of raw textual inputs, without the inclusion of any associated metadata or external contextual signals. Meta information, such as document source, author identity, timestamps, or topic tags algs-ahthough potentially valuable- are systematically discarded during preprocessing. As result, the model learns to predict the next token based on the intrinsic structure and statistical patterns present within the text itself, without leveraging any auxiliary semantic or structural cues. natural research question arises: Can the pretraining be improved if contextual information is included?
While prior studies 2 suggest that incorporating URLs context (auxiliary inputs not used in the loss calculation) during training can accelerate pretraining and enhance downstream performance, recent findings by Higuchi et [ caution that this is not always the case. Existing work lacks a comprehensive understanding of which types of metadata are most impactful and under what conditions they offer the greatest benefit. Our study aims to fill this gap, providing empirical evidence based on real-world experiments.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Fundamentally, context-aware pretraining endowed LMs with the ability to perform conditional autoregressive generation. This conditional generation capability, developed through exposure to metadata, can potentially be leveraged to steer the model's behavior using a classifier-frees guidance approach [S i.e., the distinction between context- condxx-condiiioone and context-free generation can be further amplified to highlight the influence of contextual information. We aim to investigate whether context-aware pretraining can be utilized for more controllable generation than context-free pretraining.
In response to the previously raised questions, our findings and contributions can be summarized as follows:
We conduct a comprehensive study on context-conditioned LLM pretraining, highlighting its potential benefits and limitations at both pretraining and inference stages (Section B
We examine different types of metadata and find that only URL context can speed up pretraining (Section 4.1.T). When comes to downstream evaluation, URL-conditioned pretraining only enhances the performance with longer prompts (Section 4.1.2).
We demonstrate that the veetadata-codditinn standard pretrained model. While topic and format domain information does not accelerate pretraining, it offers effective and human-intenn interpretablle steering (Section 4.2
pretrained model is more steerable than the
2
Related Work
Metadata for LLM pretraining. Allen-Zhu and Li [2 demonstrate through controlled synthetic experiments that adding special token to high-credibility data be beneficial. They further hypothesize that appending domain names (c.g., wikipedia. org) to each document in the pretraining corpus would yield similar gains. This hypothesis is experimentally validated by Gao al. (3], who report a 33% speedup in pretraining simply by prepending URL domains. However, to ensure the model remains effective on standard, domain-free text, they introduce "cool-down" phase using non-contextual data. Following this, Zhu et al. [6] formally proves that context-enhanced learning can be exponentially more sumple-efficient when the model is capable of in-context learning. that is, can effectively utilize contextual information presented during inference to perform tasks. More recently, Higuchi et al. caution that metadata conditioning not universally effective. Buidling up on the experimental setup of (2) they found that metadata conditioning only helps when the input prompt is sufficiently long reveal latent semantics. With shorter prompts, it can even degrade performance. Most prior studies conduct experiments solely synthetic datasets, where the metadata consists of the datasets' production rules. However, in real- world scenarios, metadata cannot fully capture the complexity of natural language. Therefore, we aim explore the extent to which such metadata can actually be helpful.
Guidance in LLMs. In LLMs, next-token generation is essentially a sampling process from probability distribution over the entire vocabulary. This makes it possible to steer the generation by modifying the token probability distribution. Liu et al. (7] guide the generation of base LLM by leveraging the difference between the outputs of an expert model and an anti-expert model. Li al. g adjust the sampling distribution by down-weighting tokens favored by an amateur model. used in text-to-
Sanchez et al. [9] introduced the idea of classifier-free guidance image generation text-only generation. Building on this, He et al. [IO] applied CFG to steer LLMs for personalization purposes, showing potential for controllable personalization. However, all of these approaches rely on context-frr -ntextt-free pretrained LLMs. In this work, we investigate can context-conditioned pretrained LLMs enable more effective context steering.
3 Method
3.1
Context-Conditionn Pretraining
To properly organize the contextual information, introduce two new tokens: < conteet, [context_beginl>>>>> and <| context, end I The meta information inserted between the two tokens, and prepended
for illustration purposes, <boc> and <eoc> to represent these two tokens

---

KE>
<bockotx
In 1492, Columbus saled across the Atlantic Ocean
<S>
<bockotx2<e0c>
Phoxosynt is the
<000>ct22-00>>
process by wh ch plonta make food using sunlight, water, corbon dioxide, producing exygen os procuct
<000>ct2xe00>
byproduct other organiams.
<boxkeecc>
tensd shoelaces before atarting the
DooesctxV<eoos Script is versatile language used for web development. especially interactive user interfocces online
Figure An example of our context-aware tokenization. Each document begins with default -ot-iiinn-------unnceee (
) token. For each sequence, context segment wrapped in beginning- of-context (<boc>) and end-of-context (<eoc>) is inserted after 
 and before the main text. If document is too long and split into multiple sequences, the context is prepended to each one. Although the context is added every sequence, may be empty. In 90% of the corpus, we include non-empty context; in the remaining 10%, the context is left empty. Contexts can be URL, quality score, or topic/format domains depending on user choices.
before the document contents. During training time, we mask out the loss calculation for the prepended contexts, so the loss is solely calculated over the standard texts, making it comparable to standard context-free pretraining. To avoid confusion, we use context to refer to the auxiliary meta information not used in the loss calculation during training time, and use prompls for the texts given to the pretrained model for generation.
Context is introduced at the sequence level and specific to each document. To ensure the model remains capa- ble of handling context-free text at inference time, we uniformly inter- leave pontext-prepended and context- free documents during pretraining, us- ing a 90%:10% ratio. The context enhanced corpus is illustrated in Fig- ure [ where one document can be split into several sequences and con- texts are prepended to each of them. Unlike the McCo approach (3), which applies 90% context-prepp pnneextpreeended train- ing followed by 10% context-free intermediate checkpoint to be immediately used for context-free inference.
Contact-Fre Free
90%
Conder-ondtrroll
+
Contost froe
Dontee-Sudded
10%
Pretraining
Inference
Figure 2: Diagram of two-stage investigation. During pretraining time, we feed a uniform mixture of 90% context- prepended texts and 10% context-free standard texts into the model. During inference time, we compare three different generation sampling methods.
cooldown phase, our uniform mixture strategy allows any
3.2 Context-Aware Generation
At inference time, leveraging the conditioning capabilities learned during pretraining, we can guide the LLM generate text based a given context. Next-token prediction involves sampling from a probability distribution (PLLM) over the entire vocabulary. To incorporate contextual signals, we modify this distribution accordingly. We define the three following sampling methods for generation:
1) Context- sampling: this is the standard sampling given by LLM, conditioning empty contexts.
x' ~ plMM (x|x .0)
(ctx-free)
2) Contex--Conditinned sampling: for each generation, we prepend <boc>ctx<eoc> the prompt.
PLLM (xr| 1:t-1 ctxx) ctx)
1:t-1
(ctx-conditioned)
3) Context-Guided sampling: following the classifier free guidance in diffusion models [5) we amplify the impact of the context with the following sampling probability:
(DLLM 22/2 !st-1 ctx)
x. PLLM (L)M
(ctx-guided)
(
3

---

During inference, in practice, we modify the generation in the logit (II) level, that is
xe (s|ar 0) + (x|= ctx) IT (x) a 1tt11 0))
When = O, this coincides with context-free generation and when y = 1, this coincides with context-conditioned generation. With 1, we amplify the guidance of the context.
is worth noting that the softmax output after context-guidance no longer a faithfu probability distribution, while context-free and context-conditioned generation still follow faithful probability distribution.
Experiments
Model. We adopt the Llama model architecture 11 with 16 layers, a hidden size of 2048, sequence length of 4096, and a batch size of 504 (resulting in 2.06 million tokens). The model has 1.5 billion parameters. We follow the Cosine learning schedule, applying 2000 warmup steps. AdamW optimizer is used with regularization strength 1 [2]. A max learning rate of 3e-4 is applied and cools down to learning rate 3e-5 at the end of training. To train the models, we use the Megatron-LM framework 13..
Dataset. We use Webb dataset [4]. which is a high-quality English-only dataset. The dataset has a lot of meta data available, such as URL source, quality score and token counts per document. We sub-sample Fine Web-WebFedd and tokenize it using Nemo tokenizer with two additionally introduced tokens. Throughout training, we randomly sample 100B tokens.
Evaluation Benchmarks. As standard practice, we evaluate models on general knowledge un- derstanding using LM-Eval-Harness developed by Gao et [15]. The benchmarks used are Arc- Easy [16. Are-Challenge [16), CommonSense QA (CSQA, 17), MMLU [18], PIQA [1)], Social IQA (SIQA, 20), HellaSwag 21), Lambada (LBD. (2) and Winogrande (WG,23) 23).
We compare the following runs to gain a full insight over how can contexts potentially enhance LLM pretraining.
1. Baselines
Standard Pretraining: Each document is simply prepended with a begin-of-sequence
token 

Positional Token Ablation: Standard pretraining where each sequence is prepended with <boe><eoc> tokens. This setup tests whether introducing a positional token the start of each sequence alone has a beneficial effect.
MeCo: A two-phase approach proposed by Gao et al. 3-90 of training is URL- conditioned, followed by 10% cooldown phase using standard (unconditioned) data.
2. Types of Metadata
Full URL (URL): The original source from which the document was crawled. Quality Score (QS): Provided by the FineWeb-Edu dataset, these scores are generated by classifier trained on 450,000 web samples, each annotated by LLaMA with a score from 0 (not educational) to 5 (highly educational), reflecting the sample's educational value.
Domain Information (DI): Each document is labeled with topic and format domains derived from WebOrganizer [24], where topic and format domains are returned by pretrained classifiers. There 24 categories per taxonomy. In total, have 576 different Domain Information types.
3. Combinations of Metadata
Full URL Quality Score: Combines the document's source with its educational
quality.
Full URL + Domain Information: Combines the source with topic and format domain labels.
BB faithful. we mean a probability distribution that can trusted as true representation of uncertainty, without distortion, manipulation, or miscalibration.

---

standard <bOC>
standard
14-
wy LIRL
Wy LIRI +QS
13-
13
URL-DI
Purplcstty
QS
Perplesity
W. +0S+D0I
12
McCo
12
11-
10-
18-
50
0.8
1,0
0.2
0.4
1.0
Traiined Tokens
# Traineed Tokens
Figure 3: Training perplexity versus the amount consumed tokens. Prepending the URL leads to a faster decrease in perplexity.
-X- standard
W URL
arc. challengelace
mmlu continustion/ace
piea/acc
winograndelace
average
0.36
0.73
0.32
6.32
0.71
0.30
0.30
0.52
208
408
80B
non
1008
208
408
aon
1000
208
400
s0B
1008
2011
40.8
60B
808
1000
208
408
ana 1868
Figure 4: URL-conditioned pretraining achieves the same downstream evaluation performances of 100B-token standard pretraining with only 60B tokens. The same plots with respect to all tasks are provided in Figure]
Full URL Quality Score + Domain Information: Incorporates all available metadata to condition the model on source, quality, and content structure.
4.1 Not All Metadata Conditioning Speed-up Pretraining
4.1.1
Training Speed-up with URL Conditioning
We begin by investigating whether metadata facilitate training. specifically, whether pretraining can be accelerated when additional contextual information aids knowledge acquisition. The left panel of Figure presents the training perplexity across runs with different metadata types. The two additional tokens introduced have minimal effect on learning dynamics, indicating that any observed differences stem from the contextual content itself. Among the three types of context evaluated, only the URL consistently results in lower perplexity from the very beginning of training.
Is the improved performance stemming from longer contexts? Given that URLs are substantially longer than the other two metadata types, one might wonder whether their effectiveness stems merely from increased context length. To examine this, combined the other metadata types with URLs to see additional context would further improve training. From the right panel Figure 3 the answer is No. This indicates that is not the increased context length that speeds up the training, but the context content itself.
Does the acceleration translate into downstream performance as well? With five-shot evaluation, we see better performance for the URL-conditioned model every task. For the 9-task average, the URI-coonitionee model can match the performance of the standard pretrained model on 100B tokens with only 60B tokens, achieving a 40% significant acceleration, as shown in Figure The acceleration in terms of downstream performance is greater than training loss.
Why only URL helps? We have already controlled for the confounding factor the length of metadata. However, it remains unclear why only the URL contributes meaningfully. To gain insight into this, we visualized the attention patterns of the document texts toward different parts of their corresponding contexts in Figure 5) Except for the URL-conditioned model, the other models fail to allocate substantial attention to the informative parts of the contexts in the early layers. We hypothesize that the early-layer attention to meaningful URL components may play key role in the effectiveness of URL prepending, even if later layers focus on less informative tokens. Moreover, important to recognize that URLs inherently encode both domain information and document quality,
5

---

04
62-
92
61 <hor>
qualit
topic
Figure 5: Average attention to different parts of different prepended contexts. Take https://ent //s::///n... wikipedia. wttp///aa//eeaaaaaataaaaaaaaaa
an example. en. wikipedia. org is the URL suffix. More details in Ap-
is the URL domain and /ulki//etadata*Staaaia pendix C.1.2
which may differ from human-assigned DI and QS labels. This distinction likely also contributes to the superior performance of the URL feature.
Takeaway 1.
Only URL conditioning has a positive effect in speeding pretraining; conditioning on quality scores or topic/format domains yields no noticeable change.
4.1.2 Downstream Performance Gains from Longer Prompts
When does URL conditioning work? To better understand the impact context conditioning, we perform a systematic evaluation across downstream tasks by comparing zero-shot and five-shot performance. We notice a very interesting phenomenon: The URL--cnddiioned pretrained model shows increased downstream evaluation performance with sive-shot evaluation, but not zero-shot. For the rest, we do not see substantial improvement over standard pretraining. Our results confirm recent findings from Higuchi et al. [4], where longer prompts are needed to infer latent semantics and thus to help evaluation.
Conflicting signals of different metadata types. Interestingly, the benefits of URL conditioning are negated when additional metadata introduced as context. For example, combining quality scores with URL conditioning appears to introduce conflicting signals, leading to downstream performance that is even lower than that of the standard pretrained model. This suggests that the latent clusters inferred by the language model from the URLs may differ from the predefined clusters we assigned. Notably, we did not manage to reproduce the results the MeCo baseline. Possible reasons could be different datasets and models, and that AM-Eval-Harness [15 is used as our evaluation framework used our experiments, while OLMES (25 was used MeCo paper. Nonetheless, we still observe the speeding-up effect of URL prepending.
Table 1: Oshot evaluation results. When evaluating, context-free generation is used, i.e., only <boc><eoc> tokens are prepended and the context string is empty.
Are-C
Are-E
CSQA
MMLU
PIQA
SIQA
HS
LBD
WG
Avg
standard
32.9
68.9
42.3
31.2
71.3
40.6
42.6
33.0
<ber><eac>
33.1
68.4
41.8
32.0
72.1
40.2
42.4
33.4
56.6
46.7
+ <bec> URL <eec>
33.5
69.8
41.6
32.0
73.4
41.3
42.8
33.2
$4.4
46.9
38.3
30.9
<boc> DI <eoc>
32.1
68.4
41.9
31.8
71.3
41.1
42.1
34,2
53.4
46.3
+ <bec> URL. QS <eoc>
31.2
68.7
38.2
31.7
71.3
40.1
42.2
34.6
$4.2
45.8
URL, QI
31.8
42.0
+ <boc> URI, QS, DI <eoc>
32.4
69.4
40.8
31.8
71.8
39.8
42,0
33.0
54.9
46.2
McCo
33.0
68.9
39.7
31.8
71.3
40.
41.9
33.0
55.6
46.2
Takeaway 2. URL-conditioned pretraining only benefits evaluation for longer prompts, for example, 5-oho. There is noticeable distinction for O-shot evaluation.
4.2
Context-Aware Generation
4.2.1
Context-Conditioned Generation
The results from Table 2 are from context-free generation using context-condittonel model. We now test whether the performance further be improved if correct contexts given at test time.

---

Table 2: 5shot evaluation results. <boc><eoc> tokens are prepended and the context string is empty.
When evaluating, context- generation is used, ie.. only
Are-C
Are-E
CSQA
MMLU
PIQA
SIQA
HS
LBD
WG
Avg
standard
33,9
68.6
45.1
31.8
71.7
41.5
41.9
30.6
54.9
46.7
<boc><<oc>>
34.0
69.2
43.5
31.9
71.9
41.5
42.5
30.8
55.2
46.7
<boc> URL <eoc>
35,5
71.7
46.8
32.8
73.0
41.7
42.5
30.8
55.9
47.8
<boc> QS <eoc>
32.7
69.6
41.0
31.8
72.1
41.6
43.0
32.4
55.2
46.6
<boc>> DI <eac>
34.0
69.8
42.6
32.1
71.9
40.3
42.0
31.8
55.5
46.7
<boc> URL., <e<cac>
32.5
67.8
42.8
31.6
71.4
41.7
41.8
32.0
53.0
46.1
URI.,
33,9
42.2
31.4
41.5
32.9
46.7
<boc> URL, QS, DI <eoc>
32,9
68.6
44.6
31.9
71.6
39.8
41.9
31.8
51.0
46.0
McCo
32.6
69.7
46.3
32.5
72.0
39.9
41.6
31.3
54.0
46.7
For each task, conditioned the generation contexts that match the task content. We manually select these contexts. For Quality Score context, we simply prepend Quality Score: 5 in the inference time. The contexts used are presented in Table TO in the Appendix. Conditioning on provided contexts, we observed increased downstream performance, especially in URL- and DI-conditioned pretrained models.
Table Enhanced downstream task performance with context-conditioned generation compared to context-free generation. Three different contexts are tested here, that is, URL,, Domain Information (DI) and Quality Score (QS).
Context
Are-E
CSQA
PIQA
SIQA
Avg
URL-cconttooned
w/o context
35.5
71.7
46,8
32.8
73.0
41.7
42.5
30.8
55.9
47.8
w/ URL context
35,5
72.2
47.7
32.9
72.3
43.0
43.0
32.5
56.0
48.3
w/o context DI context
34.0
69.8
42.6
32.1
71.9
40.3
42.0
31.8
55.5
46.7
DI-conditioned
33.9
70.9
46.0
32.3
72.4
41.7
42.2
33.2
55.1
47.5
QS-conditioned
w/o context
32.7
69.6
41.0
31.8
72.1
41.6
43.0
32.4
55.2
46.6
w/ QS context
32.0
70.0
44.3
32.1
72.7
42.4
42.0
32.9
54.4
47.0
4.2.2 Context-Guided Generation
Can the impact of context be further amplified? Accuracy calculation in Table 3 relies on log- likelihood that requires probability distributions. Classifier-free guidance modifies the output logits before softmax to steer generation. The logits have been manually shifted, which makes the resulting probability distribution not grounded in the model's knowledge data, thus breaking the probabilistic interpretation. Likelihood-be -bae evaluation does not work. We turn to generation- based evaluation instead. Given the limited capacity our 1.5B base models, we cannot evaluate them meaningfully using free-form generation for the QA tasks. As result, we assess their ability to generate coherent continuations when provided with a prompt.
We gather prompts in health and history domains, and let our pretrained models finish the prompts. We let gpt-4o evaluate over 4 dimensions regarding the continu- ation, that are, coherence, correctness, reasonableness, and relevance. The scores are between 1 and 10, with higher score being more desirable. With context-guided guided genera- tion, apply the steering factor 7 = 1.5 5 if not mentioned, from hyperparameter grid search in Figure Over prompts, we have the go evaluation presented in Ta- ble 4 where we compare the three different generation methods using both our caneex--onnditione pretrained model and standard context-free pretrained model. Given that LLM judges may hallucinate, we run a 3-seed eval- uation and report the mean and standard deviation. Com- pared to the standard pretrained model, observe that context-conditioned pretrained model is more steerable, as showcased by improved performance with context-guided generation.
8.5
7.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Steering Factor
Figure 6: LLM judge scores (with stan- dard deviation) vs. different values of steering factor
Which metadata type is efficient in guidance? Among the three different context types, we notice that even though topic and format domains are not effective speeding up training, they offer effective steering possibilities. We provide an example of different continuations from the three
7

---

different sampling methods in Table Conditioning on Topic: Health, Format: Knowledge Article, while context-free generation offers very generic answers, context-conditoned generation is able to offer more health-relevant technical answers. Context-guided generation is the most comprehensive and informative among the three, indicating the successful guidance of the context.
Table 4: Average LLM-as-a-judge evaluation scores (mean = std) on the different prompt completions on 50 prompts on health & medical and history topics respectively (answer length: 64 tokens). Model type denotes the type of checkpoints used, with "conditioned" being our context-conditioned pretrained checkpoint and "standard" being the standard context-free pretrained checkpoint. For each of the different contexts, we use different context-pretrained models.
topic
context
model type
etx-free
ctx-conditioned
ctx-guided
URL: htpp://meedccakgprss.com//
Conditiond
8.19 = 0.14
8,14 +: 0.20
8.70 0.14
Standard
7.98 = 0.34
1.79:0.100
8.12 0.26
& medicul
Topic: Health, Format: Knowledge Article
Conditoned
0.16
8.26 +
9.07 0.05
Standard
7.75+ 0.03
7.79 + 0.08
7.66 0.24
Conditione
8.20 = 0.19
8,20 + 0.20
8.69 0.14
Quality Score:
Standard
0.20
8.22
8.09 0.21
URL:: htts://hriuuniccacom
Conditoned
5.70
0.15
5.80 +:
6.73 0.43
Standard
4.88 + 0.15
5.60 + 0.14
6.45 0.01
history
Conditioned
5.35 0.24
5.23 + 0.26
7.15 0.26
Topic: History, Format: Knewledge Article
Standard
5.19 0.31
+
6.04 0.31
Conditioned
5.36 0.20
5.67 +
6.45 0.34
Quality Score:
Standard
5.52 = 0.19
5.68 + 0.19
7.04 0.09
Table 5: Example of different continuations (answer length: 64 tokens) given a prompt and their corresponding ratings. The Domain Information context provided Topic: Health, Format: Knowledge Article. Continuations are generated from our DI- DI-conditioned pretrained checkpoints. (Caution of Hallucination!)
Given prompt: Common causes of balance problems in older people include
Rating/Short Ratio- nale
standard
falls and underlying conditions. These range from minor health issues to more serious problems. You can get bal- ance problems from several causes, including poor nutrition, aging, and underlying health conditions like high blood pres- sure thyroid problems. It's also possible to suffer from balance problems suddenly.
5.5. Clear but general and slightly repetitive
context- conditioned
medications, ear problems, and spinal and neck problems. Vestibular rehabilitation physical therapy. which uses special exercises enhance brain signals that help maintain balance, can be used treat some types balance disorders. In most cases, medical attention is necessary to treat this type of dizziness imbalance.
7.0. More techni- cal, introduces help- ful treatment con- cepts
context-guided
ageing-related related changes in the inner ear. neurological prob- lems, and problems with medication. Other reasons for bal- anee problems include head injury, Parkinson's 's discase, mul- tiple sclerosis, and stroke. Vitamin B22 deficiency cause balance problems.
9.0. Most compre- hensive and infor- mative
Controllable guidance level. We further demonstrate various steering possibilities enabled by our method. For example, it is possible to guide the generation toward a desired topic by increasing the guidance factor value (7). When conditioning on Topic: Fashion Beauty, Format: Personal Blog and setting 7 > 1, LLMs sometimes produce humorous absurd outputs in Figure such as stating that Julius Caesar was assassinated because "of his taste in makeup" or "he wore too much jewelry." While this clearly highlights the model's tendency to hallucinate, it also underscores the strength the steering mechanism. With appropriate contextual grounding, such guidance could potentially yield coherent and relevant outputs in desired domains.
Controllable style and content. Moreover, when provided with different contextual signals, our models can adapt the continuation in ways that align with the intended framing source. As shown

---

Table 6: Example of different context- guided continuations given a prompt and steering fac- tor (t). The Domain Information context is Topic: Fashion & Beauty, Format: Personal
Blog. Continuations are generated from our pretrained DI-conditioned checkpoints. (Caution of Hallucination!)
Steering factor (7)
Given prompt: Julius Caesar was assassinated because
-2
he'd killed too many people and he'd d become too dangerous. He was assassinated by senators and officials who on the wrong side of the Senate. Caesar feared they might take power away from him and start an anarchy.
he was trying to become king of the Roman Empire. He was forced to flee Greece to escape the Roman legions. This was major mistake, because he was already an experienced general and knew that he could not defeat them in the long run.
his assassins feared he would be threat their power. At the time, the average Roman man's wardrobe consisted of lumic and short-sleeved shirt. But soon as Julius Caesar was assassinated, Rome's dress code became stricter.
of taste makeup. Back then, makeup was not only for looks. It also had medicinal purposes. The Romans used makeup to soothe and protect the skin from the sun. It also had medicinal purposes. They believed it could heal certain skin conditions and infections.
4
he too much jewelry. Napoleon that jewelry became sign of royalty and wore very little other crowns, earrings, necklaces and bracelets. Kings, Queens, royalty and those with influence wore much more than is necessary.
inspired by Alexander the Great
in Table 7, the open-ended prompt "Paris is known for" leads to varied continuations depending on the guiding URL: medicalxpress com prompts a medically-oriented response, nytimes. com results in factual and formal description by listing facts, and reddit. com encourages casual, entertainment -occued tone. A similar result with Domain Information guidance is presented in Table and Table 9 in the Appendix, where observe effective guidance in both topic and format domains. These results demonstrate that contextual steering can shape both the content and style of the generated text in meaningful ways.
Takeaway With context-conditioned pretraining. a larger improvement is observed in context-guided generation compared to context-conditioned generation.
Takeaway 4. Although domain information does not accelerate pretraining, provides more interpretable and efficient guidance during inference.
Conclusion
In this work, we present comprehensive investigation into context-conditioned pretraining for LLMs, focusing on the impact different metadata types. Our findings reveal that not all metadata types contribute equally to model efficiency downstream performance. Specifically, URL metadata stands out as the only form of context that significantly accelerates training and improves perfor- mance-partien partic-aatiillarrlyy when longer prompts used at inference time. In contrast, quality scores and topic/format domain information do not yield measurable benefits during pretraining.
Nevertheless, we demonstrate that these latter forms of metadata have strong value during inference, enabling more controllable and interpretable text generation through classifier-free guidance mecha- nisms. This dual insight, that certain metadata can optimize training efficiency while others enhance controllability, opens promising directions for more flexible and human-alise aligned language model applications.

---

Table 7: Example of different continuations given prompt conditioned on different URLs. The continuations are generated in a context guuided fashion using our URL-conditioned pretrained checkpoints. (Caution of Hallucination!)
Given prompt: Paris is known for
https:// medica medicalxpress.
150 years of medical excellence, thanks to the collaboration of doctors, nurses, medical students and researchers. It is also known as the home of major medical discoveries such as the operation of the heart, the discovery of insulin, the discovery of the HIV virus, the discovery of the X-ray, the discovery of the...
com/
https:
4 things: 1. Big. It's the capital of France and also the largest city in Europe. 2. Beautiful. It's a very beautiful city. Rich. It's one of the richest cities in the world. 4. Old. It's one of the oldest cities in the world.
/nyt /nytimes, com/
https: /ttpp://reddit..
things, its cuisine, its fashion, its nightlife and its cinema. And don't mean that only. What if I told you that there is community of actors who actually act out the scenes of French movies? This community is called REDdit!
com/
Overall, our results suggest that context-aware pretraining, especially when paired with dynamic guidance strategies inference, offers powerful toolset for building more efficient, adaptive, and steerable LLMs. Future work could explore scaling these techniques to larger models and broader metadata sources, as well as integrating contextual signals directly into fine-tuning and instruction tuning pipelines.
Acknowledgment. This work was supported part of the Swiss AI Initiative by grant from the Swiss National Supercomputing Centre (CSCS) under project ID a06 on Alps. DF would like to thank Diba Hashemi for helpful discussions.
10

---

References
[1] Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL BRLLhttps://opeereeeee 3
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws, 2024. htttsss/://rrx org/ aws.//044.44 2200.054000 05405
[3] Tianyu Gao, Alexander Wettig, Luxi He, Yihe Dong, Sadhika Malladi, and Dangi Chen. Metadata conditioning accelerates language model pre-training, 2025, URL org/abs/2500 01956
[4] Rei Higuchi, Ryotaro Kawata, Naoki Nishikawa, Kazusato Oko, Shoichiro Yamaguchi, Sosuke Kobayashi, Seiya Tokui, Kohei Hayashi, Daisuke Okanohara, and Taiji Suzuki. When does metadata conditioning (not) work for language model pre-training? a study with context-free grammars, 2025. URL hwt.h:tttt//aate http/arxivv.rrr. h/t///a///// abs/2504. 17562
[5] Jonathan Ho and Tim Salimans. Classifier- diffusion guidance, 2022. URL https:// arxiv. org/ abs/ 12598
[6] Xingyu Zhu, Abhishek Panigrahi, and Sanjeev Arora. On the power of context-enhanced learning in llms, 2025. URL
wwwp:///.rrgaab///0333 018211
[7] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pages 6691-6706 167006 Online, August 2021. Association for Computational Linguistics. doi: 10. 16653/v1/2021 wtt.://222221.1on....1 la22-lon..522. URL
org.ogg///// 2021 acl- 522/
[8] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan Boyd-Grabes and Naoaki Okazaki, editors, Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 12286-122 12266-13313 12112, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/w120233.cchl acl-ooog.... URL https:://ccaaaaaoooog..ooo acl-long.680 6877
[9] Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammana- manchi, and Stella Biderman. Stay on topic with classifier-fres guidance, 2023. URL
attps://arxiv org/ /bb//300.... 17806
[10] Jerry Zhi- Yang He, Sashrika Pandey, Mariah L. Schrum, and Anca Dragan. Context steering: Controllable personalization at inference time, 2025. URL /ttpp:://axxi.....t// abs/2405 01768
[11] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbic Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis. Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Oaacia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan
11

---

Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Va- suden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Cantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam- badur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hos- seini. Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan. Shruti Bhosale, Shun Zhang, Simon Vandenhende. Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan. Sydney Borodinsky. Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckhacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur. Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim. Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean -aptisae Gaya, Jeff Mareus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie. Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L.. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat,
12

---

Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinou,, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay. Shaun Lindsay, Sheng Feng. Shenghao Lin. Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shugiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho,, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang. Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi. Yenda Li, Yilin Zhang, Ying Zhang. Yossi Adi, Youngjn Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary De Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL URL btttps://arxvv.ooo wrggabs///00/ 21783
[12] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https //arxiv. hrg. abs 1171.005101
[13] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catan- zaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron- 2021. /ttps/ttp:///aaxi.... org/ abs 2104. 04473.
[14] Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. htKL:tttt///hugiigggaaaa co/ /
[15] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang. and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL htttps://zenod:: org/ records/ 12608602
[16] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1a0::0004305555 2018
[17] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge. arXiv preprint 18X1vv81111009)) 2018.
[18] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[19] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piga: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[20] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv: 1904. 09728, 2019.
13

---

[21] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.07830. 2019.
[22] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoe Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.0644 16666.0003,,,, 2016.
[23] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106,
2021.
[24] Alexander Wettig, Kyle Lo, Sewon Min, Hannanch Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation, 2025, URL https://arxiv.out wwwpxiiv.orgaaaae abs///2s22222... 10341
[25] Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Ha- jishirzi. OLMES: standard for language model evaluations. In Luis Chiruzzo, Alan Ritter, and Lu Wang. editors, Findings of the Association for Computational Linguistics: NAACL 2025, pages 5005-5033, Albuquerque, New Mexico, April 2025. Association for Computa- tional Linguistics. ISBN 979-8-89176-195-77 URL
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main text corresponds to what we claim in the abstract and introduction, i.e., what benefits does context-aware pretraining bring in pretraining and inference time, respectively.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See our discussions the conclusion section as well as in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We detailed our tokenization process and metadata used. Moreover, the training details are also specified Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will make training codes public when the paper is made public. The dataset we used is public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc)
Poolicc/guides/Coeeeuubiiis
www.sssinoooiiillooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtp:/cccppplgguuuiiaeoeaec
Stteeee//ooSSSSeissSioica ssissssliccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We specified our dataset, hyperparameter choices, and training details in Section
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We reported 3-seed run results with both mean and standard deviation in Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specified in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We adhere to the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We are not releasing models, but rather providing insights assist model pretraining.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited the dataset, model training framework, codebase and tokenizer.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We clearly described our usage of LLM as a judge, to grade continuations from different models.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A Limitations and Future Work
In this work, we investigated three different types of metadata. Yet it is not very thorough investigation. Furthermore, the results we present in the paper are all from 1.5B models on 100B Fineweb-Edi Edwueb--uuuu dataset. It remains to be seen if our findings consistent when scaling up the model sizes and token counts or switching to different data corpora.
Our work points out lot of interesting findings from observations in our experiments. The me- chanics behind the improved performance brought by URL conditioning are still unclear. From our experimental results, we know it is not because of the indicated topic/format from the URLs, or it could be that the LMs have different understanding topic and format domains. Future work should investigate the changes in activations between context-free and context-conditioned pretrained models to have better understanding.
B Broader Impacts
As discussed in Section can be used to guide responses in desired style or format, it also has the potential to produce harmful untruthful content. Therefore, these methods should be applied with social responsibility.
4.2
context-conditioned models are more easily steered. While this capability
C More Experimental Details
Each of the model training run takes around 800 GPU hours on GH200(120GB).
C.1 Missing figures and tables
C.1.1 Acceleration measured by downstream task performances
In addition to Figure we provide the evaluation with respect to the remaining tasks in Figure 7 The acceleration consistent across different tasks.
-*- standard -+- w/ URL
arc. easylacc
standard/acc
commonsense ga/ace
social
hellaswaglacc
0.700
0.300
0.46
0.41
0,275
0.40
0.650
0,250
0.38
0.40
206
406
608
806
1008
206
40B
608
82B
1008
208
408
608
808
1008
208
408
50B
808
1908
20B
408
50b
1008
Figure Training acceleration in terms of downstream performances.
C.1.2 Attention pattern to different parts of contexts
We randomly sample 100 documents from FibeWebEEdu and prepend each with its corresponding URL, quality score, and domain information. These prepared documents are then input to the corresponding context-conditioned pretrained checkpoints; i.c., URL-prepended documents are fed to the URL-conditioned model. We use batch size of and sequence length of 4096, without any padding. The attention weights shown in Figure 5 are averaged across all attention heads the layer and over all non-contextual tokens, and are normalized by the total attention assigned to the contextual portion.
We further visualize the evolution of attention patterns across layers using a heatmap, as shown in Figure g Starting from layer 3, we observe a pronounced attention sink on semantically uninformative and commonly shared tokens, such as the URL prefix fnttps://) This pattern persists across other metadata-coonditioned checkpoints. We therefore hypothesize that the attention the first two layers contributes meaningfully to training accelcration.
22

---

10
1-
08
06
10
044
12-
14.
16
domain
sufffffixx
boc token
coc token
vrlpprefxx
Figure 8: Average attention pattern of URL-conditioned model across layers to different parts of URL context.
Table 8:
Example of different continuations given a prompt conditioned different topics, while the format domain is kept as Knowledge Article. The continuations are generated in a context- -uiided fashion using our pretrained checkpoints. (Caution of Hallucination!)
Given prompt: Paris is known for
Crime Law
its cultural diversity, and LGBTQ+ people, women, and people of color often underrepresented. The city is home to many LGBTQ+ neighborhoods, which are underrepresented in many other cities and towns. However, these neighborhoods particularly vulnerable to violence, abuse, and harassment.
Finance & Business
its economic prosperity. In the late 19th and early 20th centuries, Paris hotbed of the industrial revolution Europe, and continues be hub for commeree. The city's 's robust cconomy is largely fucled by its high-tech sector, which includes both the manufacturing and technology industrics
Entertainment
its rich musical traditions, with thriving musical scene that has left an indelible mark on the cultural landscape. From the classical era to the modern era, Paris has been city of musicians and composers who have shaped the music of the world.
Education & Jobs
its prestigious education system. Universities are renowned their research capabilities, cutting-edge facilities, and excellent faculty. To ensure that universi- ties meet the highest standards of academic excellence, France has implemented various quality assurance mechanisms, including external examinations. These examinations assess the overall quality university's academic programs, faculty. and student body.
C.1.3 More steering results
We further present steering results on the DI-conditioned model Table Table&aand Table P Given the exact same prompt, the continuation can be steered in markedly different directions depending the contextual guidance. Topic-bases guidance is highly visible, while format-based guidance is more subtle. For example, when steering with "News Article", the output tends to include concrete details about the composer and adopts a more objective tone oone-different from the more casual and subjective tone of "Personal Blog". Similarly, when guided by "Spam/Ads", the output adopts promotional style, often highlighting a specific fair an advertising manner.
C.2 Prepended context in inference time
23

---

Table Example different continuations given a prompt conditioned on different formats, while the topic domain is fixed as Entertainment, The continuations are generated in context- guidedd fashion using our pretrained checkpoints. (Caution of Hallucination!)
Given prompt: Paris is known for
News Article
its rich music history, and city has produced many renowned composers who have left their mark on the world of music. One such composer is Jules Massenet, a French composer who lived from 1842 to 1912. Massenet is considered one of the most important composers of French opera...
Personal Blog
its music scene and it is home to some of the most famous musicians in the world. One of the most famous French musicians is Charles Dutoit, who widely considered to be one of the best French musicians of all time. He has a unique style that blends traditional French music with modern approach, and he...
Spam/Ads Ads
its rich musical traditionits many wonderful fairs, particularly the fairs at the Les Demoiselles d' Avignon, but also the many other fairs throughout the year. In music, the term "chanson" is often used, especially in the 18th century, refer a genre of music that was particularly
Table Different contexts for Table
Task
URL context
DI context
QS context
Arc-E/C
www www.ffccmonster.oom com/
Topic: Science & Tech., For-
mat: Knowledge Article
CSQA
www.factmonster.com
Topic: Social Life, Format:
Knowledge Article
Quality Score: 5
MMLU
www.reference.com/
Topic: Science & Tech., For-
mat: Knowledge Article
PIQA
www.honstuffworks.cco
Topic: Social Life, Format:
Knowledge Article
SIQA
hww.reeditt.omm///
Topic: Social Life, Format:
AskSocialScience/
Knowledge Article
HS
www.wkk.hww.cooo
Topic: Home & Hobbies,
Format: Knowledge Article
LBD
www.wow.ccom
Topic: Literature, Format:
Creative Writing
WG
www.reddit.com///
Topic: Social Life, Format:
explainlikeinfive
Personal Blog
C.3 LLM Judge
We use gpt-4o from OpenAl API to evaluate continuations given prompt. The prompt used for scoring medical & health domain continuations is presented in the following block. For the history domain, modify it correspondingly to suit the topic.
24

---

You will be given prompt and completion that attempts to continue the prompt. Your task to judge the quality of the complction based on the following criteria:
Coherence: Does the continuation flow logically and grammatically from the prompt?
Correctness: Is the information medically or scientifically accurate?
Reasonableness: Is the claim plausible and not misleading or exaggerated?
Relevance: Does it directly relate to and complete the idea expressed in the prompt?
Rate the continuation on scale from I to 10, where:
1 = incoherent, irrelevant, medically incorrect
5 somewhat relevant or plausible, but lacking clarity precision
10 = fully coherent, factually accurate, and medically relevant
You should ignore the irrelevant answer if the relevant part makes good continuation. You must not hallucinate or invent facts. Use only information in the prompt and completion.
Respond in this format:
Prompt: question
Continuation: answer
Rating: (your rating, as a float between and 10)
Rationale: (Brief explanation of the score)
25


Reviews:
[{"review_id": "B7ucgrvuFX", "reviewer": "Reviewer_RrS9", "summary": "This paper investigates whether incorporating metadata, such as URLs, quality scores, and domain info, during LLM pretraining improves training efficiency and downstream performance.\nThe paper conducts a systematic study using context-conditioned pretraining, where metadata is prepended but excluded from the loss function.\nThey find that only URLs help accelerate training and improve performance, particularly for longer prompts during inference.\nDomain metadata, while not helpful in training speed or general performance, provides strong controllability during inference.", "questions": "- Can the findings generalize to larger models?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## strength\n\n- The paper introduces a novel experiment setup by evaluating different metadata in a controlled pretraining framework.\n- The paper is well-organized with clear motivation and detailed descriptions of methods and experiments.\n- The paper conducts experiments and gives empirical findings.\n\n## weakness\n\n- The central issue is that all experiments in the paper now use only 1.5B language models (Llama). Results may not generalize to larger models. This significantly hampers the contribution.\n- The experiments only consider a single dataset (FineWeb-Edu).  Results may not generalize to more diverse corpora.\n- Why URLs help more is not deeply analyzed. This may weaken the impact of the findings.", "was_revised": true, "final_justification": "The authors' response has addressed some of my concerns.", "rebuttal": "Thank you for your thoughtful feedback. We appreciate the time and effort you invested in reviewing our work. In the following response, we refer to Weakness point i as Wi and Question point i as Qi.\n\nRegarding **W1**:\n\nWe acknowledge the concern raised. Due to limitations inherent to our academic setting, it was already a significant challenge to complete the 1.5B parameter experiments. Each experiment on 100B tokens requires over 800 GPU hours. While we recognize that results may not fully generalize to larger-scale models, our computational resources did not permit further scaling. Nonetheless, we believe that the 1B scale remains a reasonable and informative choice. Notably, the predecessor work MeCo[1] conducted the majority of their experiments using a 1.6B model, which supports our design decision. Given that their findings generalized well across models ranging from 600M to 8B parameters, we feel that our choice of scale is a reasonable and pragmatic compromise under the given resource constraints.\n\nRegarding **W2**:\n\nWe contend that FineWeb-Edu is already a sufficiently diverse corpus. It is derived from Common Crawl, first processed into FineWeb (a cleaned and deduplicated version), and then further filtered for educational quality to yield FineWeb-Edu [2]. This filtering makes FineWeb-Edu more data-efficient than FineWeb, allowing us to achieve comparable downstream performance using fewer tokens.\n\nRegarding **W3**:\n\nWe were similarly intrigued by the observation that URL was the only metadata type that consistently yielded a performance benefit. Since the submission, we have begun investigating this phenomenon and have formulated two working hypotheses:\n\n- H1: Prepending URLs significantly alters the model’s attention distribution.\n- H2: Prepending URLs enhances the structure of the learned latent representations, leading to more distinguishable and fine-grained embedding spaces.\n\nTo evaluate H1, we performed both visual and quantitative analyses of the attention patterns. \n\nFor clarity, consider the example URL (note this is not an external link, just an example):\n\n \"https://en.wikipedia.org/wiki/Metadata\".\n\nHere, \"https://\" is the prefix, \"en.wikipedia.org\" is the domain, and \"wiki/Metadata\" is the suffix. Often, URL domain carries format information and URL suffix carries topic information. \n\nSurprisingly, across all three models, we observe relatively little attention given to the semantically meaningful components of the metadata. More specifically, attention patterns in the URL-conditioned model vary by layer (see Table 1):\n- In layers 1–2, the model assigns higher attention to the URL domain and URL suffix, which carry informative content.\n- From layer 3 onward, however, attention shifts predominantly to common URL prefixes like “https://”, which are shared across most examples and carry little semantic value.\n\nIn contrast, for models conditioned on Quality Score and Topic/Format, the attention allocated to informative metadata remains _consistently_ low across all layers (see Tables 2 and 3).\n\nWe hypothesize that the early-layer attention to meaningful URL components may play a key role in the effectiveness of URL prepending, even if later layers focus on less informative tokens.\n\nTable 1: Attention weight distribution for URL-conditioned model.  Context is “\\<boc\\>\\<URL\\>\\<eoc\\>”. Non-context is the text following the context. \n\n|| non-context | context |  \\<boc\\> & \\<eoc\\> | URL prefix | URL domain | URL suffix |\n| --- | --- |--- |--- |--- |--- |--- |\n|1st layer | 79.3 | 18.9 | 5.2| 4.8 | 3.5 | 5.0 | \n|middle layer| 55.8 | 43.9 | 4.7| 37.4| 0.6 | 1.1| \n|last layer| 67.5| 32.2 |4.6| 24.5 | 0.9 |2.0 | \n\nTable 2: Attention weight distribution for Quality Score-conditioned model. Context is “\\<boc\\>Quality Score: \\<QUALITY_SCORE\\>\\<eoc\\>”\n|| non-context | context | \\<boc\\> & \\<eoc\\> | \\<QUALITY_SCORE\\> | \n| --- | --- |--- |--- |--- |\n|1st layer | 85.8 | 12.3 | 10.4 | 0.2 | \n|middle layer| 39.6 |55.1| 23.8| 0.3| \n|last layer| 59.3| 27.3 |13.0 | 0.1| \n\nTable 3: Attention weight distribution for Topic/Format-conditioned model. Context is “\\<boc\\>Topic: \\<TOPIC\\>, Format: \\<FORMAT\\>\\<eoc\\>”\n\n|| non-context | context | \\<boc\\> & \\<eoc\\> | \\<TOPIC\\> | \\<FORMAT\\> |\n| --- | --- |--- |--- |--- |--- |\n|1st layer | 83.3 | 15.1 | 5.1 | 1.0 | 1.4 |\n|middle layer| 54.0 |42.7|2.0 | 0.5 | 0.2| \n|last layer| 66.9 |23.0 | 6.1|0.4 |0.3| \n\nTo evaluate H2, we sampled 100 documents from each of 100 distinct URL domains within the FineWeb-Edu dataset, yielding a total of 10,000 documents. We did a 90-10 train/test split. We then trained linear probes on the hidden activations from each layer to classify documents by their originating domain.\n\nWhile we observed a general upward trend in probing accuracy with increasing layer depth, there was no significant difference between the URL-conditioned model and the standard pretrained model (Table 4). This suggests that, despite URL prepending, the structure of latent representations—at least as measured by domain separability—does not differ markedly between the two models.\n\nTable 4: Test probing accuracy across layers. \n|Layer index| URL-conditioned | Standard |\n| --- | --- |--- |\n|1st layer | 32.97 | 32.48 |\n|middle layer | 61.99 | 62.49 |\n|last layer | 64.18 | 65.04 |\n\nOur preliminary exploration suggests that the altered attention patterns may contribute to the performance gains observed with URL prepending. However, further investigation is needed to draw a more comprehensive and conclusive understanding of this effect. We hope this investigation provides some insights!\n\nWe hope this clarifies our decisions and findings. If you feel that our response sufficiently addresses your concerns, we kindly request that you consider raising your score. If any outstanding issues remain, please let us know, and we'll be happy to provide further clarification.\n\n\n[1] Gao et al. Metadata Conditioning Accelerates Language Model Pre-training\n\n[2] Penedo et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale"}, {"review_id": "fJufJHsUxC", "reviewer": "Reviewer_oR6w", "summary": "Recent works have shown that adding metadata to pre-training documents can be beneficial. However, it is unclear what kind of metadata is beneficial, in what way it is beneficial, and how this depends on how we evaluate/use the trained model. This paper studies the role of metadata in more depth. They find that adding URL context speeds up training, while other metadata (quality as well as topics/formats) does not have any effect. In terms of downstream performance, URL conditioning helps more on 5-shot than 0-shot evaluation. They also find that all types of metadata are able to help more effectively steer generations---for instance, asking a question about Julius Caesar conditioned on fashion and beauty blogs results in an answer that involves makeup.", "questions": "1. Is there a training acceleration figure for MeCo?\n2. When coming up with the metadata for a test sample at inference time, is there a scalable procedure?\n3. What happens when we do context-aware generation with URLs/topics/formats that are not in the training data?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nQuality:\n- The paper presents a comprehensive study on how to incorporate metadata into pretraining, and how to utilize models that are trained in this way. \n- The finding that metadata can steer generations is quite interesting and could have implications for personalization. \n\nSignificance:\n- This sort of systematic evaluation is valuable, as the behavior of metadata conditioning is not well-understood. \n \n**Weaknesses**\n\nQuality:\n- Regarding the results comparing 5-shot vs 0-shot performance, I am not sure if the conclusion is that URL conditioning only makes a difference with longer prompts, or if there is some other effect of ICL going on.\n- Is there a training acceleration figure for MeCo? (as mentioned in line 200) it would be good to see this to validate MeCo. \n- When coming up with the metadata for a test sample at inference time, is there a scalable procedure or does it involve human annotation? For instance, suppose a user has a very out-of-distribution query; what metadata would we use?\n- What happens when we do context-aware generation with URLs/topics/formats that are not in FineWeb? I.e., is the context-conditioning based on memorization in the pre-training data, or some understanding of what specific URLs/topics/formats \"mean\"?  \n\nClarity:\n- Some results in section 4 do not reference their corresponding tables/figures.\n\nOriginality:\n- The work does not exactly propose a novel approach, but this drawback is outweighed by the systematic analysis provided.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and encouraging feedback. We appreciate the time and thought you dedicated to reviewing our work. In our response below, we refer to Weakness point i as Wi and Question point i as Qi.\n\n**W1: Is enhanced downstream performance merely due to ICL?**  \n\nWe do not believe the gains are solely attributable to ICL. As shown in Table 2, only the URL-conditioned model demonstrates a significant performance improvement with longer prompts. If ICL alone were responsible, we would expect similar benefits across all models. This suggests that URL conditioning plays a distinctive role, especially with extended context.\n\n\n**W2 & Q1: Training curve of MeCo.** \n\nYes, we have recorded the MeCo training curve using Weights & Biases. As suggested, we will include this curve in the updated version of the paper.\n\n**W3 & Q2: Scalable metadata generation at inference time** \n\nIn our current implementation, metadata at inference time was produced through human annotation. In some cases, we also leveraged ChatGPT for assistance. Our used metadata at inference time is recorded in Table 10.  For scalability, we believe that automated approaches—such as using API calls with in-context examples—can replicate this process effectively.\n\nNotably, the prepended URL does not have to be real. For highly out-of-distribution (OOD) queries, a fabricated yet plausible URL can suffice. Quality Score can be set to the highest value by default, and the quality/format domain can be assigned using the WebOrganizer tool, which operates at web scale and offers sufficient coverage.\n\n**W4 & Q3: OOD metadata?**  \n\nAs mentioned in our response to W3, the metadata—such as URLs—does not need to exist in the pretraining data. Fabricated URLs can still guide generation effectively, which suggests that the model is leveraging latent semantic understanding rather than simply relying on memorization.\n\nWe include examples involving OOD topics and URLs in the Tables below (noting the potential for hallucinations). \n\nWhen given novel topics such as fruits or animals, the model appropriately centers the continuation around the topic. However, limitations remain: for instance, when prompted with Dark Humor, the model fails to grasp the topic meaningfully and defaults to generic content (e.g., a standard description of Paris).\n\nSimilarly, when we test with fabricated URLs, the URL-conditioned model still demonstrates a good grasp of their intended semantic meaning.\n\n\nTable 1: OOD topic guidance. The prompt is “Paris is known for”.\n| Topic | Continuation|\n| --- | --- |\n|Fruits | its delicious and diverse culinary offerings, and one fruit that’s particularly special is the pear. Native to the Mediterranean and Middle East, pears have a rich history and are a popular ingredient in many dishes. In this article, we’ll explore the many reasons why pears are so special…|\n| Animals | its historical and architectural monuments, as well as for its rich gastronomy. It is also one of the countries with the highest concentration of zoos in the world. The Paris Zoo is considered one of the most important zoos in the world, located in the heart of the city in the Parc des Buttes-Cha\"...|\n|Dark Humor | its numerous historical monuments and landmarks. One such landmark is the Basilica of Notre Dame. This cathedral is located in the heart of Paris and is one of the most famous landmarks in the city. |\n\nTable 2: OOD URL guidance. The prompt is “Paris is known for”.\n| Url | Continuation|\n| --- | --- |\n| https://easy-science-for-kids.com/ |  the discovery of helium which is found in the atmosphere. Helium was known to be a unique element at that time. Helium is known to be a very important element for humans and other living beings…|\n| https://www.modern-fashion-tips.com | fashion, and for a good reason, fashion is a field that involves designing, manufacturing and selling of clothing and accessories. Fashion design has a lot of components, such as the concept, pattern, fabric, color, pattern, size, and fit. Fashion design can also be characterized by the trends, styles, and …|\n\n\nWe will make sure that all the Tables/Figures are referenced in Section 4.\n\nWe hope this clarifies our decisions and findings. If any outstanding issues remain, please let us know, and we'll be happy to provide further clarification."}, {"review_id": "rZVQ90J6AK", "reviewer": "Reviewer_3pNV", "summary": "This paper studies pretraining a technique to provide metadata of documents during LLMs. While it is already known that providing URLs can accelerate and enhance pretraining, there has been limited systematic analysis of which types of metadata are truly effective and under what conditions. The authors evaluate three types of metadata: URLs, quality scores predicted by LLMs, and domain information annotated by pretrained classifiers. Their experiments show that only URLs contribute to faster training, whereas the other metadata types can even degrade downstream performance. However, the authors also find that context-aware pretraining can enable more controllable generation under classifier-free guidance.", "questions": "* Is there any intuitive explanation for the reason why URL conditioning improves performance on general benchmarks like MMLU?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n* This paper investigates context-aware pretraining, an important and broadly applicable technique for enhancing the pretraining of LLMs.\n* This paper presents an ablation study evaluating different types of contextual metadata and the conditions under which context-aware pretraining is beneficial. These findings are potentially useful for designing LLM pretraining strategies.\n* The paper offers a novel observation that context-aware pretraining can enable more controllable generation when using classifier-free guidance. \n\n### Weaknesses\n\n* Limited diversity in the studied context\n\nThere are many types of metadata that can be extracted from websites. However, this paper considers only two types, both annotated by LLMs. The diversity of the selected metadata is limited, and the rationale for focusing on these specific types is not clearly articulated.\n\n* Limited potential impact\n\nThe main observations presented in this paper are that (1) only URL metadata improves pretraining performance, and (2) other types of metadata can enhance controllability via classifier-free guidance. However, I am not convinced that these findings are substantially novel or impactful.\n\nWhile the result that only URL metadata is beneficial for pretraining could be useful for LLM development, the paper does not offer a focused analysis of this negative outcome. In addition, it has already been reported in prior work that metadata conditioning is not universally effective. The contribution would be stronger if this paper included a deeper investigation of this observation across more diverse types of contexts.\n\nSecond, although the improved controllability is a novel observation of context-aware pretraining, it is unclear whether this property is a priority for LLM developers when designing pretraining strategies, especially given the potential negative impact on downstream performance. In practice, most developers and users are primarily concerned with overall performance improvements.\n\n* Limited experiments\n\nWhile the high cost of experimentation is understandable, this paper reports results using only a single, relatively small model architecture with 1.5B parameters, trained with a single set of training configurations. It is unclear how generalizable the findings are to other settings, including larger models or different architectural and training setups.", "was_revised": true, "final_justification": "This paper presents an analysis of LLM pretraining with metadata, such as URLs, and aims to extend the investigation to broader settings. However, due to the following limitations, I consider the paper to be below the threshold for acceptance.\n\n* The paper explores two new types of metadata: quality score and topic/format. However, the motivation for focusing on these particular types is not clearly justified.\n\n* The primary contribution lies in reporting negative results, but the findings are not particularly surprising. Prior work has already shown that metadata is effective only under certain conditions. While this paper extends the analysis to additional settings where metadata is not effective, such findings add limited novelty.\n\n* The new observations have limited practical utility. Although the paper shows that quality score and topic/format metadata can improve the controllability of LLMs, it also shows that these metadata can degrade benchmark performance (Tables 1 and 2). Given the risk of performance degradation, it is unlikely that practitioners would incorporate such metadata in practice.", "rebuttal": "Thank you for your thoughtful feedback. We appreciate the time and effort you invested in reviewing our work. In the following response, we refer to Weakness point i as Wi and Question point i as Qi.\n\n**W1: diversity and rationale of the study context** \n\nWe thank the reviewer for their comment and would like to clarify a few points regarding the types of metadata considered in our study.\n\nWhile it is true that websites can contain a wide range of metadata, these are often not consistently structured or directly accessible for large-scale model training. In our work, we utilized three distinct metadata types: Quality Score and URL domain (both provided by the FineWeb-Edu dataset), and topic/format annotations, which we derived using WebOrganizer.\nTherefore, the statement that “this paper considers only two types, both annotated by LLMs” is not entirely accurate. We investigated three metadata types and explored various combinations to analyze their potential interactions and confounding effects, as detailed in our experiments.\n\nOur focus on these specific metadata types was guided by an initial empirical observation: prepending URLs to documents led to faster pretraining convergence. This raised the question of why URLs might be beneficial.\nOur first hypothesis was that URLs might implicitly signal data quality. This idea was inspired by findings in Physics of Language Models (Section 3.3, [1]), where the authors show that adding a special symbol token to high-quality data can mitigate capacity degradation caused by junk data, thereby enhancing training. To test this, we prepended explicit Quality Scores (on a 1–5 scale), but did not observe a similar acceleration.\n\nWe then hypothesized that URLs might encode information about topic and format—for example, the URL domain often reflects the format (e.g., blog, article, dataset), while the URL suffix may hint at the topic. To test this, we annotated our corpus accordingly and ran further experiments. Again, we did not see the same speed-up in training.\nThis progression reflects the reasoning behind our choice of metadata and experimental design, and we hope this clarifies our approach.\n\n**W2: potential impact** \n\nWe respectfully disagree with the statement that “it has already been reported in prior work that metadata conditioning is not universally effective.” While the study by Higuchi et al. [2], which we also cite in our paper, does raise valid concerns about the effectiveness of metadata conditioning, their conclusions were drawn from a highly controlled setting using synthetic data generated from context-free grammars. In their framework, metadata referred specifically to the generation rule used for each sample, and they found that such metadata was helpful only under limited conditions.\n\nIn contrast, our work extends this line of inquiry to **more realistic and diverse forms of metadata**, evaluated at **web-scale** pretraining levels. Given the significant differences in both methodology and experimental scale, we believe the conclusions of Higuchi et al. do not directly generalize to our setting.\n\nRegarding why only URL metadata is beneficial, we refer you to a detailed analysis in our response to **Reviewer RrS9** (**W3** part). \n\nRegarding your comment on whether improved controllability is an important consideration for LLM pretraining strategies, we agree that overall performance remains a central concern for developers and users. However, we argue that improved controllability can lead to enhanced downstream performance, as evidenced by the gains we report in Table 3 with context-conditioned generation.\n\nWe believe our work contributes a broader and more practical perspective on **an important axis of data curation**. Beyond quality filtering, leveraging overlooked metadata can further enhance the utility of training data.\n\n**W3: experimental scale** \n\nWe acknowledge the concern raised. Due to limitations inherent to our academic setting, it was already a significant challenge to complete the 1.5B parameter experiments. Each experiment on 100B tokens requires over 800 GPU hours. While we recognize that results may not fully generalize to larger-scale models, our computational resources did not permit further scaling. Nonetheless, we believe that the 1B scale remains a reasonable and informative choice. Notably, the predecessor work MeCo[3] conducted the majority of their experiments using a 1.6B Llama model, which supports our design decision. Given that their findings generalized well across models ranging from 600M to 8B parameters, we feel that our choice of scale is a reasonable and pragmatic compromise under the given resource constraints.\n\nRegarding **Q1**—why URL conditioning improves performance on general benchmarks like MMLU—we suspect this is because the model achieves a faster rate of knowledge acquisition and ultimately develops a higher knowledge capacity, as evidenced by the more rapid loss convergence. This acceleration may stem from data contextualization, which reduces the entropy of early token predictions. As shown in Figure 3 of the manuscript, the URL-conditioned model exhibits a steeper loss decline from the very beginning of training.\n\nWe hope this explanation clarifies the motivation behind our design choices and the observed results. If you find that our response satisfactorily addresses your concerns, we would be grateful if you could consider raising your score. If any questions remain, we are happy to provide further clarification.\n\n\n[1] Allen-Zhu and Li, Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws\n\n[2] Higuchi et al. When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars\n\n[3] Gao et al. Metadata Conditioning Accelerates Language Model Pre-training"}]

Author rebuttal:

Meta-review: This paper investigates the role of incorporating metadata during LLM pretraining, focusing on URLs, quality scores, and topic/format information. The authors systematically evaluate the effects of these metadata types by prepending them during training but excluding them from the loss function. Their results show that only URLs accelerate training and improve downstream performance, particularly with longer prompts, while topic/format and quality scores mainly enhance controllability of generations rather than efficiency or accuracy.

In general, the paper is well-motivated and clearly written, with careful experimental design and a systematic analysis of different metadata types. The empirical results provide useful insights into when and how metadata can contribute to training efficiency and controllable generation. However, as pointed out by the reviewers, there are some concerns about the limited experimental scale (restricted to 1.5B models and one dataset), the relatively narrow scope of metadata types studied, and the lack of deeper analysis on why URLs are uniquely effective. Despite these limitations, the paper makes a valuable empirical contribution to the understanding of metadata conditioning in LLM pretraining. I therefore recommend accepting the paper.
Decision: Accept (poster)