Title: URLs Help, Topics Guide: Understanding Metadata Utility in LLM Training
Venue: NeurIPS 2025
Accepted: yes
Keywords: Context-aware pretraining

LLM-pretraining

context guidance

controllable generation
Abstract: Large Language Models (LLMs) are commonly pretrained on vast corpora of text without utilizing contextual metadata such as source, quality, or topic, leading to a context-free learning paradigm. While recent studies suggest that adding metadata like URL information as context (i.e., auxiliary inputs not used in the loss calculation) can improve training efficiency and downstream performance, they offer limited understanding of which types of metadata are truly effective and under what conditions. In this work, we conduct a systematic evaluation and  find that not all metadata types contribute equally. Only URL context speeds up training, whereas quality scores and topic/format domain information offer no clear benefit. Furthermore, the improved downstream performances of URL conditioning emerge only when longer prompts are used at inference time. In addition, we demonstrate that context-aware pretraining enables more controllable generation than context-free pretraining, in a classifier-free guidance fashion. Although topic and format metadata do not accelerate training, they are effective for steering outputs, offering human-interpretable control over generation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies pretraining a technique to provide metadata of documents during LLMs. While it is already known that providing URLs can accelerate and enhance pretraining, there has been limited systematic analysis of which types of metadata are truly effective and under what conditions. The authors evaluate three types of metadata: URLs, quality scores predicted by LLMs, and domain information annotated by pretrained classifiers. Their experiments show that only URLs contribute to faster training, whereas the other metadata types can even degrade downstream performance. However, the authors also find that context-aware pretraining can enable more controllable generation under classifier-free guidance.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* This paper investigates context-aware pretraining, an important and broadly applicable technique for enhancing the pretraining of LLMs.\n* This paper presents an ablation study evaluating different types of contextual metadata and the conditions under which context-aware pretraining is beneficial. These findings are potentially useful for designing LLM pretraining strategies.\n* The paper offers a novel observation that context-aware pretraining can enable more controllable generation when using classifier-free guidance. \n\n### Weaknesses\n\n* Limited diversity in the studied context\n\nThere are many types of metadata that can be extracted from websites. However, this paper considers only two types, both annotated by LLMs. The diversity of the selected metadata is limited, and the rationale for focusing on these specific types is not clearly articulated.\n\n* Limited potential impact\n\nThe main observations presented in this paper are that (1) only URL metadata improves pretraining performance, and (2) other types of metadata can enhance controllability via classifier-free guidance. However, I am not convinced that these findings are substantially novel or impactful.\n\nWhile the result that only URL metadata is beneficial for pretraining could be useful for LLM development, the paper does not offer a focused analysis of this negative outcome. In addition, it has already been reported in prior work that metadata conditioning is not universally effective. The contribution would be stronger if this paper included a deeper investigation of this observation across more diverse types of contexts.\n\nSecond, although the improved controllability is a novel observation of context-aware pretraining, it is unclear whether this property is a priority for LLM developers when designing pretraining strategies, especially given the potential negative impact on downstream performance. In practice, most developers and users are primarily concerned with overall performance improvements.\n\n* Limited experiments\n\nWhile the high cost of experimentation is understandable, this paper reports results using only a single, relatively small model architecture with 1.5B parameters, trained with a single set of training configurations. It is unclear how generalizable the findings are to other settings, including larger models or different architectural and training setups.\n\nQUESTIONS:\n* Is there any intuitive explanation for the reason why URL conditioning improves performance on general benchmarks like MMLU?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRecent works have shown that adding metadata to pre-training documents can be beneficial. However, it is unclear what kind of metadata is beneficial, in what way it is beneficial, and how this depends on how we evaluate/use the trained model. This paper studies the role of metadata in more depth. They find that adding URL context speeds up training, while other metadata (quality as well as topics/formats) does not have any effect. In terms of downstream performance, URL conditioning helps more on 5-shot than 0-shot evaluation. They also find that all types of metadata are able to help more effectively steer generations---for instance, asking a question about Julius Caesar conditioned on fashion and beauty blogs results in an answer that involves makeup.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nQuality:\n- The paper presents a comprehensive study on how to incorporate metadata into pretraining, and how to utilize models that are trained in this way. \n- The finding that metadata can steer generations is quite interesting and could have implications for personalization. \n\nSignificance:\n- This sort of systematic evaluation is valuable, as the behavior of metadata conditioning is not well-understood. \n \n**Weaknesses**\n\nQuality:\n- Regarding the results comparing 5-shot vs 0-shot performance, I am not sure if the conclusion is that URL conditioning only makes a difference with longer prompts, or if there is some other effect of ICL going on.\n- Is there a training acceleration figure for MeCo? (as mentioned in line 200) it would be good to see this to validate MeCo. \n- When coming up with the metadata for a test sample at inference time, is there a scalable procedure or does it involve human annotation? For instance, suppose a user has a very out-of-distribution query; what metadata would we use?\n- What happens when we do context-aware generation with URLs/topics/formats that are not in FineWeb? I.e., is the context-conditioning based on memorization in the pre-training data, or some understanding of what specific URLs/topics/formats \"mean\"?  \n\nClarity:\n- Some results in section 4 do not reference their corresponding tables/figures.\n\nOriginality:\n- The work does not exactly propose a novel approach, but this drawback is outweighed by the systematic analysis provided.\n\nQUESTIONS:\n1. Is there a training acceleration figure for MeCo?\n2. When coming up with the metadata for a test sample at inference time, is there a scalable procedure?\n3. What happens when we do context-aware generation with URLs/topics/formats that are not in the training data?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper investigates whether incorporating metadata, such as URLs, quality scores, and domain info, during LLM pretraining improves training efficiency and downstream performance.
The paper conducts a systematic study using context-conditioned pretraining, where metadata is prepended but excluded from the loss function.
They find that only URLs help accelerate training and improve performance, particularly for longer prompts during inference.
Domain metadata, while not helpful in training speed or general performance, provides strong controllability during inference.

STRENGTHS AND WEAKNESSES:
## strength

- The paper introduces a novel experiment setup by evaluating different metadata in a controlled pretraining framework.
- The paper is well-organized with clear motivation and detailed descriptions of methods and experiments.
- The paper conducts experiments and gives empirical findings.

## weakness

- The central issue is that all experiments in the paper now use only 1.5B language models (Llama). Results may not generalize to larger models. This significantly hampers the contribution.
- The experiments only consider a single dataset (FineWeb-Edu).  Results may not generalize to more diverse corpora.
- Why URLs help more is not deeply analyzed. This may weaken the impact of the findings.

QUESTIONS:
- Can the findings generalize to larger models?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-20