When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars
Authors: Rei Higuchi
Ryotaro Kawata
Naoki Nishikawa
Kazusato Oko
Shoichiro Yamaguchi
Sosuke Kobayashi
Seiya Tokui
Kohei Hayashi
Daisuke Okanohara
Taiji Suzuki
Venue: COLM 2025 (2025)
Keywords: LLM
pretraining
metadata conditioning
TL;DR: We reveal why prepending metadata solely at pre-training sometimes helps LM's performance and sometimes hinders: it depends on whether the context suffices to infer latent semantics.

Abstract: The ability to acquire latent semantics is one of the key properties that determines the performance of language models. 
One convenient approach to invoke this ability is to prepend metadata (e.g. URLs, domains, and styles) at the beginning of texts in the pre-training data, making it easier for the model to access latent semantics before observing the entire text. 
Previous studies have reported that this technique actually improves the performance of trained models in downstream tasks; however, this improvement has been observed only in specific downstream tasks, without consistent enhancement in average next-token prediction loss. 
To understand this phenomenon, we closely investigate how prepending metadata during pre-training affects model performance by examining its behavior using artificial data. 
Interestingly, we found that this approach produces both positive and negative effects on the downstream tasks. 
We demonstrate that the effectiveness of the approach depends on whether latent semantics can be inferred from the downstream task's prompt. 
Specifically, through investigations using data generated by probabilistic context-free grammars, we show that training with metadata helps improve model's performance when the given context is long enough to infer the latent semantics. 
In contrast, the technique negatively impacts performance when the context lacks the necessary information to make an accurate posterior inference.

Published as a conference paper at COLM 2025
When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars
Rei Higuchi*
Ryotaro Kawata"
The University of Tokyo, RIKEN AIP
The University of Tokyo, RIKEN AIP kawata-ryotaro7255g ecc. www.cc...otooo.oooooo jc.
higuchi-rei714gg. wttp://rreereegge...uuooooo
ecc. www.wccokoo.ccooooooooo acc jp
Naoki Nishikawa"
Kazusato Oko
The University of Tokyo, RIKEN AIP
University of California, Berkeley, RIKEN AIP oko@berkeley.co edu
nishikawa-naoki ecc. u-tokyo ac. jp
Shoichiro Yamaguchi Preferred Networks, Inc. guguchi@preferred.i jp
Sosuke Kobayashi Preferred Networks, Inc. sosk@preferred.J jpp
Seiya Tokui Preferred Networks, Inc. tokui@preferred.
Kohei Hayashi
Daisuke Okanohara Preferred Networks, Inc. hillbig@preferred..
The University of Tokyo hayashi. kohei@gmail. com
Taiji Suzuki
The University of Tokyo, RIKEN AIP taiji@mist i. ut..i.ttokyo..c.o ac.
Abstract
The ability to acquire latent semantics is one of the key properties that de- termines the performance of language models. One convenient approach invoke this ability is to prepend metadata (e.g. URLs, domains, and styles) the beginning of texts in the pre-training data, making it easier for the model to access latent semantics before observing the entire text. Previous studies have reported that this technique actually improves the perfor- mance of trained models in downstream tasks; however, this improvement has been observed only in specific downstream tasks, without consistent enhancement in average next-token prediction loss. To understand this phenomenon, we closely investigate how prepending metadata during pre-training affects model performance by examining its behavior using artificial data. Interestingly, we found that this approach produces both positive and negative effects the downstream tasks. We demonstrate that the effectiveness be inferred from the downstream task's prompt. Specifically, through investigations using data generated by probabilistic context-free grammars, we show that training with metadata helps improve model's performance when the given context is long enough to infer the latent semantics. In contrast, the technique negatively impacts performance when the context lacks the necessary information to make an accurate posterior inference.
the approach depends on whether latent semantics
1 Introduction
Language Models (LMs) have the adaptability to a wide range of downstream tasks, backed by pre-training on datasets with diverse latent semantics (Gao et al., 2020; Arora & Goyal,
Fquul contribution

---

Published as a conference paper at COLM 2025
2023; Hahn & Goyal, 2023; Zhao et al., 2024; Zhang et al., 2025). While data from various sources and domains is one of the keys to LMs' success, it is also a cause the difficulty in pre-training, Datasets typically contain both structured, factual texts like Wikipedia (Merity et al., 2016), and unstructured, casual texts such as social media posts (Baumgartner et al., 2020). Not only the style but also the topics of the documents vary widely.
To better leverage the diversity of latent semantics, one promising solution is to condition the documents on meladata, i.e., high-level information about the source, style, or human preference of the documents that is not directly observable from the documents themselves (Keskar et al., 2019; Korbak et al., 2023; Khalifa et al., 2024). For example, Gao et al. (2025) reported that performance improved on several downstream tasks by prepending the URL or topic to each web text during LMs' pre-training, even though these cues were not used during inference. This strategy makes the model accessible to the latent semantics of data without observing the entire documents, and one might speculate that it invokes the models' ability to better capture these latent semantics, potentially leading to improved performance.
Nevertheless, it remains unclear whether providing metadata serves universal solution for LMs' pre-training. Indeed, as reported in Gao et al. (2025) improvements in downstream task performance are not consistent across tasks, and moreover, the addition of metadata does not lead to lower pre-training loss (measured by the cross entropy loss for next-token prediction). Despite the attractiveness and opacity of metadata conditioning, tracing how it influences model performance remains challenging due to the complexity of latent semantics in real-world data. As a result, the community still lacks comprehensive understanding of when metadata conditioning works.
In this paper, we systematically investigate the impact of prepending metadata during pre- training on the ability of language models. To avoid the difficulty of analysis caused by the complexity of real-world data, we conduct experiments on synthetic data with controllable and well-underston synthetic data based on context-free grammars (CFGs). We generate data using multiple CFG rules and define metadata as the information that indicates which rule was used to generate each sample. We investigate how the amount of metadata provided affects the model by conducting performance evaluations and probing analyses.
metadata. Specifically, inspired by Allen-Zhu & Li (2023), we generate
One of our key findings is that metadata conditioning can negatively affect performance depending on the task. This is intuitively illustrated in the bottom part of Figure 1.1. As shown in (A1), our experimental results revealed that when the task prompt is short, the model trained with metadata exhibits lower accuracy in inferring latent semantics compared to the model trained without metadata. This results in reduced performance on downstream tasks as in (A2). On the other hand, as shown in (B1), when the task prompt becomes longer, the model trained with metadata becomes capable of inferring latent semantics comparably to the model trained without metadata. In this case, the model trained with metadata demonstrates higher performance on downstream tasks as shown in (B2).
Our contributions can be summarized as follows:
We investigate the impact of prepending metadata during pre-training. To this end, we generate a synthetic dataset based on CFGs with controllable metadata. We conduct pre-training both with and without metadata and compare the models' performance.
We found hidden trade-off in downstream task performance: by prepending metadata during pre-training, the performance for tasks with long prompts improves, while that for tasks with short prompts does not. We evaluate the trained models using multiple metrics and demonstrate that such a trade-off arises because metadata conditioning impairs the model's ability to infer latent semantics from limited confexts.
To provide an understanding of why models trained with metadata fail to infer latent semantics and how affects the downstream performance, we introduce a theoretical framework using predictive distribution with respect to latent semantics.
2

---

Published as a conference paper at COLM 2025
Root
Texts only
Texts + motadata
ourgerfoomm
LLanguage Modelsi Atten-
JUNNs...
outperforms RNNs
Subject
Verb
Object RNNs brand recall stimuli
Training
Assention drives brand recall.
[Advertising] Attention drives brand recall.
drives
Attention
filters
Inference
Attention Elters stimalii
MASKED Actention fitems stnmali
outperforms
Ability to eslimaae latent. semantles
(BI)
(B2)
Downstream performanos
w/ metadata
w/
metadata
w/o metadata
w/o metadata
(A1)
(A2)
Context length
Context length
Figure 1.1: Overview and findings. Top: Conceptual illustration of how metadata narrows
latent semantics and how it is incorporated into training data. Bottom: Positive and negative
effects of using metadata on estimation ability of the latent semantics and downstream task performance.
Synthetic Dataset with Controllable Metadata
We constructed a dataset using probabilistic context-free grammars (PCFGs) that are sys- tematic and controllable Allee-Zhu & Li, 2023; Ahuja et al., 2025). We extend the D-Level PCFG introduced in Allen-Zhu Li (2023) by incorporating metadata.
2.1 D-Level PCFG
First, we explain sentence generation using PCFG. A PCFG denoted by G, which consists of symbol sets {S and production rules [R.) D-1 Specifically, we start with a sequence of length one consisting of unique root symbol E Sg, and, for i 0, D 1, sequentially generate next sequence of symbols of Si+1 from the previous sequence of S, Each production rule R, has length of 2 or 3, and consists of rules the form:
-= (51 + $2, 84) or F ($) + $3) for $1 S and $2, $3, S (2.1)
i-th step, we replace each symbol in the previous sentence according to the production rule R, to generate the next sentence. By repeating this process D times, we obtain the final sequence consisting of symbols in Sp, which we refer to as terminal symbols.
For simplicity, the symbols at the different levels are assumed to be disjoint (i.e., S,n for i * i). The distribution generated with PCFG G is denoted by P(G) and its support is denoted by L(g). The procedure for sentence generation is summarized in Algorithm 1.
Algorithm Procedures for sentence generation
1: Set the initial state No S E So-
2: for / = .,..,,... D do
3: Keep X; (s, 5i,....,,,,kk111 -1) where di E St for O,...,
4: for k = ., K; do
5: Randomly sample a rule R; whose input is $i, with uniform probability. Replace $;, by the right-hand side of the rule (e.g., $3, or $2, $3).
6:
7:
Output xp
where 6i, E Sp.
3

---

Published as a conference paper at COLM 2025
Rap
Res
B
Layer
B
R1
01 10
Rga
R5,
01 or 001
00)
10
001
Raa
Layer 2
00 111
Figure 2.1: An example of 2-Level CFG with hierarchical metadata (fo-ji = (0,1). Sets
of non-terminal symbols are So {S), S {A, and terminal symbols are S₂ {0,1}. The grammar (((0,,)) 1)) ([S)) (Ro,o, H. The support ((9( 1)))
11101,
2.2 D-Level PCFG with hierarchical metadata
We now introduce the notion of metadata in the context of a D-level PCFG, and describe how to generate a sequence under metadata constraints. The overview shown in Figure 2.1.
As we conceptualize metadata as high-level information underlying the sentence generation process, we operationalize
a set of meta-level information that modulate the produc- tion rules {R, Specifically, we suppose that there are multiple choices (Rij) of each production rule R₁. At each level i, we select one of these rules, whose index is represented by ji- We formally define the metadata as the sequence of indices (Jo.--.,Ju ..Ii correspond- ing to the selected production rules. Once the rule sequence {Ri,,, Ji is determined, the
PCFG grammar is constructed as ((jo.....,jj 1)) generate a sequence. Note that different sequences use different sample of (jD-....jD-I).
((S,), {Rijj 100 from which we
In this way, within the PCFG framework, we define metadata meta-information that specifies the choice of production rules. Although metadata directly defines the content of the generated sentence, it constrains the possible sentence structures compared to the full of rules f R This corresponds how, in real-world settings, labels such as source, style, or human preference roughly partition the space of plausible sentences.
2.3 Generation of data
Finally, we describe how datasets are generated. To investigate how the amount of metadata provided affects the learning of LMs, manipulate the level-wise availability of metadata labels j; in PCFG. We refer to providing metadata up to depth DM (S D) as supplying the partial metadata sequence (jo.... (jo.....jDw. DM quantifies the extent of available metadata, whereas which the model must navigate the rule space without metadata guidance. In practice, this corresponds to controlling the extent of specificity with which sentences are conditioned.
from the full metadata (jo.....iD 1). Roughly,
D DM represents the range over
To provide metadata up to depth DM, we prepend DM metadata tokens (Jo. (Jo..... 1) followed by D DM masked tokens to the sentence P(((S)) (Ri;)) The training dataset constructed by mixing sentences aug- mented with metadata as described above and sentences prefixed with masked tokens with equiprobability. For further details, please refer to Appendix A. The inference dataset, as well as the metadata-less training dataset used for comparison, is constructed by sim- ply prepending D masked tokens to each generated sequence. The purpose of inserting non-informative masked tokens is to maintain a constant sequence length while varying the amount of provided information. We take care to control this aspect, as prior work has shown that even neutral tokens, such as sink tokens, may influence learning (Darcet et al., 2024; Gu et al., 2025).
P(G j)--)))

---

Published as a conference paper at COLM 2025
3 Experiments
We now investigate when and how metadata conditioning affects model behavior through experiments on multiple tasks and metrics. We choose our Transformer model by following Allen-Zhu & Li (2023), who similarly trained on CFGs. Specifically, we adopt the LLaMA architecture (Touvron
al., 2023) with 12 layers, 12 attention heads, RoPE (Su al., 2024), and a hidden size of 768. The data generation process, described in Section 2, uses PCFG with hierarchical level D = 5. Each R2 is selected from one of Ri0 or R1, with equiprobability Only two candidates per level may appear limited, but each metadata label exponentially narrows down the set of possible grammars. Roughly speaking, full metadata reduces the ambiguity in the generative process by a factor of 2 = 32 compared to the metadata-free case. Therefore, this setup is enough to meaningfully examine the influence of metadata.
We compare the results by varying the depth of prepended metadata DM E (0,5, where larger DM means richer conditioning. We pre-train the model using next-token prediction (NTP) with cross-entropy loss. The loss is computed only for terminal symbols and the EOS token, excluding the BOS token and metadata. Therefore, the only difference among training setups is whether metadata is included, and no additional supervision is provided. See Appendix A for further discussion.
3.1 Training with or without Metadata Does Not Affect Next-Token Prediction Loss
Test loss without metadata. As a preliminary result, we examine next-token prediction performance. The left of Figure 3.1 shows the test loss across training, evaluated under inference without metadata. Although our dataset more complex than that of Allen-Zhu & Li (2023) due to hierarchical branching, the models reduce the loss well. Notably, we find no significant difference in the loss between the models trained with metadata (DM 2 1) and without it (DM = 0), which aligns with the insight from Gao et al. (2025).
Test loss with metadata. The right of Figure 3.1 shows the loss when inference is performed on data with prepending metadata. We can that it is substantially smaller than the loss in the left figure (in- ference without metadata). This indicates that models trained with metadata conditioning can indeed perform better during inference, if metadata is available. This ob- servation contrasts with the case where metadata is not used at in- ference.
over steps
w/ metadaia over steps
=
Dum3
5
1055
Dy=0
Dpm1
Dy=
Dy
032
Steps
Steps
Figure 3.1: Test loss of next-token prediction during
training. Left: without metadata. Right: with metadata.
We summarize these results in the following important observation
Observation. For average next-token prediction loss, metadata conditioning improves performance only when metadata is available at inference time. If metadata is not accessible at inference, no significant improvement is observed.
3.2 Training with Metadata Degrades the Accuracy of Metadata Prediction
Average next-token prediction loss provides only a coarse measure of model behavior and fails to capture fine-grained differences. This limitation makes it difficult for previous studies to understand the mechanism of metadata conditioning. However, our controlled setup enables more detailed investigation. To this end, to examine how well the trained model can properly infer the latent semantics from the document, we consider a probing
We also examine calibration errors Appendix which further supports this finding.
5

---

Published as a conference paper at COLM 2025
Dw=0
Dum1
Dym3
0
CFG Layer 0-0
CFG Layer 0-1
CFG Layer 0-2
CFG Layer 0-3
CFG Layer 0-4
Accuracy
0.75
0.50
0.25
100
150
100
150
Taken Position
Figure 3.2: Metadata-probing accuracy. We plot the transition with respect to the token position. We used the 12th layer of a Transformer and each column corresponds to the depth of the CFG tree. The d-th column (d 3, 4,5) represents the accuracy of classifying branches of the CFG tree from layer 0 up to layer d - 1. The chance rate of correctly predicting the metadata from layer 0 to layer d is 1/24.
task (Conneau et al., 2018; Belinkov, 2022; Allen-Zhu & Li, 2023) that predicts the metadata in order to investigate whether the trained models can properly extract the latent semantics.
Details of probing. Let E {0....11) 11) be a layer index of a trained model, and e {o,..., D } be a level of the PCFG rule. Given an input generated from (()o.....,JD 1)), let h denote the average of hidden states over consecutive tokens starting from position i at layer For each (I,d,i), we train d distinct linear binary classifiers (with all model parameters frozen); the d' classifier predicts the metadata iæ from hi, We define metadata- probing accuracy as the rate at which all classifiers return the correct labels. The hidden states of later tokens generally aggregate information over longer ranges of the input, and thus tend to yield better probing performance. We therefore evaluate accuracy across various positions/ to understand how much context is required to recover the metadata. We also conduct probing across multiple layers (specifically, the 1st, 5th, 8th, and 12th layers) to gain comprehensive view of how metadata information is distributed throughout the network, while we will present the result only for the 12th layer since the results from other layers exhibit similar results. The results for the other layers can be found in Appendix D.
Results. The results are summarized in Figure 3.2. Before seeing the results, one might intuitively expect that a model trained with metadata conditioning would associate latent semantics with document content, thereby improving metadata probing accuracy. In fact, however, metadata conditioning worsened label-probing accuracy rather than improving it. In particular, when the prompt length is short, the accuracy of models trained with metadata is notably worse than that of models trained without it. When metadata of depth 3 or 5 prepended during pre-training, the accuracy at prompt length (token position i 0) is around chance level. This indicates that models trained with metadata cannot infer latent semantics during inference if the prompt provides too little information.
In contrast, as the prompt length increases, models trained with metadata become capable of accurately predicting the metadata. Indeed, the large accuracy gap (between DM = and DM = 3)5) seen at prompt length 5 largely disappears at prompt lengths of 50 100. This suggests that models trained with metadata can extract latent semantics from the sequence, as long the prompt is sufficiently long to provide the necessary information. In summary, we obtain the following finding through the probing:
Finding 1. Metadata conditioning leads models to fail to infer latent semantics. In particular, the prompt length is short, models trained with metadata conditioning cannot infer latent semantics, while models trained without metadata can. When the prompt becomes longer, information in the prompt increases, which allows both types of models to better capture latent semantics.
In Section 4, we provide a theoretical framework that explains why such counterintuitive performance degradation occurs.

---

Published as a conference paper at COLM 2025
Table 3.1: Grammatical accuracy for various prompt length.
depth of metadata
length of prompt
1
5
10
25
50
0.000 1 0.000
0.810 E 0.059
0.834 1 0.036
0.816 1 0.027
0.834 1: 0.019
1
0.200 + 0.400
0.802 + 0.159
0.820 + 0.065
0.884 + 0.027
0.860
0.018
0.000 = 0.000
0.072 + 0.088
0.884 + 0.036
0.878 - 0.029
0.856
0.019
= 0.019
5
0.000 = 0.000
0.000 = 0.000
0.740 = 0.045
0.830 = 0.030
0.824 = 0.034
3.3 Training with Metadata Degrades the Downstream Performance for Short Prompts
Finally, we analyze the effect of metadata conditioning on downstream task performance. Although previous studies such as Gao et al. (2025) report improvements in downstream tasks, they do not offer detailed explanation for this effect. To gain deeper understand- ing, we perform sequence generation and employ a metric called grammatical accuracy, introduced by Allen-Zhu & Li (2023), which measures how accurately a model generates grammatically correct sentences.
Details of grammatical accuracy. Here, we describe how to calculate the metric. First, we randomly provide model the initial part of sequence and evaluate whether it can generate a grammatically correct sequence. Specifically, we first obtain i.i.d. sample sequences from L(9), where G := ([S)) ( R Poo1) represents the mixture PCFG. Then, we extract the first lp tokens (x,,1r..,, ).I,,,,,,,,,))... as prompts and let the model generate token sequences autoregressively until outputs [EOS]. We denote the generated sequences by (yipp+1........ MMMM, and we define == "Ki,,,..,,I,,,, Xil,, Vilptl..... ,i,M, for i ,,. Using Z; (i I,) we define the grammatical accuracy (GA) as GA(g)
Σ [[(] The value 1[z; E L())), i.e., whether z; is correct sequence of grammar G, can be determined by the (deterministic) CYK algorithm. As seen in the previous subsection, the performance of models trained with metadata is highly dependent prompt length. Therefore, we present this metric across various prompt lengths Ip.
2
Results. The results are shown in Table 3.1. Surprisingly, seemingly contrary to the findings of Gao et al. (2025) that showed improved performance on downstream tasks, metadata conditioning led to decreased grammatical accuracy when the prompt length was short. However, as the prompt length increases, models trained with metadata conditioning recover, achieving performance comparable to or better than those trained without metadata.
This pattern is reminiscent of Finding 1; models trained with metadata struggle to infer latent semantics from short prompts, resulting in degraded task performance. On the other hand, when prompts contain sufficient information, models successfully infer metadata. As previously noted (Observation), eettadata-conditionn metadata is explicitly available; thus, metadata inferred from richer prompts similarly enhances downstream task performance. These results lead to the following finding:
models achieve lower loss when
Metadata conditioning degrades the downstream task performance if the task prompt is short. This is because models trained with metadata cannot infer latent semantics from the short prompt. On the other hand, for tasks with long prompts, metadata conditioning improves the performance since the models can easily extract latent semantics from the prompts.
Finding 2.
4 Interpretation of Experimental Results and Findings
We compare training with and without metadata in how the model implicitly learns latent semantics. Without metadata, the model needs to learn the hierarchical mixture of grammars directly from data, which is challenging due to the diverse structure of possible grammars.
7

---

Published as a conference paper at COLM 2025
With metadata, grammar inference becomes easier, but accurate prediction of the metadata itself becomes necessary. We highlight this trade-off in inference and learning.
4.1
A framework for Latent Semantics Acquisition via Marginalization in the Presence or Absence of Metadata
4.1.1 Training without metadata
We begin by formalizing the role of latent semantics when training is performed without metadata. Given the prompt x, the predictive distribution of the output can be decomposed as
p(y|x)
p(y|x,91)
p (( (x) dg1,
Likelihood
Posterior
where 9L (e.g. PCFG) is latent semantics. For simplicity, we focus on the situation where 91. is PCFG. This decomposition from the principle of marginalization means that the output y can be viewed as being generated through two steps:
(A) Sampling PCFG (latent semantics) 9L from posterior p()
(B) Generating y conditioned on both x and 9L using the likelihood function p(y|x, 9L..
When trained without providing the metadata j. the model implicitly approximates the likelihood p(y|x,90 91) and the posterior p(9 internally. Prediction of p(L(|x) is not trivial even if we know the candidates for PCFG (G) jE {) because we get samples from different hierarchical CFGs: The total support of the text data is Ujc(0.11 (((())).
4.1.2 Training with metadata
On the other hand, when we train the model with metadata j = (Jo. .......DD jo-1). the predictive distribution of the output y in inference can also be decomposed as
p(yxx) / p(ylx,j)
/
xf]x
dj
((L1x,,)
dg
p(jxx)
dj.
Liikelihodd training w/
Posterior of
Likelihood
Posterior jiven
Posterior of
Likelihood in training with
G1.) in this setting (we do not need j once we know the grammar 91.. This decomposition can be expressed in three steps:
Note that p p(y|x,,9,f)) = p(ylx, (1)
(A-1) Sampling metadata (jo. jo-1)) from its posterior p(](x).
(A-2) Sampling PCFG GL from its richer posterior given metadata p(GLlx,f).
(B) Generating conditioned on both x and GL using the likelihood function p(y|x, GL).
In this case, the model focuses on the tasks (A-2) and (B) because the information is provided. Therefore, [[y]x,,] is implicitly constructed in the model. As the training dataset includes both samples with and without metadata, (A-1) and (A-2) are learned separately.
4.2 Understanding Trade-offs in Metadata Conditioning through a Formal Framework
The proposed framework highlights both positive and negative effects of metadata condi- tioning on inference:
/ The advantage of metadata conditioning is that learning the richer posterior given metadata ((,,,x,,, f) is facilitated. Once we obtain the candidates for CFG (9()) je {o, [0,1)), predicting (() in (A-2) becomes much easier than estimating (()1(x)
in (A).
8

---

Published as a conference paper at COLM 2025
X The drawback of metadata conditioning is that the estimation performance of poste- rior p(j|x) in (A-1) deteriorates. When inference is performed without metadata, the posterior of the latent semantics p( = ( (jpx)//j degenerates, making it more difficult to capture the latent semantics accurately.
These frameworks allow us to interpret the previous experimental results as follows:
(i) Interpretation of Observation. With metadata conditioning, prediction of p(j|x) is less is more efficient than without metadata. implies a trade-off in prediction of pfj(x) and f)), Therefore, the average next token prediction loss remains consistent regardless of whether metadata conditioning is used.
efficient, but prediction of p |x,
(ii) Interpretation of Finding 1. We found that, training with metadata, prediction of p(jlx) becomes degenerated in Finding 1. This negatively affects prediction of p( as discussed in (i). However, when the task prompt x is sufficiently rich, predicting p(jxx) is easy and the negative effect of metadata conditioning on prediction of p( (x) becomes small.
(iii) Interpretation of Finding 2. When the prompt is limited, it is difficult to predict x) as mentioned in (ii). In this case, the no-metadata model predicts p(91,xx) more accurately and achieves higher grammatical accuracy (downstream performance). When is rich, predicting p(jlx) becomes easier. In this situation, there is no difference between the no-metadata and metadata settings in how well p(j(x) is learned. As the metadata- conditioned model efficiently predicts p(L[|x,)
xx, j), it can implicitly construct p( = (() more accurately. Therefore, the with-metadata setting achieves higher grammatical accuracy.
Related Work
Control of Generated Text: While instruction-tuned models can follow user-written prompts in natural language, relying solely on such instructions often leads to ambiguity or inconsistent outputs. As complementary direction, Keskar et al. (2019), Chan et al. (2020), Dathathri et al. (2019), and Aghajanyan et al. (2021) address this by training models with metadata encode attributes such as domain, style, task, or source of information. Korbak et al. (2023) incorporate preference scores derived from human judgments as conditioning signals, resulting in reduced toxicity even in unconditional generation. After such training, metadata can be used to more reliably steer the model toward generating text in specific genre or a writing style.
Efficiency and Performance Improvement in LLM-training: L.M training methods that incorporate multiple pre-training and post-training stages with diverse forms of labeling have demonstrated improved downstream performance. For example, Aghajanyan al. (2021) leverage hyper-textual structures, Gao et al. (2025) integrate metadata such as URLs, and Korbak et al. (2023) utilize human preference scores during pre-training. Zhang al. (2024) reduce catastrophic forgetting during post-training by conditioning on short topic- related hints. Allen-Zhu Li (2024) show that training with a special token marking "junk" documents enables the model to allocate its capacity more efficiently. Krasheninnikov et al. (2023) demonstrate that tagging the source reliability helps the model acquire implicit meta-learning capabilities and prioritize high-quality information during fine-tuning.
These studies suggest that metadata can enhance not only controllability but also perfor- mance. However, how such conditioning influences model Satavior-paattculaarly when metadata is used at pre-training but unavailable at inference- has remained unclear. Our study directly addressed this issue through controlled experiments using PCFGs inspired by Allen- -hu & Li (2023) and revealed that the benefits critically depend on the relationship between latent semantics and the information provided by prompts at inference time.

---

Published as a conference paper at COLM 2025
6 Conclusion
In this study, we systematically investigated the impact of prepending metadata during language model pre-training, utilizing synthetic data generated from context-free grammars with controllable metadata. By comparing models trained with and without metadata, and guided by theoretical framework, we aimed elucidate the conditions under which metadata conditioning affects model performance.
Our key finding reveals a trade-off: metadata conditioning enhances performance on downstream tasks with sufficiently long task prompts that allow for the prediction of latent semantics, but it degrades performance on tasks with task prompts too short to predict latent semantics. These findingg highlight the need for more nuanced approach to metadata conditioning in practical pre-training, offering a crucial practical guideline: simply maximizing metadata not always beneficial. Instead, strategies should be tailored based on the expected nature of downstream tasks. Conditioning with excessive metadata beyond what is potentially inferable from the typical prompt can be counterproductive for downstream tasks.
Future work should focus on developing pre-training strategies that mitigate this trade- off, potentially by adaptively adjusting the use of metadata based on task complexity. Additionally, exploring the impact of different metadata types on model performance will provide valuable insights into optimizing metadata conditioning for diverse applications.
Acknowledgments
NN was partially supported by JST ACT-X (IPMJAX24CK) and JST BOOST (PPMJBS2488). RK and RH were partially supported by JST CREST (PMJCR21155. TS was partially supported by JSPS KAKENHI (24K02905) and JST CREST (PPMJCR2015) This research is supported by the National Research Foundation, Singapore, Infocomm Media Development Authority under its Trust Tech Funding Initiative, and the Ministry of Digital Development and Information under the AI Visiting Professorship Programme (award number AIVP- 2024-004). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore, Infocomm Media Development Authority, and the Ministry of Digital Development and Information.
10

---

Published as a conference paper at COLM 2025
References
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. Htlm: Hyper-text pre-training and prompting of language models. arxiv preprint ar axXi::2107.06955, 2021.
Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar, Tianxing He, Noah A. Smith, Navin Goyal, and Yulia lsvetkov. Learning syntax without planting trees: Understanding hierar- chical generalization in transformers. Transactions of the Association for Computational Lin- guistics, 2025. URL htt.:ttp////m /ettret//irrctt..t.t www.mmitteeeaa.. http///tcl/attcleeefotooao 11t.1/.1111/aa1111/ 00733/ 127877.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 1, learning hierarchical language structures. ar Xrv preprint ar arxt:22305.13673 2023.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint ar arXiv:2444.55405 2024.
Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint ar Xriv:2307.15936, 2023.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The pushshift reddit dataset. In Proceedings of the international AAAI conference on web and social media, volume 14, pp. 830-839, 2020.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1), March 2022. doi: 10.1162/coliaa.00422 URL https://aclantholoogy org/2922.c1-1.7.
Alvin Chan, Yew-Soon Ong, Bill Pung, Aston Zhang, and Jie Fu. Cocon: A self-supervised approach for controlled text generation. arXiv preprint ar Xii::2006003333 2020.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into single $&!# vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), July 2018, doi: 10.18653 20118833/v1PPL /t////v1//P////// 111888 URL https: //aclanthology org/P18 1198/.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024.
URL https://openreviewt wwwpeeeeeweeeeeereneeeeieeee www.eee/orummiic222nni1111
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint ar 1rXio:1912.02164, 2019.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, arXiv preprint arXiv:2101.00027, 2020.
Tianyu Gao, Alexander Wettig, Luxi He, Yihe Dong, Sadhika Malladi, and Danqi Chen. Metadata conditioning accelerates language model mr-ttrining.. In Forty-second Interna- fional Conference Machine Learning, 2025.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview. wttpeeewwwwwwwweeeeeeeee www.:neefooumiiime net//Forumiid-7aanAATEE..
Michael Hahn and Navin Goyal. A theory of emergent in-context learning as implicit structure induction. Xiv preprint arXin:2303.07991, 2023.
11

---

Published as a conference paper at COLM 2025
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. ar Xiv preprint arXiv:1909.05858, 2019.
Muhammad Khalifa, David Wadden, Emma Strubell, Honglak Lee, Lu Wang, Iz Beltagy, and Hao Peng. Source-aware training enables knowledge attribution in language models. arXiv preprint arXiv:240..01019, 2024.
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pp. 17506-17533. PMLR, 2023.
Dmitrii Krasheninnikov, Egor Krasheninnikov, Bruno Mlodozeniec, Tegan Maharaj, and David Krueger. Implicit meta-learning may lead language models to trust more reliable sources. arXiv preprint arXio:2310.15047, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in neural information processing systems, 34:15682-15694 2021.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomput, 2024. doi: 10.1016/ 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXio:2302.13971, 2023.
Chi Zhang, Huaping Zhong, Kuan Zhang, Chengliang Chai, Rui Wang, Xinlin Zhuang, Tianyi Bai, Qiu Jiantao, Lei Cao, Ju Fan, Ye Yuan, Guoren Wang, and Conghui He. Harnessing diversity for important data selection in pretraining large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview. net net/forum?id=bbCC t7eLRc
Xiao Zhang, Miao Li, and Ji Wu. Conditional language learning with context. arXiv preprint XXXi::240600197, 2024.
Yang Zhao, Li Du, Xiao Ding, Kai Xiong, Zhouhao Sun, Shi Jun, Ting Liu, and Bing Qin. Deciphering the impact of pretraining data on large language models through machine unlearning. In Findings of the Association for Computational Linguistics: ACL. 2024, 2024. URL hnttps://acla. http://aaaaanholgy..ooo 2024. findings-acl 559/
12

---

Published as a conference paper at COLM 2025
A More Details of the Experimental Setting
Training Setting. The number of data samples used for pre-training is 5,000,000, and the training is conducted over a single epoch. We note that the difference in results between the models with and without metadata not due to overfitting, For batch size, optimizer, and learning rate, we followed the setup in Allen-Zhu Li (2023). Specifically, we used a batch size of 96, AdamW optimizer with of 0.0003.
weight decay of 0.1, and learning rate
Data Generated from D-Level PCFG. Here, we provide a full explanation of the data generated by the procedure described in Section 2.3. The resulting input sequence during training with metadata is given by
BOS]
jo h
JD-1
$D,0
SD,1
SD
[EOS].
(A.1)
Here, Jo....... .o are the tokens that sometimes provide metadata and otherwise become mask tokens. Specifically, they are given as
jo
JDM-1
MASK
[MASK]
with probability 0.5,
Jo h ... JD-1= =
DM times
[MASK]
MASK
with probability 0.5,
times
where j; (i 0,..., D represents metadata and DM represents the depth up to which the metadata is fed into the model.
B The loss of next-token prediction for various token positions.
Table B.1: The loss of next-token prediction for various token positions.
Position of the tokens
depth of metadata
0
25
50
0 (no metadata)
0.9224
0.8867
0.7244
0.7471
0.7483
1
0.9224
0.8867
0.7246
0.7474
0.7484
3
0.9224
0.8867
0.7248
0.7472
0.7479
5
0.9224
0.8866
0.7248
0.7476
0.7479
The effect of prepending metadata during training metadata-probing accuracy and grammatical accuracy varies depending on the prompt length. is natural to ask whether similar phenomenon occurs in a pre-training task, that is, whether the performance of next-token prediction changes depending on the position within the sequence. Interestingly, not observed in next-token prediction. Indeed, shown in Table
such phenomenon B.1, the loss values calculated for each position also showed little variation across the depth of metadata, just the overall loss remains almost unchanged.
13

---

Published as a conference paper at COLM 2025
C Calibration of the trained models
Calibration Loss
0.010
D2
w/o metadata
w/ metadata
0.008
D2 =1 1
1.0
Di
Dj =
0.8
0.006
Accuracy
Loss
0.6
0.004
0.4
0.002
0.2
0.000
0.0
0.5
1.0 0.0
0.5
1.0
10000
Steps
Confidence
Confidence
Figure C.1: Left: Change in calibration loss for test data (w/o metadata) with respect to the number of pre-training steps. Right: Calibration histogram at the final step of pre-training.
To reinforce the evidence that metadata conditioning does not affect the pre-training task when not prepending metadata, we investigate whether the models capture the distribution correctly. For this purpose, we introduce the metric called expected calibration error, which is defined in Minderer et al. (2021).
We provide the definition of the metric here. We first consider dividing the interval [0, 1] into /I sub-intervals: ( ",).....,(())))
2)/m, (mn - 1)/m), [(m 1)/ml m, We then define B; (i 1,..,, m) as the set of all tokens in the test data for which the model's confidence falls within the i-th interval defined above. Expected Calibration Error (ECE) is defined as follows:
m
B |accuracy(B); - Confidence(B:)]
ECE =
Σ
where accuracy (B;) and confidence B,) denote the model's accuracy and the average con- fidence for these tokens, respectively. Intuitively, ECE measures the discrepancy between the model's accuracy and confidence. Therefore, a well-calibrated model that accurately predicts distributions is expected have an ECE value close zero.
We present the expected calibration error curve on the left side of Figure C.1. In both cases with and without metadata, we observe decrease in expected calibration error, and there was no significant difference in calibration. The middle and right plots in Figure C.1 presents a histogram showing the average accuracy for each confidence interval. This also confirms that the model's confidence aligns with its accuracy, and the presence of metadata during training does not largely affect the model's calibration, as is the case with the test loss.
D More Details of Metadata-probing
Training of Probing Classifiers. For training and evaluation of the classifier, we used 10,000 data points different from those used in pre-training. Specifically, 7,000 data points are used for training, 1,500 for validation of early stopping, and the remaining = 1,500 for evaluation.
Results for other layers. As mentioned in Section 3.2, conducted probing experi- ments on various layers. We omit the full results in the main paper, and present it in Figure D.1. We can observe that the results for the layers other than the 12th layer have similar tendencies to the result provided in Section 3.2.
14

---

Published as a conference paper at COLM 2025
Dy-0
Dy-1
Dwm3
DH-5
TF Layer 0
TF Layer
Layer 7
TF Layer
C43 sas 7-0 C7 2a 1.0
0.75
0.50
0.25
Accuracy
0.75
0.50
CFG 19
0.2
0 3V 02
0.2
50
100
150
50
100
150
50
100
150
50
100
150
Token Position
Figure D.1: Supplementary results of metadata-ppobiiee accuracy corresponding to Figure 3.2. We plot the transition with respect to the prompt length. Each column corresponds to Transformer layer index (1, 12), and each corresponds the depth of the CFG tree. The d-th row (d = 1,2,3, 4, represents the accuracy of classifying branches of the CFG tree from layer up to layer layer 0 to layer
The chance rate of correctly predicting the metadata from
1/2d
15


Reviews:
[{"review_id": "zsmJElLX49", "reviewer": "Reviewer_d5Xy", "summary": "This paper investigates the impact of prepending metadata during language model pre-training using controlled experiments with artificial data generated from context-free grammars. The authors examined why previous research has shown inconsistent benefits from metadata conditioning, where improvements appear in some downstream tasks but not in the average next-token prediction loss. \n\nSpecifically, first, the researchers create synthetic data using probabilistic CFGs, where metadata represents information about which production rules were used to generate each sequence. Second, they train transformer models under different conditions, varying the metadata provided during pre-training. Finally, authors provided a theoretical framework explaining this phenomenon: models trained with metadata become better at predicting outputs given both context and metadata (p(y|x,j)) but worse at predicting metadata from limited context (p(j|x)). This creates the observed trade-off in performance depending on prompt length.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well-designed controlled experiments: The use of synthetic data generated from CFGs allows for precise control over latent semantics and enables systematic investigation of how metadata conditioning affects model behavior.\n\n- Evaluation: The authors use multiple metrics (next-token prediction, metadata-probing accuracy, grammatical accuracy) to evaluate model performance, providing a more complete picture of metadata's effects.\n\n- \"Theoretical\" analysis framework: The paper develops a solid mathematical framework explaining why the observed effects occur, connecting empirical findings to theoretical understanding.\n\n- Findings are interesting: The research provides actionable insights for practitioners, suggesting that the amount of metadata used during pre-training should be tailored to the expected context length in downstream tasks.", "reasons_to_reject": "My major concern is the grammar complexity. The experiments use a relatively simple grammatical structure with binary choices at each level. While this simplification is justified for controlled experiments, it raises questions about generalizability to more complex real-world language structures. To that point, most of the data created or tasks are synthetic, hence I am not so sure how this method will generalize to real-world tasks.", "questions_to_authors": "See reasons to reject", "extra_scores": {}}, {"review_id": "r1rdOlUynH", "reviewer": "Reviewer_dKwt", "summary": "- The paper studies on \"metadata preconditioning\", a phenomenon where prepending metadata to pre-training sequences seem to improve language modeling, specifically downstream performance [1].\n- The paper proposes to study this phenomenon via $D$-level probabilistic context-free grammar (D-PCFG), a framework proposed in earlier work [2] that can serve to generate controlled training sequences for language models.\n  - Specifically, under D-PCFG the authors propose to use the selected production rules of the CFG as the \"metadata\" for the training sequence\n- The paper then performs a series of experiments using the above controlled setup using D-PCFG, studying:\n  1. how metadata preconditioning affects the test loss (finding: it doesn't affect it)\n  2. whether metadata preconditioning helps models learn representations (hidden states) that contain information about the metadata, by using trained probes (finding: it doesn't help, and in fact makes probes worse)\n  3. how metadata preconditioning affects downstream performance, as defined by the model's ability to generate grammatically correct sequences (finding: metadata preconditioning hurts if prompt is short, but helps if prompt is long)\n\nRefs\n- [1] https://arxiv.org/abs/2501.01956 \n- [2] https://arxiv.org/abs/2305.13673", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A1. The paper studies a relevant problem, as a direct follow-up of Gao et al. (2025) [1] using the techniques from Allen-Zhu and Li (2023) [2].\n- A2. The paper is well-written and well-executed. Experimental setup and takeaways are clear. The formatting of the paper is nice and clean.\n- A3. The analysis provided in section 4 provides useful information of the conducted experiments.\n- A4. The paper is strong along the \"Understanding Depth, Principled Approach\" dimension for reviewing guidelines (https://colmweb.org/ReviewGuide.html) \n\nRefs:\n- [1] https://arxiv.org/abs/2501.01956 \n- [2] https://arxiv.org/abs/2305.13673", "reasons_to_reject": "R1. As with other attempts to reduce real-world natural language modeling into toy tasks, a general weakness of this work is that it is unclear why the result necessarily transfer to language modeling for natural text.\n  - Yes, the PCFG + metadata construction is intuitive, but the exact correspondance to real-world data seems unclear\n  - That is, for the claim in the conclusion (\"metadata conditioning enhances performance on downstream tasks with sufficiently long task prompts that allow for the prediction of latent semantics, but it degrades performance on tasks with task prompts too short to predict latent semantics.\"), the paper justifies why this is the case for D-PCFG and makes an inductive reasoning step that this phenomenon must also extend to natural language. My concern is that this inductive step isn't valid. \n\nR2. (As a result of R1) The paper would benefit from providing more examples and more intuition on the problem. \n- While section 4 provides interpretations on the D-PCFG results, I'd be more interested in intuitions that map the provided setup to a realistic, natural language task. \n\nR3. Even within the D-PCFG setup, it seems reductive to equate \"Downstream Performance\" in natural language modeling to \"grammatical accuracy\". I.e., the task setups feel contrived and detached from real-world settings. \n- For example, one counter-argument is that Gao et al.  measures \"downstream performance\" on multiple tasks, whereas in this paper's setup there is only one possible downstream task", "questions_to_authors": "My main question would be whether there's anything else that the authors could add to make clearer the correspondance from the toy tasks to the real-world language modeling setting.\n\nMinor:\n  - L57: \"As shown in (A1)\" --> perhaps better explicitly refer to the bottom left plot", "extra_scores": {}}, {"review_id": "HROSHDWUFl", "reviewer": "Reviewer_StBC", "summary": "The paper investigates the effect of metadata conditioning in training language models. The authors conduct a series of experiments using probabilistic context-free grammars (PCFGs) to analyze how prepending metadata in training data influences the model's ability to learn semantic structures. They find that metadata conditioning improves performance in downstream tasks when task’s prompt is long, but degrades performance when the prompt is short and latent semantics cannot be inferred.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides an in-depth analysis of the effect of metadata conditioning, highlights the conditions under which it helps learning semantics and when it does not. This is a valuable contribution to understanding the role of metadata in training language models.\n- The paper presents both clear empirical results with controlled interpretable data and a theoretical analysis, providing novel insight into the underlying mechanisms of metadata conditioning.", "reasons_to_reject": "The experiments with PCFGs, although well-designed, still falls short in experimental rigor. Hyperparameters that could affect the results are not sufficiently explored, making it hard to draw strong conclusions from the results.\nSome improvements to the experiments could strengthen the conclusions:\n- Model size: due to the well-known phenomenon of \"emergent abilities\" in LLMs, larger models may exhibit different behaviors in learning semantics with or without metadata. The authors could consider testing models of different sizes.\n- Large-scale pretraining: the authors should consider the impact of large-scale pretraining on the model's ability to learn semantics with metadata conditioning. Large-scale pretraining typically include a large amount of code data and structured data, which could significantly affect the model ability to model grammar in PCFG-generated data. For example, this can be done by performing continual pretraining using PCFG data on top of Meta's LLaMA model weights.\n- Size of training data: the author mentioned that \"Without metadata, the model needs to learn the hierarchical mixture of grammars directly from data, which is challenging due to the diverse structure of possible grammars\" in Section 4. Data size is a critical factor determining whether the model can successfully learn the grammar without metadata. Therefore, the authors should consider training the model with different data sizes to see if the observation generalizes.\n- Longer prompt length in Table 3.1: The results in Table 3.1 does not seem to show a consistent advantage of metadata conditioning even at longer prompt lengths. The authors could consider testing the model with longer prompt lengths, like in Figure 3.2, to check if the conclusion \"for tasks with long prompts, metadata conditioning\nimproves the performance\" (Finding 2) holds.", "questions_to_authors": "- The metadata is provided starting from the top-level rules. Could the result be different if the metadata is provided starting from the bottom-level rules? The bottom-level rules are applied last in the generation process, so the model may be able to infer those rules easier from data.\n- In line 223, it is not quite clear why \"learning the richer posterior given metadata p(G_L|x, j) is facilitated\". The posterior is sharper because of the metadata j, but it does not necessarily mean that the model is learning better.\n- This also links to the conclusion in line 296 \"When x is rich, ... it can implicitly construct p(G_L|x) more accurately\". Even when x is long, predicting p(j|x) may still be imperfect, and this error will propagate to the posterior p(G_L|x, j). The advantage of modeling posterior through metadata j versus directly modeling thus seems not very clear.", "extra_scores": {}}, {"review_id": "HAPG7VKYCk", "reviewer": "Reviewer_xHQg", "summary": "This paper investigates when providing metadata at the beginning of text documents during pre-training of large language models improves the results. The authors generate synthetic data using PCFGs for their experiments and conclude that prepending metadata is useful only when the context is long enough.\n\nThe goal of the paper is certainly interesting.Being able to understand when a technique will improve the results is a worthy goal. However, it is not clear to me whether the setup that the authors use is realistic. Using artificial data generated by PCFGs can be biased. \n\nFurthermore, the paper lacks important details. For example, there is no information about the size of the data generated.\n\nFinally, the section on the interpretation of the results and findings lacks concrete connections to the actual results (and contains no data).", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main reason to accept this paper would be the attempt it makes to investigate the impact of prepending metadata to the training documents during pre-training.", "reasons_to_reject": "The main reason to reject the paper is the limiting setup relying on PCFGs to generate artificial data. The overall setup is not realistic.\n\nFurthermore, the paper lacks important details. For example, there is no information about the size of the data generated.\n\nFinally, the section on the interpretation of the results and findings lacks concrete connections to the actual results (and contains no data).", "questions_to_authors": "- Could you please clarify how much data is used? What is the effect of the amount of data?\n\n- Is there any data to backup the interpretation provided in Section 4?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the impact of metadata conditioning in a synthetic dataset generated from PCFGs, where the latent variables are known and can be added as metadata. They vary and ablate the metadata and find some conclusions on the impact on test loss, metadata prediction, and impact on downstream performance in the context of prompt length. While reviewers felt that there could be more connections between the findings on PCFGs to real data, they noted that this was generally a clean and interesting setup to study this problem. Some reviewers felt that there could be a more comprehensive sweeps of different aspects such as data size and model size. Generally they felt that the paper provided interesting findings for the community, although only on synthetic setups.
Decision: Accept