ICML 2025

Metadata Conditioning Accelerates Language Model Pre-training
Tianyu Gao 1 Alexander Wettig 1 Luxi He 1 Yihe Dong 1 Sadhika Malladi 1 Danqi Chen 1
Abstract
The vast diversity of styles, domains, and qual-
ity levels present in language model pre-training
corpora is essential in developing general model
capabilities, but efficiently learning and deploy-
ing the correct behaviors exemplified in each
of these heterogeneous data sources is challeng-
ing. To address this, we propose a new method,
termed Metadata Conditioning then Cooldown
(MeCo), to incorporate additional learning cues
during pre-training. MeCo first provides meta-
data (e.g., URLs like en.wikipedia.org)
alongside the text during training, then transi-
tions to a cooldown phase using only standard
text—enabling the model to perform well even
without metadata. MeCo significantly acceler-
ates pre-training across different model scales
(600M to 8B parameters) and training corpora
(C4, RefinedWeb, and DCLM). Notably, a 1.6B
language model trained with MeCo matches the
downstream task performance of standard pre-
training while using 33% less data. Addition-
ally, MeCo allows us to steer language models
by conditioning the inference prompt on either
real or fabricated metadata that encodes the de-
sired output properties—for example, prepend-
ing wikipedia.org to reduce harmful gener-
ations or factquizmaster.com (fabricated)
to improve performance on common knowledge
tasks. We further demonstrate that MeCo is com-
patible with various types of metadata, such as
model-generated topics. MeCo is remarkably sim-
ple, adds no computational overhead, and shows
promise for producing more capable and steer-
able language models. Our models, data, and
code are available at https://github.com/
princeton-pli/MeCo.
1Princeton Language and Intelligence, Princeton University.
Correspondence to: Tianyu Gao <tianyug@princeton.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PM...