Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation
Authors: Amanda Myntti
Erik Henriksson
Veronika Laippala
Sampo Pyysalo
Venue: COLM 2025 (2025)
Keywords: Register
Genre
Large Language Models
NLP
LLM evaluation
TL;DR: We investigate the effect of register (also known as genre) as an explainer of LLM performance, and show that it has a substantial impact on model accuracy on standard benchmarks.

Abstract: Pretraining data curation is a cornerstone in Large Language Model (LLM) development, leading to growing research on quality filtering of large web corpora. From statistical quality flags to LLM-based labelling systems, datasets are divided into categories, frequently reducing to a binary: those passing the filters are deemed as valuable examples, others are discarded as useless or detrimental. However, a more detailed understanding of the contribution of different kinds of texts to model performance is still largely lacking. In this article, we present the first study utilising _registers_ or _genres_—a widely used standard in corpus linguistics to model linguistic variation—to curate pretraining datasets and investigate the effect of register on the performance of LLMs. We train small generative models with register classified data and evaluate them using standard benchmarks, and show that the register of pretraining data substantially affects model performance. We uncover surprising relationships between the pretraining material and the resulting models: using the _News_ register results in subpar performance, and on the contrary, including the _Opinion_ class, covering texts such as reviews and opinion blogs, is highly beneficial. While a model trained on the entire unfiltered dataset outperforms those trained on datasets limited to a single register, combining well-performing registers such as _How-to-Instructions_, _Informational Description_, and _Opinion_ leads to major improvements. Furthermore, analysis of individual benchmark results reveals key differences in the strengths and drawbacks of specific register classes as pretraining data: _How-to-Instructions_ excels at physical reasoning and sentence completion while barely crossing random baselines on world-knowledge benchmarks, while _Narrative_ boosts performance on social interaction tasks but struggles with scientific questions. These findings show that register is an important explainer of model variation and can facilitate more deliberate and detailed future data selection practices.

Published as a conference paper at COLM 2025
Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation
Amanda Myntti, Erik Henriksson, Veronika Laippala & Sampo Pyysalo University of Turku
(amanda. a. myntti, erik. www.hhkssnn.aaeeeeeeee htt.iisssosmaelllspsssslllllo spyysalo)@utu.. fi
Abstract
Pretraining data curation is a cornerstone in Large Language Model (LLM) development, leading to growing research on quality filtering of large web corpora. From statistical quality flags to LLM-based labelling systems, datasets are divided into categoríes, frequently reducing to binary: those passing the filters are deemed as valuable examples, others are discarded as useless or detrimental. However, a more detailed understanding of the contribution of different kinds of texts to model performance is still largely lacking. In this article, we present the first study utilising registers genres widely used standard in corpus linguistics to model linguis- tic variation-to to curate pretraining datasets and investigate the effect of register on the performance of LLMs. We train small generative models with register classified data and evaluate them using standard benchmarks, and show that the register of pretraining data substantially affects model performance. We uncover surprising relationships between the pretrain- ing material and the resulting models: using the News register results in subpar performance, and on the contrary, including the Opinion class, cov- ering texts such as reviews and opinion blogs, is highly beneficial. While a model trained on the entire unfiltered dataset outperforms those trained on datasets limited to a single register, combining well rell-performing registers Instructions, Informational Description, and Opinion leads to major improvements. Furthermore, analysis of individual benchmark results reveals key differences in the strengths and drawbacks of specific register classes as pretraining data: How-to- to-Instructions excels at physical reasoning and sentence completion while barely crossing random baselines on
such
benchmarks, while Narrative boosts performance on social interaction tasks but struggles with scientific questions. These find- ings show that register is an important explainer of model variation and can facilitate more deliberate and detailed future data selection practices.
1 Introduction
Recent work on scaling laws and the limits of human-generated training data available for Large Language Model (LLM) training have caused an uptick in research pretraining data quality as opposed to quantity (Kaplan et al., 2020; Hoffmann et al., 2022; Muennighoff et al., 2023; Villalobos et al., 2024). While state-of-the-an on large web corpora, there is increasing focus on filtering such data to remove potentially detrimental material. The selection of pretraining data often relies on simple heuristics or quality signals (Longpre et al., 2024; Ostendorff et al., 2024), sometimes together with LLM-created quality flags (Weber et al., 2024; Henriksson et al., 2025). However, language use varies along number of dimensions other than the presumed dichotomy of positive or negative value for LLM training, ranging from persuasive, narrative, to informative texts. The effects of this variation on LLM capabilities remain unknown. In this paper, approach pretraining data curation from a novel perspective: how do different registers (or genres) of training data affect LLM performance commonly used benchmarks.
models are still trained predominantly

---

Published as a conference paper at COLM 2025
Register studies is widely applied paradigm in corpus linguistics that examines language variation across different situations, from casual spoken conversations to persuasive and informational settings (Biber, 1988; 1995; Biber & Conrad, 2019). Registers are defined as situationally characterised text varieties, with typical classes encompassing news, reviews and song lyrics. Register has been shown to be one of the most important predictors of linguistic variation (Biber, 2012) and, as put by the founder of register studies, Douglas Biber, "register always matters" (Biber, 2013).
Web register classification has long aimed to characterise different kinds of web texts in order to better utilise and understand the web as a corpus (Kilgarriff & Grefenstette, 2003; Baroni & Kilgarriff, 2006). However, early studies focused on pre-selected sets of registers presumed to be typical of the web, and thus lacked generalizability (Santini, 2007; Sharoff et al., 2010) and subsequently failed to capture the full range of linguistic variation found on the unrestricted web. Egbert et al. (2015) addressed this gap by creating corpus representing the full spectrum of English web registers, with Laippala et al. (2023) presenting the first deep learning-based web register classifier trained on the data and targeting the entire unrestricted web. Other studies expanded this approach to multilingual settings (Repo et al., 2021; Rönnqvist et al., 2021; Kuzman et al., 2023a; Henriksson et al., 2024), showing that web registers are identifiable in web-scale data. These advances now enable the integration of corpus linguistics frameworks with LLM pretraining data curation- -our study being to the best of our knowledge the first to bridge these areas, despite the well-documented role of registers in linguistic variation.
To understand the characteristics of web registers in the context of LLM pretraining, we train small generative models up to 100 billion tokens with web data with datasets curated to only contain texts from one register class. We evaluate these models using well-known benchmarks and analyse the results through average accuracy and performance on individ- ual benchmarks, revealing how each register impacts model capabilities. Our results show that register has substantial impact on model performance. Additionally, we find that training on a combination of register-specific datasets that displayed strengths in different benchmarks can lead to major improvements: registers like Opinion boosts model perfor- mance, while incorporating examples from register Interactive Discussion or Spoken at the expense of drops. With these findings, our study aims to open up discussion about linguistic selection of training data, and to show that register classification holds potential as tool for analysing pretraining data.
Informational Description and Opinion leads to performance
2 Related work
Ablation studies, where small generative models are trained with different parts of large dataset and evaluated on downstream tasks, are a well-established method for evaluating dataset quality and optimising data curation methods. Gao et al. (2020) validated the quality of their Pile datasets by evaluating models trained on the Pile, CommonCrawl,, and CC-100. Longpre et al. (2024) pretrain small models on data filtered by different qualities (data age, data domain, data toxicity) and analyse the effects of this curation via model evaluation. Similarly, Sachdeva et al. (2024) use an ablation framework to find that diversity and coverage of different topics in the pretraining data are crucial to models that succeed in multiple benchmarks. Likewise, Burchell et al. (2025) released a multilingual dataset and performed ablation studies to measure the quality of different versions of their datasets. Henriksson et al. (2025) use an LLM to line-by-line classify training documents to optimise dataset composition and evaluate with downstream task.
The methodology of our approach most closely resembles that of the FineWeb ablation studies (I'nedo et al., 2024a). The Febe dataset was created by conducting analyses of tens of heuristic filters to select the data curation methods that result in the best pretraining data with respect to the downstream performance of the trained model. Currently, datasets produced by the FibeWeb pipeline are available for over a thousand languages (Penedo et al., 2024b).
2

---

Published as a conference paper at COLM 2025
Current leading work on dataset composition optimisation includes Su et al. (2024) and Li et al. (2024). Lee et al. (2022) established deduplication as a standard procedure in pretraining data selection pipelines. Other common approaches for selecting pretraining data from web sources include URL-based filtering (e.g. Penedo et al. (2023)), text statistics such word count, repetition, or presence of blacklisted words (e.g. Raffel et al. (2020)), filtering based on perplexity (e.g. Muennighoff et al. (2023),Weber based on similarity against a selected good-quality corpus (e.g. Brown et al. (2020), Tirumala et al. (2023)). According to survey by Albalak et al. (2024), it remains an open question whether quality filtering data improves model performance, and in which cases: Dodge et al. (2021) show that url-based pretraining data filtering leads to bias in the resulting data, while reference corpora used for similarity selection can be biased in language, topics, or demographics (Rae et al., 2022; Gururangán et al., 2022).
Weber et al. (2024)), and selection
While register studies is a widely applied standard to analyse language variation corpus linguistics, the term genre is more often used in the NLP field (Sharoff et al., 2010; Petrenz & Webber, 2011; Sharoff, 2020; Kuzman et al., 2023b). However, both approaches target text categories such opinionated, informational, and lyrical.
Register studies has a long tradition in mapping the relationship between typical linguistic features associated with texts and different situational contexts where language is used, in both online and offline settings (e.g., Biber (1995); Biber & Egbert (2018); Ansarifar et al. (2025)). Biber (2012) even shows that register is one of the most important predictors of linguistic variation. Similarly, web register (or genre) classification is a widely applied approach to classify web content (Kilgarriff & Grefenstette, 2003; Santini, 2007; Petrenz & Webber, 2011; Myntti et al., 2024).
Since the release of the first web register corpus covering the full unrestricted web (Egbert et al., 2015), automatic register classification using machine learning has been successfully applied to the unrestricted web, achieving nearly human-level performance (Laippala et al., 2023; Kuzman et al., 2023b). Henriksson et al. (2024) also show that register characteristics transcend language boundaries, allowing classification models to identify registers even in languages that were not present in the training data (zero-shot languages). These advances enable the current study to select pretraining data by register. Our approach also aligns with calls for greater transparency in foundation models (Bommasani et al., 2023), as it provides structured metadata about the linguistic composition of training data, helping researchers and users to understand what types of texts model has been exposed to during training.
3 Methods
3.1
Data
We use HPLT version 2.0¹ (Burchell et al., 2025) as our source data. This collection includes datasets for 193 languages and additional parallel datasets for 50 languages paired with English. HPLT v2 datasets have been processed from a combination of Internet Archive and Common Crawl using Trafilatura (Barbaresi, 2021) with language identification performed using OpenLID (Burchell et al., 2023). Two versions of each dataset are provided, deduplicated, with Minl-ash deduplication (Broder et al., 1998), and cleaned, with additional cleaning heuristics applied. An evaluation by Burchell et al. (2025) showed the cleaned English dataset to be of high quality, with models trained on the data achieving performance comparable to ones trained on the FineWeb dataset (Penedo et al., 2024a). In this study, we nevertheless chose to use the deduplicated English dataset, which yields slightly worse performance in LLM pretraining. This choice allows us to examine the effect of register with minimal interference from cleaning procedures that might disproportionately affect certain registers and thus bias our analysis. This decision is also motivated by the uneven distribution of register labels in web data; with fewer cleaning steps, we retain more data, which is especially important for the less frequent register classes.
1 Available at hhttsss//mmll--pppeetrrrraaaaatt
3

---

Published as a conference paper at COLM 2025
Individual registers
Description
Available
How-to wor---o--nstruccion (HI)
Recipes, instructions
100 B
Interactive Discussion (ID)
Discussion forums
314 B
Informational Description (IN)
Wikis, information sites
695 B
Informational Persuasion (IP)
Advertisements, commercial sites Song lyrics, poems
421 B
Lyrical (LY)
20 B
Machine Translation (MT)
Machine translation
306 B
Narrative (NA)
News, blogs
545 B
Opinion (OP)
Opinionated texts, religious sermons
416 B
Spoken (SP)
Interviews, speeches
32 B
Instrncciie--nyooraatinall (HII-N) News (ne)
Hybrid; Documents predicted both Hi and IN
70B
Subregister NA; news
404 B
Description (dtp)
Subregister IN; decription of thing or person
781 B
Combinations
HI-IN-HI-die
1/3rd sampled from HI-IN, HI, dtp
HI-IN-HI deT-dt--PPP
1/4th sampled from HI-IN, HI, dtp,,
HI-IN-HI HI--HHH-OPPO Wil--dppOP--N-
1/5th sampled from HI-IN, HI, dtp, OP, NA
HI-N--HI-dy--OP-NA--
1/6th sampled from HI-IN, HI, dtp, OP, NA, ID
HI-N-HH--tp-OP-NA---D
1/7th sampled from HI-IN, HI, dtp, OP, NA, ID, SP
Baselines
HPLT v2 deduplicated FineWeb
Burchell et al. (2025) Penedo et al. (2024a)
Table 1: Datasets used in our experiments. Each dataset corresponds to one trained model. The Individual registers section describes the main register labels and hybrid & subregisters. The Combinations section contains combined register classes, and Baselines lists the baseline datasets we in our experiments. The Available column contains the approximate token counts available in the HPLT v2 deduplicated corpus.
The HPLT v2 dataset includes predictions of register classes as probabilities for each docu- ment, generated using the multi-label register classifier by Henriksson et al. (2024). This classifier was fine-tuned from XLM--RBERla--arge (Conneau al., 2020) using the multilin- gual CORE corpus introduced in the same study. The CORE register scheme is hierarchical, with main registers divided into 25 subregisters that further describe the content of each document. Documents can have none, one, or multiple main registers (referred to as hy- brids), along with any associated subregister labels. The full register scheme can be found in Henriksson et al. (2024), and example documents are given in Appendix A.
Although the register labels are available for multiple languages, we limit our analysis to English due to the availability of varied and well-established benchmarks. For our experiments, we assign register labels to documents from the given probabilities using classification threshold of 0.4, optimised for English. From the register classes, select all main registers, 2 subregisters, and one hybrid for our experiments. We included the subregisters and hybrids for specific reasons: the subregisters News and Description are among the largest subregister classes, each with over 100 billion tokens svailablessuffici data to train models and draw meaningful comparisons with their parent registers Narrative and Informational Description. Moreover, there has been particular interest in using news articles (e.g. de la Rosa et al. (2025)) and web encyclopedias (i.a. Wikipedia) to train LLMs, which correspond to classes News and Description in the register scheme. The hybrid class was included study the differences between individual registers and hybrids, based on preliminary results showing that register-specific models trained on these classes perform well overall but differ across individual benchmark tasks. The selected register classes and their abbreviations are described with examples in Table 1. Subregister classes not selected for our experiments are present in the data; however, they merged to their main-level class.
Using these labels, we sample 100 billion tokens per register, using all available data for registers with fewer than 100 billion tokens. We limit our analysis to documents over 200
4

---

Published as a conference paper at COLM 2025
characters and remove documents with extremely high token counts (corresponding to over 300,000 words?) due to hardware constraints. These excluded documents represent less than 0.3% of the total data and primarily belong to the Informational Description class.
We also experiment with training on combinations of certain registers, which are listed in Table under "Combinations". These combination datasets are created by sampling equally from individual register datasets: for example, in the combination dataset HI-IN-HI-dlp, 1/3rd of the training samples are from snstrucciieeInnfomaiiiin Instructions (I-I), and 1/3rd from Description (dtp), with our dataset creation step ensuring no duplicates are present in the combined data. See Section 4 for further information and motivation for these datasets.
(HI-IN), 1/3rd from How-lo-
3.2 Model training
We replicate the training setup of Penedo et al. (2024a), using the same model architecture (Llama), model size (1.71 billion parameters), and other training parameters. total, the models are trained up to 100 billion tokens. See Appendix B for further information the training setup. As a baseline, we use a model trained on a random sample the HPLT v2 deduplicated data. This allows to compare the effects of using single register as opposed to using data with the natural distribution of registers in the source dataset. Finally, as our setup mirrors that of Penedo et al. (2024a), also include results for a model trained on a sample of the FineWeb dataset as further point of comparison and as a reference for interpreting our findings.
3.3 Evaluation
To maintain comparability with previous work, we also follow Penedo et al. (2024a) in our evaluation. Specifically, we use the LightEval (Fourrier et al., 2023) evaluation harness with the following benchmarks in zero-shot setting for all tasks:
HellaSwag (Zellers et al., 2019) contains logical sentence completion tasks the form of presenting paragraph and an incomplete sentence with alternative sen-
tence endings.
WinoGrande (Sakaguchi et al., 2021) evaluates commonsense reasoning in binary classification setting using pronoun resolution problems.
PIQA (Bisk et al., 2020) focuses physical commonsense reasoning, with examples involving manipulation of physical objects.
SIQA (Sap al., 2019) contains tasks on reasoning about social situations.
OpenBookOA (Mihaylov et al., 2018) focuses on tasks requiring multi-step reason- ing with additional context given.
ARC Easy and ARC Challenge (Clark et al., 2018) are multiple choice benchmarks
with science exam questions.
Commonsense (Talmor et al., 2019) consists of multiple-choice questions requir- ing prior world knowledge.
MMLU (Hendrycks et al., 2021a;b) is multiple-choice question answering dataset covering 57 distinct categories, such as US politics and anatomy.
These benchmarks were chosen by Penedo et al. (2024a) due to their compatibility with small model size, showing reliable, low-variance and monotonic signals even with limited training data, which is crucial for experiments. For MMLU, we consider all 57 categories as one task and average our results accordingly. This ensures that models reaching high accuracy on MMLU but mediocre results on other benchmarks do not dominate the average results. See Appendix D for further discussion.
The selected benchmarks reflect the most common present-day expectations placed on LLMs: fluency, scientific and real-world knowledge, problem solving, and reasoning. However, this selection inevitably disadvantages certain register models. For example, models trained on the Lyrical register are likely to show poor performance, as none of the benchmarks
Defined as white-space and punctuation separated segments
5

---

Published as a conference paper at COLM 2025
measure poetic capabilities that would be expected from model trained on lyrical data. This does not mean that lyrical capabilities are unwanted or unnecessary for LLMs, but rather highlights the importance of benchmark diversity and the significance of selecting benchmarks that measure abilities relevant to specific use cases. In our case, although the benchmark selection disadvantages some registers and their expected performance more than others, using the full range of registers covering all kinds of web texts yields a complete picture of registers' effects on LLM pretraining.
Experiments
Our experiments are divided into two parts: First, we examine the effects of register on model performance by training models for the 9 main-level registers, one hybrid class cestrrufiie-Infornattii
and two subregister classes (News and Description), presented in Table 1 under "Individual registers". These experiments are meant to highlight the differences of registers as pretraining data.
Second, we investigate the effect of further mixing registers by selecting tor-performing registers from our first experiment. This allows us to observe how adding or removing specific registers affects overall model performance and whether the strengths of individual registers are preserved in combined models. The selected top-scoring register combinations are presented in Table 1 under "Combinations". When sampling data for register combi- nations, we ensure that no duplicate documents are included, even when combining the Instructive-Infornattinal hybrid class Instructive- Informational (HII-NN with the register con-tto--nsttruciions (HII). We chose to use the subregister Description (dtp) instead of the main-level register Informational Description (IN) because Informational Description showed signs of performance degradation in later training steps, and noted in Section 3.1, we had to exclude some documents from the Informational Description dataset. To maintain comparability, we train all models up to 100 billion tokens, although the larger amount of combined data would allow for longer training. Our aim is not to identify optimal register combinations, but rather to explore how register-based data curation might influence model performance different tasks.
Some of our datasets contain fewer than 100 billion tokens, which necessitates repeating data during training. This raises questions about the validity of our results, as Hoffmann et al. (2022) argue that that repeating data can lead to performance degradation. However, Taylor et al. (2022) report validation loss continues to improve up to epochs for 120 billion parameter model, and Muennighoff et al. (2023) find that for the GPT-2 (Radford et al., 2019) architecture with 2-8 billion parameters, multi-epoch training has negligible effects when limited to 4 or fewer epochs. Xue et al. (2023) further demonstrate that smaller models are less susceptible to the negative effects of repeated training examples in T5-architocture models (Raffel et al., 2020). Therefore, we are reasonably confident in the results from models trained on Instructiae-lnforraiionn (1.4 epochs) and Spoken (3.1 epochs). The model trained on our smallest dataset, Lyrical (5 epochs), performs poorly even within the first epochs, suggesting its low performance is due to register characteristics and benchmark selection rather than data repetition. None of the combination models exceed 1 epoch of training, since we sample the 100 billion tokens from combinations of multiple datasets, together exceeding 100 billion tokens.
We additionally compare our register scheme to three other quality classifiers, FineWeb-edu classifier (Penedo et al., 2024a), the DCLM quality classifier (Li et al., 2024), and NVIDIA's NemoCurator Quality Classifier DeBERTa (He et al., 2023). These experiments and their results are presented in Appendix C.
Results
5.1 Average performance
The results for individual register models, averaged over all benchmarks, are presented in Figure 1 and Table 2, and detailed in Appendix D. The evaluation shows striking differences in accuracy between the models, demonstrating that the register of the pretraining data

---

Published as a conference paper at COLM 2025
Instrutive-Informational (HI-IN)
0.46
HPLT v2 (all registers)
Description (dtp)
0.44
How-to-Instrrutions (HI)
Opinion (OP)
0.42
Narrative (NA)
Informational Description (IN)
0.4
Interactive Discussion (ID)
Spoken (SP)
0.38
News (ne)
Informational Persuasion (IP)
0.36
Machine Translation (MT)
Lyrical (LY)
0.34
20B
40B
60B
80B
100B
Figure 1: Individual register and HPLT v2 (deduplicated) model accuracies as function of the number of training tokens. Rolling average over 6 billion tokens adjacent checkpoints) applied for ease of reading overlapping lines, see Table 2 for numerical results for the final checkpoint. Dotted lines indicate training continuing over one epoch, legend in order of last checkpoint performance.
0.48
HI-IN-HI-dtp-OP
HH-N-HHI-tt--P--NA
HI-IN-HI-dtp
-H-IN-HH-dtp-OP-NNA--
0.44
FineWeb
0.42
HPLTT v2 (all registers)
0.38
0.36
20B
40B
60B
80B
100B
Figure 2: Combination model and baseline accuracies function of the number of training tokens. Rolling average over 6 billion tokens (3 adjacent checkpoints) applied for ease of reading overlapping lines, Table 2 for numerical results for the final checkpoint. Legend in order of last checkpoint performance.

---

Published as a conference paper at COLM 2025
Pretraining dataset
Accuracy
Pretraining dataset
Accuracy
HI-IN-HI-aft HI-HH--tppOP
0.475
Opinion (OP)
0.447
HI-N--HI-Itt-OP-NA HI-IN-HI-dtp
0.472
Narratioe (NA)
0.441
0.466
Informational Description (IN)
0.437
FineWeb
0,466
Interactive Discussion (ID)
0,431
HI- JI-NN-HI-ttp-OP-NA-DD
0.465
Spoken (SP)
0.422
Instractiie-hnfoouutttnnl (HI-IN)
0.465
News (ne)
0.418
JI-IN-HI-dtp-OO--NAA
0.464
Informational Persuasion (IP)
0.393
HPLT v2 (deduplicated, all registers)
0.457
Machine Translation (MT)
0.374
Description (dtp)
0,452
Lyrical (LY)
0.358
(HI)
0.447
Table 2: Results by the last checkpoint performance in numerical format. Note that the values differ slightly from the Figures 1 and 2 due averaging over adjacent checkpoints in the figures.
has a substantial effect on model performance. We find that the model trained on the full HPLT v2 data outperforms models trained on individual registers, demonstrating that mix of registers is important for LLM training. The performance ranking of individual registers generally aligns with intuitions about their content and what typically considered high-quality pretraining data: registers like Lyrical (LY), Machine Translation (MT), and Informational Persuasion (IP) yield worse-performing models, while models trained on registers containing informational content and instructions, such as Description (dtp) and How--o-Inssee Instructions (HT), perform the best. Surprisingly, Opinion (OP) yields the 4th best performing model. Both Informational Description (IN) and Narrative (NA) models perform worse than performance drop, are the models trained on Interactive Discussion (ID), Spoken (SP) and remarkably, the subregister News (ne), consisting of news articles, which performs among the worst.
Description and Opinion. Following these, with a noticeable
As opposed to the main register models, the hybrid class model instructive-Inforraation (HI-IN) outperforms the model trained the full HPLT v2 data. This shows that hybrid documents do not simply result in averaged performance between the two hybridised registers, but can show better performance than either alone. This finding motivated our second experiment, investigating register combinations. The subregisters News (ne) (subregister of Narrative) and Description (dtp) (subregister of Informational Description) show divergent patterns when compared to their parent registers: News drastically underperforms its main-level register Narrative, while Description achieves higher accuracy than Informational Description. We analyse differences between main-subregister pairs in more detail in the following section.
As the results show, pretraining model with data from only one register class does not provide improvements compared to training with the full dataset. This aligns with the established notion that pretraining data should have variability to yield capable models. The results for our combination model experiment are shown in Figure 2, Table 2, and again with more detail in Appendix D. The results clearly support the above idea: all combination models outperform the individual register models and the baseline HPLT v2 model. Additionally, the best models in our study (HI-IN-HII-dtp-PP and HI-IN-HI- dIp-OP-NA) reach higher average accuracy than the models trained on the HPLT v2 and FineWeb datasets. HI-IN-HI-dtp model shows much worse performance than HI-IN-HI- dtp-OP, which further includes Opinion (OP). This suggests that opinionated texts provide valuable pretraining data and are key to high model performance, at least as measured by the selected benchmarks. Further adding the Narrative (NA) dataset, seen in the HI-IN-HI- dtp-OP-NA model, slightly worsens performance, though the difference is small. Significant drops occur when including Interactive Discussion (ID) and Spoken (SP) in models HI-IN- indicating either that these registers contain qualities that harm the abilities measured these benchmarks, that they do not provide benefits that outweigh the reduced proportion of Instructive- dnderuciie-Informattonn (HI-IN), Description (dtp), and Opinion (OP) in the training mix.
HI-dtp-OP-NA-ID and HI--N--I-dtp-OOP-NAA-------
8

---

Published as a conference paper at COLM 2025
ARC Easy
HellaSwag
MMLU
HI-IN-HI-dtp-OP
HI-IN-HI-dp-O--NAA
HI-IN-HI-dtp
HI-N--HI-Ct--OP-NN--ID
HI-NN-HI-dtp-OO-NNA--DSSST
FineWeb
0.3
0.4 0.5
0.6
0.3 0.4 0.5 0.6
0.240.260.28
0.3 0.32
Iestruciveelnforraatinalll (HI-IN)
ARC Challenge
SIQA
PIQA
HPLT v2 (all registers)
Description (dtp)
Now-to-Instructionn (HI)
Opinion (OP)
Narrative (NA)
Informational Description (IN)
0.2
0.25
0.3
0.38 0.4 0.42
0.6
0.7
0.8
Interactive Discussion (ID)
WinoGrande
CommonsenscQA
OpenBookQA
Spoken (SP)
News (ne)
Informational Persuasion (IP)
Machine Translation (MT)
Lyrical (LY)
0.5
0.55
0.25
0.3 0.35
0.25 0.3 0.35
Figure 3: Change of accuracy from first to final checkpoint on individual benchmarks shown range, with grey indicating the first checkpoint and colours indicating the last checkpoint. The random-guess threshold is shown as a grey vertical line in cases where at least one model falls below it. Bars and legend shown in order of average accuracy.
5.2
Per task
The results for each benchmark separately are shown in Figure and detailed further in Appendix E. The differences between benchmarks are remarkable, both in the performance ranking of models and in the way accuracy trends with respect to the number of training tokens. This shows that registers have strengths in different tasks. Most benchmarks display the monotonicity condition that influenced the selection of Penedo et al. (2024a), as they tend towards increased accuracy with more training tokens. A clear outlier is the SIQA benchmark, where increases in accuracy are very small, with the largest increase seen in Narrative (NA) and cnstructive-nformaatti by 0.03 during the training.
(HI-IN) models, both increasing approximately
Generally, the trends observed Section 5.1 hold. The models with the lowest average accuracy-I.yrict (LY), Machine Translation (MT) and Informational Persuasion (ie)-eeform poorly on all benchmarks, and, notably, worse than a random guess on ARC Challenge. The model for subregister News (ne), despite weak average performance, improves the MMLU and WinoGrande benchmarks, while Spoken (SP) enhances results on Commonsense The differences between benchmarks become more pronounced with models trained Narrative (NA), Informational Description (IN), Opinion (OP), now--ooInnstructonn (HI), and Interactive Discussion (ID), which display great variation in ranking across tasks, highlighting their strengths. The Narrative model performs well on SIQA and WinoGrande but barely surpasses the random baseline on ARC Challenge. Informational Description trumps other models on MMLU and both ARC benchmarks, but ranks low to mediocre elsewhere. The model trained on the peru-tt-Innsructiie PIQA and HellaSwag. This might be because PIQA measures reasoning in physical tasks, which aligns with the instructions related to physics included other hand, woru-to-Instructionss ranks among the worst MMLU and CommonsenseQA. The Opinion model excels in WinoGrande, SIQA and CommonsenseeAA This signal also appears in the combination models' performance CommonsenseeAA, including Opinion in the training data greatly increases accuracy on this benchmark, unlike the HI-IN-HI-dtp model, which excludes Opinion.
dataset outperforms almost all other models on
noow-to-Instructinn On the

---

Published as a conference paper at COLM 2025
The hybrid mnstractioe-Informattin (HI-IN) model generally performs between individual Nhow-to-Instruche (HII and Informational Description (IN) models, and is notably closer in performance to whichever yields better accuracy. Exceptions to this pattern are Open- BookQA, CommonsenseQA, and WinoGrande, where the mnstrucciove-ffoomaattii shows better accuracy than either Specifically in the case of WinoGrande, this is surprising, as both Hoow-to-nnstruction and Informational Description struggle with this benchmark. From this result, we conclude that hybrids may contain qualities not found in individual registers and require further study. The News (ne) model trails behind Narrative (NA), except on the MMLU and ARC Challenge benchmarks. This is likely due to news articles covering topics related to MMLU tasks, such as politics, business, science, and the environment. Similarly, the other main-subregister pair, Description (dtp) and Informational Description (IN), shows differences in performance across benchmarks, with the Description-model outshining Informational Description in ARC Easy, HellaSwag, PIQA, WinoGrande, and OpenbookQA. Informational Description, on the other hand, excels in ARC Challenge and MMLU. Specifically for the WinoGrande benchmark, the performance difference might be explained by the Description register containing more texts focused on things and people, thus resulting in better pronoun-resolution abilities.
model Informational Description eeeaaateeee separately.
The combination models lead in performance on most benchmarks. On MMLU specifically, we observe a huge drop between the HI-IN-HI-dtp-On model and others: including Opinion in the pretraining data increases performance, while removing Opinion or further adding Narrative greatly disrupts model performance on this benchmark. Notably, the models HI- IN-HI-Atp-OP and Hl-IN---I-tt--OOP--AA are trained with registers that individually perform on HellaSwag and ARC Easy,
well on wide range of benchmarks: cnstructiveInformatti Hooo-to-Instruccie on Hellaswag and PIQA, Description on ARC Easy, ARC Challenge, and OpenBookQA, Opinion on CommonsenseQA, and Narrative on SIQÁ and WinoGrande.
6 Conclusion
In this paper, we presented the first study investigating the effect of linguistic register on the performance of small generative LLMs. Our findings show that register is an important explainer of LLM performance, and we were able to show surprising relationships between the register of pretraining data and model accuracy on different benchmarks. While using the whole dataset yielded better performance than limiting the pretraining data to any individual register class, we found specific combinations of registers that outperform the full uncurated dataset. These findings can be used to increase pretraining data transparency and to optimise data selection methods, potentially addressing specific performance gaps by incorporating appropriate registers.
Limitations and future work
As a first study in the intersection of two broad lines of research the linguistic registers and LLM training - our work leaves a number of questions incompletely answered. Our experi- ments are conducted in English only. While register classification has shown multilingual prowess, our results might not generalise to other languages, and we leave multilingual analysis for future work. The benchmarks used, though standard in studies of this type, do not fully measure all model capabilities, as discussed regarding the Lyrical class. Evaluating with more diverse benchmarks would provide more comprehensive results and might reveal strengths and drawbacks we were unable to uncover.
Bias from data curation methods is known to propagate to trained models (Rae al., 2022; Gururangan et al., 2022), and register classification is not exempt from this issue. To mitigate bias, we used the deduplicated version the HPLT v2 data. Although this version has been processed with Trafilatura, P'enedo et al. (2024a) note that Trafilatura can leave "undesirable" content in the dataset, potentially affecting some registers more than others. The possibility of benchmark contamination also exists; register classification could systematically assign leaked benchmark material to specific classes, potentially explaining some performance differences.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
This project has received funding from the Finnish Doctoral Program Network in Artificial Intelligence, AI-DOC under decision number
21877/2224-KK--6 the European Union's Horizon Europe research and innovation programme under grant agreement No. 101070350, the Digital Europe Programme under grant agreement No. 101195233, and the Research Council of Finland under grant No. 362459. The contents of this publication are the sole responsibility of its authors and do not necessarily reflect the opinion of the European Union. We wish to acknowledge CSC IT Center for Science, Finland, for computational resources.
Ethics Statement
Training large language models is resource-demanding This fact guided our decisions: we used small model sizes and trained only up to the point where clear conclusions can be drawn. In total, we used approximately 11,000 GPU hours per model. See Appendix B for calculation of the total computational cost. To further mitigate the environmental impact of training, we used LUMI supercomputer, which is among the most environmentally friendly supercomputers in the world?. The models trained for this study are planned to be used in future research.
Releasing our models publicly for reproducibility opens up the possibility of our models being used in contexts other than research. Although we do not evaluate the possible toxicity of the generations they produce, the model sizes small, and they trained only up to 100 billion tokens, which makes their generative capabilities weak and thus likely unsuitable for applications outside the scope of this study.
Reproducibility statement
We have prioritised reproducibility in work and make all artefacts created part of this study, including code, model training configurations, and the trained model check- points, available on nttps://hugging iagegecnn.cc...... co/ uurkoNlP/register-ablltttt licenses. Our work only makes use of data openly available from https: https://hplt-t/ hhtphhhtppttptec..... org/datasets/v2.c 0, and our model training setup is detailed in Appendix B.
under open
References
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language models, 2024. URL https:ttt///xi...ooo abs 2402. 16827.
Ahmad Ansarifar, Hesamoddin Shahriari, Shelley Staples, and Mohammad Ghazanfari. Linguistic variation two written academic sub-registers. Revista Española de Lingüfstica Aplicada/Spanish Journal of Applied Linguistics, 38(1):162-191, 2025. ISSN 0213-2028. doi: https:/ / htt.://ooooorggggo orro/iooo//////oo 10.110011 wew.la.22052...o journals/10 10. 1075/resla 1tt.//.....11/1.1.111 22052. ans.
URL https: //ww:::://ww.../// wtt. jww.:j/tbpppateeeeee paee--plaffrm..mmm com/ content/
Adrien Barbaresi. Trafilatura: A web scraping library and command-line tool for text discovery and extraction. In Heng Ji, Jong Park, and Rui Xia (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 122-131, Online, August 2021. Association for Computational Linguistics. doi: /222/2/////222ccl///// 15. URL https:tpp/so http::::/aaaaaaaaaaaa http///caaaaatatoa..a/ooo or2/2021.22222 acl demo. 15/.
Marco Baroni and Adam Kilgarriff. Large linguisticall-ppooessed web corpora for mul- tiple languages. In Demonstrations, pp. 87-90, 2006. URL https: /tops://achh http///aaaaannty.....oooooo org/ E06-2001/. Phttps://lum
httts://lumi-supprrompttee supercomputer. wwtprpppppueeeeeeaaaaaaaaaaaaaaa future/
11

---

Published as a conference paper at COLM 2025
Douglas Biber. Variation across Speech and Writing. Cambridge University Press, 1988.
Douglas Biber. Dimensions of Register Variation: Cross-Linguistic Comparison. Cambridge University Press, Cambridge, 1995.
Douglas Biber. Register predictor of linguistic variation. Linguistics and Linguistic Theory, 8(1):9-37, 2012. doi: doi:10..515/cllt2202-0000 URLttpss:/ddoi.rrg 1515/ cllt-2012-0002
Douglas Biber. Interview with Douglas Biber. Journal of English Linguistics, 41(4): 359-379, 2013. doi: 10117700755422215022333 URL https://doi.org/// 1177/ 0075424213502237.
Douglas Biber and Susan Conrad. Register, Genre, and Style. Cambridge University Press, Cambridge, 2019.
Douglas Biber and Jesse Egbert. Register Variation Online. Cambridge University Press, 2018.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. The foundation model transparency index, 2023. URL httttttt///xxiii org/ abs/a/as///3// 12941.
Andrei Z. Broder, Moses Charikar, Alan M. Frieze, and Michael Mitzenmacher. Min-wise independent permutations (extended abstract). In Proceedings of the Thirtieth Annual ACM Symposium on Theory of Computing, STOC '98, pp. 327-336, New York, NY, USA, 1998. Association for Computing Machinery. ISBN 0897919629. doi: URL https:///oii.org org/10 10111114626688 276781
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference Neural Information Processing Systems, NIPS '20 Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Laurie Burchell, Alexandra Birch, Nikolay Bogoychev, and Kenneth Heafield. An open dataset and model for language identification. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volune 2: Short Papers), pp. 865-879, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 /v1 2222/// orgpollooooor//2222 www. acl moolsshhor..77 75/.
URL
Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Jindřich Helel, Erik Hen- riksson, Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhlum, Bhavitvya Malik, Farrokh Mehryary, Vladislav Mikhailov, Nikita Moghe, Amanda Myntti, Dayyán O'Brien, Stephan Oepen, Proyag Pal, Jousia Piha, Sampo Pyysalo, Gema Ramirez-Sánchez, David Samuel, Pavel Stepachev, Jörg Tiedemann, Dušan Variš, Tereza Vojtěchová, and Jaume expanded massive multilingual dataset for perfopperormance language technologies, 2025. URL https://arxix. org/abs/// /b////////// 2503.. 10267.
Bernabeu. An
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL htttps://arxis / 055.05577....
12

---

Published as a conference paper at COLM 2025
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanow. Unsupervised cross-lingual representation learning at scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440 8451, Online, July 2020. URL
Association for Computational Linguistics. doi: 10.18653 2t166/33/1/2200cl/m//// https://aclanthol
org/2020...... acl mmiin. 747/.
Javier de la Rosa, Vladislav Mikhailov, Lemei Zhang, Freddy Wetjen, David Samuel, Peng Liu, Rolv-Arild Braaten, Petter Mæhluu, Magnus Breder Birkenes, Andrey Kutuzov, Tita Enstad, Hans Christian Farsethås, Svein Arne Brygfjeld, Jon Atle Gulla, Stephan Oepen, Erik Velldal, Wilfred Østgulen, Liljia Øvrelid, and Aslak Sira Myhre. The impact of copyrighted material on large language models: A norwegian perspective, 2025. URL 09460.
https://arxix: wwwparwww...o..rbrrraeee.reeee/e
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groen- eveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: case study the colossal clean crawled corpus. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (,ds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-13388 1305, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: (0.18653/v1] 2021 1021.emnlp-main main..9..
URL https://aclante /aaas///llaaa org/2021- emnlp-main n9/.
Jesse Egbert, Douglas Biber, and Mark Davies. Developing a user-based method of web register classification. Journal of the Association for Information Science and Technology, 66, 02 2015. doi: 10.1002/ 23i.23
Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for IIm evaluation, 2023. URL / httpps////ggttbbb.co/ggggggnggg com/huggingface// lighteval.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arv preprint arXiv:2101.0000 2020.
Suchin Gururangan, Dallas Card, Sarah Dreier, Emily Gade, Leroy Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A. Smith. Whose language counts as high quality? measuring language ideologies in text data selection. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Lanuuge Processing, pp. 1562-2580 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653 10.18653/v1/20222 wtt//////1//222emm-mmmeeeee //aclanthology 2022 een.l-mmain 165/.
URL https:
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with madieentdiisettagleed embedding sharing, 2023. URL https://arxiv.org// http://og////////////o/oo 099443.....
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
Erik Henriksson, Amanda Myntti, Saara Hellström, Anni Eskelinen, Selcen Jrten-Johansson, and Veronika Laippala. Automatic register identification for the open web using multilin- gual deep learning, 2024. URL /
19892.
Erik Henriksson, Otto Tarkka, and Filip Ginter. Finerweb-10bt: Refining web data with -lmed line-level filtering, 2025. URL https::/aarxv..or//aaat 073114.
13

---

Published as a conference paper at COLM 2025
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: //axxiv. org/ hetp//bbs/00..... .83611.
Adam Kilgarriff and Gregory Grefenstette. Introduction to the special issue on the web as corpus. Computational Linguistics, 29(3):333-348, 2003. doi: 10. 10.1162/09910000332271111 URL https://aclanttoooggr
3001/.
Taja Kuzman, Igor Mozetič, and Nikola Ljubešić. Automatic genre identification for robust enrichment of massive text collections: Investigation of classification methods in the era of large language models. Machine Learning and Knowledge Extraction, 5(3):1149-1175, Sep 2023a. ISSN 2504-4990. doi: 10.3390/ make5030059. URL http://dx doi. make5030059.
3390/
Taja Kuzman, Peter Rupník, and Nikola Ljubešić. Get to know your parallel data: Per- forming English variety and genre classification over MaCoCu corpora. In Yves Scherrer, Tommi Jauhiainen, Nikola Ljubešić, Preslav Nakov, Jörg Tiedemann, and Marcos Zampieri (eds.), Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pp. 91-103, Dubrovnik, Croatia, May 2023b. Association for Computational Linguistics, doi: (0.18653/v1/2000 v.r.aall.......1... URL httpp::/acclattholooe . org org/2023. vardial-1.9/.
Veronika Laippala, Samuel Rönnqvist, Miika Oinonen, Aki Juhani Kyröläinen, Anna Salmela, Douglas Biber, Jesse Egbert, and Sampo Pyysalo. Register identification from the unrestricted open web using the corpus of online registers of english. Lan- guage Resources and Evaluation, 57(3::1045-177, September 2023. ISSN 1574-020X. doi: Funding Information: Open Access funding provided by University of Turku (UTU) including Turku University Central Hospital. Funding was provided by Academy of Finland (Grant No. 331297), Emil Aaltosen säätiö, National Science Foundation (Grant No. 1147581). Publisher Copyright: c 2022, The Authorrs).
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424 8445, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ /v11 a022.acl--ong acl-long.571 577/
URL https: / /ttp://///clllthhooo/oo or2/2022.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: In search of the next generation of training sets for language models, 2024. URL ht //tp///axvv.///////////ee//aaav www.voog//aaao orgg/bs//205 11794.
14

---

Published as a conference paper at COLM 2025
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pre- trainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 3245-3276, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1. 2024.naacl-Iong. 179. URL https://aatt acl.nhhoooon org/2024 naacl-long. 179/.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018.
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS 22, Red Hook, NY, USA, 2023. Curran Associates Inc.
Amanda Myntti, Liina Repo, Elian Freyermuth, Antti Kanner, Veronika Laippala, and Erik Henriksson. Intersecting register and genre: Understanding the contents of web- crawled corpora. In Mika Hämäläinen, Emily Öhman, So Miyagawa, Khalid Alna- jjar, and Yuri Bizzoni (eds.), Proceedings of the 4th International Conference on Natural Language Processing, for Digital Humanities, Pp. 386-397, Miami, USA, November 2024. Association for Computational Linguistics. doi: 10.18653 10.18653/v1/20222 /1/0/////1/24...mm..11..... URL https: http:::///claathoooggg.o org/ 2024. n1p4dh-1.38/
Malte Ostendorff, Pedro Ortiz Suarez, Lucas Fonseca Lage, and Georg Rehm. LLM-datasets: An open framework for pretraining datasets of large language models. In First Conference on Language Modeling, 2024. URL https://openreviiie net:/oorumidd-5RMMMl...
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobei- dli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon I1m: outperforming curated corpora with web data only. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS 23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: De- canting the web for the finest text data scale. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neu- ral Information Processing Systems, volume 37, pp. 30811-30849 Curran Associates, Inc., 2024a. URL. https::/proceee http:s:////rrcceenng..eeee neurips. cc/paper 3f
pdf.
Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. Fineweb2: A sparkling update with 1000s of languages, December 2024b. URL https://hugging: co co/datasess/
Philipp Petrenz and Bonnie Webber. Squibs: Stable classification of text genres. Com- putational Linguistics, (7(2):355-393, June 2011. doi: 10.1122/COLL..00022. URL
https://aclanthologyy wttp://ooooooo/1111//00000
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl, 2019. URL https://cddi http::///ddoooonii....oomm comm/better-lanuggee--ee wwweemoddllllaaeeee languuag models are. pdf. Accessed: 2024-11-15.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbi, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia
15

---

Published as a conference paper at COLM 2025
Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Sran-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher, 2022. URL https: www.a..vvooogo. ab/////s2...... 11446.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21),), January 2020. ISSN 1532-4435.
Liina Repo, Valtteri Skantsi, Samuel Rönnqvist, Saara Hellström, Miika Oinonen, Anna Salmela, Douglas Biber, Jesse Egbert, Sampo Pyysalo, and Veronika Laippala. Beyond the english web: Zero-shot cross-lingual and lightweight monolingual classification of registers. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, 2021. URL btttps://aclanttologg. (/g//221.eacc-9000 24, .pdf..
Samuel Ronnqvist, Valtteri Skantsi, Miik Oinonen, and Veronika Laippala. Multilingual and zero-shot is closing in on monolingual web register classification. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 157-165, Reykjavik, Iceland (Online), 2021. Linköping University Electronic Press, Sweden. URL https: //aclanthology.org org.ggy..orr/// org.oogg.. wrw.://..//2...mm11dar... modalida-main.11
Noveen Sachdeva, Benjamin Coleman, Wang-Cheny Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data- efficient lims, 2024. URL https:
orggabs//4422 09668.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106 August 2021. ISSN 0001-0782. doi:
URL https://doi. org/10. 1145/3474381.
Marina Santini. Automatic identification of genre in web pages. PhD thesis, University of Brighton, 2007. Unpublished doctoral dissertation.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNNTP-IJCNLPP) pp. 4463 4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653 10116533v1/DD1-111 1454. URL https://aclantholl org/D19-1454/.
Serge Sharoff. Know thy corpus! robust methods for digital curation of web corpora. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. May 2020. European Language Resources Association. ISBN kttps://aclantholl.0 org/ 2020. 1rec-1.298/
Marseille, France, URL
Serge Sharoff, Zhili Wu, and Katja Markert. The web library of babel: evaluating genre collections. In Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner, and Daniel Tapias (eds.), Proceedings of the Seventh International Conference on Language Resources and Evaluation (IREEC1))) Valletta,
16

---

Published as a conference paper at COLM 2025
Malta, May 2010. European Language Resources Association (ELRA). URL https://
aclanthology www.ey..rrgg1---111 L101111-
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming common crawl into refined long-horizon pretraining dataset, 2024. URL https:
1/arxiv.orr//as//4112. 02595.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseeA:
question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computatioonal Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4149-4158, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. org/N19- 1421/.
URL https://aclantthlloee
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: large language model for science, 2022. URL htttp:://arate http:///axxxvv.../aabb.o
wrg./bs////// hrgg/ab//2211.90055.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improv- ing Ilm pretraining via document de-duplication and diversification. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 53983-53995. Curran As- sociates, Inc., 2023. URL httpes:/pproceedings pewreeddiiinnnnnniii.eee....eeee cc/paper filles/pperr/222// file/a8 /ttaa//ccaa777ffffc7777777777771 -atp://eecceeet-----a-----aaaaa Datasets aat.aasttttaandd.eenaaarrsrr.... pdf.
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn, Position: will we run of data? limits of Ilm scaling based on human- generated data. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
Maurice Weber, Daniel Fu, Quentin Gregory Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Re, Irina Rish, and Ce Zhang. Redpajama: open dataset for training large language models. In The Thirty-eight Conference on Neural Information Processing Systems Dalasets and Benchmarks Track, 2024. URL https: /ttp///pppeeeeeeeeeeeeeeeeeeeeee peet/forumiid=lnukee
Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To re- peat or not to repeat: Insights from scaling Ilm under token-crisis. In A. Oh, T. Naumann, A. Globerson, K. Saenko, Hardt, and Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 59304-59322. Curran As- sociates, Inc., 2023. URL https://prooceeding.. neurinsp cttpcccppppppeeeeeeeeeeeee httppppe..illleppppee///0/000/ http///222222712222222a444442223
pdf.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
17

---

Published as a conference paper at COLM 2025
A Examples of texts in register classes
Below we show examples of texts classified as selected registers. These texts were chosen randomly but with focus towards brevity (with some texts additionally truncated), general topics, and non-toxic language use. We also omit new-line characters.
How--to-nntrrutions (HI)
Home & DIY How To Cut Your Own Hair At Home With countless non- essential businesses closing due to the govemneet-imlenenneed lock- downs, many people have been forced to be creative and resourceful in order to get by. Case in point, not everyone of us are lucky enough to be quarantined at home with professional hairdresser. So we do the best we can - which often means cutting our own hair. [...]
Interactive Discussion (ID)
tried last night to compile standalone which I've compiled approximately 1, 000, 000 times before, and have been met with a long series of very lengthy crash reports. As far as can tell, something in the compiler is trying to install every framework on my computer, and sometimes wind up with 1 gig+ .mxf files. Naturally, little dinky tests compile fine, and can compile all of the modules that make up my patch, but not the whole thing at once. I did super- clean Max reinsiall, no third third-party objects aside a couple of Lobjects which are up to date, and I'm stumped. The only difference can think of since I compiled this few weeks ago is upgrade to 4.5. Anyone have any light to throw on this? Im happy to send crashlogs, but they's re big. 10.5.2, Max 4.6, Jitter 1.6.3 thanks, M
Machine Translation (MT)
Incredible Interior Design Sketches Good Looking Interior Designer Re- markable Interior Design Exceptional Interior Design Sketches Cozy Inte- rior Design Sketches Chic Interior Design Sketches Cool Interior Design Sketches
News (ne)
Wal-Mart has signed a storm water settlement agreement with the Connecti- cut Department Environmental Protection concerning alleged violations at 20 Wal-Mart stores and two SAM'S CLUB locations in the state. Under the agreement, Wal-Mart will pay $600,000 in civil penalties for violations alleged to have taken place between 1996 and 2003. Wal-Mart also will contribute $550,000 to two different supplemental environmental projects- $500,000 to assist municipalities in addressing storm water issues, and $50,000 for environmental projects in the Connecticut River Watershed. [...]
Opinion (OP)
Once again a very nice stop for the night. Staff has been very well trained, all are friendly and helpful with one exception/ the young man who checked us in. Breakfast was so appreciated after 12 nights at Holiday Inn Express which was ok but does not have your choices. Lynne does an especially good joboddoobppppeee aabaappecciteed all her help. They... More
B Model training setup
Models were trained following the setup of Penedo et al. (2024a). We used the GPT-2 tokeniser, Llama architecture and the same training settings: 1.71B parameters (1.82B with embeddings), sequence length of 2048 tokens, and global batch size of ~22 million tokens.
18

---

Published as a conference paper at COLM 2025
Models were trained for 50,000 iterations, which amounts to 100 billion tokens. Training was done on LUMI supercomputer on 16 nodes, each with a single 64-core CPU and 4 Ml250x GPUs with dual-GCD (graphics compute die). We used NVIDIA's Megatron-LM training framework instead of HuggingFace's nanotron framework used by Penedo et al. (2024a).
Training each model took approximately 84 hours, amounting to 10,752 GPU hours per model with the node setup described above, with average performance of 25 teraflops per second. Evaluation was carried out on a single GPU, and with 19 models, each with 50 checkpoints requiring approximately 20 minutes to evaluate, this added 300 GPU hours to the total computational cost. Other processing tasks, such as sampling and tokenisation, required no GPU resources and used comparatively negligible amount of CPU hours.
C Comparison to other data curation schemes
As discussed in Section 2, many different training data selection schemes and tools have been introduced in the literature. To quantify the connection and possible similarities between our register scheme and three other publicly available quality classification tools, FineWeb-edu classifier (Penedo et al., 2024a), the DCLM quality classifier (Li et al., 2024), and NVIDIA's NemoCurator Quality Classifier DeBERTa (He et al., 2023), we classify sample of over 50 000 documents from our pretraining material with all four classifiers. We compare the selected tool to our register classification scheme by conducting x²-test to evaluate dependence and also report metrics pertaining to the possible overlap.
The FineWeb-edu classifier assigns documents an ordinal label ranging from 0 to 5, the NemoCurator label 0, 1, or 2, and the DCLM classifier uses binary label (0,1). For registers, we use the classes in Table under "Individual registers". Obtained results from x²-test (considering FineWeb-edu and NemoCurator's 's labels nominal) show that all quality classification tools are dependent on the register classification results with p < 0.01. This dependence between the classification schemes is to be expected, as all schemes are used as a proxy for "quality" in LLM pretraining, To evaluate the strength of this connection, we calculated Cramer's V, which yielded the following results:
Registers- DCLM classifier: 0.136
Registers- FineWeb-edh classifier: 0.221
Registers- NemoCurator classifier: 0.419
In the context of machine learning, these values indicate low to moderate connection between the classifiers, with the NemoCurator showing the greatest association, which can be considered somewhat substantial. To account for the two ordinal classification schemes, FineWeb-edu and NemoCurator, we also calculated the 17²-metric which measures the amount of impact variable has the variance of another. These results were 0.173 and 0.273, for FineWeb-edu and NemoCurator, respectively, which similarly show a strong connection between the schemes but do not imply direct overlap.
Although the above analysis shows associations between the studied schemes, the register classes coincide with all classes of the three tools, with the exception of FineWeb-edu's highest quality class 5, which only occurs in conjunction with register classes Instructive- Informational and Informational Description. We visualise this in Figures 4, 5, and 6. This means that our register scheme can be seen as mutually reinforcing when combined with other established quality labelling schemes. We specifically see from our results that the 3 tools we compared to the register scheme often assign high-quality labels to documents from registers Informational Description, Description, and the case of NemoCurator, News, but neglect classes that we found beneficial in this study, such as fooo-to-Instructinn and Opinion. This highlights the value of using the non-binary but linguistically motivated and versatile scheme that the registers offer for curating data.
19

---

Published as a conference paper at COLM 2025
Description (dtp)
Narrative (NA)
Informational Description (IN)
Informational Persuasion (IP)
News (ne)
0
Machine Translation (MT)
Opinion (OP)
Iontrto-nstrucctinss (HI)
Spoken (SP)
Instructive-lnformational (HIIN)
Interactive Discussion (ID)
Lyrical (LY)
Figure 4: Visualised cross-classification for our register scheme and the DCLM quality
classifier. DCLM label corresponds to low quality and label to high quality.
Informational Persuasion (IP)
Machine Translation (MT)
Interactive Discussion (ID)
Narrative (NA)
Lyrical (LY)
Description (dtp)
Opinion (OP)
Informational Description (IN)
News (ne)
Spoken (SP)
2
How-to-Instructions (HII)
Instructive-lnlommatinna (HI-N)
Figure 5: Visualised cross-classification for our register scheme and the FineWeb-edu classi-
fier. FineWeb-edu label 0 corresponds to the lowest quality and label the highest quality.
Opinion (OP)
Spoken (SP)
Narrative (NA)
How-to-Instruce Instructioss (HII)
Description (dtp)
Sntrructive-nformmitinn (NN)-N))
News (ne)
Informational Description (IN)
Informational Persuasion (IP)
Lyrical (LY)
Interactive Discussion (ID)
2
Machine Translation (MT)
Figure 6: Visualised cross-classifccation for our register scheme and the NVIDIA NemoCu- rator classifier. NemoCurator label 0 corresponds to the highest quality and label to the lowest quality.
20

---

Published as a conference paper at COLM 2025
D Numerical results
We present the results in numerical format in Table 3. As stated in Section 3.3, we reweigh the accuracy outputted by Lighteval, acc_norm, the calculation of acc_norm is heavily biased against MLLU. MMLU consists of 57 tasks, which are considered separate benchmarks by LightEval. This is why values of acc norm are lower than "Accuracy" in the table: average performance of our models on MMLUU is just under 0.3.
Pretraining dataset
Step
Tokens
Accuracy
acc_norm
stderr
1000
2 B
0.348450
0,269707
0,030520
FineWeb
50000
105 B
0.465567
0.330767
0.031929
1000
2 B
0.351006
0.264036
0,030214
How-to-Inntrructions (HI)
50000
105 B
0.446558
0.294467
0.030880
1000
2 B
0.352893
0.266001
0.030249
Instructioo-lnformaation (HIIIN)
50000
105 B
0.464502
0.314607
0.031506
1000
2 B
0.354339
0.265349
0.030291
HI-IN-HI-dtp
50000
105 B
0.465616
0.320815
0.031642
1000
2
0.352104
0.267529
0.030357
HI-IN-HI-dtp-OP
50000
105 B
0.475093
0.331857
0.032058
1000
B
0.354445
0.264463
0.030214
HI-N-HII-tt-OP--NA
50000
105 B
0.472279
0,319987
0.031603
1000
B
0.348062
0.263200
0.030155
HH-N-HI--Hp-OP-NNA--D
50000
105 B
0.463591
0,324652
0.031859
1000
2 B
0.353098
0.268480
0.030369
HH-N-HUI-IIp--P--AA------
50000
105 B
0.464738
0,325085
0.031820
1000
B
0.341536
0.265107
0.030274
HPLT v2
50000
105 B
0.457484
0.328140
0.032064
1000
2
0.336598
0.264045
0.030252
Interactive Discussion (ID)
50000
105 B
0.430686
0,298429
0.031214
1000
B
0.344821
0,270357
0.030517
Informational Description (IN)
50000
105 B
0.437347
0,333771
0.032217
1000
0.336226
0.261104
0.0301 46
Informational Persuasion (IP)
50000
105 B
0,393026
0,276895
0.030480
1000
0.322654
0.261681
0.030248
Lyrical (LY)
50000
105 B
0.357626
0,271479
0.030604
1000
2
0.331829
0,262588
0.030269
Machine Translation (MT)
50000
105 B
0.374349
0,281125
0.030821
1000
B
0.336791
0,263844
0.030236
Narratioe (NA)
50000
105 B
0.441469
0,297377
0,031149
1000
B
0.343446
0,262086
0.030109
Opinion (OP)
50000
105 B
0.446629
0.318284
0,031720
1000
2
0.340778
0.266882
0.030336
Spoken (SP)
50000
105 B
0.422314
0.303859
0,031381
1000
2
0.347940
0.268266
0.030463
Description (dtp)
50000
105 B
0.452347
0.329816
0.032027
1000
2
0.332720
0.257563
0,029931
News (ne)
50000
105 B
0.417933
0.304612
0.031457
Table 3: Accuracies for all models, first and last checkpoint. "Accuracy" denotes accuracy weighted by benchmark, which present in our figures. Reweighing was done to mitigate the dominance of MMLU in the results, see Section 3.3. acc_norm and stderr stand for average accuracy and standard error as given by LightEval.
21

---

Published as a conference paper at COLM 2025
E Performance by benchmark and register
Figures 7, 8, and 9 present the results over all checkpoints, with Figure showing results by register, others by benchmark. To augment readability, we again applied a rolling average. In Figure 7, the values are normalised with respect to the average benchmark score on the final checkpoint; for example, the Description (dtp) model outperforms the average of all registers by 0.1 points on the final checkpoint on the ARC Easy benchmark.
Instrutiive-nnoormatinna (HI-IN) Description (dtp)
constt--nstrucctons (HI)
ARC Easy
0.1
0.1
0.1
ARC Challenge
0
0.
Hellaswag
MMLU
-0.1
-0.1
-0.1
SIQA
-0.2
-0.2
-02
PIQA
20B 40B 60B 80B100B
20B 40B 60B 80B100B
20B 40B 60B 80B100E
WinoGrande
Opinion (OP)
Informational Description (IN)
Narrative (NA)
Commonsense
0.1
0.1
OpenbookQA
0
-0.1/
-0.1
-011
-0.2
-0.2
-0.2
20B 40B 60B 80B100B
20B 40B 60B 80B100B
20B 40B 60B 80B100B
Interactive Discussion (ID)
Spoken (SP)
Informational Persuasion (IP)
0.1
0.1
0.1
-0.1/
-0.1
-011
-0.2
-0.2
-022
20B 40B 60B 80B100B
20B 40B 60B 80B100B
20B 40B 60B 80B100B
News (ne)
Machine Translation (MT)
Lyrical (LY)
0.1
0.1
0.1
0
0
-0.1
-0.1
-0.1
-0.2
-0.2
-02
20B 40B 60B 80B100B
20B 40B 60B 80B100B
20B 40B 60B 80B100B
Figure 7: Models' performance by register. Results are scaled by the average last checkpoint performance to highlight the models' differences. Dotted lines to increase readability.
22

---

Published as a conference paper at COLM 2025
ARC Easy
HellaSwag
0.6
0.6
Instructie-Innoormationl (HI-IN)
0.5
Description (dtp)
0.5
How-to-Instructions (HI)
0.4
0.4
Opinion (OP)
0.3
Informational Description (IN)
20B
40B
60B
80B 100B
20B
40B
60B
80B
100E
MMLU
ARC Challenge
Narrative (NA)
0.32
0.32
Interactive Discussion (ID)
0.3
0.3
0.28
Spoken (SP)
0.26
News (ne)
0.28
0.24
Informational Persuasion (IP)
0.26
0.22
Machine Translation (MI)
20B
60B
80B 100B
20B
60B
80B
100E
Lyrical (LY)
SIQA
PIQA
0.42
0.8
0.41
0.75
0.39
0.65
0.38
0.6
0.37
0.55
20B
40B
60B
80B
100B
20B
40B
60B
80B 100B
WinoGrande
CommonsenseQA
0.56
0.54
0.35
0.52
0.3
0.5
0.25
20B
40B
60B
80B
100B
20B
40B
60B
80B
100B
OpenBookQA
0.35
0.3
0.25
20B
40B
60B
80B 100B
Figure 8: Accuracies of register models on individual benchmarks. Random -uess thresh- olds shown in grey when at least one model falls under it. Rolling average over billion tokens applied for ease reading overlapping lines.
23

---

Published as a conference paper at COLM 2025
ARC Easy
HellaSwag
HI-IN-HI-dtp
0.55
HI-NN-HI-dtp-OP
0.5
HI-N-HI--Ip-OP-NAA
0.45
HH-N-HI--m--P--AA--D
0.4
HL-N-HI-dmp-PPNN--IDSS
20B
40B
60B
80B
100B
20B
40B
60B
80B
100E
MML.U
ARC Challenge
FineWeb
0.31
0.3
HPLT v2 (all registers)
0.3
0.28
0.29
0.28
0.26
0.27
0.24
0.26
20B
40B
100B
20B
40B
60B
80B
100B
SIQA
PIQA
0.8
0.415
0.41
0.75
0.405
0.4
0.395
0.65
0.39
20B
40B
60B
80B
100B
20B
40B
60B
80B 100B
WinoGrande
CommonsenseQA
0.58
0.56
0.35
0.54
0.52
0.3
0.5
20B
40B
60B
80B
100B
20B
40B
60B
80B
100B
OpenBookQA
0.35
20B
40B
60B
80B
100B
Figure 9: Accuracies of combination models on individual benchmarks. Random Random-guess thresholds shown in grey when at least one model falls under it. Rolling average over billion tokens applied for ease reading overlapping lines.
24


Reviews:
[{"review_id": "kHI2i9pxiF", "reviewer": "Reviewer_4S5f", "summary": "This paper presents how documents of different registers in the pre-training data may impact the performance of the trained models. Register is a linguistic concept for the variety of language used for a particular purpose or particular communicative situation. This paper obtains register labels for documents in a large-scale pre-training data source HPLT v2 using a BERT-based classification model, and trains multiple 1.7B models on datasets of different register mixtures. Via evaluating the trained model on multiple standard LLM evaluation benchmarks, the authors observe varied LM performance across different register mixtures, and find certain register mixture can lead to better performance compared to the original document mix.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper addresses an important problem in training LMs for better understanding the pretraining data and how it impacts the downstream performance. It is clearly written, with proper details for the experimental setup and training/evaluation seems to be reasonable. \n- The analysis (and visualization of the results) is nice, and it may offer a new angle to study the pre-training data.", "reasons_to_reject": "- I have some questions in terms of experimental details and how it is generalizable. I think one key question I have is that to what extent the register mixture is different from existing data selection and mixing methods, or whether ablating registers offer a very different angle for people to analyze the training data. Right now it seems the paper doesn't compare with other data mixing methods, which could be further strengthened with additional details and comparison. (See more in the question section below.) I am happy to increase the scores if the main concerns are addressed.", "questions_to_authors": "- First, can you provide examples of documents for each of the registers in the dataset? In addition, it seems the categories in table 1 is not the full set of register labels produced by the model (see ErikHenriksson 2024) -- what happens for those removed categories? are they removed? \n- I think one important missing category seems to be \"code (or programs)\" -- existing literature shows that code has a significant impact for language models during pre-training. None of the categories seem suitable for code (that's the reason why I ask about the first question), and it seems in this paper it doesn't talk too much about code. It would be helpful if the authors can provide details for this. \n- Further, can the register labeling mixture strategy results in very different mixtures in other datasets? for example, in the dolma paper (dolama: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research) in appendix M & N, they report detailed ablation results comparing how different data mixtures impact the training. I feel doing the exact same replication studies might be challenging, but some analysis along that direction should be helpful. \n\nAlso some questions about figures: \n- The visualization of the training is nice -- for figure 3, can you have a different version where you can plot different benchmark results for the same data mixture in one plot, and it would be interesting to compare the trend across different mixtures? \n- In figure 4, why there are dotted lines?", "extra_scores": {}}, {"review_id": "ERJGYYjC13", "reviewer": "Reviewer_ak3C", "summary": "This paper studies the effect of pretraining data curation by register on language model performance. Register refers to different genres of text, such as how-to-instructions, narrative, and lyrical text. The paper performs a large number of data ablation studies by pretraining many 1.7b-parameter Llama models on different registers and on combinations of registers. The suite of trained models is evaluated on multiple general-purpose downstream tasks, such as HellaSwag and MMLU. The paper finds that some registers, like instructive-informational, lead to higher performance when trained on in isolation, and that combinations of registers lead to even higher performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper performs a large data curation ablation study that can help model pretrainers decide how to curate data. The experimental setup is clear and sound, with appropriate baselines and care taken to account for different dataset sizes.\n2. The paper offers clear findings, such as training on instructive-informational data leads to higher downstream performance on the evaluated tasks than training on other registers.", "reasons_to_reject": "1. The paper would be made stronger by situating your results against those of other pretraining data curation papers that train on different kinds of data. While this is the first paper to my knowledge that curates based on register, some of the papers you mention in related work do curate based on similar notions of topic distribution or data source. For example, how do your findings complement the data domain ablations in Figure 5 in Longpre et al. 2024 or the coverage experiments in Sachdeva et al. 2024?\n2. You evaluate all of your models on general-purpose benchmarks like HellaSwag and MMLU. What about more targeted evaluations that might highlight strengths and weaknesses of different registers? I know you mention this in the limitations section already, but it does seem relevant to determining the potential benefits of different registers.", "questions_to_authors": "1. In Table 1, you list descriptions for the combinations as, e.g. '1/3rd of HI-IN, HI, dtp'. What does this mean? The number of available tokens for the combinations don't match up with 1/3rd of each of the individual registers.\n2. Your conclusion mentions that some relationships between register and downstream performance. Could you elaborate on this?", "extra_scores": {}}, {"review_id": "Cmopk4Hlpe", "reviewer": "Reviewer_syA5", "summary": "This article reports an interesting observation: it is possible to train an LLM on a smaller corpus, provided that the data is filtered to include only certain linguistic registers, without significantly affecting its performance on standard benchmarks.\n\nIt joins a long line of research on corpus filtering training, to which it makes a relevant contribution.", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A well-written, interesting article that reports an interesting observation that might be useful for people who want to train LLMs.", "reasons_to_reject": "This article raises several concerns that, in my opinion, currently prevent it from being suitable for publication, despite the fact that I find the main conclusion interesting.\n\nFirst, regarding matters of form: I believe it is essential to provide a detailed description of the model architecture in the main text of the paper. This is not a minor detail that can be relegated to an appendix — it is central to understanding and evaluating the experimental setup.\n\nSecond, I do not understand the rationale behind presenting evaluation results across all training checkpoints rather than focusing on final performance (e.g., the last or best checkpoint). The evaluation strategy is never explicitly discussed by the authors, and this choice makes it particularly difficult to interpret or compare results — especially between Figures 1 and 2, which are central to the article. A table summarizing final metrics would have enabled a clearer and more quantitative comparison.\n\nIt would also have been valuable to include a comparison of training costs across models. This would help assess to what extent reducing the training corpus size actually translates into practical efficiency gains.\n\nMore fundamentally, the article feels incomplete: the authors report an intriguing observation — that some corpora yield better-performing LLMs — but make no attempt to explain it. While it may seem trivial that a poorly constructed dataset would lead to underperformance, and while it is well known that training and evaluation distributions should align, it would have been insightful to explore which properties of the data might influence model quality. Addressing this would also contribute to a more foundational question: how can we select training data without having to train models on all possible registers or combinations thereof? As it stands, the work merely shows that such a selection might be beneficial, without providing any guidance on how to make it. A straightforward experiment, which could be easily conducted, would help begin addressing my concerns: it would be informative to examine how the classifier used for data filtering categorizes the various examples in the test corpus.\n\nFinally, a broader question: would it not have been relevant to investigate the impact of the model architecture — especially its size? Could reducing the training data also be a strategy for justifying or enabling smaller models?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yjzd2ImHyq", "reviewer": "Reviewer_W4AW", "summary": "This paper describes experiments on training LLMs on data from different registers (genres) and how it affects performance on a given selection of tasks. The authors separate the training data into different subsets based on genre/register and train models both on individual subsets and combinations thereof. All models are evaluated on 9 benchmarks (all English).\n\nThe paper is well put together and easy to follow, but I'm left a bit unsure as to what the conclusion is: A mix of genres seems better overall, but performance on the individual tasks varies depending on the subset of data used. This seems quite intuitive? Models trained on certain registers perform bad across all tested benchmarks, e.g. lyrical which consists of song lyrics and poems. But the benchmarks all require the model to perform some sort of reasoning, so it really is not surprising that lyrical texts do not work well as training data here?", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is a good read, easy to follow and with good graphics to condense the large number of results.", "reasons_to_reject": "As stated above, I'm a bit unsure as to what the conclusion or finding is - a lot of it seems quite intuitive:\n1. genre/register matters for performance on a given task \n2. some combinations perform better than using the entire dataset\n\nRegarding 1., this is not surprising and for 2., how specific are these combinations to the selected benchmarks?", "questions_to_authors": "As stated above, the paper is really well written and clear, so I have only one question:\n\nWhat exactly is the MT register? Is this machine translated text? The English side of parallel corpora used to train MT models? But both of those would consist of different genres/registers (news, instructions, opinion texts, etc.), so I'm not sure how this constitutes its own register? Or is it something else?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is a cool paper! I'm fairly familiar with pretraining data practices, so I can see how this overlaps with some related work, but I agree with the authors that most of the notion of "register" in modern language modeling is not know. The paper is well written, the experiments make sense, and it is bringing a classic linguistics idea into relatively modern pretraining experiments (~2B models, 100B tokens). 

3 of the 4 reviewers agree this is a strong paper and the authors have worked on the manuscript to address their concerns. The one low review score seems to be focusing on more minor issues, i.e. the points raised by reviewer syA5 seem minor detractions to the COLM reviewer / score guidelines where overall the paper checks many boxes

I'm not sure if the method will catch on as being "central" to language modeling development, but I see it being adopted in complement to "domain" sampling that's done today. 

Regardless, I am happy to recommend accepting this paper. This seems like a very solid COLM contribution.
Decision: Accept