DataMan: Data Manager for Pre-training Large Language Models
Authors: Ru Peng
Kexin Yang
Yawen Zeng
Junyang Lin
Dayiheng Liu
Junbo Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Data Selection
Pre-Training
Primary area: foundation or frontier models, including LLMs

Abstract: The performance emergence of large language models (LLMs) driven by data scaling laws makes the selection of pre-training data increasingly important. 
However, existing methods rely on limited heuristics and human intuition, lacking comprehensive and clear guidelines.
To address this, we are inspired by *``reverse thinking''* -- prompting LLMs to self-identify which criteria benefit its performance. 
As its pre-training capabilities are related to perplexity (PPL), we derive 14 quality criteria from the causes of text perplexity anomalies and introduce 15 common application domains to support domain mixing.
In this paper, we train a **Data** **Man**ager (**DataMan**) to learn quality ratings and domain recognition from pointwise rating, and use it to annotate a 447B token pre-training corpus with 14 quality ratings and domain type.
Our experiments validate our approach, using DataMan to select 30B tokens to train a 1.3B-parameter language model, demonstrating significant improvements in in-context learning (ICL), perplexity, and instruction-following ability over the state-of-the-art baseline. 
The best-performing model, based on the *Overall Score l=5* surpasses a model trained with 50% more data using uniform sampling. 
We continue pre-training with high-rated, domain-specific data annotated by DataMan to enhance domain-specific ICL performance and thus verify DataMan's domain mixing ability. 
Our findings emphasize the importance of quality ranking, the complementary nature of quality criteria, and their low correlation with perplexity, analyzing misalignment between PPL and ICL performance. 
We also thoroughly analyzed our pre-training dataset, examining its composition, the distribution of quality ratings, and the original document sources.

Published as a conference paper at ICLR 2025
DATAMAN: DATA MANAGER FOR PRE-TNAINING LARGE LANGUAGE MODELS
Ru Peng Kexin Yang² Yawen Zeng' Junyang Lin" Dayiheng Liu Junbo Zhao
'Zhejiang University AAlibaba Group
trupeng, J. z ) 82 82ju.edu.co
liudayiheng wawdyyhellaabbiinnadn com
ABSTRACT
The performance emergence of large language models (LLMs) driven by data scaling laws makes the selection of pre-training data increasingly important. How- ever, existing methods rely on limited heuristics and human intuition, lacking comprehensive and clear guidelines. To address this, we are inspired by "reverse thinking" " prompting LLMs to self-identify which criteria benefit its performance. As its pre-training capabilities are related to perplexity (PPL), we derive 14 quality criteria from the causes of text perplexity anomalies and introduce 15 common application domains to support domain mixing. In this paper, we train Data Manager (DataMan) to learn quality ratings and domain recognition from point- wise rating, and use it to annotate a 447B token pre-training corpus with 14 quality ratings and domain type. Our experiments validate our approach, using DataMan to select 30B tokens train 1.3B-parameter language model, demonstrating significant improvements in in-context learning (ICL), perplexity, and instruction- following ability over the state-of-the baseline. The best pest-performing model, based on the Overall Score I=5 surpasses a model trained with 50% more data using uniform sampling. We continue pre-training with high-rated, domain-specific data annotated by DataMan to enhance -omain--peccfic ICL performance and thus verify DataMan's 's domain mixing ability. Our findings emphasize the importance of quality ranking, the complementary nature of quality criteria, and their low correla- tion with perplexity, analyzing misalignment between PPL and ICL performance. We also thoroughly analyzed our pre-training dataset, examining its composition, the distribution of quality ratings, and the original document sources.
INTRODUCTION
As large language models (LLMs) demonstrate performance emergence driven by data scaling laws, the significance of data has become increasingly evident (Kaplan et al.) 2020) Brown et al. 2020
Chowdheery al. 2023). This trend prompts researchers to explore how to select pre-training data,
including deduplication (Lee et al., 2022), domain mixing (Gao et al. 2020 Shen et al., 2023). heuristic- based data selection (Rae et al.) 2022 Wenzek et al., 2019), and data sampling using LLM quality signals (Gunasekar et al.] 2023 Wettig et al., 2024a). Although these efforts aim to enhance data quality and diversity, deduplication and domain mixing are solely used as a priori or post-hoc steps in the data selection process. Furthermore, existing data selection methods typically rely on limited heuristies and human intuition, lacking comprehensive and clear criteria for data selection, thus making the selection of ideal pre-training data for LLMs unresolved challenge.
Following this line, this paper provides guidelines for selecting pre-training data, including quality cri- teria, application domains, and Data Manager (DataMan) with a comprehensive quality rating and domain recognition, equipped by data sampling strategics to enhance the LLM performances. Firstly, we believe that excellent quality criteria must: 1)-apply to a wide variety of texts; 2edemmssttette deep understanding of content, capturing semantic levels; and 3)-complement each other. However, existing studies (Rae et al.) 2022 Wettig et al. (2024a) rely limited heuristics and human intuition, while grounded in empirical findings, lack generality and comprehensiveness. To address this issue,
"This work done during internship at Qwen team, Alibaba Group.
Both are corresponding authors.

---

Published as a conference paper at ICLR 2025
Reverse Thinking
15 Common Domains Medicine, Finance, Low....
TextA: Hello, what are you doing today! Top 2% PPL
14 Quality Criteria Accuracy, Coherence, Lunguage Consisteccy Semantic Density, Knowledge Novelty, Tople Facus, Creativity, Professionalin, Style Consistency, Grammotiedl Diversity, Structoral Sundardizution, Originaiit, Sonsitivity Overall Score
TextB: Colls can be divided into prokaryotic cells and eukaryotic cells
Super LLM
Human Expert
Bottom 2% PPI.
95% Agreement
Train DatuMan
Accuracy:
Book
Gitbub
Coherence:
Ardiv
Wiki
CommonCCrood &
Domain Type:
Web-scale Corpus
by DatuMan
Select and Mix Subset
Train Lunguage Model
The pipeline of the Sample- with--wtaaleeeee model: We derived 14 quality criteria from LLMs' reverse thinking and used DataMan annotate the quality rating and domain type of pre- training data. By employing data sampling strategies to select subsets, the performance of trained LLMs outperforms the state-of-the data sampling baseline.
Figure 1:
we are inspired by "reverse thinking" prompting LLM to self-identify which criteria benefit its performance, as its pre-training capability is related to perplexity (PPI.) (Muennighoff et al. 2024 Marion et al., 2023b). We extracted documents with the top 2% and bottom 2% PPL from different sources and used Super LLM to identify the reasons for anomalous in document perplexity. Through iterative refinement, we derived 13 quality criteria related to LLM performance: Accuracy, Coherence, Creativity, Grammatical Diversiry, Knowledge Novelty, Language Consistency, Original- ity, Professionalism. Semantic Density, Sensitivity, Structural Standardization, Style Consistency, and Topic Focus, and formed comprehensive criterion called Overall Score. Further, introduced 15 common domain types in the LLM application industry (Naveed et al. 2023) for domain mixing and developed full prompt curate a fine-tuning dataset. We employ a small-scale LLM fine-tune DataMan via text generation and apply pointwise rating over pairwise rating (Liu et al. 2007) for more applicable, cost-effective inference on vast datasets. Using DataMan, we annotated 447B tokens in the Slimpajama corpus (Soboleva et al.) (023)) with quality ratings and domain types to create the DataPajama dataset. While ensuring the diversity of sources and domains, we maximized the representativeness of quality criteria, sampling 30B token subset from the DataPajama and trained the Sheared-LLaMA--1.. 1.3B language model (Xia et al. 2023) from scratch.
In our experiment, we initially validated the feasibility of using prompt as it achieved over 95% agreement with human preferences. We then revisited the limitation of pairwise rating (Wettig et al. 2024a), bounded pointwise rating error by its model loss, and fairly ranked the annotation results from both ratings to explain, from three perspectives, why we chose pointwise rating. While training the DataMan model, we thoroughly analyzed its fine-tuning datasets, hyperparameter search, inference accuracy and efficiency of three model versions, and misclassification issues. Further, We evaluated language models trained on 30B tokens sampled with different data selection methods from the DataPajama dataset. In ten downstream tasks, the Sample-with-Dee DataMaD model, trained on data sampled with Dataman's 's 13 quality criteria, outperformed the existing state-of-the- thettheartt (SOTA) bascline (Educational value - = 2) in in-context learning performance by 0.4% to 4.3%, showcasing the effectiveness of these criteria. As the Overall Score rose from to both ICL performance and PPL improved significantly. confirming the necessity of quality ranking. Our strongest model was achieved at Overall Score 1=5, even surpassing the model trained on uniform sampling with 50% more data, as the Overall Score encompasses attributes of all criteria, supported by its high correlation with other quality criteria. To solidify our results, we sampled a larger 60B data subset and compared the strongest Sample-wint with-Watahhan model Overall Score 1=5 against the existing SOTA baseline. Mcanwhile, we found PPL and ICL performance are not aligned strictly. Through correlation analysis and visualization, we reveal that PPL reflects general understanding, whereas
2

---

Published as a conference paper at ICLR 2025
ICL captures downstream generalization ability. In the instruction following tasks, all Sample-witt DataMan models overwhelmingly surpassed the existing SOTA baseline with a win rate between 67.1% and 78.5%. Additionally, We continued pre-training the strongest Overall Score I=5 model using high-rated, domain-speciffi data annotated by DataMan, achieving superior ICL, performance in specific domains, thereby verifying DataMan's capability for domain mixing. We also conducted an in-depth analysis of the DataPajama dataset, explored the distribution of DataMan's quality ratings from various sources, and inspected the original documents corresponding to each quality rating=11,,3,4,555 we also identified complementary relationships among quality criteria, while their low correlation with perplexity further confirms the novelty of DataMan's an' quality criteria.
Our contributions are summarized as follows:
1. Based on the relationship between LLM performance and PPL. "reverse thinking" allows LLMs to self- elff-ideniff which criteria benefit its performance, assemble with typical application domains to guide data selection.
2. We introduce DataMan, offering comprehensive quality ratings and domain recognition, along with data sampling strategies to optimize LLM pre-training. Vast experiments validate its efficacy, setting new performance records.
3. We will release the code, all models, and the annotated DataPajama dataset, paving the way for the community to explore the guidelines between data and LLMs further.
2 RELATED WORK
Deduplication. Deduplicating training data is now standard in managing pre-training data for LLMs, as it greatly impacts performance (Lee et al., 2022). While Kaplan et al. (2020) and Hoffmann et al. (2022b) examine scaling laws with unique data trained for one epoch, some studies (Hernandez et al. 2022 Muennighoff et al. 2024] Xue et al.
2024) suggest that repeated data can harm performance, particularly as repetition epochs and model size grow, Additionally. fuzzy or semantic deduplication improves LLM performance (Jiang et al. 2022 Abbas et al.) 2023 Tirumala et al.) 2023). But deduplication should precede quality-basee selection and domain mixing, and is unsuitable for
sampling fixed-sin size subsets.
Heuristic-based Selection. This selection approach includes two methods: Rule-based heuristics, which apply manually crafted
such as mean word length, stop word fraction, and word repetition thresholds (Laurençon al., [022 TogetherAl] 2202 Penedo al. 2023 Soldaini et al.) 2024) to filter data, exemplified by the C4 filter (Raffel et al. 2020) and Gopher rules (Rac al. 2022). These reduce noise effectively but need complex manual design. In contrast, Model- based heuristics use models like binary grammar discriminators (Chowdhery et al.] | Touvron cet al., 2023a) to select data resembling the target domain, alongside techniques like importance resampling (Xie et al. 2023) and perplexity filtering (Wenzek et al.) 2019 Muennighoff et al. 2024
Marion et al. 2023a) However, they require more precise quality ratings for optimal selection.
Domain mixture. Most pre- training datasets, like the Pile (Gao et al. 2020), comprise mixed data from various sources and domains (Nijkamp et al. 2023 Zhang al. 2023] Yang et al., 2024bb Maini et al. 2024 Li et al.) 2024). As LLMs gain traction, domain
data for improved functionalities are increasingly used in model training (Du et al.) 2022 Gao et al., 2023). Identifying the optimal domain mixture ratio is essential for effective LLM pre-training (Wang et al. (2023). Early attempts to define this relied on experiments and intuition (Gao et al. 2020 Thoppilan et al. 2022). Recent studies have begun use automatic methods, such as domain generalization (Xie et al. [023 2024)) domain gradients (Fan et al. 2023). and loss evaluation (Xia et al., (023). to assign domain weights and assess model performance across various mixtures. This inspires our work to introduce domain types to assist in pre-training data mixture.
LLM quality signals. Appropriate LLM quality signals are valuable for selecting pre-training data (Gunasekar et al.) (003).. Research shows that data enriched with facts and trivia aids LLMs in accurately addressing niche topics and fictional worlds (Petroni al.) 2019 Korbak et al. 2023). Other studies have emphasized the educational value of data as key for enhancing LLMs reasoning
capabilities (Wei et al. 2022 Kojima al. 2022). Recent efforts have integrated these insights,
3

---

Published as a conference paper at ICLR 2025
proposing four quality wrrting style, facts and trivia, educational value, and required expertise- to assess pre-training data and enhance LLM capabiliiiess (Wettig et al. 2024b). However, these criteria depend heavily on human intuition and thus lack generality and comprchensiveness.
ANNOTATING TEXT BY DATAMAN
We developed DataMan (see Figure T. data manager that comprchensively annotates 14 quality criteria and domain types, enabling ideal data selection and mixing.
3.1 OVERVIEW OF THE ANNOTATION
Let {4 (l,,...... b,. represent all documents to be annotated. The annotation results of DataMan correspond to querying the ratings of document in across all quality criteria and its domain type. Assume that the quality ratings and domain recognition are in multi-level annotation format as ........ (n); where 1, E {1..... is the label for the C'tth criterion or domain of document l. For this paper, K is for quality ratings and 15 for domain recognition. Let F be a class of functions, and f E F be the annotation function. We use super LLM as the annotation function, recording the annotation results for each criterion and domain, expressed as: {(1, [(,,2)= for DataMan, S [L:, learning to rank (I.2R) model (Liu et al., 2007 2009). We minimize the loss function based on the
L= ==( .1),, .1))................ )..... (L,
".l,,...... 14) (.)....,........
(6n, 1 ........ ( Thus, we can quickly create fine-tuning dataset 19)}. Essentially, our method uses pointwise rating by pointwise
document, annotation labels, and function:
L(fit,C) = Σ - 4))
i=1
This trains DataMan to learn the annotation functions for each quality criterion and domain type.
3.2 PROMPT CURATION
How to define the quality criteria and domain types of texts? We believe that excellent quality criteria should: 1)-apply a wide variety of texts, 2)-demonstrate deep understanding of content, capturing semantic levels, and 3)- complement each other. However, prior studies largely relied on human intuition, such as educational value in Gunasekar al. (2023), writing style in Wettig et al. (2024a). and toxicity and privacy in Korbak et al. (2023). which lack generality and comprehenstveness. To address this issue, we undertook an in-depth exploration of text quality, motivated by "reverse "hinking"-prompiipp prropting the I.IM to self-identify which criteria benefit its performance. Specifically. since LLMs' pre-training abilitics are closely related to their PPL [Muennighoff et al. 20244 Marion iet al., 2023b). where "high PPL indicates data is difficult to learn, and vice versa". we focused on the training text from various sources with the top 2% and bottom 2% of PPL. We devised an analytical prompt for Super LLM investigate the reason behind these perplexity anomalies in documents, aiming to analyze the traits of both casy-to-leand and difficult-to-lean data. Through iterative refinement, we derived 14 quality criteria as shown in Figure ④ Additionally, we identified the 15 domain types that need to be assessed from typical LLM application industries (Naveed et al. 2023), integrating the "ler's think step by step" chain-of ofain-of-thought prompting strategy (Wei et al. [0022 and thoughtfully designed system prompt. Further details of the entire process are in Appendix A
Prompt validation. We validate prompt effectiveness using clear cases before prompt use. Initially, we gathered a pool of documents preliminarily rated by an independent group, splitting them into two sets of ten documents high--igh-rated and low-rated-too to ensure distinct quality gap. Table 4 of Appendix A details these document sources. These 20 documents were randomly shuffled and then rated on a scale of 1-5 based on each quality criterion by five independent human annotators who had not seen them before. Subsequently, using the super LLM with our prompt, we evaluated the same documents and compared super L.LM ratings with human majority votes, finding over 95% agreement. Also, we ensured inter-rater reliability among human annotators by calculating the Kappa coefficient (McHugh) 2012). which validated the consistency of human ratings. However, this human validation remains subjective, we hope the community develops more rigorous method.
'The Super L.M can independently refer to any advanced L.M.M In this paper, We use the state-of-the-art LLM available that time, GP1-4-0125 255preview
4

---

Published as a conference paper at ICLR 2025
3.3 WHY USE POINTWISE RATING?
Using LLMs to rate text falls into two categories: explain why we chose the pointwise rating from the following three aspects:
pointwise rating and pairwise rating. Here, we
Pairwise rating limitations. We revisited pairwise rating (Wettig et al. 2024a) and observed minimal quality differences among the top-3 documents ranked by writing style, even humans struggle to discern differences (see Table (5) This highlights the limitations of pairwise ratings when quality differences are marginal, as documents meeting an "acceptable" quality threshold should be accepted, where pointwise rating aligns well with human judgment. Appendix B.1 notes that creating fine- tuning dataset requires Super LLM to make 40 predictions per criterion and document pairs, then use LLM preferences as labels. Table 5 shows that document pairs need a 250% selection probability difference for quality criteria, also required for inference in Table 6, thus questioning the practicality of pointwise ratings. For N documents, pointwise ratings only need N ratings, whereas pairwise ratings demands N * (N - 1) comparisons with 40 predictions each, greatly increasing costs. Despite both ratings capture different aspects, pointwise rating's broader applicability and cost-effiectiveness in fine-tuning dataset curation or pre-training data annotation make our preferred choice.
Bound pointwise rating error. We present the mathematical connection between rating errors and loss the pointwise rating model (Chen et al. 2009).
1/2
1 1 NDDG(;;4) <
15v2 Nn
Σ D(O)) " D(UPPN)) (LU;
(2)
NDCG(fit,C) = Nn
G(((T
(3)
Na max G(
where NDCG is rating measures defined with respect K-level ratings f. G is the gain function, T is the rating list produced by the rating function f, and D is the position discount function. One usually sets G(z) = 2 1, D(2) = [og.[1+2] if v M, and D(2) = 0 if 2 > M (M a fixed integer). Thus, as the pointwise rating model minimizes loss via training, the rating error also reduces, validating the feasibility of pointwise ratings theoretically.
Fairly compare both rating results. To ensure a fair comparison despite differing evaluation metrics, we ranked annotations from Qurating's pairwise and DataMan's 's pointwise ratings on the same 58,824 documents from the first split of Qurating's IB analysis se@ We then presented the top, median, and bottom ten documents based on Qurating's 's four criteria writing style, facts and trivia, educational value, and required expertise- and DataMan's 's Overall Score with sum of the remaining criteria. All results are provided herd"] Our conclusions are: H)-Higher Qurating criteria correlate with higher DataMan's Overall Scores for the top and bottom 10 documents, and vice versa. Med-an 10 documents show minimal changes in Qurating's 's criteria, yet their Overall Scores vary from 3 5, supporting that "Pairwise rating struggles with minimal quality differences, while pointwise rating matches human judgment." iii)-Rankings by Overall Score highlight top-quality documents, especially STEM fields. iy>-The "application domain" field accurately categorizes domains, aiding in Nomain-specinf continue pre- vre-training, as shown in Table
3.4 TRAINING THE DATAMAN MODEL
After developing the prompt and rating method, we collected documents from both in-source and out-of-source of the SlimPajama corpus (Soboleva al. 2023) train DataMan. By prompting the Super LLM, we obtained 14 quality ratings and domain types for each document, creating a fine-tuning dataset of 35,700 documents at a cost of $13,858. Documents were limited 2,048 tokens (averaging 810 tokens), surpassing the [256, 512] token range of Wettig et al. (2024a) when handling sentences with broader length variations. Subsequently, we fine-tuned the DataMan model with
5

---

Published as a conference paper at ICLR 2025
Qwen2-1.5B (Yang et al., 2024a) using text generation loss. In Table & the DataMan model achieves near 80% average test accuracy across all criteria, with an Overall Score test accuracy of 81.3% and domain recognition test accuracy of 86%. Appendix B also provides details the fine-tuning dataset statistics, average scores for each quality criterion per domain, correlation among quality criteria, three DataMan model versions, the chat prompt for text generation, the hyperparameter search, reports on inference accuracy, misclassification analysis and inference efficiency.
4 MANAGING DATA BY DATAMAN
In this section, we apply the DataMan model manage data by selecting high-quality. diverse document subset from the pre-training corpus. Each document di in the corpus D, with source st, is an- notated by the DataMan model with 14 quality ratings and domain types as C = ((.,....,4 1-111 gg) where q is the domain type. Given the source and domain distribution probabilities P(s) and P(1) respectively, we perform top-k (k is the selected subset size) sampling without replacement for each quality criterion across source and domain distribution, using the probability:
P(d. 1, s, 9) P(s.9) P(d.) E Edy P(dj 9) P(s.q)
and P(dj())
(5)
EdyeD
This method maximizes sample representativeness based on quality rating while ensuring diversity in source and domain distributions, and sampling without replacement prevents duplicate data. To assess the Overall Score covers all criteria, we substitute top-k with uniform sampling based on fixed Overall Score ratings. These techniques implicitly steer language modeling toward reward- weighted regression (Peters & Schaal [2007] Korbak et al. 2023), expanding maximum likelihood estimation loss with data reward mechanism.
EXPERIMENTS
We empirically validate the DataMan method by training the language model from scratch.
5.1 EXPERIMENTAL SETUP
DataPajama. We used the Llama tokenizer (Touvron al. 2023a) to segment Slimpajama cor- pus (Soboleva et al., (003), a cleaned and deduplicated version of the RedPajama (TogetherAT 2023). into 1024-tokens documents. The DataMan model then annotated these documents with 14 quality ratings and domain types, creating the 447B token DataPajama for pre-training. Although the annotation process is expensive, it can be reduced via large-scale parallelization and cost-effective DataMan models. The quality ratings and domain types in DataPajama can serve various purposes, such as data selection, data mixing, or continued pre-traiming in specific domains. Detailed statistics of DataPajama can be found in Table Of Appendix C
Data selection methods. For each baseline, we select 30B token training dataset from DataPajama with one of the following methods, while retaining the original source proportion as the overall dataset. We leave it as future work explore combinations of quality criteria broaden our method.
Uniform: We select randomly with uniform probability across documents. For comparison's sake, we train an additional model on 45B tokens, requiring 50% more compute.
DSIR: We apply data selection with importance resampling (DSIR) (Xie al. 2023) and select examples that resemble either English Wikipedia or the Book domain (TogetherA] 2023)
commonly used as proxies for quality (Brown) 2020) Touvron et al. [003a]] Xie et al.] (023)) We
follow Xie et al. (2023) and train hashed bigram models on Datal Pajama and the target data.
Perplexity Filtering: We implement perplexity filtering (Wenzek et al. 2019 Marion el al. 2023a) and select the documents with the lowest/highest perplexity scores, computed by a pre-trained Sheared-Llama-2.10 7B model (Xia et al. 2023 -2x the size of our DataMan model.
Sample with Qurating: We sample 30B tokens according to each of the four criteria described in Wettig al. 2024a): writing style, facts and trivia, educational value, and required expertise. Specifically, we normalize the variance of the quality ratings to be and then sample with temperature E {0.0 (i.e., top-k selection), 2.0). Additionally, we merge the Qurating-sampled

---

Published as a conference paper at ICLR 2025
data for 7 2.0 of the four criteria as criteria mix, and subsampling is randomly to 30B tokens, ensuring that we exclude duplicate documents.
Sample with DataMan (Ours): For each of 13 quality criteria, we perform top-k sampling based on the quality ratings and domain type described in Section 4 Additionally, we explore sampling 30B tokens with overall scores / E {1, 2, 3, 4, 5} to demonstrate the effectiveness of all criteria.
Training settings. Using different data selection methods, we select a 30B token subset from DataPajama and train randomly initialized Sheared-Llama- for one epoch in a randomly shuffled order. We train on slightly more data than the compute-optimal ratio (Hoffmann et al.) (0222a) because more training tokens better reflect the performance gains attributed to data quality, Further details are in Appendix C
language model (Xia et al. 2023
Evaluation metrics. We provide holistic evaluation of the language models trained on 30B tokens:
We measure the perplexity over SlimPajama's 's validation set and test set, 500M tokens each.
We evaluate the in-context learning (ICL.) performance using 1m-eva aaa i oo-harness. (Gao et al. 2224). We study 10 tasks, comprising reading comprehension tasks -ARC-easy/challenge (Clark et al. 2018). SciQA (Welbl et al. 2017). LogiQA (Liu al. 2020). BoolQ (Clark et al.
2019)), commonsense reasoning tasks (HellaSwag Zellers et al. 2019). PIQA (Bisk et al. 2019)
WinoGrande (Sakaguchi et al.) 2019)) and 2 iooteigg-intensive tasks (NQ (Kwiakkoski et al.
2019), MMLU (Hendrycks al. (02T)). We report the detailed settings in Appendix C
We evaluate the instruction-followin capabilities of our models, borrowing the setting used by Xia et al. (2023). We perform supervised fine-tuning on 10,000 instruction-responss pairs from the SP dataset. We evaluate another 1,000 instructions and use the AlpacaFarm codebase (Dubois et al. 2024) to judge the responses from two models with GPT-40.
5.2 RESULTS
We report the model's perplexity and ICL results Table [ the instruction-following evaluation in Figure 2
the domain-specific continue pre-training performance in Table A provides comprehensive results for all models, including validation and test perplexity across sources in and ICL results for individual task in Table 14
Tables 12 13
Traditional methods perform poorly. Table I indicates that DSIR and perplexity filtering perform worse than random uniform sampling. This indicates that model output output-based methods, despite their widespread use, are ineffective for data selection.
Clear quality criteria work, but Qurating's 's criteria mix fails. Table shows that selecting data with Qurating's s four criteria (Wettig al., (024a) improves performance compared to uniform sampling. highlighting the importance of clear LLM quality signals. Educational value = 2.0 is the current SOTA baseline. However, the criteria mix of Qurating's 's four criterion did not perform well, possibly due to a lack of complementarity among the Qurating's 's criteria.
DataMan's criteria surpasses SOTA baseline and Overall Score works best. In Table compared to the SOTA baseline (Educational value = 2.0). our 13 individual quality criteria improved ICL performance by 0.4% to 4.3%. For example, the sumple-with-creatiiee impressive score of 60.6 in commonsense reasoning tasks. As the comprehensive criterion Overall Score increased from to 5, performance gains significantly, highlighting the necessity of quality ranking. Our Overall Score |=5 works best, even exceeding the Uniform +50% data baseline. validating the feasibility of combining the 13 quality criteria into a composite criterion via LLM weighting. This approach not only avoids the disruption of manual adjustments but also achieves optimal results. The correlations among all criteria in Figure 5 further confirm this point.
model achieved an
PPL and ICL are not strietly aligned. Our results in Table I reveal a trend where PPL and ICL metrics correlate to a degree, increasing or decreasing simultaneously, but they are not aligned strictly. This meets with the intuition that PPL implies general understanding. while ICL focuses more on downstream generalization. Further analysis see Figure 6 in Appendix|C Notably, DataMan's high-rated Overall Score achieves an optimal trade-off between understanding and generalization.
7

---

Published as a conference paper at ICLR 2025
Table Sample-with-DataMeee models improve perplexity and in-context learning (ICL) results, and the Overall Score 1=5 perform best. We report the validation, test perplexity, and ICL performance of 10 downstream tasks. We highlight the best result in each column and improvement over uniform sampling with the 30B token budget. In Appendix C 13 and ICL in Table
we report full results of perplexity in Table T2
Reading Comprehension (5 tasks)
Commonsense Reasoning (3 tasks)
World Knowledge (2 tasks)
Val Perplexity
Test Perplexity
Selection Method
Average (10 tasks)
Uniform
10.7
10.75
50.9
14.9 15.9 t10
44,9
+50% data
10.09 inst
10.14 Las
52.9 120
57.0 f20
46.8 T1.
with Wiki
13.34
13.37
50.1
49.8
14.7
42,9
DSIR
with Book
13.60
13.59
47.9
56.6
14.1
43.8
lowest
15.98 taas
16.04 1829
48.3 126
49.6 13A
13.7 432
41.7 182
Perplexity
highest
11.32 2062
11.34 10.99
49.6 413
53.5 41.5
13.4 415
43.5 4144
Writing Style
top-k
13.01
12,97
49.3
53.3
13.5
43.4
7 2.0
10.60
10.64
51.0
55.8
14.1
45.0
top-k
14.38 tia
14.33 t338
54.3 134
51.7 133
15.5 THA
45.8 TWV
Facts & Trivia
10.68 10.22
10.72 J088
52.7 018
55.6 106
15.6 107
46.2 013
top-k
13.54
13.49
54.7
54.9
14.4
46.7
Educational Value
7 2.0
10.67
10.72
53.3
56.3 113
15.7
46.7
top-k
14.97 1832
14.92 153
52.8 T1M
48.7 463
14.3 Une
43.9 L19
Required Expertise
10.7
10.74
L0o1
52.7 018
55.5 163
15.0 T0U
46.0 (1.
Critcria mix
2.0
10.63
10.68
52.1
55.5
15.2
45.7
Accuracy
top-k
10.82
10.80
fiee
53.8 T2M
58.2 t32
16.6 t12
47.7 12k
Coherence
top-k
10.72
10.71
L004
54.9 140
58.8 138
16.1
48.3 134
Creativity
top-k
11.08
10.88
11.00
7035
53.0 131
60.6 18A
15.2 1e3
47.7 838
Grummatical Diversity
top-k
10.87
10,86
55.1 182
133
16.5 fia
48.5 138
Knowledge Novelty
top-k
11.01
11.01
54,6
56.9 11.
15.5
47.5 12.6
Language Consistency
top-k
10.35 2038
10.35 1040
54.1 132
59.3 143
16.7 f1.
48.2 133
Originality
top-k
10.68 unde
10.67 Lons
53.9 130
58.6 136
16.4
47.8 12.9
Professionalism
top-k
11.27 1057
11.26 TA51
54.6 137
54.8 483
15.9 11.9
46.9 13.9
Semantic Density
top-k
11.10 70400
11.09 1104
54.4 138
16.7 f1.
48.0 133
Sensitivity
top-k
10.11
10.13
54,7 138
59.2 143
16.1
48.3
Structural Standardization
top-k
12.11 110
12.11 11.36
53.7 178
57.0 130
17.1 173
47.4 135
Style Consistency
top-k
10.74
10.73
55.1
59.6 146
16.2
48.7
Topic Focus
top-k
10.41 20.39
10.41
54.6 137
58.4 TM
15.6 TAY
47.9 15.
|=/
1131
23.95
113.0
428
133
13.1 11.8
38.4 U6.
Overall
1=2
12.84
82.14
12.91
02.16
50.3 986
50,9 UAT
14.7
43.4 41.
Score
I=3
11.75
11.78
01.88
50.7 403
54.1 163
15.2
44.6 403
1=4
10.21
10.22
53.5
60.1
16.0
47.9
1=5
10.52
10.50
55.2
60.2 152
17.4
49.1
DataMan's instructionnfollloinng abilities also well. As shown in Figure 2 we compare the instruction-following win rates between the Sample-with-Dttaaan model and the SOTA baseline (Educational value 2.0). The results indicate that, under the same supervised fine-tuning conditions, our model consistently surpasses the SOTA Baseline, with the Overall Score reaching an impressive win rate 78.5%, further validating the superior performance of our method.
Training domain-specine models. While the Overall Score |=5 achieves the best performance, its performance in specific domains can still be improved, as shown in Table 2 To address this, we applied DataMan's 's domain recognition to filter data with high Overall Score the medical, law, and financial domains, and continued pre-training domain-specifer models, achieving ICL performance gains in the corresponding MML.U subtasks. This validates DataMan' capability for domain mixing.
8

---

Published as a conference paper at ICLR 2025
Overall Score55
78.5%
21.5%
Educational Value
Catetence
72.0%
27.4%
Educational Educational Value
Senslisitt
72.0%
0%
Figure 2:
Instruction following win rates of Hample-with-DttaMMee models v.s. the state-of-the-art base- line (Educational value = 2.0) af- ter instruction fine-tuning on 10K ShareGPT examples. The results indicate that, under the same SFT conditions, our model consistently surpasses the SOTA Baseline, with the Overall Score reaching a im- pressive win rate at 78.5%.
Darsity
Creativity
71.4%
26.8%
Eouc and Value
Knowledge Navelty
70.3%
Educational
Straciunal Standandicaiinnn
70.0%
30.0%
Educational Value
Grammetical Diversity
70.0%
30.0%
Educational Valuw
Topl: Facus
30.2%
Educational
Consistency
69.5%
30.3%
Eoucational Value
Origiality
Educational
Language Consistency
69.2%
36.0%
Eoucational Value
Educational
Accuracy
67.1%
32.9%
Educational Value
Table 2: The Sample-with DataMan model further improves perplexity and in-context learning (ICL) results when continuing pre-training on high Overall Score, domain-speen -specff data. We report the validation, test perplexity, and ICL performance of corresponding MMLU subtasks.
withata-it
Val Perplexity
Test Perplexity
Anatomy
College Medicine
Medical Genetics
Overall Score 1=55
8.10
8.05
28.1
24.3
22.0
Medicine CPT
8.11
8.11
30.4
26.6
30.0
Val Perplexity
Test Perplexity
International Law
Professional Law
Jurisprudence
Overall Score 1=5 + Law CPT
7.92
8.13
25.5
33.9
22.2
8.06
8.34
35.5
24.7
24.1
Val Perplexity Test Perplexity
Econometries
High School Macroeconomies
Marketing
Overall Score Jm.5 + Finunce CPT
9.51
9.49
23.7
33.3
22.2
9.59
9.63
25.4
34.9
23.9
5.3 ANALYSIS OF QUALITY RATINGS
Distribution of quality ratings, In Figure 3 we present the distribution of quality ratings across different sources in DataPajama. Overall, the quality ratings for each source primarily concentrated at 4 and 5, indicating generally high sample quality. This may be related to the fact that DataPajama a subset of the curated and deduplicated slimpjama corpus. However, for the criteria of Knowledge Novelly and Creativity, there is a higher proportion of samples scoring 2 and 3, which is consistent with the lower average scores for these criteria found in Table 7 Across all domains, only a few scientific domains like mathematics and medicine have Knowledge Novelty scores above 3, while in Creativity, only culture and entertainment scores were high at 3.64 and 3.56, respectively. Nevertheless, in DataPajama, the combined share of domains like mathematics and medicine is only 11 11.5%, and similarly, the combined share of culture and entertainment only 25%, both of which are relatively small. This explains the modest ratings of the DataPajama dataset in terms of Knowledge Novelty and Creativity.
Correlation between quality ratings and log-likelihood. In Figure 4 we illustrate the correlation between quality ratings and the loglliilddood scores computed by Llama-2-7b (Touvron et al. (023b). Most quality criteria do not show significant correlation with perplexity. except for the criteria of Structural standardization, Professionalism, and Creativity, which have Spearman correlation coefficients ranging from 0.47 to 0.55, indicating weak correlation. This indicates that our 14 quality criteria and sample-with-datamann method are independent traditional perplexity metrics and filtering, indirectly showcasing the sophistication of the "reverse thinking".
5.4 DATA INSPECTION
Furthermore, examined examples of original documents from each source under Dataman's quality criteria and ratings. Specifically, for each criterion, we randomly selected samples with ratings ranging from to 5 from different sources and presented them in the Appendix Notably, these

---

Published as a conference paper at ICLR 2025
Conemabe
Larguege Danu
Crowladge
Tpp Pecz
Drabbaa
313
Cons
Clammotkel Dee
221
Seuttare Sund
Smothby 38 zaa
DeeviSSond
Commeniraai
Gbub
Wicppeds
Stassiichange
Figure 3: The distribution quality ratings across different sources in DataPajama
Assuracy
Comerance
Languaga Contboany
Semwnts Danily
Knonklege novily
Tapic focas
Creatiity
w
Profascralon
Giammatiical Stuendy
Seuctari Standardladian
Oriimaityy
Sensibity
Overal Scora
Figure 4: Correlations of quality ratings and negative log-likelihood scores by Llama-2-7B (Tourron et al. (0233) over 30B tokens training documents. The negative log-likelihoods averaged over the number of tokens, and are the logarithm of the perplexity score a single sequence. We observe that perplexity scores not good approximations for any quality criteria.
samples represent only small snippet; nonetheless, they exhibit significant quality differences. We invite readers to review these differences in detail, which compares high and low ratings.
CONCLUSION
In this work, we introduce DataMan, a comprehensive data manager capable of quality rating and domain recognition, developed to facilitate the pre-training data selection and mixing. By using DataMan to annotate pre- pre-training datasets and sampling data based on quality ratings while balancing domain diversity, our trained models demonstrate significant improvements in language modeling, task generalization, and instruction following. Through extensive experiments, we provide valuable insights for the community studying the relationship between data and large language models.
10

---

Published as a conference paper at ICLR 2025
7 LIMITATIONS
We acknowledge several limitations in our work. First, DataMan's reliance on LLMs for text quality assessment and domain categorization may inherit biases from these models. Second, DataMan's inference accuracy is not yet optimal, sometimes causing misclassification, highlighting the need for large-scale collection of documents with diverse quality differences for fine-tuning. Third, using SlimPajama alone as a pre-training corpus limits result reliability, incorporating additional corpora would better. Fourth, the model size is restricted by data and training resources, resulting in models with only 1.3B parameters, whereas increasing parameters might reveal interesting phenomena. Lastly, the considerable costs of developing data managers, data filtering, and pre-training experiments could hinder further research in this domain. We aim to address these in future work. Despite these limitations, DataMan remains a powerful tool for data selection and mixing.
ACKNOWLEDGEMENT
This work is supported by the Fundamental Research Funds for the Central Universities (226- 2024-00049), the NSFC Grants (No. 62206247) and the Pioneer R&D Program of Zhejiang (No. 2024C01035).
REFERENCES
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari Morcos. Semdedup: Data- efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piga: Reasoning about physical commonsense in natural language, 2019. URL https::/arriiv.or////b/s
1911.11641.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14166 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https
alaxxxiorggabs/////oo http://///aab/0001111110 2005.14165.
Wei Chen, Tie-Yan Liu, Yanyan Lan, Zhi-Ming Ma, and Hang Li. Ranking measures and loss functions in learning to rank. Advances in Neural Information Processing Systems, 22, 2009.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson. Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Heiir-Hellstemm, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Christopher Clark, Kenton Lee, Ming- Ming-Weiii Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL httpes///axxivaorr///////11510.0

---

Published as a conference paper at ICLR 2025
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. /ttp://///aaa....o.//aaaa///////
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experee In International Conference on Machine Learning, pp. 5547-5569. PMLR, 2022.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15392 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite. Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.000 1000000777 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou, framework for few-shot language model evaluation, 2024. URL
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304. 2ne.:2334.55111 15010, 2023.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint wrXiv:2403.13257. 2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306. 1644, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Mcasuring massive multitask language understanding, 2021. URL https:
//arxiv.org/abs 200900030000000
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage. Zac Hatfield-Dodee Tom Henighan, Tristan Hume, al. Scaling laws and interpretability of learning from repeated data. arXiv preprint 2rXiv:2205.10444 10487, 2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendrieks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint
2022a.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35:30016-30030, 2022b.
Tao Jiang, Xu Yuan, Yuan Chen, Ke Cheng, Liangmin Wang, Xiaofeng Chen, and Jianfeng Ma. Fuzzydedup: Secure fuzzy deduplication for cloud storage. IEFE Transactions on Dependable and Secure Computing, 20(3):2466-2)
2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 20Xiv:2001.08800 2020.
12

---

Published as a conference paper at ICLR 2025
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-222133 2022.
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pp. 17506-17533 PMLR, 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452-466, 2019.
Woosuk Wwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: 1.6 th composite multilingual dataset. Advances Neural Information Processing Systems, 35:31 1809-3033300 319933116....00 2022.
Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better, 2022.
URL /ot.://cpss//eeeivvrrraa/a////./
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang. Xingxing Zhang, Haoyang Huang. Shao- han Huang, Xiaolong Huang, Zeqiang Huang. Dongdong Zhang, et al. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402. 13064, 2024.
Jian Liu, Leyang Cui, Hammeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning, 2020. URL https
/ //aaxii.org//////eeiee /bt////a///////// 200700822
Tie-Yan Liu, Jun Xu, Tao Qin, Wenying Xiong, and Hang Li. Letor: Benchmark dataset for research on learning to rank for information retrieval. In Proceedings of SIGIR 2007 workshop on learning to rank for information retrieval, volume 310. Citeseer, 2007.
Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends@ in Information Retrieval, (()):225-311, 2009.
Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephras- ing the web: A recipe for compute and data-cfficient language modeling. arXiv preprint arXiv:2401. 16380, 2024.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint (tle:2309.04664, 2023a.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang. Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale, 2023b. URL https: 7 //axxiv.or/abbs http:org//abb//3..44555.55 2309.04564
Mary L McHugh. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-282 2012.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv preprint arXhv:2307.06433, 2023.
Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. Codegen2: Lessons for training Ilms on programming and natural languages. arXiv preprini MrXXix3335.003000 2023.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon I1m: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 16, 2023.
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weightt regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp. 745-750, 2007.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint 2019.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Pagamini, Laurent Sifre. Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, lean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong. Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li. Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart. Simon Osindero. Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis insights from training gopher, 2022. URL https::/aarxi..orr/aab.. http:///aaab///.14.......
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https://arxiv.org///////
1907.10641
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.
Zhiqiang Shen, Tianhua Tao, Liqun Ma. Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, et al. Slimpajama-de: Understanding data combinations for Ilm training. arXiv preprint 230.. 2023.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPa- jama. http://pps////eebrraaa////ggg///
jama- a 627b toweettoeeeeeeeeeeeeeee jama 2023. UtL.https./huugiigaaa..c. (ataaetss/cerebraa51111e... 627B
Recupp_ccaee--verssoooo--eeeee_. jama
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint 2:xXiv:242200/555 2024.
14

---

Published as a conference paper at ICLR 2025
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 568: 127063, 2024.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- -zze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.082) 2022.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving IIm pretraining via document de-duplication and diversification. Advances in Neural Information Processing Systems, 36:53983-53995, 2023.
TogetherAI. Redpajama: An open source recipe to reproduce laaaa training dataset, 2023. https: //git
hwwpeeerrccomuutteeeeeeeeee amm Saaaa-Daa -Dtta
TogetherAI Redpajama: an open dataset for training large language models. 2023.
https:/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a. URL htt.::tp//// http://arxvvvooooo hwtp//o/oor//aaa.oooooo/...bbbsa abs/ 23302119977
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Zige Wang, Wanjun Zhong, Yufei Wang, Qi Zhu, Fei Mi, Baojun Wang, Lifeng Shang, Xin Jiang, and Qun Liu. Data management for large language models: A survey, arXiv preprini arXiv:2312.01700 2023.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11123-1112 1302, 2019.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of- prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2498 2022.
Johannes Welbl, Nelson F, Liu, and Matt Gardner. Crowdsourcing multiple choice science questions, 2017.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Cenet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:
2019.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739, 2024a.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739, 2024b.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre- treiirann via structured pruning. arXiv preprint arXiv:2310.0m 10.06694, 2023.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34227, 2023.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024.
Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling Ilm under crisis. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
An Yang. Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li. Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024a. URL
httts::/aarxivorgg abs/ 2407.10671
Rui Yang. Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems, 36, 2024b.
Rowan Zellers, Ari Holtzaan, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019. URL httttss///axxx..org////a// http://.wrr//aa//155.00700
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init 2023.
attention. arXiv preprint
16

---

Published as a conference paper at ICLR 2025
A
PROMPT CURATION
Full Prompt
Please carefully read and analyze the following text, score based on fourteen evaluation criteria and their respective scoring definitions. Additionally, select the most appropriate category from fifteen domain types that best matches the content of the text. Let's think step by step.
Text:(tex)
Domain Types: [A]Medicine [B]Finance [C]Law [D]Education [E]Technology [F]Entertainment [G]Mathematics [[]]Coding [I]Government [J]Culture [K]Transportation [L]Retail E-commerce [M/Telecommunication INAArricutture [OJOther
The Higher The Score, The Evaluation Criteria:
[AAcuuryy:: the fewer grammar, referential, and spelling errors the text contains, and the more accurate its expression. _/5
[2]Coherence: the more fluent the content is expressed, and the stronger its logical coherence. _/5 L3nganee Consistency: the consistent the of language the text, with less mixing of languages. _/S
[4]Semantic Density: the greater the proportion of valid information in the text, with less irrelevant or redundant information. _/5
[5] Knowleege Novelty: the more novel and cutting-edge the knowledge provided by the text, with more insightful views on the industry or topic. _/5
16/oopic Focus: the more the text content focuses on the topic, with less deviation from the main theme. /5
[7]Creativity: the more creative elements are shown in the text's expression. _/S
[8|Professionaiimm:: the professional terminology appears in the text, with more accurate of terms and more professional domain specific expression. _/5
19]Style Consistency: the more consistent the style of the text, with proper and appropriate style transitions. _55
[10]Grammatical Diversity: the more varied and correct the grammatical structures used in the text, showing richer language expression ahility. _/5
[1]]Structural Standardization: the clearer structure followed by the text and the more standardized its format. _/5
[12]Originality: the fewer repetitions and similar content in the text. _/5
[13]Sensitivity: the appropriately sensitive topics handled in the text, with less inappropriate content. //
[14]Overall Score: the better the comprehensive evaluation of the text, with superior performance all _5
Our full prompt shown above, where [text] represents the text to be annotated. Next, we will elaborate the entire prompt curation process, including obtaining all quality criteria, domain types, and system prompts, which were conducted by experts with the assistance of Super LLM:
Initializing quality criteria. Inspired by "reverse thinking" -promiptin- the LLM to self-identip which criteria benefit its performance, as its pre-training capabilities are closely related to perplexity (PPL) (Muennighoff et al. 2024 Marion et al. (023b). To this end, we devised an analytical prompt for Super LLM to investigate the reasons behind textual PPL anomalies (in the top and bottom 2%) from each source, and extract the initial quality criteria below. For clarity, Table provides several examples illustrating how these initial quality criteria are derived.
Enhancing quality criteria. Next, we utilize Super LLM to rank the initial quality criteria by importance, eliminating, merging, and supplementing them. The resulting second-step criteria include: [1] Text accuracy (grammar, references, spelling), [2] Semantic coherence and consistency, [3] Language consistency, [4] Effective semantic content ratio, [5] Knowledge novelty, [6] Topic focus, [7] Creative expression ratio, [8] Proportion of technical terms, [9] Style variability, [10] Complexity of grammatical structures, [11] Content regularity, [12] Content redundancy, [13] Proportion of sensitive topics. We then revised the prompts under the principle that higher scores indicate better criteria. With Super LLM's assistance, we simplified the criterion names, detailed the criteria for each rating level, and ultimately derived 14 quality criteria in the paper.
17

---

Published as a conference paper at ICLR 2025
Initial Quality Criteria
[1] Semantic FlueneyyCohereecc/llggee Evaluate whether the text is smooth and easy to read, whether the content is coherent, and whether the logic is clear.
[2] Content Aaasistency/Variabllii in Language Style: Evaluate if the information within the text is
contradictory and if the language style is diverse.
[3] Topic Diversity: Determine the richness and variety of topics addressed in the text.
[4] Content RegularityFFommating: Consider whether the text adheres to a certain structure or format.
[5] Content Redundancy: Analyze the extent of information repetition within the text.
[6] Proportion of Domain-Specifie Vocabulary: Measure the frequency of professional terms or specific vocabulary used in the text (such proper nouns, technical terms, Classical Chinese).
[7] Proportion of Sensitive Topics: Examine the percentage of content that addresses sensitive topics
(e-g., involving politics, toxicity).
[8] Proportion of Creative Expression: Assess the degree of creative or innovative expression in the text
(e.g.. use of thetorical techniques).
[9] Degree of Language Mixing: Analyze the extent which different languages are used within the text (i.e.. the ratio of text in various languages).
[10] Complexity of Text Structure: Evaluate the overall complexity of the text's structure.
[11] Proportion of Long Sentences: Assess the ratio of long sentences within the text.
[12] Proportion of Grammatical, Referenee, and Spelling Errors: Evaluate the ratio of grammatical errors (e.g., incorrect punctuation, unclear sentence breaks), reference errors, and spelling mistakes in the text.
[13] Proportion of Content Lacking Semantics: Determine the ratio of parts within the text that lack meaningful content (e.g., garbled text, HTML tags, XML elements, navigation bars, incomplete chart numbers, or disjointed citations).
Identifying domains. We identified the 15 domain types that require assessment, based on factors such as typical application L.M industries (Naveed et al., 2023), the number of existing industry LLMs, and the level of attention they have received, indicated by metries like GitHub stars
Chain-of-though and system prompts. We incorporated the chain-of-thougeee prompting Wei et al. 2022)-"ler's think step by step" wep"-whilk avoiding using few-shot example prompting, the diverse sources of text could introduce rating biases. We allowed the Super LLM to generate system prompts to enhance the accuracy and confidence of quality ratings, "You are an expert to evaluate the text quality with high accuracy and confidence. Don't II hesitate to use the full range of the score scale, including extreme scores the text warrants it." " Results indicated that this prompt facilitated more nuanced text annotations compared to the typical prompt, "You are a helpful assistant.". To ensure stable output from Super LLM, we set the temperature 0.
the
Table We provide several examples of excessively high and low perplexity, use analytical prompts to investigate the reasons for these anomalies, and extract initial quality criteria from the LLM's responses. The analytical prompt is: "Read the following document, which has a (high/low) perplexity of (apuee for LLM inference. Please analyze the reasons for the PPL anomaly."
Text
Response
March: 1.77 inches- April: 5.31 inches- May:
Stractural consistency: Data prw in a fiaed Fornat. ([4) Content hegularity/Fomaatiing Redundancy: & large aumber of repeating elencats. [[5]Content Redundancy) back complex syntaxs The text lacks complax sentence scrustcres. [[10]Comlaxxity of text Structure)
5.9 inches Junes 5.9 inchwa July: 5.9
Inches- August: 2.38 inches- Saptenber: 5.31
Inches- october:
inches- Hevenber: 2.36
incliss becember: 2.36 inclaw2009, Sennwvill
26 Presipitation: 56- Jansary: 1.58 inchas-
Febrnerys 2.24 inchex- Harch: 2.24 inches.
Agril: 5.04 inches-
5.5 inches Junes 5.6
Mininal context depenience. ([n]annattic
inches- Julys 4.48 inchea- Ascuat: 2.24 inches-
September: 5.6 inobes Detober: 4.48 inches
Novenbee: 2.3 inchwa- December: 2.24 200,
Sannevilla gc Frecipitation
Continne on nest page
18

---

Published as a conference paper at ICLR 2025
PPL
Text
LLM's Responee
1,07
4But the wice took oil in their vessels with their lamps. lw bridegroon tarried, they clumeered and ciept. sind at midright there was ary made, Behold, the bri orioegroon comethy go ye out rThen all those virging
Textual Standardizationn Rib' ([3]Topic Diversityl Structura. and Redundant: The nerretive has stsuctore and redundancy, [[4]Content Regularity/Fomattinn [[5] Content Redundancy) Contextual sonsistency: The parts et the text have internal coherence. [[2]Content Consistency/Variabll texical Gramaticaal Consietency: Bible asc fixed vocabulary and gromaticaai strJatures. (110)Compexity of Text Structure)
is
excerpt
and trimeed theig lamps. 8ina the toolich soid unie the wiee, Give us ot your oily for our lamps are gene out. seut the xise anzwered, saying, Not soi Lest there be rather then that sell, and buy for yourselves.
for and your
ye
Language style)
185.1
An ragent bas, nd eelisbig lint lakes as, test ollemrchinng moliivaliiona ideal avoids, tafe prolbl since 11: entlllle that peisort, sho are conll'nt lchrou.ghhit their litetine xithout axcnoviedlging zare Hin any way are initntolibly criticized telr exaibitirng; sleplombble lstck ot ot Inler-regeirs, and in part loufar, Fad. of enga.con.,.ll2 with and gratilede tox, experitir persons who conntriaule to their lives it verinls ways. dodidings's s our view la tha't eluprisingly little ettort to sieciluuge our to receive or Voomplet, linosher P, Py-roolll's cazes wa niny respend with an approprriale alihlde such recognition and/ ree gratitude lbid:: 19;
Complex Contents The text deeply discusses philosaphy Terminology Denze: Understanding reguiies in degth donain knovledge. [[6)
ethics.
receive envtet tron others. neribigiinng
Proportion Demain-Specific Vocabulary)
Incoherent Text: tiple parts incoharenee. ([3]Topic Diversity) Randon Charecters: There may be encoding errore. [[1]]Proportion of Gramatical, Reference, and Spelling Errors)
167.5
The Indian Exprese is now Telegran, Click here to tein our channel (eeneianexpreeet stay updated with the latest inadlinessor all the latest Solkata News, downlaed Iodian Pxpress App- Midnchengiachees Michsel Caisance ist vorzestig xieder ins Training eingessiegen. will in der neuen Saison mehr spielen als in der abgelauzener, Rorzum: Der 19-Jhd' brige will eich bei Torossia Käönchengl adbach durchsetzen chael kif rgers nit Frankreich bei der 320-M.. Fobo: AP/Darko Vajinew coul sance startet bei Borussia seinen Angrite aur die Startelrbala too rmsed tacSacte des eb-IPtarteosi and eanst was tar illnagen apenaps chontges reenwe islc.
Ri Rixed languages: À nix of various languages and spelling eysteme. ([9]Degree of Language Mixing) Insoherent Sentences: Poor cornect ivity betveen sentences.
([2) Content consieteney/Variablll Language Style) Anbiguous Context:
und
Lack clear contextua informat ion.
[13]]roporttin of Content backing Senantics) Formatting Krrors: Contairs fornattiing ing isques, typos, and garbled text. ([12)Proportion of Granmatical, Reference, and Spelling Brrors)
ki josr
frkijhe
uas med weda, eelgeaMisket da
72.19 property on sard the autura the token and ridding it of dispirigednese coldy
Text content is complex ex and varied: The text you provide looks "ike it powns literaty works, with lots et metaphors, symbols, and abat exgressions. S[n]Santttic Suueesy//olarenne//aagee licegular
bath pernitted and the knight)l also the noumaine ine had reviving al firslliinetiin in the hollows from peak Lo golden peak and there veretWWlais for reabers which alteres the lonel conelinesse Fise always bhe wished for his exan even its chasow evenlilla sandow on stones once vere story cherished along
Iny melles Lips retreshed then
cornet seructure: Peetry otten has di Ferert fornet and structure than regular prose, such as Hine breaks, irrepuiar punctuation, blank lines, Structure) Proper noune and archaions: The text my cortain proper rouns, archeisns, or vorde coired the aucher, which not be in the model's s vocabulary. ([6) Proportion of comain-ggecifi Vocabulary) Greative expression in texi: Because postrry ofter conteine lot of creative expression experimental use of language, beyond the seape of the modcl's predictions nssed or conventions. language usage. Propoottt of Creative Expression)
and sarmlWlaad the menory or his ride that as a basialllli tilled then erptied yet ice curve rensins and its dept would never forget nor the exact threw quarter WeiseWWehhee on tae tringes et cloues and the enbrace in the chasm) dewn erasing any doubt the act was modemn nountaia climbing Jooking
indentatior.
[[1]]aoplexiity
snow and
surplns wrinkle centiansllllam mackWWWted with diches of With knight was no exception he even woze his hair like legend' W l qustatnecs ha plied hinselrk\WStack pretending he had an occupation like Wlet course it was the samelilleve the ni litary and the religious wtareves there crowd
lot from need for
forth
Contanee on next page
19

---

Published as a conference paper at ICLR 2025
PPL
Text
LLM's Response
1.91 torner 1970s antivar redical togitive wanted for murder, Afte- living for years underground, Grant mast now go on the run, With the FBI in hot pursuit, be sete ott track down person that can cleas his asme, Shepard knews the tignt ot the nationel newe story he has expezed and, fox a journallst, this is an opportunity ot litstime. Hell-bent maxing for hinselt, he willing stop nothing to capitclizo on it. Bo dies deep into Grantsies past, Despite warnings tron ais editor and threats the F2I, Shepard releatiesaly tracks Grant across the country. Vs. As Grant reopens sid weunds and recornects with comer nembers of his antivar group, the Neathez Undezaround, Shapard realizes senethine
model, can't talk about toplar involving politios rel gion,, poreggrphhy, vioicnet, etc, (17]eroportion of Sensitive Topice)
cross-countey jsurney
this
jast
up.
118.75
Weddoocr'1569 = 258dsess2188g 11 c--o- 0- Id-o
(-(d-
Non-stardard characters and symbols:
contains aarge number worde, such as "3"," a*," -t, ";" Such special coaracters and punctuation marks that
13810c2271850 serli-111 2211--1310-410--t----1
h-y-t-ip-ix----fpe
commn in nornal language exprossicn, ([13)Propoxtion tacking Semanticn) Seroled characters and randon strings: egcurrences in the text similar "dsoci7,, 'p q-rl Soca garbled or candon stringe are unpredictable to the language model besouse they are meaningless in normll languaçe represertation. ((13)Proportion of Content Lacking Semanties)
--11-- azl a11---11100111300-t---1
Content
c-Id-o
c-1d-o
arll-figa:- gil--fgg---i-gg- (A--fijgg-4.-.. 44. 1-10-
c-10--
Table 4: For each quality criterion, we collected a set of documents initially rated by an independent panel and divided them into two sets of ten documents cach-biihhrrated and low-rated-to ensure distinct quality gap. We used this data for prompt tuning and to validate the agreement between the prompt uses and human preferences. This table describes the sources of the documents.
Criterion
Sources
Accuracy
High
Academic joumals, technical manuals, professional reports.
Low
Social media posts, personal blogs, informal emails.
Coherence
High
News seports, research papers, essays.
Low
Forum comments, random lists, random collections of paragraphs.
Languago Consistency
High
Business documents, logal contracts, academic essays.
Low
Bilingual posts, casual conversations, mixed language blogs.
Semantic Density
High
Research reports, market analyses, white papers.
Low
Adverising copy. social media posts, forum Q&AA.
Knowledge Novelty
High
Cutting-edge rescurch papers, conference presentations, expert interviews.
Low
Common tutorials, listicles, ouidated press.
Topic Focus
High
Specialized textbooks, academic papers on spocific topics, focused industry reports.
Low
Miscellaneous blug articles, off topic comments and social media content, unthemed discussion drufts.
Creativity
High
Poetry, creative writing, artistic critiques.
Low
Technical documents, routine business communications, standard emails, lengthy legal texts.
Professionalism
High
Formal technical reports, industry white papers, legal documents
Low
Personal blogs, informal tweets, children's litcrature.
Style Consistency
High
Published novels, professional speeches, maguzine articles.
Low
Articles with mixed styles, drafts of lettors, hastily written online reviews.
Grammatical Diversity
High Literary warks, academic articles, formal speeches.
Low Emails composed of simple sentences, children's 's reading materials, transcriptions of oral presentations.
Structural Staddadization
High Formal reports, standard operating procedures, structured propesals.
Low Free form writings, scattered notes, rough drafts.
Originality
High Research roviews, detailed analyses, varied essays.
Low
repetitive comments, simplistic online articles, redundant adventisig copy.
Sensitivity
High generic content, informed articles on sensitive topics, guidelines.
Low Crude social media content, unthoghhffl internet jokes, superficial news headlines.
20

---

Published as a conference paper at ICLR 2025
Table 5: We follow Qurating's Table 4 (Wettig et al. 2024a) by using 10 documents from different sources, ranking them by writing style, and using them to analyze pointwise and pairwise ratings.
Rank
Text
DataMan's Annotation
Anery Blaine interited from his nother every trwit, except the stray inexpressinle tew, that made him worth while. Bis zather, an the frectull inersioulate nan with a taste for Byron and a habit of drewsing ing over the Encyclopedia Britamios,, grew weal thy at thirty through the death ot two elder brothers, successtu. Chicago brokers, feel ing that
nccancy: 5 coherence: 4 lan-
gunge comsistency: semamic density: knowledge novelty: 2 topic Focus: 5 creativity: 4 profescionaliam: 3
style comsisency:
5 grammasi- diverity: stmectual standrdiizionn:
world was his, Bar Barbor and nes Beatricw of laza. In conseguence, Stephen Blaine handed down to posterity his heigat of
First
to
3 originaliy: 5 sensitivity 5 over- all scure: 4 domiin: culture
2.
Technologi
logies es for naking and nanipalatt in bielogy ever sinow the discovery of the bMA double belix. But introducing site-apecific medicicationa in genonea cells and organi ans remsined sluxive. Rarly approaches relled on the pe principle of aile apeci secugaition Lion of DNA sequenees by origonuclestides, small molacules, or solf-splicing introns. More ly, the site-di rected sing finger ruclesses 1798)) offectos sucleases (PALENa) using the principle of wice specific ...
his have enshled advances
renmey: s coherence: 5 lan- guage consistency: 5 semantic demitty: 5 knovledgr_novchy
topic focus 5 creativity: 3 profesionalism::
syye cousisaney:
5 gammati- cal.diisitty stoucural Concuura__tadddddzzaa standrdizzzion annggnuliyy sunsitiityy owerall
TAI.
5 domain: techuology
3.
The winter 1905-07 was the coldest in Alberta's a history and was exacerbated by a stortsge of cocl, Ona cause of this shortage was the streined cela.ionship belween usal nimwra and nime operaloza the beginaing
lan- guage comiisency: 5 semaniic density:
accuncy; 5 coherence:
5 knowicdge_novally 3 topic focus:
in the province. Goal and Coke Company locked out the niners from iss nine near Taber.
April 1907,
Canada Mest
5 creativity: 3 professionaism:: 4 stle consitency:
5 gammasi- cal divasityy 4 stuutural stmuuuua__ttddddddaee standurizzzon: unginality 3 seasitiiity: 5 soore dostain:
The sans company was also facing wozk stogpage at its mine in the Crew's Bast Fass, wheze minars vare refusing to sion new contract. problen spread until by April
4
on Decenber 3, Venezuela held a controversisl referendum over stain to the sil-rich Rasequibo secion contzolled by Quyana. that same day, the Vice President of Venezucla, Deicy Rodriquez, shares video on X, Cornerly Twittes, showing group of Indigunous people hoisting Vonazualan flag in its staad over the territory, which is alse known 25 Suayana Esecuiza. 'Clary to the brave people!' she arote, which la the fiest line of the countzy's a natioral anthem. The post cans
suumyy:: 5 coherence: 5 lan-
guage_ consistency: semanric unwleedeennvelly: 3 topic focus:
5 creativity 3 professianalim:: 4 style romessoosss
lowering Guyarase flae
5 grammati-
diversitty suucturral snussurl_s stanbrriatio ongginaliy 5 seasitivity: 3 seorall_scoeee 4 domain: goverament
5.
The Godtather is one or the nost praised novies in cinema history. sudiences alike ask gata 2ll the attention it gets for being
accancy: 4 coherence: J lan-
It gives everything that critics movics. In ny opinion one ot, or the best novies ever, One of the bast things The coatather does is its incredible casting and its iconie pertornances acters
gugge comiitency: semmiic density: knowledge, novelty: 3 topic focus: 5 creativity: 4 professimmlism:: syle consisency:
4 grammsti- cal diversiiyy; 4 stmuuurrl staddrdizaion:
frem each and evary sonvincing that wor the movic severai academy swarde. It a1so jurpetarted several actors, acting careers, and gave an ...
its characters.
originality: 5 sensiivity: 5 over-
all score: 4 domsin: entertaimmet
6
The food is good, bus not great value. Up front, xill just say,
lan- guage comistency: 5 semamic semamic_denslly:
setmey:
coherence:
do not wasto your time getting traditionsl sushi here because tht ie'ss not really that maki nigiri and while it was good, it vagn't that than cur tave sushi places. Insters, come here for their signature di and you'll 11 probsbly be heppler, He really enjoyed some of their signature diches. Ne dined edaname: tasty and spicy! Yellewtail
better, For exanple,
ordered
4 knowledge novelty: 2 topic focus:
5 creativity 3 professionalimm:
4 grammaii- ral diversity: 3 semucural stndrrdintionn
style comsisemuy:
party of 4 and had: spicy
3 uriginait:: 4 sensitvity: 5 scure domsin:
7
My Pather worked the ranks as softwase engineer and sanagement, has palents for the company that saved is nillions of dellars. Heess almost to panaion and suddenly HIB starts making his life miserao's e, lie not loed this Urend was happening to
Porbes 500 company since
Noved up
semmey:: 4 coherence: 4 lan-
guage. consisteney: semaatic denilty: 4 snowledee_naved 3 topuc focus:
4 cremivity: 3 professinalism: 4 stle cousiseney:
of his covorkers when they getting close to age 60 as well. ER Sady cails bin into the ottice and says that hhe was not punching in and out at the correct time. My Putheee ergganess, is vway vary.
4 grammaii- diverity:: simectural stimbrlicafion 3 originality: 5 sencifiviy: 5 ower- all score: 4 domain: technology
8
THE ADVENTURE OP LIMA AND HER ADVENTURCUS DOG SHERO Lina was a nornel girl like any girl.She lived of the hilla and ahe looked behind a special bush undes the cwarwal of pine trees.ate sav nany pines behind but whan she neved the pines she Found large piace of weilen. Lene,,,Lina, Line said his mother GeT de! late for school! t0l nonilfa too cired.Come on you have no, arquementa, a Lina was fron rich fami family. She lived in Los Anji lous
aceancy: 2 coherence: 3 lan-
hills, the
top
guage crmsisency semantic density: kanutedge_novset
topic focus:
4 creatiitty 3 professionlism: I style consisency:
in which sinething was
3 gammati- cal diversity stmenura stmaural staddrdizzoon 2 originalit: 4 sensitvity: over- all seore: 2 domain: other
9
"Bunshine Quis Mkly of Win a top Bony DVD player it u know which countzy the Algarve is in? ThE ansz to 82277. 50 SP: Tyrene Cust service arnorcement. for you. Pleasu call 07046744435 new to arsange dulivery You are vinner I have been spacially selacted 2 seceive AE1000 cash or a 4 holiday (flichte incl speak to live operator 2 clain 0871271810810 URGANT! Be you have von AE3CO prize ...
accumey. 2 coherence: 3 lan- guage_ consistenyy: 2 semantic density:
Now Years delivery vasting
topic fopucfoocus creatiity: 2 protessioaalsm: 2 style pousistnnyy: cal diversity; 2 stmuuural standrrdizatoon
3 knoeledgt_noviiie
2 grammati-
contact you. Last weakends
shovs that
2 originality: 3 sensitaity: 5 score 2 domain: retail e-commerce
CRID"toowEONERRPRh Htw. www.aaccccooceeeocccccc. BaMOGMe wutpaatyyaatooooc..ooo..a. GRAD i www.aatttoe
sccuacy: coherence:
lan- gunge comiisencyy I semmic demiity: knowlodge I topic focus:
wttp://oowoettteeoeeoeeeooo. wtwpeeeeweeoooooeeeeeeeeeeeeeeee Sedeg SiM.tttt ht www.w/cccewccccccccccc.cccccecee Too www.eeeawoooooooouu2..o......
creuivity: 1 professumaism:: I syte cousistency:
...
gramati- cal diversityyy scrutural staddrdiizio originaity: I sensitivity: 5 soore domain: other
21

---

Published as a conference paper at ICLR 2025
B DATAMAN MODEL
B. FINE-TUNING DATASET
To create the fine-tuning dataset of DataMan, we collected documents from both in-souree and out-of- source within the large pre-training corpus SlimPajama (Soboleva et al. 2023 For each document, we used the full prompt to instruct the Super LLM to generate scalar scores (I ~ [] 5] across 14 quality criteria, along with [A O letter grade indicate its domain type. The fine-tuning dataset has 35.7k documents. Each document was limited to 2,048 tokens, averaging 810 tokens. It outperforms the [256, 512] token range of Wettig et al. (2024a) when handling documents with broader length variations. Table 6 lists the number and proportion of documents categorized by domain, Overall Score, and source in the fine-tuning dataset.
Domain analysis. Domain Other accounts for nearly 25% indicating that the fine-tuning dataset encompasses domains outside the existing 15 domains, providing DataMan with rich domain-specifie prior knowledge. Domains that account for between 15% and 3% involve texts related to web crawling (such entertainment and culture) well typical vertical domains (like medicine and coding), enabling DataMan to better address both general and specialized knowledge. Finally, the collection of data with high barriers entry, such as mathematics and long- tail telecom data, remains a challenge for data management.
Overall Score analysis. Considering the imbalance in the collected documents between high and low scores, we performed up-sampling on low-scoring documents (< 3) to avoid biases in the quality ratings for DataMan. In practice, we divided the sources into five equal parts based the difference between high- and low-scoring documents and performed a fourfold up-sampling on low-scoring documents, ultimately reaching total dataset size of 425,794.
Source analysis. While ensuring adequate data within the SlimPajama domain, we also introduced 19% of out-of-domain data (Other) to enhance DataMan's source generalization capability.
Table 6: The number and proportion of documents categorized by domain, Overall Score, and source in the fine-tuning dataset.
Domains
# Documents
Proportion
Overall Score
# Documents
Proportion
Other
84,373
24.83%
5.0
100,242
29.50%
Technology
45,094
13.27%
161,225
47.45%
Entertainment
40,696
11.98%
3.0
51,571
15.18%
Culture
31,595
9.30%
2.0
22,423
6.60%
Government
24,075
7.09%
1.0
4,293
1.26%
Medicine
21,146
6.22%
Sources
# Documents
Proportion
Coding
19,861
5.85%
CommonCrawi
228,000
63.8%
Retail E-commerce
16,880
4.97%
C4
8,000
2.24%
Law
15,989
4.71%
Wikipedia (English)
10,227
2.87%
Education
13,629
4.01%
Book
12,000
3.36%
Finance
8,915
2.62%
StackFxchange
10,348
2.90%
Transportation
6,891
2.03%
Github
10.386
2.91%
Mathematics
4,875
1.43%
ArXiv
10.152
2.85%
Agriculture
4,627
1.36%
Other
67, 67,865
19.05%
Telecommunication
1,132
0.33%
Overall
356,978
100%
We present the average score for quality criteria across each domain in Table 7 with analyses below:
Knowledge Novelty excels in mathematics and medicine, closely linked to cutting-edge scientific research, enhancing the model's scientific abilities.
Creativity is highest in culture and lowest in legal domains, reflecting the openness of literary works versus the stability of legal texts, thereby improving the model's literary skills.
High professionalism indicates data from specialized fields likemathematics, law, medicine, and finance, enhancing the model's performance in these areas.
22

---

Published as a conference paper at ICLR 2025
Table 7: The average score for quality criteria across each domain in the fine-tuning dataset.
Professionatism
Standar Standardization dizatioon
Overall Score
Accuracy
Coherencs
Language
Consistency
Semantic
Density
Knowledge
Novelty
Topic Focus
Creativity
Consistency
Grammatica
Diversity
Structoral
Originality
Sensitivity
Style
Domains
Mathematics
4.66
4.59
4.82
3.94
4.86
4.84
5.00
4.71
Law
4.68
4.62
4.85
4.53
2.79
4.74
1.94
4.59
4.63
4.15
4.38
4.59
4.86
4.40
Medicine
4,50
4.50
4.76
4.40
3.36
4.65
2.73
4.39
4,47
4,21
4.14
4.47
4.87
4.33
Coding
4.31
4.33
4.83
4.62
2.63
4.88
1.98
4.52
4.68
4,99
4.21
Culture
4.49
4.43
4.68
4.18
3.13
4.37
3.64
3.69
4.38
4,21
3.74
4,50
4.82
4,20
Agriculture
4.41
4.78
4.37
3.14
4.61
2.83
4.42
3,92
4.52
4.96
4.19
Education
4.43
4.39
4.74
4.15
2.90
4.52
2.89
3.97
4.36
3.94
3.85
4.39
4.93
4.07
Government
4.48
4.33
4.78
4.11
2.86
4.44
2.49
3.99
4.34
3.98
3.80
4,36
4.68
4.01
Technology
4.26
4.16
4.64
4.10
3.17
4.46
2.69
4.07
4.16
3.77
3.67
4.33
4.93
3,99
Transportation
4.34
4.22
4.70
4.13
2.73
4.56
2.57
3.83
4,21
3.66
4.34
4.95
3,91
Telecommunication
4.29
4.66
4.05
2.44
4.22
3.90
Entertainment
4.16
4.13
4.46
3.87
2,68
4.28
3.56
3.22
4.07
3.80
3.37
4,26
4.62
3.82
Other
4.11
4.02
4.47
3.86
2.60
4.09
3.08
3.32
4.01
3.34
4.17
4.65
3,71
Retail E-commerce
4.20
4.02
4.59
3.91
2.38
4.43
2.86
3.46
4.02
3.52
3.41
4.14
4.95
3.70
Coding exhibits the least grammatical diversity and high structural standardization due to its fixed grammatical formats. In contrast, low values in retail e-commerce for these criteria suggest that they lack correlation.
Data from specialized domains showcases strong originality and semantic density, with low content redundancy and meaningful content, improving the model's performance in vertical fields.
The government and entertainment domains exhibit lower sensitivity, likely related to free speech on social media and politically sensitive topics, aiding the model in filtering harmful speech and sensitive content.
Other criteria perform well across all domains, ensuring basic requirements are met and enhancing the model's s general capabilities.
In general, specialized domains tend achieve higher Overall Score, while long-tail and general domains are relatively lower.
Figure 5 shows the Pearson correlation heatmap among various quality criteria. All criteria are positively correlated, with Pearson correlation coefficients generally below 0.8, except for Style Consistency and Structural Standardization, which we speculate may adapt basic requirements with other criteria. Notably, since the Overall Score is derived from the remaining 13 quality criteria, it highly correlates to each individual criterion.
23

---

Published as a conference paper at ICLR 2025
Accuracy
1.00
0.80
0.76
0.66
0.50
0.59
0.22
0.69
0.83
0.74
0.74
0.62
0.51
0.81
Coherence
0.80
0.67
0.74
0.31
0.73
0.81
0.59
Language Consistency
Semantic Densky
0.06
0.81
0.59
1.00
0.54
0.76
0.13
0.72
0.76
0.59
0.00
0.64
0.53
0.84
Krowledge Novelty
0.50
0.55
0.41
0.54
1.00
0.39
0.33
0..4
00.48
0.66
0.49
0.39
0.36
0.67
Topic Focus
0.59
0.74
0.59
0.76
0.39
1.00
0.09
00.00
0.75
0.46
0.73
0.48
0.49
0.72
Creativity
Professionalism
Style Consistency 0.83
0.87
0.78
0.76
0.48
0.75
0.22
0.69
3.00
0.67
0.84
0.61
0.51
0.84
Grammatical Diversity 0.74
0.73
0.60
0.59
0.67
0.46
0.49
0.62
10.67
3.00
0.60
0.50
0.38
0.78
Structural Standardization
0.74
0.81
0.62
0.49
0.73
0.05
0.76
0.80
3.00
0.59
Originality
Sensitivit
0.51
0.52
0.55
0.53
0.36
0.49
0.07
0.53
0.51
0.38
0.44
0.41
0.53
Overall Score
0.81
0.86
0.71
0.84
0.67
0.72
0.30
0.77
0.84
0.78
0.
0.62
0.53
1000
Accunccy
Languege Consestancy
Caherncc
Semantic Deristy
kroowledgee Novelty
Topic Focus
Creativity
Professionalton
Style Conisteacy
Grammaticnl Diversity
Struciunt Sandardization
Oriirality
Sensitbity
Overal Score
Figure 5: Pearson correlation heatmap between 14 quality criteria in the fine-tuning dataset.
B.2 DATAMAN TRAINING
We fine-tune the DataMan model using Qwen2-1.5B (Yang et al. 2024a), ad advanced open-source 1.5B parameter language model, based on text generation loss. To meet diverse application needs, we offer three DataMan model versions, named according their chat prompts and applicable scenarios:
Chat Prompts
Score-only: Please give an overall score for the text: Text: [text] Overall Score://5
Domain-only: Please specify an domain type for the text: Text: (text) Domain:_
All-rating: Please score the text on fourteen evaluation criteria and specify its domain: Text: (text) Domain: [1]Accuracy:_] /5 [2]Coherence:_/5 [3]Language Consistency: /5 [4]Semantic Density: /5 [5]onoee Novelty: /5 To]io Focus: /5 [7]Creativity: [8]Professionalism:: /5 [9]Style Con- sistency: /5 [10]Grammatical Diversity: /5 [11]Structural Standardization: /5 [12]Originality: /5 [13]Sensitivityy_5) [14]Overall Score:_/5
Score-only DataMan rates the Overall Score of text (1 token), ideal for large-scale dataset filtering.
Domain-only DataMan identifies text domain (1 token), ideal for large-scale data mixing.
All-rating DataMan rates the quality criteria of text and identifies the domain (15 tokens), ideal for refined data selection and mixing.
Hyperparameter search. Next, we conduct hyperparameter search using a validation set of 8.6k documents. The search grid included: seed e442, 1024, 3400y..earning learning rate e(1 x 10-6 7 10-6 1 10 10-5.2 2.2x x 10 5 5 x 10 5]. number epochs c(2, 3, 4, 5). batch size e(256, 512, 1024), data size c{82k, 164k, 246k, 312k. 357k), up-sampling fold e(1, 2, 3, 4, 5). model size c(0.5B, 1.5.), and inference temperature c(0.0 (grredy decoding). 0.1, 0.3, 0.5, 0.8, 1.0). Model selection was based which model achieves the best accuracy the criterion of Overall Score. The selected model
24

---

Published as a conference paper at ICLR 2025
Table 8: The test accuracy of the three DataMan model versions. Here, X indicates not applicable.
Domain Arg. Acc
Domain Aceurary
Quality Avg. Aet
Qualitt Accuracy
Medicine
Finance
Law
Accuracy
Coherence
Language Cuncisency
99.4
78.8
Education
Technology
Entertalnment
Semantic Deneity
Knowledge Novelty
Topic Focms
86.8
89.0
66.2
67.2
78.4
78.6
uxing
M044
90U6
Government
$1.0
79.5
Creatiiity
76.5
Professlonatism
Style Condstency
76.4
Culture
Transportation Retail commerce
Grammatical Diversity Siructural Standardization Originality
$4.9
73.9
92.3
Telecommmmkation
Agricutture
Other
Seusitivity
Overall Soore
$3.4
75.6
Medicine
Finance
Law
91.7
79.9
91.5
Education
Technology
Entertainment
90.
BR.
66.7
Damannally 65.9
Mathematicss
Coding
Government
Culture
Transportation Retail E-oommeree
Telecommmicattonn Agricutture
Other
82.6
65.9
Scere-only
77.3
Table 9: The detailed analysis of testing accuracy of the Overall Score criterion.
Accuracy
Error Rate
Overall Score
5-level
<3,
<3
<2 but
22,
>3
22, <3 but 24
23, but <3
81.3
97.5
98.5
81.6
0.2
2.7
15.5
0.2
1.4
parameters were: seed 1024, learning rate 10 trained for epochs, batch size 512, data size 357k, 4-fold up-sampling ratio, 1.5B model size, and utilizing greedy decoding for inference.
Inference accuracy. Subsequently, we evaluated the accuracy of three DataMan model versions test set comprising 8.6k documents, as shown in Table Leveraging gold-labeled fine-tuning data from Super LLM, all model versions exhibited excellent performance. All-rating DataMan achieved nearly 80% accuracy across quality criteria, with grummatical diversity and structural standardization being the most challenging criteria to predict. We found performance limitations in quality rating using the Overall Score criterion, which showed five-class accuracy of 81.3% and a binary accuracy of 97.5%. The accuracy for high-quality documents reached 98.5%, but for low-quality documents, it was only 81.6%, due to insufficient samples. We aim to collect more low-quality documents to improve DataMan's 's accuracy in this area.
Misclassification analysis. In Table D we analyze the test accuracy of the Overall Score criterion to verify that DataMan rarely makes unreasonable decisions, making unlikely to cause "snowball effect". In addition to the 5-level classification accuracy (5-level Acc) used in the paper, we classify samples with score of or above as positive samples, and vice versa as negative samples, thus obtaining 2-level classification accuracy: (<3, 23 Acc). We detail the accuracy for positive samples (23 Acc) and negative samples (<3 Acc), as well as error rates for specific misclassification cases: extreme false negative samples (<2 but >3 Error Rate), moderate false negative samples (22, <3 but >3 Error Rate), marginal false negative samples (22, <3 but =3 Error Rate), extreme false positive samples (24 but <3 Error Rate), and marginal false positive samples (23, <4 but <3 Error Rate). Results show the error rate for DataMan in the two extreme misclassification cases of the Overall Score is very low, at just 0.2%. This indicates that DataMan rarely mistakes poor-quality documents for high-quality ones, and vice versa. Considering the strong fault tolerance of pre-training, the snowball effect will not become a bottleneck.
Inference efficiency. Finally, Table 10 presents the inference FLOPs and memory usage for three DataMan model versions evaluated single A800 GPU using vLLM (Kwon et al.] (023). To reduce DataMan annotation costs, we recommend cost-effective models like Score-only DataMan or fine-tuned Qwen2-0.5B and suggest changing the training objective from text generation multi-task
25

---

Published as a conference paper at ICLR 2025
Table 10: The inference FLOPs and memory usage of three DataMan model versions.
Input Speed (Toks/S)
Output Speed (Toks/S)
Processing Speed (Docs/S)
Memory (G)
All-rating
31822
868
30
72.9
Score-Only 63644
1736
60
72.9
Domain-Onie 63644
1736
60
72.9
classification. However, to avoid parameter transfer issues due to the different learning paradigms between pre-training and fine-tuning (Wang et al. 2019), we continue using text generation loss. Furthermore, heuristic pre- pre-processing.. such as deduplication using Fuzzydedup (Jiang et al.] [022) and Semdedup (Abbas et al. (023), or rule-based and model- based selection methods like C4 filter (Raffel
al. 2020). Gopher rules (Rae et al.) (2022), and binary grammar discriminators (Chowdhery et al. can pre-reduce data annotation needs.
2023
C EXPERIMENTAL DETAILS
DataPajama statistics. Table 11 shows the domain, Overall Score, and source statistics of the 447B DataPajama corpus, from which we select 30B tokens using different data selection methods. Firstly, from a domain perspective, the proportion the mathematics domain in DataPajama has significantly increased compared to the fine-tuning dataset, while the coding domain has seen a slight rise. In contrast, the proportions of all long-tail domains (such as Transportation, Agriculture, Retail E-commerce, and Telecommunications) have fallen to their lowest levels. Secondly, regarding overall scores, DataPajama, as a subset of Slimpajama, has undergone extensive cleaning and deduplication, resulting in high proportion of samples rated 5 and 4. Conversely, low-quality texts (rated below 3) account for only 7. 7.86%. We chose to retain these low-quality texts to allow researchers for in-depth analysis. DataPajama is a curated subset of SlimPajama, which is itself subset of RedPajama. Both SlimPajama and RedPajama are released on HluggingFace under the Apache 2.0 License.
Training details. Using different data selection methods, we select the 30B token subset from DataPajama and train language model from scratch for one epoch in a randomly shuffled order. We employ Sheared-Llama-I.30 transformer architecture with RoPE embedding (Su al. 2024) and SwiGLU activations (Shazeer) 2020). This model is trained using a global batch size of 2048 sequences and a learning rate of 10 with a cosine learning rate decay warmup for the first 5% of training steps. We use weight decay of 0.1 and train with Adam (Kingma 2014) with hyperparameters 3 (0.9, 0.95). Finally, we save a checkpoint every 1,000 steps and merge the last three using mergekit (Goddard et al.) [024] as the desired LLM, eliminating biases from step fluctuations. Each model is trained on 32x NVIDIA A800 over 228 GPU hours.
10 5 and a linear
In-context learning settings. We choose a different number of few-shot examples per task to ensure that all demonstrations fit within the context window of 1024 tokens. We use the following number of demonstrations (given in parentheses): ARC-easy (15), ARC-challenge (15), SciQA (2), LogiQA (2), BoolQ (0), HellaSwag wag (6), PIQA (6), WinoGrande (15), NQ (10), MMLU (10). We report accuracy for all tasks, except for NQ, where we report EM. When available, we use the normalized accuracy metric provided by 1m evaluation iarratiioonaannssssse harness.
Full results of perplexity and ICL. In Tables 12 and 13 we report the full validation and test perplexity results for all models, including those for each RedPajama source. Table 14 contains the ICL performance of all models across 10 downstream tasks. The lowest perplexity reveals how quality criteria enhance L.LM performance in specific data sources: i)-Sensitivity excelled in web domains (CommonCrawl, C4), emphasizing the importance of avoiding sensitive topics to improve web content adaptability, ii)-Semantic Density, Originality, and Topic Focus showed superior performance in Wikipedia, suggesting the benefit of informative, original content for world knowledge absorption. iii)- C-eatiiiiiy scored lowest in book sources, indicating its role in enhancing the understanding of literature. In ICL tasks, the criteria's influence is as follows: i)-High Semantic Density improved performance in elementary science tasks (ARC-E, ARC-C) and, alongside high Professionalism. in more advanced questions (SciQ). High Creativity aided in summarization and subtitle tasks
26

---

Published as a conference paper at ICLR 2025
Table 11: The number and proportion of documents categorized by domain, Overall Score, and source in the 447B token pre-training corpus, DataPajama.
Domains
# Documents
Proportion
Overall Score
# Documents
Proportion
Other
100,395, 132
22.97%
5.0
169,558,487
38.80%
Culture
64,774.739
14.82%
4.0
098,088,1 168
45.33%
Technology
44,947,278
10.29%
3.0
36, 156,824
8.27%
Entertainment
43,543.874
9.96%
2.0
29,504,959
6.75%
Government
38,157.053
8.73%
1.0
3,681,879
0.84%
Coding
31,900,509
7.30%
Sources
# Documents
Proportion
Medicine
30,021,105
6.87%
CommonCrawl
263,494.321
60.30%
Mathematics
10,108,505
4.60%
C4
70,289,855
16.08%
Law
19,463,871
4.45%
Wikipedia (English)
13,282,740
3.04%
Education
44,6632298
3.36%
Book
27,674,520
6.33%
Finance
10,138,552
2.32%
StackExchange
8,518,050
1.95%
Transportation
6,430,573
1.47%
Github
10,386
2.91 %
Agriculture
5,739.330
1.31%
ArXiv
10,152
2.85%
Retail E-commerce
5,355.667
1.23%
Other
67,865
19.05%
Telecommunication 1,350,806
0.31%
Overall
436,990,312
100%
(HellaSw., W.G.), while complex reasoning tasks (PIQA) benefited from mix of high Semantic Density. High-High Originality was effective for Wikipedia-relater tasks, where redundant knowledge was counterproductive.
Results on larger 60B tokens. Despite the 30B token subset exceeding the compute-optimal ratio of data-to-model suggested by (Hoffmann et al.) 2022a), solidify method, used the strongest DataMan variant Overall Score 1=5, the existing SOTA baseline (education value 2.0), and uniform sampling select larger 60B subset to train the 1.3B language model from scratch. The model's PPL and ICL performance of 60B subset in Table 15 the full validation and test PPL across sources in Tables [6 17 and the full ICL performance of ten downstream tasks in Table 18 The comple-wwth--eee -attt--attaaan model significantly outperformed the SOTA baseline in ICL performance tasks and showed modest improvement in the validation and test PPL. This further confirms the effectiveness of the DataMan approach.
Misalignment between PPL and ICL. In Figure we plot the relationship between perplexity and ICL performance for all models across 10 downstream tasks, including the Pearson and Spearman correlation coefficients, to investigate the misalignment between PPL and ICL. The results indicate that the misalignment is most pronounced in the LogiQA and MMLU tasks. Deeper analysis identifies two main causes: i)-domain mismatch: pre-training often uses extensive general corpora, which enables the model to exhibit lower perplexity common text. However, tasks like MMLU, which span 57 distinct specialized domains (such as abstract algebra and anatomy), may suffer in ICL performance due to domain mismatch; ii)-CL task complexity: Many ICL tasks require complex reasoning rather than simple text generation, which perplexity assessment struggles to capture. This is particularly evident in LogiQA, where the task assesses human logical reasoning skills through expert-written questions from Civil Servants' Exams.
27

---

Published as a conference paper at ICLR 2025
Table 12: Validation per-token perplexity per RedPajama source across all our models. We highlight the best result in each column. Abbreviations: CC = CommonCrawl, Wiki Wikipedia, StackEx StaakExchange.
Selection Method
CC
C4
Github
Wiki
ArXiv
StackEx
Book
Overall
Uniform
11.09
13,93
3.04
10.41
5.69
6.15
12.42
10.7
+50% data
10.47 Uoez
13.12 Jon
2.88 10.6
9.43 JU3
5.42 uam
5.84 4031
11.70 Jam
10.09 l0s
with Wiki
13.02
1191
18.66
1473
3.62
1958
24.07
6.63
7.28 11:13
15.39 1399
13.34 12:1
DSIR
with Book
13.11 1200
18.16
1928
3,50
1936
38.97
13856
6.55 1ew
6.83 tore
13.18 10.6
13.60 12:9
Perplexity
lowest
16.20
21.51
17.58
4.41
18.26
7.12
9.10
20.26
15.98
highest
11.92
14.34
10.01
3.21
11.38
5.90
6,20
12.38
11.32
Writing
top-k 2.0
12.77 1168
18.87
1196
3.40 19.56
25.61
01530
5.82 1e/
7.03 88
12.48 10.00
13.01 02:3)
Style
10.94 18is
14.09
19.16
2.99 Jole
10.32
5.60 1ow
5.60 00:s
12.01 Un:41
10.60 2000
Facts
top-k
12.60 11.11
19.15
1528
3.52 19.8
64.82
1500
5.91 162m
7.23 1106
15.90 13
14.38 13:6
Trivia
== 2.0
10.98 40m1
14,25
10338
3.00 V00
10.65
5.56 40/3
6.11 60.00
12.32 40.00
10.68 4003
Educational
top-k == 2.0
13.26 T21
18.84
14M
3.45 1941
27.20
116.9
5.63 4ow
6.90
15.45 tim
13.54 12.91
Value
11.02 Toa
14.10
10.17
2.98 Vam
10.49
Toxs
5.53 48M
6.09 Jane
12.34 Jame
10.67 4003
Required Expertise
top-k
15.13
21.83
17.00
3.59
18.87
5.54
7.63
16.38
14,97
T 2.0
11.06
14.17 1938
2.98
10.25
5.54
6.10
12.29
10.7
Criteria mix
2.0
10.97 4803
14.10 t9.17
2.99 Va1
10.57 18
5,56 4833
6.10 unme
12.19 40:33
10.63 Line
Accuracy
top-k
10.73 4836
16.59
12.66
2.94 40.10
9.96 J0.5
5.28 40.4
6.15
11.64 49.78
10.82 10.13
Coherence
top-k
10.70 4839
16.36 1248
2.90 V0.14
9.32
5.27 4642
6.01 00.14
11.48 40.94
10.72 T002
Creativity
top-k
11.27 1018
16.41
12:88
3.19 19.15
9.70
5.38 4831
6,26 10.11
10.87 41.55
11.08
Grammatical
top-k
10.85 JRM
16.72 13.99
2.92 40.
9.84 201
5.25 4R/8
5.91 0031
11.27 4105
10.87 2003
Diversity
Knowledge Novelty
top-k
11.06 t000
16.42 1240
2.86 48.8
9.59 J032
5.18 4651
5.87 40:38
12.33 40.0
11.01 103
Language
top-k
10.34 1875
15.43 11.00
2.90 10.16
9,33 01.38
5.28 1641
5.84 00:3
11.36 11.06
10.35 0035
Consistency
Originality
top-k
10.73 1839
16.16 t223
2.84 10:00
8.98 -140
5.25 Lows
5.82 0033
11.29 41.33
10.68 2002
Professionalism top-k
11.23
17.00 tan
2.88
9.52
5.23
5,94 1831
13.26
11.27
Semantic
Density
top-k
11.22 TRx
16.61 1368
2.84 1820
8.96 $1,45
5.22 lewm
5,85 60300
12.13 1039
11.10 Tt499
Sensitivity
top-k
10.30
14.16 1923
2.83
9.01
5.29
5.76 40.39
11.42
10.11
Structural
Standardization
top-k
12.15 1106
17.95 Mim
2.91 4913
10.40 Joml
5.37 4833
6.07 4016
14.72 13300
12.11 000
Style
top-k
10.71 4834
16.39 13366
2.92 1n1
9.65 2036
5.28 Un:r
5.94 0031
11.42 41:0
10.74 2081
Consistency
Tupic
top-k
10.48 486i
15.39
1146
2.84
8.97 -1.44
5.27 4642
5.81 10.34
11.40 4000
10.41 4029
Focus
l=/
22.22 11133
24.93
15.58
11158
69.54
119133
21.71
19.95
113300
25.41
f13.00
23.83
19315
Overall
12.80
1121
15.01
1U0
5.40 f2.56
18.89 11.8
10.73
13564
8.57
1242
14.97 1258
12.84 12114
Score
11.61
1832
15.83
1130
4.05 1UM
14.25
1334
8.78
1329
6.54
10.39
13.20 to.s
11.75 1145
10.16
46a
14.93
1339
fa11
9.02
1133
6.05
1636
5.67 Ja.4
11.36 11:6
10.21 (04)
10.56
Iass
16.36
t248
2.66 18.38
8.51 -1.9
4.74 Vaas
5.92
40:3
10.79
11.s
10.52
28

---

Published as a conference paper at ICLR 2025
Table 13: Test per-token perplexity per RedPajama source across all our models. We highlight the best result in each column. Abbreviations: CC CommonCrawl, Wiki Wikipedia, StackEx StackExchange.
Selection Method
CC
C4
Github
Wiki
ArXiv
StackEx
Book
Overall
Uniform
11.1
13.82
2.97
10.29
5.26
5.75
13.05
10.75
+50% data
10.47 loss
13.03 1an
2.82 joir
9.33 JuM
5.47 tea
4.94 Jan
12.29 tam
10.14 lMs
with Wiki
12.97 tup
18:50
11/8
3,50
1955
23.91
01363
6.71 fi/s
6.36 t0.1
16.11 tim
13.37 1243
DSIR
with Book
13.08 fis
17.96 1:
3.41 1934
39.03
13831
6.62 11:
5.93 10.18
13.38 1033
13.59 03:11
Perplexity
lowest
16.15
21.34
17.53
4.21
18.14
7.21
7.43 11.88
21.51
16.04
highest
11.91
14.27
19.15
3.14
11.24
5.96
5.34
12.74
11.34
Writing
top-k 2.0
10.95 4815
18.65
1483
3.33
19:56
28.18
714
5.85
10:9
6.10 10:35
12.81 00.44
12,97 00:35
Style
10.95 L8is
13,96
19.14
2.93 18M8
10.20
5.60
1638
5.22 01:
12.62 Un:43
10.64 1003
Facts
top-k == 2.0
12.50 11/00
18.89
1500
3.40 19455
64.81
18853
5.95
6.30 1955
15.91 1786
14.33 03.54
Trivia
10.98 4013
14.11
19.79
2.93 5900
10.52
192
5.60
1933
5,20 40.55
12.99 00.00
10.72 40:3
Educational
top-k == 2.0
13.18
1206
18.61
1479
3.29 1952
26.33
1M604
5.69
tour
5.92 1M37
15.86 1281
13.49 13:44
Value
11.03
1o0
13.97
10.15
2.91 Uam
10.36
Tour
5.58 1632
5.17 40.5
12.97 Jam.
10.72 1003
Required Expertise
top-k
15.04
21.58
17.76
3.46
18.37
5.59
6,54
16.70
14.92
T 2.0
11.07
14.04
2.91
10.12
5.59
5.17
12.91
10.74
Criteria mix
2.0
10.97 4803
13.97 15:15
2.92 V818
10.44 teis
5.61 1e3
5.26 40.09
12.82 40:3
10.68 Unee
Accuracy
top-k
10.68 40:22
16.42
12.00
2.82 40.15
9.84 J0.5
5.33 100
5.20 40.55
12,10 49.95
10.80 10:5
Coherence
top-k
10.66 4044
16.14
12388
2.81 40.16
9.20
5.32 Tom6
5.08 40.57
11.97 41.6
10.71 4004
Creativity
top-k
11.12 1o07
16.23
12411
3.09 10.10
9.61
5.42 1036
5,42 40:33
11.39
11.00 00335
Grammatical
Diversity
top-k
10.81 JRM
16.53 1371
2.82 19.55
9.72 JMn
5.30 700
4,97 0078
11.73 41.3
10.86 700
Knowledge Novelty
top-k
11.01 to0l
16.23 124)
2.81 4816
9.46 J03
5.22 400
4,93 40:5
13:00 4016
11.01 0022
Language
top-k
10.31 1079
15.27 1EAS
2.78 10.10
9.21 C1.36
5.33 160
4.87 00.58
11.91 11:4
10.35 0043
Consistency
Originality
top-k
10.69 Ioat
15.96
1216
2.77
10200
8.87 1143
5.30 Tou
4.86 Uta9
11.91
41.34
10.67 2005
Professionalism top-k
11.18
16.81
13
2.79
9.40
5.28 teur
4,90 loxs
13.81
11.26
Semantic
Density
top-k
11.16 Thos
16.41 13.9
2.78 18.99
8.84 $1,45
5.27 frm
4.89 l036
12.83 103
11.09 TM434
Sensitivity
top-k
10.28 1ear
14.04 1029
2.77
8.90
5.34
4,81
11.98
10.13
Structural
Standardization
top-k
12.08 ThM
17.76 1398
2.81 49.6
10.25 4001
5.43 1007
5.12 00:6
15.49 17:1
12.11 0135
Style
top-k
10.67 48/1
16.20 13338
2.80 4117
9.52 2033
5.32 1008
4.98 40.77
11.95 41 I0
10.73 J08
Consistency
Tupic
Focus
top-k
10.44 486
15.22
1140
2.77
8.85
5.31 1eas
4.83 40.92
11.96 41.00
10.41 4034
l=/
22.22 11132
24.92
110.3
15.03
11316
69.85
22.46
13123
19.22
26.26
23.95
11320
Overall
12.83 11.23
14.92
11:0
5.22
12.55
18.74
15:33
11.00
1534
7.81
1200
15.74
12:0
12.91 f216
Score
11.59
184)
15.66
t134
13.44
14.15
13.86
8.97 1321
5.71 J0.44
13.97
1052
11.78
1183
10.13
14.78
taxe
3.06
8.91
-1338
6.13
1687
4.75
41.0
11.93
11:22
10.22 (03)
10.51 1839
16.18
1236
2.56
18.11
8.40
-1.9
4.77 (e,a)
4.99
44.6
11.28
11.77
10.50
4025
29

---

Published as a conference paper at ICLR 2025
Table 14: The in-context learning performance for ten downstream tasks across all models. We report accuracy for all tasks, except for NQ, where we report EM, and highlight the best result in each column (before rounding), Abbreviations: HellaSw. = HellaSwag, W.G. = WinoGrande.
Reading Comprebension
Commonsense
World
Selection Method
Reasoning
Knowledge
ARC-E (15) ARC-C (15) SciOA LogiQA (2) BoolQ (0) HellaSw. (6) PIQA W.G. (15) NO (10) MMLU (5) Average
Uniform
24.1
57.5
44
44,9
+59% data
60.6
24.3
90.3
24,4
60.1
47.7
69.0
54.4
5.8
26. 1
46.8
DSIR
52.8 442
26.3 613
85.9 41
25.2 11
60.3 114
35.8 222
61.4 472
52.2 463
4.7 136
24.7 4/
4na
with Book
49.5
25.3
83.6
23.5
57.9
44.8
69.4
55.6
3.1
25.2
43.8
P Perplexity
lowes highest
49.2 483
25.1 223
83.7 540
22.0 123
61.4 113
34.6 194
65.0 558
49.1 434
2.7 214
24.7 010
41.7 033
53.5 149
25.6
84.6
105
124
65.6
53.4
2.9
414
Writing Style
top-k
$2.7 143
27.3
79.7 130
26.4 123
60.5 113
41.1 124
66.1 123
52.3 102
2.5 116
24.4 113
43.4 015
56.4 911
28.4
85.8
Tox
118
44.9 103
68.6
4.5 734
TE1
Facts &
App-2
65.6
35.1
87.9
24.1
60.9 11
30.4 145
62.5
53.1
5.7
25.
45.8
Trivio
2.0
59.3 113
24.8 123
88.1 TM
25,0 105
61.4 133
43.9 100
68.3 003
54.6 123
1.1 183
26.9 11.
46.2 113
Educational
kop-k
66.6
34.6
89,6
24,6
58.3 11
45.5
66.4
52.9
3.8
25.0
46.7
Value
2.0
60.7 t32
30.4 F3
88.8 113
26.6 193
60.1 135
45.4 11a
69.1 10
54.2 113
4.3 192
27.1 154
46.7 11
Roquired
60.4 12>
30,9
86.8
25.0 705
114
36.1 479
57.8 4M
2.4 407
26.3 13.
419
Expertise
2.0
59.6
29.8
89.0
25.8
61.4
43.2
67.4
56.0
4.6
25.4
46.0
Criteria
2.0
59.2 m12
30.2 136
88.0 193
24.3 162
58.7 112
44.5 105
68.7 103
53.5 113
5.3 112
25.1 188
45.7 TE3
Accuracy
kpp-
62.8
29.4
90.3
24.7
61.7
49.8
69.3
55.6
7.0
26.2
47.7
Coherence
kxp-
63.6 to
32,5 t38
90,3 F76
26.7 TT
61.6 1X)
50.8 TH
70.6 TH
55.1 17
7.0 1799
25.2 4K3
48.3 fe
Creativty
60.8
30.6
87.8
102
123
11>
71.2
58.6
4.7
25.6
Divensity
Grammarical
top- k
61.6
33.4 89,3
26.3 133
62.0 113
50.6 tr9
69.8 113
56.1
7.7 136
25.2
48.5 T1S
Knowledge Novelty
63.5 to
32.8 T5Y
90.5
62.1 14x
47.2 112
67.9 501
55.6 TXT)
6.2 489
24.8 489
ft
Consistency
Language
k
63.0
31.0
89.7
25.3
61.4
50.2
70.1 57.6
7.6
25.8
48.2
Originality
top-k
64.0 165
31.7 181
90.7 130
25.3 112
57.7 102
49.0 153
70.5 118
56.2 133
8.0 7AP
24.7 11)
47.8 110
Professiocaliom k
61.4
32.2
91.1
24.0
61.2
45.0
66.1 53.3
6.6
25.2
16.9
Domity
Semantic
top-k
66. 182
31,9 16
91.4 FAT
25.2 THX
$7.2 103
48.4 148
71.2 T3A
54.7 122
7.5 TM
25.9 182
48.0 10
Sensitivity
63.2
32.4
91.1
70.6
6.8 LRV
25.3
mi
Strucurra
Standardization k
62.1
31.9
89.8
25.3 11
59.3 11X
45.9 T1X
70.6 THR
54.5
7.1 130
27.0 11
47.4 1E
Style
Consistency
63.0 th
32,0 T54
90.3 126
61.6 14)
70.7 TX1
57.8
7.2 T7X
48%
Topic
Focus
kpp-
61.6
30.8
91.2
27.3
61.9 tr.
50.0
69.0
56.3
7.1
24.0
47.9
I-]
42.0 :15
23.0 666
69.4 11K3
25.7 TE
55.2 133
31.0 1158
61.3 113
50.3 122
0.8 033
25.4 182
38.4 IES
Overall
1=2
53.7
26.0
83.8
26.4
61.8
38.2
63.9
50.5
4.3
25.0
43.4
Score
1=]]
54.3 amz
26.2 514
87.1 5M6
24.1
62.0 tu>
42.0 673
68.3 9m
52.0 8m3
4.7 tm
25.7
44.6 989
60.7
31.3
90.6
24.1
113
163
71.0
57.9
7.7 136
24.2
1=5
66.1
31,0
90.7
26.1
59.2 tix
$1.5 t6S
70.7
58.3
7.8
26.9
49.1
Table 15: The wrip-l--wtt-Dattaaaa model (Overall Score 1=5) improve perplexity and in-context learning (ICL) results on larger 60B tokens. We report the validation, test perplexity, and ICL performance of 10 downstream tasks. We highlight the best result in each column and improvement over uniform sampling with the 60B token budget.
Reading Comprehension (5 tasks)
Commonsense Reasoning (3 tosks)
World Knowledge (2 tasks)
Val Perplexity
Test Perplexity
Selection Method
Average (10 tasks)
Uniform
10.81
10.79
53.7 54.2 T05
58.4
16.4
47.6
Educational Value
2.0
9.81 11:00
9.85 Ja34
58.7 tus
16.0 704
47,9 10.
Overall Score
1=5
9.93 Jus
9.91
J038
56.5 128
62.9 145
17.5 mM
50.6 TM
30

---

Published as a conference paper at ICLR 2025
Table 16: Validation per-token perplexity per RedPajama source across three models trained on
60B tokens. We highlight the best result in each column. Abbreviations: CC = CommonCrawl, Wiki WWikipedia, StackEx StackExchange.
Selection Method
CC
C4
Github
Wiki
ArXiv
StackEx
Book
Overall
Uniform
10.81
16.50
2.89
9.54
5.22
5.88
11.43
10.81
Educational Value
r =
2.0
10.15
12.94
2.78
4003
9.11
5.20
5.68
11.36
9.81
Overall Score 5=5
9.94 lax
15.58 4us
2.52
7.71
4.53 1aw
5.62
10.13 nJ
9.93 Jus
Table 17: Test per-token perplexity per RedPajama source across three models trained on 60B tokens. We highlight the best result in each column. Abbreviations: CC = CommonCrawl, Wiki Wikipedia, StackEx StackExchange.
Selection Method
CC
Github
Wiki
ArXiv
StackEx
Book
Overall
Uniform
10.76
16.31
2.78
9.41
5.27
4.93
11.92
10.79
Educational Value
T
2.0
10.16
12.83
La48
2.73 Uuee
8.99
5.25
lazz
4.76
11.92
9.85
Overall Score
1=5
9.90
15.40
2.43
7.61
4.56
4.68
10.61
9.91
Table 18: The in-context learning performance for ten downstream tasks across three models trained on 60B tokens. We report accuracy for all tasks, except for NQ. where we report EM, and highlight the best result in each column (before rounding). Abbreviations: HellaSw. = HellaSwag, W.G. = WinoGrande.
Rending Comprehension
Commonsensee
World Knowledge
Selection ARC-E (15) ARC-C(15) SciQA (2) LogiQA (2) BoolQ (0) Hellašw. (6) PIQA (6) W.G. (15) NQ (10) MMLU Average
Reaconing
Uniform
63.0
32.4
89.1
22,3
61.6
19.1
70.7
55.0
6.4
26.3
47.6
Educational Value
64.5 113
31.1 133
90.9 115
24.1 113
60.6 119
50.2 10a
69.9 TM
56.0 T10
7.0 1E6
25.0 113
47.9 133
Overall Score
1=5
68.6
36,8
91.9
24.4
60.6
56.1
72.4
60.1
8,6
26.4
50,6
ARC-e
ARC-C
SclQ
LogiQA
BoolQ
65
2-0 0.29
0.665
0.52
-027
90
F. 0.885
00.0.60
28
p: 0.118
0.1014
0.442
00.09
85
26
55-
80
25
50
75
23
56
70
22-
HellaSwagg
PIQA
WinoGrande
ri -0.0
-0.055
P. -0.59
r. 72-0.69
58
n--064 P0-0.68
-0.67
-0.58
27.0
0.005
00.08
68
95
26.0
25.5
64
25.0
62
35
24.5
58
50
24.0
10
20
10
20
20
Perplesity
Perplecity
Perplecity
Perplesity
Perplesity
Wiltom
wikting
Rejuired
Grammatical diyy
Semariic
Scoe
DSm Wi::
Facts top
Criara
Knowledge hovelty
Senutivity top
Oveal Score 1n2
n5R Book
rhmm & Pish t-2
tiitom earl dets
Language Convinency top
Smmmraa Shreaet topi
Guell Stowe i-]
Purplec lowaat
Edu. talus sopic
Accunco poy cop-
orla nall by
Style Consalaney tapis
Overl Score tnd
Pertilecty 7 givert
Edi. valon 2
Coherone top-
Protecrtons dio topik
Teo Focue mop-
Ouell Sooe
Wiingg Styte tap-c
Rezssinad bxp. lopk
Cruatiik topk
Figure 6: We plot the relationship between perplexity and in-context learning (ICL) performance for all models across 10 downstream tasks, as shown in Tables and along with the Pearson and Spearman correlation coefficients. The misalignment between perplexity and ICL is most pronounced in the LogiQA and MMLU tasks, guess, possibly due to domain mismatch and ICL task complexity.
31

---

Published as a conference paper at ICLR 2025
D INSPECTING RAW DOCUMENTS AND RATINGS
Finally, we present snippets from the raw documents of Wikipedia, Books, Stack Exchange, Github, ArXiv, CommonCrawl, and C4 subsets of DataPajama. These documents correspond samples with quality ratings of 1, 2, 3, 4, and 5 across 14 quality criteria, as shown in Figure Notably, although these samples represent just small random snippet, they display significant quality differences. We believe it is essential to provide an unfiltered view of the training data; therefore, we have not applied any filtering to these documents. A small number of documents contain % potentially sensitive content.
From the visual examples, we found notable distinction between the data rated and and those rated 3, 4, and 5. For instance, the score of corresponds to an example like "...83 510 s 311 548 m 305 546 301 540 299 530 299 ...", whereas the score reflects " Systemm recognizes hierarchy of events from the measurements, not exactly in the sense of physical reality..." However, the discrepancy between scores 4 and 5 is not as pronounced, as seen in the example "...have been augmented with terms that quantify the user satisfaction the ad relevance..." which corresponds to a score of 4. This further supports our rationale for choosing pointwise evaluation over pairwise, as humans also find it challenging determine superiority based on subtle differences.
In terms of domain adaptability, most of the evaluation criteria we established are semantic-focused, allowing for effective differentiation of documents within the C4 domain. However, also observed that our criteria have some relevance in the code domain (e.g., GitHub). Specifically, code that features more detailed comments and follows structural conventions tends receive higher scores. while disordered code is typically rated lower.
32

---

Published as a conference paper at ICLR 2025
Table 19: Raw training examples selected to have 14 quality ratings from to within ArXiv.
Accuracy y) (2,
Accuracy =2
Avcuracy =3
Acvuray
Accuracy This ...haaural moshodology is particulerly inseresting the putposes of ouz anslysir since cite mhake-1982reeaxi shoved that for
This
30 put 140,0
...an oider PUT, then there will be versions dbject
10 110 (9,0
primacily happens with commente,
(uo yi
120, y1
line (1, 70
1ine(1,01
inotatiions, and imports
1ineil,,0
1in 20
det0 put (20, y
stored in SM0RS
det0 put 120, yl
Dirdelbegin birdel birdel There are seven false
Li,
60
8. But the nost
linetl, 01 70 100
put (20,0 pot 120,
recent will always be returned in GET requeats, and the space occupled by the
110,0 put (20,
y1 (i, 20
1ine
70
positives that
1ine 11,
(1, 30
texti: DiFdel missing refactoring
are
detu put (20, y)
sone paransters the game has a unique
dst0 put (2C, yl
11, 0) 70 70
line Line 01, 10 90
put (50,0 put 120,
eaclier version
put 120,0 put (20, yl
references Dirdel. DiPeeland...
1ine(1,0) 70
will eventual Ey
subgame perfect
1inel1,01 01
(1,0 60
reclai by...
equi eiuilibrium
Coherence
Coberence
Coherenec=
Coherence
Coherence
1. 02.06 540
r.-put (1,
sections 1nt roductitnnn ion The
trrlabels. For
cr the proposed
emline 131.16
line 11 100
instance, in the
neural progran
82,00 125.16
put 123, 90)
HK (dordan-Thiry has been developed (see Rafs citel, cite2, cites, citeéi. Thee theory unifies
graphical nodel Fig. zef
models xapace.
90.00 542 emline
(2, 20
A comprehansiva understanding of the extent of generali parrslissblttt neural program models xepace would help
126.16 90.03 543
(39, 50)
Fimodels
121.16 02.00 544
line 11 50
rectangle camed ropiariiure defined, is referred by the node emph diegran label raned amph TopicName also dafined. Such
emline 136.16
(43, 10)
74.00 132.16
line (9, 11 70
emline
put 20)
thecry described by NGT { eu nos eu gr al Theory) (see Rer. cites) and Blectrodyaanica.
132.16 02.00 547
line (o, 11 70
126.16 74.00 548
put (60, 01
(0, 11
Topic
devel opera to know when data-driver: approaches and wher. to resort....
smline 141.16
40 put 170, 30)
82.00 549 131.16
1ine (3,1) 40
82.00 92.00 549
put (82,
The theory has been proposed by...
192.16 82.00 ...
[9, 50
graphical...
Creativity =1
Creativity =1
Creativity =3
Creativit =4
Creativity =3
.../88 21:16
7645.386 -0.053
...) 10
11,0.760)
improveneat
reasenable
... Introduction
....dies: atsention. Be maintained his incerest in Darwin and Darwinisn: in reading
0,028 500 2.93
addplot [style" ppurple, mark-s,
compared with traditional
Stereo algorithms
629 1,2CA punkts
benefit enornously
13/10/88 22:12
mark opnsons-
nethods such
from benchmazks
7448.425 -0.017
fiil-xbite coordinates
matrix factorization..
They provide quancitative
1875 nodern physics as well. idots In May 1875 James read and revieved book called sl The Unseen Universe by physicist and nas hachenation inian Peter Guchrie Tait idota and physicis: and...
0.065 500 2.63
996 1.2 punkt
(0.01, 0.476)
The rating-
evaluation
16/10/88 23:47
10.1,0.660
nethods suffer from key
to encourage
7451,491 -0.048
00.4,0.7261
conpet ior and
0.030 500 1.75
0..7,0.757)
limitation, which is sparsity of the data.
track prograss. Despite great
1,208 punkt
(69)
27/06/89 03:36
addplot [style- rred, rred, markeaqua rex, mark options-
progress over the past years, many challengea still unsolved, zuch as...
7704.633 0.028
Specifically wher collecting data real-world piatforms, the...
0.012 500 2.37
1936 1.2C%
fiiitiitt coordinates ...
02/08/9 34:02...
Grammatical
Grammatical
Grammatical
Grammatical
Grammatical
Diversity 1
Diversity =2
Diversity =3
Diversity #4
Diversity =
510
311
65 ...dd tine :50, 140, yocord)
consiiir the human brain
...5o fill questionnaire
...ma ead
n
figure. sectior conclusions
301 540 299 530
lesrning process aga in. Say the information 19 of the sensory scimuletion. A child lesrned an animal looks just like dog but having the sound
with Likert scale
299 524
301
(1, 60
and open-erded
label secicone
514 1 303 503 1 311
line (1,0) 40 deto 149, ysoord)
queat and daseribe any problems stay experienced.
subsection Sunmary
215
The zonal
322 508 226 514
line (1,01 60
surface aiz temperature response to abrupt coo/ increases ot 4, tines : in 3,000-yr simulations
1 328 524 1 328
160,0 put 110, ycoord) line (1 60 (1,e
530 1 326 540 1
subsection Results subsubzaction
322 546 1 316 548
311 508
t0 dafo put (40, ycoord) 60 50 put :30,0 put (40, yooord)
Round 1 figiteaser (top) shows the handover for all the 10 jbjects in the set textit Houssheld-A by participants...
red
15
ta 250 937 244
935 240 928
of the eat from the nanipalated noviea and this kid
238 919 238 913
performed in a low-zasolution
1 240 503 1 244 897
line 01 60
line (1,0) (1,
version of the
895 897 1
line (1,01 (1,0
leatned the dog and cat in a....
CSSMI 6CH exhibit an in...
265 903 1 2....
50 de50 pu...
Knowledge
Knowledge Novelty 2
Knowledge
Knowledge Novelyy #4
Knowledge
Novelty 1
Novely =3
Novelty 5
...speetdxei-ypp
...cne bead kind of structure is useless for our aim, being fully unstable. let go directly the interesting one. in Fig. ref PC:
heve been augrented with terns that Lfy the usez satisfaction in the ad relevance. Bids receive adept ive discounts in order to deal with situstions where the perfect infornat for as susuptt ion is unrealiatic unrealistic rituations....
...tyten recognises
gp ét solid
(311 (311.36,215.57)
gpset 1 inexidsh
hierarchy of events from the not exactly in the sease of physicsl reality -- if crezture never neasured/eww black svan, it does not nean there aren't: t any bus in the sense manually
eycle draw
2.00 draw[ge path] 12.39,, 5.33,,500333 12.255,5.033) node (ap node right, font- zontsize 0pt
(359.2,216.35)
(256.1),
2522.08,226.11
; drew [E:1]-
zgt, 255:red, 155;
wa shew how this lattice is made and textit unitary structure made up by the highlighted yellow bonds plus the...
green, 135; blue, opecity-l
Spt selectfont at 12.164,5.033) ggooler rgh color-
1 (359.2,205.79) 1376,7,216.35 (359.2,228 226.2,26.993 93) 242.6,)16.36)
0, 00000000000000002,2222 apsetlinetype go lt axes ....
cycle...
33

---

Published as a conference paper at ICLR 2025
Table 20: Raw training examples selected to have 14 quality ratings from within ArXiv.
Language Consistency -1
Language
Language Consistency = ...baseadog no LUCP, Rick
Language
Langaage
Condstency 2
Condistency
Consistncy
60
section
sectior
line line(l,)) 20 def0 put (40, ycoord)
iten fix:]
roductt ion
roduct oo Nowadays, travelara use
texthf Siis pids
hing, caiddo do AROOP, Muething realizou una anslise das opcôes de moden footnote Muething Meden nests contexto signidica solução de transni esāo digital que provs canal de comunicação sam perda Kuething...
kiologi
Line 03
90
iganks igabks
infornat iona about
put (10,0 p40140,
1ga0ks textbf and
genes and proteina are stozed into biologicsl
varicus online services and recomender
ycoord) line (1, 0)
vabazinerela textbi registreerins
50 (ine
def0 put (40,
iten(gloss:]the
ontologies cite
systens plan their trips. Recomender systems allow deal with dasé overload and make better decisisas in personalized
yccord) line 0)
had-to everyone
cannstaro201350
60 80 put (30,0
sneself
cite Guzzi 0uzz12012 euch Gene ontology (60). has gained a wide diffusion in bisinformarics and computational biology...
put (40, yeoord)
to-free-courses
line
register
deto put (40, ycoord)
Lina ()
aaunerate and anunerate Results also shax two main downsides of...
Line (2,0) 0) 60 70 put (20, p40
yocord) ... Originality =1
wey... 0rginality
Originlity
Originalit==
Originality =4
...circler 3. 5.000001
...09886278768
...6 draaw opacity=o. [Line 015th-0.75]
...3.86125 105
sectior Introductior
(55.
0.7388
6,8678 106 0.86801
circie. 0.000001
9210.52550281299
0.86903
label iatroduction IsSEPARstart I ime series forecas:ing, which conalats
(26.),
0. 7212
(10.93,-3.29) controls
0.0683 109 0.06791
circle. 0.030001
5769373529
110 0.66793 111
put (57.18,-228.44)
0.7212
(6.95 and
8.86888 112
cirsle. 0,000001
590553700
(3.31,-0.31
0.0689 0.06003
of aralyzing historical signels patterns to predict future cutcomas, is important pxchlen with scientific, business, and Indostrial... Prolesommimn
(57.666-22222
0, 742
(0, controls (3.31, and
114 0.68858 115
circles 0,000001
9388.6673514562
0.85782 115 0.8675
(58.55, 1.)
9219.0119989241 7.7676
(6.95,1.4)
117 0.86748 118
circie. 0.330001
(19.93,3.25 3,,3,,290 drax [colog- rgb, 25S:red, 126;
0.86755 119
(69.3,,,2222
0.77
C.86805 120
circle. 0.000001
279097299
0,06006 121 0,06051
put (70.00,-229. 1.03
0,754...
green, 126; blue, 126...
122 0.66844 123
C.85789...
Profesionalism
Prolesionalom
Profesionalsm:
Prolesionalismm
...472) (292, 472)
138,9 120
U..5.55
...iscics netwoek was studied in which shippers collaborate
label appli::results
(202, 1721 1203,
line (1, 01 70
controls
4203 (204,672)
1ine 30
(270.11, 370.90)
rigure ref groph:
4204,4721 4225
def0 put (20, y)
(269.53,37)
4205,472) 472)
1ine(1,0; 70
(258_81,371,43
bandle their
represents the
(205, 422)
p2t, 0. put (20,
controls
shipment requesta
three transition incersities
(207,472) 4727
line (1, 01 70
(268.09, 071,0,3,7355 371.57)
mate better with common carrier and cost-allecation
4208,472)
(267.5, 97)
[Fart the three cunolative incidances [Fart B) and the estimazed 15MCS tra trajectories C) for men with a low lavel of education (eith
4728
472)
1ine 30
(257_5, 270.26)
(209, 4721 (209,472)
line 0: 30
eyale 1 draw
(210, 172) (320,
Minet1, 30
1.70.11,370.20
mechanisms preposed ensore the susta austainabil of the coll con.ation.
(210,172) (211,472)
det0 put (20,
controls
4211,472 (212,472)
linetl, 10
1,.0.11,39..5.
(212,472) (213,472)
put (40,0 put 120,
and (, 95)
(213,
(213,472)
y) 1ine (1, 70
(273.42,358 358.959
(214, 4721
(214,472)
} 1ine 01 40
controls
In cite ganaterer2021 p02 sonerse prinary school the prisoners'
(215,472) (215,432)
dero put
y)
42 358. 95)
(215, 4721 (215,472)
1ine 03 70
and (272.72, 359..55)
dipioma), in each class. The first
(215, 472...
put put 120, y)
(272.12, 27.
dilenna was spol...
class i...
Semandic
Semantic Deasity
Semantic
Semantic
Semantic
Deasity
Demsity
Deasity
Densit
...961 (3),24)))
.).1,0) 70
...513(2]
...., fill-c]
sectior
0. 4.10541183205527951222
(1, 30
palina [iiiinn:inncoooor--aaa htt.:cc..orrbaaa..622277.
itnewidthet. 21w.2222a27.223722.6677888
INTROCICTONN
(2.448,
de50
In
years
0.10800112213270001
1ine 30
(10.4.13.151013)
(7.8022.2.55217)
we have seen a dranstic increase
[1] 12.449,
140,0 put 120,
(15.4:11.951012)
definecolor zgo
0. 0.108660 08866692221141881
y3 (1. 01 70
(1,.4211,351042)
6,116418,0 0,666666,, 0. 0.7a66.0777.9.... 7.79996teeesst
[1] (2.45,
1snne 50
paline (3inecolor-blaan
[color=s, fill=e]
within the area of transpartation and its associated research...
1.552944212205228
dato (20, y)
1inewidth=tt
[1] (2.451,
1ine 70
(15.2,13,151042
(7.8022,2.6478) restangle
0. (.18992206699119)
(10, 0 put 120,
(15.2.11.951012)
[1] (2.452,
yi line 01 70
(15.2.11.951042)
(7.97022,2.75277.
0.10625340192292..
(1, 01
psiine...
put 60,
Semsitiity==
Semsiinily =2
Semilivity =3
Senstivity
Semilivity =3
...464)
....2,
...1) (0:1)
...and the thermodynanics
...Diveree individusls age at different rates and display variable susceptibilition to tissue eging, funct ional deeline and agiag-relased disezses.
rcie|-0.500pt]
274.09052: (17.2,
(180s1); (blae]
1.000pt 1.556pt
211-9906611114421
(-60:1) 1180:1:
statistical machanics of
1.000pt 1.556pt
198.9,
:60::) (-50:1)
1.000pt 1.556pt
305. 0332200976922077)
(60:1) (-50:1)
the classical
1.000pt 1.556pt
196.
(60:11 (-60:1)
reale becomes
put 4388
305.210487692307)
(60:1) (-60:1) ;
part icularly
rule [-0.000
196, 8,
draw[aluel (3:1)
delicate
1.000pt 1.556pt
335. 204.220487)9923777
(-60:1) [-128:11
cite phyazop
put (352, 421;
198.9,
(180:1) (120:1)
Regarding
mile|-0.50pp.
335.2103976923077)
(60:11; draw(zed)
the netheds of calculation, among the rlectronic
Contenarions, exemplićying ext longevity, serve as models for healthy aging. field of hunan...
1.000pt 1.566pt
147.1999999999999
(150:0. 07: (60:1)
put (363, 425:
282.685463 (48.0.
(-30=0.87)
milel-0.500g
293. 233_95226307652333
1-90:0.07;
stracture teshniques, Dersicy Punct ional,oral, ...
1. 0300pt
136.
(180:1) (90:0.87)
288.4655492307688)
34

---

Published as a conference paper at ICLR 2025
Table 21: Raw training examples selected to have 14 quality ratings from I to within ArXiv.
Structural Stondardization
Stenetural
Structoral
Siructural
Structoral
Standordization
Standardiation
Stantardbation
Standardization
S6E5017
... statls sett
... GEGE200
Tast yeara
...aELack. color
12AIqI3 UE10707:1
The algorichan
BL112006,
binger devices
black Buch
CDX=w1K28BTVX
chat do net Learn
Iribarren2009,
like blood
fiqure summarizes
qaau JEMtnn LLC
(Spazzow and Pol)
Java2006,
refzigeration
what happens with attacked
gB3Ao, 86VSSKePU
do not degrade for the same
Leek Leekovec2 007e,
unite, CT scan
ROXWM81 n
Tee kovek2 006,
systens ens and X-ray systens are Internet, in ordar to check remotely their opesational State and make adjustment ia needed (e.9., lowes the blood unit inside se...
STPT and MP regramas the difference the adversarial and legitimate spectrogram ie imper ibpl to the human viausl ayatem. color black This aspect is very Lmp...
reason diacussed above. ittalse begia fiquie incladegsaphices
rang 998, Mar Two Fundanental
gEAs0FCawKt2r 187
4D60yn By sy7kK
types of models of information difffusonn used in the literature are models threshold...
1JLq0oQ vłkfZEx
33e lugc 19hyuh5
(widch-0.45
VgP5*61 growcelld
textur dth, height=0.
WJJaBvX] quOwoelld
G4ELaQETa5...
textheight]
pdg
Style Consistency
Style
Style
Styk
Style
Consistency
Consistency =
Consistency
Consistency
...29 o NY n Id
...) 1ine (2, () 70
...0 put (60,0
...) (-1,1)
section Introduction int THF) 3D racing game that was released in 2008 by video gane developer Nades. It past the racing came series Trackfanii. it was designed for the roncc Sports World Cup, which yearly inzernati...
chas FY char
su put 148, 0 put (20,
put (20,
papolygon
BY pchar FY
iine 70
11,
[eLiistylorseldd,
p. p PX
line 01, 50
line 01
fillcolor
PX m destheUtild PY PY pV
de50 put (20, y)
def0 put (20, y)
(0,0) :0,-6) (1-6)
put (20, y) put (20,
(fre(1 70 50
(1,-1) pspol ygon
newInfect lousChid
put line (1, 70 TO
put (10, put 120,
jdiltatyle-oola
PY p p. p
put 120,
y1 1ine
esllcolor-
p
y) line (1,
[1,1] [6,2] (6,0)
p PY p PY P BY p PY PY PY PY Pz PY PY PV
I line 1inell1,0 20
put (20, y) put (20,
( 2,00 pspolygon
deš0 put (20, y) put (20, y)
yi deE0 put (20, y1
[ellstyle=solidd
(20,
(r, 70 20
singghhild
line(1, put (20, 0 puz 120,
60
put (50, put 120,
(-5,1)
id chaz PY p FY p BV PV ch...
yi 1ine (1,0) 30
(-6,0) (-2,0)
y) 1ine(1,00 01 70 lin...
(1, 60 dato
papclygon
Hillssyle--olid,
fillco...
Topic
Topic
Topic
Topic
Topic
Focus 1
Focus
Focus
Focus
Focus
.5.55 0,2
put (20, y)
8033
...8m] 163) node [nadel, right 0.25en and
betweeen varehouses which alnost entirely cbatacle- fzee. and therefore
12490313.292500
line 03 70 10
(2.572499277557189)
0.20666688666657
put (50,0 put 120.
1.417611987501175)
33225C57.530733
y) 11, 70 1 11, 50
draw
0.25cm az, 1ine width=o. 1im] (=0) node [noell, left 0.2em and 0.25en
0.2056666556665
wiath=lpt,
13225C57
01225C57.5367320 5.5301381
deł0 put (20, y)
colce=gezett!"
only zequiring
0,21333:3333333
(20, yi put 125, y1 fine .0)
put (20, y)
(2. 22577199275557109
low precision
13231616.6440612
1.4177611977511776)
nevigation, By
0.213533333333
Line hidt (s5) node Inode2, right a 0.25cn
ignor ing this
1323:816.5440602
put (30, 0 guz 120,
(2.579999266833992,
he eroergenitty..
0.22
1ine 70
1.91122191920022:7)
mobila robcts forced worst-case decisions to
13282162.5630116
line line (0
40
draw[line
and 0.25cm line width-0. .1m) (561 node (node2, right 0. 25on and 0.25cm of a4, 1ine wiath=0, 1nm] (a6)
put (3, 40 put (13,50 50) Tine (), 50
width-ipt,
13262462.5634116
coler=gggeff"
0. 0.225668866667
(2.57999266633199,
enaure their safety. For example, for she aboy...
13454C96...
put (20, 90) line...
1.4192818792042009)
12.5...
Overall
Overall
Overall
Overal
Overall
Score 1
Score 2
Score =3
Score =
Score
.7.4 (202,
(30,0 pu:
work
... Tabel
(202,472)
(202,472)
y) 1ine
contxola
was acudled
appli: :reautta
(202, 472)
(202, 4721
70 1ine
(270.11,
which shippers
Figure ref splot: represence the three cransition incensities
(203,472)
(203,472)
30 der0 (20,
and 53,3,.3,31..88
collsborate and
(204,472)
1201,172;
line
(268.01,31..33
bundle their
(205,472) 4205,472;
Ci,eC.,) line
controls (268.69,371..77
pnent requests
(205,472)
4206,472;
Line 70070 80
to negot late better
(206,472)
(207,472)
put (20, 0 pu: 120,
and (267.5, 3,30, 98)
ratea with a
[Fart A), the three cunulative
(207,
(208,
1ine 01 70 line
(267.5, 3,57,5,370.29 35)
comman carrier and
4208
(208, 472:
30
cycle draw (270.11,
cost-allosation
incidaaces (Part and the estinated mear 18M2 tradectories (Part Ci for nen wish level of education (wish ne prinscy...
(209,472)
1209,172
de50 put (20, y)
nechaniens vere proposed to ensure the bity of the collaberation. In cite gans....
(210,472)
4210,472;
Line () 0) 70 70
controls 1270.11,369, 369.53)
(210, 4721
4211,472;
put (40, pu: 120,
(211,472) (212, 472;
1ine(1, 70") Line
and 3,38,,
(212, 472)
472)
42,
(213, 472)
(21...
(2...
controls...
35

---

Published as a conference paper at ICLR 2025
Table 22: Raw training examples selected to have 14 quality ratings from I to 5 within Book.
Accuracy =1
Accuracy =2
Accuracy =3
Accuracy -4
Accuracy =
BAERK SMAT..
..-npe Hell upon dis, Pharissea
..-ountai 34. 26
-'eentyy years ago Jarry Milson was known as the
cana shurch ateeples, and a =hrae-atory brick opera house. Instead, they found a balf-burnt town ruined by a devistat ing blaze that lay waste twenty blocks fow months earlier. Holcemb had taken leave of a proud, 250-year-ola city touted
LETTER DI GREEK
London Season
SNALL LETTER
picked up dar frails and cut away and as day passed by hin he felt sich
33. 27 Turning
SPSILCH GREER
Forty 36. 28 The
castle king of the Platte River, Bis cactle rosmed for hundreds of milles
SNALL LEVTER
right by
clarsical style 27. ncknowlesgments
AHD GREEK SMALL
LRTTER TOTA WHTH
38. Not 39.
VARTA GRBRK SMAT.
queer pain in his head as if sonebody had 31
up and down the
LETTER TAU
GREEK
Credita 40. A
maic. river and all its tributaries, and, as the cowboys used to say, no one man could count then
SNALL LETTER
Note About the Author 1. i 2.
ONICRCN NIIA
lanentable hard thump wad a hemeer, dat knooked him down flat as floun...
VARIA GRSBR SMALL
3. iii 4, iv 5,
LETTER NU GREEK
6.6, vi 7, vii 8.
SHALL LETTER GREEK...
viii 9.
it they
the ... Coherence =5
Coberence 1
2
3
4
14.
15.
16.
d..de NonE4s allaient
....ett Lawzeace half-trish,
Eo.Eo lecture.
Sovie: bloc dipicnatic and trade missicns heavily staffed by intelligenoe operat ives. The age-old of aspionage had not faded away, despite the world entering the apparertly softer, gertler
Billy continued neekly listen. The old man knew that sex was at the on of it, Be sav that Billy had sensusus nature whi ch needed to be curbed. ailly giving up to seek out Queen syyddningsala--ya he knew Billy's desires...
22.
22.
23-
20.
veraient, ez
half-Eqyptian copt software engineer az a campus concert 1el tolk mualo, Ananda had gane on a lark, friend's s urging. The chenistry had been despite
25.
26.
27.
20.
quelque chose
29.
30.
31.
32.
d' d'impuiseant et d'indigné dans qut pouvait apercavols du
33.
34.
35.
36.
37.
38.
39.
40.
41. 42.
43.
44.
45.
46.
47.
48.
visage au-desrus
19.
50.
5L.
52.
de in serviette conti invit presser main
53.
54.
55.
56.
60.
62.
62.
63.
64.
déchaznče, 4t plus the fact that Lazry resembled Woody Allen with dark hai....
65.
66.
67.
68.
tard encoze à ce monent ce deveit
phase of the Cold Maz associated with the détente be:ween...
69. 70.
71.
12.
73. 74.
75.
éare la seule, l'a...
Creativity 1
Creativity
Creativity =3
Creativitt =4
Creativity =3
47.47, 150-53,
and Fletcher
...qu'
za.z and pear
...ank into his chair, suller pout
188-89, 229--7, 282-312,
reck (1810),
n'oubliers
portographic
the
pas. Pile
1'pa.
vere angzy over the
Prrigine da son pragnations on politique. La question dans monde ensorcelé n' est pas de savoir qui a ralson, qui va 19 plus droit, à la mesure Grand Tronpeur, quelle action seta assez souple, asses...
private livea.
And don't t you think about going behind my back, I warred husband number two as eat at Baniel's right. You will not be arranging a handy accideat, understood: Hell, my lover, I've found T...
369-70, 375-73
Court'a assertion
Desective Sergeant Nornan Pilcher craved fane and recognition. Back ing them up were a bunch of minoz dyspeptic personalities, self-styled
law, 15-81, 247-82
of pawer to strike laws;
147-02. 147-02.
down
147-82.. See
and both Virginia and Kentucky had made dark threata foreshadoxing a constitutiena
a'so natural lax legality, 154-55,
lagal status,
ieitniz, N.,
mais
shat
liberalisn, x9,
crisis in 1923, according to Robertson, In d...
273-75,
worthies, suburiann officials and legal types. This
2573-75' liberty.
99-10...
Grammatienl Diversity
Grammatical Diversity =2
Grammttical Divensity 3
Grammatical Diversity 4
Grammatial Diversity =5
28.
29.
...5), 308 (a3)
Gunnyy La was fighting lcoked up, and the old corkecrew right vent over and the dunny started
& nornal life, whatever that now watching many of ny friends enter their second marriages and, while I that happy have waited, to have missed that first of divorces, hope find soneone special when the tine is right. T...
...ed the canon Dead Mhite Europsan Males, the Anglo-tnerican core in Rnglish depa stments was supplanted minimized-aand along with it z consciousness of the ancient and redieval ynol ogy Bnglish worda, a complex Linezge that is wittily
20.
21.
32.
33.
Leonard, (n53) Reszen, Leo, Rothweiler,
36.
25.
36.
27.
18.
39.
40.
41.
42. 43.
44.
45.
Monika, lr13, all)
46. 47.
48.
49.
Buhlen, Herritt,
52.
53.
(n23) Kules, Ses also conbinstorial systems; Reads;
trilling the daisies. And the haseball games in the old days of spike Shannon,
54. 55.
56.
57-
58.
59.
63.
61.
62.
63.
66.
65.
66.
67.
68.
69.
Symbols and
23.
synbol processings
Pred Ternay, Jimny collina, Cy Young, Pat Deugherty,
Mike
74.
75.
75.
77.
Mord atructure
78.
79.
30.
92.
Buneihart, David,
92.
83.
84.
103-11...
Fielder
evokes
Knowledge Novelty =1
Knowledge Novelly=2
Knowledge
Knowledge
Knowledge
Novely=3
Novelty
Novelty
...I joined them for while and ai aloked up little, you know, cab fare. Then forced everybody, including the conductor, to get the lest car, and I pulled the
...of the affals,
La faveur
that would train people to and useful citizens
...IPvs, proposed
prefecring pretty face poverty. scupid devil, to throw away such a birthright! lucky doo, xho his successo27 let the rogue win the race. I an so tixed of the dodeas, the tvists, the sliases, the larkings, that 1 wl11 put...
d'obtenir un peu Devenons
originally in the wid-1990s. When would IDve run out? If you vere to divida the totel number of pessible addresses within a 128-bit space by 7 billion people, it would be abie thseretically allocste
ces ânes
cholsies Que 2f godto travers leurs corps contraignons,
in the nodera technological socicty. short, popular culture movementa вaи caiture as the means by which the individua''s relstion to society night be smel moratedd Many of these fourdatio...
en souffrant, Paltiire
pin and left them back the zunnel. sonetines that's
fantaisie, - Almar moins si fost encor ! Il e'sst pas, pour nover une divine astache, Que oea
the only way you can get seat. A...
approxinately 5 k 1288 addresses
36

---

Published as a conference paper at ICLR 2025
Table 23: Raw training examples selected to have 14 quality ratings from 1 to 5 within Book.
Language
Language
Language
Language
Language
Consistency =1
Consistency =
Consistency =3
Consistency 4
Consistency =
...tott,, mutta hän valstesi, etts kuski pettyisi, Jos hän paljastaied olevanes vaills Gloria
611 inngttnn len, tréa chronatiqun refrain d' on aucre monday voluptueux, onirique, lui tient compagnie retous. M'rtt ou pas, 11 efen sait riea, at eux, qu'en savent-ils, si L'onbre de aroix zait loi 2 La censation d' une repéti...
11.. Pt
No, necesito ayuda. Boy nuava aqui, Execuerdas? Arte tacto, senti un tiblo escalofrio que con la anistad. I' por quá no? Bus mejillas se sonrojaron, ojos centellesban; bajo los focos el cielo azul verdose de crepisculo...
ex.epi-oni se Athena'
pendant longtenps ja arréttai
general shift from oraiity to the increasing of writing, This lopment un also Locked in Solon's less actractive legislation, of sereed 'peculiar' by Plutarch, the epidikasi example, procedu...
cotte conclusion,
qu' a moins d'avoir seize poches, chacune sa pierre, je of acriverais
ollut koakaan
voinut käsictss,
ett' toke
jamais au but que ie n'ctaie proposé, moins d' un hasard extraordimaire. be s'il était concevable que je double n...
halusi köyt:&8
kiduzukeen ja
kucleman välinetts koristeena, zhtá hyvin olisi
kantaa Mirtiosi
Originality 1
Originality =2
3riginality=
Originality =
Originality =
...2013, B.S.
...., elections Seokleeeur---oheee
... heart hanners in the
... ... 2Ux nivesux scolaires les plus Al sléé ob I'fffet de suraélection tend neutrallser Les différences de tre que, premierement, fait moins sppe1 une compátenca atricte et strictament contrólable davantage une sosta familiarite...
-6n1, 188n7,
jenua=i 2014,
"y 259n33
inuerking reding-
rib cage nuch
scknowiedgment Ty ... 61n10, "
30 ap=il 2014
state isserbly (1927), elactoral pezfornance 1929 state parlianents, Danzig, Lipge state
longez than it should, but unmolested, for now. I walk calnly away feom the bus and change clothes find asivation Army kin. I select a nan's apparel, compresa hair inso woolen hat, and di...
K.3. 4
Ty ....
april 2014, B.S.
action
29 spril 2014)
115n21, -, 45m2,
I hoooddstuk 5
1115821
bepal i
elections (1938), Preaidential
<. 125n21,
**l... Opschrift ingevoegd bij art. 2 wet 15 2013, B.S.
128222,
(1932), - Reichstag
,115a21
(1928), Reichstag
(1930), Reichstag
115m21,
9 fanuari 2014, inwerkingtrrsding:
(1932), Ba...
agee.
Ja.s
Frofesionalim =1
Professoollsm 2
Profesionalimm
Professinulimm #
Profesionalism
... Jagered in each other's e scms after a conjugel visit bus station toilet stall. They call darvis an rUncle Ton of Finland, ' said Centry. Delicious arked, what's that nean? Sentry shrugged and shoos his head. Something about too many mascie...
.oE faceva
to Heaven dicectly, where you may join hes your convenience, Your nost humble and obedient servant The Prince of Evil Zane stared at the message, abeorbing its every impl ication. Suddenly it burst Into flane hand. droppad it,...
... For the
.tes the problen of freedon in Adan's e choi ce
venize 1' l'ffaann,
purpose of the object on which enter, consulted a great mass of He can agree with telbnis that
un affanno insoppersabile, per cui avrebbe valuto halzare in piedi anani ma non poteva. 11 cuore le batteva precipitoso gome il ga leppo of cavalle anappato. 11 cuora non reggeva plc, fo... Semantic
of hinself world.
in
documenta, and have had recourse the personal experience of gentl who has made this kind of research his buainess. statement make, ve shall
another gesture of Adan, implying another Adan, implies another world, but only In the sense that another face
every
world xill
corres...
Semantic
Semantic
Semantic
Semantic
Deasity
Density
Density =3
Density
Density =
36.
31,
35.
..-ancholy bait,
...adlen, Bunist datzu urad schreybst in alle welt, du kundist allis zuszwendig und geprauchst keiner pucher. bu dorfftist Rumis nit, nan siht aa nehr dan du gleabst, das du allis an buther schreybst und lerist: wen du die augen so fleysalg...
..d FeSDLEGRS,
...Pro:est asta
39.
40.
41.
42.
60. seld for
bus as ainners who ware lost. Io Ulaf different than Conor except that Conor had accepted the escrifice made for kim. You're still here. Conor looked up. Haidoz stood over him, holding dripping bastle-ax, face, ha...
king in Izeland, Protestant
43.
44.
45.
L5.
more than an ex,
17.
10.
19.
50.
734.734 fry,
parlisment,
52.
53.
56.
ocher, 772,
Protestant
35. 56.
57.
58.
172, 192,
hissarchy, Protescant
59. 60.
61.
62.
790, 190. what
63. 64.
65.
66.
elestors and governnent, the benches of justice, the arny and revenue, through all their branches dessils, Protes:ant....
69.
20.
790,772, 796. with the
381,
71. 12.
73.
74.
75.
76.
77.
78.
worn, man may nay,
79.
80.
82.
82.
141. Fistes gnaxed that, 96. 772, live
84.
85.
96.
89.
90.
91.
92.
93.
94.
the sea, live in
25. 95.
the sea,
Senstivity =
Seusitrilyy =2
3e
Sumiiindy
Semilivity =3
... and Incorrect
.t. hers ha stared a7 the fire. The saft light did not touch his eyes, Not Arthur?
...ca the face and cried, Tradizore! Father Faciere covering
not like it. "But it wasn't t all that much, Just walls
...6 Bichard svistly, but silently, raised the avoré before hinselr preparation for an attack-wwhat kind of attack he but he fully intended to be ready. Be touched the cold steel of the blade his sweat-sliex forehead. Be spoke the wo...
ways to be born. And without question. the
sone busted glass. Bardin cane back to his main problem. ' 'The gner. bid you ever nost hin?' 'No. Biggie set neeting for tonight in case had sonething trade. That's why...
powerless to stop it. The solda:i finished their part of the cezemony. Babe, red cape slowing in the drafty place, held up her hand, Glorgio rushed forward. Ee gave haz SHo...
dlaguating
The air huamed with Pover. This was
deformity to be born with was gayness. Walch was why IJP always deleced his browser history and hated The "Hought and hated gay people but couldn's z hold it aqsinst chen...
no place for lies. touched his hand, and when he finally surned, she qazed into his eyes, Rapecially Arthur. He e...
37

---

Published as a conference paper at ICLR 2025
Table 24: Raw training examples selected to have 14 quality ratings from I to 5 within Book.
Staddura
Strnctural
Strictural
Structural Standardization +
Strucural Standardization
Standardizatinn 1
Standardization 2
Standardization =3
...bevall, bochoo, grieve, lament, regret, snivel, aqual: blubbar, depiore, trickle, ulvlate, whimper 8 complain break down, make fuss, percolate ahed pity bamoan, lanent (for): feel over: beusil, repent ready to...
...0f scotch END
..s Pene.ope
... Rarns
lieutenant he*s
nights before. Tenpting as dropping the liquor have been, decided he'd rather eat paint thinner than pointed questions fron locals about Susan, There had beer. some changes the Martin househnld over 111...
lsttene af llenllly
SK, Hokhopachyay
got a neck like
like a girl in
A, Aggarval Ba. Resveratro:
Buisa ateez,
dream. Now
tried to
nay
Mrs.
splits
Fairwsathez's
IME-induced activation of nuclear transcrige ipg lon factors NP NB-Kappa activazor protein-l, and apoptosis:
beauty in the Catholic House and gave nyself
soft old hand
affect affectionatell It sounda like atorybook, she You must came and see Doris. She is suct
herria, which lantt bad
gaily.
man, 2 man makes anything look good, but when maider lovesick...
potential role of oxygen internediates
darling
wouldn't have had trash the pazzon....
and 1ipid peroxisation...
Style Consistency
Style Consistency ... T'ext réne du sertir-ore orratteu.. Ailer au bout des puirsancer du seul à deux. Reconst
Style
Style
Consistency
Comsistency
...my buggy vus cozn to places, an! knocked
...their names appeazed or.
fo.s and both castured. In 1513, just before battle ct Floddon, its walls were at length Caid low by Janes TV., but not until the famous cannon Hons Meg-still, I selieve, to be Edinburgh lastle-had brought against it. Ona of th...
Beircipe, Production, mode capitalist, -plantatien, Proletacian,
the
aiz. De
where possible. Unbilled playere and technical persornel are they can ba reasenably docunented. Various
first time day run into ne dey killed hoss. De third dey paralized ny arn busted the linin' of stonach. learned read write surrendar by atudying in s...
x... Proletaziat, Protein, x Frussian, sudding, r.. Puerto svi, Paerto Rico, xe, svi, avill, xix, 2x, xxi,...
a'au-del é:sts privés et cornus. a La prison du miroir enfin inctile, grasait delivzer cat:e voix peut-être cncendue, prise
webeites--for
example, Insernet
Movie Dacabase www.indd. con) --fatuuee
credits for Micheaux f...
Topic
Topic
Topic Focus
Topic
Topic Focu=
Focus
Focm=
Focus
TONGA) kankin (23.4.
ssid,
blens,
Mission
...Ohlo Railroad,
ninute oz century
San Antonio, label. Made in
103-8, banking industry, 26-27, 88, 93, 201, 212-13, 298, 305-8, 305 359-60. also specific banks bank notes, 40-11, 90 Bank of Earle Augueta 93-103, i15, 132, 145-45, 181, 430 Bank of England,
109
DAGAARE, SOUTHERN)
ago. Give him
in Honterey,
kankkenn
a booster shot. Te can't hurt people ware sitting en a sheif, f,, looking down at him. A bare-bressted
the philippines. She flinge is and screach, Is
at Purisina de Cadegoné, in Sierra Gorda, -25 as scudent of Sezra, -32 travel to Alaske, -65, travel
dagoeblad ad (36.2.
VIAAMB)
(36.2. INDONESIANI kankri (38.13a.
on ny shoulder and I threw it at Shma, who ducks aside. grab again into
RCNEAR))
kankri (38. 7a. KINDUSTANT] kankro (38.13a.
fornie, to Korld, Croix, Karquia
,
white-haired oldster, a young Negro girl, flabby execative-type man, Goonto...
chese the kitchen. Baze, here. I throw the canisale
NEPALI) kankro 13b. SEPALI) kankrol...
Carios de,
Croix, Zesdoro
Overall
Overall
Overall Sowr 3
Overall
Overall Score
Score =1
Score Barkedale, Brianna (chacacter),
Score
... 02-03,
...63 Chepizhny 1961 v, chepi thny 1984
...You wozk for free, 1'll put you
... ...-ended also crucial here. I
... ...uncie's life, but changed things enough for my dad that he married my mom? I pondared that for an extrenely long time, and kept coming back was thiss What 12 I did? Selfishly, I had adnit liked being slive, but also ...
up nice hotel. You can eat great food I'1l
have sddressed hes important concerns
r Rarkadale,
v. Chegishny
orAngelo (chasactex), ,
1983/84
Chapter Sea La Dauff (1985: 126-8). *+20..
Cheralavsky 1976
give you bookez.
E. B. Cook 1958
*ADA DUBIN:
205-95
N. n. after 3.
always give the
Propositional and assertive modes, le valuable for certain phatosophical cal work, arquably restrict the open-ended inquiry that t...
and Avon, 84-05, chess, and dealre to escape the Same, MiNggsts, and Stringes, walls...
cook 1913
desetie Boys
Conk
C-
because nobody
B. Courtanay 1868
in the record business, even people who knew them, would' given...
and
c. E. Courtenay 1870 J. Curpe...
38

---

Published as a conference paper at ICLR 2025
Table 25: Raw training examples selected to have 14 quality ratings from 1 to 5 within C4.
Accuracy =1
Accuracy =2
Accuracy =3
Accuracy =4
Accuracy =5
zon data of data the quantity of prestitutes or the planet exceeds millions today. Your mail uz! should contain appropriate infor- nation on such web page. Prequently is the cal message, Some pager provide you with few 1i...
-. day services availabla in many instances, to make sure the waate preducts tend to taken care of as quickly and effectively as poasible. Otherwise the najority places quarantes their storage containers to be right now there by the subs...
Facta IE
... ccorrosion and ansi-rust agents. The oil has Proprietazy toshnology add on that provides an extra layer of protect ion that pelps the while
...31140 (2016).
decidas to ignoxe.
AIP Conference Broceesings, 1711, 050016 (2016)
Honzs is not Serzoriat orgenization. is pelitical party, which has ges control of Gana by legitimate imate denocratic
European Journal of Inorganic Chemietry, 4273-4274 (2018). J. Am. Chen. Bee., 137, 12498-11306 (2015).
day,
engine
means, during an electica which was witnessed by forner Presi dent Carter. In fact Carter Cente...
maintaining propar temperature
Chemical Science, 6, 4306-(310 12015). Inorgario Chemistry, 54, 11593-11595 (2015),
stability and oxidation. That
feature you will not find with
Cuherence
Coherence= =]
3
Coherence #4
Coherence =5
..x Eron your comittee into a plasma. The
... lake from en old channel of the Missiesippi River. flosting casino is located on the lake neaz the downtowa area knewn the Teep Ussino Sreenville, with a second juat weat of the city near the Greenville Bridge known 6s Harlow'a Caaino...
...ing plate drive mechanisn. 0001 The stitching hone position not
... SHOCKING
...lbis 15
Package Seised
resporse to your
and Destroyed
Request for Set Aside of Denial of Porther Reviev of Droteat and to Void the Derial of Protest Humber
client chared the
Rayal Saill
sensor go on when the atitch notor (reaz) has baen rotated in reverse for 0.5 gec or more.
use nainly simed at international
Unittdd with clearinghouse
interne: the
aiz shipmants but since then further rules have been int reduced by the Insernational
contraction of pushing your traffic and security pp.. naterial of the law for Japan's
(103-99-100302
filed behalé
Seh notor (reat; / itching hone pesition
of the inporter, Mast Induscries, Inc. (Prozestantl, seeking set aside the (art de...
Air Transport Association, or TATA for short, with their Danger...
(reat)
anlid affect?
N338) Raplace the atitcher
takes cone > with. covalen...
Crentivity
Creativity
Creativity
Creativity -4
Creativity 3
... Matervlllll area police ooe reports for June 14, 2016, AMBON, Manday
". .e.2e the deal before it's gone. Check out 709 off Fine Jewalry. Bealis Department store now. Find more discounte and offers from Bealls Department Store fust at Cougonianie in Agril 2019. Save 73% off Pine Jeweizy. Seize the dezi before it's gone...
...A hore bullding project can be the chance for couple
..o worked and he behaved in the rannnz of idle danéy. would even conduct interrogations lying on 5 sectes draped in rich chinese silks, nanicuring hinsel: while he put his guestions. Yet he had inapized trust and was tolerated with amuseme...
... d6i-War. stands at his Master's 9 funeral, he remimbera all that happened.... and all that could have been. The menories that zampage ny mind are painful, yet so beaut iful in their
3:24 p.m., a bacassment complaint was invest igsted ailltop Road. CLINTON, Monday at 9:45 a.m.r report of barasament led an oral warning on Diamond Avenue. 3:57 p.m., thr...
their drean cone true. The fanily has grown, there has been professional
auccesa for whataver zeason, it's time to hard-earned funds renovated kitchen, an extension or bré...
New can this be? Now can he be gene?
Everything he
Grammatkal Diversiy=1
Grammatical
Grammaticca
Grammatical Diversily =
Grammatical Diversity =$
Diversily =2
Diversty=3
...aked to
...IE you are the typical inventes, it very much possible that you would probably iike to license all your invantion. and receive seyalties, or even sell that it outright we'll lllll dub that person royalty
.r.e sale
...be against their
..-Hatthias
where this
colculation
Schoeracrts and Diane Kruger
contributes us.
Sales, ypplyriieeee
should have
Our radictory
to allow essier
listened. In
head! ine sharp,
notwork contains
modding.
the book, I could not count the tines obasanjo slapped. his wife, Madan Bemi, or the number of tines she herself rlapped
slinky dive into genre territory for
really ssid as inglies when fee funcsions. in Time to know a induccion to present in us, the registered compary has the exposure, Vanities elininated been
Fixed: gdt-modAPI: Checks. Checks.checklissa wtw.kktissoonoveeeeeee did not retura true when it should. Fixed: when using windowed mode the window sise starts sealler than 1024x788.
sepa helner Alice Kinoccur. Maryland is the original title of Disorder, the secend feature by Parisian writer-director Allie Winocour, and while not one
hothor. Rut you really are noze notivaded
nany of Obasanjo's
nunerous
Fixed: End
nistreases
paramours,
including...
Knowledge Novelt= 11
Knowlegee Novelty =2
Knowledge Novelty =3
Kaowledge
Knowledge Novelty CR sequencaa fxom seactive CD4
Novely =4
Free parking lot to 2ian and the Village Alpiaz, near to the footsteps. Gevardine reet,, near the Body shop Livingstone 2, in the place Bettolecto. Anple parking lot before the Blue Camping Borco, next to the ski footsteps, Parking lot of the
before washington journal. * en tuesday, the present the unit european parlianent
...ray'a The
...heped construct one piace in what will be much larger conversation. And certainly didn't start anything ourselves; people were talking abous just wanted to giva it bit of a pash and inspire others to take what needs happe...
Legenda of Serusalen that Rabbi Lariz supposedly knew
I cells from 22 individusls with
day supernaturel way where Jeremiah was placed in the Court the Guard mant Loned in Ceremiah 32:2. This was the key. and when He-Ari the Holy saw hin. There waa also sid...
discuseed efforts to conbat the islan' state and terrorist groupa rollowing the recent actacks in brussels. intelligenceerr sharing, islamig radioal...
Mycobacteriun tuberculesie infection.. Me feund 141 mcR spacificity groupa, including is dissinct groups conseining TC8s from mpltiple individual.. These TCK groups typicall....
this,
39

---

Published as a conference paper at ICLR 2025
Table 26: Raw training examples selected to have 14 quality ratings from 1 to 5 within C4.
Language Consistency=
Language
Language
Langange Comistency =4
Language
Consistency= =2
Consistency =3
Consistency =
A...Anyyhhaaa. While the nanoscale saved for the pui ignore, the service of the efficacy were one of his cexts to rotation for la. The Blias Stadiatis, taught the breaks of bacterial elenent at the knowledge of the side, 60 act...
... Waich user
...good quality
...Aro you aware of any new trestmente being researched? Rov you feel about participa:ing in clinical trisls? what would prevent involving your child in a medical research study? Interact and reply in the conments belox! No: sure about new t...
...ōn Arrival:
Lovas Freedon
furniture
Presumine you are already within
th most or Betes
affordable
freedom the nost?
quality furniture
italy prioz
only you could see hov aqainst freedon you are.. I love freedon anyone who doesn't I love it should go dia! You vanna back that up? Being against abertion doeen't nake me anti-fr...
foordablle
arriving in the Amalfi coast, mest comennl you will errive by train in
quality furniture affordbble quality furniture high quality afoordabe
you
high quality
Naples! in Salarno. We buy our train tickets online through Frainlire, From here you can get to the v...
furniture brands satas. good quality furniture high quality
furnitere brands high quality furniture bra...
Originaity=1
Originality =2
Oriiiality=3
Originlity
Originality
tobteble belew, if you want your kids longer
... Spinit beasts
incllde
...0h yeah, that meaningless, well-worn caschphrase used by influencers and felf-help noaches. What's a in it for people fact,
.. ...iteer 5 years of a continuous Lawful
choice of ganes and slots and also a brand new caeino. Supplying a varsety of payment choices offered some Languages, its prevslence has been increasing among fans.
services such as
sine, please
and social
in Laxenbourg, thi thi-d-country nat fona's (1.8. from country that is naither EU Member State
choose bigger size,
act aetivities Seajoy cla provides asalated living only to residents, but also to all Dekalb county residents as well. If yeu need assistance with daily tasks, Gordon Care can help you or your
Factory direct
aell, quality
Made in china, fabric is incredibly soft and cemfortable. direct sell piece retail hot sell /free ah shipping. Machine H...
though? personal branding is something you should develop and control.
nor a country treated as such seeland, Norway, Lichtenstein and Switze=land) may make an spplication to obtain Professoanlsam =
itfs fun and conterporary, and nore i important ly foz casin...
Especially when it conea to your grofessionsl
lif...
Professiomllkm
Profesiionalimm -2
Profesimmalimm
Professionllmm
PPrograns, rector of adnissior, inhabit floundara, and build a relavant besondere for conaequancea. using the CAPICHA gives you involve giant and coincides you
...50 long. But yesterday lunch, high hill in a rolling green pazk, undernezth
the majority of the work was done during week. Brainstorn or creste content upgrades (freebies give away the end of the postl A perfect
oilf you're rr thinking of selling your propezty, our qualified real esiate tean is the process as easy as possible. We have a friendly and dedi ested staff devoted the profession and staying on top ot the narket and trends. aze...
..riann Eartin, 00, FACOS, CCDS, explaina
confusion sehind the various sepsis definitions and providea guidancs to ceders when reporting sepsis in ICD-10-08, For natiencs who suffer from frequent аyлotona of gastzcesophageal reflux disease (G3RD)
huge, old tree, listening to the church bells. And then i vent back work. I an working
here
example would be the After 9
asper free
Theive Guide that is incloSed in this past. Brainsteim or tripwi...
jugaae gradient nethod without
web designer for a college in leak. As this is the week
zo the concrete inanity. What
Semmatic
Semantic Density -2
Semantic Density =3 ...6 5 -6 2014 29 llancalach faur manor blaneslach Fawz Manor - Locations Guide to Uoctor
Semantic Density 44
Semantic Density = In reaurrect tua Christe, alleluis. In resurrection, thrist, alleicis. Cwli et terra Sttnttur alleiuia,
Density -1
... I found your liating on Padlist and I'a intereszed in coning see it: ww..pd.iit..com
...Prssident Trump has a prospect lve ally in the war on Morlin immigration, and
.. After googl ing an at empt to get Matzex 5402 fully operational, came across this forun, which appears to have ive and koowledgeable connonity. soneone can help ne with ny cilenna. The adaptes card resognized Kas G...
talking about
listings /
the patriots the J.S. Border Patrol (UEEP] Innigrazion and Customs Enforconent
Who location: Llancalach Pawr Manor Decails The Locatioas Guide to Doctor Wao, Torchwood, and the Sarah Jane Adventures. 9 -3 2014 sep 05 shice hall m...
m00-menroe-
avence -n-ritne-
let heaven and earth refoice, alleluia, Jesus has providad for everythingy he has chosen twelve whon calls Apos...
98056-1233
Can you please set no know if atra still available, and when might be able to it? Thanke! 85, foun...
hope
and Removal operstions (ICS BRO), as previourly dieuussed in...
Semilivity =1
Semilivity
3
Semilivily =4
Semilivity =5
...euracce.
scconc stage engine should be ready by 2022-23,
...How can
... "oommally', and иe have expanded then with the positiws lfberties
... ...dininng job in the healthcare industry will not be di difffeu't if your New Grad Hursing resume has all the imporeant detaila on your skills and other qualifications Looking for work can be daunting task for new Nursing grads. Expazience is big d...
Shanks for the info. 1 vae just wondering about that. :-] But even sonecne agrees with my paint of view, atill of deliberately never patronizing any entity that
get the best
73-461 Prep Guides
if this is true imagine that all she units of the second cant would
Guaranteed Success training Drep Guides? Now ciick in Ignoexan. nom] is work. NCSA How get the bast 79-461 Prop Suides Guaranteed Success tcaining frep Guides? Now ef lok in Ignoexan. is work. latest
socialian
pronisaa. Na have real Bocialist denocracy, popular denocracy,
pollcy
izd. 30, either retrofit or directly instelled, it does net make to have half aquadron with engine an...
genuinely free elections, etc. In this sence say it besrayed its prenisea, because the whole thing
uses guerrilla ialentiaing cechniques, apan etc., so this act...
40

---

Published as a conference paper at ICLR 2025
Table 27: Raw training examples selected to have 14 quality ratings from 1 to 5 within C4.
Structural
Siructural
Siruciural
Siructural
Siructural
Sumbelliztion 1
Standardization= 2
Sumbardization =3
Stamaaddiation= 4
Stamlanlization =
Succh-lveed
...Fantastic Kubin
So.So, the latest
kel Na5 Forted cancel
The Fresident India, Shri Pranab Mukherjee wi viats the
charactes actor
Sofa Woodhaven
who specialised
ingortant have in almost any home. You need
Sernar domestio agy agenoy, officially called the Fedaral office for the Protection of the Constitut ionion (Rundesant für beetasungaachuss is the ravelation that it has been spying on at least 1/3 of the feder...
pl planned
to hhe nevaroon of Southern Weekend,
playing 1y
aleaayaligh:ly
Rashtrapat.
ccceatric and
for the greatest you to actually never overpey for them. Noises listle complex, propez? arfestively, this post is on this page to help. Keep readin....
Guangzhou- paper known for outepsken reporting on corrupt fon and /BCeecee sensitive issuea.
NL Mllayan,
often ... Konentar Br. Poslao
Secunderabad for ten day southern sojourn from June 29- July 08, 2015. During his southers setorr, xill visi: Tisupati in Andare Pradesh on July d2, 2015. On Joly...
Severin Datun
: 03. 2005.
Najbllje je da vas muz provjeri
kolssterol. Velo cesta povisen
schadale of Merkel's three-day trip so China was altered to ineert
solesterol je uzrok kamencima, keko
Style
Style Consistency =2
Style Consistency =3
Style
Style Consistency =5
Consistency=1
Consistency =4
-.. sone
...I? you are anazing therapy services call
child accidentally
Image Tron
... ...I sad dostar's
1ight
vistual sculpture that was created from :90 people traeing the frent side of their bodiea.
obligations,
knocks the spaghettics and mestballe off the table and it lays in giant red heap on the fioor. The last paper towel ured only seconds prior to incidert. When all wlse
appciatment coday, and in the office, they bad poster asking their for patiaace aa thay woek to implamant electronic reserds. The signs have been last few visits, so I think the bulk of the efforts are complete....
others getting to bleating, absolute pain, yoghurt and long Inward, la the othar preparation INo weeks pricr to energeney, This process energy and helps ald your tolerance body treatment. Its 1o...
the grofessionsls
Body Central Physical Therapy
pat
soon pessible so they can provide you the
resul: which scuipture of the space between the two bodies. virtual sculpture that was created fron tracing my body as
best solutions
your
natter You'll of work with these guya because
up
fails, you use a dishtovel, Whan the child that acc... Tople
they really do
Topic
Topic Focms
Topic Focns 4
Topic Focms 5
Focms
Focms
iacllce
the direction. the objective furet on he cost inuoua in avery parametes, which not always the case. in the specialized
Raw sockete
Home Fitness
...ang-i1se.
person who.
allex program
Can Lenon Water
Squeese water out the
Probably, the
or aggli ina to
Really Felp You
Orient conea
provide suacon headers for
Lose Reight?
gelat in, and add the gelatin to your hazslaut creme anglaise, stiering until has me ted and has been incorporated. stiz
second as faz as the popular fruit nachine thenes go. Golden uotus is slot play developed by RTG, the company that famous for production of the quality rofiwace for the online essincs. mise. cra...
and elinical
specidic
trials need be conducted in people before any claims car be made. Drinking lemen water reçarded by nany professional nut nutrisioniste ax having real and palpable waight-loa... Overall
protocol (top ip)
which are otherwiee provided by the kernel/os necwork stack.
literature ve car fand other alternat ives the Gauss-Hevtas nothod peoblem. lem. for example, the sirgu' ar value decomposit (...
the ice bath until the nixture is chilled. sold 1/3 of the whigged crean into
simple torns its for adding custon
headers inscesd of headers provi ded by
Overall
Overall Score=2
Overall
Overall Score
Score 11
Score =3
Score
... ...nive:alsy.for
This Bee Bav penay slot nzchine Free Games
...Kunning
a..ut high-maintenarce lettuce wraps here. Not the kind of High-nainttneee that would you away. You know I don't t like things to be complicated
...Wad,, September 19, 2018 10:51
more infiemation contacy me at wexignail.com or 095336403 i am looking forward heazing fron you. I zeally do approciate your kind gesture for reply inanting ne this opportunity.An a Nigarian. 32 years old of ha...
Festivel
Nychwood December
a.m. Wednesday, September 19, 2018 10:51 a.m. In
sar manuel casino
12/12/2019 to 01/0//2020 with
Rocus. Pind short
apactments, housea and rooma posted by Nassau
day 1000 ni1e/1000 kn 6 day with shorter races. a Runcing ieetivel kychwood June 66/36/2019 to 22/06/2019 with 6day/72 hour and shorter Races. is THERS ANY RUWNERS BRAVE ENOUGH ro...
this early morning Sept. 8, 2018
photo, 16-year-old
Paradise Island
dialysis patient Gliss Salgado prepares for his trip the Puerto Blean nainland, at his home in Vieques. Salgado Aerr
lardlords, Our
Strategy Games include and 747 more.
Just the kind of aigh-naintenace that cakes sonetàing that needs a.
Trailways offers lexury notorcoach
trans...
41

---

Published as a conference paper at ICLR 2025
Table 28: Raw training examples selected to have 14 quality ratings from to 5 within CommonCrawl.
Accuracy I ... last Friday and he angry!! Subaerize. oz, least they WILL 130-story treehouse once they finish building all Sinow
Accuracy =2
Accuracy +3
Accurncy -
Accuracy =5
-.. The Piral
...in the porest Light. dae the
ana Shiva
...ee big es a oa",
Solut iolution (Hiatory
presents in the repext The Putcze Daily Bread: Regeneraticn or Collapse new
big rounded pota of ctzyaanthamuns And had cascading
Basay Samplet /
1iat below to get information en najers, minors, and the depertments and programa that adninister then. Black incledas African Americen, Rispanic includes Latino, and Facific Islander includes Netive lawelian, which
samples Histozy
The zinal Solution
" The Mexican
chryeanthenung
War The Irish
evidence on the immi nent collapae food systems if contiaue on the path of industrial egrien ture.
coning down off the kall, and then there's wall that leads down into our big reflecting pool that terninates And he would grow these cascading chrys...
Republiod loon Arny
then itra coascant leazning and experiencing through mistakes and growth, all provide our customers with the be
Check Out The Zinal Solction Bocay More then Eiity yeazs ago, over four year periad noze than eLeven nillion people...
2303698567 (066686666666611680 (11////666616aeaeaaaaaaaaa
10 demands for an en...
Coherrnce-
Coherenge -
Ccherence= 3
Coherence
Cohereace
folf of insightful information and entertaining descriptions. Your point of view ia the best amorg many. hpornvideo Rey, thanks for the blog Much thanks again. Auesome.
*.. will ourselves happlly
cr.1tion, the threat of à mass dano and very
... The goods or white Bet on chzeme MBzwag's Caroma paint has becone the benchmark in the narket thanks to its tectnology edge,
...Moouaid holds off Fenfield with deferse in Class AR baseball semitinal
jump on the it screws over the othez group. Itta fairly easy 20 hav Js then nentality can be dest ive to both h curselves and socisty. It is striking think hox susceptible we are to this nertality....
political
row. In the sane year Braddock was made Dane far her services
JunicE pitcher
Huntez Kaish nearly pizched a complete during the Knights' vin ier Pield. MeQuaid id holda off Fentield with deferse class AA baseball semitinal Junier..."
in the wake of the row over the veila Braddock stated that she asgrosching 60 and had always planned to retire. So St...
hov did it come about? A look behind the aceres of the devel department shows that invention has 8 let to d...
poundest2 I think this real
great post.
looking forward to
read
Creainily)]
Creativiy= ... China,, Bargladesh to
Crealnt
4
Cralbiiy
...29, 1992 bid not seek slect ion.
... Tvitter
... HomeOpara Rector Berlios, La Damnation
...uuut Sanoa:
Chat SMT Experta
The Goldenith's Munerous
KruegerRcbert C.
build overland crada routa? By
Pacebook
Krueger (D-7X) 19930121Jen 21, 1993 No, defeaced Junuary 5, 1993. Frahnchell la Prahn (R-K31 199506110un 11, 1995 Ko, defeased for nonination. fha D. Cha fee (R-RT] 19991102Hov 2, 1999 Yes, o...
Really Blame
Motsopolitan
tales and legends exist in Zzgreb and about Zegreb;
Patrick Scally in
for Increasing
Opera, November 7, 2000 Novenber 10, 2008Michael Miller Piled UrderOpera Susan Grahan as Mazquerita in LaSusan Grahan Marguerite tor'ioz's
News on Uctober 19, 2012 Ongoing
Political Division?
Andrew Hutchinson Badhutchinnsn There is a lot to take in from the latest New York Timzs' latest report an internal neno sent by Facebook's s head of VR and...
efforts
remenbered through the ages, ether sink into eblivicn, and even the origin of some is forgotzen over inne August Banoa' a The Coldsmith's 's Ire..."
Surming to markets in India and central Aeia cook apparent
detour yesterday in Bei Beijing. China and Bangladeah contatively
Rannat ion de Pauat. Photo Ken Boward/M...
agre...
Grammatical
Grammatical Diversity
Grammatical
Grammatical
Grammatical
Diversity 1
Diversity =3
Diversity 4
Diversity =
34.04 (18)
...Fuel Krievel
...with
...R. had knows her socially before
Annio fluid
03/20 03/27
1971 Mar 2019.
jihadist Islam. Sterestyping can
calla prevent development of ascites in neonstal rat model of necrotising enterocelitis
02/13 03/2C (15)
BCSTORY and Nitzo
wa went to Nozth Males. He had been letting me spend 16 monche my life working on this scheme, and but Iding chances of nyseld. seceived El, C03 from H. because I had been more than 600 wronge...
03/05 03/13 (15)
circus announce
have very dize consequences) just sek Pastor Kefonnell, the out spoken clerio who is retiring from North Balfast's Whitewell
02/28 03/06 (20)
the seturn of the
02/2: 02/26
(20)
Live television
02/14
Rvel Tave premiering Sunday, July 7 at 8PM ET,
02/07 02/11 (20)
Aim: it has been denorstrated that in neonatal model of necrotizing en:erocelitis (NEC), ammiotic fluid sten (APS) cal's decrease..."
01/3: 02/07 (19)
01/24 01/31 (19)
Rird high-quality
01/17
Svel Enievel
01/10 01/13 (15)
stock photos and editorial news
01/03- 01/1C (19)
Netropolitan Tabernacle, He branded aslan
12/27 01/ 12/27
pictures from Getty
01/ 2/227 C17...
Inages. Dox...
...
Knonledge Novelty =1
Knowlelge Novely =2
Knonlegee
Knowledge Novely =4
Knowlelge Novelty
Novelty =3
..akoncy Censtense Baker
...counczy, and shcever xas coming was the buyer.
Jesss?
...-2aditoon step (i.e., the pozertial initial add ion of the Regina SK 54R 2x8
...Shirley
iClassic Fack)
Hactarlane 461 Broad St reet N.
Mey Esther
CoLD STORAGE
Peterson dearnette
smake a nove now
Opera: ique
Rankin Ellen
shait, hide and observe Shark
(Clasaic Fack)
25 Things You May Not Know About Saskatchexan Vol.
nucieopaile to the of sis-electrophile) has to be slover than the incerno add the nucieoshile to the catalytically generated 2-a-lyipalladiun complex, it to be...
Sxallow Richards
Takkyu Ishino
Elaine Reulet
night be another super, Set' is not do anything resh
Jingle WIRE05
Ratherine siva
(HIRE05 Pack (JF))
12 issue By
Saubel Madan C,
Kerra Ishihare
Lorne incor
J. Malker Paye
now. Tend be nice the buyer
Breaking the
Daring the widd Devenian Period, between 375 and 403 million years ago, much of saskatobexan covered b...
Matcleton Roaalyn
(Continue Pack (JBII Akira Ishihara open the P.A. (continue Pack [JP]I oblivicn Records
s. Yalox
coo. From last
Yerkovich Bella
thread Yoo hear
Abzug Syra Bradwell Annie Jump cannor.
the one about the shark-man the d...
Jane Cuaninghan...
42

---

Published as a conference paper at ICLR 2025
Table 29: Raw training examples selected to have 14 quality ratings from to 5 within CommonCrawl.
Language
Language Consislency =2
Language
Language
Lauguage Comistency =
Consislency =1
Cousislency =
Consistency =4
Meixin,
r..Figh= side, Ive
....a Land,
-..ha had adnicted
....eta-aaca
xeixin, weixin,
long
Koevatcnturs, up north, where the
it, though
Lung Cancer
weixin,
that 1 know to use my hands better, said
could still
47:61-03, 2005.
we weixin, weixin.
Bock family lived 200 From 1050 to had bet ne.
pretend, but he
37. the PURE Meta-analysia
we ixiin
wel xin,
we weixin,
wolkin,
Schwarta, whe
1248. 9hh Stone
That evening we greated differently than usual, 1 felt Frozen, manipulated, wanted be slone, dida't wanz to know anybody anymore. After 15 days, the longest interval be...
Group:
wesxin, welxin,
playad nine at right tackie for Caroline in 2009 and 2010. 1 dont think about it.
"Lnage" of Neah''s
Fastopara:iva
xeixin, weixin,
Ack
radiotherapy in
weixin, weixin,
THE ICE-AGE THE ATLANTIS-T19E Uden's land, Atlantis Hel, isfe capical cut off from the other world 10 tropical
noa-emall-cell
weixin, weixin,
lung cancer: Systenatic reviev and meta-onalysie of individual pat ent data fron nine randonised controlled trials. Laacet 252:2... Originality =5 ...65 the 1y2decans,124,13
weixin, weixin,
weixin, weixin, we weixin, weixin, we ixiin, wel xin google, googie. google info...
You just gozta go play..ra DY Cul len Jerkins (ealf) and 1B Jacquian Wil...
Originality 1
Originality- =2
Originality- 3
Originaiity =4 HameSullt
(5) Tan 26 (6)
...5earch By
...sone tall Into year? Yes, juat think 19ss than 22, If i2 the Fed policies and the way, BUB starte to tone it, the Bank of England, at cetera. Mell, you talk about the Fed coming off boil. it looks like at least are going to slow
27 (5) Can 28
office Search
SpacesKllokheis
(2) Jan
[4] Can
by Office Baton
The lost city of
193a (Discoidia Domain
30 (2) Tan 31 (2)
Rouge Dailas/Fort
Delhi By Arya
Feb 01 (5) Feb 02
Horth Fort Worth
Sethi 15 august the
Receptorst, 149 and intogrins..C,
Feb
(4) Feb
Golfgers Eouston
2021
04
(2)
Jackson London
list of cities of belhi, this city does find its place, as its ruins not gurvived the revages of time, und like the others. While ve have works avai....
The latter are noncovalently
Peb 06 (3) Feb 88
Mobi New Or leans Releigh Tampa
(6) Feb 09 [4] Peb
linked haterodimers and ene subunits. integrins have a short eytopl aplassiic donain (except for the 190....
10 (3) Teb 11 (5)
Tupalo Seazch By Adnissions Bearch
12 (3) Feb
(9)
ay Admissions
16 (2) reb 17 (5)
Alaisama Arkaneas
Peb 18 (6) Peb Teb
Call forn's Diatsie: Court for the Borth...
18 (6) Feb Peb 18
Profesiinaliam
Profesdonallmm =2 431 In addit MSCs are known cransciffeeention into neuronal and glial cells. MSCs been shown to n' grate to danaged neuronal tissues and to alleviate the definits expar mental aninal nodela of cerebral ischenia (10), spinal cord ins...
Professionalim =3
Profesionalism 4
Professooallomm =
clupp of smoke,
Travel and Tourier
Sobral, Cears Sobral
snoke descends from above the burning road to get closer to the fat boy so despices, Her enher eyes burn into ais fat boy soul, Helter agony. He waddlea backwarda toward that side of the house where the
Alabans a 21-14 lead on
Taings To Do In Slovania Poz an otharworidly Experienze .oo about trip the green Reaven on Barth? by
morisipality in the state of Ceass,
Chio State with
Brazil. city waterfront Princess korte Whorth's Princess) Sohral cada vez melhor Location in the state Ceard and Brazil Location Brazil coordinataa: 03-60268 40"14209 / 3.67...
9:30 left in the
second
Alabama is -700 en the 1ive 1ine
lahio State +475), spread -13,, total 84s. 6:09 p.m.: Alabama fumbles, and Ohic State makes the ride pay. Teaque scores...
Yanns N. Nikey (CD0), sixaeey If you' alek and sired of only doing the same old touristy stuff, think about visit...
Semantic Demsily 1
Semamiic Donsily =
Semantic
Semanlic Densily =4
Semantic Deusity =5
Density =3
...Scioct
...a meaninggul dialoque among xll the interested perties. Rut infortunatell this was not to happan... or course, a meaningful only
...lee cannos always rely upon the viedon of a leader, nentor,
... .i.ii Buggy
... ...evision Notes for the Paris Peace Treaties Euring the peace process, the Reatern govaramenta had a chance to reflect on what wrong and desige peace setslement that would
postFeke Saint
and he was like
Draee Saint 2rake
Fetey on the back
Saint 3Pake Saint the Year 5aae Saint
and, Rot he up
4Fake Saint
Sone: truth escapes these who serve, but consultation with attorney useful if you have access to one. Much of this naterial is developed to help these who" ve been denied leg...
ny the laat queation. JT harely beat hin out, But yeah he is anazing, 1 did t'inkk he was gonna be battling for the Bookie scoring lezd. I 2150 thougat ha would be m...
the Year 6 iFazt 1)zake Saint of the Year (Part 2irake Year Taae Saint the Year Fake Saint of the Year 9Fake Saint of the Year 18Fa...
dialogue take place if the parties naintain abfectivity. You don't eare marks for bredibility when you give credance ...
had
Saint
restore stability
and confidence
Buroрann
leadership. I.
The E...
Sensilivity =1
2
Sensitivity =3
Semilivity =4 ...go vitness Lily and James' murder and the fie dowan't deal with then
Se
5..30s Daily
...is to say looking at lying-ass dogs
.n.on bringing water bostle so you can sip it
...nnnan Rights in Econonic Policy Ruman Rights in Sustainable Development Rights Claining and Accountahl lity cocountabllity is corneratone the human rights franewock. It has both a corrective and preventatiiv funetion. addresses individual and
2618-11-21 23:31
https://pmmoddeev
50% naily
lkke sitting where
throughout the Energy arinks are popalar because of their
2013-11-21 23:31
sitting and telling hin I'1 kinds of crap about being reformed, finding religion, getting an honest-to-God job, and settling down. No more neth, boaze, cooze, brawling, and knockinn the
hrtga://gmmodineaa
agenta should have good resson for doing it or be willing face ceprimand from a Flower. Hoze answers by Bilipsis Plood
50% naily
2013-11-21 23:31
branding and
hetpa://smmoiieea
association
httga://mmodaaawa
with sports and physical stanina. In addition, when your museles have energy, givea you the ability to wozk
hstps://ummdinne.
hatps://pmmodinev
https://pmoodnnews
httpa://pmmdinnwa
2012-01-22 18:34:00 VTC Link to this !) Kell,
attp::/smmoddnaaa
hatpa://jmmodieee .xnockkk
43

---

Published as a conference paper at ICLR 2025
Table 30: Raw training examples selected to have 14 quality ratings from 1 to 5 within CommonCrawl.
Structural
Struetural
Structural Standardiration
Structural
Struetural Stundordizatinn
Stundardization 1
Stundardization 2
Standardization
ing i2:
... Trump' 5
mi mi la Uvetkovion Andy Hall Sarbia
into the realn of a Super Sa Saiyan. It'a fair to say,
... The Potential of Tropi ca Agre-Induat Industrial By-Buatatta a Funczional Fesd for Foultry Document Type : Review Articles Bugtharto T. Yudiarti Iszoll E. Midiastuti Department of Animal Science, Faculty Animal and Agricultural Science, Dipcnego...
fawniing ng compl fance with 2ioniam, and the atrocities
crackdown on free press in Alabana
and it nei ghioorrrs
step in the zight disection' President
Rosovo and Albania have descended into bitter diplonatic tuckus after officials in Belgrade Invoked slur... Albania, Greece Must Reflest on Past Mistakes Settle ineitine Ta... By Akes
without
Nanek Frieza saga making little boys and girls tune in after school, that anine night
committed by that gacist, genocidal and inssne protect Terael. then lose bys Zioniam is pretty nuch Christisnity
Dorald Trung's admin tat fon working to seatrict the
been auch huge hit here in the UK. 2) klingons star Trek Riker dines ingon....
press in states around the country effort to keep country's reputation
without the
all. The not fon of discounting phys.....
cransparency,
Style
Style Consistency =3
Style Consistency ...e, it energed that nonrespenaes and absences cid not reselt in disciplinary sanct lons. other factors account ware the autenonous organitation as regards tine and space, the absence of exclusivity clauses and nonconpete agreenents.
Condistncc -1
Consistency ...with a
Conditenccy ...ductory esseys, (shrink) Review of Aristotle Desire.
...5 becenber
...time had cone to take conron But the
2014 Rovember
vehicle at the
2014 October 2014
intersection Grand Forest He. Anderson 5aw a dack, heoded
September August 2014 July 2014 .une 2014 May
Israeli armad forces would not tolerste what
[REVISH] Theraton Lockwood - 2013 Pryn Maur haview The
2014 April 2014
figure reach
they vieued inaubordins They turned their weapens on the pretestors, kiiling six outright, injuring dozens and acresting hundreds who persisted in thei...
March 2014 Pebruary 2014 Canuary 2014
through
window, grab small parcel and run nerth on After colliding with a vehicls at the intersection of Grand and Forest, Na.
Decenbez 2013
imags of a copy of Fraxitzles' Aphrodite-nude demurely shielding her pubic regien--which adorns the dus...
Nevenbes 2013
October 2013
Septenber 2013
August 2013 July 2013 .une 2013 May 2013 April...
C...
Topic
Topie Facus =2
Topic Focns 3
Topie
Topie Facms -5
Focus 1
Feeus 4 Pr.Pr, Ootober 2020, 12:59 FM Two priests are going trial the Vazican's criminal tribunal this week, one accused of sexual Ty abuaing an altar boy who served at papal Masses in St. Peter's Bagilica, and the other secused of covering up, T...
Scre partlsan zayulu a'
...CFF nst, protested under the conssitution. grant I might have misstated when said rulsd. but the fact rensing that the court agreed with As stated, the only person who scande to gain from pursuing th's lawsuit Mr. Barko...
...Eng_oy staz, you could
and small secoads. This
pas un péché, dat
arque, is as proscriptive relicigus observation.
géneral Kasonga policiers qui ont matraqué Serge welo, an des généraux du dent alv Arrestation
coniniscent fake a jet take Luxucy watches engine, is ence sed in complex reaturing alternating polished and eatia-finished opt
They arrive with expectation and nandate, and any deviation fron the ascreted
that
Ngoys Mulunda
zélix Rabange
screen history is ganble. We drawn them they recognizable Kayza the mo...
mérite le néme
selon
firishes. 3oth the orange accenta on the dasl a...
Jean-Claude Racerde
Union sacréo
Overall
Overall
Overall Score 3
Overall Score 4
Overall
Score
Score=2
Score
... less. Seatt King, Weshington, US, 98101, 1363! 754-.... Haike, Fewless. Phoenix, Maricops, Arizone, us, 85255, (480) 375-.... Carlera, Pevlesa. Kissimse, Oscaola, Florida, US, 30741, (107) 742-.... Jason, reviess. Pateros, Ckarogan, Mashington, U...
...Steroids for sale new zealand, what is taking Staroids for sale new zealand, what is taking sarns Ruy anabol is online Steroida for sale new zealand six spocts supplements
...BEACH BOY
...stands at the door, with the reat of the Company and the behind hin. Meon Shadow is so happy to see then, he hugs Oncle. Unote talla hew the Company heard ot the Lees' trouble and showed help with
... hunan rights. à Taudry Diace of Salvation: The Art Jane Boxles Edited by Jeanis skerl Southern
ARDASHIR VAXII
PDP Azdaahie
Vakil in Bonbay and now lives in Sondon. Ardashir Vakil's wward-winning first novel, Beach Boy (), is Bonbay's 5 answer to James. Sarrying
illinoss Universicy
Press, 1997 Library
Congresa
P83552.0837188
1997 I Devey
Decimal 010.5409
sale in New Sealand found to contain anabolic ste...
universel story (an adol adolescent bay'a off-ng-of-agel with speci...
through these
the flight bragonkinga. They are deter...
assays-which deal with Bovleafa publishes...
44

---

Published as a conference paper at ICLR 2025
Table 31: Raw training examples selected to have 14 quality ratings from I to 5 within Github.
Accuracy =2 ...Comeent
Accuracy
Accuracy
Accuracy
eiccion
... rarcokk
...pesvato VoLs.
... pecczge
sMi suge saggeasiiinn ions
supposta Lopie
1sdata2 new ArraySiet<String
W Final
ensell Lopla. colla.deaa.attii
eliminates
1zons</Lis
Class viserClans,
counkers import
unknowas
dataaa..
final long
akka. aaaasatremm.aaag.
nangulaids N
page tlink Fron vi viesppotilc and viavtopic
add Inttin Intens (h-
cnd bawnbelyy
import acke.strmem. Actributes,
N 865 125 X N 869 inclactiin
getstringsstrr (
Lpe
inaoeAdaptez
achni meliiaaa task (naw
FlowShape, Inlet,
treshines hops junkeleers
Inagsidaptor
sunaabla
Cutlet import
C requiment
(Lightlbeary..
public
isolstors resucing netheznoac
thic, satal;
1StmpleVUser
import
naybe disabled via ACPc/1li
callery. sellary.eethaacer
Ley vilser
csalaccellection ablutabll import
nonfletton fan
(inwgekiaptent)
(15 inSimplettase eVUser)
R 918 250 X 86 Zocbeazez
<ll>Corncats
Leacaladeress,
seals.ccllocttin
</1i> <t CiiComenta requiment naybe
(1i. getaddress Lastnane. sotToxt.........................
nww wewiinaaaionn antaaareee ()
immut abwe. Vec.eoobbb.b.de.
snesthetization
catch (Exception
import Boal...
underenntoneed Canking
dizabled via s/Ei/
el threx nex Rum: ins (e);
mereasstumbetiveeeeee (1 (1
Caberence
#4
... vGr spt type-Syrehsire Byrohsirska
.ig xI
1x4pseaalu lonze
...C5
P.Resei..getBIP11 (1, 2, magges//wt.ppp pag
optisnal, tesolutions optionall
abezeda
bqReaci:.
alt=S:ories on
TVBB)1>
est hatgeaggl /ngggi//////ptttt
1e1c1.511113;
Camper Nesas
assertPalze (POOERAN
<script type Byr5?>
(IIoxtVics)
<d19 class-inag
resolution:"
C3JECT
xahothos ixi jaki
niuyyaan
uverlay? <e
opLionel,
11P8..iSubbyype (NO
</scrigt> <soript
getCal Ldet
hret-httpi/ /todeppe/////e...//////
pesoluttion
CBJECT TYFR1);
typa-Syzn>
FindViowEyId
suresz-blnke View
optional,
assextFalse (B033EE
vehodmusiz jakk
id.v). sex
Pull Project c/ax
jpounti. axciesh
CSJRCT
</acrist> <aczipi
(beRwaul g.t.ar (1)
aobyld (d. ionppiiine wtt..l.................eee (ARRAY
type-Tagh>
(iTextViex) nBuyCanrainer.
FindVi
www..ctoo..boocoood..... TYPE):;
tagaenski
<script typa-Taiu
ssolution::-e iogtiiona,,, 3ssertFalcc
</scripl> camr
beichillkt (1). b.
csoript LypesTaiu
react awectar.bloukkee bbluzhhundd CBJRCT
(twols
integration/ resolucion:-optionn
Creativity I1
Creathilyy =1
Cranivily
Creatiiity =4
Creativity =3
Abstract
... 1/ R0vezzide
impor Ise fron fa-eext ingort uglifyas fron
...id: 514 title:
... the Xind products of tountcous years</1>
beletior, Abst ract Delction, Abstract beletion, IsReadState
publ exit
Heppy New Year! date: 2016-01-077
(Listenar:
anglen 4s /Loola/
12:23:04 100:00
<i>il mote the date, with novy
void it dlistener,
index. conet
autaor: Jerei
ChanoesExist,
oeltoptianValue)
ninidy uelifyes
Glever lsyout:
bloons crown de/1> <lottt Bain preads
soteeddtaee changesExist, Finall csstate, RyMessage Deliverylime, ....
Lixterer.
function aylify
guid: hctps//
vettoptionValue (this) (useriptions)
.on//
her baletul tires around. s/ap>
edvezride public wonegt
aptiona
7p-514 sommalink:
eesiigg
/2018 /01/07/
caps kapea
scarcetts: true
aaspy-nav-geard
BolidtolozBrush
al L2): Reaull 15 (visit...
aewropt iuna)
caleguriest General Rappy
return name:
CetColorsy
fugli
20161 Me 'ape you all cajoyad your hol...
TypeKame
transtornaundio...
[ImerrsiveC2......
Grammatical Divesity
Grammatical
Grammatical
Grammatiol
Grammatical
Diversity
Diversity
Diverslly
Diversity
...><//pan></1>>>>>>
...6p Factora
... lamersive
...Caaracteret
<li>-aaaan class
content: f1ler
Obrlene ElNunezo
Coloz.Ga .Gat
fsetBecin>
-Fleg-17 title-
ffa-archorsbeffrt
De Documentos
Cyle-RyTyeeKane
<CharacterOffsee
.Elag 7P by
content: f13d;
vil, true] SauullCCopyyle (1:
[Imers Ve
2864
</span></Li>
-o--nniock-alte wtt.aoooos-llttttttetttoteee dt;
Colo:Names.
c/ChacacterOttsetEnnn
<11><taba a
content: flle:
public DataTable
DarkChrone
<208>SN</EO
classe FTag-To
-fe-bull beye:befare
Get (int
Medium)) Opwal
<NER>O</NER>
2140;
LEmpreso,
WindoxcTacne.
</apsn><//1>>
wlppall felsfornent iluwzz))
PSR0</Speakee>>
<1i>>cegaan
content: 1141;
Current
</tocen>>
classe flag-Ia
-cs-ellipeis-pe Veb.pooo---bbtorraeee (atelable
1 1.0 7 public
<Loken 1d=24>
fitl .flg
content: ....
301.Doee...
Brush Not iflcst ion
Swords, </word> ...
</open>C....
Zoreground =>
Knowledge
Knowkdge
Knowlidge
Knowledg
Knomledge
Novelty
Novelty =2
Novelty =3
Novelty =4
Novelty =5
(feastal)
egeen-ngAvM
Lype-ko>1si-2 Sops-ko-isi-ooeall
tangisain cheela:
aesinsleTa
(fedate))
playhooks
</language>>
que escoreena
pereodraz-qip-
2ddapase (solf.
The varianles
<lanquage type- kuskurdiat:
jezecurche
Skal Lenes
May pro pxo//et// suaton
ne, icaacuruha
eigFormesveedd
(tedsta))
stutt/ playbooks/
</Language>
chaciai que
lociistooi
self.croatotrbee
ansible.cfo"
<lanquage
Comajelanealave
sricstpri grp 4167>
(ferew) def
would use
typ= h/ggsss/yyyyyyseegggee..
guine. s/pece?
Cheawg taen lvittl
cswate Jawle," //apeaatacc anwibl redsta):
Lype=la>
</listiten>
Sxpe-Seneitiv??
<listitons
eaforns:ing ref=/Sk jena/B....
an The openstak it trequency: dep deploy/env.s" dicectozy ...
typo-1n7
quare>llhe chu chaen, cacheauru sal rasli,, tengierasurd ereauru
fedetie get (frequenny)
<iangusge
return
type=1o-
st lfrequency /
nonac joreniha ne,
1000) return...
45

---

Published as a conference paper at ICLR 2025
Table 32: Raw training examples selected to have 14 quality ratings from 1 to 5 within Github.
Language
Language
Consistncy =1
Consistecc =
Consisteccy =3
Consistency =
Comistency =
jucesor maa
adiplacing
exepeckaeg
<value>con.gg
Necos Thena
clis. Intoger
controller; /+-
Consig pretix Zoz leal
Ia- Pa-paaa- fuEexoza.
Monacal</a> </h2>
posusru esat
Created by tangxhiging on
wahot to honec. implementation.
<div classebl pok
s/po
queue template
style-marcia: D>
</b-cckguote>>
2016/8/31. +/
condiquestions
mithra,
Liebe pige-Uler
<blockquote>
import android.app.
* #erivate
BeaclBruya 1iV
und Userinner,
<p>lozen ipsum
Applica:ion;
public String
und Userinner and Uscrimenner /s diesen Monas erhalt uneer bel lebtes Thene des Honats ciner. wige...
dolor wit anet,
ingort ardroid, Contooxt
get Autteetett
ritilbrub<//on /valuc>
edgueucTempl atoCo
<vel wellueoonn.gg.
edipi suiigg
ingurt android,
offzefix
tx-pors. wersstutexoee
Integer posuere
content. Content.Incente. Incentz inport jave.text .JetoFornst import Java. Lext, Simpietatefornat/ Import sava. M.i..Asr... Originality
queueRathi retorn
xubot lou homee. implemetstion.
erat ante. </p> smmsi>ssooooe Canous in soise
quenePath nOT
ADTO CREATED LEAF QUEUE TEXPLATE PRETIX; SPrivate
nichra.
SoGabeWooluss
sit-e-Souree Tit...
public 5...
Originality 11
Originality =2
Originalit-
Originality =5 ...// returned. cheracter set coliniterSet is used as word doliniters. // if the rege iver
...ap.RPS:10151>
...Ibelia
... <tane-Daily
RP61015: 5%,
Cordition ion
Minimum
x.Trin(' (f1),, Microsctt.Tee
<policySe Recerence
y</aber></a>
<vs
usw.uus3///// 3</va lue>
olata, Servi ces, GataaM:SSrrvcce
xet=urn:ntas:
y</aber></>>
wraiuo-13</vaaus
:05:1
yc/abbr><//>
weilue)33</vel uee
Ge Aocoo Info,
:manager>PP510111.co
wra_ue>13<//p
truel; f3/ There MnCieshDeaaullP
</pollcy8at
ys/abbe></a>>
<vaiua>75</vall
empty // wEiGs comdl iunsz // it the reseiver is ail, n=l is returned IssString cest GfNordalsing DolinitersF (NEC...
ys/abbra</e>
yc/alb>></a>> P
cvalue>71 2is/value>
</td> <ta> Kanage
<ys_ue>/3<//we lue>
in the schens,
10151ianploye>>
how Conditior.
<vaiu>>87</ (ess)777/vaaa
131 </surnarys
PP51015 .5.mm
could interact
</ </twnger aree </peranctors>
publ in glabelst
s/polley@tRetssreeeeee with the curzent
Microscft.O...
<policySetRe.eer..
application state (Thelia] </td>
Profesdonallim
Profesionaliom =1
Profesionallnn 3
Profesionalisan er trom base import Negi o import os class Boat (Nagiosiutool This class have
Profesdonalimm
.. 10EBBM
<div
then on chai ne.
v.. lesve.e 11, 01.0 asyne: reaune
classecol an 12x
The menory jave la
V2NgT EKayЕWFоΣX
<div class main
vero colezed
IL 0096 1darg.0
Ig www.ee31leeeeeeeeeoo JEGH
image <div
like zubisa and
IL D097: ldfld
classenn--ng innen Image
eneralds, pesrls
Eysten.Runt Runtine.
FSžn9y
Utamas/diy>
and anothysts, and ropes
three options howt.
comp Lerseerviis...
g6A10Eс1.R
lngs? Iagge
TaskAaelle
Q03NTONPIENB
ULanas/dive
than coloza
file in nagioa.
tustaaaed 0.
xgshtel 0.2M
classenn--ie inge "mage
to suit every
You can speci fy the templace you
TI 009e: et loo.2
Utamas/div>
outfit. sont ot
IL posd: idarg.9
P10921
classeon Lnga? Eaage
the alighta aeze Comenche's, e, put besause Conancha wanted
need. if you create a lots at bast file at one time, tais Ifecian.....
IL D0 ldrlda
yBSUVE;RE9E YM
Ur Urama</div> <diy class=kck-ings>
Systen, Runtinennnnn...
luifih bG1KYXR
p=24g02VJJKK11EN
fsiunda with Kathy she cid...
Semantic
Semanile
Semantlc
Semantic
Semanik:
Density 1
Demity =2
Densily =3
Density =4
Density 5
Span> copar
spana
<asV class-
.x.langage
1d=1792-1192</apaa
</td> <cd> <p>A82
Lightooldenrodre
LinkAEAD-<a hzef-
<languaça typent1>
1. DP 1, TPP 3, HP
is dl
joongonert ht
</languege>
1:8-179011999</ppmn
1</p> </te> <td>
sty_e= rackground:
IComponent
<language type-tin> </language>
1F9 2</p>
Lightrreys? <apan>
Class</a>
16-175001796</apan>
<p>ANC 2, 12B 2</p>
<div
<lanquaga Eypontli>
Cagien
<poANIC 2, 190 2</p>
<ii styles
dlexselirkAlBAAn
</Languege)
1d-179581 /954/sppn>>
<p>ANC 2, 1F9 2</p>
background:
bret-text tettttext.tti>>>>>
<languags type-tmb>>
<apan
<p>AlI 2, 138 2</p>
Liahtdreenpss span>
Using Text
</lanquäqo>
1d=f19661196<//ppnn 7966/span>
<gAANC 2, 1Yp 2</p>
Lightrreens//gamn///
(aaes////ooaappae Nempomenssc/ax///
<language
<span
<PAN 2, IFP 2</p>
<11 style
<div
type tap=-taix<aammuaae
1d-1/9/81/9911 19/////99///pp/
clees=1irtBHEAD-G
<language
1d-1793>...
href-ccheraltest.co
SSendiivity =1
Seniiifitt
Sencifiiity
Sensifiilyy
Sandibiit
...955>aadd</main /ea.ai/d///aaeeebb.....
...akilssocaa ski shote,
hic olnazsa
crai neeeeeeent
<score32, <
arylary senden
regular, argular.ent!!: 1,
bcgia-1936
<orabled>
ski schot, Milllhh
joiniioinnibs
mmd=1142>>Bom Inouyw
Ywat/enabled>
yok yadyrap
'diat/ Iunta/
</nsinReterent>
</gane> <game
ski skirmiches
veriyoriar. decin, yadgrag
glyphicons
triples!
ski
fng reguler.avg")
annotator gold
incex" imagens
skpoticism, akept iuism,
verdikgo, benin halin dala hop oluyor. By Hawrast Ben benin cskeidin aykkat azylargay bolver, sen tat yenek arzusuna kendindea
pata. jointes.
s*Type=REPORT
Adbsesiptionn jong
Edist/fnnte
mainkeferent
Triple Hers
5i sughterd, www.staaaggeeree
glyphicoons-h
Generisity=
(Japan) s/description
NON GENERIC
sel ocloneaf /s
al ipper iws, al ippppra, smarpthone, snartphenes, Lipanee sna rhanen,, anadkichos diiichos, sandwi....
egek
peth. join
labitua ity IS0DIC 1130111000133/rroo
'dist/oants/
naiinee-bA teinVe--AAppettna
<manufacturer>
glyphioons--
class "DSWII/>>>
Michioutsu
hal ),
s/nemffeclurecs
annot...
46

---

Published as a conference paper at ICLR 2025
Table 33: Raw training examples selected to have 14 quality ratings from 1 to 5 within Github.
Structural
Streeuraa
Structural Stamdandization =
Structral Standardization
Structural Standardization
Standardization
Standardizatonn #2
...B
years
..-pera> Spectries naw velue fcc cantzol's bound property obtained In the <see tre De::::eellsppress Xizalipporaa UE MRConErooL walasteBinding/>
</cmes/ed >ecd>
Busty Milts Got Merel leaaly
ago. The daca on thesw alide labels are invelisble
id calinfaa
<Ldd <Lud>
Lwe ammenio
</ts></tr> derestd>
Pucked By 2 tvil
va -o
<te><tt>> <a
Teensas Boys><h3>2
they can help us to understann rec how bur unvizorment. and ilinate have changed,
id=calinfras
hrefeitparlanin
B1 Blacknal 1 feed
the rament
-Feat Hood-
Buaky Mills Got
value walue=KOK>st
Mood</a>
Mecol lessly Focked
lintra
</tt><t>><Ind
BviL Teanage
</Lt>]
Boyws/h /h
rense have saitted, and also tell La the geologicel hi story of the woma Le.r
vi
handler </para> </sumarys>
hret-itpariamint
</aivs
idngalintract
-feat Seat-Hunber.Dor htmi>
atvlo-baight:
ercamento2
<value>A sase
34pep3 <div
ve ce
ore Sryyat en..sttt.t.t...//// specifyl...
Sings/LL21 Ga
classesaling ing? <ing alt-Rating 9zomi...
classegl mlgglyihio cepphees--eenove remove test -ak--danee></1>>
hret- itpazlamint
-faat-Parsen....
Stvk
Styl
Consitency
Convistecc
Consisenncy
Consitency
Consistency
1..8 mentas:
...23eva Tableccoopensvetttooo 1:
et then
...1] D7 7+ the 0 aiready in Lie string o/ ceHenopy (cataptr, strine]i],,
instelineats,
</dop> <dep
iafs.csetasroggia))
...</p> </divee!-
instals: installs, tуpиеpeир?
brwak; Gaae
courlay-untlen ->
instil: instilll
<governor
2: into,
</a>ci- overlay
instils: instills,
-> </diy> <div
inst ional lest lontedeeenienn
break; info.
classeiten lasts
plocelIengl wlaw (phoos
institat ional fsatloa,
<figurebciing
</dep> <dep
masterSotebook D) break) case info.
src-https://
institational
typerag> <governoz
anteraethestade
CRASSERT (plocalSon
inst Lonal iue
idix Sommnieee
githuu. iu/ ing
CRASSE (plocca
institationalli
<governar
monterdarrints
postt-inages
CRASSERT Splocallen
mnttitational inst tha fonal iseds
2aa-225mmmbbzz
breakj case
/as:erscuate, altes </fig...
1pq
CRASSER7 (plocallen....
egevernor
5: info,
inst ional Find,
mentwrCecrinhe
Topie
Topic
Topic
Topic
F iarea CWILiME
Fecus
... 20) 01) Ita Eacuizoage +2 mod Fic aquire re Requirename ago
... (00
X, y.
(moynihan /128. (pa,
dix class-fa
<lanruago typosia>
e.qPr0r21
fa behance square
inter
)1: p-sh
/citaces/
aria-hidser crrpodserrcupp/////
odystatoanimationn
faces/Lwitter?
roll Eaqulzwage modit. 7 nodi 610 bea/bctzon> </div> saivo <label acolee! displays inline-b ook;
<li>s classeta
<language
derplleege/ 128- 1pg.
fa- bitbuckes
type=ibs>iaen
Twoilanded,
<lisel tobee//ruus</>>>>
aria-hidsen
EandStato:
ALlCacee/
clangaage
Sialess
faces/twister?
fa-bichucket
typeeibas bolbboo//anggeeee
2odyštates Knee lineling..
seol-hirvi.
<liscl claes=fa
<language type-ids
128.1pq,
viden:130pxx''
fo-bitbucket
indonesieno
BodyStats::
Biast <abbl>
clisei classela
</language>
II x, y
/uitaces//
<imput style
ta-pitauccet
<language type-ies
bigetAnimationn
facas/twiiter/
or disppay :imliney'
<li>i classefa
interlinque
(dataAD, daraOA,
wws twwleestankinn
fa bitbucket
</langusge> <la...
128. 1p8.
<lise classete
ta-pitaucxet...
Overall Soore
Overall Score 2
Overall
Oveall Score
Ovesll
1
Score
Score
26, 31, 35,
SpPPudd pesaoa
... </d.0s sdly
39, 40, 41, 42. 43,
0 </pactezn>
ton drêto di entri
ciass=cantents <h2
Bio. Awg.zithha Aliganant.
45. 46, 47, 48
</dat=tinneoreet
fungon pihl lee sé téra,ó paia.s/p> </1i> spVortedd pobo quel
class=cant ent head
Mult plle@egueneeed iggnent
</dateTine
is-cmater>
System;
58,
Pornatlangtho
show
System.
65,
coding 8774//25
Generiez
66,
55,
66,
67,
66,
Foumsticnngth
<div class puro-g
using 2i07 using
69. 56, 71,
75,
Lygenahort>>
Lickroe Stozidadi di poder di públ onz à debe mostrdao-el cs eleignn seziu qui réto,
anchie ideeventa
blo. Alguri
74, 75. 75, 77, 78,
edateTline oomat>>
<div class-l-ban
Aliqunent:
79, 60, 81, 82,
83,
spattern
pure-n-1
usi ng SH Bio.
86, 85, 86, 87,
96,
pratt-provicional1
pure-u-nd-1-2 1g <h3
89. 90, 91, 92, 93,
I D s/patLmen>
96...
</ostetine
II! <sumasry> /...
Format> </dateTina lengthy...
47

---

Published as a conference paper at ICLR 2025
Table 34: Raw training examples selected to have 14 quality ratings from to 5 within StackExchange.
Aecurac =1
Accuracy =2
Accuracy m3
Accuracy #4
Accuracy =8
...0: Row can
...Gr Now write jbjjctt
... AcDess
2..3.0.0,377.11
4URM0nelcs
the original
violation inaida
Ln
30pmAoC182-4mPv
array
resdable
ctype imported
c34F98ie100
shuffled array?
want to write
windows Funct Fon
/envs/myenu//
VIoC-mPin2 J048y
Thia ia the Pronpi: This i5 what
abjact in huran
Rt RuIDe:ive lve
Lohhon3.7/
5DF5 Babrtu --NIND
readable torn
Capability
site-gackages
CRRT1PTCATE
wrote: ingort
in text file, the file
Sidsl contann
(from app) 17.11
---2631H
gava.util. Scanery
importing
Requirenent
CERTIFICATE---
public class ContegianCent nol public static void args) Bysten.out.
saved as nornal serialisn object
ntdll.dil
siresdy patistied: python-jose
MITCCMOCAagg
Rt Ine-ive
AwIangIQoHivus
with unwanted
CapabilitySide
[cryptogeapty]
XK-3LF 1-gb
characters instead. Box do I zewrite the pregram saving
Prontiame sunction and having accesa violation inside can't
<9,0,0,5-3.2
cetten.out.prnntee printiat Picase Enter
In /bene/ubuncu/
1825x ogYDVggSEw
anaconda3/erya
1ib/pythen..7
fiqure what an doing wronq- inport vin32fi....
site site-packagia
(fron app) (3....'
Coherence =1
Coherence =1
Coherence= 3
Coherence 4
Coherence 5
7579
in
...2722
...0: Why i this excor
... jave.utill Arraytiist;; impost
1525736201
ync26Lmgei:cDlor
DEBUGI DD9U]
1525736229
ge6xplin30
Domain: 2
on JSF page? I an nexble in jat. I have a maven project and webasheze use 102 and richfaces. 1 an getting this exres:
1525736426
SEurlbattoons io
18:56:23.1
Inpost. static android.
1525726763
list modelisty
(93147530) JUBER
1525737360
ap-settings-tine->
DEBUGI [14]]
Manifest rest.pemmietion.
1525736174
1421309968;
DEBUG/Enail:
READ CONTACTS;
1525726723
up-saving-
18::6:22..1
login screen that offers login
1525736207
post-S saved;
(831688561 08F3
1525736839
vordps 524356
DEBUG [14]]
vie snail/paszwerd
1525737132
Regllaader e Cookie:
DEBUGI Snail
Brror Parsing
/ public clase laginkot ivity
1525736460
up-settings
2: 2 18:56:23.1
vieuleee adtt
1525737437
deleteunde:11266.
(8320563811 USED
index, shial:
exlenda
1525737050 ....
DEBUGI [14]]
reaced 2]
Applorpattctivvity
DRNUGI Rxee Types
The elenent type
ingl erents sesdrrCalibse...
2 18:56:23.1
html
(0323555391 IUSER
DEBUGI [14] [DEBUC]
Creativity =1
Creativity =
Creativily =3
Creativity
Creativity =5
...0: Ecroz
...0: Can This
...C= How
use
... sepresen an evant that haggened in Paris
...., svegllandos una mattina da sogni agitati, si Lzavó Lrasfornato, nel lesto, in
instantiatiag
Reviov-Based
v) and notify
serviet class in
Git fook
conrect ly with
Relipse Buzopa?
Be Enforced by Gerrie? there makes pull-rsqueste and combined reviews tast safe in git? I that there couple related quastions, that have already been asked at github (Se0 a...
threada In java?
in 1250. Legend probably
First.java this
I'm naking 20 gane android
my first java
toal,
word event, but in any case it is a very at sange origin for a famcus nathemacical. Mathonatiod is the ast of giving the same nane to diffesent thi...
slass package
scudio and I've
insetto innondo. Riporava sulla schi ene, dora pone una corazza, sollevando i1 capo vedeva suo ventre arcuato, bruno diviso in tanti segment...
com; inport
been eitting on a probl en for a few days nov. I want ny thread that
javax.aervlet. Request
import
serviet,
fervietSxcaption
Gravity, ny player can jump. When the player done jump...
impart
aevax.
Cookie; import
jarax.sarviet. hetp....
Grammatical
Gramiatticl Diversity 2
Grammatical
Grammatical Diversity
Grammatical Diversity
Diversity
Diversity
..- 11.00.511851103713
20066800111522....
JPA Spring
I.. 212 -2.360e406
38. 38.52939380595251
SitayitanDSvtk.
6.001953107322153366
ignores Lazy
1-90.5152219032225
0.0374915660901055
loading insida
3.246u-06 -2.
38.52006087299111,
"paper filed':
-.096118C2790067,
#Iranszcati
0.480 poly txt,
--90.57:605270.00
True, 'type': s306", "pages";
0.01533061269222244
have spring
caN 175
38.9224307263241,
-..17927377945522,
service class
8.537a405
1-90.590773122558,
B, "bascoda'
0.22853557133624,
where I'm leading a
1.150c+05 0.740
38.5055282054255,
vacEx262'
0,
JPA ab oblect (target)
0,468 poly
-990.663977772288
"Leansaction ion id';
0. 0.0923733533009221,
via CaDD, This
15, saw - 157
38553923377252241
-0
0L04000623204499435
target clase has
-2.184e405
1-0066652127639577
CarkaXP 54214'
0.8101552578422421,,
mee-to-nay may nappingg ng
2.880es05 -5.
38.6200698088575,
Faction datef:
0.0204079876558867,
that set lasy
0.458 poly 15,
1- 1-90.4351277034001..
222222
0-17",
0.0640597650633233,
loading.
would like to query this object insid...
raw 710 3. 80994404
ccttegory:: p...
4.910wi04 0.776
0.447 poly
15...
Knowlede
Knowledgy Novelt =1 ...0: Bitonicsort code
Knoelelge Novelty 33
Knowledge Novelty
Knowledge Novelty 5
Novelty
...21330ogga
...C: xinpu: setting not sistent during linux Hangaro Snone XLL, like to have special setting of mouse buttone, which 1 obtain with xinput. In order to make setting parsistent accoss sessions, I write the xinput in /xxpcofile
...0: Why do combinaterial
RECR ynk19/
while (sroMidth / 2 > desi redWidch) aseMldth /- erclleight
YOF1-ETáyj*:
segnentat ion
abst nact ians of geonetric jects behave ss This quest ion inspited by talk of June Ruh trom the recent Curren: Develepnente in Mathenatics conference. Baze are two examples of the kind...
running bitonic sort sequent fal code on waw.aaahinncceeeee.. It
#/4ghl gehittoo
inSempl esine - 2;
flest deriredScsic (fleat)
vigin2w100MM
runs tine for array size upto 16 cloments soon as incgease the aize 32 It gives the fol lowing error while executions lons MARNING: Process
/82nF3i Rwi Saneewo
desiredMideh
sxckidth) //
aKLLMMwGk
Decode with
inSampleSize
AnveWLAnaaaee
options.
quCSATTOw j/
induatDeeodeRounde talse; // now downl...
/DRdeR /122A..
48

---

Published as a conference paper at ICLR 2025
Table 35: Raw training examples selected to have quality ratings from to 5 within StackExchange.
Language Considracy
Languagy
Language
Language
Language Comistency
Consideacy =
Cunsidenry 3
Conostency
>-+QZ caanoda
2x Hov to get
6.6667 6266662718888122
Go ISTENT não
Lankss /optiinn
AppBar Text Color
dxidFane and
297755001433313
enccntirao
con value-
Azent: Changing
dolunn IDs on
aplicatiwo delpht
ase
can't change the
Mouse Bntered
1o1 ty
s/ppt fory
androsd
9928 66442
copt Nevix
(:3505:30441:00 67593,
java
teying
0..47633719022 90251725,
agee
587050559550291
havia Lono
the this.
androtd. content,
copt
previding
my apell lost
331410070889473,
Ivityler
you
connect ing
0.64:539066285255
PoundExcapp
understard.
objjects
0.8433775511221.
Lomble find exolici sillisity activity class...
pedro, saper coaarca pedzo.
the image: Sereenshot
auquance))
0.6777579555
could find any
-0.279...
searched
ay
Originality
Originality =1
Originaliy= 3
Origiallyy
Orignallyy
[0,d,=]
...0z Holumnayyuu
... Shacwpgint
x2))/
1-
Xemarin. found
Node
threw Unknown
pow14636657211
[o
trying
50% Faception
8896234
bug
Anyone
1:1237580.
Flxllayuut
diles
familler this?
3927:1878.16998..v
ressage.
tharePsint
-147053464
Fexiayouts
follovind the
16259850.
[o,
the ahould
File deleted
-1256.
(-35024-33223322222
13,
Folder Cleared!!!
attenptiot
088978038961688
(o, 5,33 Last, query
colunn containes
ACTUAL CUTPOT
aceass dsta
+101122266
and inner the container,
Folder Cleared!tt
through Contont
1966903359
File deleted The
Query Tool: 04/02/2010
4026224C062748
stays
acde
44334846754
ceneralization
enpty. filed Poleesinalsm
les funation
10:45:06.12
2- clico
claaCCvertedd))
vaNp.cxo (0x062C)
Professmmnliom =1
Profesionalics =1
Professonalism
Profesiionlism 3
...0: batveer.
...g:
[204.9).
1154
spacial arable
Konitoriag
fila:/accunnl//
(214.5),
205.
Charasters from.
controllezs
chdvisoz
accumalo11.44
(217.
[2C1.
input
Excal
Pronctheas
1ib/connos-jci--1
[218.
[159.
This netasd.
file that 1 load
bosker tried
(157.5',
194.
in tableView
inglenent
(178.5',
1144.51,
Algerithn is taken fran the database praceduro already
the date thie
nonitoring
-1.4,0/115/
(210.5).
211-
file
solution
11ine-0.9.54 9...4. 1oz
(194.5). 1194.5).
vericble data, this variable to another controller
opensaift
mlla:/accunale. accumalc
(194.5)
114..5).
bean
cluster based
-1.4.0/1157
(194.5' 114..5
blacklist.
Froretheas
wemmlee-stnppeene
(194.5 114..5.
Sparan nareTnArable
node-exporter
pomplee--snpplee se--s-ppe--11
(194.5,, (194.51,
in Arable of ieali
Cont rol wrtho to do zome stuff on is it retues mull
grafans cadvigor, I have huge probler with SAdvi conponent. I did let of confiqu...
have huge probles with cAdviaur compocent.
(194.5,, (194.5],
[194.5), 1194.51,
First
did lst of conficu...
Samaniic
Somantic
Semanti Dussity =3
Semantic
Semantic Dansiyy =5
Deneity =1
Densily =2
Demityy =4
<zd Gla6s"
actton allgar
.. 0..0.11444000
0i output
...0: Erroz when trying lssue
text-right
left classuuttayou
8.40761575499
of the call method uxing xostegger? I have created RyLoggez class and it pazzes paranatera to XCGLooger to output logs. have speci Fied true for the xciloqoer's a showE llealane and dateshevPuncti wnw.uuunnniinnnree...omm the righ...
text--nwrrap
EPadless
915644
dats nar lst statements with
coucton cl claxa htn
styles harder
13.1781459659
ban hin info
collagse seearaat;;
1C.205134367
bedit<</batis c/battonk
width:i00s;;
15.0854695726
myagh. jdbe. except
<bucton cless- btn
height: 5047
1.03013513987
1dbe4 wtype4.yygg_vvoo exirroctixapttion
bin-xa bta-warning
shite-apace:
6.46506592699
You have an ezzoz
sapan clasa-
coznal; > <div id-
#.27542399822
in your sgu syntax;
glyphicom
NDXA <sable
11.387166818
check the namell
glyphicom trwah
cw wellapacing ou cellpadding
5.34088296113 00782290755
that corewsponds to your HysgL server verslon for
></apan> </butien>
7,35750199406
id- "NDEA
11.8693581818
ct- 12 lsdata
10.8:57524873..
CarIDER 77:"1NNE"
zut it alkays outs...
clas
Seualiiitt 1
Nemstiilly =1
Sumiiilly +3
Semalivily 4
Semsiiiily =3
Tacddnii
...0: hoat does
2: Eupheriia
20356200
..7JBU57E20
probiena':
play the trunpet nean7
and Colloquialian
drteafte da:
NA C3 -C_03510888
". "dexcribe
titwrery saarry/ppeecc
707s 037b
0.29160742
problame's 6,
zeaest Asadenia
Deviows
xu: 3069 Procezz
0.220716441
pprobluna dater:
ss cuestion, oaer noonman239 krites:
is pls
Kournald loid:
0.25246176
". "prebleno yaf:
for sonething
3146, tio-dEt5a000
0.218160478
read 9260lema's
Khat is proper
to be both a cusheni on and colloguialion It so, what would be sone exemples es
tap-drtaBaa) too 4c462003)
0.49141939
"icabilition"'
ctiquatia for college stascnts reading to leave the lecture roon for any reasen?
D. D.603998956
diooaility""
Stack: (9276d20
1.660I657T0
ccnnerrns"
64757374 20137161
C4 0.25073595
'bet things's
12766564 2c205466
-0.23513014
reoe young"
of this? A: Mell,
20666774
2010616e
-0.217313407
(a ",,
Exsegle: arthr breaks, an urgent
los of a slang sords (which are
69632379
65742.20
0. 0039090466
tbecribe aleidyef: ", "argue...
<9206361
7655206e
2.211241734
phone call, or ret...
colloquial by definition),
6r702065 756...
-8.57995366....
49

---

Published as a conference paper at ICLR 2025
Table 36: Raw training examples selected to have 14 quality ratings from I to 5 within StackExchange.
Structural
Structural
Structural Standardization =
Structural
Structoral
Standardization 11
Stundardization 2
Standardization =
Standardization 5
IXDHaVtio
...] JFEOF?
...eeeneard
...Q: result shell spript
XP3
120231 u2613
instead pausing
Keyboard. 0.05
#270C J2700]
ask infer
ob jbjectiveee
Ray-KP4 KP4
une 3C1 oDFFB-
who the travel lers ware. He was doubttul, teartul, and cotally
would like to run
Keyboand 0.05d
gM000+222
uDEZF]11 [ u270N
shell script, from
--> Ray Ray_KP5
To/p
u270B] (2: u38Ic
tile or from an ob jectivene string
Keyboard. 0055
P+y5<1020138601
uDFzB- LBFFF]121
--> Ray. KP
u0049 2004K ud0
nacking in the skill woodcraft.
(within the codel
Keyhoand 2aaamd.000g 0005
Xike /190:155VV55
u2049 u2122 u2139
would also
Key.KP7
M2194- u2199 u21AS
Gandalt contimm
Like the shell script's result to be storad to a variable, I would not like the shell script t...
Keyboard.
u21AA 52328 x23CP
this, "The victor
--> Key. Key-KP8
u2330- u235F u23F1
would energe
Kayboard, C951
a2392 .2378 U23PA
auzonges than
Key KK2
201//aFI
u2402
u25AA
u25AB
eithes, and free
Seyboard.CDB
etk1/ 150C......
u25R6 v2500
u25Pn
From douot," eaid
--> Key KPO
u25...
Gandalf.
Keyboerd.0066
Key KeDo
Kayboard.CD56
Koy.
Style Consistency
Style Consistency 2
Style Consistency 3 ...Q: Box to generale randon nurberz and distribute then randonly to some buttons?
Style Consistency 4
Style
Consistengy 5
0C x1b x61 xlac
Kollawers:
Desktap
...2: Framennedd
800
kay: Jenny <lext: JJanyy
playing ing in
x00
x00 x00 x00
lacissakop rest-g erver node modul le
Ballay/GuieTinnnSkeeee buz in VLC I ercoded a 1824-Video (HEVC) in Swift using Videotcolbox
x00
x60 x00 x00 x00
ank Hess' vvalues
x0c
x60
x00
x00 x00
"Jenny Heaas 1,
nongoase lig
x00 x60 x00
x00
x00
repeate:
11:3941::
x00
x00 x00
xod XGD
comindara: 11,
rookle in android pregramning and generate some nunbers
at Function. Findone Theodos loa DeskLop Lariese-node larissahgp res s-server node
xoc
xo0
x03
Leakt ags:
CoreNedia.
x00 x00 x05 x03 a00
Lexts Thailand
The swaulting fragmented MPC only playing in VLC, The library Laing to welle the EMP4 is
St x847P4 x00 x00
id: 3, text:
randonly y in a specific range. Provided that, the sun of tao of then equals certain
x00 x50 x00
x00 xC0
In...
X00 xtc x00
x01 x01
nodulea mongooae lib model.
Ja... -odel.js....
Topic
Topic Focus =2
Topic
Topic Foems -4
Topic
Focus
Focms
Facus
2554
uBCP4
Shwy claxse
...01: P 2
...0:
besw 11;
8C558 U32E4.
navber <al
(22.0, 305.0,
actionPertorred nethod (Java] is not working and I have no clus why Bare is my whole progran, don't t wender about the using,
STYLE container
uC815 u3900 4
classe 0l
9.01: [f-2(),
width: 1140sk)
WAB69 u3644 uC300
class- textheador
(27.0, 310.0,
marçia: auto:
sCllc u3093
uC628
asta classe logu
20.01: [12(],
Busdia (nax widths
uD553 u3093
uC758
Steieginsspan
(18.0, 302.0,
1200px) .ontaiine
#02b0 u1PP3 u/99C
classe vind
14.0): [*2r],
width: 960pp:
UC544 u32C9 UB17C
>emn< >amn<//pan-<//><<11
(28.0, 293.0,
Inedia inax-xiath:
03098 03λ38 uC900
<li class-
4.011 ('-2")
992pxi ccnnsiine
uC2DC 434E4
4
toxtheader
296.0,
Down from action pentorned methad, (T only aid the System. priaaiitt))))
width: 720pxi
un558 u3098
>ca0s/a26/1i>
[12"),
@nedia
uC774 u3CF4 uAB30
classe Lextheades
(23.0, 307.0,
103
0C883 uC143 9025C
><o><//></13> ...
19.01: [*-2"1, I.
084F4, 3AB78
(28.0, 294.0,
10.0): ('1",,
(27.0, 233.0, 0.01:
1-2'1, (33.0,
307.0,
Overall Score
Overall
Overall
Overall S
Overall Score 5
Score 2
Score 33
PRbEEEE AB5AG
Finishoodd
...Q: The code works but it laga
QQ Bow to keep
..-2: Concurrent Quece in Java that
<d8pl sact ael Finisho
luual di sicooryy
BSASHAbABw
25/39/2015
it works. with
autonstically
the last retains
AllgAbABAA
HAAAARBA
</a3p/
synced with
<d3pl iaslinnttcnn
toroach and losp
remote, without Latency issues?
each child
SAdABAAHAAG
12/12 12/12/211<<d5gpl
When the code
Lhread have
ACEATAAGAC
beselineStart>
is wozking so lacay it wou l be very good that it is not
develop a git git-tracked
main shroad which
AATORJACTATAAA
</d :ha Resestart
starts up child
AGAAFIAAA
FinishMudeel
codebase that has
threads. Each
BOSSWADABEA
when
lot of tiles. This code remote nachine.
these child threads continually produce new event and add it to the shared quece, That event ropresents
EXAGQA
FinishModaas>
works. How the code worke: swarches the corputer for a Fi is then wher
VABGAA OA WABYЛEAAUA
<dbplaactuslFnnoos
run
BCAPYALABOA
27/03/2015
evwcy tine I make a change loca lyy, I must then ayae
30A8A...
</d3p1 </alpl.accuuslle wattuaattttiieeess
<d5p1:
50

---

Published as a conference paper at ICLR 2025
Table 37: Raw training examples selected to have 14 quality ratings from I to 5 within Wikipedia.
Accuracy =1
Accuracy =2 ...fe, cárui
Accuracy =3
Accuracy =
Accuracy =5
...5) 1998
.... Thstaskoku),
-.-ond hadden.
R1 Gri que, por eapiritt
VR231
1998
nant raşie
appointed queen regrant Japan.
Hez huia vervaigens verhuard aan burgensester
1-)1 LINEAR 1-
eate observazz de cuplu Insā. de cand Setita se oprise anterior la brut rentro cumpára paine ş1 duiciuri pa drumal spre casa bunicli. Urmeazã Rapunzel, eu påcul blond, pe 18ngä al cárei turn din pādure trece nevasta
I 140065) 1998
Haniko died in
ascapista y su optimi sno infartil, al épocs, fue el novelista Kar Jokai, autor de voltmenes grosa de fiec: ón rent novelas cuent en los que respira
VI231 1990
the 200s, and the van Sappeneer
[->1 LINEAR 1-
next king of uapan male, but civil war broke
lenk Eikens, die echter
logró
144467) 1996
/ 1998 II
1927 plotseling
1-)| LINEAR 1-
out again, and the rebellion ended when
overlaed. Mogell 1
I 143063) 1998
werd het huis vanaf
VE201
1920 verhuurd
snother fenale, Teegeo/Ichibay Isee Calyel, become queen of Japan.
Johanaes Jurgens die vanaf dat jear werksaan was als griffier bij het
concn...
Coberence= 1
Coberence= =] Aided
3
Coberence
Coberence =3
...James Middleten
e... la perratio-tijttlloriieeii.
Lrbaine eatre
...Paul Finch is English ausher and ecciptwriser, le began his uriting career the British television Proprame The Bill. His estly reripts vere for children's an ion, He writter
Archibald Miller
naic Boig.
Millian Miller
2006. Aided
d' d'Aqques qui devenue Are Internationn lonal tera travailler
Arques. De nouvelles rostes sont * gull rement construites
Nornan Craig
Guill rele Garbada ocus Aided Gairb
M: M:l1man Saurence
Minot Hugh
Glinne Rige Cles
Fitzgarald Noora
violentes de Guill Mac
directenent Ladirectenens juequ' plus de dane 1' l'Andemarois,
larçies (recade, voic nouvelle de la vall e de La 1VAVAL, voles expressea Saint-Ches Dunkezque
Gorald Ewart Nash
Srnest sdward
Carbada et Gairb Glinne kige) Aided Taegairi Ruadaig tLa violeate Loagaize Bandachl, trad. Guyonvarc' h, La mort violen...
owen Augustus
Paget Nedley
causant une mutation spaciale az paysag za M'dddmarols, svec une ferte riurbanisstion...
Parlee Lacrence
Pearsen Geatfrey
at Boulege-Sariat-Deee desserte par l'actoroute A26
330 short stories which have appeared in magazines, suzh
2idcock Sydney
Zope Frederick
Powell T...
Creativit
Creativily
Creativity
Creativiyy =
Creainily =3
.x.87231 -
rr. Arendt (a 10, ober 1854
rist a neta
Da-iinggeendde
I. 30 de juliol,
Placára, state definità Lovinsacu come uns serie di quadri dells nostra antica toni
battuta 12 1881 se alla cadenza perfatia impone the i1 solista riprenda suonaze dopo doveroes pausa. Seque un ponta di roglegamento al secendo tenz imisure 13- 19). carattere de prino teme e*a moito cantabile, 11 secondo tena
Beutungen hat auch
2000 Socorro
Barling
Rip Van Minkle, die andere in Amerika spielende Kuragaschi chte des skizzerbuches, erfahren, Protagoniat in der Kolonialzeit in einen swanzigjähzigen. zauberschlaf lt den
LINEAR |- 87232
April 1936 obenda]
I 1I 30 da guliei.
war ein doutscher publizist und
2000 11 Socorzo II
L.NEAR 1- 87233
treikenserva
11 30 jul
iolititer, leben
2000 1I Sccorro II
Arend: stannt aus einer 1ddischen ramilic und konrectieste splter zun christencum,
arcaici ,
LINEAR 1- I 89234
Ion Vianu come una storia pittoresca della Valzcchia. Cálinescu ha notato come, in nolte delle sue poesle in partiso... Grammatall Diversity 3
II 30 da juliel,
2000 Socorro
L.NEAR 1- 1 I 11 al, 2300
Nach den Resuch des Gyrnas iuna.
Socorro 11 LIHEAR
1- 87235 11
Grammatienl Diversity 1 ...11
Grammatical
Grammatical Diversity =4 endencia
Grammatkal Diversity = C..Ciiiiicc 1996 ckredlenic nadene fali strajköw protestów, do ktorych dosalc PaL pod koriec 1976, po ogioszeniu przez ra4d Piotrs Jeroszevicoe uprowadzenia
Diversity =
...al Stadium Stadio Artemio
...frique de l'Est, Brésil, irate-Unie es en Aust custralie, lees deux eapéces descendent de l'aurochs (Bos primigenius), dont le dernier représent ant auropéan sfeat sceint 1627, alors que la sous-espècs ayant mené au zébu surait disparu Tnde env...
btyle-background;
perpetua
1:58 June 1. -Mariners
Franchi Uhelanco
neutrslidsd del
Arena Arubaane
estado. .as paredea de la fortaleza fueron derrinadas la quarnicion prusiama fue retirada. Los visitantes fanosos Lexenburgo al y al inclayezon al posta zlemán coothe, a los escritores
-1111
voetbalel
1-11 -
Arcbaana
szyle=backgroundi
voetbaleiftai
59 June
(vzouven)
Cardinals
Ackton Sate
1. 1-111
Asian Pootball Confederat ion Moraiz Aslanyan
-1
ssyle=background:
drastysanych podwyiok cen uzzedzwych na nicktsre artysuly konsunpoyjne. Prayosyny...
I 60 Juno Cardinals II= -I1
Cassama Assaidi Fernando Astengo Asten villa PC Asturiech stallffaal...
11 -- 1- ssyle....
fmi' Bo....
Knowledge
Knowledge Novelt -2 32" J.H.
Knowledge Novelty 3
Knowledge
Knowledge Novelty =5
Novety =1
Novelty #4
50
...Vonga 1638-1690),
aoiembric 2300
Bossi apartak
to ba sone sort of non-trivia. medium to which associate certain energy. This is because the concept of absolutely space contradicts postulates quart un nechanics es. According to OFT, ever. in absence of real particles...
puzchased equipment pment and wrote software that allew him to analyse beats, and began studying his own heartbeat thythms well those fri ends and osher mallciana. After decadas of study, Graves used sone the ...
1. Socorro 1|
Boskwa Valencia ea 0:3 (0:1)
figlic di Ton Khan, Sotto 11 regno, Lan Xarg conobee mass imc splandoze.
L.NEAR |-| 178961
11 roienbrie
Angulo 6', Mieta Sánches a5 kolajka paddziernika 2002 E. Liverpcol F.C. Spartak Bookwa 5:0 (3:01 Heskeyy
2000 Socorro
LINEAR ||| 178762
1. 11 roienbrie
ALla sua morte scoppiarons muovamente diseidi interni de' aarastterrrria Tian Thala (t. 1590-1695), prime ministro di aarigna Vongsa
2000 11 Socorzo II
L-NEARI- 178763
11 roienbrie
2000 Secorra II
7' 89', Cheyrou
LINEAR 178764
15', Hyypia 28r, Diso 81" Valoncia
I 11
51

---

Published as a conference paper at ICLR 2025
Table 38: Raw training examples selected to have 14 quality ratings from I to 5 within Wikipedia.
Language
Lunguage Consistency 4
Language
Consistency =1
Consistency 2
Consistency =
Consistency =5
...que desti d'aooka Ree
rekord vátázstvi v sezons, ve ktcré ale neziskal Litul nistra svšta.
...Ni: 1skousatva
scenter> <conter>
(er. x Monde
dabba avare una contrepacte
scentor> <center>
ambigu und mica abert encaca que la seva dobladora
do l'Art w] est
Scenter? [coette>>] -
une association ion
adeguata: la
d' drtriistes russes Fondée 1898 1' 1'ide
tarra alcre attivita
859; Tudelanol
Rokstein orela que personalge viu. 3n copital Un món re
Ste jnk depedl
style=backgroun
1984 1988 Alain Prost a roce 2006 Micheel Schamaches. V roze 2005 se Kimi Räikkönen podalil rekord nejrychlegbich kol sezdne. rekord so de11...
produltive.
:FFB0801144]
prénez un renopweau pictural de 1' art zuase en synthétican: plusi eure formes artistiques dont théacse, décoration et P'art da livee....
idea una noneta che prende la forna di biglietti bancaci aganciats ata da netallo presiose nolto moderns, Ai nostzi gioxni,
styles background;
nona, de la quarta temporada el desti d' Ahsoka revela finaimsat. Ezra Bridges, que ha afffeeie batyely a...
[P5020]18-]13
style=/2ackrronm
:PPE4R5; (1318)
<sstyle"
background:
:effcef/4lsty4 stylee...
o'oro e l'argento hanno co...
Orginalty 1
Origiality =
Originaliy::
Originallyy #4
Originally =3
Trogrest
... s..aquad-a. bar 11 bariata si
11
11 12 ottobre
is an exteraion
guaso, bascante
Goodricke-Pigott
2002 1I Hount
of Tropfest, the warld's
avvicine a Brooke.
neuczo y de alta
11 R. A. Tucker
Temmon Survey 1-
la regazza penss cha voglia un autografo, lui dice che eli serve una firma per il conto e che non à i1 tipo da autografi specialnent sa chieda. Lei gli dioe chi lui ai presen....
pureza, can un Ligeze saboz dulce resultante axúiarr de caña. Cen cate destilade finol, añadiends diversos ingrodientes, elabora tanbien el Colorado la Extraconcha, que obt ienen concluis al p...
[- I 112515
552175 - II
largest short
ottobre 2013
film festival. Tropfest began
Survey
I1 R. A. Tucker
552176
1993
scccening
1- 215645 -1
II 9 ottobre 2013
for 200 people in a cafe in Bydney but bas since becone the largest platforn for ahost films in the world. Propfest Arabia....
11 II RitE Peak
II Mount Lennon
Spacevatch J-
Survey |- I 552177
215646 II I1
II 1I 31 narzo
NEAT
2008 Hount
215647
1I
Lermon survey 552178
11 Kitt
Deak 1I
Spaceratch
2...
Profesionalsm
Profesionalimm #2 sasr va comenpar nostrant que aquestas millores
Priessionllsan =3 ...arr
Frofeskonllim
Professioallim 3
2717777
...Caml Christian
i..inamen
24 settambze 2300
nozts. cracait
Bernez [+
Schanskha erhinlt,
11 LIREAR |-
à o'oriiine une guilde légale
Novenber 1841 in Christianis;
122778 - 1I 1113 24
eren comporais.
beschrieben
settembre 2000
Pez exemple, arab llanper nissil aucleas, augmentant un con cte a ialila escala fent que les encinss mundials es rearmin, els tons d' d'teernttivve Earth van ser malvers...
naia qui acceptait
25. Hai 1918
nachden 1856/57 Sagend Yasykköl See beauchte.
LINEAR 1-
(Eligos) suriouts
obenda) ein norvegischer
122779
nissions
settembre 2000
arassaainaat
Polilikes Laban Seine Bltern wazen der Richter an BLifCsobergezia olut Steen Julius (1809-1855) und dessen Frau Marie Louise Palkenberg...
11 LINEAR [-
des missions ont Ar4 décla rdea
Sem Senjjnow--iann-Shheee konnte beveisen, dass wa sulbat in den trockenwisten große Sobisggleetscher gibt, vas ez und andere Wiszenehhfft.cr... Semanlic Densily
122780 11 1111 24
seltembew 2000
intardites par le Conseil la Nagie, le
LINEAR 1-
122781 11
settembre 2000 11
d'Eison Hald été arcété et mie on prison. La guilde
LINEAR 1- 122782
Semantic Donsity
Semantic
Semanicc
Semanlic
Donsity
Density
Demiiy
...T Krenot
c.. (1902)
...lee tonale
...
Spacevatch 1-
Ruinze Erehof
Nozae (1905-1906)
delle Arti
sang:iento
1390397 - 11
11
Vollenaove 3rehot
Mozse (1910-1916)
Figurative presse
(titulo original:
Spacewatih 1-
Millensoo-d
No-se (1914-1916)
P'aula
Blood Al Alley)
1390398 11
Espelo Fanfare
Notor Bob
dell' Issitute
peliculs
Lemmon Survey
[tiim) zenny
(1914) Motorette
Hagistrale Statale S. Satta di Nuore,
estadouniderse 1955 di-igide por Willian A. Wellman protagenizade poz John Mayne Lauren Bacall. Arquncato La guerza eivil China ha tezminado. Las cenunistas han bonado el ...
I- 1398399 -
Mlanke-s-Koen
(1911-1914) Noyea
11 11 Spaczwatch
Stadion
(19C3-1934) Koyar
Mel 1957 Nuoro partecips alla
1398000 - 11
FC Twente-
(1911-1915) M.F.M.
11 11 Spacswatch
Berman Finkers
(1914-1915) Muelier
Biennale Nazionale
399401-398300
Friezenberg
(1896-1899) also
Pittura
1398401
Gamnelke
Mueller
Prenio Sardegna, prenossa dail' previnciale par i1 turiano. Nel 1958 espone...
Spacavatih
Ganzendiep
Mulford (1915, 1922) Holtiplex (1912-155 19...
1398002 - 11 1I
Goidernan
Genenuiden Genne
Oveznatees
Geech...
Semultvity
Sensiivily
Sensliviyy
Sensiiiity ... Antisenitonliaa was eine der crsten
Semsiivity ...l, Batezreichischee rischtomicspieler
Sonfessionn 13
... Marur
...ostrale de1 biaachi (1a
(2000) Onderatudy
begeh: Helen
(2000) Net Drezms
Zoch schwcren
cosiddetta
(2000) Net Dreems
Krastwwsgendiaastan
cousasica)
Vereini gungen
Kellet, Heinrich (1924-2013)
(2300) When
ein deutachez
abitanti deila
Sannlung ung van Judengegaeza
Boyz Aze Away The
Kurzflin. untez der Regie von Kos Geises
Scandinavia, con i capelli biondi e
batesreichischee Bistor iker Keller, Beinzioh ich (x 1941), Schweizer Jurist, Becht skonsulend und Direklos dea Bundesentes For Jastix ....
Girlx Will Play
Deut schtn Raiserreidh und die eeste, die das Schi Schlagus lagvrrt Antiseni zun politischen Progrann erhob. Sic wurde an 25. Septembez...
1 (2000) When The
gli oochi azxurii.
Boyz Are Away The
aue den Jahe 2022.
Charroux sostiene
Girlx Will Play
Swine Urauffuhrung fuierte dez Piln an 21. Januez 2022 auf den Fllnfestival Nax Ophale Preis 2022...
inoltre che questo
(2000) White
popalo
Panty Caronicles
evuto un' origine ext raterrerssst re, proveniente da un...
(2000) Girls
12000) Calerdor
l...
52

---

Published as a conference paper at ICLR 2025
Table 39: Raw training examples selected to have 14 quality ratings from 1 to 5 within Wikipedia.
Structural
Structural
Structural Standardization
Structural
Structural Standadization
Standordization
Standandiaaionn =2
Standodiiation
.4val 6:6
t--type: Point,
he allucinasion vedendo l'ansessing no mascherate in lortoranza na aubito dopo cozce tra 1e braccis di Kierany due al baciano tirisoono poi col nella
The pair always
La di ti de Sheey lune,
#connamen-
coordinetea:
had intent Liona bringing
guverte joar-la.
-155,5988931 6558844
chanteuse
Do
le tunnel rétractable
23.9707875297782
Christaptar Beeve
françaico,
, type: Feature, properties:
onta the snow, and whan they found out that ha enjoyed watchingg ing the show ii Gouch and Mi lar decided that they
compose quatze albuna atudio, dix singies
juaqu' au millec
du terrain pour protager les
geonstry:
type: Point,
et dix el clips
joueurs pfavail
cuocdinaless
video. Albuns
été déplié,
95.167531967:63
Chansons Singles class
Le la president
8.726969207992207
1' maccaina. Al Loro citorno sia Kioror comanica a quest' alt ino che conLingesd
JSK, Meaznd Chésif Bannachi déclare que selen las nédecins du club, Albe...
I a type: Feature, properties: geonstry:
going bring
atylletooxtallinnncaatee It Liste des singles et positions dans la...
seasen tvo. They had already crafted chazacler, Dz. virgil Swant, they would seveal...
him
type: Point,
evocdinaless
12.8279014599444,
Style
Style
Style Consitency 3
Style Consistency #
Siyke Consisteccy 5
Consetenccy =1
Consisencyy =
1 [13,0 0 1
kath.,, 280 ref.,
33da, Malta,
...hav2 bccr.
286209
120,4 1264 1131
zsidó lak. Set.
1631 (prestavi jen 18931 Obalisk
ottobre
1274 13,70
anyaszanteaydá,,
[Location
2001 HEAT
13,65 121,, 142,3
keth. kapolna. Bablábway,
Thenasa Jeffersona Mant ioelln, 1823 Cbelisk
for four other
286210
11
12,22 122
sunga missing Lhis
cltobre 2001
122 5,38
4
Reds. 192 hold srant6-old. F. u.
Linal) (hyriisst Sean Rafferty:
XEAT |- 286211
10,69 127,0 10,329
levov, Tays, Romunija, 1824 villa Tosloria,
I 16 otcobre
1642,4 [14,2 113,3
ar. Andrasy Gyula.
Thase are net at the besause Flayers' Theatre privaze club and was not censored, Your to the Bar
2001 II LINEAR I-
121,8 1207 1165
Borevezky
206212 17 ottebre 2001
1341 1346 12,93
ameeorfflasozzzatie
dva oballska, 1842 Obelisk
12,99 118,3 6
Zemplén värnagyd:
LINEAR
286213 I1 17 oltobre
10,955 [23
Largyald céaze
pokra line Ruilije spenir na poroko Francescs V., vojrode Modzae, in princese fd....
123 5,33 6
szerint: Sz516ske, bodrogrent magyar kicko2sé...
2001 1I LINEAR J-
10,6 124, 10,258
286214 II II
1509,5 116...
(1951) Diedre includes in Topkc
Topic
Topic Fucms
Toplc
Topic
Pocus
3
4
Foem=
(Da:lingEon.
und Hotels in Bri Lanka kamen mindestens 253
r..clasg-note
E..Eacy Victue
Sh.Sherddan
Yupadeo 2obkul2oonsies, Thai
I- clagg-veard
stazring Ber. Barnes, Jassics Kristin
Jobbins (boza
class-tn In
Jully 1960) Australian journelist,
Kenachen westere worden serletat... 2.
Isben,
NorLhover
artist and jewelry designer, COVTD-19. 19, James Hihoncy, 62, Anericar poimonalogist internist, COVID-19. Mark Motaanata,
(Slestonbury)
Themas Cal Pirth (anong othore) is duc for release in Earope on 7 Rovenber 2008. Jobbins has also comtribulen ed Lo, Breast Wisses, conedy musinal about 'breasts, and the people who support then'
classeadr
televi sion presenter and
Juni: Ermordung des CDU Politikere
Sonerset
class=note
screenxriter: Life and carser Jocbins Be Balbourne, Australis. was educsted at Acchan School, Edgeclity itt She is third generation
Halter Libhekk
classmnote
Oktober:
Asschlag dalls ISaslet
clasa=fa
North Oversby
stskethall player 60, Anericer
and Gesel Iicchftt 6. Januar: 76,
classeadr
Lincolashire |
Verleihung der Colden
class-note
TEaES, San Ancenic Spurs, lo...
class=note
Australian film naker... Overall
Overull Store
Overall
Sooe =
Scoee
Sowe
Sowe=5
2, Sú, 12, 36,
1..l Koog 15
(2.11011)11 2003
...3-2001) Adoun
...E1 tormino
13, 25, 35, 371,
10 8 Raph
#9511
Youaousni, lccing
latino (en latin:
59, 196 81, 92,
de las Casas
alignerightt 14,811
Prine mirister (2007)
int hace
181, 106, 273, 60,
6CM
aligneright 0,0281
referencia de las etniar origen indoeuropeo y de1 grupo itálico que se asentazon lo largo de la costa Lirrénics Lation, Italia, en el curso del 11 milenica. durance la Edad del Bronce.
194, 620, 270, 220,
+2 kcogs 25
aliga=right 0,0541
Kassiré Kounskoye, Prime miister 12907-2098)
106, 388, 287, 63,
11 6 Arnand
alignerigtt 0,0211
3. 6, 191, 122, 13,
Moz 1 Privatnik
sight S,266
234, 490, 166, 290,
"Sasereti 4CM 1 23
Il 83B 11 26.
Yousceut Salch Abbas, Prins minister 2010)
314, 47, 48, 31,
+7 krogov 21
biezna 2003 [Campo
96. 26, 115, 99,
|- ods 18
Teperstore |ICTNESS
158. 191, 110, 77,
Clerente Blondetsi
(1201022) |2000 FU aligneright 115, 111
85, 197, 45, 10,
Alta Gozse
Ennancel Nodingar, Prime miiister 120:0-2013) Bjinranger Dadnadj-, Prine minister (2013) Kelzeu...
113. 140, 353, 48,
Alfa Romeo Tipo 308 I 1 1- Oda...
120, 195, 2, S...
aliga-richt 2, 857)
aligneright D, 047] al...
53


Reviews:
[{"review_id": "f5AQq0Nd2F", "reviewer": "Reviewer_8xFT", "summary": "The paper introduces \"DataMan,\" a comprehensive data manager designed for pre-training large language models (LLMs). DataMan aims to improve data quality and selection by introducing 13 well-defined quality criteria and an overall quality score. The approach leverages reverse thinking by prompting LLMs to self-identify the factors that influence performance, focusing on perplexity outliers. The paper conducts experiments on several benchamrks to show how DataMan effectively annotates and selects high-quality training data, resulting in improvements in in-context learning (ICL) and instruction-following tasks. As described in the paper, the implementation of the framework would be released, which could be beneficial for real practices.", "questions": "1. How do the authors plan to resolve the situation where model itself made unreasonable decisions on criterias? The hallucinations generated during the process could have snowball effects for some training scenarios. One approach that can be tried is using Mixture-of-Experts to reduce the possibility of such situations, but I would like to see authors' insights and if they did any experiments on this direction.\n2. Is there a way to mitigate the high computational cost of DataMan’s annotation process, perhaps by leveraging lower-cost models or more efficient sampling techniques?\n3. The paper mentions that perplexity and ICL do not align strictly in some cases. Could the authors provide more insights or analysis into the scenarios where this misalignment is most pronounced and how it impacts downstream tasks?\n4. I think \"vasrtdatasets\" at line 26 is a typo?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of using LLMs to self-select the criterias that are beneficial for model's performance is straightforward and easy to understand. The overall framework makes sense.\n2. The description of the method is clear and easy to follow.\n3. Data selection for both pre-training, instruction-tuning, and SFT is an important direction for LLM applications, where I think the direction in this paper is interesting and could be beneficial for downstream applications. The experiments show that the proposed method can also improve the model's performances in different benchmarks.", "weaknesses": "1. I think the motivation to use 14 scores is still a little bit unclear. It would be beneficial to have an analysis of why these scores were chosen and how these scores can distinctively differentiate models' abilities when evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "deyNdiVLdf", "reviewer": "Reviewer_Tq6k", "summary": "The authors propose DataMan, a pipeline for pretraining data quality filtering. First, PPL-thresholded data are used to extract 13 different quality rating criteria. Then, a smaller LM is fine-tuned to annotate documents with scores for each quality rating (and domain.) Finally, the LM is used to produce a higher quality small subset of RedPajama. Training on this subset shows accuracy gains across numerous tasks and significant improvement over baselines.", "questions": "- SlimPajama is already quality-filtered and deduplicated. How does this affect your analysis and results? If DataMan's improvements come mainly from filtering already clean data, the impact on raw web data could be quite different. Could you provide results or analysis on a raw, unfiltered dataset?\n\n- How is the \"overall score\" calculated from the 13 criteria? Could you provide the exact formula or methodology? Is this determined by the annotating LLM?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper uses a novel \"reverse thinking\" concept which derives quality criteria using documents filtered into high and low perplexity buckets, which avoids pitfalls of criteria based on human intution.\n- Introduces 13 detailed criteria for quality, which moves beyond typical heuristic approaches which threshold on perplexity, deduplication, etc. \n- The focus on 13 specific quality metrics and their annotations makes it clear why a particular document was chosen by the model for inclusion in the dataset, thus making the method interpretable.\n- DataMan demonstrates a significant performance improvement over numerous heuristic and model-based baselines across multiple metrics: PPL, benchmark accuracy w/ ICL, and instruction following.\n- The method performs well against impressive baselines, including statistical approaches like DSIR and models trained with 50% more data.\n- Empirical evidence shows benefits of DataMan for domain-specific continued pre-training.\n- The presentation is clear.", "weaknesses": "- The authors apply DataMan to a 30B subset of SlimPajama, which is a relatively small dataset. Further work should be done to validate that the method works for large pretraining-scale datasets.\n- The SlimPajama dataset has already been filtered for quality with duplicate documents removed. It’s not clear if this influences the analysis of the method or if similar improvements would be seen applying DataMan to unfiltered web data.\n- The paper does not discuss the inference FLOPs required to run DataMan. This would be useful to understand the tradeoff between spending more FLOPs on annotation versus pretraining for more tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l8drxRh4Fr", "reviewer": "Reviewer_ViAr", "summary": "This paper presents DataMan, a Qwen2-1.5B fine-tuned to perform pointwise evaluation across 14 quality criteria on a 356K dataset annotated by GPT-4-Turbo. The authors then use this model to create the DataPajama dataset from annotating a 447B tokens subset of the SlimPajama dataset. Pre-training from scratch on a subset of DataPajama show improvements compared to other baselines.", "questions": "- The message in Section 3.4 is overall confusing: \n\ni) I'm not convinced that pointwise ratings can capture the same essence as pairwise ratings. Suppose you are evaluating the factuality of an LLM response given a question about Albert Einstein, the LLM gives two answers (A) and (B): both are factually correct but (A) adds a few more details while (B) simply gives the answer. Pointwise rating would not distinguish the subtleties of both responses while pairwise would. To support the opposite view, the authors perform validation through human judgments. However, I'm concerned about the validation process (Table 5), as it appears limited, having been conducted on only 10 samples. \n\nii) For prompt validation, the authors claim that human agreement exceeds 95%. Could you clarify the validation process ?\n\niii) Could you provide more context and an explanation to the equation (L.246-255) ? I can imagine that the error in rating decrease with the loss when training. But, how does it justify the use of pairwise over pointwise ?\n\n- As a parallel work to [1], is there a reason behind choosing Qwen2-1.5B instead of Sheared-Llama-1.3B to train your DataMan ? Plus, the authors in [1] have disclosed their training dataset. This would have enabled a direct comparison between pointwise and pairwise ratings. \n- For the \"Sample with Qurating\" baseline, could you clarify the sampling process ? My understanding is that you sampled with DataPajama and not the original QuRatedPajama. In this case, is the comparison fair ? A quick look at results in [1] showed lower perplexity.\n\n[1] Wettig et al., QuRating: Selecting High-Quality Data for Training Language Models, 2024.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Use LLMs for filtering pretrained data as opposed to using heuristics.\n- Promise to release code, models and datasets, which would benefit the research community especially given the cost of obtaining such models and datasets.", "weaknesses": "- **Novelty**: [1] already proposed to train a model to rate and select high-quality pre-training data based on four qualities: \"writing style\", \"required expertise\", \"facts & trivia\", and \"educational value\". The main differences seem to be on the considered criteria, the model used (Qwen2-1.5B vs Sheared-Llama-1.3B) and using pointwise instead of pairwise ratings.\n\n[1] Wettig et al., QuRating: Selecting High-Quality Data for Training Language Models, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tFOZJGNDzF", "reviewer": "Reviewer_JcNL", "summary": "This paper addresses the need for comprehensive data quality criteria for pre-training large language models (LLMs). It introduces \"DataMan,\" a tool that evaluates text quality and domain types using 13 quality criteria related to LLM performance. The paper demonstrates the effectiveness of these criteria by fine-tuning LLMs on a subset of the SlimPajama corpus, showing performance improvements over state-of-the-art benchmarks. Contributions include the establishment of quality evaluation criteria, the creation of the DataMan tool, and the release of models and annotated datasets to facilitate further research on the relationship between data quality and LLMs.", "questions": "- How does this method compare with other methods (e.g., deduplication, heuristic-based selection, domain mixture, other LLM quality signals) mentioned in the related work?\n- How does this method compare with these large open-source efforts described in the following two papers?\n  - Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research [[paper](https://arxiv.org/abs/2402.00159)] [[code](https://github.com/allenai/dolma)]\n  - The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale [[paper](https://arxiv.org/abs/2406.17557)] [[code](https://github.com/huggingface/datatrove/)]", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper offers a comprehensive set of quality criteria for evaluating pre-training data, which helps improve LLM performance.\n- Though not comprehensive, the approach of leveraging LLMs to self-identify anomalies in perplexity is an effective method for deriving quality metrics.\n- The release of annotated datasets and models is a valuable contribution to the community, promoting further research and development.", "weaknesses": "- The dependence on perplexity as a primary measure for quality assessment may limit the generalizability of the approach. While perplexity is an indicator of understanding, it may not fully capture other essential aspects of data quality, such as creativity or sensitivity. It is also bounded by the knowledge and capabilities of the LLMs that are used to compute perplexity.\n- It fixates comparing with Wettig et al.,2024 and lacks comparison with other related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Eb0Aiou7ap", "reviewer": "Reviewer_Pb2j", "summary": "This paper proposes DataMan, a perplexity-driven data management system that annotates the quality and domain of data points, which could be used for many applications such as data selection in the pre-training stage. The authors proposed 14 data quality evaluation metrics and trained a model for data quality based on Qwen-2-1.5B with GPT-4o-turbo annotations. Utilizing the data quality annotations, the authors selected a subset of data and trained a Sheared-Llama-1.3B model from scratch. The model trained with the best overall score demonstrates superior ability in various downstream tasks compared with baseline methods. The authors also conducted various analyses of DataMan and revealed several interesting findings.", "questions": "1. In lines 231-235, the authors claim that pointwise assessment is better than pairwise one by presenting results in Table 5. I don't quite understand how content in Table 5 could be related to this. Moreover, I don't think handpicked results could really support this claim.\n2. It would be better if the authors could write how they picked these 13 quality criteria and all the domain categories. For example, the authors might provide some statistics on the outlier causes mentioned in lines 92-94, and how the iterative refinement is done. Is it done manually? By student or expert?\n3. In line 285, what is the value of $k$ in top-k?\n4. In lines 201-203, can you explain how the designed 14 quality criteria correspond to the three principles listed here?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The proposed system DataMan is overall sound for assessing data quality and this is an important research problem in the current research community.\n2. The annotated data is an expensive but valuable resource for the community.\n3. The experiments, along with analyses, are solid and demonstrate the great performance of DataMan.", "weaknesses": "1. The presentation of this paper needs significant improvement. I can find typos almost everywhere. I just list a few here. In line 193 \"minimizing\"; A missing space in line 241; Figure 1 caption is too close to the figure;  line 110 \"require\"; line 234 incorrect citation format; line 420, \"quating\";\n2. Some claims made in this paper are not well supported. The one that mostly concerns me is that the authors claim \"previous works are intuitive and lack generalizability\". However, the 14 quality criteria seem also intuitive and manually designed by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)