MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models
Authors: Jiazheng Li
Lu Yu
Qing Cui
Zhiqiang Zhang
JUN ZHOU
Yanfang Ye
Chuxu Zhang
Venue: ICML 2025 poster (2025)
Keywords: Large language models
Data selection
Pre-training
Primary area: deep_learning->large_language_models

Abstract: High-quality data plays a critical role in the pretraining and fine-tuning of large language models (LLMs), even determining their performance ceiling to some degree. Consequently, numerous data selection methods have been proposed to identify subsets of data that can effectively and efficiently enhance model performance. However, most of these methods focus on general data selection and tend to overlook the specific nuances of domain-related data. In this paper, we introduce MASS, a Mathematical data Selection framework using the Skill graph for pretraining LLMs in the mathematical reasoning domain. By taking into account the unique characteristics of mathematics and reasoning, we construct a skill graph that captures the mathematical skills and their interrelations from a reference dataset. This skill graph guides us in assigning quality scores to the target dataset, enabling us to select the top-ranked subset which is further used to pretrain LLMs. Experimental results demonstrate the efficiency and effectiveness of MASS across different model sizes (1B and 7B) and pretraining datasets (web data and synthetic data). Specifically, in terms of efficiency, models trained on subsets selected by MASS can achieve similar performance to models trained on the original datasets, with a significant reduction in the number of trained tokens - ranging from 50\% to 70\% fewer tokens. In terms of effectiveness, when trained on the same amount of tokens, models trained on the data selected by MASS outperform those trained on the original datasets by 3.3\% to 5.9\%. These results underscore the potential of MASS to improve both the efficiency and effectiveness of pretraining LLMs.

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
Jiazheng Li
Lu Yu " Qing Cui
Zhiqiang Zhang?
Jun Zhou f2 Yanfang Ye
Chuxu Zhang
Abstract
High-quall sigh-quality data plays critical role in the pre- training and fine-tuning of large language models (LLMs), even determining their performance ceil- ing to some degree. Consequently, numerous data selection methods have been proposed to identify subsets of data that can effectively and efficiently enhance model performance. However, most of these methods focus on general data selection and tend overlook the specific nuances of domain- related data. In this paper, we introduce MASS, MAthematical data Selection framework using the Skill graph for pretraining LLMs in the mathe- matical reasoning domain. By taking into account the unique characteristics of mathematics and rea- soning, we construct a skill graph that captures the mathematical skills and their interrelations from reference dataset. This skill graph guides in assigning quality scores to the target dataset, enabling us to select the top-ranked subset which is further used to pretrain LLMs. Experimental re- sults demonstrate the efficiency and effectiveness of MASS across different model sizes (IB and 7B) and pretraining datasets (web data and syn- thetic data). Specifically, in terms of efficiency, models trained on subsets selected by MASS can achieve similar performance to models trained on the original datasets, with a significant reduc- tion in the number of trained tokens ranging from 50% to 70% fewer tokens. In terms of effi- cacy, when trained on the same amount of tokens, models trained the data selected by MASS out- perform those trained the original datasets by 3.3% 5.9%. These results underscore the poten- tial of MASS to improve both the efficiency and effectiveness of pretraining L.LMs.
Reference Dataset
Prompt LLMs
Collect Math Skills
Construct Skill Graph
Target Dataset
Score & Rank w/ Skill Graph
High Quality Subeet
Train LLMs
Figure I. The pipeline of MASS. (1) We first employ prompt en- gincering to extract mathematical skills from reference dataset, thereby constructing a skill graph. (2) We then score and rank the target dataset using the constructed skill graph. The top-ranked subset can further be selected and used to train L.LMs.
1. Introduction
The success of large language models (LLMs) is closely tied to the scaling up of model size, training data, and com- putational resources (Brown, 2020; Hoffmann et al., 2022; Qwen Team, 2024). Among these three factors, training data serves the foundation of LLMs' rich knowledge and capabilities, enabling them to excel in creative writing, complex reasoning, and even agentic planning. Recently, there is mounting evidence that pretraining on high-qualiçe and diverse corpora can significantly enhance L.LM perfor- mance (Penedo et al., 2024; Li et al., 2024a). For example, Microsoft's 's Phi series of models has been renowned for their pretraining corpus's quality and efficiency. The latest Phi-4 model (Abdin et al., 2024), trained 10T tokens, has surpassed other models of the same size, including Qwen 2.5 (Qwen Team, 2024), which was trained on 18 trillion tokens. This has led many researchers to focus on L.M data curation methodologies, which include the selection of high-quality sub-datasets (Wettig et al., 2024), deduplica- tion of extensive datasets (Tirumala et al., 2023), and the balancing of data from various domains (Xie al., 2024), among other techniques.
Eqqul contribution 'Unieersity of Connecticut, USA AAnt Group, China 'University of Notre Dame, USA. Correspon- Jun Zhou
dence Chuxu Zhang saaux..zang@@ucmm. <jun.zhoujun @ anteroup.com>>.
In terms of data selection methods, the objective is to ex- tract high-quality subsets from the original datasets such that training LLMs on these subsets can achieve similar or even superior performance on downstream tasks compared
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
to training on the entire original datasets. There are two main categories of methods: heuristic-based approaches and model-based approaches. For the former, a variety of manually designed filters, including mean word length, stop word fraction, and word repetitions, are frequently applied to preprocess raw web data (Tirumala et al., 2023). For the latter, models similar to BERT (Devlin et al., 2019) are com- monly used to evaluate and select data with higher scores, while other approaches directly employ GPT-like models to assess the data's s value.
synthetic dataset (Jiuzhang..0-a T. (Zhou et al., 2024e)). Using the skill graph, we identify and extract high-quality subsets from each of these target datasets. We continue pretraining TinyLI Lamnylaaaall... 1B and Mistral-7B on the original and selected subsets. Experimental results demon- strate that with merely 50% to 70% training steps, models trained on selected subsets can achieve similar performances than those using original datasets. When trained on the same amount of tokens, models trained selected subsets can outperform those using original datasets by 3.3% to 5.9%. This highlights the efficiency and efficacy of our data selec- tion method in enhancing the mathematical and reasoning capabilities of LLMs.
While the emergence of OpenAI o-series models (OpenAI, 2024), DeepSeek-R1 (DeepSeek-Al et al., 2025), and sim- ilar models, focusing on advanced reasoning capabilities, highlights the growing emphasis on the mathematical and reasoning abilities of LLMs, current data selection methods usually concentrate on general domains, neglecting the spe- cific attributes of mathematical domains. AutoDS (Zhang et al., 2024b), as an exception, uses Qwen2-72B to score the Open WebMath (Paster et al., 2024) dataset based on whether each text demonstrates mathematical intelligence and suitability for educational purposes mathematics. Al- though it employs tailored prompts for mathematical texts, it still overlooks the deeper nuances of the mathematical domain when compared to the general domain, particularly from the perspective of math skills.
2. The Proposed MASS
2.1. Overview
Given target dataset Digt = 22. ,,N) where each data point is piece of text, our goal is to select high quality subset Dng with budget b E Z+, where b = [Digl < |Pegel- When continuing pre-training or fine-tuning LLMS on Drg. we can achieve similar or even superior downstream performances compared to training on Digt. As illustrated in Figure 2, our proposed MASS can be divided into two main steps. In the first step, we collect small-scale high quality reference dataset Dref and instruct strong LLM to identify the mathematical skills present in Dref. Next, we construct a skill graph Gakat that encap- sulate the compositional relationships among the various skills. In the second step, we then use G to guide data selection. Specifically, we compute the similarities between the target dataset and the extracted skills. We then perform aggregation and pooling through the skill graph to generate final quality scores for each data point in the target dataset. These scores are eventually used to rank and select the high- quality subset Dng- Details of these two steps are described in the following.
Therefore, in this paper, we delve into data selection for LLMs and propose MASS a MAthematical data Selection framework using Skill graphs, to enhance their mathemat- ical and reasoning abilities by considering the underlying skills (or knowledge points) and their interrelations embed- ded in mathematical datasets. MASS has two selection guidelines: (1) a data point that encompasses more impor- tant math skills should have a higher quality score; and (2) a data point that covers more important compositional information of math skills should have a higher quality score. As illustrated in Figure 1, we begin by utilizing a high-quality reference dataset and instructing strong LLM to extract embedded math skills. These skills are then used to construct a skill graph, where nodes represent skills and edges denote the co-occurrence of skills. Next, we calculate the semantic similarities between these math skills and the target dataset to selected. We further aggregate these sim- ilarities on the graph structure to integrate the compositional information of the skills to obtain final quality scores for each data point in the target dataset. Finally, these scores are ranked, allowing us to select a high-quality subset from the original dataset.
2.2. Skill Graph Construction
Skills Extraction. We randomly sample 100K data points from NuminaMath dataset (LI et al., 2024) as the refer- ence data Dref = {d1,d2, .d... where |Drejl = 100K. NuminaMath, which comprises 860K pairs of competi- tion math problems and their solutions, is considered to be of high quality (i.e., each text is meticulously collected and annotated with accompanying chain-of -fi---f-ttf-oo thought traces) and diverse (i.e., it includes problems ranging from high- school-level to advanced compreeddooomptttioo-eeeelll) We then in- struct Qwen2. Team, 2024) identify to 10 tested mathematical skills in each data point of the reference dataset due to its excel- lent accuracy and latency. The prompt template we use is shown in the Appendix C. As a result, we collect 865K
In this work, we select NuminaMath (LI et al., 2024) as the reference dataset to construct a skill graph based on its mathematical skills. We then apply our method to three tar- get datasets: two web datasets (Open (OpenWebMath (Paster et al., 2024), OpenWebMath-pro (Zhou et al., 2024a)) and one
4 (Qwen
2

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
skills from Pres in total. An example of the data sample and its corresponding skills is shown below.
1, 2, VI), the values are similarly normalized from the number of occurrence of the skill pair (v,, vy):
Skill Extraction Example
exp
Aig - (elf .))
CONTENT: Find the greatest value of z such
Σ (aIsEE) exp
z+7. The expression when sim- plified by factoring the numerator transforms into: (a- (a-9)[++1))
that -r- 90
2
8-9
By far, we have constructed the skill graph Gska that en- capsulates the skills tested in the reference dataset and their compositional relationships. A visualization of a sub-skill graph and more statistics of the entire graph can be found in Appendix B.
-. Canceling the (2- factor on both sides, provided x + 9, we get: x + 10 2_
9-9
z+7
Multiplying both sides by (z + 7) to eliminate the fraction yields: (xr + (0) 7)
2. Ex-
panding and rearranging this equation results in: + 17x + 70 2 + x² + 17x + 68 = 0. Fac-
2.3. Data Selection via Skill Graph
toring the quadratic gives: (x++4) +) ( 13) = 0. The solutions to this equation are x 4 and x = -13.
With the reference dataset Dref = {dl, d2.... dn. and the skill graph Geau constructed in the previous step, we propose to select high quality subset Dig from the target dataset Digt (21, ,NN] The detailed process is outlined as follows.
The greatest of these solutions is -4 4
Equation solving, Factoring polynomials, Fraction manipulation, Quadratic equations, Root identification, Expression simplification, Algebracc transformation, Polynomial division, Inequality con- sideration, Solution verification
SKILLS:
Semantic Similarities Computation. Given a target data point from target dataset x, E Digt- and specific skill from the skill set vy E V, we compute their semantic similarity:
sim (Zt, Uy) = max cos(Emb(z:)) (mmb(dd))))m((())))) (3) kgu!d
Skill Graph Construction. Once we have identified suffi- cient set of mathematical skills, we construct a skill graph Gessh- = (V, E, A) where V represents node set of skills, E denotes the edge of skill co-occurrence, and A is the adjacency matrix. To reduce noise, we preprocess these extracted skills by merging those with semantic similarity .5 (Xiao et al., 2024) as the text encoder. As a result, 46,490 unique nodes (skills) are retained to form the node set V {e1, 02..... , tivi} (V) = 46, 490). Each node i/ has two attributes: (1) agntt a scalar indicating the skill's number of occurrence; (2) ofde, a list containing the indices of the sub-reference dataset where this skill ap- pears. For edges, we create an edge between two skills when they co-occur in at least one data point, resulting in 1,184,007 edges in total, forming the edge set E = ferj = U;) feo(U,, Uz) > 0}, where (1, uj) de- notes the co-occurrence count of (vi, U;). Each edge eij has an attribute: Gj occurrences of the skill pair. Next, we define the adjacency xVV
where cos denotes the cosine similarity, ofde is the list of indices of the sub-reference dataset where this specific skill fy appears, de is the k-th data point in the reference dataset and Emb() is the text embedding model (we use bge-larges eeaaargeee-ll..... (Xiao et al., 2024) in this work). The intuition behind this approach is to calculate the co- sine similarity between the target data o, and each of the corresponding reference dataset {dx k E wife of the node (skill) øy. The maximum similarity value is then se- lected as the similarity score sim(2;, Uy). Similarly, we compute the similarities between each data point of the target dataset and each of the skills, forming the similar- ity matrix S E N where N is the size of the target dataset and [V] is the number of nodes in the skill graph.
greater than 0.9, utilizing
Aggregation on Skill Graph. With the above formulation of sim(x;, 0y), we aggregate the computed similarities on the skill graph using the adjacency matrix A. The aggre- gated similarity simagg (I;, Uz) between the target data æ; and the skill by is calculated as:
eont, a scalar indicating the number of co-
matrix A E x for the skill graph. Specifically, for diagonal elements {Ai, i 1, IV],, the values are normalized from the number of occurrence of the skill i using softmax function with temperature coefficient T:
simago (Ii, Uj) Ag.sim(mi vy)
+
Σ (LEA(1e)
A,ssim(Z ox), (4)
exp
where (v,)) denotes the neighbor set of Uy in the skill graph. The intuition is that the aggregated similarity simagg (12, 22) is not solely based on the original similar- ity sim(a;; U2) and the skill importance Aj.- but also in-
A,, = ou ont T)-
IVI
pone
exp
For non-diagonal elements {Aij 1 i * J, i.j =
3

---

(8) MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
Reference Dataset
Skill s:
Node: Calculus
a fo - 36
Solution:
that $gix)
Prompt Engineering
Extract 1-10 math skills
Inverse sunction fonaaional number operation B.sinplsEying expressions
Construct Skill Graph
Dount: 1234
I0s: 3,3,00...
Flass, ne need to Sunction
Edge: (Calculus, Probebilityy Count: 1234
ID:1
ID:1
TD:2
10:2
I0:n
Target Dataset
Skill Graph
High-quality Subset
Sty
Probasitity of getting aoes, rive cards poler hard (Part
Rank Select Top %%
coin has been placed in each the ielld 2ut aton top. leaee many cot newd to be oves,
kings and Quees cards polnt
Compute Scores
Pretrain Language Model
semcsise Soarala nethed
retied
ID:1
1D:1 Score:0.9
ID:2 5coze:1.1 T0:1 Score:0.
2
Score:1.
2
ID:0
IDin Score:0.
10: Scoress
0.6
0.9
0.2
Plity gettinn Accs, 2 Kings ane 1 Quson in
Compute Similarity
0.8
(8x,
Aggregato
Asa A,
Sum
9. 2-1-0.220.44 0.140.1
(Peat II) 5o zowocked iy foundaa So
0.8
Ars
0.9 Ass + 0.7 Ass
0.9
Are
I0:1
0.5
0.1
0.7
Figure 2. An illustration of MASS, In the first step, we instruct Qwen2 extract mathematical skills from each data point the reference datasets. Using these extracted skills, we construct a skill graph that captures the compositional relationships among them. In the second step, assign quality score to each data point in the target dataset. This score calculated by pooling and aggregating the similarities of the data point and the skills the skill graph. Finally, the top-ranked data points can be selected for training the large language models.
Instr-nttttuutereet Int (Qwen Team, 2024) to
cludes the weighted sum of the similarities of the connected skills, incorporating the compositional relationships of these skills. In practice, we can compute the aggregated similarity simagg (I,, 0U) efficiently using matrix multiplication:
Data Selection. Given the score vector score(Dtgl) and a selection ratio k%, we determine whether each target data point x; should be retained or discarded as follows:
keep(a;) if score(æ;) ranks in the top k% of score(Digt).
Sagg SA,
(5)
op(æ;, k)
where Sagg and S represent the aggregated and the vanilla similarity matrices respectively, with A being the adjacency matrix. For each target data point 24, its quality score is obtained by summing the similarities between zj and all the skills Ey. which is formulated as:
drop(x;) otherwise.
Finally, by applying op(æ,, k) to to all data points in the target dataset Digt, we obtain the high quality subset Diyi
Dng = {o)((z,,)) (p) k), ...,..
(9)
IV]
score(x:)) = simagg (Ii, U3)-
(6)
2.4. Why MASS Works?
As stated in the introduction, the goal of our method is to en- that each data point receives higher score if satisfies two key conditions: (1) it covers broader range of impor- tant mathematical skills, and (2) it captures a richer, more nuanced compositional understanding of these skills. These two principles serve as the foundation for evaluating and ranking the quality of data points within the target dataset. Here provide an explanation of how MASS operates based these two foundational principles.
Next, we can compute the score vector score for the entire target dataset Digt = {I1, D2, [NN in matrix form as follows:
score (Digt) (score(x)) score(zz)))) score((xx) (..oorrr))
vl
S
Sggg.''
/=1
Given the adjacency matrix of the skill graph A, according

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
to Eq. 4 and Eq. 6, the quality score of given data point
Table Hyperparameter settings.
consists of two components:
Hyperparameter
Vinyllama 1B
Mistral-17
M
Peak Learning Rate
8c-5
2c-5
score 2))
simagg og (I, U/)
Context Length
2,048
4,096
Batch Size
512
256
[V| A sim 13)
M
Learning Rate scheduler
Cosine Annealling
Σ
Σ
A A,ssim(z, mk)-
Optimizer
Adam
Warmup Ratio
0.01
Weight Decay
0.1
(10)
The first component weighted sum of similarities to these skills, where the weights are given by the diagonal elements of the adjacency matrix. As described in Eq. 1, if skill vi is more important (i.e., it appears more frequently), the corresponding element Aj.. has larger value. Conse- quently, the more important skills data point covers, the greater the value of the first component.
our experiments are provided in Table 1.
Baselines. We compare MASS with various data selection baselines which are from different method categories. They include (1) AutoDS (Zhang et al., 2024b): This method uti- lizes meta- prompted language models as zero-shot verifiers to autonomously evaluate and select high-quality mathemat- ical content. (2) DSIR (Xie et al., 2023): This approach estimates importance weights in a reduced feature space for tractability and selects data through importance resampling based on these weights (3) RHO-1 (Lin al., 2024): This method selects suitable tokens during the training process using selective language modeling and training dynamics. (4) ProX (Zhou et al., 2024a): This approach refines pre- training data at scale through program generation using language models. (5) BM25 (Robertson & Zaragoza, 2009); This method ranks and selects documents based on query term occurrence and rarity across the corpus that extends TF-IDF by considering term frequency saturation and doc- ument length. (6) Heuristic rules used in the construction of FineWeb corpora (Penedo et al., 2024): This method employs heuristic rules for data selection. Note that we mainly implement these baselines for TinyLlama-1, 1B considering the compute resources.
Similarly, the second component also a weighted sum of similarities, where the weights correspond to the non- diagonal elements of the adjacency matrix, which encode the compositional relationships between skills. According to Eq. 2, if a skill pair kj. is more important (it appears more frequently). the corresponding element A has a larger value. Therefore, the more important compositional information a data point covers, the larger the second com- ponent becomes.
3. Experiments
3.1. Experiment Setup
Training Corpora. We utilize different types of pretrain- ing datasets, including math-related web data Open Web- Math (Paster et al., 2024), OpenWebMath-pa (Zhou et al., 2024a) and synthetic data Jiuzhang3.0 (Zhou et al., 2024c). OpenWebMatt a dataset that encompasses large portion of mathematical text from the internet. It has been filtered and extracted from over 200 billion HTML files on Common Crawl, resulting in refined set of 6.3 million documents containing a total of 14.7 billion tokens. OpenWebMath-poe an enhanced version of Open using the ProX framework, and contains approximately 5 bil- lion high-quality tokens. Jiuzhang3.6 -. 1---0O--oou-PPP PT-CoT is synthetic dataset composed of 3.8 billion high-quality math- related tokens, including question-answer pairs sourced from a variety of origins.
Evaluation Setup. We compare the model performances on nine mathematics-relann benchmarks used in RHO-1 (Lin et al., 2024) and ProX (Zhou et al., 2024a). The evaluation is conducted using the same implementation and includes few-shot ofain-of-though (CoT) examples.
further refined
The code available: github..om//ljjazheenn 17/MASS.
3.2. Main Results
The main performance comparison results are presented in Table 2. On average, across nine math and reason- ing downstream tasks, the vanilla TinyLlans ama-l. 1B and Mistral- 7B achieve performance scores of 13.3% and 53.0%, respectively. To assess the effectiveness of our ap- proach, we continue pretraining these two models using both the original datasets and the selected subsets. Specifically, for Open WebMath, Open WheenWehttthpoe pro, and Jiuzhang3.0, we apply MASS to select the top 30%, top 60%, and top 70% of tokens, respectively, based on the varying qualities
Training Setup. We continue pretraining (CPT) TinyLLama 1. (Zhang et al., 2024a) and Mi Mistral-7R (Jiang et al., 2023) using both origi- nal datasets and the selected subsets identified by various methods. To accelerate the training process, we leverage DeepSpeed Zero-2 (Ren et al., 2021) and Flash Attention 2 (Dao, 2023). The specific hyperparameter settings used in
5

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
Table The main experimental results. '' nyl. ama-1. and Mi selected subsets of Open WebMath, OpenWebMath pro, and Jiuzhang3.0, 0. The bolded entries indicate the best results within each setting. * indicates that results from ProX (Zhou et al., 2024a)
continuously pretrained using both the original and
Unique
Trained
MMLU STEM
SAT MATH
Dataset
Method
GSM8K
MATII
SVAMP
ASDiv
MAWPS
TAB
MQA
Avg.
Tokens
Tokens
TinyLi amaammamma. -1.15
w/o continual pretraining
2.7
2.8
10.9
17.9
20.5
12.5
14.0
16.3
21.9
13.3
14.6B
14.68
5.2
3.0
20.7
31.4
41.0
10.1
19.5
37.5
20.3
RULE"
15B
4.5
2.8
17.5
29.4
39.3
12.4
25.0
18.4
RHO-1
5.0
23.5
41.2
53.8
18.0
Maih
14.6B
3.0
40.2
19.6
14.9
14.6B
2.6
24.1
37.8
12.1
22.3
Aut
4.9B
14.6B
2.4
22.9
39.2
52.7
18.4
13.8
23.2
24.1
22.7
MASS
4.9B
14.6B
4.4
24.9
41.4
21.5
13.9
20.3
25.0
23,9
5.1B
14.6B
8.6
3.0
23.8
40.2
51.6
19.6
14.9
26.1
25.0
23.6
OpenWchMath
DSIR
3B
14.6B
8.8
3.2
24.1
41.5
53.1
18.9
14.3
27.6
27.5
24.4
-pro
AutoDS
3B
14.68
9.1
4.5
22,4
40.8
54.3
23.2
13.1
26.5
28.0
24.7
MASS
3B
14.68
10.2
5.8
23388
42.3
57.9
25.3
15.3
27.0
34.4
26.9
3,4B
6.8B
22.3
19.0
46.4
60.1
73.2
29.6
19.1
24.0
34.4
36.4
DSIR
2.4B
6.8B
24.5
21.3
48.2
63.9
74.4
28.8
19.2
22.1
33.6
37.3
Jiuzhang3.0
AutoDS
2.4B
6.8B
26.7
20.8
51.3
66.7
73.5
31.1
19.3
22.4
32.8
38.3
MASS
2.4B
6.8B
30.1
24.8
52.5
69.1
80.7
32.9
20.4
22.7
34.4
40.8
M1 Balsraa a-1-
w/n continual pretraining
41.1
10.6
64.9
68.5
87.3
54.8
33.9
49.9
65.6
53.0
14.4B
9.6B
44.5
19.0
60.6
68.4
87.8
50.5
44.5
50.9
56.2
53.6
OpenWeb.Math
MASS
4.8B
9.6B
47.7
23.2
64.6
74.7
90.5
55.7
50.7
$2.6
65.6
58.4
5.1B
47.1
21.8
63.2
73.7
89.5
42.6
52.2
56,2
Math
BM25
5.1B
44.7
24
63.1
73
86.1
49.1
49.8
52.6
67.1
-pro
5.1B
42.1
21.6
73.4
86.8
55.3
51.9
70.8
MASS
3B
5.1B
53.2
25.6
67.0
76.8
90.4
57.6
51.8
$4.5
81.2
62.0
3.8B
3.8B
66.4
39.4
82.9
85.9
90.8
35.3
61.8
40.1
50.0
61.4
Jiuzhang3.0
MASS
2.7B
3.8B
70.0
43.8
84.3
85.7
93.7
35.7
63.5
46.9
65.6
65.5
of the datasets. The selected data are repeated proportionally ensure that the total number of tokens used for training remains consistent.
grammatical correctness, noise reduction, or textual coher- ence), MASS operates deeper semantic level. By alyzing reference data, MASS identifies the specific skills model requires or lacks and strategically curates training subsets to address these gaps. This targeted bisee se- lection contrasts sharply with existing approaches, which often overlook semantic efficacy in favor surface-level text quality.
Table 2 demonstrates that all of these math datasets enhance the models' mathematical reasoning capabilities while our approach achieves the best performance across these methods. Moreover, compared to the original datasets, us- ing MASS-selected datasets further improves the results by 3.6%, 3.3%, and 4.2% for TinyLlama-i 1B, and 4.8%, 5.9%, and 4.1% for Mistral-76 1- Web- Math, OpenWebMath pro, and Jiuzhang3.0, respectively. Additionally, we present the performance of the interme- diate checkpoints trained on both the original and selected datasets, as shown Figure? 3. Clearly, terms of effi- ciency, our selected datasets achieve the same results the original dataset, but with 43%-71%% % fewer trained to- kens. In terms of effectiveness, approach outperforms the baseline by 3.2%-5.9%.
3.3. Analysis MASS
In this section, we delve into an in-depth analysis of our data selection approach.
1. COMPUTE EFFICIENCY OF MASS
We conduct a detailed brcakdown of the computational costs associated with MASS, using OpenWibMaa t the target dataset and Mi distral- -7B as the target model. As summa- rized in Table 3, the pre pre-processing steps (including skill extraction, embedding, graph construction, and subset se- lection) account for less than 3% of the total computational budget, with the majority allocated to the actual model train-
What distinguishes MASS from other baselines is its fine- grained, skill-centric selection approach. While conven- tional methods focus on superficial linguistic quality (e.g.,
6

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
CPT OPT'TinyLiama on Open WebMath
CPT Tinyl LoyyLJamn Jaama on Open Mabt-at-rrr
CPT TinyLlamaa on Jiuzhang3A1
24-
CPT Cligg CPT MASS
CPT on Ohip
20-
+3.6%
71% less tokans
3.25
CPT MASS
57% less tokons
Arg. Performnne (%)
71% less tokensk
16
16
20-
CPT on Orig.
CPT Mistral on Mathh
CPT Mistral on OpenWebMatt-ppro
CPT Mistral Jiuchang
62
Ong CPT MASS
CPT on MASS
CPT on MASS
64
Arg. Performance (55
4,8%
-5. 5.9%%
43%
66% less tok.
tolsggs
40
52-
20-
Takens
Trained Tokens (R)
Trained Tokens (B)
Figure The average performance continual pretrained Tin TinyLlama-1, 1B and Mistral with respect to the number of trained tokens OpenWebMath, OpenWebMath- pro and Jiuzhang3.0 datasets.
ing phase. This lightweight pre-processing overhead makes MASS highly practical for large-scale applications.
superior performance. We provide case studies of the data points with the highest and lowest scores from these three datasets in the Appendix D.
In comparison, existing baselines either do not disclose their pre-processing costs or rely solely on CPU-based opera- tions, making difficult perform a precise, end-to-end computational comparison. However, in Figure 3 of our paper, it is clear that MASS achieves more than 40% greater efficiency than all baselines, which means even if we do not consider the pre-processing steps of MASS, it remains the most efficient and effective approach.
Avg. scores: 1.07
Avg. scores: 0.95
Avg. scores: 0.92
Density
Jiuzhang3.0
OpenWebMath-prro
3.3.2. QUALITY OF DATASETS
3
OpenWebMath
Since we have computed the quality score for each data point in the target datasets, randomly sample 10K data points from each and present the score distributions in Figure 4. It is clear that Jiuzhang3.0 0 is the highest-quality dataset, with an average score of 1.07, followed by Open WebMath- at 0.95, and Open WebMath at 0.92. This ranking aligns with the model performance results shown in Table De- spite being trained on only 6.8 billion tokens, Jiuzhang3.0 achieves an average performance of 36.4%. significantly out- performing Open MatMa (20.3%) and OpenWebMath-ppm (23.6%), which were trained on approximately 15 billion tokens. This highlights the effectiveness of synthetic data in the pretraining of large language models, demonstrating that even with a smaller token count, synthetic data can drive
0.5
0.6
0.7
0.8
0.9
1.0
1.1
1.2
Data Scores
Figure 4. Distribution of scores of different datasets.
3.3.3. IMPACT OF SELECTION RATIO
Given the varying quality of the datasets, we conduct series of experiments to examine the impact of the data selection ratio our method. We continue pretraining TinyLlama-1e 1B using subsets of OpenWebMath-ppp and Jiuzhang3.0 selected different ratios. The subsets are

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
Table 3. Compute efficiency of MASS
Operations
A100 GPU Hours
Extracting skills from reference dataset
~4
Embedding reference dataset and target dataset
005
Constructing the skill graph
2 CPU hours
Selecting high-quality subset from the target dataset Training Mistral- -Bral1-BB on the high-quality subset (~10B tokens)
4
990
repeated to reach a total of 15.3 billion and 6.8 billion tokens, respectively, as reported in Table The results, presented in Figure 5, clearly demonstrate that for Jiuzhang3.0, rel- atively higher selection ratio leads to higher performance, while for Open WebMath-pra prot, relatively lower selection ratio leads to higher performance. This pattern aligns with the quality of the datasets, indicating that for higher-quality datasets, maintaining a higher selection ratio is more advan- tageous. Performance trends exhibit a spike-shaped pattern, which can explained by the quality-diversity tradeoff: (1) very low selection ratios, the high quality of the selected data comes at the cost of reduced diversity, which nega- tively impacts model performance. Prior research (Zhang et al., 2025) has consistently highlighted the critical role of diversity in data selection methods; and (2) at excessively high selection ratios, the inclusion of excessive noisy and low-quality data degrades performance.
diagonal elements. This could be because more important nodes tend to have more connections with other nodes in the graph. As a result, even in the absence of the diagonal elements, the compositional information derived from the non-diagonal elements still provide valuable insights into skill importance. This allows the model to retain some level of understanding of the skills' significance, which helps mitigate the performance drop.
45
Jiuzhang
DebtWebMatl-pro
40
Avg. Performance (%)
35
Jiuzhang Orig.: 36.4%
30
3.3.4. IMPACT OF SKILL GRAPH
25
As discussed in Section 2.4, key step in our method for ad- hering to the two principles is the aggregation of computed similarities on the skill graph. This aggregation enables the incorporation both the importance of individual skills and their compositional relationships. To assess the effective- ness and necessity of this step, we perform an ablation study with two variants.
OWM-pro Orig.: 23.6%
20
30
40
50
60
70
80
90
Selection Ratio (%)
Figure 5. The average performance of TinyLl ama-1 1B with respect to different selection ratios. Note that models are trained the same amount of tokens.
(I) w/o Diag. In the adjacency matrix A, we set the diagonal 2, ... IV]] to O. This means that
elements (Ai, we do not account for our first principle, which pertains to the importance of individual skills.
Table 4. The average performance of 'ii nylll ama-1 1. with respect to different adjacency matrix forms.
Method
Open WebMath-pr
Jiuzhang3.0
(2) w/o Non-diag. In the adjacency matrix A, we set the non-diagonal elements {A,, + j. i,j = 1,2. [V] to O. This indicates that do not take into account our second principle, which emphasizes the importance of the compositional relationships among skills.
MASS
26.9
40.8
w/o Diag.
26.5(0.4%.))
39.6(1.2%()
w/o Non-diag.
24.1(,.8%..
37.2(2.1%) (%))
3.3.5. IMPACT OF SIMILARITY CALCULATION
From the results in Table 4, we observe that performance consistently declines when any part of the adjacency matrix omitted, highlighting that both of our principles play a crucial role in the data scoring and selection process. Ad- ditionally, non-diagonal elements have more significant impact, as indicated by a performance decrease of 2.8% and 2. % with their omission, compared to 0.4% % and 1.2% for
In Eq. 3, define the similarity between data point and skill by taking the maximum value of the cosine similarities. Below we present two alternative approaches:
(I) Mean Embedding. Compute the mean embedding for the relevant sub-reference dataset associated with a skill, and define the similarity as the cosine similarity between
8

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
the skill's mean embedding and the data point embedding:
Graph-based methods are increasingly valuable in LLM data curation. Skill-it (Chen et al., 2024) demonstrates that LLMs learn skills in a natural order and constructs a skills graph for efficient training. MathScale (Tang et al., 2024) builds a concept graph and uses random walk for instruction data generation. Unlike these approaches, our work focuses on using skill graphs for data selection during continual pretraining to enhance LLMs' reasoning abilities.
Emb(n;)= = mean Emb(da), kEv
(11)
s2m 0z) = cos(Emb(z;)) Emb(ry)).
(12)
(2) Name Embedding. Use the embedding of the skill name directly, rather than the corresponding sub-reference dataset, and define the similarity as the cosine similarity between the skill name embedding and the data point embedding:
5. Implications and Future Works
sim(r;, 12) = co)(Fam((:;),, Emb(name(w)))
(13)
A key distinction between LLMs' mathematical and reason- ing capabilities, compared to domains like academic writing or role-playing. is that they can be analyzed through the lens of specific underlying skills. The success of MASS high- lights the advantages of incorporating these skills and their compositional relationships into the data selection process, rather than relying on general selection methods.
From the results in Table 5, we observe that both alternatives reduce model performance, but to different extents across the two datasets. For Open WebMath-ppppa -ro, the average de- crease is 0.6%, while for Jiuzhang3.0, 3., the average decline is 3.2%. This discrepancy might be attributed to the dataset for- mat. Specifically, Jiuzhang3.0 a question-anse annserr pair for- mat, similar to the reference dataset NuminaMath. whereas OpenWehMath-po consists of web-based math-related data. This may result in less discriminative similarities between a data point and the sub-reference dataset of skill. As result, the use of mean embeddings or name embeddings has has a greater impact on the Jiuzhang3.0 dataset compared to Open WebMath-pra
There are several potential directions for improving our method in the future. For domains with distinct and well- defined skills, such as coding or biomedicine, MASS could be naturally suitable and helpful. However, adapting MASS for areas like creative writing or role-playing, where skills are less clear-cut, remains a significant challenge. Also, to enhance the quality and correctness of extracted skills, we could filter generated skills using a predefined taxonomy. Another crucial aspect not yet addressed by MASS is di- versity. While this paper focuses on a continual pretraining paradigm, future applications in small-scale data selection for instruction tuning reinforcement learning will neces- sitate combining skills with greater diversity. Additionally, incorporating skill difficulty could boost the model's perfor- mance. Currently, relying solely on skill frequency as an importance metric might be limiting; less frequent but more complex skills may require more data for large language models to effectively master.
Table The average performance of Ti TinyLlammml 1B w.r.t. different similarity calculation methods.
Method
Open WehMath-pro
Jiuzhang3.0
MASS
26.9
40.8
Mean Embedding
26.4(0.5%L) 26.2(0.7%L)
37.4(3.4%%
Name Embedding
37.8(3.0%%)
4. Related Work
See Appendix A for detailed related work discussion.
Data selection for LLM training falls into heuristic-based and model-based approaches. Heuristic methods, such as language filtering, and deduplication, serve as preprocessing steps, while model-based methods further refine data by improving quality, diversity (Li et al., 2024b; Liu et al., 2024). However, despite the variety of methods available, they often focus on the general domain and neglect the specific aspects of mathematical reasoning. Recent work (Zhang et al., 2024b) evaluates text quality for mathematical intelligence using LLMss, whereas our study approaches the problem from a skill-based perspective.
6. Conclusion
In this paper, we propose MASS, a novel and data-efficient framework designed to enhance the mathematical capabil- ities of large language models (LLMs) by leveraging skill graph-based data selection. First, we employ prompt engi- neering techniques to extract and categorize mathematical skills from high-quality reference dataset, constructing a structured skill graph. This graph is then used to score and rank the target dataset, selecting the top-ranked subset for training LLMs. Our empirical results show that MASS sig- nificantly improves both the efficiency and effectiveness of the training process, leading to models with stronger math- ematical reasoning abilities and faster convergence. This approach offers a more data-efficient way to enhance LLMs, particularly in specialized domains like mathematies.
Mathematical reasoning is crucial for assessing LLMs' intel- ligence (Shao et al., 2024). High-quality datasets play key role, such as OpenMathInstruct-2 (Toshniwal et al., 2025) and MathPile (Wang et al., 2024), which offer curated math text. Furthermore, our approach leverages a mathematical skill graph to efficiently select higher-quality data.

---

Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., Li, X., Su, X., Lin, X., Li, X. Q., Jin, X., Shen, X., Chen, X., Sun, X., Wang. X., Song, X., Zhou, X., Wang, X., Shan, X., Li, Y. K., Wang, Y. Q., Wei, Y. X., Zhang, Y., Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Yu, Y., Zhang, Y. Shi, MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
Impact Statement
This paper presents work whose goal is to advance the data efficiency of large language model pretraining, which could facilitate wider accessibility of AI technologies and con- tribute to more sustainable AI development practices. There are no negative societal consequences of our work, as it is specifically designed to improve the data efficiency of ex- isting technologies without introducing any harmful ethical, privacy, or social risks.
Y., Xiong, Y., He, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Ou, Y., Wang, Y., Gong, Y. Zou, Y., He,
Y., Xiong, Y., Luo, Y.. You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z. Z., Ren, Z., Sha, Z., Fu, 7., Xu, 7.,, Xie, 7.,, Zhang. 7., Hao, 7.,, Ma, 7., Yan, Z. Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Pan, Z., Huang. Z., Xu, Z., Zhang, Z., and Zhang, Z. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
References
Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gu- nasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. Phi-4 technical report. arXiv, 2024.
Barbaresi, A. Trafilatura: A web scraping library and command-line tool for text discovery and extraction. In ACL, 2021.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In NAACL, 2019.
Brown, T. B. Language models are few-shot learners. arXiv, 2020.
Fan, S. and Jaggi, M. Irreducible curriculum for language model pretraining. arXiv, 2023.
Chen, M., Roberts, N., Bhatia, K., Wang. J.. Zhang. C., Sala, F., and Ré, C. Skill-it! a data-driven skills framework for understanding and training language models. In NewrlPS, 2024.
Gu, Y., Dong, L., Wang, H., Hao, Y., Dong, Q. Wei, F., and Huang, M. Data selection via optimal control for language models. In ICLR, 2025.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. In NeurIPS, 2021.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv, 2021.
Dao, T. Flashattention-2: 2: Faster attention with better paral- lelism and work partitioning. In ICLR, 2023.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute- optimal large language models. In NeurlPS, 2022.
DeepSeek-AI, AI, Guo, D., Yang, D., Zhang, H., Song. J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai,
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C.,
Chaplot, D. S.,, Casas. d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv, 2023.
D. Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao,
G. Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang.
Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. Bag of tricks for efficient text classification. In ACL, 2017.
H. Ding, H., Xin, H. Gao, H. Qu, H., Li, H., Guo, J., Li, J., Wang, J., Chen, J., Yuan, J., Qiu, J., Li, J., Cai, L., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., Gao, K. Guan, K., Huang, K., Yu, K., Wang. L., Zhang, L., Zhao, L., Wang, L., Zhang, L., Xu, L., Xia, L., Zhang, M., Zhang, M., Tang, M., Li, M.. Wang, M., Li, M., Tian, N., Huang, P.. Zhang, P., Wang, Q. Chen, Q., Du, Q. Ge, R., Zhang, R., Pan, R., Wang. R., Chen, R. J., Jin, R. L., Chen, R., Lu, S. Zhou, S. Chen, S., Ye, S., Wang, S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou, S., Wu, S., Ye, S., Yun, T., Pei, T., Sun, T., Wang. T,, Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Xiao, W. L., An, W. Liu, X., Wang, X., Chen, X., Nie, X.,
Kang, F., Just, H. A., Sun, Y., Jahagirdar, H., Zhang, Y., Du, R., Sahu, A. K., and Jia, R. Get more for less: Principled data selection for warming up fine-tuning in llms. In ICIR, 2024.
JI, Beeching, E., Tunstall, L., Lipkin, B., Soletskyi, R., Huang, S. C., Rasul, K., Yu, I., Jiang, A., Shen, Z., Qin, Z., Dong, B., Zhou, L., Fleureau, Y., Lample, G., and Polu, S. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions, 2024.
10

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
Li,J, Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E., Keh, S., Arora, K., al. Datacomp- Im: In search of the next generation of training sets for language models. In NeurlPS, 2024a.
Toshniwal, S., Du, W. Moshkov, I., Kisacanin, B., Ayrapetyan, A., and Gitman, I. Openmathinstruct-2: Accelerating ai for math with massive open-source in- struction data. In ICLR, 2025.
Li, M., Zhang, Y., Li, Z., Chen. J., Chen, L.,, Cheng, N., Wang, J., Zhou, T., and Xiao, J. From quantity to qual- ity: Boosting LLM performance with self-guided data selection for instruction tuning. In NAACI, 2024b.
Wang, Z., Xia, R., and Liu, P. Generative ai for math: Part i-mathpile: A billion-token-scale pretraining corpus for math. In NewrIPS, 2024.
Wettig, A., Gupta, A., Malik, S., and Chen, D. QuRating: Selecting high-quality data for training language models. In ICML, 2024.
Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C.,, Yang, Y., Jiao, J., Duan, N., and Chen, W. Not all tokens are what you need for pretraining. In NuerIPS, 2024.
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., and Nie, J.-Y. C-pack: Packed resources for general chinese embeddings. In SIGIR, 2024.
Liu, W. Zeng, W., He, K., Jiang, Y., and He, J. What makes good data for alignment? a comprehensive study of automatic data selection instruction tuiigg In ICLR, 2024.
Xie, S. M.. Santurkar, S., Ma, T., and Liang, P. S. Data selection for language models via importance resampling, 2023.
OpenAl. Introducing openai ol, 2024. URL https://
openai wwm.c///oooo.ooo. com/ol/.
Xie, S. M., Pham, H., Dong, X., Du, N., Liu, Lu, Y., Liang, P. S., Le, Q. V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pre- training. In NeurIPS, 2024.
Paster, K., Dos Santos, M., Azerbayev, Z., and Ba, J. Open- webmath: An open dataset of high-quality mathematical web text. In ICLR, 2024.
Yang, A., Zhang, B., Hui, B., Gao. B., Yu, B., Li, C., Liu, D.,
Penedo, G., Kydlíček, H., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., Wolf, T., et al. The fineweb datasets: Decanting the web for the finest text data scale. In NeurlPS, 2024.
Tu, J., Zhou, J., Lin, J., Lu, K., Xue, M., Lin, R., Liu, T.,
Ren, X., and Zhang, Z. Qwen2.5-man technical report: Toward mathematical expert model via self-improvement.
arXiv, 2024.
Zhang, C., Zhong, H., Zhang, K., Chai, C., Wang, R., Zhuang, X., Bai, T., Qiu, J., Cao, L., Yuan, Y., et al. Hamessiig diversity for important data selection in pre- training large language models. In ICLR, 2025.
Qwen Team. Qwen2.5: A party of foundation models, 2024. URL https: /tttttss://qwee http///wwnlllmm www.ee..m.iithu..... qwen2.5/.
Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S.. Zhang, M., Li, D., and He, Y. (Zero- offload]: Democratizing (billion-scale) model training. In USENIX ATC, 2021.
Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model. arXiv, 2024a.
Zhang, Y., Luo, Y., Yuan, Y., and Yao, A. C.-C. Autonomous data selection with language models for mathematical texts. In ICLR, 2024b.
Robertson, S. and Zaragoza, H. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 2009.
Zhou, F., Wang, Z., Liu, Q., Li, J., and Liu, Program- ming every example: Lifting pre-training data quality like experts at scale. In NuerIPS, 2024a.
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. Deepseek- math: Pushing the limits of mathematical reasoning in open language models. arXiv, 2024.
Zhou, J., Ghaddar, A., Zhang, G., Ma, L., Hu, Y., Pal, S., Coates, M., Wang, B., Zhang, Y., and Hao, Enhancing logical reasoning large language models through graph- based synthetic data. arxiv, 2024b.
Tang, Z., Zhang, X., Wang, B., and Wei, F. Mathscale: Scaling instruction tuning for mathematical reasoning. In ICML., 2024.
Zhou, K., Zhang, B., Wang, J., Chen, Z., Zhao, W. X., Sha, J., Sheng, Z., Wang, S., and Wen, J.-R. Jiuzhang3.R O0: Efficiently improving mathematical reasoning by training small data synthesis models. In NuerIPS, 2024c.
Tirumala, K., Simig, D., Aghajanyan, A., and Morcos, A. D4: Improving I1m pretraining via document de- duplication and diversification. In NewrIPS, 2023.

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
A. Related Work
A.1. Data Selection for LLMs
Data selection for LLM training can generally be categorized into two approaches: heuristic-basee and model-based. Heuristic- methods, such as text extraction (Barbaresi, 2021), language filtering (Joulin et al., 2017), and data deduplication (Tirumala et al., 2023), are typically the initial stages of pre-processing web-crawled data. Model-based data selection methods follow these stages to further refine the data by considering various factors, such quality (Li et al., 2024b), diversity (Liu et al., 2024), and distribution matching (Kang et al., 2024). Model-based selection can be conducted at the sample level (Fan & Jaggi. 2023; Gu et al., 2025), where a complete piece of text is retained or discarded, or at the token level (Lin et al., 2024), where only useful tokens are incorporated into LLM training. These methods are generally targeted at the general domain, often overlooking the specific features of mathematical and reasoning problems. Closest to our work, (Zhang et al., 2024b) proposed using designed prompts for LLMs to evaluate whether a text exhibits mathematical intelligence and if it is suitable for educational purposes in mathematics. In contrast, our work addresses this problem from the perspective of skills.
A.2. LLMs for Mathematical Reasoning
Mathematical reasoning is a cormerstone for assessing the fundamental cognitive capabilities of human intelligence. Recently, there has been a notable surge in the development of LLMs (Shao et al., 2024; Yang et al., 2024) aimed solving mathematical reasoning problems. The release of OpenAl's o-series models has further highlighted the potential, significance, and opportunities for enhancing the reasoning capabilities of large models. From the dataset perspective, numerous nigh-qualij datasets have been developed to bolster the mathematical reasoning abilities of LLMs. For instance, in the SFT stage, Optrutathnstrruct222 (Toshniwal et al., 2025) features 14 million question-solutiin pairs, using various augmentation methods from the MATH (Hendrycks et al., 2021) and GSM8k (Cobbe et al., 2021) datasets, making nearly eight times larger than the previous largest open-source math reasoning dataset. In the pretraining stage, MathPile offers diverse and high-quality math-centric corpus comprising about 9.5 billion tokens. This corpus was developed through meticulous data collection and processing, including comprehensive suite of preprocessing, prefiltering, language identification, cleaning, filtering, and deduplication. Our method can be further utilized to select these existing datasets through the math skill graph, thereby efficiently and effectively training LLMs.
A.3. Graphs in Data Curation for LLMs
Graphs, as form of data that encapsulate compositional relationships, can be valuable in analyzing the acquisition of knowledge during the training process of LLMs. Skill-it (Chen et al., 2024) demonstrates that language models follow a natural order when learning a set of skills from their training data and builds a skills graph for data-efficient LLM training. However, the skills they extracted are relatively general and human-designed, not specifically centered on mathematical skills. MathScale (Tang et al., 2024) instructs LLMs to construct a concept graph and uses a graph random walk algorithm to sample a sub-graph, which is then used generate an instruction tuning dataset. Similarly, Zhou et al. (Zhou et al., 2024b) further demonstrate the effectiveness of baahh-baeed instruction data generation by constructing dataset-related graphs. These two methods differ from ours, which focuses on data selection for the continual pretraining stage of LLMs.
12

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
B. More Information of Skill Graph
B.I. Visualization of Skill Graph
We show sampled sub-skill graph with 10 nodes.
Sequence of terms
Tetrahedron Properties
mathematical matilling
Fractional representation of parts
Exponentiar probability
system lineaf equations
Event probability
Optimizatioo-problemms
theory
function définition
Figure 6, A visualization sub-skill graph.
B.2. Statistics of Skill Graph
Table 6. Skill Graph Properties
Property
Value
Number nodes
46,490
Number edges
1,230,497
Density
0.001
Clustering Coefficient
0.776
Modularity
0.587
Average degree
52.94
Maximum degree
11,691
Minimum degree
Degree standard deviation
199.69
13

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
C. Skill Extraction Details
The prompt used to extract mathematical skills from the referenced dataset presented below.
Prompt Template to Extract Skills
Please assume the role of math teacher and analyze the provided question with the following steps:
1. Determine if the text involves mathematical knowledge, reasoning, or problem-solving skills. Respond with "YES" or "NO". Identify 1-10 concise, general mathematical knowledge points being tested.
Note:
Ensure each knowledge point abstract and generalized.
2. Avoid using verbs; focus noun phrases (e.g., "function symmetry" instead of "amalyzing function symmetry"),
3. Avoid specifics like numbers, angles, or exact values. Focus overarching concepts and techniques.
4. Keep each knowledge point to maximum of 10 words.
Provide your assessment in one JSON format:
"jjon
"math relevance": "YES" or "NO", "knowledge points"; ['pointl", "point2", ...]
*.*
Provided text: (TEXT) JSON output:
14

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
D. Case studies
show the data samples with the highest and lowest scores in Jiuzhang3.0, ApeenWeMatt--ao and Open WebMathh datasets, respectively.
Text with the highest score in Jiuzhang3.0 3.0 dataset
Problem:
Consider sequence of numbers where each term is the result of applying the function ã5, which maps any integer to its smallest nonnegative integer equivalent when divided by 5. For example, 75 = 2 because 7 + 5 has remainder 2.
Let S(a,r) represent the sum of (a 5)5 for & ranging from 0 r, where and are integers. Given a 17 and 12, calculate the value of S(17, 12).
Solution
To solve for S(17, 12), we use the formula provided in the proposition:
a(r + 1) - r(r+1)
S a rr> that Therefore, use the second case of the formula.
r(a,)) =
a(rr + 1 5) 5r r(r+1)
Since 12 and 17,
First, calculate a(r + 5):
17 (12 5) = 17. 8
87-8 = 136
Next, calculate 5r r(r+1),
2
5 12 12-(12+1) 60 12:13
60-78 78 -18
Now, sum the two parts:
136 + (-18) 136 18 = 118
Thus, the value of S(17, 12) is: 118
Explanation
In this problem, we needed to apply the formula for the sum of terms involving the function - l5 The key step was recognizing which case the formula to use based on the values of a and r. By breaking down the problem into simpler calculations and carefully handling the algebraic expressions, we able to logically determine the correct value for the This required clear understanding of the function properties and the conditions for applying different formula cases. This approach of systematically analyzing the problem using established methods will help students in their mathematical journey.
Text with the lowest score in Jiuzhang3.0 dataset
Sven Erlander was born in 1934 and died in 2021. How many years old was Sven Erlander when died?
Solution
To find out how old Sven Erlander was when he died, we need to follow these steps:
1. Determine Sven Erlander's birth year: Sven Erlander was born in 1934.
2. Determine the year he died: Sven Erlander died in 2021.
3. Calculate the age difference: We subtract his birth year from the year he died:
2021 1934 87
(14)
So, Sven Erlander was 87 years old when he died.
15

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
Text with the highest score in OpenWebMath-prig dataset
New exercises and problems in MathematiccSSettembrr 2003
New exercises
Maximum score for each exercise (sign "C") is points.
coin has been placed in each field 3x3 table, showing tails on top. At least how many coins need be turned over, so that there are three collinear (row, column, diagonal) heads or three collinear tails?
Is there regular polygon in which the shortest diagonal equals the radius of the cireumscribed circle? Peter's telephone number (without area code) is 3 12837, that of Paul is 310650. If each of these numbers is divided by the same three-digit number, the remainders will be equal. That remainder is the area code of their city. What is the remainder? (Note: Area codes are two-digit numbers Hungary.)
The angles A and of a convex quadrilateral ABCD are equal, and angle C is a right angle. The side AD is perpendicular to the diagonal BD. The lengths of sides BC and CD are equal. What the ratio between their common length and the length of side AD?
Solve the equation 2x log
0 the set of real numbers. (Suggested by É. Gyánó, Budapest)
New problems
The maximum scores for problems (sign "B") depend on difficulty. is allowed to send solutions for any number of problems. but your score will be computed from the largest scores each month. We have coloured each positive integer either red blue. The sum of two numbers of different colours is always blue, and their product is always red. What colour is the product of two red numbers? (3 points)
Find the locus of those points in the plane of given square at which the square subtends an angle of 30% (3 points) Prove that if m and n are integers, m + n² + m + The convex hexagon ABCDEF cyclic, and AB = BC = a, CD = DE = b, EF F.A = c. Prove that the area of the triangle BDF half of the area of the hexagon. (4 points)
cannot be divisible by 9. (3 points)
The number in base-a notation is 0,3737 ... 0,37 and the number G in base-a notation is 0,7373... 0,73. The same numbers written in base-b notation F 0,2525... ... 0,25 and G 0,5252 ... 0,52. Determine the numbers and b. (4 points)
Is there right-angled triangle such that the radius of the incircle and the radii of the three excircles are four consecutive terms of an arithmetic progression? (4 points)
The point P lies on the perpendicular line segment dropped from the vertex A the regular tetrahedron ABCD onto the face BCD. The lines PB, PC and PD are pairwise perpendicular. In what ratio does P divide the perpendicular line segment? (3 points)
Given the real number write the expression x² + Lx² 1 as product of two quadratic factors of real coefficients. (4 points)
The points X, Y and Z divide a circle into three ares that subtend angles of 60%, 100 and 200 at the centre of the circle. If A, B and C are the vertices of triangle, let MA and MB denote the intersections of the altitudes drawn from the vertices A and B with the circumscribed circle, and let FC denote the intersection the bisector angle C' with the circumscribed circle. Determine all the acute triangles ABC for which the points M A, MB and FC coincide with the points X, Y and Z in some order. points)
Let x; 1, 3/1 2, 21 = 3, and let Fn+1 Vn + 1. Mn+1 nn + zu .n+1 Fn Vo.
for every positive integer Prove that at least one of the numbers T200, y/o0 and 2200 is greater than 20. points)
New advanced problems
Maximum score for each advanced problem (sign "A") ) 5 points.
is the isogonic point of a triangle ABC (the point in the interior of the triangle for which LAIB LBIC /CTA A 120%). Prove that the Fuler lines of the triangles ABI, BCT and CAT are concurrent. Prove that if b, are positive real numbers then
1
a(1 b) +
+
2
b(+c)
c(1 + a)
1 abc
We have selected few 4-element subsets of n-element set A, such that any two sets of four elements selected have at most two elements in common. Prove that there exists a subset of A that has at least V6n elements and does not contain any the selected 4-tuples as a subset.
16

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
Text with the lowest score in pro dataset
Article Content
Ieo McKern
Reginald Leo McKern (March 16, 1920 July 23, 2002), better known simply as Leo McKern, was an Australian actor who appeared in numerous British television programs, movies and over 200 stage roles.
Some notable roles:
17

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
Text with the highest score in OpenWebMath dataset
Part
Question List
For all nonzero integers and m, let the operation She defined by
m
Im =
m72
1
Quantity A
3 2
Quantity
There are 30 students in Mr. Peterson's gym class. 14 of them play basketball, play baseball, and play neither basketball baseball.
Quantity A
The number of students who play both basketball and baseball.
Quantity B
6
In regular n-sided polygon, the measure single angle is
(n 18080
71
The degree measure of an angle in regular 10-sided polygon is how much greater than the degree measure of an angle in a regular 6-sided polygon? For all real numbers a and b, the operation @ is defined by
+ b 2a b
What is the absolute value difference between (3 o 1) @ 2 and 6 6 3?
Quantity A
The units digit of 72
Quantity B
The units digit of 327_
Of the employees at company, 60 percent were men and, of these, were still employed after recent corporate restructuring. If the number of women who were still employed after the restructuring was five times the number of men who were employed after it, what percent of the women were still employed after the restructuring? q is even, then #g = -2; If q is odd, then #q = 4. a and are integers such that odd.
#(aa)
18

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
Text with the highest score in OpenWebMath dataset Part 2 Quantity
#b
f(x) 3x
g(z) + 1
is an integer such that -10 v r - -1.
f(g(z))
Quantity B
s(((z))
Three digits have been removed from each of the following numbers. If 25, which of the numbers is equal to 3 2(n-1)?
a
90
r
Quantity
8
X is the center of the circle above, then what the sum of the measures of ZWXY and ZVX In the figure above, e of d. What is the value of c? What is the value of x??
+ b
Quantity B
180
In the figure above, what is the value of w??
What is the area of a regular hexagon with side length 8?
||
95
Quantity B
8
In the figure above, line is parallel to line k. If f -130 and g 70, then h
12.
10 11 12 13 14 15 16
36 37
19

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
Text with the lowest score in Wath dataset- Part 1
MAS S H (television)
Inspired by the film of the same name, MH (Mobile Army Surgical Hospital) was an American television series that aired on CBS from September 17, 1972 to February 28, 1983 (251 episodes). The sitcom was about outfit of medical workers stationed in Korea during the Korean War. Much like the movie, it combined elements of "zany" comedy and darker antiwar message.
20th Century Fox head William Self gave the show to producer Gene Reynolds and comedy writer Larry Gelbart. This combining of genres was unusual for television series of its time, and was in fact an early example of what later became known as a dramed. The show's producers did not even want a laugh track in the show, but this proposal was rejected by CBS; however, as compromise, the emergency room scenes were shown without laugh track, and in fact the show was shown in the United Kingdom entirely without the use of canned laughter. The DVD release offers choice between laugh-enerusted and laugh-free soundtracks. Many of the stories were based on real life tales told by hundreds of actual surgeons interviewed for the show.
At the end of its first season the show ended 46th in the ratings. CBS responded by moving the show to Saturday night between hits All in the Family and The Mary Tyler Moore Show. MH ended the next 9 of 10 seasons in the top 10.
The series used the theme song "Suicide is Painless", which was taken from the film, though without the lyrics. Some said the series seemed to be more about the Vietnam War, given the attitudes of the characters, than about the Korean despite its Korean setting. However, even the movie somewhat anachronistic, given its of such early seventies fashion the fu manchu mustache. The show's producers have said that the movie was really about all wars, not just Korea Vietnam.
The series was followed by After starting Morgan, Farr, and Christopher reunited in a midwestern hospital after the war. It was not well regarded, and was quickly cancelled.
The show featured Alan Alda, who wrote and directed some of the most emotional and wward-winning episodes; Out of all the starring characters Hawkeye, Hotlips, Klinger and Father Mulcahy were the only ones in the show for its entire series. McLean Stevenson left the show at the end of the third series, and his character Henry Blake was discharged and sent home. In the final scene of his last episode it was reported that Blake's ' plane had been shot down and he was killed. Actor Wayne Rogers left the series after the end of series three due to disagreements about his character. At the beginning of the fourth series Hawkeye was informed by Radar that Trapper had been discharged, and audiences did not see Trapper's S departure. At the same time Col Potter was assigned to the unit as Commanding Officer replacing Blake, while BJ Hunnicut was drafted in as Trapper's 's replacement. Larry Linville left during the first episode of series as Frank Burns became mad and was drafted away from the 4077th. Charles Winchester, a snobbish but highly skilled surgeon, was his replacement. A couple of episodes into series eight Gary Burghoff left the series, and Radar discharged. Existing character Klinger took over Radar's post, the character thereafter enjoying more prominent position in the series.
Gary Burghoff (Radar O' O'Reilly) was the only M A "'S" actor to reprise his role from the movie, retaining his extraordinary ability to detect the arrival of choppers transporting wounded long before anyone else could hear a thing. When Burghoff left the series, the company clerk role was taken up by Jamie Farr (Corporal (later Sergeant) Klinger, whose cross-dressine dressing never got him the discharge he wanted). Viewed one of the most popular sitcoms in history, ensemble show, became increasingly centered around Alan Alda's character, Hawkeye Pierce. The show survived many personnel changes over the course of the show, and in fact the series changed its tone over the years. Initially, the series placed more emphasis on the "zany" elements, while in the later years it focused more on serious elements and character development; however, both the serious and the comedic elements were present throughout the history of the series. In the later years the story lines began to also stale and the show's comedic edge had dulled even though the show was still in the top of the ratings. Alda and his fellow actors then voted to end the series with the 10th season but CBS and 20th Century Fox offered the actors shortened 11th season leading up to an opportunity for them say goodbye in grand finale.
still a very popular syndicated series. Originally seen
20

---

MASS: MAthematical Data Selection via Skill Graphs for Pretraining Large Language Models
Text with the lowest score in WatnWe dataset Part 2
Goodbye, Farewell, and Amen The final episode was titled "Goodbye, Farewell, and Amen" and was first broadcasted on February 28, 1983. The episode was 2.5 hours long and was viewed by over 125 million Americans (77% of viewership that night) which made "Goodbye, Farewell, and Amen" the most watched television episode history up that time.
The finale started in the waning days of the war with Hawkeye in mental hospital who was finally driven over the edge by a bus ride gone terribly wrong. The bus passengers, who were refugees, were in danger of being discovered and executed by a North Korean patrol. Hawkeye scolds the refugees to be quiet but a baby begins to whimper and its mother responds by smothering the child. Hawkeye repressed this by replacing the memory the baby with that of a small animal.
Winchester captured rag-tag bunch of Chinese musicians who he teaches Wolfgang Amadeus Mozart's "Quintet for Clarinet and Strings" to them. However, he later sees all the musicians killed and as a result views classical music as stained for him (classical music his number one solace during the war).
Corporal Max Klinger, best known for constantly trying to be discharged via Section 8, finally decides to stay in Korea to be with his new wife even though he finally got his release papers (along with most of the 4077).
The final and perhaps most memorable scene was between Hawkeye and BJ Hunnicut. Hunnicut was not able to say goodbye and Hawkeye mocked him for this failure. Both men lament that they will on opposite sides of the country after they go home and conclude that they will probably never see each other again. They tearfully embrace for the last time and Hawkeye boards helicopter and lifts off. Hunnicut rides off motoreycle and as the helicopter ascends Hawkeye sees final message from his longtime friend spelt with stones the sandy soil, "G.ODBYE."
21


Reviews:
[{"review_id": "sTdjqifro6", "reviewer": "Reviewer_NxxW", "summary": "This paper introduces a method for math data selection in pre-training. It begins by extracting a skill graph from a high-quality reference dataset, then utilizes this graph to score a larger dataset and filter out high-quality samples.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NxxW,\n\nThank you for your thoughtful feedback and positive recognition. Below, we respond to each of your comments in detail.\n\n1. **Weakness 1.1:** It would be better to use **stronger base models.**\n\n    **A:** Thank you for your suggestion. We chose Qwen2.5-7B as a stronger base model, and are currently pretraining it on both original OpenWebMath and MASS-selected OpenWebMath. The training is still in progress, and we will share the results in the discussion phase once the training is complete.\n    \n    Additionally, we would like to explain why we did not opt for these SOTA models in the first place. The datasets we use are all open-sourced and have probably already been used in the pre-training stage of these strong models. For instance, Qwen2.5 was trained on 18 trillion tokens [1]. So, even if we selected high-quality data from OpenWebMath and continue pretraining Qwen2.5 on it, we would likely observe minimal to no performance improvements. Conversely, a relatively smaller and less capable model might be more suitable to test the data selection methods effectively.\n\n3. **Weakness 1.2:** More comprehensive evaluation: It would be beneficial to include results on corresponding results after **instruction tuning**.\n\n    **A:** Thank you for your suggestion, but we would like to emphasize the three reasons why MASS prioritizes the data selection of pre-training over instruction-tuning and did not provide results after instruction-tuning:\n\n    - First, pre-training datasets are typically much larger than instruction-tuning datasets (e.g., OpenWebMath with 14.7 billion tokens vs. MetaMathQA with 103 million tokens). Due to their scale, pre-training datasets often contain a significant amount of noisy, repetitive, and irrelevant low-quality data, whereas fine-tuning datasets are generally more compact and high-quality, gaining little benefit from selection.\n    - Second, most of the knowledge LLMs acquire comes from the pre-training stage, while the instruction-tuning stage primarily focuses on aligning with human preferences and formatting. Thus, refining the vast pre-training datasets through the lens of skills is more effective and directly impacts model performance.\n    - Third, in our experiments, we used the Jiuzhang dataset, which follows a QA format similar to an instruction tuning dataset but at a much larger scale. This successfully demonstrates MASS's effectiveness even in an instruction-tuning-like setting to some extent.\n\n2. **Question 1:** It would be benificial to include results on performance on **additional tasks, such as general instruction following and coding**. / How can this method be adapted for **other data domains?**\n\n    **A:** Thank you for your suggestion. While we cannot fully implement the method across other domains during the rebuttal phase, our future direction is to adapt MASS to other tasks and domains. \n    \n    Since the core of MASS is a skill graph so it would be naturally suitable for domains where distinct and clear 'skills' exist. For example, in the coding area, we may extract skills such as *['CSV file handling','data visualization','SQL query construction',...]* (generated from *iamtarun/python_code_instructions_18k_alpaca dataset* by DeepSeek). In the biomedical area, skills may include *['alcohol-related disorders','neurotoxic substance effects','thiamine deficiency',...]* (generated from *FreedomIntelligence/medical-o1-reasoning-SFT* dataset by DeepSeek). \n    \n    After skill extraction, we can construct the corresponding skill graph and apply MASS’s data selection approach. Some minor adjustments (e.g., prompts, graph construction) may be needed for domain-specific adaptations. However, for domains like creative writing or role playing, where skills are harder to define, the current MASS framework may not be suitable.\n\n4. **Question 2:** How would higher-quality math data impact the **overall model capabilities**?\n\n    **A:** We believe higher-quality math data can impact the model capability in two ways:\n    - Improving math ability: High-quality mathematical data can directly enhance a model's mathematical reasoning skills, just as shown in our manuscript.\n    - Improving general reasoning ability: mathematical data is inherently logical and structured, so it can also help models better grasp logical relationships in complex tasks such as scientific document processing and code generation. For example, pre-training on datasets containing mathematical proofs and formulas can improve a model's ability to handle tasks requiring logical reasoning.\n\n\nWe thank you again for your thoughtful advice, which has strengthened our work. We will include relevant details and analysis in the next version of our manuscript. Should you have any further questions or require additional information, we are happy to address them.\n\nSincerely,\n\nMASS authors\n\n[1] https://qwenlm.github.io/blog/qwen2.5/", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes, they evaluated the models before and after continued pre-training on various math benchmarks. However, it would be beneficial to also include results on:\n\n1. Corresponding results after instruction tuning for each model.\n2. Performance on additional tasks, such as general instruction following and coding, to provide a more comprehensive assessment.", "theoretical_claims": "There are no proofs.", "experimental_designs_or_analyses": "Yes, the experiments on continued pre-training are well designed. However, in addition to the evaluation concerns mentioned earlier, it might be better to use stronger base models, such as DeepSeek-Code, which serves as the starting point for DeepSeek-Math's continued pre-training. Other 7B models stronger than Mistral is also better.", "relation_to_broader_scientific_literature": "This paper presents a practical and straightforward pipeline for pre-training data selection, particularly for math. It would be interesting to see this approach applied to stronger base models and larger-scale training.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength:\n\n1. The writing is clear and concise, without unnecessary information.\n2. They conduct extensive ablation studies.\n3. The method seems to be simple and practical.\n\nWeakness:\n\n1. It would be better to use stronger base model with larger-scale training and more comprehensive evaluation.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. How can this method be adapted for other data domains?\n\n2. How would higher-quality math data impact the overall model capabilities?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DLjRpISllx", "reviewer": "Reviewer_CEvW", "summary": "This paper proposes an approach called MASS for selecting mathematical training data. The paper takes a high-quality reference math dataset, obtains each problem's skills (by prompting a LM), and constructs a skills graph, from which we can read off how frequent each skill is in the reference dataset and which skills commonly occur together. MASS uses the skills graph to select samples from a target dataset, prioritizing samples that are associated with the most commonly occurring skills in the reference dataset, and samples that have compositional knowledge. Experiments show that continually pretraining with MASS outperforms other common data selection methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer CEvW,\n\nThank you for your thoughtful feedback and positive recognition. Below, we respond to each of your comments in detail.\n\n1. **Weakness 1 Q1 / Question 5:** Does this approach work for **other domains**?\n\n    **A:** Yes, it does work for other domains. Please see our reply to reviewer NxxW on Question 1.\n3. **Weakness 1 Q2 / Question 6:** Does this approach work when a practitioner wants to define their **own set of skills?** What sort of **skills work the best** for this method? (i.e., topic-based, reasoning-based, style-based).\n\n    **A:** For the first question, yes, it does work with minor adjustments. We propose two potential solutions:\n    - We can directly prompt LLMs to identify relevant skills from a pre-defined set for each math sample. However, this faces challenges when the skill set is large (e.g., our 46,490 distinct skills) due to: (1) prompt length constraint and (2) model's instruction following ability. \n    - We can opt for a BERT-like classfier model instead of autoregressive models. So we need to collect a sufficiently large training set of math data and their corresponding skills pairs to train the BERT model which can further be used at scale for pre-training corpus. The data collection can be done either by utilizing LLMs or manual annotation by experts.\n\n    For the second question, while our current version doesn’t specify skill types during extraction, this might be a direction worth trying. We believe that hierachical topic-based skills may work well. Specifically, we first identify math data point to algebra, geometry, statistics and so on. Next, we further classify algebra data to abstract algebra, lie algebra and so on at a finer level. Thus it also incorporates flexibity and hierachical knowledge.\n    \n    However, this remains speculative and what works the best still needs comprehensive designs and experiments to figure out. We leave this for future work.\n5. **Weakness 1 Q3:** Does the **choice of LM** matter for extracting skills?\n\n    **A:** During our initial implementation, we compared the SOTA proprietary model GPT-4o and open-sourced Qwen2.5-72B-Instruct-GPTQ-Int4 and found that the extracted skills are similar. Using the same example in our manuscirpt, here we show the outputs:\n    \n    *GPT-4o:* [\"rational expressions\",\"factoring polynomials\",\"root identification\",\"solving rational equations\",\"quadratic equations\",\"expanding algebraic expressions\",\"solving equation\",\"excluding restricted values\",\"considering inequality\"]\n    *Qwen2.5:* [\"Equation solving\", \"Factoring polynomials\", \"Fraction manipulation\", \"Quadratic equations\", \"Root identification\", \"Expression simplification\", \"Algebraic transformation\", \"Polynomial division\", \"Inequality consideration\", \"Solution verification\"]\n    \n    Actually what matters more is the prompt template and the output parsing process. After careful evaluation, we selected Qwen2.5-72B-Instruct-GPTQ-Int4 for its optimal balance between cost and performance. This choice allowed us to focus resources on extensive prompt engineering, which ultimately contributed to MASS's significant performance improvements.\n    \n    While the current results are satisfactory, we acknowledge that skill extraction could potentially benefit from stronger LLMs and prompt template. \n8. **Weakness 3 Q2 / Weakness 3 Q3 / Question 7:** A clear intuitive reason regarding **why MASS should outperform these other approaches**; **why are skills a better axis** for data selection than, i.e., directly prompting an LM to score things or a token-level data selection approach?\n\n    **A:** Please see our reply to reviewer 2U9L on Weakness 1.\n11. **Question 2:** What is the **impact of prompting for different skills** (i.e., a pre-defined skills taxonomy, or having the model output 5 or 20 skills rather than 10)?\n\n    **A:** For a pre-defined skills taxonomy method, please see our reply to Weakness 1 Q2.\n    \n    For the number of skills extracted, we need to emphasize that our prompt asks the LLM to output 1-10 skills (not a fixed 10), so the ouput actually varies based on the sample. Moreover, the core contribution of our approach resides in its skill-based framework rather than in quantitative aspects of skill extraction, so we chose not to focus extensively on determining the exact number of skills.\n\n## Due to space constraints, we are unable to address all questions in this section. We will provide other responses during the discussion phase. Thanks for your understanding.", "claims_and_evidence": "I have no concerns about the claims made regarding whether MASS works; however, I do have some questions about the claims regarding *why* MASS works, and would appreciate any clarification. The paper claims that MASS encourages selecting samples that cover compositional information, but I am not fully convinced that this is the entire explanation of what is actually happening. \n\nWhen looking at the first term in equation 10, my interpretation of MASS is that it uses the frequency of skills, $[A_{11}, A_{22}, ...,]$ as a representation of a reference dataset. With just the first term, the score function encourages selecting points whose associated skills (as captured by $sim(x, v_j)$) are those that frequently occur in the reference dataset; i.e., we select target dataset samples that are similar to the reference dataset samples in terms of the skills they exhibit. The second term in the scoring function appears to cover a neighborhood of v_j, which feels like its a robustness mechanism, rather than something that explicitly encourages compositionality. That is,using $\\mathcal{N}(v_j)$ in addition to $v_j$ feels like we are scoring the sample based on a coarser grained skill/\"skill family\" in the skills graph, which provides robustness against if individual skills are too narrow/poorly defined. Mathematically, the point I am trying to get across becomes clearer if we rewrite equation 10 slightly: $score(x) = \\sum_{j=1}^{|V|} \\sum_{v_k \\in v_j \\cup \\mathcal{N}(v_j)} A_{jk} sim(x, v_k)$. For each skill $v_j$, you are summing over the skills graph entries corresponding to its neighborhood (including itself). \n\nTo further understand why MASS works, I have two questions/thoughts:\n- For equation 10, what if the second term was $A_{jk} sim(x, v_k) sim(x, v_j)$? That is, we'd increase the score of $x$ a lot if it was similar to both $v_k$ and $v_j$, and $v_k$ and $v_j$ tend to occur together a lot in the reference dataset. This feels to me like it would capture more of this compositional knowledge you claim.\n- It would be interesting to investigate if enforcing a looser skill taxonomy and only using the first term would essentially result in the same sort of data being selected.", "methods_and_evaluation_criteria": "See \"Claims and Evidence\" above. The evaluation criteria makes sense, and the proposed method makes sense overall, besides the following two points (rephrased from above):\n- For equation 10, what if the second term was $A_{jk} sim(x, v_k) sim(x, v_j)$? That is, we'd increase the score of $x$ a lot if it was similar to both $v_k$ and $v_j$, and $v_k$ and $v_j$ tend to occur together a lot in the reference dataset. \n- I see the appeal of making the LM identify skills in an unsupervised manner---you don't require much domain knowledge. However, I wonder if making the LM list 10 skills per sample results in hallucinations/poorly defined skills. That is, is there some benefit to defining a set of skills/level of granularity (for example, skill = the name and level of the math class you would learn about this problem in), and getting the LM to adhere to this taxonomy?", "theoretical_claims": "Not applicable, no theorems in paper.", "experimental_designs_or_analyses": "The experimental setup appears well-structured, although I had a few concerns:\n- Compute efficiency analysis: how do these data selection methods perform if we control for compute in the data selection process? It would be interesting to compare the performance of MASS to the performance of allocating the compute used for MASS's data selection to just train on more random data from the target dataset.\n- AutoDS baseline: do you prompt using the same LM for AutoDS and MASS? If I remember correctly, AutoDS uses Qwen-72B base language model while MASS uses Qwen2.5-72B-Instruct-GPTQ-Int4. More details on implementation of baselines would be appreciated. \n- Additional baselines: additional baselines that select data based on matching some representation of a reference dataset, like LESS https://arxiv.org/abs/2402.04333 or BM25, could be helpful.", "relation_to_broader_scientific_literature": "The contributions are highly relevant to broader scientific literature, since data selection is a critical part of the LM development pipeline. The idea of skills as a way to capture different characteristics of data and matching along skill distributions is interesting and builds on important lines of work that examine how models learn from data. It provides an alternative to existing, less interpretable data selection algorithms, which involve embedding the reference dataset, training on it, computing gradients on it, and so on.\nI do think the paper would be more interesting/stronger if the authors investigated if skills graphs could be applied to other domains, such as code, science, or general natural language.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths: \n- Doing data selection by extracting the skills from a reference data and selecting the target samples that are most aligned with these skills is a very interesting and novel idea. It provides evidence towards this skills-based view of how models learn from data, which is both scientifically and practically important. \n- The method performs very well and outperforms other data selection approaches. \n- The paper is generally well-written and easy to read.\n\nWeaknesses:\n- It is not completely clear when this approach works; the paper is lacking recommendations for how to use this approach in practical settings. \n     - Does this approach work for other domains?\n     - Does this approach work when a practitioner wants to define their own set of skills? What sort of skills work the best for this method? (i.e., topic-based, reasoning-based, style-based). \n     - Does the choice of LM matter for extracting skills?\n- It is also not completely clear why this approach works; see my comments above regarding my interpretation of the score as encouraging fuzzy neighborhood-based skill alignment, rather than compositional knowledge. \n- There should be more information on how baselines were implemented and why these baselines were chosen. Moreover, I don't think there is a clear intuitive reason stated regarding why MASS should outperform these other approaches; why are skills a better axis for data selection than, i.e., directly prompting an LM to score things or a token-level data selection approach?", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. Understanding the claim of encouraging compositional information: For equation 10, what if the second term was $A_{jk} sim(x, v_k) sim(x, v_j)$? \n2. What is the impact of prompting for different skills (i.e., a pre-defined skills taxonomy, or having the model output 5 or 20 skills rather than 10)?\n3. How does MASS compare against other approaches when we match the amount of end-to-end compute used to train the model?  \n4. Clarification: do you prompt using the same LM for AutoDS and MASS? \n5. Can MASS be applied to other domains, like code, science, or natural language?\n6. What types of skills work well with MASS, and what skills do not? \n7. Why were these baselines chosen and why should one expect MASS to intuitively outperform them?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N3kqIqHnJm", "reviewer": "Reviewer_2U9L", "summary": "The paper proposed MASS, a novel mathematical skill graph construction method for selecting data for pretraining LLMs in the math domain. MASS prompts a strong LLM to generate nodes of skills from a reference dataset, and then construct an adjacency matrix (as a graph) using the dataset statistics. Then the graph can be leveraged to calculate quality scores for the pretraining data by their representational similarity with the skills. Empirically, MASS outperforms other pretraining data selection baselines and full-dataset training. More broadly, MASS demonstrates the great potential of using graphs for data selection in general.\n\n\n## update after rebuttal\nI have increased my score as my concerns are well addressed by the authors.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 2U9L,\n\nThank you for your thoughtful feedback and positive recognition. Below, we respond to each of your comments in detail.\n\n1. **Weakness 1:** There is a lack of **theoretical understanding** on why the proposed approach works better than other baselines.\n\n    **A:** First, we emphasize that this work prioritizes model architecture and performance improvements over rigorous theoretical analysis, which we defer to future research.\n    \n    Second, we present some initial thoughts and general insights. In Section 2.4, we analyze why MASS works through the lens of skills. Here, we highlight two features of the MASS-selected data: \n    - It encompasses a wider range of important mathematical skills.\n    - It captures a richer, more nuanced compositional understanding of these skills. \n    \n    In contrast, other baselines adopt fundamentally different approaches: \n    - Rule-based method focuses more on pre-processing, such as language filtering and deduplication. \n    - Rho-1 improves the data quality within each sample at token-level.\n    - ProX, following Rho-1, uses programming to refine data. \n    - DSIR calculates the weight of each sample using bag-of-ngrams estimator. \n    - AutoDS prompts an LLM to select data simply based on its general math intelligence.\n    \n    We believe what seperates MASS from other baselines is that we select data at a finer semantic granularity—skills. Having the reference data, we decide what skills models really need and lack, so we can purposely select a subset that contains these skills to teach LLMs. Other baselines, meanwhile, prioritize linguistic-level quality (e.g., grammar, noise, or coherence) without considering the sementics and skill-level efficacy.\n\n2. **Weakness 2:** The **computational complexities** of constructing the skill graph and selecting from the skill graph remain unclear.\n\n    **A:** Here we break down the computational complexities of the whole approach of MASS. \n    \n    | Operations  | A100 GPU hours | \n    |--------|------|\n    |Extracting skills from reference dataset|~24| \n    |Embedding reference dataset and target dataset|~0.5| \n    |Constructing skill graph|~2 CPU hours| \n    |Selecting high-quality subset from target dataset|~4| \n    |Training Mistral-7B on high-quality subset (~10B tokens)|~960| \n\n    As shown, the computational cost of pre-processing steps is relatively low compared to model training (<3%). \n\n3. **Weakness 3:** Show **a comparison with DSIR on the larger Mistral-7B**.\n\n    **A:** Thank you for your suggestion. We began training Mistral-7B using DSIR, and the model is still in training. We will post the results in the discussion phase as soon as complete.\n\n4. **Question 1:** Clarify the **correctness** of line 323 / Explain what could **cause the spike**.\n \n    **A:** We confirm this statement is correct. 'Higher' is used when we compare Jiuzhang3.0 and OMW-pro, not with Jiuzhang3.0. In figure 5, it is clear that the best ratio for Jiuzhang is higher than that for OWM-pro. \n    \n    As for the spikes, we offer the following explanation:\n    - At very low selection ratios, while the selected data maintains high quality, the reduced diversity negatively impacts model performance. Prior research [1] has demonstrated the importance of data diversity in selection methods. \n    - When the selection ratio is too high, too much noisy and low quality data is remained thus harming the performance. \n    \n    Thus, the observed spike-shaped relationship between selection ratio and model performance is well-justified from both quality and diversity perspectives as a trade-off.\n\n\n5. **Question 2:** More **statistics about the graph**.\n    | Property   | Value|\n    |--------|------|\n    | Number of nodes   | 46,490  | \n    | Number of edges   | 1,230,497 | \n    | Density  | 0.001 | \n    | Clustering Coefficient | 0.776 |\n    | Modularity| 0.587 |\n    | Average degree | 52.94 |\n    | Maximum degree | 11691 |\n    | Minimum degree | 4 |\n    | Degree standard deviation | 199.69|\n\n6. **Question 3:** Why is the **normalization in Equ. 1 and 2** important?\n\n    **A:** We normalize the entries in equations (1) and (2) with different temperature coefficients because these original entries (raw counts of skills and co-occurrences) follow different statistical distributions. This normalization ensures both measures are on comparable scales before being combined in Equation (4) to compute similarity. Without such scaling, one type of entry could disproportionately influence the final similarity measure due to its inherently larger magnitude. The temperature-based normalization prevents either component from dominating the results while preserving their relative importance.\n\nThank you again for your thoughtful advice. We will include relevant details and analysis in the next version of our manuscript. \n\nSincerely,\n\nMASS authors\n\n[1] Harnessing Diversity for Important Data Selection in Pretraining Large Language Models, ICLR 2025.", "claims_and_evidence": "Most of the claims are well-supported.", "methods_and_evaluation_criteria": "The method is brilliant and the evaluation criteria are properly chosen.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "The experimental designs for evaluation and ablation studies are comprehensive. However, more baselines (e.g., DSIR) could be included for pretraining data selection on Mistral-7B.", "relation_to_broader_scientific_literature": "Selecting pretraining data using a carefully constructed graph that contains knowledge and statistics is a novel and brilliant idea, which is under-explored by prior works according to my knowledge. Although this paper only demonstrated its practical usability in the domain of mathematics, I believe it has a great potential for data selection for other domains in general.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well-written and well-structured. \n2. The methodology is novel and interesting. Using the adjacency matrix of the skill graph to select pretraining data not only accounts for their direct usefulness to learn individual mathematical skills, but also accounts for their potential usefulness to other skills when multiple skills are required to solve a problem. \n3. The ablation studies are well executed to demonstrate the performance gain from the graph construction. \n\n**Weaknesses**\n\n1. Although the design of the method, the empirical performance, and the ablation studies are commendable, there is a lack of theoretical understanding on why the proposed approach works better than other baselines. \n2. The computational complexities of constructing the skill graph and selecting from the skill graph remain unclear. \n3. The comparison with other baselines is only performed on TinyLlama-1.1B. It will be great to show at least a comparison with DSIR on the larger Mistral-7B.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. In line 323 (right column), the author said “for Jiuzhang 3.0, a higher selection ratio leads to better performance”, but when the ratio is too high (>70%), the performance decreases. Could you please clarify the correctness of the sentence? In addition, could you please explain what could be the reasons that caused the spike? \n2. Could you provide a bit more statistics about the graph to understand how dense is the graph, e.g., clustering coefficient?\n3. In equations (1) and (2), the entries are normalized with different normalizing factors so that they can sum up to 1 respectively. Why is the normalization important?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes MASS, a novel framework for mathematical data selection using a skill graph to enhance LLM pretraining specifically in mathematical reasoning. The authors effectively demonstrate the method's efficiency, showing substantial reductions (50%-70%) in required training tokens without sacrificing performance. Moreover, models pretrained using MASS-selected data notably outperform those trained on original datasets by up to 5.9%.

The reviewers unanimously recognize the manuscript's strengths, highlighting its clear writing, practical novelty, and solid empirical validation. Reviewers in particular note the rigorous experimental design, including comprehensive ablation studies and scalability assessments across different models (1B and 7B). Additionally, the proposed skill graph method offers interpretability and adaptability, potentially extending to other structured domains such as coding or biomedical fields.

Weaknesses pointed out by reviewers include the need for stronger base model benchmarks, additional comparative evaluations after instruction tuning, and theoretical underpinnings explaining the superior performance clearly. The authors have adequately addressed these concerns through rebuttal.

Overall, the paper offers a compelling combination of empirical results, methodological innovation, and practical relevance. Given the positive feedback from reviewers and the thoroughness of the authors’ response, the submission is recommended for acceptance at ICML.
Decision: Accept (poster)