STAT: Skill-Targeted Adaptive Training
Authors: Yinghui He
Abhishek Panigrahi
Yong Lin
Sanjeev Arora
Venue: ICLR 2026 Poster (2026)
Keywords: metacognition
out-of-distribution generalization
dataset reusability
skill-level training
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce a new fine-tuning strategy, STAT, that select or synthesize training data targeted on models' missing skills.

Abstract: Language models often show little to no improvement (i.e., “saturation”) when trained via vanilla supervised fine-tuning (SFT) on data similar to what they saw in their training set (e.g., MATH). We introduce a new fine-tuning strategy, STAT, to train such a student model by using the metacognition ability of a stronger large language model (LLM) as the teacher. The teacher uses the task dataset to create a list of skills needed for the task, and then labels each data point with its required skills (Didolkar et al., 2024). By monitoring the student’s answers, the teacher creates a Missing-Skill-Profile for the student, tracking how often they failed to apply each skill in their responses. We use this idea to build a modified training set in one of two ways. In STAT-Sel, the teacher uses an existing set of training examples but adaptively reweights them according to the Missing-Skill-Profile. In STAT-Syn, the teacher synthesizes additional examples involving missing skills. Across extensive experiments on Llama and Qwen models, our methods yield improvements of up to 7.5% on MATH, whereas SFT provides only limited gains. Furthermore, STAT enhances performance on out-of-distribution benchmarks (e.g., AIME24/25, AMC23, etc.) by an average of 4.6%. Crucially, we find that STAT is complementary to RL via GRPO (Shao et al., 2024): after the model is improved using STAT to address skill gaps, GRPO continues to add further gains. We conclude that skill-targeted adaptive training should broadly improve current training pipelines.

Published as a conference paper at ICLR 2026
STAT: SKILL-TARGETE, ADAPTIVE TRAINING
Yinghui He Abhishek Panigrahi" Yong Lin Sanjeev Arora Princeton Language and Intelligence, Princeton University {yh0068, ap34, y17690, arora)@princeton. edu
ABSTRACT
Language models often show little to no improvement (i.c., "saturation") when trained via vamilla supervised fine-tuning (SFT) on data similar what they saw in their training set (c.g., MATH). We introduce a new fine-tuning strategy, STAT, train such student model by using the metacognition ability of stronger large language model (LLM) as the teacher. The teacher uses the task dataset to create a list of skills needed for the task, and then labels each data point with its required skills (Didolkar et al., 2024). By monitoring the student's 's answers, the teacher creates a silssing-Skiln
for the student, tracking how often they failed to apply each skill in their responses. We use this idea to build a modified training set in one of two ways. In STAT-Sel, the teacher uses an existing set of training examples but adaptively reweights them according the Mi ss missings masing-Skiin 1arl111-Proflllo In S7A7-Syn, the teacher synthesizes additional examples involving missing skills. Across extensive experiments on Llama and Qwen models, our methods yield improvements of up to 7.5% on MATH, whereas SFT provides only limited gains. Furthermore, STAT enhances performance on out-of-distribution benchmarks (e.g., AIME24/25, AMC23, etc.) by an average 4.6%. Crucially, we find that STAT complementary to RL via GRPO (Shao et al., 2024): after the model is improved using STAT to address skill gaps, GRPO continues to add further gains. We conclude that skill-targeted adaptive training should broadly improve current training pipelines.
Models
MATH MATH MATH² GSM8K AMC23
MATH-perturb
AIME
Avg.
simple
hard
2024
2025
Llama-3.2-3B-Inse
44.0
18.2
21.9
73.0
21.7
33.7
12.2
33.3
16.7
30.5
+SFT
44.8
22.9
21.0
75.1
20.8
33.0
12.2
30.0
20.0
31.1
+GRPO
45.4
24.4
23.3
77.4
25.8
38.4
11.8
33.3
6.7
31.8
STAT-Sel
51.5
26.6
25.7
80.2
24.7
39.8
13.3
43.3
23.3
36.5
STAT-Syn
50.2
31.7
26.2
79.2
23.9
39.1
14.7
40.0
30.0
37.2
Table STAT significantly euhances performance of Llama-3,2-3B-Inss Instruct on various math benchmarks by targeting its missing skills in solving MATH. See Table 3 for full evaluation results.
INTRODUCTION
Language models have demonstrated remarkable success at acquiring knowledge from large-scale natural text corpora through the next-token prediction objective (Shannon, 1951). Subsequent supervised fine-tuning on domain speciffc tasks such as mathematics. However, this process often inefficient and data hungry (Kaplan et al., 2020; Muennighoff et al., 2023; Zhang et al., 2024a; Villalobos et al., 2024), with models quickly reaching a saturation point for a fixed dataset whereby further training does not help performance. Several works have suggested that this saturation happens because the loss is an average over data points, causing the training signal to diminish as the model becomes adept at most of the training examples (Chen al., 2023; Xie al., 2023a; Lin et al., 2024; Tong et al., 2024;
curated data using the same objective then leads strong performance
Equal contribution.
C Code: http://ghhh...oommmrccccooop/p/t

---

Published as a conference paper at ICLR 2026
Stage
Stage 2
Stage
SkillMap
Skill based
training question distribution
Student model
Small
dataset
Model-Specific Missing-Skin Beeiinn-Skkil--PP Prilll-Proflle
SkIII
Skri22
StiTe
1kk MATH test samples
emphasszz Skollt."
Reward fltering
Nap
Q1
Q2
QF
Q+
QUi
Synthesize
Difficult questions
Missing skill identification
skill-targeted questions
STAT-sel
Model responses
"identiff the skills chot this solution fails to laverage."
STAT-syn
Misingskil
9.5k SFT Data
ogaraar
nxbber
Figure STAT a shree-stage skill-based data selection/generation method for supervised fine- tuming (SFT). Stage 1: Identify difficult questions for each model using reward filtering model responses. Stage 2: Use frontier LLMs to analyze the model responses and build a model-specific Missi Sng 1 Profi le. Stage 3: Use pre-constructed 5' 11 1Maap map the missing skill distribution to training question distribution, which constitutes the STAT-Sel data. STAT-Syn synthesizes new questions using frontier LLMs targeted the missing skills.
Jiang et al., 2024; Xue et al., 2025; Zhang et al., 2025). In addition, there is a mismatch between the "average" next-token prediction loss used during training and the auto-regressive generation process used to evaluate performance. As result, the average loss may fail to capture the specific generation errors that remain in saturated model (Arora al., 2022; Fang et al., 2024).
To tackle this saturation, prior works have shown that adapting the training data distribution can boost performance on inference-time tasks. The key idea is to focus the next-token prediction loss on an adapted set of examples targeted towards good generation (Xia et al., 2024; Yu et al., 2024b; Lin al., 2024). This is primarily done by using embeddings or gradient- based estimates to pick training examples most relevant to reducing loss reference validation set. While these methods show benefits, they remain anchored to validation-set loss, which only coarse proxy for a model's actual generation errors. In fact, our experiments reveal that unbedding-basee methods, which adapt training data by measuring similarity to validation questions the model fails on, can be ineffective (Section 3.2) saturated models that have undergone extensive post- training, e.g., haama-nsttuct models.
We propose to address the saturation problem by drawing inspiration from pedagogical prac- tices rooted in cognitive science, which customize training to specially target the student's skill- deficiencies (Bandura Walters, 1977; Hattic & Timperley, 2007).
How can we effectively today's strong teacher models design better training strategies to help small models overcome their saturation?
We turn to a growing line of research in LLM meta-cognition (Didolkar al., 2024; Kaur al., 2024), which leverages the predictive abilities of frontier LLMs to reason about the high-level skills required to solve given task, as well the skills actually being used in the student's answer. Thus, in principle, frontier LLMs can act as the teacher who guides the training process of the student model, actively monitoring the student's competence on individual skills and adjusting their training examples.
Informal description of data design: Our pipeline starts with list of relevant skills for the problem created via teacher metacognition (Didolkar et al., 2024), and adds three stages. First, we use the teacher to evaluate the student model on a small validation set of questions and use a reward model to identify the questions that are difficult for the student. Second, create a model responses. Our first method STAT-Sel simply up- up-weights training examples using the -innnnniil-ppoaa
by using the teacher to check the missing skills in the
in effect, this guides the student to focus on their deficiencies. Our second (more expensive) method STAT-Syn uses the teacher generate synthetic training
2

---

Published as a conference paper at ICLR 2026
data using in-context examples from the validation set associated with list of deficient skills in -lsssing-Skill-Prorte
Key findings: Applying STAT-Sel and STAT-Syn teaching on Llama and Qwen models with the popular MATH (Hendrycks et al., 2021) data shows the following:
Substantial in-distribution gains: STAT achieves improvement on MATH by up to 7.5%, whereas naive fine-tuning yields negligible gains. Previous ambedding-based data selection strategies that adapt to the student's 's validation errors are found ineffective (Section 3.2).
2. Strong out-of-distribution (OOD) generalization: Improvements difficult and OOD benchmarks such as AIME24/25 and AMC23 highlight the general utility of skill-targeted training (Section 3.2).
3. Adaptivity evolving tasks: Extending the previous observation, show that STAT-Sel and STAT-Syn can be continually adapted to new, harder evaluation settings, i.e., new validation set, while still leveraging the same training set (Section 3.3).
4. Supplementary benefits over reinforcement-lerrning (RL): We show that STAT followed by RL improves upon RL-only training. such as GRPO (Shah et al., 2024) (Section 3.2). This suggests that STAT is likely prove relevant most training pipelines today.
We conduct extensive ablations to pinpoint the success of our proposed methods. A fine- ane-ggained skill-level analysis reveals that even when models have been extensively trained on MATH, they struggle on basic operation skills such as basic algebra (Section 4). By explicitly targeting these basic skills, our methods reduce such errors and improve generation performance, including on methods
out-of-distributiin tasks. In contrast, alternative approaches such embedding- often underperform, as they do not explicitly address these skill gaps (Figure 2). Thus, our findings demonstrate the robustness and broad applicability of skill-aware targeted training.
2 STAT: ADAPTING TRAINING TO MODEL'S S MISSING SKILLS
Let Q be the set of test-set questions, out of which we use subset Qual validation data and Qtest= Q) Quai evaluation data. We also have access to a set of training questions P, which has been utilized to train the student model during its pre- pre-training or post-training phase, and naively fine-tuning the model on offers little to no improvement. In our experiments, we use the test and training dataset from MATH, denoted as Q and P respectively. We aim to build targeted training dataset Ptargeted to train the model further.
Our work builds on using metacognitive abilities of frontier models from Didolkar et al. (2024), which we describe here. While hard to define precisely, skill informally defined as a basic computation necessary to solve task at hand. For example, necessary skills to solve arithmetic tasks could be addition, subtraction, multiplication and division. We will use S, a set of skills that are necessary to solve questions in Q and P. These skills are enlisted from large model like GPT-4o using an appropriate prompting strategy (Didolkar et al., 2024; Kaur et al., 2024). Then, we create Ski 11 -Map: S P be a map from skill to the of training questions that require applying the skill, which we will also get by prompting the same LLM (Achiam et al., 2023). We use the skill set S and the Skill -Map from Didolkar al. (2024).
To develop STAT-Sel and STAT-Syn, we first identify difficult questions for the student model on a validation set by analyzing its own responses. For these questions, we then use the teacher model to infer the skills that are missing from the student's 's reasoning. A skill-targeted training set is constructed by emphasizing examples corresponding to these missing skills, either via up- up-weighting samples or synthesizing new ones. Unless otherwise specified, all of our experiments use GPT-4o-mint as the teacher model.
2.1 STAGE DETECTION OF difficuli QUESTIONS VIA REWARD FILTERING
In this stage, will label question q Q difficult not for the student model. We could simply define difficult questions as the set of questions that the model gets wrong after evaluation. However, this requires access to the ground truth labels. Instead, to make technique more broadly
3

---

Published as a conference paper at ICLR 2026
applicable, we reward model classify the responses of the student model. The reward model need not be a perfect reward model; we give more ablations in Section E.1. Given a question we reward model score the response of the student model.
Reward filtering. As we primarily focus on math datasets, we assume that the model's response is composed of steps for a question g and contains the answer in its final step. We will use the reward model to output reward scores for each step. For simplicity, we will refer to the scores of the reward model frg.1 "", gg,t). Then, we use thresholds 71, 72 filter out difficult questions for the student model. We will refer to the threshold filtering function R
{0, 1}.
0,
(if)
Tgtt
(final step has low reward)
1
Σ
(or) 7 q. s T1
(average low reward across all steps)
R(g)=
(or) 3i < s.t. Tg.a $ otherwise,
(low reward at any step)
Identifying difficult questions. We define Qdifiiutt as a model-specific subset of the MATH dataset, consisting of questions with low-reward model responses Rt. To avoid training directly the test data, we use two splits of Cameut as validation and test sets:
: Difficult questions in the validation set, given by Qdifficutt n e @aal are used to label missing skills in Stage 2.
val
: Difficult questions in the test set, given by Qdifficutt n o tost are used for MATH
evaluation in Table 3.
2.2 STAGE 2: CONSTRUCTING MODEL-SPECIFIC STSSTNG-SRTTIININO
For each difficult question q in Qdiffiuutt we use a frontier LLM (GPT-4o-mini) to pre- dict the set of skills in S that are missing in the model's responses. We call this map Missing-Skiin Skiil-kroffiillll----------
S. This map will be used build STAT-Sel and STAT-Syn. See Section 4 for examples and an extensive analysis of Missing--Skili Swiik-kkoofiiiii across models, and Section C.3 for prompts.
Q
2.3 STAGE 3: SELECTING SYNTHESIZING SKILL- BASED TRAINING DATA
In this stage, we construct our skill-targeted training dataset, Ptargeted-
STAT-Sel. We create this by directly sampling questions from the training dataset P according the skills listed in the Mi ssinggskill--rree examine Mi Missing Soiig-Skki---- 1all--rrofee((( and, for every skill it contains, sample multiple questions from P that are linked to the same skill via the Skill 1ii111MM0 1-Ma. Consequently, the frequency with which skill contributes to the selection process is proportional to the number of questions associated with that skill in the wwwssiingsiiiiiprrooo
Specifically, for each question g Qdifficult we
STAT-Syn. We generate new synthetic questions using the teacher model. For each question D
g E Qdifficults we examine Mi siissiggssiil--- 1offlllrrrlle For each skill it contains, we randomly sample 3 questions from P that are linked to the same skill via the Skill-Map and ask the teacher model to create new questions and responses by referring to the sampled questions. We keep only those questions where the teacher model is consistent across at least 2 of its responses, and keep only those question-ansswt answer pairs in our training set. Detailed procedures are given in Section B.
EXPERIMENTS
3.1 EXPERIMENTAL. SETUP
Datasets. All training data for STAT and the baselines are either selected or synthesized from the MATH dataset (7.5k train / 5k test) (Hendrycks et al., 2021). In addition the original solutions
4

---

Published as a conference paper at ICLR 2026
Unique
# QA Synthetic
Method
Training Data Description
Questions
Pairs
Data
MATH-Train
7.5k
7.5k
X
MATH original training set.
(Hendrycks et al., 2021)
MATH-Augment
Augmented MATH training set with multiple teacher-rewritten solutions per question.
7.5k
9.5k
X
(TIGER-Lab, 2024)
MATH-Hard
3k
9.5k
X
Subset of MATH-AugII with Level 4-5 MATH questions.
(Sun et al., 2024)
Embed-Sel (Li ct al., 2025)
Reweighted set of MATH-Augment via upweighting training questions similar the difficult questions embedding spuce.
4k
9.5k
Embed-Se Embed-Syn
Synthetic MeTe-level QAs generated by the teacher, using training examples from Embed-Sel Sel as references.
4k
9.5k
(Jung et al., 2025)
Reweighted set MATH AugmeAt via upweighting training questions related to model's missing skills in solving the difficuli questions.
STAT-Sel
4k
9.5k
X
(Ours)
Syuthetic MATH-level QAs generated by the teacher, with train- ing examples from STAT-Sel and their associated skills as refer- ences.
STAT-Syn
4k
(Ours)
Table 2: Comparison of training data construction methods. We attach details of data construction procedure Section C.2.
provided in the dataset, we also collect three alternative versions of each answer by prompting the teacher model to re-write them three times. We further report performance of STAT and each baseline after continuing training with GRPO (Shao et al., 2024) on the same MATH training set. We randomly split the MATH test set into 1K validation and 4k test subsets, with both MATH and MATH evaluations drawn from the 4k test split. See Section 2.1 for design details.
We also evaluate our method on extensive OOD benchmarks including GSM8K (Cobbe et al., 2021), MATH² (Shah et al., 2024), MATH-perturb (Huang et al., 2025), AMC23 (AI-MO, 2025), and AIME2024/2025 (HuggingFaceH4, n.d.; HuggingFaceH5,, n.d.).
Model & Training Configuration. We focus on smaller models as testbed, as their performance remains noticeably weaker on MATH. We employ mim as the teacher model, and apply STAT on student models -llmma33.233B-Insiiii Qwen2.5-3B (Qwen et al., 2025), and evaluate under O-shot settings. We fine-tune models for epochs, with learning rate chosen separately for each method based accuracy MATH. We provide detailed hyperparameters in Section C.I, ablations on threshold sensitivity in Section E. and discussion of teacher model variants in Section E.3.
1aama-3.2-IB-Instree
(Meta AI, 2024), and
Baselines. We compare skill-aware training against several baselines. We begin with MATH-Train where the model simply trains the original MATH responses, and MATH-Augment, which substitutes the responses with teacher re-written answers. We also compare against MATH-Hard, restricting training only Level 4-5 questions. Finally, test whether skills really matter in approach", selecting training
STAT-Sel and STAT-Syn, we swap them out for an embedding- questions by their similarity to difficult validation questions from Stage 1. Please find summary in Table 2. We have attached detailed data creation procedure Section C.2 and prompts Section C.3.
3.2 EVALUATION RESULTS
We present results for Llama-3.2- Instruunt and Qwen2.5-3B in Table 3 and for 1lama-3.2-11 Instruct in Table Section D. We refer to each untrained model as 'Base Model'. Our findings can be summarized as follows.
Naive SFT provides little to no benefit. Both MATH- -raii and MATH-Augment yield at most a 1-2% gain over the base model, showing that naive SET offers negligible improvements. It is worth noting that we have systematically tuned haper-parameters for naive SIT (details attached in Section C. In fact, we observe that Qwen2.5-3B can even degrade under MATH-Train Restricting
We use Aithabb-NNIPPgtte-Qeee2--------
(Li et al., 2023)
5

---

Published as a conference paper at ICLR 2026
Methods
MATH MATH" MATH GSM8K AMC23
MATH--prrturb
AIME
Avg.
simple
hard
2024
2025
Llama-3.2 3.2ua-3.--BB. 3.2-3B--Instteeeee 3B-Instructt
Base Model
44.0
18.2
21.9
73.0
21.7
33.7
12.2
33.3
16.7
30.5
MATH-Train
44.8
22.9
21.0
75.1
20.8
33.0
12.2
30.0
20.0
31.1
MATH-AN MATH-Augment
45.2
23.9
23.8
77.8
23.8
35.1
12.5
30.0
13.3
31.7
MATH-Hard
45.6
24.9
23.3
78.2
21.6
38.0
11.8
30.0
26.7
33,3
Embed-Sel
46.0
26.5
20.5
21.6
14.7
36.7
16.7
32.8
Embed-Syn
48.8
27.3
19.5
78.4
22.7
36.9
13.3
26.7
23.3
33.0
STAT-Sel
51.5
26.6
25.7
80.2
24.7
39.8
13.3
43.3
23.3
36.5
STAT-Syn
50.2
31.7
26.2
79.2
23.9
14.7
40.0
30.0
GRPO
Base Model
45.4
24.4
23.3
77.4
25.8
38.4
11.8
33.3
3.3
31.8
MATH-Train
46.4
28.4
28.6
80.7
29.7
37.6
12.5
36.7
10.0
34.5
MATH-Augment
47.4
31.6
28.6
81.4
30.6
37,6
14.0
36.7
33.3
37.9
MATH-Hard
49.4
33.2
28.6
31.3
39.1
15.4
13.3
37.1
Embed-Sel
50.4
37.5
23.8
80.5
32,0
38.0
16.8
36.7
20.0
38.8
Fmhed-Syn
49.7
37.8
19.5
80.6
33.9
39.1
16.8
36.7
23.3
38.6
STAT-Sel
52.2
35.0
32.4
34.2
42.7
17.6
43.3
26.7
STAT-Syn
51.0
39.1
29.0
82.0
31.9
43.0
15.8
46.7
33.3
41.3
Qwen2.5-3B + SFT
Base model
55.8
45.3
34.8
80.9
26.4
43.7
24.0
23.3
20.0
39.4
MATH-Train
50.0
44.2
32.9
80.1
33.6
42.3
23.3
26.7
26.7
40.0
MATH-Augment
56.6
45.6
37.1
80.4
33.0
40.9
21.9
16.7
26.7
39.9
MATH-
56.7
45.6
31.4
33.6
23.7
30.0
16.7
Emhed-Sel
57.5
46.4
34.3
80.4
33.6
43.7
21.9
30.0
26.7
41.6
Embed-Syn
56.4
47.4
34.3
80.4
35.2
43.7
24.0
26.7
26.7
41.6
STAT-Sel
58.4
47.6
39.5
35.5
24.0
33.3
30.0
44.1
STAT-Syn
59.4
49.2
40.5
81.3
34.4
44,8
25.1
36.7
30.0
44.6
GRPO
Base model
61.6
49.8
41.0
85.1
37.7
49.8
25.8
33.3
30.0
46.0
MATH-Train
61.6
51.1
34.8
84.8
36,9
51.6
26.5
33.3
30.0
45.6
MAgm-Augment
61.0
48.2
40.5
36.3
26.2
26.7
MATH-Hard
59.0
51.1
35.7
84.2
37.7
49.8
26.5
33.3
23.3
44.5
Embed-Sel
59.7
48.9
41.0
84.3
38.4
46.6
25.8
26.7
36.7
45.3
Embed-Syn
61.4
52.3
40.0
38.8
47.7
28.0
26.7
30.0
STAT-Sel 62.8
52.1
44.8
84,8
38.8
48.7
30.1
36.7
33.3
48.0
STAT-Syn
61.8
52.4
41.9
85.6
39.2
50.9
26.9
40.0
36.7
48.4
Table 3: Improvements on various math benchmarks from applying STAT. Results under "+SFT" show the performance of SFT models trained with each method, while '+GRPO' shows the performance after applying GRPO on top of the corresponding SFT models. Our methods, STAT-Sel and STAT-Syn, achieve average gain of up 6.7%
the base model, with strong OOD performances (AMC23 results reported average 64, AIME on wass@64). Applying GRPO on top of fine-tuning with STAT further enhances these improvements by 4%. Full results are provided for Llama-3.2 1.aee-..------ttteeee --I-nnsttuut in Table 7, Section D.
supervision to only the most difficult MATH questions (Levels 4-5) also fails to produce meaningful gains. A natural idea is then adapt training toward the model's mistakes by selecting training questions semantically close to difficult validation examples. Using embedding similarity, Embed-Sel achieves only marginal over MATH-Train and MATH-Augment. Synthetic augmentation via Embed-Syn provides a small additional boost, but the overall gains remain modest.
Skill-targeted adaptive training shows substantial improvements. STAT achieves average gains of to 6.7% on Llama-3.2 3.2-3B 1---B-IInnee Instruct, 5.2% on Qwen2.5-3B and 3.4% on Llama-3.2 2-1 I-sstuuct,,
the performance of base model. On closer analysis on MATH" test set of questions, STAT-Syn substantially improves the performance of the model on difficult questions, compared to STAT-Sel, which leads improved performance overall for Llama-3.22 3.2--1..1111- 2- IB-Insrruct and Qwen2.5-3B.
show that
Benefits extend beyond MATH. On out-off-istribee benchmarks, we observe consistent improve- ments across datasets, ranging from simpler problems in GSM8K challenging competition sets

---

Published as a conference paper at ICLR 2026
STAT-Se missing skills)
Embed-Sel
MATH-trraa
Selving Equacions
Prims Number Theiry
Polynomidd Concepes
2nd
Baatic Arihhmeti Uperatione
Logarihnic and Expenratial
Primer Namber Throry
3rd
Algebratc Manipulation
Combinacarial Operntioas
Cor Knshrial Operations
40
Calculaion Convurdan
Factoring
Tranglc Grometty
5th
Alpommi Exprassion
Number Thoory
Undemtandinng Application
60.
Cantinaaris Opeutiors
Trisnal
Expananial
76
Cambinatorios Conscpta
Inequallyy
Selving Equalims
8th
Modular Anthosetc
Solving Equations
Pythagerzia
90.
Liscur Equation
Polynonial Concepti
Combinutatis Knowledge
10th
Nambber Theory
Bose Conversion
Matnx Operaioos
Figure 2: Comparison among Top 10 frequent skills present STAT-Sel, Embed-Son and MATH- questions selected on Llama-3.2-IIB-1H Sel, align poorly with the actual Top 10 missing skills of the model (i.e., skills in STAT-Sel).
The skills emphasized in both baselines, MATH--Tran and Embed-
such as AIME Specifically, STAT-Sel and STAT-Syn improve averaged OOD performances by 5.3% and 5.8% respectively, with STAT- Syn generally excelling harder tasks such as AIME and MATH This demonstrates that targeting skills generalizes extensively beyond the source training set.
Compatibility with GRPO. natural concern is whether methods work well with RL-based methods such as GRPO, which typically follows SFT (Dubey et al., 2024; Guo et al., 2025). For both Llama and Qwen, improvements from SFT on STAT have carried over subsequent GRPO, yielding average gains of up to 9.5% over GRPO on base model. Surprisingly, on Llama-3.2-IB-Instte and Llama-3.2-3B-Insttee
where GRPO alone does work well (improving #2.4%), SFT alone on STAT already produced better results than GRPO, and adding GRPO on top further boosts performance by- 44%
3.3 CONTINUAL LEARNING ON CHALLENGING BENCHMARKS
As our earlier results show, STAT already generalizes strongly to wide range of OOD tasks while using only MATH data for training. But in practice, models often face evaluation settings that grow harder time. A natural question then is: benchmarks while still using similar questions as MATH?
continue adapting the model these tougher
For our study, consider MATH-perturb-bardd We report performance for two model variants of Llama-3.2 3.2-3B.-Insttuuuu STAT-ConSel takes model trained with STAT-Sel, and trains further with a data creation pipeline identical to STAT-Sel, but ss ssing-nnni..o wwwnnngSkillPPpe. Prrofile built validation questions
17.6
17.2
17
Aocuracy
15-
with Mi wissii from MATH-perturb-Bee hard. STAT-ConSyn builds on STAT-Syn model with the same idea. both cases, the evaluation bench- mark only gives the skill profile, and the training examples still come from MATH.
13.3
13
12.2
Buss
STAT Sel
STAT Sym
STAT ConSel SunSyn
STAT
Figure 3: Continual learning results on MATH-perturb-hard. Further fine- tuning STAT models based on their missing skills unseen data yields 3-4% gain (STAT ConSel/ConSyn),
As shown in Figure 3, STAT-Sel and STAT-Syn trained models show only 1-2% improvement on MATH SeHH--peturb-haee hard over the base model performance, which reflects the difficulty of this benchmark. However, continual trained models show a larger gain of 3-4%. This shows that framework can be readily adapted to unseen test-time datasets by constructing Mi Missing-Sker Saing-Skkill Meiii--Profi file directly on them, while still using MATH training data. Thus, swirl-aware training provides flexible solution to adapt the models with more challenging evaluations while still relying on existing training datasets.
4 WHY SKILL- TRAINING WORKS
In this section, we dig into the effectiveness of our proposed skill-aware targeted training. We conduct all the ablations and analyses Llama-3.2- 3.2- B-Inrrtrcct due to limited computational resources. First, we present the wwwsinnnsiiiiipraao the student's performance uniformly across these skills. Finally, show that the bascline strategies are ineffective because of misalignment in the skill distribution in their proposed training data and the missing skills.
across all models. We then show that STAT improves
7

---

Published as a conference paper at ICLR 2026
Base model
Embed-Syn
MATIH-tran
GRIO
Soliing Equations
+2.
+72
+4.5
+1.3
+4.7
+3.5
-0.5
4.0
MATII-aagaaan augmented
STAT-Sel
Artiimetic
+13.6
-03
+7.5
+12.2
+0.2
+5.6
+1.0
462
MhTd--
STAT-5p
Embed-Sel
Algebriie
+1.2
+4,4
11
Maciprilaioa
Calculation
Algebraic Manqulation
Conversion
Calciaaionn and Conversoo
+12.7
+8.8
+6.1
-10.0
Algehmic
Dascc Arthmmticc Operations
+14.6
+5.3
+2.8
+5.4
+7.3
+4.4
+0.1
-288
Papression
Expresson
Cunbirateia
+13.5
+14.1
+10.7
+7.2
-9.7
+4,4
+9.1
+4.7
Aae moddd (6))
Contiraneis Consepss
+74
Comkinstoi a
Solving Equations
Operations
Modalar Aribmmii
668
+3.9
+4.9
+5.0
-1.8
+3.2
-0.3
-7.7
Liner Equation
+121
+0.8
+8.5
+11.3
+2.0
+6.5
+3,9
+5.5
Combinstoniee
Number Theory
Concepts
Sunber Theoy
+3.0
+8.3
-1.1
-0.9
+5.7
+3.3
-2.0
Modular
Linear Equation
STAT
Emdhdd Syn
MATH MATH MÄTH hard augmented
Antimetic
Syn
Sel
Figure 4: Trained model performances (Left) and performance gain over base model (Right) on Top 10 frequent missing skills, across training strategies on Alama-3.2-1B I.-trut- Accuracies on the left plot are normalized per skill axis for better visualization. Our approaches STAT-Syn and STAT-Sel most effective in enhancing model performance across nearly all the skills.
Models struggle with basic skills. First, we closely examine the Mi Missing- across different models, obtained at the end of Stage (Section 2.2). We present the Top 10 frequently missing skills for each model according to their Missing-SSki Skill-Profile in Figure 2 (Left) and Figure 6 (appendix D). The key observations are:
Algebra-centric skills appear at the top, e-g., manipulating equations, handling expressions, and solving linear forms. This suggests that even though both Llama and Qwen models achieve high performance on MATH, they systematically struggle with operation computations.
Most missing skills are shared across models, c.g., equation-solving skills and basic arithmetic operations are missing different model families (Llama and Qwen) and sizes (IB and 3B). However, smaller models show more frequent weaknesses in basic computational skills like arithmetic.
STAT effectively addresses models' frequent missing skills: We take Llama-3.22 3.2-1BBInsttrct as case study to examine how different training strategies impact performance across skills. frofiles le, we select the 10 most frequently missing skills and build
From its Mi wiisinggskiil corresponding evaluation sets, each containing questions annotated via the Skill Skill -Ma.. We then measure both absolute performance and performance gains under each method.
As shown in Figure 4 (Left), STAT consistently outperform all baselines across all 10 skills, whereas baseline models can even fall behind the base model on skills such as Algebraic Manipulation and Modular Arithmetic. Figure 4 (Right) provides quantitative breakdown, showing that STAT can deliver over 10% accuracy gains on skills, with the largest improvements on basic skills like Calculation & Conversion, Algebraic Expression, and Combinatoric Expressions. Notably, STAT also brings clear improvements on mnowledge-intensien skills such as Number Theory and Combinatorics.
Misalignment between baseline training data and missing skills. To investigate the rea- son behind the ineffectiveness of our baseline strategies, we adopt a skill-based evaluation by comparing the skill distribution of their training data with the model's missing skills in the -iii--irri....ee....-.----- Figure 2 highlights a clear misalignment between the model's actual missing skills (STAT-Sel) and the basclines: Neither MATH- Train nor Embed-Sel targeted the basic algebraic skills the model struggles with, even though Embed-Sel selects data directly via embedding similarity difficult questions. They prioritize more advanced and conceptual areas such as polynomials, prime number theory. and trigonometric or matrix operations. We provide concrete question examples in Section D.3 to illustrate the distinct emphasis of each skill.
8

---

Published as a conference paper at ICLR 2026
Comparing STAT to GRPO. One of our in- teresting findings in Section 3.2 was that STAT could outperform GRPO training Llama in-
Original Question (on Ellipse Properties)
Let and the foci the ellipse ku is a constant. Suppose that there is a circle which passes through Fi and Fg and which lies tangent the ellipse at two points on the axis. Compute &.
where k
struct models. Here, we compare these two approaches from a skill-hased perspective. As shown in Figure 4 (Right), although GRPO on Bama--.2--BBInnstuue also yields positive gains across nearly all the top skills, the overall ef- fect remains less pronounced compared STAT. A possible reason is that GRPO provides only coarse feedback to the model by contrasting correct and incorrect responses, whereas skill- targeted training pinpoints model weaknesses in fine-grainn line-graieed way. In light of this, one future direction is to develop a GRPO variant that in- corporates skill-based feedback into the reward.
Model Response & Missing Skill (on Sulving Equations)
y²
We rewrite in the standard form ellipse: 1 (1
We can see that the length of the semi-borizontal axis is and the semi vertical
(Correct
Therefore, we have
[ vk
Simplifying this equation, we
This equation is all values of l. Therefore, the value of uniquely determined by the given conditions. (Incorrect X, Missing skill: Solving Equations)
goti
Embed-Syn Question (on Ellipse Properties)
STAT-Syn Question
(on Solving Equations)
22 2 The ellipse "s 4 1 foci located along one the cuordi- nate axes. What is the distancce between the foci?
Solve for 0:
Case study on synthetic data. To understand why our training samples are sargl-targeted, conduct a case study of the training data.
2.
Va +
Here we compare STAT-Syn with Embed-Syn, as their data are both created with a spe- cific focus (e.g., embedding-basee similarity or missing-skill targeting).
Figure 5: Comparison between synthesized questions from Emhed-Syn and STAT-Syn.
In this example (see Figure 5). the original question centers on ellipse geometry; the model handles this part well, but showed a gap in the final though highly relevant, captures only the main topic (Ellipse Properties) through embedding similarity. By contrast, STAT-Syn leverages the missing-skill information (Solving Equations) and generates targeted question to practice it.
step. The new question in Embed-Sy
This case study demonstrates that semantic similarity, as captured by embedding based methods, is not always the right approach. Skill-targeted adaptive training provides direct way to constantly improve the model.
5 DISCUSSION
Related Works: We provide a more detailed discussion of related works in Section A. Broadly, prior approaches can be grouped into three directions. First, several skill-aware algorithms improve language models either by designing more targeted inference-time instructions or by generating synthetic data instill new skills (Kaur al., 2024; Gandhi et al., 2025; Didolkar al., 2024). In contrast, our method adapts training data toward skills that the model continues to struggle with, even after having been extensively trained.
Second, performance-awart adaptation methods adjust training data to improve efficiency and performance (Xia et al., 2024; Yu et al., 2024b; Xie et al., 2023b). However, these techniques largely focus on minimizing validation loss on target set, which is only an indirect proxy for generation -iree errors. Some attempts to remove dependence on explicit validation sets instead optimize implicit properties such as embedding or gradient diversity (Jung et al., 2025; Wang et al., 2024; Yu et al., 2024a; Ni et al., 2024). By contrast, our approach explicitly targets the model's generation mistakes through a metacognitive framework.
We also relate our work to recent approaches that adapt training by profiling model weaknesses (Zeng et al., 2025; Khan et al., 2025). These methods construct skill representations to measure student performance and guide synthetic data generation. For instance, Khan et al. (2025) study adaptive data generation agents that allocate training data across skill categories based student performance, while Zeng et al. (2025) build hierarchical skill trees by clustering instance-leve skill descriptions to generate targeted synthetic data.

---

Published as a conference paper at ICLR 2026
While these works and ours all leverage skill- profiling, the role of skills differs. Prior approaches use skills as organizational categories for summarizing correctness across instances and guiding synthetic data generation to those categories where models struggle. In contrast, to identify missing components in failed reasoning generations. We further show that modifying question distribution through missing skills, without generating new data, improve performance. Our approach is complementary and provides finer-grained strategy on adaptive training.
skills
Finally, while existing works have shown effectiveness of teacher-in-the- training (Zhou & Ai, 2024; Gu et al., 2024; Zhang et al., 2024b; Wang et al., 2023; Zhou et al., 2023; Xu et al., 2024), where a teacher provides repeated feedback to student through logits or targeted generations, our skill-aware targeted training provides an effective and cheaper way to adapt student with the teacher.
We investigate whether targeted skill-based training can improve language models when naive re-training yields little benefit. Using frontier LLM to analyze responses, we construct a skill profile and selectively re-train on relevant examples, achieving significant gains on both in- and -utto--distribution tasks. Ablations show that models often fail basic skills like algebraic computations, and STAT efficiently addresses such gaps with carefully adapting training data.
Conclusion:
Our work points to two promising directions for future research. First, since the general skill feedback identified by a frontier L.MM can effectively guide student training. it would be valuable to investigate whether these skills correspond to specific mechanistic circuits within the model. Second, while our focus has been on mathematical datasets, exploring whether STAT can also improve dimensions such as safety and interpretability presents interesting avenue for further study.
ETHICS STATEMENT
All authors of this work have read and agree to abide by the ICLR Code of Ethics. We affirm that this research was conducted in compliance with the principles of research integrity, fairness, and transparency outlined therein.
Our study focuses developing and evaluating novel fine-tuning approach for language models, targeted at improving mathematical reasoning benchmarks. The work exclusively utilizes publicly available datasets such as MATH, AMC23, and AIME24/25. These datasets are widely used in the research community and do not involve human subjects, private data, or personally identifiable information. No sensitive, proprietary, or confidential data were accessed released.
We acknowledge that research in language model training can have broader societal impacts, par- ticularly regarding potential misuse (e-g.. generating misleading harmful content). However, our contributions are specifically focused on mathematical problem-solving and skill-targeted fine-tuning, which pose minimal direct risk of harmful applications. The methods proposed are not designed for, nor evaluated on, domains involving sensitive personal, social, or political content.
We have no conflicts of interest external sponsorships that could bias the reported results. All experiments were performed under standard academic conditions with openly available resources. Our work complies with legal and ethical standards for dataset usage, algorithm development, and reporting.
We used an LLM solely to improve the clarity and readability of the manuscript text (e.g., grammar and style polishing). The model was not employed for designing experiments, analyzing data, or generating results. All scientific contributions, methodologies, and findings reported in this work are the product of the authors.
Use of LLM:
Reproducibility Statement: We have taken several steps to ensure the reproducibility of our results. A detailed description of the STAT algorithms, including pscudocode, is provided in Section 2 and Section I. The datasets used in all experiments (MATH, AMC23, AIME24/25, GSM8K, and others) are publicly available and fully cited in the references. We describe experimental setup, model configurations, training hyperparameters, and ablations in Section 3.1 and Section C.
To facilitate replication, we will release code repository along with all the STAT-Syn data if we proceed to the camera-ready version. Together, these resources provide sufficient detail for independent researchers reproduce our results and extend methods related benchmarks.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
AI-MO. AIMO Validation AMC [dataset]. Hugging Face Datasets, May 2025. URL https: / buggingfacee wwooddataset... /AT- a im- dat ion- amc. Accessed: 2025-08-
26.
Kushal Arora, Layla EI Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. arXiv preprint arXivv2204.01117 2022.
Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936, 2023.
Albert Bandura and Richard H Walters. Social learning theory, volume 1. Prentice hall Englewood Cliffs, NJ, 1977.
Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang. Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 13360000330440 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Jimenez Rezende, Yoshua Bengio, Michael C Mozer, and Sanjeev Arora. Metacognitive capabilities of IIms: An exploration in mathematical problem solving. Advances in Neural Information Processing Systems, 37:19733-19812 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Lizhe Fang, Yifei Wang. Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? arXiv preprint arXiv:2410.23777 2024.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307 2025.
Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Miniplm: Knowledge distillation for pre-training language models. arXiv preprint arXiv:2410. 172155, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprins arXiv:2501. 2025.
John Hattie and Helen Timperley. The power of feedback. Review of educational research, 77(1):
81-112, 2007.
Yinghui He, Abhishek Panigrahi, Yong Lin, and Sanjeev Arora. Adaptmi: Adaptive skill-based in-context math instruction for small language models. arXiv preprint arXiv:2505000447, 2025.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv: 003.0000000 103.03874, 2021.
Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, et al. Math bathh-erturb: Benchmarking Ilms' math reasoning abilities against hard perturbations. arXiv preprint arXiv:2502. 06455,, 2025.

---

Published as a conference paper at ICLR 2026
HuggingFaceH4. aime-2024 [dataset]. Hugging Face Datasets, n.d. URL https://
huggingface. cttaaacccccoooeeaoeoooeeooooeeee httpa//o/aaaaaaae.a datasetsss htw.eeeteeggggggeeeeeeeeeee wwwpgiiggaaeee///aaaaaaaaaaaeeae 2002 2024. Accessed: 2025-08-26.
HuggingFaceHS..
aime 2025 [dataset]. Hugging Face Datasets, n.d. URL https://
huggingface co/ dataset waaeeets//ttaaaat/eeeeeeee /tt///maaaaaaaaaaaa a mme25. Accessed: 2025-08-26.
Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and Zico Kolter. Adaptive data optimization: Dynamic sample selection with scaling laws. arXiv preprint arXiv:2410.11800 2024.
Jaehun Jung, Seungju Han, Ximing Lu, Skyler Hallinan, David Acuna, Shrimai Prabhumoye, Mostafa Patwary, Mohammad Shoeybi, Bryan Catanzaro, and Yejin Choi. Prismatic synthesis: Gradient- based data diversification boosts generalization in Ilm reasoning. arXiv preprint arXiv:2505.20161, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini arXiv:2001.0666 100361, 2020.
Simran Kaur, Simon Park, Anirudh Goyal, and Sanjeev Arora. Instruct-skillmix: A powerful pipeline for Ilm instruction tuning. arXiv preprint arXiv:2408. 14774, 2024.
Zaid Khan, Elias Stengel-Fskin, Jaemin Cho, and Mohit Bansal. Dataenvgym: Data generation agents in teacher environments with student feedback. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL / wtwp:rewwwww.ee/ooumeme.... ftr.mmorummiiee 00SnKBGTs2.
Jiazheng Li, Lu Yu, Qing Cui, Zhiqiang Zhang, Jun Zhou, Yanfang Ye, and Chuxu Zhang. Mass: Mathematical data selection via skill graphs for pretraining large language models, 2025. URL
petps://arrxee
org/ abs 159.11
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023. URL https://arxix org/abs / 2308. 03281.
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. Rho-1: Not all tokens are what you need. arXiv preprint arXiv:2404.0743 07965, 2024.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. arXiv preprint arXiv:2308.0777 07074, 2023.
Meta AI. Customizable
Llama 3.2: Revolutionizing
Edge AI and Vision with Open, URL https: / //ai www......c.om.o com/blog/
1lama- Models, 2024.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances Neural Information Processing Systems, 36:50358-500 2023.
Xinzhe Ni, Yeyun Gong, Zhibin Gou, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Explor- ing the mystery of influential data for mathematical reasoning. arXiv preprint arXiv:2404.01067 2024.
OpenAl. Gpt-4o mini: advancing cost-efficient intelligence. gpt-4or ialoominn--.. www.inniavvaaninnnncoeem -wwoost-fffiiientt wtw.iiiiiienntteellleeeeeeeeeeee
com/ woom/index... index/ 2024.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan
Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025, URL https: / rxiv..vv............. orgo arg/ 2ooo/220 2241 151
12

---

Published as a conference paper at ICLR 2026
Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Jiatong Yu. Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, et al. Ai-assisted generation of difficult math questions. arXiv preprint arXiv:240722000, 2024.
Claude Shannon. Prediction and entropy of printed english. Bell system technical journal, 30(1):
50 64, 1951.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Yang Wu, al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision, 2024. URL https:/ /tas://aaxiii ht./v.iiiiv. org/ abs 20nb//2403 2403 09472.
TIGER-Lab. Math-plus.
https:/
MATH-plus, 2024. Dataset available Hugging Face.
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficult--wwree rejection tuning for mathematical problem-solving. 2024. URL https://arxiv,, org. abs// 2407. 136900.
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? limits of Ilm scaling based on human-generated data. In Forty-first International Conference on Machine Learning. 2024.
Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. Scott: Self- consistent chain-of-thought distillation. arXiv preprint arXiv:2305.012: 01899, 2023.
Peiqi Wang, Yikang Shen, Zhen Guo, Matthew Stallone, Yoon Kim, Polina Golland, and Rameswar Panda. Diversity measurement and subset selection for instruction tuning datasets. arXiv preprint arXiv:2402.0233/8 2024.
Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora. Conceptmix: compositional image generation benchmark with controllable difficulty. Advances in Neural Information Processing Systems, 37:86004 86047, 2024.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333. 2024.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoe Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, (6:69798-6981 2023a.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34227, 2023b.
Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang, An implementation of generative prm. https: //gttbb//// www.coo////.../ /RLFF low RLHF R wwweaardd-Mooiinnn 2024.
Wenda Xu, Rujun Han, Zifeng Wang, Long T Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen- Yu Lee, and Tomas Pfister. Speculative knowledge distillation: Bridging the teacher-studen student gap through interleaved sampling. arXiv preprint arXiv:2410.11325, 2024.
Boyang Xue, Qi Zhu, Hongru Wang, Rui Wang, Sheng Wang, Hongling Xu, Fei Mi, Yasheng Wang, Lifeng Shang, Qun Liu, et al. Dast: Difficulty- self- self-training large language models. arXiv preprint xxXv:2503.000029 2025.
Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: A flexible and expandable family of evaluations for ai models. arXiv preprint arXiv:2310. 17567, 2023a.
13

---

Published as a conference paper at ICLR 2026
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023b.
Simon Yu, Liangyu Chen, Sara Ahmadian, and Marzieh Fadaee. Diversify and conquer: Diversity- centric data selection with iterative refinement. arXiv preprini arXiv:2409, 1378, 2024a.
Zichun Yu, Spandan Das, and Chenyan Xiong, Mates: Model-aware data selection for efficient pretraining with data influence models. Advances in Neural Information Processing Systems, 37: 108735-108759, 2024b.
Zhiyuan Zeng. Yizhong Wang. Hannaneh Hajishirzi, and Pang Wei Koh. Evaltree: Profiling language model weaknesses via hierarchical capability trees. In Second Conference on Language Modeling, 2025. URL https: : wwwp//vew.r.eeeeeeeeeee... net/
Biao Zhang. Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets Ilm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193, 2024a.
Jia Zhang, Chen-Xi Zhang, Yao Liu, Yi-Xuan Jin, Xiao-Wai Yang. Bo Zheng, Yi Liu, and Lan-Zhe Guo. D3: Diversity, difficulty, and dependability-wwar data selection for sample-efficien Ilm instruction tuning. arXiv preprint arXiv:2503.1144 2025.
Yifei Zhang, Bo Pan, Chen Ling, Yuntong Hu, and Liang Zhao. Elad: Explanation -uidedd large language models active distillation. arXiv preprint arXiv:2402.. 13098, 2024b.
Haoyu Zhao, Simran Kaur, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Can models learn skill composition from examples? Advances in Neural Information Processing Systems, 37: 102497- 2024.
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-Françços Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. arXiv preprint arXiv:2310.0846/ 2023.
Yuhang Zhou and Wei Ai. imperfect teacher models in low-budget scenarios. arXiv preprint arXiv:2406.05322, 2024.
Improving knowledge distillation from
14

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
Recent line of works have shown that cognitive theories relevant for human learning can also be utilized to improve performance of language models. Arora & Goyal (2023) argue that language models learn to generalize beyond training data, by learning generalizable skills that connect the text tokens. This argument was later used by Wu et al. (2024); Yu et al. (2023a); Zhao et al. (2024) to create evaluation benchmarks to compare how different LLMS can generalize beyond training. Furthermore, Didolkar et al. (2024); He et al. (2025) utilize this framework to create effective instance specific in-context learning examples for language models. More related to our work is the work by Kaur et al. (2024) who create synthetic instruction following datasets using arbitrary combination of skills and show that small language models learn more efficiently from such synthetic tasks. Similarly, Gandhi et al. (2025) show that certain cognitive skills are necessary for models to explore during reinforcement learning, and one can similarly enforce such skills by targeted continual pretraining. In contrast, show that can the skill-based framework create targeted training datasets by analyzing the missing skills in model's responses after training and even unlock further gains.
Related work is line of research that leverages model gradients to construct targeted training datasets (Xia et al., 2024; Yu et al., 2024b). These approaches utilize gradients on both training and test sets to identify effective subsets of training data. However, key limitation is their reliance on ground-trute truth solutions for the test data. In contrast, our work offers a complementary strategy: we employ a frontier LLM to analyze the model's responses and generate targeted training data based on its feedback.
We also compare our proposed method to embedding based data selection. Embedding based methods Wang et al. (2024); Yu et al. (2024a); Ni et al. (2024) have primarily been used to improve diversity of training dataset. As shown by our results, embedding similarity could also be utilized (albeit with drop in performance) to get questions from training dataset that are similar to difficult questions.
Finally, we introduce STAT-Syn, an approach analogous to STAT-Sel, which synthesizes additional training data specifically targeted to the identified missing skills. Synthetic data generation has recently attracted significant attention as a practical alternative for augmenting real-world datasets, and has been shown to improve language model performance both in-distribution and outtt--distribution (Jung et al., 2025; Yu et al., 2023b; Lu et al., 2023; Li et al., 2023; Kaur et al., 2024). comprehensive comparison of STAT-Syn with prior work synthetic data generation is left to future study.
B DETAILS OF STAT DATA CREATION
B.1 ALGORITHM FOR CONSTRUCTING STAT-SEL AND STAT-SYN DATA
Algorithm outlines the procedure to construct P Prorgeted in Stage 3 (Section 2). For each question in the test set Q, the algorithm first identifies the associated missing skills using the Missing-Ske Skill Profile. For each missing skill, a small set of examples is retrieved from the Skill-Map, which links each skill corresponding training data. In STAT-Sel, these retrieved examples are directly added to the target training set. Otherwise, the examples are used as seeds to prompt GPT-4o to generate new, skill-specific questions, which are then included instead. This process ensures that the resulting training set Pt Ptargeted is adaptively enriched with examples that directly address the model's weaknesses.
B.2 TRAINING DATA CREATION PROCEDURE OF STAT
We now provide a detailed interpretation of our training data creation approach outlined Algorithm 1.
STAT-Sel. 4k unique questions, 9.5k QA pairs. We begin by filtering 500 difficult questions from the validation set using our process reward model. For each such question, the teacher model identifies 2-3 missing skills in the student's 's response. As described in Section 2.3, we then create the training set by selecting 5 questions for each missing skill in the question's 's Miissnng-Skin - We use 3 answers for each question and randomly sample a subset of 9.5k question-anae answer pairs as training set.
15

---

Published as a conference paper at ICLR 2026
Algorithm 1 Skill-based data selection/geenraaion
Input: Test set Skill-Map 8-P, Missingsk1lPProfiie Q-8, STAT-Sel: bool Output: Ptargeted
1: Piargeted
2: for in do
3: skill.list WissinghhiiPrrfllleel.
4: if sskillliist is not not empty then
5:
for skill in skill_list do
6:
Pakalt Skill-Map [skill]
7:
selected rrandm_ssapll (P skill if STAT-Sel then
8:
9:
P Ptargeted targettd +P selected
10:
else
Pne GPT-4o(P eclected: skill, prompt="Propose new question based on the given questions and the given skill.")
12:
13:
Ptargeted Prargeted + P new
14:
end if end for
15:
16:
end if
17:
end for
18:
return P Ptargeted
STAT-Syn. 4k unique questions. 9.5k QA pairs. We begin by filtering 500 difficult questions from the validation set using our process reward model. For each such question, the teacher model identifies 2-3 missing skills in the student's response. For each pair of (difficult weccccuttuuusst.. ion, missi ng skill), we retrieve questions from MATH training set. We input these 3 questions, along with the mi missine ssi.nsinsskkk. i illl, to the teacher model, prompting it to synthesize 2 new questions. The teacher further generates 3 solutions for each new question. We then filter the newly synthesized data by:
1. Compute consistency scores for each set of (new www..quetionn.. solution) pairs, according
to the number of solutions agreeing the final answer. For example, question with 2 solutions agreeing on the final answer has a consistency score 2.
2. Keep only the new question with consistency score of 2.
3. For each filtered question, keep only the solution that agrees on the final answer. 3
This process enables our approach to generate diverse data, as we input 3 questions to the teacher model as references each time. The consistency-filterii step filters out both invalid questions and solutions, ensuring the quality of STAT-Syn.
'For STAT-Syn, after filtering teacher-generated answers using consistency, obtained 9.5k valid ques- tion-answer pairs. To comparability, we standardize the training data size to 9.5k pairs for all experiments.
16

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL DETAILS
C.I MODEL & TRAINING CONFIGURATIONS
Model Settings, All inferences are under 0-shot settings, with temperature 0.1 for pass@ sampling. and temperature 1.0 for average @ 64 or pass @6 sampling. For the process reward model in Stage 1 (Section 2.1), we use RLHFlow/LLamaa.. 1--8B--RM-MMsttall reward model fine-tuned from 1.lama-..1-8B with filtering thresholds TI = 0.85, 72 = 0.7. We use seed=0 for evaluations.
(Xiong et (2024)), an 8B process
SFT configurations. For SFT, we adopt QLoRA with rank 16, scaling factor a = 32, and dropout 0.05, applied to the attention and MLP projection modules. Models are trained in 4-bit NF4 quantization with 6íloat16 compute, using the paged AdamW (8-bit) optimizer. We train for 3 epochs with cosine learning rate schedule and a 3% warmup ratio. Peak learning rate is chosen separately for each method among {5e-4, 2e-4, le-4, 8e-5, 2e-5, based aceuracy MATH. The effective batch size 8 (per-device batch size of 2 with gradient accumulation of 4). We apply gradient clipping at 0.3, weight decay of 0.1, and enable group-by-length packing for efficiency. Other configurations follow the official code base from Llama" and Qwen"
GRPO configuration. We train for 6 epochs using a constant learning rate of 5e-7. The objective includes only the policy update loss, without any KL--divergence term, and the entropy coefficient fixed at 0.0. Each batch contains 256 questions, with 4 rollouts generated per question. Responses are truncated at maximum length of 2048 tokens. We set the PPO mini-batch size to 64, which implies that each batch of 256 questions is split into four mini-batches. The model performs four gradient updates before refreshing the reference model.
C.2 TRAINING DATA CREATION PROCEDURE OF BASELINES
We compare STAT-Sel and STAT-Syn with the following baseline models fine-tuned with various data selection/generation methods, to measure the effectiveness of skill-aware training:
7.5k unique questions, 7.5k QA pairs. We naively train the model on all question from the training dataset, with a single answer from the original dataset for each question.
MATH-Train:
7.5k unique questions, 9.5k QA pairs. In order to make fair comparison to our proposed methods, we pick 3 answers per question to create 22.5k question-answer pairs and then randomly sample a subset of 9.5k question answer pairs as our training set.
MATH-Augment:
3k unique questions, 9.5k QA pairs. We include all questions from the levels 4 and of the MATH dataset. We use responses per question to create pool of 1 12k question-answer pairs and then keep a random subset of 9.5k question answer pairs.
MATH-Hard:
Embed-Sel: 4k unique questions, 9.5k QA pairs. Here, we compare the effectiveness of skill-
based training data selection to embedding- bbased training data selection 6 We use our difficult question set from stage I and for each question, we pick similar questions from the training set using an embedding model based similarity score. We pick 3 answers per selected questions and keep a random subset of 9.5k question answer pairs.
Embed-Syn: 4k unique questions, 9.5k QA pairs. For each question in the difficult set identified during stage 1, we retrieve question-answer pairs from the training set P using an embedding- based similarity measure. The teacher model is then prompted to generate 5 new questions, each accompanied by candidate responses, conditioned on different groups of 3 retrieved pairs as in- context examples. We retain only those generated questions for which the LLM produces at least consistent responses, and add the corresponding consistent peestion-answer pairs to our training set. Finally, we keep a random subset of 9.5k question answer pairs to create our training set.
"htpp://gihuu..com//// httpp//ggghhbbbooommmmm/ll/////e
hhtp:://gitubbccoo//eeenM////O/ We use Alibaba-NN.Plltt-Wweeii innstruct as our embedding model (I.i al., 2023)
17

---

Published as a conference paper at ICLR 2026
C.3 PROMPTS
C.3.1 CONSTRUCTING SKILL-MAP ON MATH
Statistics of skill lists. We adopt the list of mathematical skills obtained in Didolkar et al. (2024) using an L.M labeling--clustrringg pipeline. The skill list contains 128 skills in total, divided into subsets across 7 subjects. Each subject includes ~18 skills.
Skill-Map construction procedure. To construct the Skill-Me Map (see Section 2), follow Didolkar et al. (2024) to label skills on both the training and test sets MATH using GPT-4o-mini (OpenAI, 2024). We enlist all skills that used annotate the questions in MATH and dataset in Tables 5 and 6, which have been taken from Didolkar et al. (2024). We ask the LLM to read the question and provide up to five skills required solve this question, from the given existing skill list. We show an example prompt for annotating MATH Number Theory questions as follows.
Example skill annotation prompt for MATH Number Theory questions
[TASK]
You'l 'll be given math question. Your task is output:
(1) < skill> list here up to five skill(s) that are required to solve this problem, seperated by commas </skill>.
(2) <reason> reason here why these skills are needed </reason>.
[SKILL LIST]
You should only choose the skills from this list:
"arithmetic sequences",
conveesion,"""""
"basic_arithmetf"
"division_and aaadiaanddddiiaae remainders""
"exponentiation".
"factorization".
"gretestt common crreates_ccommoncoo
fommon__ivisor..oooo Iivisor.calculationss,,
"modular "modular.arithmeete arithmetic"".
"number_manipulaaton"
"number.theorr"..
"prime number theory",
"sequence analysis".
"solving_ equations".
"solving_equations"" "understanding of fractions"
[QUESTION]
(question)
[REASON AND SKILL(S))
Table 4 shows some example MATH questions and their corresponding annotated skills. From the skill annotation, we construct a Skill -Map (see Section 2) that stores the required skills for each question.
18

---

Published as a conference paper at ICLR 2026
Question
Annotated skills
What is the units digit of
exponentiation, modular arithmetic, sequence analysis
3 + 3
+ 3 + 3" +... 3 ?
In the addition problem each letter represents a distinct digit. What is the numerical value E? [Figure]
basic arithmetic, number manipulation, solving equations
triangle ABC, tan Lan(<CAB) = and the altitude from A divides BC into segments of length 3 and 17. What is the area of triangle ABC?
geometry and space calculation, trigonometric calculations, arithmetic operations
Table 4: Example MATH questions, and the annotated skills generated by GPT-4o-mini.
Subject
List of Skills
Per subject split in MATH
Algebra
algebraic..erreesioosiiii...
algebraic manippaaioonii skills, arithmetic skills,
itt.ulatiioonanndcoooerrrrrsssss skills,
camplaaacccaaaeeeettttonnaaaaaae compleexnumberrss1111...1
geometzicc.sqqueecc.......
skill1 skills,
ouleeeathoor.ssll...ppppooooaoee quadbaaicceepaatton33111111,,$
falicaanappprrrio..aall11,,,,,,,
Counting and Probability
combinattorrccssggeraattiooss
Geometry
Algeraacccklllsarreccccllllllsss
circle.gemmtrryy33111.,,,
skills.
skills,
queerildteeraaaandddooooooo.....
raticaandpprporrrions.k111,,.,.,
Table 5: List of skills used for annotating questions in each subject in MATH dataset
19

---

Published as a conference paper at ICLR 2026
Subject
List of Skills
Per subject split in MATH
Intermediate Algebra
absellure.valle.skiill.
calcuuls..ppiimm.att.........l,.
polynomial_conceppsss_
functions,
Number Theory
axponerttiaticcn.factoooiaaa..a,
prr.ieee.nudeeeloooyyyeeeeeeeaa. of fractions prime
o
Pre-algebra
mattpppllartonnnnn.dddsiiiooeeer and area,
htt.://.a..nommmmmmoaaaooo....o. raiicantpprprrriooonnneeeesexoop
Pre-caleulus
ca calculus,
complexx_umbers,
parametric.eqqutiions.
seeueences.errllsss.dddmmatttt
three sttree.dimenssoonalgeeemettrrr
otg.:p/e...........a.........oor orrigepoomtrriccccalloolltt..oos
vector..perraiioss
Table 6: List of skills used for annotating questions in each subject of MATH dataset (continued from
Table
20

---

Published as a conference paper at ICLR 2026
C.3.2 MISSING SKILL IDENTIFICATION FROM MODEL RESPONSES
In Stage 2 of STAT, we use GPT-4o-mini label the skills that are missing from model response. We ask the LLM to read the question along with the SLM response and provide the skills that the model fails leverage in the response, from the given existing skill list. Below we show an example prompt for labeling missing skills for MATH Number Theory questions, well as an example LLM output.
Example missing skill identification prompt for a MATH Number Theory question
[TASK]
You '1l be given math question and a step-by-step solution written by a Small Language
Model. Your task is to output:
(1) <judge> judge here whether the solution correct incorrect </judge>
(2) <reason> if i'' incorrect, reason here why the solution is incorrect </reason>,
(3) <skill> list here what skill(s) should the SLM enhance in order to answer correctly,
seperated by commas </skill>.
[SKILL LIST]
[list of 15 number theory skills}
[QUESTION]
{question}
[OOUEOSOOLUTIO]
(model_response,
[YOUR REASON AND SKILL(S))
Example GPT-4o-mini output for missing skill identification
(1) <judge> incorrect </judge>
(2) <reason> The solution incorrectly simplifies the expression and misapplies properties of modular arithmetic. The solution claims that the sum of the inverses from to ( (p-1 is (-1 pmodp), which is incorrect. The correct approach involves recognizing that each term
(a (a-). (a+ should be evaluated individually, and the sum should be calculated directly without assuming incorrect simplifications. The solution also fails to correctly apply the properties of modular inverses and does not properly evaluate the sum. </reason>
(3) <skill> modular arithmetic, number ttheor, understanding of fractions </skill>
21

---

Published as a conference paper at ICLR 2026
C.3.3 PROPOSING NEW QUESTIONS FROM SKILL-BASED TRAINING QUESTION DISTRIBUTION
In Stage STAT-Syn, use GPT-4o to propose new questions based on existing questions in the skill-based training question distribution. Below we show an example prompt.
Example prompt for proposing questions using GPT-4o
[TASK]
You'll Il be given three math questions (e.g., [QUESTION IJ), with their solutions for reference.
Your task output new, novel math question that emphasizes the use of [SKILL]. Important Note: the new question should not be very similar to any of the given questions (e.g., naive adaptions like altering variable names values from given question strictly prohibited). Meanwhile, the new question should have similar difficulty with the given questions.
Output format:
(I) <reason> reason here how the given questions relates to the [SKILL] </reason>
(2) <draft> reason here how to design new, novel question while emphasizing the [SKILL]
</draft>
(3) <question> your newly constructed math question </question>
[QUESTION 1]
(train.set.question] }
[QUESTION 2]
deiin_set.questtonn
[QUESTION 3] (train_se__questionn}
[SKILL]
(missing skill}
22

---

Published as a conference paper at ICLR 2026
D ADDITIONAL RESULTS
D.1 EVALUATION RESULTS ON LLAMA-3
Table 7 shows the evaluation results on Llama-3.2 I-Istrutt Similar Table 3, STAT consistently
outperforms both heuristic-based and embedding-ban based data augmentation baselines on in-distribution dataset and most OOD benchmarks. We presented more discussion in Section 3.2 and Section 4.
Madels
MATH MATH MATH GSM8K AMC23
MATH-perturb
AIME
Avg.
simple
hard
2025
Llama-3.2-1 3.2-1IB-nttttrctt
Instrrccct SFT
Base Model
26.0
15.1
9.1
40.7
11.1
17.2
6.5
20.0
10.0
17.3
MATH-Train
27.0
14.5
10.0
42.8
8.8
19.0
6.8
26.7
10.0
18.4
MATH-Augment
27.8
14.2
8.1
43.4
11.1
17.9
6.8
26.7
3.3
17.7
MATH-Hard
28.4
15.4
8.6
44.6
10.8
18.6
7.2
23.3
3.3
17.8
Embed-Sel
15.6
8.6
8.8
6.8
3.3
Embed-Syn
28.4
17.2
11.0
44.3
10.0
20.1
7.9
23.3
6.7
18.8
STAT-Sel
32.4
15.6
11.0
45.0
12.0
19.4
7.9
26.7
16.7
20.7
STAT-Syn
18.3
12.4
11.0
20.8
7.5
23.3
10.0
GRPO
Base Model
31.8
14.4
13.3
23.3
8.2
6.7
MATH-Train
32.0
16.0
11.9
50.8
10.0
23.7
7.9
16.7
6.7
19.5
MATH-Augment
31.2
15.0
9.0
49.1
13.6
24.7
7,9
23.3
13.3
20.8
MATH-Hard
32.2
14.8
11.0
50.6
11.6
22.9
10.0
20.7
Embed-Sel
32.8
16.2
11.4
49.9
12.0
21.9
6.5
23.3
13.3
20.8
Embed-Syn
32.6
15.0
10.5
51.0
13.9
21.1
6.8
26.7
3.3
20.1
STAT-Sel
34.8
16.6
13.8
50.1
14.8
23.7
9.0
30.0
16.7
23.3
STAT-Syn
35.2
21.1
13.8
51.0
14.8
24.7
7.9
33.3
16.7
24.3
Table Improvements on various math benchmarks from applying STAT. Results under "+SFT" show the performance of SFT models trained with each method, while "+GRPO' shows the performance af- ter applying GRPO on top of the corresponding SFT models. Our methods, STAT-Sel and STAT-Syn, achieve an average gain of up to 3.4% over the base model, with strong OOD performances (AMC23 results reported on average @ 64, AIME on pass (ess66)), 64). Applying GRPO on top of fine-tuning with STAT further enhances these improvements. See Table for results on Llama-3 3.2-3B 3.2-3B-InsHe 3-IInstruct and Qwen2.5-3B.
D.2 BEEIING-SKILLPROOT
Figure 6 shows the snippets of model-specific Missing-Skill-Proofee of Llama-9.223B-IInttnnc Llama- and Qwen2.5-3B, obtained at the end of Stage 2 (see Section 2.2). These profile snippets include the Top 10 frequent missing skills of the models. As discussed in Section 4, most of the frequent missing skills in both models are algebra-related, such as solving equations, manipulation, and calculation. In addition, both models also demonstrate noticeable weaknesses in conceptual and -oriente mathematical skills, including combinatorics, understanding and application of functions, and number theory. Compared to Llama-3.2-38 Instuutt the missing skill profile of Llama- Innttutt concentrated more towards basic operations (e.g., solving equations), suggesting that smaller models have more pronounced limitations in fundamental computational abilities.
23

---

Published as a conference paper at ICLR 2026
1.ama-3.2-38-Imma
Qwen2.5-3B
Solving Eqgattion
Solving Equations
Algehraic Manipalation Equatioes
2nd
Algobmic Manipulatt intipuaaiion and Equations
Basic Ar trimeete Operations
Basic Anihmetic Operations
1000
3nd
Basse Arilhmeic Openaiios
Algobraic Manipulatioe
Cumbinatorics Concrpts
Calculaton Cunversion
Calculation
Solving Equations
300
Algebmaic Manipulation
Algobrzic Expresiin
Simplificatio and Dosic Operatoon
6db
Combinatorios Concepes
Combinatories Operations
Multiplictiion and Diviston
7th
Indestaading and Application Functions
Combinatorios Concupts
Calculation and Conversio
Algebazic Expression
Arthmeric
Probcklitt Concapts
400
9dt
Simplificttoon and Basie Operations
Lineer Equatien
Polynomic
108:
Counting and Number Theory
Number Theoy
Algehraie Min
Figure 6: Top 10 missing skills of Llama-3, 1amma--3.2-33- 3- Instruct, Llama-3.22 I---trrcctt and Qwen2.5-3B The models struggle most with fundamental mathematical skills such as solving equations and basic arithmetic operations.
D.3 CASE STUDY: EXAMPLE QUESTIONS WITH DIFFERENT SKILLS
We observe in Section 4 that the baseline training data distribution largely misaligns with model's actual missing skills, with baseline data emphasizing more advanced and conceptual skills (e.g.. Prime Number Theory, Polynomial Concepts) while the model lacking more basic operational skills (e.g., Solving Equations). To better visualize this misalignment in data distribution, here we showcase three example QA pairs from the MATH training respectively associated with the three skills above.
Question with skill "Solving Equations"
Question:
The inverse f(x) 2x-1 may be written in the form f= .+5
where a, ez+d'
or+b
b, and d are real numbers. Find a/c.
Solution:
we substitute
(x) into our expression for f, we get
2/ 25 -
f (()
f (x)
Since f (f(x)) x, we obtain
2f (x) -1
(1) +5 =
2f (x) -1 x 5)
= 25 (x) - xʃ (x) + 5x.
Moving the terms involving (x) to the left-hand side and the remaining terms to the right-hand side, we get
(x) - (x) = 5r +
(=) r) 5r
5x +
f (x) - +2'
Now we can see that (a, b, d) = (5, 11,2) for this representation of (2), so
5
5
c -1
24

---

Published as a conference paper at ICLR 2026
Question with skill "Prime Number Theory"
Question: The positive integers A, B, B, and B are all prime numbers. The sum of these four primes is
A. even
B. divisible by
divisible by
D. divisible by 7
E. prime
Express your answer using letter, A, B, C, D, E.
Solution:
The numbers A and B are both odd or both even. However, they are also both prime, so they must both be odd. Therefore, one of A and B is odd and the other prime between
and A + B, must be the odd prime. Therefore, B = 2, the only even prime. So A - 2, A, and A + 2 are consecutive odd primes and thus must be 3, 5, and The sum of the four primes 2, 3, 5, and 7 the prime number 17, so correct answer is (E), prime.
even. Because
Question with skill "Polynomial Concepts"
The polynomial P(a:) 2n + az + br + has the property that the mean of its zeros, the product of its zeros, and the sum of the coefficients are all equal. The y-intercept of the graph of y = P(z) is 8. What is 6?
Question:
Solution: The y- g-intercept of the graph is the point which = 0. At that point, P(x) which we are told is equal to 8. Thus, c = 8. The product of the roots of the given polynomial - -4. The problem states that the mean of the
must also equal -4, the sum of the three zeros (this is cubic equation) is equal to 3 -4 = - -22. The sum of the zeros also equal - so 24. Finally, we are given that the sum of the coefficients, is also equal to - 4. Plugging in our known values of and c, we have 2 + 24 + = -4.
+ a bbb
Solving for b, we get b -38
25

---

Published as a conference paper at ICLR 2026
D.4 EFFECTIVENESS OF STAT ON EACH SUBJECT
To evaluate whether STAT enhances general subject-level competence, we measure model accuracy across the 7 subject categories in MATH. These subjects are: prealgebra, algebra, intermediate algebra, geometry, precalculus, number theory, and counting & probability. As shown in Figure 7, both STAT-Sel and STAT-Syn consistently outperform the base model and data augmentation baselines across nearly all subjects. Notably, STAT-Sel achieves the strongest improvements in precalculus and number theory, while STAT-Syn excels in intermediate algebra, prealgebra, algebra, geometry and counting & probability. It is worth noting that STAT brought most improvements on the algebra-related subjects. This aligns with our observation in Section that Llama-3.2 1-n--tttrett shows its most pronounced weaknesses in algebra, and confirms that our approaches effectively target the skills the model fundamentally lacks.
precalculus
intermediate algebra
counting & probabilits
Base Model
MATH-Tran
MATH-Augment
MATH-HHard
goometry
Embed-Sel
Embed-Syn
STAT-Sel
number theory
STAT-Syn
algebra
prea Igebra
Figure 7: Fine-tuned model performances on MATH subjects, across different training methods. For better visualization, accuracies are normalized per skill axis, with the base model drawn as a uniform circle and the segghestperformmngg method each skill placed at the outer edge. STAT-Syn and STAT-Sel are most effective in enhancing model performance across nearly all the subjects.
E ABLATION & ANALYSIS
E ABLATIONS ON THE REWARD FILTERING METHOD IN STAGE
Recall that in Stage of the STAT pipeline, we use an off-the-shelf process reward model [RLHFllo/Lllam33. 10ama3.1-88 der filter out a set of difficult questions for each model. Here, we conduct various ablation studies on the reward filtering process.
to score small language models' responses, in or-
Effeet of threshold values on the reward model prediction. We investigated the effect of Fı and T2 (defined in Section 2.1) on the classification performance of difficult questions. Specifically, measure whether our classification of questions as difficult also corresponds to the correctness of responses assessed using ground-truth labels. In Table 8, we report four metrics (accuracy precision / recall / F1) evaluating the prediction accuracy resulting from different filtering thresholds. Note that 7 0 T2 0 means completely removing the constraints of T₃ or T2. Across all evaluated combinations of threshold values, our choice of the threshold values (π = 0.85, = 0.7) gives good combination of prediction scores. To further visualize this effect, we conduct STAT on top of all combinations of thresholds, and report the final accuracy in Table 9, Our choice of threshold values yields the highest final accuracy among all the combinations.
(OOD) prediction performance of reward model. Although we primarily evaluated STAT on MATH and GSM8K, our method can potentially be extended to other math datasets. While the reward model we used in Stage 1 was only trained the MATH and GSM8K distribution, we show that it capable of scoring responses for various OOD math datasets. Table 10 reports the reward model's performance classifying correct/incorrect responses from Qwen2.5-3B
26

---

Published as a conference paper at ICLR 2026
71 72
T2 0
T2 0.6
T2 0.7
T2 0.8
T1
53/0/0/0
80/ 78 79/
80
79
75
95
78
71 0.8
80/79 78/ 79
76 80
79 72
90 80
66
96
78
71 = 0.85
79 74 88 80
79 72
/0 80
78 70
92 80
74 /
65
96
78
0.9
73/
64/ /9////950 95
73 64 9
72 64 9 77
70/ 62 97 75
Table 8: Reward model performance (aceuracy precision / recall FI) on classifying correct/incorrect responses from 1..02.55-1.BB-strrrr means completely removing 7 or 72- Our choice of threshold values (71 0.85, 72 0.7) gives good combination of prediction scores.
MATH, accross different thresholds.
0 72 = 0
T 172
T2
T2
0.6
T2
0.7
T2
0.8
T =
52.8
55.7
55.9
55.7
T1 0.8
55.1
56.3
56.2
55.6
7 0.85
55.3
56.4
56.4
55.6
0.9
55.7
55.7
55.6
55.2
Table 9: Final STAT performance of Qwen2.5-1.5B-mmm
on MATH, with different thresholds.
Our choice of threshold values (T1 0.85, T2 0.7) leads to the highest accuracy.
on four popular math benchmarks: AMC23, AIME24, AIME25, and MATH The reward model achieves comparably high performance on scoring model responses on these OOD, significantly more difficult benchmarks, indicating that the model highly generalizable. This implies the potential to extend our method to new datasets without the need to train a specialized reward model for each one.
Metric
AMC23
AIME24
AIME25
MATH
Accuracy
92.5
86.7
86.7
84.8
Precision
90.9
92.6
86.7
95.2
Recall
95.2
92.6
100.0
88.5
F1
93.0
92.6
92.9
91.0
Table 10: Reward model prediction metrics across four OOD math benchmarks. Despite not being trained on these benchmarks, the reward model's prediction capability is largely generalizable to them.
Reward Filtering vs. Simple Heuristics for classifying difficult questions. Considering the computational overhead of calling separate PRM, we explored alternative approaches classifying questions that rely on computation-free simple heuristics. Specifically, we experimented with two heuristic strategies:
Consistency heuristic: We measure the consistency of the model across five sampled gen- erations per question and classify questions with lower consistency as difficult. Specifically, question is difficult if, among sampled generations, the most common response appears
< 2 times.
Length heuristic: We the length of the model's responses a proxy and classify questions with longer responses as difficult. Specifically, a question is difficult if the average model response length on this question is 800 words.
Table 11 shows that both heuristics yield reasonably accurate predictions. Moreover, applying STAT on top these heuristie-classifedd difficult questions can improve the final accuracy by 2%. However, we leave a more thorough investigation into the robustness and generalizability of these strategics in relation to PRM-based classification for future work.
Process Reward Outcome Reward. We also compare the prediction accuracy of our process reward model (PRM) with threshold filtering (see Section 2.1) against directly loading the reward model outcome reward model (ORM). Our preliminary experiments indicated 0.9 as the optimal
27

---

Published as a conference paper at ICLR 2026
Classification method
Classification accuracy
Consistency Heuristic
79.8%
Length Heuristic
74.2%
Reward Filtering
78.0%
Table 11: Performance of consistency heuristic and length heuristic on classifying difficult questions. The classification accuracy of simple heuristics are on par with the reward filtering method.
threshold for the outcome rewards. With 0.9, the prediction metrics the ORM are: Precision 0.54 Recall 0.90 / F1 0.68, whereas the prediction metrics of the PRM with optimal thresholds are Precision = 0.70 Recall = 0.92/ / F1 0.80. Therefore, our method using PRM with threshold filtering is superior to directly using ORM.
E.2 STATISTICS OF DIFFICULT QUESTIONS
In Stage of STAT (see Section 2.1), we identify a set of difficult questions for each individual model using a process reward model along with a filtering heuristic. Table 12 reports the proportions of difficult questions classified for different models each math domain. Compared Table 3, the proportions of difficult questions closely correspond to the accuracy numbers of each model, even though we did not access the ground truth the whole pipeline. Notably, our classification method captures not only questions that the model gets wrong, but also questions that the model passes with a flawed solution process.
Model
Geometry
Precaleulus
Algebra
Prealgebra
Intermediate Algebra
Qwen2.5.38
61.8 8
70.1
29.7
33.2
75.9
J.ama-3.2-1BBImms
92.0
91.4
89.7
99.0
Llama-3.2 I.2.1BBInstr
68.2
82.7
45.5
48.9
85.7
Model
Count.
Count&&Prob. &Prob.
Number Theury
MATH Arg.
Qwen2.5-23
56.1
Llama 3.2 1-21B-Baae Instruct
97.9
95.2
94.0
3.2.3B-Instte
62.3
Table 12: Proportions of difficult questions (%) classified by STAT for each model. Although our method did not access the ground truth, the proportion of classified difficult questions still closely mirrors each model's s accuracy (see Table 3) in each domain.
E.3 ANALYSIS OF THE TEACHER MODEL
Teacher model need not be overwhelmingly stronger than student. One feature of STAT is the demand of a substantially stronger teacher model to supervise the student. In this section, we evaluate this demand by directly comparing teacher and student performances on math reasoning benchmarks. Due to resource constraints, our evaluation is limited to a representative set of benchmarks, but the results are sufficient illustrate the key trend: the teacher is not strictly dominant, and the student can approach or even match the teacher's performance within a manageable gap.
As shown in Table 13, although teacher models obtain higher absolute scores, they are not over- whelmingly stronger than the students. In particular, the gap between GPT-4o-mini and Qwen2.5-3B only around 10 points across GSM8K and MATH, a margin that significant but manageable. This suggests that STAT does not strictly rely teacher and student are relatively close in ability, the student can still benefit and recover most the teacher's 's performance. This opens up the possibility of self- iiff-mmprovement, where model iteratively teaches and refines itself without requiring access to an external teacher that is substantially stronger.
much stronger teacher to succeed. Instead, even when
Agreement across different teacher models. Since our approach relies on a frontier LLM as teacher, a natural concern is potential bias in the missing-skill labeling process. In light of this, present a preliminary investigation into the level agreement among different LLMs in missing skill labeling, using an A.M--s-a-judge approach. We first evaluate GPT-4o-mini' ability to self-verify
28

---

Published as a conference paper at ICLR 2026
Teacher
Student
Benchmark
GPT-4n
GPT-4o-mini
Qwen2.5.3R
1.ama-3.2.3B--naa. I.ama-3.2-IR-mmae
MATH
73.0
69.1
55.8
44.0
26.0
MATH--oruub--implle
N/A
17.2
MATH-petuub-aard
39.4
N/A
24.0
12.2
6.5
Table 13: Math reasoning accuracy (%). Comparison between teacher models (GPT-4o, GPT-4o- mini) and student models (Qwen2.5-3B, Llama-3.. 3.-.-3.--In-ttuuuce Llama-3.2- IB-Instruct) on GSM8K, MATH, MATII-pertur--simple. and MATH-perturb-harce
the correctness of its own predicted missing skills and find that it judges its predictions to be correct 70% of the time. further assess the reliability of these predictions, we compute the agreement between GPT-4o-mini and Claude-3.5-Sonnc, where agreement defined as the average fraction of overlapping skills relative to the total number of skills predicted by GPT-4o-mini. Given the fine fininnriined nature of our skill list, we consider this level of agreement significant.
The models agree 43% of the predicted skills,
29


Reviews:
[{"review_id": "hl2UrG91OL", "reviewer": "Reviewer_PEvi", "summary": "Supervised fine-tuning can often result in limited gains. This paper proposes an approach, STAT, to transform an SFT dataset (via reweighting or synthetic data generation) based on skills. Given a validation dataset, the current model's failures on it are analyzed to determine which skills are missing from the model. These skills are linked back to the training data, and data most associated with these skills are upweighted or synthetically generated. Empirical results show that STAT-Syn and STAT-Sel significantly outperform other data-centric approaches on several model families and sizes. It can also be used in continual learning, and also can be used with RL downstream.", "questions": "(Taken from the Weaknesses section above)\n1. Have you considered modeling or empirically testing for interactions or uneven learning rates among skills? How might such dynamics affect the assumptions behind STAT?\n2. In the continual learning setup, how does optimizing on MATH-perturb-hard impact performance on other math benchmarks? Do you see any sort of forgetting going on and could that be fixed?\n3. Do you observe cases where the true missing capability is a composition or union of multiple skills (e.g., solving equations related to geometry)? Could STAT handle such compositional skill structures?\n4. To show STAT's improvement at the skill-level, can you consider an oracle-style evaluation where you identify difficult test samples (those currently misclassified) and measure how many become correct after applying STAT?\n5. How do you expect the method to generalize to other domains such as coding, reasoning, or agentic workflows? What guidance could you provide for defining “skills” in new domains, given that the math taxonomy here is well structured?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n- This paper builds on several existing skills-based works, e.g., using their techniques for skill discovery and skill mapping. However, this is not a significant drawback. Moreover, this work focuses on the SFT setting and uses teachers/skills exclusively for alignment, rather than embeddings or gradients. \n\nQuality:\n- Experiments show that STAT can significantly improve performance across model sizes and families for math. \n\nClarity:\n- The paper is well-written, and the method is easy to understand.\n\nSignificance:\n- A common heuristic for curate good training datasets is to simply create or select samples that \"look like\" the evaluation data. E.g., if you want your model to do well on science, you need to train on science data. However, this heuristic is fairly imprecise and can fail to capture the underlying skills that samples teach the model. Moving towards skill-based data curation is an important direction, and I foresee datasets being annotated with more and more fine-grained skill taxonomies in the future to facilitate greater gains.", "weaknesses": "Quality:\n- The method operates under a reasonable assumption that the training data distribution should be aligned with the test data distribution in terms of skills (Figure 2). However, this does not take into account the possibility that 1) skills may interact with each other and 2) skills are learned at different rates (as suggested in Chen et al 2023 https://arxiv.org/abs/2307.14430). \n- The continual learning extension showcases the flexibility of STAT; however, I am also curious about if optimizing on MATH-perturb-hard negatively impacts performance on the other math benchmarks. More generally, I am interested in seeing if the model forgets previous skills, and if so, whether you could make some simple adjustments to STAT (e.g., doing an exponential moving average for STAT-Sel over time) to address this.\n- I am curious about the role of composition in these results. In Figure 5, STAT-Syn determines that the missing skill is solving equations, while Embed-Syn determines that the missing skill is Ellipse Properties. Is it ever the case that the actual missing skill is a union of these skills? For instance, what is needed is not just more data focused on solving equations, but instead data focused on solving equations related to geometry? This may also touch more fundamentally on a question of skill granularity/combinatorics of skills. \n- I am curious about an \"oracle\" experiment that could track whether or not the missing skills are learned by the model. Here's a proposal: instead of using a validation set, take the test set, and use the ground truth labels to determine the difficult samples, which are just the ones that the model gets wrong. If we apply STAT-Sel/STAT-Syn, how many of the questions in the test set will flip and become correct? Does STAT interfere with the test set questions that were already answered correctly? \n\nSignificance:\n- The empirical evaluation is somewhat narrow - it relies on a single dataset and a specific skill taxonomy. This makes it unclear how well the findings generalize to other domains (e.g., coding, agentic workflows) or to different definitions of “skill.” In particular, the analysis in this paper suggests that skills are deeper than the topics identified via embeddings (e.g., in Figure 5), but it is not completely clear how to construct these skills from first principles on a new domain besides math.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mmoyLaxnEf", "reviewer": "Reviewer_6kx2", "summary": "The paper proposes STAT, a teacher-student fine-tuning method that uses a stronger LLM to diagnose which skills a smaller model lacks on a math validation set, then adapts training data to target those deficits. Two instantiations are presented: STAT-Sel (re-weights existing examples) and STAT-Syn (synthesizes new questions). On MATH and several out-of-domain (OOD) benchmarks, STAT yields up to 7.5 % absolute gain over vanilla SFT and 4.6 % average OOD improvement. Ablations show that (i) models saturate because they still fail on basic algebraic skills, (ii) embedding-based data selection does not target those skills, and (iii) STAT is complementary to subsequent RL (GRPO).", "questions": "Based on the comments above, a few questions for the authors:\n\n- Can the skill-map pipeline be automated for non-math domains? \n- How many independent training runs (seeds) were performed? If only one, can you provide variance estimates?\n- The method's success seems to rely on a predefined list of skills (Section 2, lines 141-145). How sensitive is STAT to the quality and granularity of this skill taxonomy? For instance, would a coarser or finer-grained skill list significantly alter the results?\n- How important is using a powerful teacher? Have you considered alternatives which might be better for scaling and help improve fronteir models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n- First work to operationalize LLM metacognition for training-set adaptation rather than mere evaluation or prompt refinement.\n- Skill-map + Missing-Skill-Profile + targeted data construction is a clean pipeline motivated from how humans actually learn.\n\nEmpirical Quality\n- Experiments on Llama and Qwen families (1B–3B sizes) with MATH as the primary dataset show substantive in-distribution and OOD improvements over naive SFT and embedding-based baselines.\n- Careful ablations vs. strong baselines: MATH-Augment, MATH-Hard, Embed-Sel/Syn, and GRPO.\n- Hyper-parameters openly tuned per method (sec. 3.1 & C.1).\n\nClarity\n- Algorithm box (Alg. 1), reproducible prompts (Appendix C.3), and released data and code pledge (sec. 7) at camera ready version facilitate replication.\n\nSignificance\n- Provides a practical recipe to push small models beyond SFT saturation without extra human labels.\n- Demonstrates that basic skill gaps (solving linear equations, arithmetic manipulation) remain even after extensive training. I believe these are valuable insight for the community and pushes algorithmic progress towards better data/curriculum design.", "weaknesses": "- Dependence on a Powerful Teacher Model: The entire STAT framework is predicated on the availability of a significantly more capable \"teacher\" LLM (GPT-4o-mini is used) that can accurately diagnose skill deficiencies. How would one go about using this method for improving the skills of frontier models? Is self-improvement a possibility here by using either the same model (prompted in a different way) or by using specialized but smaller models  as teacher?\n\n- Scope limited to MATH-style reasoning (minor): results are strong for math, but applicability to language or safety tasks is only discussed as future work. If the authors could discuss a bit about the generality of the method and possible way to expand it beyond the Math scope, that would be helpful.\n\n- No comparison with gradient-based data selection methods (e.g., RHO-1, Lin et al. 2024) or difficulty-based rejection tuning (DART-math, Tong et al. 2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q6P6xKHgm4", "reviewer": "Reviewer_yjVy", "summary": "This paper introduces STAT (Skill-Targeted Adaptive Training), a novel fine-tuning framework for language models that leverages the metacognitive abilities of large teacher models to identify and target missing skills in student models. STAT improves generalization and performance on mathematical reasoning benchmarks by either reweighting training data (STAT-Sel) or generating new skill-targeted questions (STAT-Syn). Across multiple benchmarks, STAT consistently outperforms standard SFT and other adaptive data selection methods, and complements reinforcement learning approaches like GRPO.", "questions": "1. How robust is the STAT framework to inconsistencies or ambiguities in the skill taxonomy generated by the teacher model? Would the approach degrade if the teacher’s skill decomposition is noisy or if the skill set is too coarse-grained?\n2. Given that STAT relies on a strong LLM teacher to guide training and even generate new examples, how do you mitigate the risk that the teacher’s inductive biases dominate the student’s learning?\n3. How do you balance granularity vs. utility when generating the Missing-Skill-Profile? For instance, does distinguishing between \"algebraic manipulation\" and \"linear equation solving\" actually improve downstream performance, or introduce noise?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The work leverages frontier LLMs' introspective abilities to identify specific skills lacking in student models. This goes beyond error analysis and introduces a more human-like teaching dynamic.\n2. STAT methods (both Sel and Syn) demonstrate consistent and significant performance gains over SFT and baselines across in-distribution (MATH) and out-of-distribution tasks (AIME, AMC, MATH-perturb), with gains up to 7.5% on MATH and 5.8% on OOD datasets.", "weaknesses": "1. The entire STAT pipeline depends on access to a strong teacher model with metacognitive abilities. This could limit replicability or scalability in low-resource settings or with proprietary LLMs.\n2. The Missing-Skill-Profile is built using validation set responses, which might introduce bias if the validation set is not representative of future deployment scenarios.\n3. The STAT-Syn variant, while effective, incurs significant computational and inference cost due to on-the-fly synthetic question generation and filtering, which may not be inefficient in large-scale deployments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)