MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes
Authors: Changsheng Zhao
Ernie Chang
Zechun Liu
Chia-Jung Chang
Wei Wen
Chen Lai
Sheng Cao
Yuandong Tian
Raghuraman Krishnamoorthi
Yangyang Shi
Vikas Chandra
Venue: ICLR 2026 Poster (2026)
Keywords: On-device LLM
Primary area: foundation or frontier models, including LLMs

Abstract: The paradigm shift in large language models (LLMs) from instinctive responses to chain-of-thought (CoT) reasoning has fueled two prevailing assumptions: (1) reasoning capabilities only emerge in sufficiently large models, and (2) such capabilities require training on massive datasets. While the first assumption has already been challenged by recent sub-billion-parameter reasoning models such as Qwen3-0.6B and DeepSeek distilled variants, the second remains largely unquestioned. In this work, we revisit the necessity of scaling to extremely large corpora (>10T tokens) for reasoning emergence. By carefully curating and resampling open-source datasets that we identify as beneficial under our designed metrics, we demonstrate that strong reasoning abilities can emerge with far less data. Specifically, we show that only ~2T tokens of high-quality data are sufficient, and pre-training with 4.2T tokens on the dataset resampled from these ~2T tokens, followed by a established post-training procedure, enables the development of MobileLLM-R1, a series of sub-billion-parameter reasoning models that substantially outperform prior models trained on fully open-sourced data. For example, MobileLLM-R1-950M achieves an AIME score of 15.5, compared to just 0.6 for OLMo-2-1.48B and 0.3 for SmolLM-2-1.7B. Remarkably, despite being trained on only 11.7% of the tokens compared to Qwen3’s proprietary 36T-token corpus for pretraining, MobileLLM-R1-950M matches or surpasses Qwen3-0.6B across multiple reasoning benchmarks. To facilitate further research in this direction, we have made the models (https://huggingface.co/collections/facebook/mobilellm-r1) and code (https://github.com/facebookresearch/MobileLLM-R1) publicly available, along with the complete training recipe, data sources, and data mixing ratios.

Published as a conference paper at ICLR 2026
MOBILELLM-RI: EXPLORING THE LIMITS OF SUB-BILLION LANGUAGE MODEL REASONERS WITH OPEN TRAINING RECIPES
Changsheng Zhao" Meta Al CB cazhao@metad com
Ernie Chang
Zechun Liu
Meta AI ern erniecyc@meto com
Meta Al zechun wwwietue.....m
Chia-Jung Chang
Wei Wen
Chen Lai
Rick Cao
Meta AI
Meta AI
Meta Al
Meta Al
Yuandong Tian Meta AI
Raghuraman Krishnamoorthi
Yangyang Shi Meta Al
Vikas Chandra Meta Al
Meta AI
ABSTRACT
The paradigm shift in large language models (LLMs) from instinctive responses to chain-of-thoughh (CoT) reasoning has fueled two prevailing assumptions: (I) reasoning capabilities only emerge in sufficiently large models, and (2) such capabilities require training on massive datasets. While the first assumption has already been challenged by recent coo--illion-parammett reasoning models such as Qwen3-0.6B and DeepSeek distilled variants, the second remains largely unquestioned. In this work, revisit the necessity of scaling extremely large corpora (>10T tokens) for reasoning emergence. By carefully curating and resampling open-source datasets that we iden- tify as beneficial under our designed metries, we demonstrate that strong reasoning abilities can emerge with far less data. Specifically, we show that only ~2T tokens of high high-quality data are sufficient, and pre-training with 4.2T tokens on the dataset resampled from these ~2T tokens, followed by a established post-training procedure, enables the development of Mobi 1 1LLMM-R,, a series sub-billion bibbbilioo--parameter reasoning models that substantially outperform prior models trained fully open-sourced data. For example, Mobil 1 score of 15.5, compared to just 0.6 for OLMo-2-1 despite being trained on only 11.7% of the tokens compared to Qwen3's proprietary 36T-token corpus for pretraining. MobbileLMM-R1-99000 reasoning benchmarks. To facilitate further research in this direction, have made the mod- els (htttps:/huugiigg ctt.fffccccc////eo hntp:///coccccccoooooooooooooe http://eeeeeeoooooooooooooo////r lmmr) and code R1) publicly available, along
R1-950M achieves an AIME and 0.3 for SmolLM-2-1.77 Remarkably,
matches or surpasses Qwen3-0.68 across multiple
(https: /tttps::/giite //githubbb com/ facebook with the complete training recipe, data sources, and data mixing ratios.
1 INTRODUCTION
MoNHeLLMA-1--55MMbbaa
Qwer2.5-1 SB-bae
Large language models (LLMs) such as GPT (Achiam et al. 2023), Qwen Yang et al. 2025 2024), and
HumanEvall Accuracy
MobileLLMM Www.LLM---R--eO--baa.
DeepSeek (Guo et al. (2225) have demonstrated re- markable progress in explicit reasoning. Advances
30
Owenā-O
have been driven by scaling model size. expanding training data, and applying post-training techniques such supervised fine-tuning (SFT) and reinforce- ment learning (RL). Reasoning LLMS are capable of tackling complex problems by following long chains of thought that incorporate reflection, backtracking, and relf-validation. At the same time, reasoning traces have evolved from prompt- bosed based chain-of ohain-of-thought (CoT) in-
20
LLaMA3.2
MobiieLLM-AA1--00--bbse
CLM6-2-0425-18-taa
Gemma. Ki Cemmmaa--27aM..p
SmroiLLM.-360M..0 SmollLME
SmolLLM211788bbil
Approx Pretraining FLOPs (Size Tokens
Figure 1: Pretrained model accuracy training efficiency trade-off.
"qqual contribution, authors listed alphabetical order. 31edd overall data curation efforts Coorresponding author

---

Published as a conference paper at ICLR 2026
context learning (ei et al. 2022) to models explicitly optimized on long reasoning traces to generate multi-step reasoning sequence (Jaech et al. 2024).
deployment. Large models already strain resource- constrained devices (Liu al. 2024), and long-context reasoning further exacerbates memory usage KV cache growth sharply increases the footprint (Sadhukhan al. 2025). Looking ahead, one can envision a future with personal assistants, smart homes, and robots increasingly relying on on-device reasoning for complex tasks. In such a world, deployability and portability will become inevitable trends for the next generation of LLMs. This motivates our central question: Given strict capacity constraints, what is the most effective recipe to endow small reasoning models with strong capabilities and unlock their hidden potential?
However, this paradigm poses increasing challenges for
Developing small reasoning models poses unique challenges beyond simply scaling down large ones. For large models, expanding the corpus often drives stronger generalization. In contrast, small language models are far more sensitive: noise in the data can easily overwhelm their limited capacity, making data quality and curation paramount. As models shrink, neurons must encode more overlapping knowledge, increasing the risk of interference and conflicts (Zhu al. 2025) supp-pppsiiiiii provides an intuitive lens for understanding this challenge (Elhage et al. 2022). Mitigating these risks requires carefully optimized data, objectives, and training procedures.
While extensive research has explored how postinraining objectives and data curation can elicit reasoning from pretrained models (Wang al., 2025 Li al. 2025), far less attention has been paid to more funddmental question: How can we endow pretrained models with the latent potential for reasoning in the first place? This work addresses this gap by investigating two critical questions: (1) What kinds of data are most effective for instilling reasoning capability, and (2) How can diverse forms of reasoning- suuhhhgg-sshhhhu as coding, mathematics, and logical problem-solving be embedded into compact model without overwhelming its limited capacity?
Through capability aware data curation and probing into the latent factors that govern reasoning, we achieve highly token-efficient pretraining compared to prior work. With only 4.2T training tokens, just 11.7% of Qwen's 36T, our MobileLLM 1e--eee-LM---99000 R1-950M model, matches surpasses Qwen3-0.6B Yang et al. (2025) multiple reasoning benchmarks, placing itself on the Pareto frontier of accuracy-traaning-ton ttreennin-tooen efficiency trade-off curve (Figure I Beyond introducing high-performing small-scale reasoning model, share both the insights and the pitfalls encountered along the way, offering a first-hand glimpse into the complex yet fascinating mechanisms behind reasoning models.
Our contributions are as follows:
We introduce benchmark-free, self-evolving data optimization for pre-training data curation, a principled dataset-level weighting approach that leverages cross domain influences to tailor the data mixture. This facilitates robust reasoning generalization on held-out benchmarks. achieved without exposing them during training or data mixture optimization.
We further propose a data-model co-evolution strategy to adapt to rapid changes in model capacity during mid-training. We show that this process converges as most samples reach zero negative influence, indicating that the dataset's s information has been largely exhausted and offers minimal further improvement.
Compared to existing fully open-source models, Mobi 1eLLM-R1-------- R1-950M model attains higher MATH accuracy than Olmo 1.24B (Allal et al. 2025) and 2x higher than SmolLM2 1.7B (Allal et al.) 2025), while significantly outperforming both on code benchmarks despite having fewer parameters.
We have disclosed the complete of open-sourced datasets employed in our study and have released all trained models and accompanying code to enable full reproducibility and foster future research.
2 PRE-TRAINING: BALANCE OF CAPABILITIES
The notion of reasoning in large language models (LLMs) remains both complex and contested. While the term is sometimes used to describe model's ability engage in structured, multi-step inference (Wei et al. 2022 Kojima
Pre-training
Mid-training
Post-training
Phase (2T token)
Phase (100B token)
General (866k samples)
Phase (2T token)
Phase 2 (100B token)
Math/Code SFT (3.2M samples)
Random Initialization
MobbileLLM-R1-bast
MobileLLM-R1
Figure 2: Overall training pipeline of Mobi 1LMMM-1..
2

---

Published as a conference paper at ICLR 2026
et al. 2022), it has also become proxy for improved performance on challenging benchmarks (Srivastava et al., 2022 In this work, we adopt a pragmatic stance: we treat gains on reasoning-centrie benchmarks as reasonable evidence of enhanced reasoning behaviors, while remaining cautious about equating such gains with genuine reasoning ability in the cognitive sense (Bender Koller, 2020). Concretely, this entails selecting informative datasets that most effectively enhance the target capability (Section 2.1 and optimizing their combination ratios to maximize knowledge acquisition within the fixed token budget (Section Figure 2 illustrates the training pipeline with the full procedure in Appendix A
2.1 SELECTING INFORMATIVE DATASETS FOR TARGET CAPABILITY
To systematically assess which pre-training distributions most effectively support downstream reasoning behaviors, a naive approach would be to pre-train separate models on all combinations of candidate datasets, followed by mid- training and post-training, and then measure performance on reasoning benchmarks. However, this strategy is both computationally prohibitive and prone to overfitting to specific benchmarks.
Instead, we design a lcave-onc-out (LOO) analysis. We train models from scratch on the entire set of pre-selected high-quality datasets, excluding one dataset at a time. We then trace negative log-likelihood (NLL) on curated copability-probiiie datasets throughout training. Each capability-probing datasel can be viewed as defining a token distribution that implicitly induces the necessary preconditions for reasoning to emerge. Importantly, these distributions are heterogeneous: when learned, they contribute unequally to different reasoning-related capabilities, such code understanding, general knowledge, and mathematical problem solving (Chen et al. 2021, Hendrycks et al. 2021 Cobbe
et al. 2021).
2.1.1 CURATION OF REPRESENTATIVE DATASETS AND CAPABILITY-PROBIE DATASET
Curating the sapability-prabinng datasets is critical: it must be representative of the desired capabilities and sufficiently comprehensive to cover each reasoning category. We describe the process of preparing capability datasets as follows.
Hierarchical Rejection Sampling. To derive a compact capability-probiiie datasets for each domain, we employ a hierarchical rejection sampling pipeline that integrates multiple classifier- and model-based filters. The objective is to construct a small yet representative target dataset for each capability, such that it can serve as a faithful proxy for reasoning performance while dramatically reducing overall volume during evaluation. For each corpus in Table 5 we first apply the FINEWEB-EDS classifier (Penedo et al. 2024) to select samples with high educational value, retaining only those with classifier scores above 4. Next, we incorporate model-based evaluation by scoring each remaining sample using the Ask-LLM paradigm (Sachdeva et al. 2024).. The evaluation prompt asks the model to judge whether a sample should be included in a reasoning- proiiinggg-oonggggggog dataset, framed as a binary classification task ("1" for inclusion, "0" for exclusion). Rather than relying solely on the hard prediction, we record the probability assigned to "1" as graded measure of the model's confidence in the example's 's reasoning relevance. For all Ask-LLM scoring, we select the top 10% samples within each dataset. This step complements classifier-based quality filtering by directly capturing signals of reasoning relevance, consistent with recent findings that costly, fine-grained quality samplers can outperform simple maximum-coverage approaches in terms of data efficiency (Sachdeva al. 2024 Pang et al. 2025 Chen & Zhou 2023). Next, we apply a dlomain-specife prompt to Ask-LLM for each capability with specific emphasis on code, math, general knowledge or combined. Finally, we perform semantic deduplication across corpora, shrinking each dataset in Table to a subset of roughly 10,000 examples. This yields the representative datasets DR, ;, each containing highly representative samples for its corresponding corpus.
We categorize them into three domains according to their composition: Code (C), Math (M), and Knowledge (KC):
C = (StarCoder, StackFxchange, Nemotron-Code, Cosmopedia, Natural Reasoning, pes2o}
.M- = [OpenWebMath, FineMath, Algebraic Stack, Nemotron-Math, Cosmopedia, Natural Reasoning, pes2o)
IC = |Fine Web-Edu, Wikipedia, Arxiv, Cosmopedia, Cemotron-Sciencee Natural Reasoning, pes2o)
Note that a single dataset may contain data relevant to multiple domains, in which case its representative subset is included in more than one domain. In this way, we construct three filtered, domain-spednt speraaai-seii capabilitypprobing datasets, E.M. by combining the representative subsets from all datasets assigned to each domain. We use (C, M,,. to denote a mixture of original datasets prior to down-sampling.
2.1.2 DISENTANGLING THE IMPACT OF DATA SOURCES
We then evaluate the impact of different pretraining corpora on the emergence of reasoning ability by measuring the negative log- licechooooo (NLL) on the cepability-probbing datasets. To isolate the contribution of each corpus, we perform
3

---

Published as a conference paper at ICLR 2026
General Knowledge
Math
Code
4.8
2.7
Normaizzed
4.6
wiki
1.55
1.8
wio starcoder
open web math
w/o stackexchange
3.3
finewebedu
w/o algebraic stack
300K 400k
500k
100k
Training Steps
Training Steps
Training Steps
Figure 3: Leave-one-out analysis of pretraining data. The y-axis represents the normalized negative log- leellikelioood (NLL) the curated capability probing datasets. systematically exclude individual Sttasets-StttCodder, OpenWebMathh FineWeb-Edu, Wiki, Arxiv, StackExchange, and Algebraic Stack to quantify their per-token contributions to downstream performance by comparing them with using full set of data. These trajectories reveal how beneficial each dataset is and how its impact evolves throughout training. Removing FineWeb-Ede yields the largest cross-domaln degradation, likely attributable to its board web-based composition that connects diverse domain.
rigorous leave-one-out ablation studies, systematically removing individual datasets and measuring the resulting change in NLL across the three capability- prribit datasets corresponding Code, Math, and General Knowledge capabilities.
Group Impact via Loss Delta. We define the impact of dataset Dj reasoning capability as the change in loss it induces on the corresponding capability-proiiie dataset D.,,,.. Let denote parameters trained on the full dataset D U,D, and و_, denote parameters trained with Dj removed. The group impact of Dj on De, MC, K. is
AC(D, = BENDE ((=;0=,)(((2;;
(1)
where is the evaluation loss. A positive value indicates that removing D increases the benchmark loss (i.e., Dj is beneficial), while negative value suggests that its presence may hurt performance.
Leave-One-Out Ablations. We operationalize Eq. [ by training models under leave-one-our settings and measuring the resulting differences in loss across benchmarks. Together, these analyses highlight not only which sources matter most, but also how much marginal benefit additional data from a given source provides. This methodology allows us to disentangle the contributions of heterogeneous data sources to reasoning-related performance in code, knowledge, and mathematics.
Figure 3 presents the results of our leave-one-out (LOO) experiments across the three evaluated capabilities. To ensure fairness, tokens from each dataset are sampled with equal probability, and no example is repeated during pretraining. Without this normalization, larger datasets such as FINEWEB- EDU would otherwise dominate exposure. We find that excluding FINEWEB-END
results in the largest degradation across all capabilities, including knowledge, math, and code. We attribute this to its web-based composition, which provides broad and diverse coverage across domains. This result highlights the central role of large-scale web data form of "glue" that binds heterogeneous domains together.
In contrast, domain-specific datasets primarily strengthen their respective domains: STARCODER substantially improves code performance (and, interestingly, math), while math-focused corpora primarily benefit math. However, their transfer to general knowledge is limited. An unexpected observation is that STARCODER benefits math more than OPEN WEB- MATH benefits code, reversal of the commonly held view that mathematical data contributes disproportionately to coding ability Lewkowyez et al. (2022). Finally, WIKIPEDIA appears to contribute little to math or code compared to web or domain-specitf data, yet remains necessary as a structured and reliable source of factual knowledge.
2.2 DATAMIXING VIA CROOS--AAPABILIT INLF-INE
we demonstrate that the pre-selected datasets yield measurable utility, as evidenced by reductions in NLL on sapabilityproobii datasets. Building on this, we study token budget allocation: given fixed training budget, how should tokens be distributed across heterogeneous datasets to maximize downstream reasoning performance? Uniform sampling provides natural baseline but ignores the varying marginal utility of different datasets. Our key insight is that more informative datasets should receive proportionally larger sampling ratios. To operationalize this, we leverage the influence score quantify each dataset's contribution and guide principled token re- reiieighting.
In Section 2.1.2

---

Published as a conference paper at ICLR 2026
General Knowledge
Code
7.75
Original Datamix
Original
5.50
Original
7.50
Datamix
Datamix
27
5.25
Paplexity
7.25
5.00
7:00
4.75
26
8.75
4.50
6.50
4.25
23
4.00
100K
200K
400K
B0OK
B00K
500K
100K
200K
300K
400K
500K
Step
Step
Step
Figure 4: Comparison of data mixture strategies using averaged perplexity. Math is averaged over MATH-500, GSM8K, Code HumanEval, and General Reasoning is an average 9 tasks, including ARC-easy, ARC-challenge, BoolQ, PIQA, SIQA, HellaSwag, OBQA, WinoGrand, and MMLU. We compare Original (uniform) sampling with our derived Datamix. This mixture consistently lowers PPL on Code, Math, and Knowledge benchmarks, despite these benchmarks not being used during training or data selection.
Generally, let 0 denote parameters obtained by training dataset D, 24 training example, Ztest example from the target set, which in our case is capability probing set, and L(x, 8) the loss function. The influence score of r; on the test loss can be approximated as
[1I
0) Voo 0-) H VoL
where Hg- the Hessian of the training loss at 0" While directly computing the Hessian matrix Hg- is computationally prohibitive for large models, AutoMixer (Chang al.) 2025) proposes an efficient approximation method that bypasses explicit Hessian inversion and makes influence score calculation scalable.
We extend the AutoMixer framework by treating influence scores quantitative proxies linking individual training samples to capabilities. Concretely, the influence of a sample on the validation loss of a capability -poobing dataset measures the connection strength between the sample and the corresponding capability. Rather than using benchmark test sets, we employ samples from capability-probing datasets and compute influence scores separately for Code (C), Math (M), and Knowledge (KC) domains.
For each training sample x; from a source dataset, we compute its influence on the validation loss of all three capability- probing datasets. We term this "self-influence" when training and validation samples originate from the same capability and "cross-influence" if they target different capabilities. Because the source datasets are substantially large, we develop efficient influence estimation algorithm that operates on the representative dataset (defined in Section 2.1.I) of each source in Table 5) yielding computationally scalable surrogate that faithfully preserves cross- coos-cappbility contribution signals. Concretely. if x4 E DR StarCoder C DE we evaluate
Self-influence: I(I;; Trest E : Cross-influence: I(x;, Ptest O,,,), I(Fi, DR;
(3)
Here, checkpoints Og.e, O,,t, and 0x,r are obtained by training separate models to convergence on the full training sets of domains C, M,,,K, yielding domain-specializee parameters. Following the AutoMixer protocol, a single checkpoint is insufficient to capture the full training dynamics. We therefore compute influence scores at T = 10 evenly spaced checkpoints, weighting each score proportionally to its training step to emphasize later-stage training. These weighted scores quantify the evolving influence of example x; on the Code, Math, and Knowledge domains throughout training.
Then, the joint influence of sample is computed
T
Ijoint (xt)
Σ OF (C. t=1
Σ
aet I(z;; 0c,z,
(4)
where 0.,, is the checkpoint / for capability e, and Ocet are blending factors reflecting acquisition speed across checkpoints. We assign linearly inereasing weights dee x across the T checkpoints, and maintain uniform weights across capabilities c.
Each source dataset g is then assigned a sampling weight (w,)):
Wg
Eg Pg"
Pg
Pg
N
1
Σ
Izoine (I:)
(5)
2:E9
with Ng the token count of dataset 9 and 8; the length of sample T;. The resulting mixture respects the global budget N while prioritizing datasets whose samples show strong self- and cross-capability connections.
5

---

Published as a conference paper at ICLR 2026
DCLM
Math
Flan
0.0020
Stage General
Ganeral
Stage 1
Stage General General Math
0.004
Slage
0.003
Density
0.0015
General
Density
0.003
Ganeral
Stage
Density
Math
0.002
0,0010
0.002
0.0006
0.001
0.001
0,0000
0.000
0.000
-4000
-2000
2000
-4000
-2000
0
2000
-4000
-2000
2000
Influence
Influence
Influence
Wiki
PES20
StackExchange
0,0025
Stage
0.0025
Stage 1
0.004
Stage Genenal
Gerenal
General
0,0020
General
Stage
0,0020
Garena
Stage 2
0.003
General
Stage
Densty
0.0015
Math
0.0015
Stage Main
Density
Math
0.002
0.0010
0.0010
0.001
0.0005
0.0005
0.0000
0.0000
0.000
-4000
-2000
2000
-4000
-2000
0
2000
-400
-2000
2000
Influence
Influence
Influence
Figure 5: Histogram of influence scores for the general knowledge and math capabilities. In phase data samples exhibit varied influence scores. As training progresses, most samples eventually attain zero or negative influence scores, indicating a convergence point in the model-data co-cvolution. At this stage, the dataset's 's information has heen largely exhausted and can no longer contribute to further model improvement.
In this setup, we derive a closed-form solution for the data mixture ratio, enabling effective utilization of the limited token budget while enhancing each dataset's contribution to model performance. Using the representative datasets and capability probing datasets sampled from the training corpus, it makes influence score computation tractable and exposes how strongly each source dataset (Table (5) contributes to Code, Math, and Knowledge capabilities. This formulation enables principled weighting at the datasel level, grounding the mixture in empirically measured cross-domain influences rather than heuristic allocation. As illustrated in Figure 4 the resulting mixture consistently outperforms uniform sampling on Code, Math, and Knowledge benchmarks- none of which are accessed during training or mixture menstruction-ddmmonnttr the potential for benchmark-fire -free, self-adaptive data optimization.
3 MID-TRAINING: KNOWLEDGE COMPRESSION
After the model has been exposed to broad knowledge during pretraining, the mid-training phase focuses on compressing this knowledge and maximizing performance on target tasks. We design each mid-training phase with a limited budget of 100B tokens. Unlike pretraining, mid- -id-irrining induces dramatic shifts in weight distributions and necessitates a more sophisticated, co-evolving model-data mixture strategy. To this end, we propose a novel mid-training paradigm that enables self-boosting: the model trained on a given data mixture is used to compute influence scores for samples, which are then leveraged to dynamically remove negative influence samples and adjust the data sampling ratios for the next phase. As training progresses, the influence scores of data samples increasingly concentrate around zero or negative values, indicating near-complete utilization of the informative content in the dataset and convergence of the process. Notably, this self-evolving scheme requires access external benchmark datasets, yet it substantially improves performance on target benchmarks relative to uniform sampling.
We build upon the Dolmino dataset, which has been shown in the OLMo 2 (OLMo al. 2024) to be effective mid-training corpus. To enhance domain specialization, we augment Dolmino with additional mathematics and programming data, aiming strengthen the model's math and coding capabilities. Given a training example x; from the mid- mid-training dataset and a probe example Tiest from capability probing dataset DE,M, we calculate the influence score I(x;, Zwas 0). Here, rather than relying on separately trained models for domain-speciffe corpora, we leverage the pretrained model capture the dataset requirements at the current stage of training. The data-model co-evolution proceeds iteratively through the following steps:
(1) Sample-Ievel influence for rejection sampling. Intuitively, this step acts as a filtering mechanism: only training examples that positively contribute to the target capabilities are retained, while neutral or detrimental samples are
6

---

Published as a conference paper at ICLR 2026
discarded. Given the raw mid-trainie -iiiiiiinning dataset D(raw), at compression phase t we define the retained dataset as:
Dj { E D [(x;; 0,) > 0},
where 0, is the model state phase t. This rejection sampling can be interpreted as iterative data distillation process: the model continually refines its training distribution, focusing only on samples that yield positive transfer toward the target probing dataset.
(2) letaset-level influence for adaptive data mixing. Beyond sample-level filtering, we aggregate influence scores to the dataset level, enabling adaptive control of the mixing ratio among mid-training datasets, according Eqs. andd
(3) Train the model on the curated data and repeat the iterative process. The compressed dataset with the updated mix ratio used for continued mid-training:
0++1 MidTrain(Or, Dz)-
and the updated model 0.+1 provides refined influence scores for the next stage. This iterative compression continues until no additional samples yield a positive influence score. In practice, we find that two stages suffice to produce a well-compressed dataset that balances generality with targeted capability improvements.
Intuition: Distributional Compression of Influence. The compression phases can be viewed as iteratively distilling the mid-training dataset in alignment with the model's evolving capacity throughout training. In early phases, the influence scores are more varied because the model 0g is still under-trained. However, better aligned with the target distribution, and its estimates of sample importance are narrowed down (See Figure 5 This recursive interplay produces increasingly refined datasets: uninformative (or negative-influence) samples are discarded, thus amplifying the impact of high-value samples. Conceptually, compression phases mimic an iterative denoising process, where each step sharpens the signal from D(target) against the noisy background of D(raw), We terminate the iteration until the distribution of influence converges to approximately zero.
increases, the model becomes
Figure 5) shows histograms of influence scores for gen- eral knowledge and math across stage 1 and stage 2 of training. During stage 2, the distribution of influence scores undergoes a pronounced "compression": the range of values narrows, and extreme contributions become less pronounced. Intuitively, as the model becomes more capable, the influence of individual data samples con- verges toward zero, indicating diminishing impact on downstream reasoning performance. We further high- light the effect of this influence compression in Figure 6 Subsampled mid- mid-training data consistently outperforms the original mid-training set under both standard cross- entropy training and knowledge distillation. Notably, the original data experiences pronounced performance dip around 30K steps, whereas the subsampled data maintains higher downstream performance throughout training. A similar trend occurs with knowledge distillation using the ILLMM3-8B teacher model, though the performance gap is slightly smaller than under pure cross-entropy. These results indicate that compressing influence scores effectively identifies and preserves the most informative samples, leading to more robust and stable performance trends.
47.5
45.0
37.5
35.0
32.5
original
subsempled
original + 86
subsampled
10000
20000
30000
40000
50000
Step
Figure 6: Comparison of the impact on MMLU benchmark
between the original mid-trainne teaiiiiinng data and the subsampled data, with and without knowledge distillation.
4 EXPERIMENTAL RESULTS
Using the datasets from Sections 2 and we obtain 1ob Given that our primary goal is to elucidate how data curation in pre-training and mid-training builds strong small reasoning models, we leverage established supervised fine-tuning (SFT) datasets. We first apply Tülu-3-FFT (Lambert et al. (024) dataset for instruction alignment and AppenScienceReusonnngg. 2025) for -orientte SFT to extend context and elicit long of reasoning. We validate our training process and data choices, compare our model against prior work trained on the same reasoning SFT datasets.
OpenCodeReasoniige- (Ahmad al. (225) and Opessnatn (Moshkov al.]
Our ablation studies (Table I reveal several key insights into the two-stage post- training pipeline. (1) cnstuuctionfollowinn supervision (Tulu-SFT) provides crucial alignment signals that make subsequent reasoning adaptation significantly more effective than starting directly with reasoning data. (2) Domain- specific reasoning corpora (math, science, code) yield consistent gains on their respective benchmarks, while scientific reasoning data further exhibits strong cross-domain transfer to math and code. (3) Symbolic reasoning improvements
Post-training process ablation:
7

---

Published as a conference paper at ICLR 2026
often trade off with factual knowledge retention, introducing math or code data reduces MMLU performance, particularly in smaller models with limited capacity. (4) Decoupling alignment and reasoning proves essential: a staged approach (Tulu first, then reasoning data) consistently outperforms joint training, especially on math and general reasoning benchmarks.
Comparison with baselines identical reasoning SFT: To disentangle the contribution of curated pre-training and mid-training data from that of high-quality post-training data, we conducted an ablation study. Specifically, we finetune all baseline instruct models, as well as the Mobi 1 leLLM --R general supervised fine-tuned model (trained for epochs on the Tulu dataset), on the joint reasoning SFT corpus (OpenMathRResonnin) + OpenSciencceRezsoningg + OpenCodeReasonie
for one epoch. Our results in Table show that, under identical supervised fine-tuning, models with stronger pre-training and mid-training exhibit more robustly embedded knowledge, which in turn facilitates the elicitation of reasoning capabilities during post-training. In that sense, Mobi 1 eLLLM-R1 consistently outperforms prior models trained on fully open-source corpora, such as OLMo-2 and SmoILM, on reasoning benchmarks, Notably, our 140M and 360M checkpoints achieve substantial gains over SmoILM baselines, while our 950M model surpasses both OLMo-2 1,48B and SmolLM-1.77B despite its significantly smaller size.
4.1 FINAL RESULTS
HumanEval
GSMBK
While the model undergoes the pre-training, mid mid-training, and post-training stages, track its reasoning ability by measuring perplexity on two reasoning-focte focnssed benchmarks: HumanFval and GSM8K. Our results in Figure show that the perplexity on math sees a significant drop early during the second phase of pre-training. Interestingly, the same model, when subjected to the second phase of mid-training with limited data, exhibits dramatic perplexity decrease in HumanEval. This suggests that the knowledge acquired from math training is transfer- able to coding, enabling the model to develop coding abilities subsequently.
Perplexity
Perplexity
3.5
Premsá Ind--ee--Sleee
Preosin
Mal-ma--Sleeel MM mill Stage
Mid valn Sugee
SF1
SFT
2x100
10"
Training Steps
Training Steps
Figure 7: Evolution of reasoning capability during training, measured by perplexity reductions on caasoning-ocuseed benchmarks: HumanEEal for coding and GSM8K for math.
In the following, we position our trained Mobi MMLMMR111 within the context of prior state-of-the-art models and compare their performance. We present results for two sets of models: the base models, evaluated after pre-training and mid mid-training, and the final models after the complete training pipeline. The experimental settings and full training pipeline can be found in Section
Base Model Figure g compures base reasoning models across multiple benchmarks. We group them into fully open- source models (OLMo OLMo et al. (2024), SmolLM et al. (2025), Mobi 1 LLM-R1), RR),, with weights, data, and training recipes available, and partially open-source models (Qwen Yang et al. 2025). Gemma Team et al. (2025) LLaMA Dubey et al. (2024)), which released model weights and partial training procedures. Compared open-source models, Mobi 1ebieeLM--R consistently outperforms both OLMo and SmolLM across all parameter scales. For example, the 140M scale, MobileLLM-PP R1 achieves 16.3% GSMSK and 15.9% HumanEval, dramatically
fully
Table 1: Ablation studies on post-training stages, We use M, S, C to denote Cassonnh, OpenScienceecassonngg and OpenCoddRcasoningg datasets respectively.
Table 2: Evaluation of reasoning capabilities elicited by different models when fine-tuned on the same reasoning supervised finetuning (SFT) datasct. Bascline models their instruct checkpoints; our model uses interme- diate Tulu3-SFT checkpoints, denoted with All models are trained for one epoch on the joint reasoming SFT cor- pus (OpenMathReasoning + OpenScienceeeassoning OpenCode 2).
Stage 1
Stage 2
MATH GSM8K LCBv6 MMLU
Ablation on Stage I
w/o Tulu-3
M C+S
56.2
68.2
13.1
44.0
w/ Tulu-3
C+
57.8
68.5
13.7
43.7
Ablation on Stage 2
Model
Size
MATH
GSMSK
LCBv6
Tulu-3
M 57.4
68.2
0.0
43.1
Tulu-3
C
16.2
31.0
12.0
39.9
Smoll M2- 12-133M-Innttuct
Instruct
135M
3.2
1.6
0.6
Tulu-3
23.8
62.2
Mobil BabileLLM-RIL R1-140M
140M
3.7
L.1
Tulu-3
M+C
58.4
65.6
10.9
40.4
Smill 560MT Instrict"
"362M
3.2
".44"
33."
Tulu-3
M
60.0
66.9
0.6
45.0
MobileLIM 1A--IM------MM--- R1-360M*
359M
19.2
23,8
4,0
Tulu-3
C+S
14_3
OLM--2-04255 IB-SFT
"1,888"
53.0
"58.8
11.4
Tulu-3
M+ S
57.8
68.5
13.7
43.7
EmnlLLM2-I.7B-Instte
1.71B
41.4
50.5
7.4
Joint Ablation
MobileL.MMRR1-99
R1-950M*
949M
57.8
68.5
13.7
Tulu-3 (M S)
56.2
53.1
14.9
44.0
Tulu-3
(M + S)
57.8
68.5
13.7
44.0
8

---

Published as a conference paper at ICLR 2026
GSMBk 61.6
HumanEval
MMLU
52.4
30
40
42.4
30.4
30
10-
26:0
20
10
3.0
5.
Mobiles 3.-44--MMMMM Note
LMN 12534
MosifeLL HooieeLLM----- 23000 1500--t01se
Mo MANIeLLLMM--R g3pe 250ae-los
OLMo 1a23 Ialane Fl 1-250
Gatema fottet
tase
Oheen Carona- 1.
LaMee 338 bain (1348)
MobileLLMM. 444--20------- 2440----- te
MD) 1750 taice
Maisted M4- M4--330----55
Smeat stalt take
Mobitet soo Jonse
28225 18 iI 150
Smotad 7B lave
Gamma a 2034
Dieedl Genma 3.1
Uaaets baie (--308
42 Lapa
Mobiles M.R. seole dorka Small Lari
Mob MobifeLLM-RRL lase
oute 1.23 take (1 1-408)
Smatas 1.1 tasse
German 3otto Gorma 3-18pt
Owend 2-.---00 base
Lattt 1i8-aaee 2426
Open-source weights
weights
training recipes
Figure 8: Performance comparison of base models across three tasks: GSM8K, Hvma and MMLU. Models are grouped by parameter size and color-coded by model family: Mobi 1eLLM-R. R1 (purple), SmolLM (orange). OLMo (yellow), and other partially open-source models (gray). Labels indicate model name and size for select models. Mobi 1eLLM-BB R1 consistently achieves strong performance across tasks while remaining parameter-effiiine A comprehensive comparison is presented Table
Math
24ME
LCBv
25.1
20.0
74.0
23
15.0
18.0
10
7.5
20
25.22
5.0-
4.4
5
2.5
0.6 0.3
00
0.
0.0
Smuttd Mn metner
MobiiaLLM- LM-81-14--11
MobitaLM 300M
SmalLMG intrrc
oute 10 ioitttn 439)
Mobiled LM- 93-300
Smolktt 7B matrrad
Laaiss a- (-224)
Deep Baok shert- diertl--heeem 1.10
Orem 68
Gesine-3 28-2
Ou 2425 28 Instrct 4000
MobiteLMM 30-3
Smos 788 Paantrrcct
LaaMt au dulramect (4-20))
Dremilent Glatee Owern 488
clnsumee 18.3
Smodt 1554 Mobilel Lak -4-11.440M
Mabite LM 41 Smuitat Mottt-nstred instrect
GLM 2023 Inieeret 4800
ModaleL 13300
Gemmu 3-27044
LEA L. everict (1.24g)
Despleetat ii D.w.k.kkkooooo Coeem- 1.88
Corm 8-1-a
Gumma
Open source weights
Open source weights source training redpes
Figure 9: Performance comparison of post-trained models across three tasks: MATH, AIME'24, and LiveCodeBench-vv. The full comparison results are provided in Table 9
surpassing SmoILM2-135M (1.8% and 0.0%, respectively). Compared to prior partially open-source models, such as Qwen3-0.6B,
achieves comparable or superior results despite being trained on substantially fewer tokens (4.2T for Mobi 1elb-e1 vs. 36T for Qwen3). Notably, Mobile MerbileeLMM-1-9000 attains the highest HumanEval score (46.3%) among all sub- IB models, significantly outperforming Qwen3-0.6B (30.5%).
Post-trained Model Figure 9 presents the performance of post-trained models. Notably, on LiveCodeBench, small models below 400M parameters struggle to produce reliable outputs. In contrast, Mobilel..M--1110000 points, surpassing even models with over IB parameters, such as Smo1LM2-1.77 Gemma3-1B, and LLaMA3.2-1B Remarkably, Mobi le LLLLM--1--- R1-950M demonstrates a substantial accuracy gain over Qwen3-0.6B on LiveCodeBench and even matches the performance of much larger state-of-the-art models, such as Direoseee-R--isiil-------- Across Math and AIME benchmarks, Mobile RobileLLae LLLMMR11 consistently outperforms other fully open-source models and achieves scores comparable to the partially open-source Qwen3 series. See Appendix B. for detailed comparisons.
achieves 5.1
5 RELATED WORK
The advent of GPT-3 (Brown al. 2020) highlighted the transformative potential of large language models (LLMs), spurring research on both proprietary models (e.g., Claude (Anthropic 2024) and open-source alternatives (e.g.. LLaMA (Touvron et al. 2023a b. Dubey et al., 2024). Gemma (Team al. 2024a b (025), Qwen (Team] 2024 Yang et al. 2025) ). Scaling laws have been central to understanding how larger models elicit emergent capabilities and potential performance singularities, while afficiency-accuracyy trade-offs in smaller models are increasingly studied to optimize computational resources. MobileLLM Liu et al. (2024) pioneered on-device LLM deployment, followed
9

---

Published as a conference paper at ICLR 2026
by high-performance small models such as OLMO et al. (2024) and SmolLM Allal et al. (2025). reflecting broader trend toward transparency, including open-soureing weights, training data, and full pipelines.
Research has also shifted from instinctive response to explicit reasoning thinking, exemplified by OpenAl's 01 (Jaech et a 2024) and DeepSeek-RI [Guo et al. (2025). Qwen Yang et al. (2025) another example of high-quality reasoning model, with smaller variants achieving state-of-the-art benchmark results. Prior studies suggest reasoning emerges only after extremely large-scale pretraining Yang et al. (2025). In contrast, our work shows that a small model can attain strong reasoning abilities using only 4.2T pretraining comparreeeee to Qwen trained on 36T tokens. We release our full data collection and training pipeline with a detailed rationale for each data selection, ensuring maximal transparency and reproducibility.
6 CONCLUSION
We present data-centric framework to maximize reasoning in small language models under limited parameters and tokens. We introduce benchmark- free, self-evolving data optimization, a principled dataset-level weighting method that leverages cross-domain influences to dynamically tailor the data mixture. This approach enables strong performance code, math, and knowledge benchmarks without exposing any benchmark data during training or mixture construction. Trained on 4.2T tokens drawn from 2T curated open-source data, McbileLLM-RI achieves state-of-the of-the-rrt results among small models with a fully open-sourced recipe, and matches Qwen3-0.6B with only 11.7% its 36T-token traniigg data. Our findings challenge the conventional belief that small reasoning models require massive data, instead underscoring the pivotal role of data quality, token efficiency, and principled data curation.
ETHICS STATEMENT
Our work investigates methods for optimizing the training and deployment of small-scale language models. The models and datasets used in this study are publicly available and widely adopted in the research community. We did not collect new human subject data, nor did we rely on sensitive or proprictary sources. Potential risks primarily concern the general risks associated with language models. While these risks are not specific to our contributions, acknowledge them and emphasize that our focus is methodological rather than on downstream deployment. We encourage future applications of our techniques carefully assess ethical considerations in their respective domains.
REPRODUCIBILITY STATEMENT
We follow the ICLR reproducibility guidelines. All datasets used are publicly available, as we revealed in Section A.3 We describe data processing procedures. model architectures, training configurations, and hyperparameters in detail in Sections A We will release code and trained model checkpoints to support full reproducibility.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint urXiv:2303.08774 2023.
Wasi Uddin Ahmad, Somshubra Majumdar, Aleksander Ficek, Sean Narenthiran, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Vahid Noroozi, and Boris Ginsburg, Opencodercaasoinngii ii: A simple test time scaling approach via self-critique. arXiv preprint arXiv:250700007,, 2025.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydliček, Agustín Piqueres Lajarin, Vaibhav Srivastav, al. Smollm2: When smol goes big-data-centric training of small language model. arXiv preprint arXiv:2502.0273:1 2025.
Anthropic. Claude. https://clauch a. 2024. Large language model.
Emily M. Bender and Alexander Koller. Climbing towards NLP's Everest: Avoiding the pitfall of understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5185-5198, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-199 1901, 2020.
10

---

Published as a conference paper at ICLR 2026
Ernie Chang, Yang Li, Patrick Huber. Vish Vogeti, David Kant, Yangyang Shi, and Vikas Chandra. AutoMixer: Checkpoint artifacts automatic data mixers. In Wanxiang Che, Joyce Nabende. Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 19953, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 119-8-89176-251-0, doi: 1999
10. 120116
ptittps://aclanthhlogyy..
Fei Chen and Wenchi Zhou. Quality over quantity: An effective large-scale data reduction strategy based on pointwise v-information. arXiv preprint urXivv2507.00038, 2025. URL https::/ /ttp:///aaxxvvv.. wwwp://..ggg...o......o.o abs/ abw.aa/...././///../// abtpa//////0000 00033
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al. Evaluating large language models trained on code. In arXiv preprint arXiv:2107.3377, 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, et al. Training verifiers to solve math word problems. In arXiv preprint 4rXiv:2110.14668, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv e-prints, pp. arXiv 2407, 2024.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravee, Zac Hatfiel- Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.06652 2022.
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun, Rankme: Assessing the downstream performance of pretrained self--upprrised representations by their rank. In International conference on machine learning, pp. PMLR, 2023.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang. Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in iims via reinforcement learning, arXiv preprint arXiv:2501. 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference I Learning Representations (ICLR), 2021.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models zero-shot reasoners. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannanch Hajishirzi. Tulu 3: Pushing frontiers in open language model post- traininningn..n. 2024.
Aitor Lewkowyez, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206. 14858, 2022. Utt
//arxiv. //arxiv.org/absl
org//rgg//sso.oooo abs/ httpa/bb///222e222 220...286688
Dacheng Li. Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir Patil, Matei Zaharia, et al. Llms can easily learn to reason from demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.0733 2025.
Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first International Conference on Machine Learning, 2024.
Ivan Moshkow, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman, Aimo-2 winning solution: Building statc-of-thc-art mathematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504.1o0 2025.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint vrXiv:2501.00650 2024.
Xi Pang et al. Fine-grained data selection for Ilm supervised fine-tuning, arXiv preprint arXiv:2502.01968 2025. URL https: //axxiv.org/bbst 2et.bb//222201163333
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37: 30811 1 108849, 2024.
11

---

Published as a conference paper at ICLR 2026
Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-eflicient Ilms, arXiv preprint (XKi:2402.09663 2024.
Ranajoy Sadhukhan, Zhuoming Chen, Haizhong Zheng, Yang Zhou, Emma Strubell, and Beidi Chen. Kinetics: Rethinking test-time scaling laws. arXiv preprint (tti::250600333333 2025.
Aarohi Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. In arXiv preprint 220.:222004663... 2022.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint urXiv:2403.08299 2024a.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint 4rXiv:2408.00116 2024b.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report, arXiv preprint arXiv:2503. 19786, 2025.
Qwen Team. Qwen2 technical report. arXiv preprint urXiv:2407.1067) 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:230..1391, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv22307.09288, 2023b.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint 2rXiv:2506.20512, 2025.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quac V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
An Yang, Beichen Zhang, Binyuan Hui, Bofci Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.1211 2024.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.00 0583888800 2025.
Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. Reasoning by superposition: A theoretical perspective chain of continuous thought. arXiv preprint A.Kv:2505.12544 2025.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A FINAL RECIPE
To ensure reproducibility and provide a transparent foundation for future research, we begin by detailing our model architecture and training setup, including the curated data sources and the complete recipe. Building upon fully open- source data, Mobi eLLM-R1 establishes a new standard among transparent reasoning models, achieving substantially higher accuracy than prior fully open-source efforts such as OLMo and SmolLM.
A.1 MODEL ARCHITECTURE
Our model architecture is based on designs from MobileL.LM Liu et al. (2024) and LLaMA3.2 Dubey et al. (2024). We adopt the LLLMA3.2 tokenizer with a 128k subword vocabulary. We also incorporate QK-norm to mitigate training instabilities in the self-attention block. Following MobileLLM, adopt weight sharing between the input and output embeddings to improve parameter efficiency. A complete set of architectural specifications is reported in Table 3
Table 3: Detailed architecture specifications of Mobil 1 --. "Dim" denotes the embedding dimension and "Hidden Dim" represents the dimension inside the beod-forward network.
Model
Layer
Head
KV-Head
Dim
Hidden Dim
Params
Moo1 1et.IM--1-
15
3
576
2048
140.2M
SobileeLMM-R--66
15
1024
4096
359.4M
MooileLLN-RI R1-950M
22
24
6
1536
6144
949.2M
A.2 TRAINING RECIPE
The complete set of training hyperparameters is detailed in Table with stage-specific configurations as follows:
Pre-training phase: Models are initialized from scratch and optimized with Adam (31, B2, c) = (0.9, 0.95, 10-8) and weight decay 0.1. The learning rate employs 2k-step warmup followed by linear decay to 0.1 the peak value. Mid-training phase: Optimization continues with Adam, where the learning rate decays linearly to zero. Knowledge distillation is applied with Llama-3.1 1----3..------- --8--nnstruct model as the teacher, where the student is trained by minimizing the KL divergence between its output logits and the teacher's logits.
Post-training phase: Adam used with zero weight decay. The learning rate warmup ratio is set to 0.03 for general- pprepoe- SFT and 0.1 for reasoning-specific SFT, followed by linear decay zero
Table 4: Training setup across different stages.
Stage
Phase
Tokens / Samples
BS
Seq. Len.
Steps Epochs
LR
#GPUs
Time
Phase
tokens
16
500k
4.00F.03
16x8
4-5 days
Pre-training
Phase 2
2T tokens
16
500k
4,00E-03
16x8
4-5 days
Phase 1
100B tokens
4
4k
50k
3.60F-04
16x8
1-2 days
Mid-training
Phase 2
100B tokens
4
4k
50k
3.60E-04
16x8
1-2 days
General SFT
866K samples
4
4k
2 epoch
5.00E-06
16x8
~22h
Post-training
Reasoning SFT
6.2M samples
8
32k
4 epoch
8.00E-05
16x8
~25 days
A.3
DATA
A.3.1 PRETRAINING
To balance general language understanding with strong reasoning capabilities, we design a two-stage pre-training curriculum with carefully curated data sources (Table Phase emphasizes broad coverage through large-scale web and educational corpora such as FineWeb-Edie which provide linguistic and domain diversity. At the same time, we seed the mixture with reasoning-rich corpora, including OpenWebMee model early to mathematical and scientific discourse. In Phase 2, we deliberately shift the weighting toward specialized reasoning datasets such as FineMath, OpenWebMath, Algebraic Stack, and Facebook Natural Reasoning- while reducing the proportion of generic sources. This skewed allocation ensures that the model continues to benefit from general pre-training signals while increasingly focusing on structured reasoning tasks, ultimately aligning the training distribution with our goal of developing compact yet strong reasoning model. The total token count of the datasets
Arxiv, and StackExchange, to expose the
13

---

Published as a conference paper at ICLR 2026
Table 5: Pre-training datasets and mixture ratios for the two-stage curriculum, with 2T tokens sampled per phase.
Dataset
Rows
Tokens (B)
Phasel Mix
Phase2 Mix
StarCoder
206,640, 614
263.8
10.66%
0.52%
WebMath
6,117.786
12.6
6.93%
23.33%
Fine FineWeh-ldu
1,279,107 107,432
1300
63.75%
54.83%
W1K1
7,222,303
3.7
5.03%
0.14%
1,533,917
28
6.36%
1.32%
StackExchunge
29,249.120
19.6
5.03%
0.86%
Algebraic stack
3,404,331
12.6
2.25%
1.26%
Nemotron science
708,920
2
0.03%
Nemotron code
10,108,888 883
16
0.72%
Nemotron math
22,066,39 397
15
3.01%
Costuupelaa
31,064,744
25
2.70%
Facchook natural reasoning
1,145,824
1.8
3.18%
rine Naath
48,283,984
34
8.01 %
peS20
38,800,000
50
0.08%
Total
100%
100%
used in per-training is 1.8T. We sample data from each source according the predefined mixture weights, drawing total of 2T tokens per phase.
A.3.2 MID-TRAINING
For mid-training, we construct mixture that complements pre-training by targeting benchmarks and reasoning-intensivg domains (Ta The first phase emphasizes coverage of general-purpose datasets (c.g., Dolmino DCLM baseline, FLAN, and peS2o) alongside curated knowledge sources such as Wiki and StackExchange. This configuration is intended to improve the model's broad knowledge, reflected in the performance in general benchmarks like MMLU, where factual recall is critical. In the second phase, we deliberately skew the mixture toward math and coding corpora, particularly Dolmino Math, Nemotron CCCCnath,, and Nemotron-Codee while reducing the weight of general-purpose datasets. We also introduce small but targeted of benchmark-style datasets (e.g., GSM8K, ARC, OBQA) align training with downstream evaluation. This two-stage weighting scheme allows the model to first consolidate broad knowledge and language understanding before specializing on sussoning-intesivv math and code tasks, which are central to our objective of building compact yet competitive reasoning models. The reported data mix ratios correspond to post sub-sau sampling. We will also release the code for the sub-sampling procedure ensure full reproducibility for anyone following our protocol. For each mid-training phase, we sample 100B tokens from the sources following the predefined mixture distribution.
Table 6: Mid-training datasets and mixture ratios, with 100B tokens sampled per phase according the mix ratio.
Dataset
Subset
Rows (M)
Phasel Mix
Phuse2 Mix
DXLLM Baseline
606
37.03%
6.51%
FLAN
57.3
4.10%
0.72%
peS2o
11.41%
2.01%
Dolmino
Wiki
6.17
2.66%
0.47%
StackExchange
2.48
2.12%
2.00%
Math
11.63%
29.10%
Nemotron- Cemoorron-Prettaiiing SetrainiingCoode...
882
20.69%
29.10%
Nemotron
Nemotron- C-Mahh-vl
144
3.45%
19.40%
StarCoder
StarCoder
206
6.90%
9.70%
TriviaQA (train)
OBQA Hram)
SaturalQuestaonn (train)
PIQA (triin)
Benchmark Set
~0.01
0.97%
GSK (Iram)
BonlQ (Irai))
AKC-Fasy Fasy (Irain)
ARC-U AKC-Uhallenge (frain)
Total
100.00 %
100,00
14

---

Published as conference paper at ICLR 2026
A.3.3 POST-TRAINING
In the post-training stage, we leverage established post-training datasets. Following standard practice, first align the model with instructions through general supervised the context length and promote long chain-of thain-of-thought (CoT) reasoning style. The datasets used in post-training are shown in Table
(SFT) and then apply reasoning-specifii SFT to extend
Table 7: Post-training data.
Stage
Dataset
Rows
General
Tulu3-SFT
866K
OpenMahhResoning
3.2M
Reasoning
UpenScuenceKcssonnng
802K
Opent odeKeusoning-2
2.2M
B COMPARISON WITH PREVIOUS METHODS
B.1 DETAILED ACCURACY COMPARISON
Table 8 demonstrates the performance of sub-billion and near-billion parameter reasoning base models across six widely used benchmarks. The results demonstrate that the proposed Mob IeLLM-R1 models consistently outperforms prior open-source models of comparable or larger scale. In the <150M regime, MobileLIM R1-140M delivers substantial gains over SmoILM2-135M, particularly GSM8K (16.3 vs. 1.8) and HumanEval (15.9 vs. 0.0). Similarly, MobileLIM MeweeLMMM--6660 R1-360M surpasses both Gemma-3-270M-p and SmolLM2-3600
achieving more than double the accuracy MATH500 and GSM8K. At the higher end, Mobil meviieeLi LLM-R1 LLM-R1-950M matches exceeds the performance of larger models: it improves upon Qwen3-0.6B in code generation (HumanEval: 46.3 vs. 30.5) and commonsense reasoning (58.6 55.3), while remaining competitive math and programming tasks. These results highlight that carefully curated training data, rather than sheer scale, is the key driver reasoning performance lightweight models.
In Table 9 we present a comparison of reasoning-orienter models across diverse benchmarks, including MATH500, GSM8K, AIME'24, AIME'25, and LCBv6, all evaluated in zero-shot settings. The results show that MobileLLM L-LMM]- RR models deliver substantial improvements over prior sub-billion billiillon parameter instruct-tuned baselines. In the < x550M and 150M-4000 ranges, Mobil LLM-RR1 achieves up to 8x higher accuracy on MATH500 and GSM8K compared to AmnolLM2-Inssrree and Gemma- 3-it variants, while also providing consistent gains on LCBv6. At the higher end, MobileLIM -950M reaches an AIME'24 score of 15.5 a level of reasoning performance previously unseen
Table 8: Performance comparison of reasoning base models across multiple benchmarks. Here, CommonSense Avg. denotes an average of 8 tasks in CommonSense Reasoning benchmarks including ARC-casy, ARC-challenge BoolQ, PIQA, SIQA, HellaSwag, OBQA, and WinoGrand. Models with fewer than 150M parameters do not yield reliable MMLU scores and are therefore denoted as
MATH500 (4-shot, em)
GSM8K (8-shot, em)
MBPP (3-shot, pass@) 1)
HumanEval (0-shot, pass@) 1)
CommonSense (0-shot, avg acc.)
MMLU (S-shot, acc.)
Model
Size
<150M
Smoll MM-555M2-335
135M
0.4
1.8
3.8
0.0
50.7
MobileLIM-BR 140M-base
140M
4.6
16.3
S.4
15.9
44.3
150M 400M
Gemmmu--227M4.
268M
0.6
1.1
2.0
3.1
48.4
26.5
SmolLM2-360M
362M
1.8
5,0
19.4
0.0
56.6
24.7
359M
13.4
39.4
20.8
32,9
51.0
26.8
400M - 1B
Qwen2.5-0.5B
494M
14.8
41.8
29.6
28.1
52.3
47.5
Owen3-0.6B-Bazz
596M
29.8
60.9
39.0
30.5
55.3
52.4
MobileLLM-R1-990N
R1-950M-
949M
26.8
61.6
39.2
46.3
58.6
47.4
>/8
Gemma--3-1Bppt
1.00B
0.6
2.4
9.4
6.1
57.3
26.1
LLaMA3,2-1B
1.24B
26.6
17.1
58.4
32.0
0LMMo-2-0425115
1.48B
39.8
7.8
6.7
Qwen2.5-1.50
1.54B
31.0
68.4
44.6
36.6
58.7
61.2
SmolLM2-11.7B
1.71B
11.6
31.8
35.4
0.6
62.9
50.0
Qwen3-11.7B-aac 1.7B-Base
2.03B
38.5
76.2
56.4
47.6
60.9
62.1
15

---

Published as a conference paper at ICLR 2026
Table 9: Evaluation results of different reasoning models MATHS00, GSM8K, AIME'24, AIME'25, and LCBv6 benchmarks. For AIME, evaluate models across rums and report average accuracy. For LiveCodeBench. results are reported the average accuracy across 16 runs. Models with fewer than 400M parameters do not produce reliable AIME scores and are therefore denoted as '
MATH500
GSMSK
AIME'24 0-shot pass n=64
AIME'25 (-shot pass
LCBv6
Name
Size
0-shot pass
O-shot pass
O-shot pass n=16
n=64
<150M
Smoll Smoll.M2- SobileLMM-R1-1400
135M
3.0
2.4
0.0
140M
6.2
4.1
1.7
150M - 400M
Gemma-3-270M-it
268M
6.8
8.4
0.0
SmollLM2-960M--Isttruu
362M
3.4
8.1
0.7
SobileLiM-R1-300
359M
28.4
24.5
5.1
400M 1B
Qwen2.5-0.5BB
Bren2.550.5B-Inntruue
494M
31.2
48.1
0.3
Qwen3-0.6B
596M
73.0
79.2
11.3
17.0
14.9
MobileLIM-R1-998
949M
74.0
67.5
15.5
16.3
19.9
>/8
Gemma-3-iB..
1.00B
45.4
62.9
0.9
0.0
2.0
LLaMA3.2-IB-nnmme
1.24B
24.8
38.8
1.1
0.2
ULMo-2-0425 I---steeB-l
1.48B
19.2
69.7
0.6
0.1
0.0
DpenRessonngg-eemotoe 1.5B
1.54B
83.4
76.7
49.7
40.4
28.3
DeepSeck-RI PrrnSeck-RI-Diseee Distill-Qwen--
1.54B
83.2
77.3
29.1
23.4
19.9
Qwen2.5-1 Instruct
1.54B
54.0
70.0
2.5
0.9
7.9
SmoILMM-11778 1.---...---nstuuc
1.71B
19.2
41.8
0.3
0.1
4.4
Qwen3-1.7B
2.03B
89.4
90.3
47.0
37.0
29.8
in open models trained on fully transparent datasets -whiee maintaining competitive accuracy on GSMSK (67.5)
and surpassing Qwen3-0.6B in LCBv6 (19.9 vs. 14.9). Although larger proprietary models such Qwen3-1.7B achieve stronger absolute results, MobileLLM-H
and OpenReasoning-Nemotee reasoning emerge with far fewer training tokens when coupled with careful data curation and resampling. These findings underscore the central claim of this work: reasoning capabilities in lightweight models are not contingent on scaling to massive proprietary corpora.
demonstrates that strong
C ON-DEVICE PROFILING
We benchmark the latency of Mobi 1eLLM-R1 models (140M, 360M, 950M) and LLaMA-3 models (1B, 3B, 8B) using ExecuTorch on a Samsung Galaxy $22 (8 GB RAM). In this experiment, we run each model with warm-up phase and then average the results over three iterations using the same prompt. On-device applications usually adopt quantized settings. To reflect typical on-device deployment, models are benchmarked under quantized settings: linear layers are quantized using 8-bit dynamic activations and 4-bit weights. For the 125M model. group size of 32 is adopted since its hidden dimension not divisible by 128, whereas all larger models use a group size of 128. Emhedding layers are quantized 4-bit with group size of 32.
Latency is reported in tokens per second across generation lengths ranging from 1k to 32k. As expected, latency decreases context length grows and model size increases. When context length becomes large, the KV cache size approaches the size of the weights. Since we are already memory-bound when loading weights, the additional cost of loading the KV cache is no longer negligible.
For long-context reasoning, the sequence length can rapidly extend to 32k tokens. At this scale, larger models quickly hit memory constraints. Thus, enabling efficient long-context on-device reasoning necessitates smaller models. Beyond memory efficiency, smaller models also provide significantly higher generation throughput, as speed scales approximately inversely with model size, underscoring the importance of compact models for practical on-device reasoning.
Table reports the measured generation throughput (tokens per second) of Mobi LobileLLM-RI models and LLaMA-3 baselines across different context lengths the Samsung Galaxy S22. The results confirm the clear efficiency advantage of smaller models: the 140M variant sustains over 100 tokens/s up to 8k tokens, while even the 360M and 950M models maintain practical throughput across medium-length contexts. In contrast, larger models such LLaMA-3.2-33 and
16

---

Published as a conference paper at ICLR 2026
Table 10: Generation speed (token/s) across context lengths for different model sizes. OOM indicates out-of-memory.
Model
Size
1k
2k
4k
8k
16k
32k
Mabi 1e1.MM-R1114M
140M
129.67
116.16
111.42
110.47
96.38
79.71
MobileL1MM-R13300
360M
77.23
61.54
61.24
61.27
61.13
O0M
Mobilee11M-R1-900
950M
31.05
27.59
25.09
25.36
00M
00M
LLaMA3,2-1B
1.24B
28.71
24.99
21.99
22.00
0OM
OOM
IJaMA3.2-33
3.21B
9.04
8.61
0OM
COM
00M
0DM
LLaMA3.1-8B
8.03B
2.76
3OM
00M
COM
30M
ODM
LLaMA-3.1-8B encounter out-of-memory failures beyond short contexts, and the LLaMA-3.2-1B model runs at less than one-quarter the speed of Mobiles MeeileeLMM-BB 1LLM-R1-140M, R1-140M. Moreover, long-context settings (16k-32k) quickly exhaust memory capacity for billion-scale models, highlighting the prohibitive cost of VV-cache storage at scale. These findings demonstrate that sub-billion parameter models not only enable reasoning with reduced memory overhead but also achieve far higher generation speed, making them far better suited for on-device long-context inference.
D ABLATION ANALYSIS AND INSIGHTS
D.1 LEARNING RATE AFFECTS REPRESENTATION LEARNING
We conduct ablation study the effect of learning rate during pretraining. The setup employs 16 nodes with 8 GPUs each, per-device batch size of 8, and training for 500k steps with a sequence length of 2048, corresponding to approximately IT tokens. The learning rate follows a linear decay schedule from its peak value 10% of the maximum, with an initial warmup phase of 2k steps. The mid-training stage follows the same optimization recipe as used in our final configuration.
Our results indicate that, while pretraining MMLJ performance remains largely unchanged across learning rates, the differences become significant after mid-training, as shown in Table Models pretrained with larger learning rates exhibit superior downstream accuracy, suggesting that they acquire stronger intermediate representations during pretraining. Importantly, pretraining accuracy itself does provide predictive power for downstream outcomes.
Therefore, we investigate structural diagnostics of pretrained representations that can act as low-cost proxies for Marrido
downstream generalization. We draw inspiration from RankMe Garrido al. (2023). originally proposed in the context of vision self self-supervised finetuning:
Min(N,K) Σ
(2 o2
RankMe(Z)
=exp
Pk log Pk
where pk
+
(8)
min(N.K) i=1
where dx (Z) denotes the singular values of output embeddings Z. Our analysis indicates that downstream performance is closely tied to parameter utilization during pretraining: models that activate broader, task-relevant subspaces and maintain high-rank embeddings facilitate fine-tuning to more effectively recover and exploit useful representations.
Table 11: Performance comparison between pre-training and after mid-training under different learning rates.
Pre-training
After Mid Mid-training
Learning Rate
MMLU
RankMe Score
MMLU
4e-3
27.92
21.98
36.31
2c-3
27.67
20.19
33.95
le-3
26.87
14.22
29.02
4e-4
27.50
7.81
27.55
In our case, the RankMe score correlates strongly with post mid-training MMLU accuracy, supporting its utility proxy measure that could potentially provide early signals and save resources. Furthermore, analysis of the RankMe score shows that higher learning rates lead to higher representational rank. This implies that larger fraction of parameters are effectively utilized during pretraining, which turn enables stronger generalization and improved final task accuracy. We note that this is preliminary study and represents a promising direction for future investigation.
17

---

Published as a conference paper at ICLR 2026
MobileL LM-b1e 950M-base
MobileLLM -MM-1-9-----M-
70
450
54
2800
68
53
GSMBK Score
GSMBK Scate
66
Length
300
2400
48
64
200
46
2200
100
20 40 80 80 100
20
40 60 80
100
Step
Figure 10: Accuracy and sequence length during RL fine-tuning on Mobi 1eLLM-R1 base model and final model after supervised finetuningSSTT). Small models benefit from RL when pretrained suitable corpus, with 1ELLMM-R119000 11---01------- showing steady reasoning gains, However, small model trained with SFT data outperforms base+RI,, and additional can degrade small models optimized with SFT.
D.2 DISCUSSION; WHETTHER TO USE RL OR NOT
A central question in the development of small reasoning language models is whether reinforcement learning (RL) is beneficial. Recent works such as DeepSeek-RI -1 (Guo et al. 2025) and Qwen3 (Yang et al. (2025) adopt two-stage paradigm: (i) train large model with RL, and (ii) generate strong reasoning traces to distill smaller student models, either via sequence-level distillation rationale is that large models, equipped with greater exploration capacity and self self-improvement potential, can discover more effective reasoning trajectories, while small models, being capacity- consttiyc.nttrrinedd are better suited to imitating curated demonstrations rather than performing exploration directly.
combination of logit-level and sequence-level distillation. The underlying
To assess the impact of reinforcement learning (RL) small reasoning models, perform ablation study by applying an RL stage to both the base model and the final model, i.e., fine-tuned with SFT data. The base model is finetuned for 100 steps on the TULU3 dataset (1% of total data) cold start, solely to learn the correct output format. Subsequently, both the base and SFT models undergo GRPO training on the NuminaMath-TTI dataset with a learning rate of 3 10-6, 100 training steps, batch size of 32 prompts, and 8 generations per prompt. The KI. coefficient for the reference model is set to 3 0.
The results Figure highlight several key observations: (1) Small models can also benefit from RL-based fine-tuning when they are well-pretrained on a suitable corpus. Results show that Mobi leeLeLLMM WELLLLM-R--000 R1-950M-base achieves evident improvements in reasoning accuracy as the average length gradually increases. (2) Supervised fine-tuning (SFT) data distilled from large models, with data source listed in Table consistently yields higher performance than directly applying RL to small models, corroborating prior findings. For instance, the final GSM8K accuracy of RL-optimized MobileIIMMPR eL..-RR ------1--5--M-- performing small models fully fine-tuned on SFT data, additional RI. does not observe a significant performance improvement. This suggests that SFT provides more structured and reliable supervision than the noisy self-exploration signal accessible to small models, which often lack the capacity to further refine their reasoning policies beyond the distilled demonstrations.
is 57.0, compared to 74.0 for the SFT-trained Mobi eet.leeMMRRRRR 1.L..--R1-990 1950M.. (3) For high-
D.3 COMPUTATIONAL OVERHEAD THE DATA CURATION PIPELINE
To provide detailed characterization of the computational overhead of our method, we measure and report the data curation costs, including representative dataset construction, influence score computation and leave-one-out ablation.
Hierarchical Rejection Sampling for producing representative datasets: ~00 GPU hours.
Pre-training influence computation: 100 GPU hours.
Mid-training influence computation: 11 100 GPU hours.
Leave-one-out ablation: ~0 GPU hours per dataset, ~ 400 GPU hours in total.
Total: ~6,800 GPU hours.
Training cost:
Pree-trinning ~28,000 GPU hours.
18

---

Published as conference paper at ICLR 2026
Mid-traine Mid-training: ~ 10,000 GPU hours.
Post-training: ~600 GPU hours.
Total: ~-8600 GPU hours.
The detailed training cost can be found in Table Our analysis indicates that the data curation pipeline itself is also computationally efficient: the combined cost of data curation and ablation experiments constitutes only approximately 15% of the total cost of data curation and model training.
E
USE OF LLM
We leverage large language models (LLMs) to assist in drafting and polishing written content. Specifically. LLMs are used to improve clarity, coherence, and readability of the text, while ensuring that technical accuracy and intended meaning are preserved. All outputs generated by the LLM are carefully reviewed and edited by the authors to maintain factual correctness and align with the scientific content. The LLM serves tool support writing efficiency, generate original research ideas or conclusions.
19


Reviews:
[{"review_id": "pLUD2002up", "reviewer": "Reviewer_GtCo", "summary": "The paper presents a details of data-centric framework for creating high-performing sub-billion-parameter language models (LLMs) with strong reasoning capabilities, named X-LLM-R1. The central finding challenges the assumption that reasoning requires massive training corpora (>10T tokens) by demonstrating that models trained on only 4.2T curated tokens can match or exceed models trained on 36T proprietary tokens, like Qwen3-0.6B. Key methodological contributions include a benchmark-free, self-evolving data optimization approach using cross-domain influence scores to dynamically manage the training data mixture, along with a phased pre-training and mid-training curriculum focused on token efficiency. The authors highlight the critical role of data quality and structured post-training (SFT) over sheer scale, ultimately achieving state-of-the-art results among small, fully open-sourced reasoning models.", "questions": "1. Can you please test in details the long context inference performance with these models?\n\n2. Please compare with SoTA reasoning models for the reasoning tasks (not only the instruction tuned models).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novel data curation and training strategy\n\n2. Challenging the conventional belief of training with large corpora of tokens for reasoning capability emergence\n\n3. SoTA reasoning on SLMs, and making models open weight. The performance gain over other open source SLMs are convincingly better.", "weaknesses": "1. Inherent Limitations of Small Model Capacity still remains a questionable issue along with the constraint of performance on long context data.\n\n2. Ineffectiveness of RL on SLMs is a concern here.\n\n3. The concept of reasoning remains vague in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mkjNOZ3dLd", "reviewer": "Reviewer_3Hy6", "summary": "This paper aims to enhance the reasoning capability of sub-billion-parameter language models through data-centric training strategies.\nTechnically, the paper presents two key contributions:\nInfluence-based DataMix for pretraining, which adaptively re-weights training datasets based on their estimated contribution to different reasoning domains.\n\nMid-Training Knowledge Compression, a dynamic filtering mechanism that removes samples with non-positive influence during mid-training, enabling co-evolution between model and data to reduce redundancy and stabilize learning.\nEmpirically, the proposed 950M-parameter model outperforms OLMo-2-1.48B and SmolLM-2-1.7B, while matching or exceeding Qwen3-0.6B, using significantly less data and compute. All training recipes, datasets, and checkpoints are reported to be fully open-sourced.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* Clear motivation and practical significance: The paper challenges the common “scaling is all you need” assumption and demonstrates that strong reasoning capability can emerge in small models through carefully optimized data recipes.\n* Technical contribution is sound: Extends AutoMixer from dataset-level weighting to multi-capability, sample-level influence estimation. The proposed method leverages internal capability-probing datasets to guide data weighting without relying on external benchmarks.\n* Experimental validation is concrete: Provides impressive empirical results and demonstrates strong efficiency, achieving competitive or superior performance to larger models with substantially less data and compute.", "weaknesses": "* Approximation reliability: The Hessian–vector product (HVP) approximation for influence estimation may introduce noise. Validation is indirect, without small-scale ground-truth comparisons or formal variance analysis.\n* Capability definition: Evaluations are limited to three domains—Code, Math, and Knowledge—oversimplifying the diversity of reasoning skills such as planning, commonsense, and multilingual reasoning.\n* Some missing baselines: The study lacks comparisons with strong alternatives, including AutoMixer, In-Run Data Shapley.\n\nChang, Ernie, et al. \"Automixer: Checkpoint artifacts as automatic data mixers.\" arXiv preprint arXiv:2506.21910 (2025).\n\nWang, Jiachen T., et al. \"Data shapley in one training run.\" arXiv preprint arXiv:2406.11011 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T23ZnAs4Xb", "reviewer": "Reviewer_dhHj", "summary": "In this paper the authors explore whether it’s possible to obtain strong reasoning capacity in sub-billion parameter models, without using proprietary data or an enormous compute budget. They introduce the X-LLM-R1 family of models ranging from 140M to 950M parameters, trained only on ~ 4.2T tokens. (NB: compare to Qwen-3, this is ~ 12% of the data). The achieve state of art reasoning results among open models, and demonstrate this ability comes more from data quality than pure scale, and show how to co-evolve the model and data using an influence score to focus the training procedure on data that will still contribute to the learning.\n\nThe authors also will (/ have but can’t link due to double blind review) release the data, training recipe, architecture implementation, making this an extremely transparent piece of work.\n\nOverall it was a pleasure to read this paper, and I hope my comments provide a useful perspective.", "questions": "1. Would it be possible to add a set of evaluations on the models general competence compared to Qwen (ideally others, but specifically as the closet similar model) to show if there is a trade off for general ability vs reasoning specific ability? (e.g. HeelaSwag, PIQA, NaturalQuestions… etc)\n    \n2. The leave one out analysis is really nice, but I was curious about the impact on downstream benchmarks rather than just the NLL? Would it be possible to see the impact of leaving out specific datasets on the reasoning tasks rather than just the model NLL? Alternatively could the authors comment on the effect / importance of the LLO beyond just the increase in pre-training loss and what this says about pre-training data mixtures?\n    \n3. You reweight the data at two boundaries. Do you expect more frequent reweighting to help? And where would the diminishing returns point be?\n    \n4. What is the total compute overhead of training a model in this way? (In GPU days) Relative to one epoch of normal training? i.e. can you show the cost of running the evaluations for the influence as part of the analysis?\n    \n5. I would also really appreciate to see how much of the dataset is required to be sampled when computing the influence to compute the weights?\n    \n6. Can you comment on / provide the impact of weighting on each of the datasets? ie. how much do different datasets get weighted up / down at different times, and how predictable is this? Do you need to compute the influence for new models, or is this relative proportion more or less static? In which case would it be possible to take the proportions and train a similar model without the influence step?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This paper presents an extremely clear example of the separation of pre-, mid-, post-training and fine-tuning - and gives strong evidence of the importance of each. This makes for an extremely valuable resource for others to build upon.\n    \n    - As does the open release of the model.\n        \n- The performance on key benchmarks is extremely good - especially for an open source, sub-billion parameter model.\n    \n- The LLO analysis offers valuable insight into the data mixtures required during an efficient pre-training phase, and which of these are the most influential.\n    \n- The key novelty presented in the paper is the use of an influence sampling / weighting method that computes the impact of training data on a approximate sample wise basis on the “probing datasets” - this allows the training corpus to be resampled between each phase during mid training to focus on high utility datasets, making the training process extremely efficient.\n    \n- The removal of low / negatively impacting data samples offers a clear and seemingly extremely efficient way to train models using significantly fewer tokens than naively would be expected.", "weaknesses": "- The focus of the paper is on the reasoning ability of the model - and makes only very passing comment (lines 362/379) on the impact of the knowledge retention / general usability of the model. Could further comment be provided on how the language understanding, factual retention, and general instruction alignment is affected with such a strong emphasis on reasoning during training?\n    \n- The method as presented is powerful - however the authors only show the influence sampling at two points during training (I believe) - between the two pre-training phases and between the two mid training phases. Could comment be provided on whether the authors think additional weighting phases would be more impactful? Or how frequently one could plausibly use such a method? To only compute the influence and weight twice might not be so different to hand curated weightings of data mixtures based on intuition. And that the real power of such a method might be in the ability to compute this frequently?\n    \n- There is obviously a compute cost associated with this sampling / probing approach - could the authors comment on this and put it in the context of the relatively small token budget required? One assumes the cost of this influence approach is not excessive, but it is noted the influence is computed only on a subset of the data.\n    \n- Additionally the weighting is applied on a dataset by dataset basis - what scope is there for differentiating between samples in a dataset which are well understood by the model vs individual samples which are poorly described?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vvLy3VufR3", "reviewer": "Reviewer_9Hby", "summary": "This paper challenges the prevailing assumption that achieving strong reasoning capabilities necessitates massive training datasets (e.g., >10T tokens). The authors propose a data-centric and highly curated training pipeline, introducing a series of sub-billion parameter language models called X-LLM-R1 (140M, 360M, 950M).\n\nThe core contribution of this work is a fully open-sourced \"recipe\" for efficiently training small reasoning models, which features two main stages of innovation:\n1.  **Pre-training Stage:** A \"Datamixing via Cross-Capability Self-Influence\" strategy is proposed. This benchmark-free method automatically optimizes the mixing ratios of various open-source datasets by calculating the influence scores of data samples on \"capability-probing datasets\" (covering code, math, and knowledge).\n2.  **Mid-training Stage:** A \"data-model co-evolution\" strategy for knowledge compression is introduced. This strategy dynamically computes sample influence scores during training and iteratively removes (filters out) samples with zero or negative influence, enabling the model to absorb knowledge more efficiently.\n\nExperimental results show that the X-LLM-R1-950M model, trained on only ~2T tokens of curated open-source data (4.2T tokens total pre-training), matches or even surpasses the performance of Qwen3-0.6B (trained on 36T tokens) on several reasoning benchmarks, particularly AIME and HumanEval. The authors commit to releasing all data sources, mixing ratios, models, and code.", "questions": "1.  **Regarding Curation Compute Cost:** Could the authors provide an estimate of the computational overhead required for the full data curation process (LOO analysis, influence score calculation, etc.)? How does the *total compute budget* (curation + training) of this method compare to the budget needed to simply train a baseline (like Qwen3-0.6B) on 36T tokens?\n2.  **Regarding Sensitivity to Probing Datasets:** The pre-training mix depends on the \"capability-probing datasets.\" How much would the final data recipe and model performance change if the construction of these datasets were altered (e.g., different Ask-LLM prompts, or a different sampling threshold than 10%)?\n3.  **Regarding the Knowledge vs. Reasoning Trade-off:** The MMLU performance in Table 8 suggests a potential trade-off where optimizing for reasoning (especially in pre-training) may reduce general knowledge. Was this an intentional design choice, or do the authors view this as an unavoidable trade-off for small models with limited capacity?\n4.  **Regarding Mid-training Convergence:** In Section 3, the model undergoes two \"knowledge compression\" stages. Why were two stages chosen? As shown in Figure 5, the influence scores are indeed compressed in Stage 2, but have they fully converged (i.e., reached the \"zero or negative influence\" state)? Could the authors provide the percentage of data that was filtered out in each stage?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  **Important Research Problem:** The paper tackles a significant research question: how to train small, efficient, and deployable reasoning models. Its argument for \"quality over quantity\" in data provides a valuable path for researchers with limited resources.\n2.  **Novel Methodology:** The core contributions (influence-score-based pre-training mixture and mid-training knowledge compression) are novel and principled. The \"benchmark-free\" nature of these methods is a key strength, avoiding overfitting to downstream benchmarks.\n3.  **Strong Empirical Results:** The results are highly competitive. X-LLM-R1 significantly outperforms other fully open-source models (like OLMo, SmolLM) at all parameter scales. The fact that X-LLM-R1-950M (4.2T tokens) achieves performance comparable to or better than Qwen3-0.6B (36T tokens) on key reasoning benchmarks is a strong demonstration of the recipe's effectiveness.\n4.  **Commendable Transparency and Reproducibility:** The paper excels in its commitment to openness. The authors promise to provide full details of their training recipe, including all data sources, mixing ratios, architecture, and hyperparameters. This transparency is of great value to the community.\n5.  **Insightful Ablation Studies:** The paper provides valuable insights through detailed ablations, such as the LOO analysis of data sources, the effect of learning rates, and the SFT vs. RL discussion.", "weaknesses": "1.  **Implicit Computational Cost of Curation:** The paper emphasizes its \"token efficiency\" but does not explicitly discuss the computational cost of the data curation process itself. For instance, (1) the LOO analysis requires training multiple models; (2) calculating influence scores, while scalable, also requires significant compute (e.g., training domain-specific checkpoints). A comparison of the *total compute* (curation compute + training compute) versus a \"brute force\" approach (like Qwen3's 36T) would make the \"efficiency\" argument more complete.\n2.  **Sensitivity to \"Capability-Probing Datasets\":** The pre-training data mix relies heavily on the constructed \"capability-probing datasets.\" Although this process is \"benchmark-free,\" the construction itself (e.g., using Ask-LLM with specific prompts for hierarchical rejection sampling) introduces designer priors. The paper lacks a sensitivity analysis on how robust the final data recipe is to changes in the design of these probing datasets.\n3.  **Trade-off between Reasoning and Knowledge:** The results in Table 8 show that while X-LLM-R1-950M-base excels on GSM8K and HumanEval, it lags behind Qwen3-0.6B-Base and SmolLM2-1.7B-base on MMLU. This suggests that the curation strategy, which is highly optimized for code and math reasoning, might come at the cost of retaining broad factual knowledge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)