Tina: Tiny Reasoning Models via LoRA
Authors: Shangshang Wang
Julian Asilis
Ömer Faruk Akgül
Enes Burak Bilgin
Ollie Liu
Willie Neiswanger
Venue: ICLR 2026 Poster (2026)
Keywords: Reasoning models
efficient reasoning
LoRA
RLVR
Primary area: foundation or frontier models, including LLMs
TL;DR: Our work reveals the surprising effectiveness of efficient RL reasoning via LoRA, and and provides hypotheses, supported by experiments, about why it works so well.

Abstract: How cost-effectively can strong reasoning abilities be achieved in language models? Driven by this question, we present Tina, a family of tiny reasoning models achieved with high cost-efficiency. Tina shows that substantial reasoning performance can be developed using only minimal resources, by applying low-rank adaptation (LoRA) during reinforcement learning (RL), to an already tiny 1.5B parameter base model. This minimalist approach produces models that are competitive with, and sometimes surpass, SOTA RL reasoning models built upon the same base model. Crucially, this is achieved at a tiny fraction of the computational cost employed by existing models. In fact, the best Tina model achieves a >20\% reasoning performance increase and 43.33\% zero-shot Pass@1 accuracy on AIME24, at only \$9 USD cost (i.e., an estimated 260x reduction). Our work reveals the surprising effectiveness of efficient RL reasoning via LoRA. We validate this across multiple open-source reasoning datasets and various ablation settings starting with a single, fixed set of hyperparameters. Furthermore, we explore the hypothesis that this effectiveness and efficiency stem from LoRA rapidly adapting the model to the structural format of reasoning rewarded by RL, while largely preserving the base model's underlying knowledge. In service of accessibility and open research, we fully open-source all code, training logs, model weights, and checkpoints.

Published as a conference paper at ICLR 2026
TINA: TINY REASONING MODELS VIA LORA
Shangshang Wang, Julian Asilis, Ömer Faruk Akgüll, Enes Burak Bilgin, Ollie Liu, Willie Neiswanger
University of Southern California (shangsha, akgul, ebilginn, zwiu2898, nei wwwnneeissneeeemee Buscg edu
ABSTRACT
How cost-effectively can strong reasoning abilities be achieved in language models? Driven by this question, we present Tina, family of tiny reasoning models achieved with high cost-efficiency. Tina shows that substantial reasoning performance can be developed using only minimal resources, by applying low-rank adaptation (LoRA) during reinforcement learning (RL), to an already tiny 1.5B parameter base model. This minimalist approach produces models that are competitive with, and sometimes surpass, SOTA RL reasoning models built upon the same base model. Crucially, this achieved at a tiny fraction of the computational cost employed by existing models. In fact, the best Tina model achieves a >20% reasoning performance increase and 43.33% zero-shot Pass@1 accuracy on AIME24, at only $9 USD cost (i.e., an estimated 260x reduction). Our work reveals the surprising effectiveness of efficient RL reasoning via LoRA. We validate this across multiple open-source reasoning datasets and various ablation settings starting with single, fixed set of hyperparameters. Furthermore, we explore the hypothesis that this effectiveness and efficiency stem from LoRA rapidly adapting the model to the structural format of reasoning rewarded by RL, while largely preserving the base model's underlying knowledge. In service of accessibility and open research, we fully open-source all code, training logs, model weights, and checkpoints |
INTRODUCTION
Language models (LMs) demonstrate increasing proficiency across a variety of tasks, but achieving robust, multi-step reasoning remains frontier challenge (Xu et al. 2025 Liu (222a)). Enhancing complex reasoning via supervised fine-tuning (SFT) is a well-adopter technique, often utilizing distillation process (Min al. 2024 Huang et al. 2024) by which the model learns mimic reasoning traces generated by more advanced models such as ol (OpenAl) 2024) and R1 (DeepSeek) Al 2025). This approach, while effective, can run the risk of instilling shallow form of imitation in the learning model. In contrast, reinforcement learning (RL) enables models to learn directly from verifiable reward signals derived from curated data (DeepSeek-Al) Al, 2025| Lambert al., 2025). In doing so, RL can lead the model to explore a greater variety of logical paths and discover more robust solutions. However, RL pipelines are often complex and notoriously resource-intensive, typically involving substantial compute. This raises fundamental question anchoring our research:
How cost-effectively can one perform RL to efficiently instill reasoning abilities in LMs?
This question is motivated by the need establish efficiency limits for RL reasoning, i.e., under- standing the minimal requirements needed to achieve meaningful reasoning improvements. We focus on low-resource settings with tiny models and minimal parameter updates probe the fundamental limits of instilling reasoning via RL, isolating training dynamics from sheer model scale. Although quantized small models offer an alternative path to efficient deployment, our aim to understand how far RL-based reasoning can be pushed under tight compute budgets. (RUCAIBox STILL Team 2025 Luo et al. 2025 Dang and Ngo 2025) Moreover, understanding these efficiency limits ad- dresses fundamental questions about the nature of reasoning in LMs. Specifically, recent studies
httpp::/ggitubb.cmm//saaagg
wang/Tina

---

Published as a conference paper at ICLR 2026
Data Amount (Thousands)
Computational Cost (9)
Time Cost (Hours)
Reasoning Performance (%)
10*
3623
Value (log scale)
56.00
10
150
44.47
41.00
10a
40
10a
10a
62
82
24
24
24
20
101
14
104
STLL-3
DaspScaleft
Open
Open-R52
Opan
Tina
STILL-3
Deep
Open-AS1
Open-A52
Open-R53
Tima
STILL-3
Deep
Open-A51
Open-R52
Open-R53
Tina
STILL-3
DeepScaled
Open-RDt
R81
Open-R52
Open-A53
Tina
Figure I: Comparison between Tina and baseline models Reasoning performance denotes the
average score across AIME24/25, AMC23, MATH500, GPQA, and Minerva. The calculation of each comparative metric is detailed in Appendix A.I
indicate that reasoning and knowledge storage are distinct capabilities: while knowledge capacity scales primarily with model size, reasoning performance may be less coupled to parameter count alone (Allen-ZZ -hu and Li 2024). This decoupling suggests that smaller models may possess untapped reasoming potential. Furthermore, evidence shows that efrfmeter-efficient methods can adapt models for specific capabilities without degrading existing knowledge (Han et al. 2024), thereby offering promising avenue for reasoning.
Therefore, in this paper we combine two key efficiency strategies: employing compact tiny base models (e.g.. 1.5B) (DeepSeek-A) All 2022)) and applying low-rank adaptation (LoRA) during RL. LoRA enables us to modify model behavior by training only a small fraction of parameters, making it an ideal technique for efficient reasoning. The synergy between these approaches forms the foundation of our Tina (Tny Reasoning Models via LoRA) modeling framework. Importantly, this paper not only introduces the Tina models but also uses this minimalist setup to characterize reasoning improvements and explain why such low-cost approach is effective. While approach is straightforward synthesis of LoRA and GRPO, the core contribution lies not in the method's s complexity, but in the surprising discovery that substantial reasoning can be achieved such a minimal cost, supported by an analysis of the training dynamics and a novel hypothesis to explain this effectiveness. We summarize our contributions follows:
Surprising Effectiveness of Efficient RL Reasoning via LoRA We show that our Tina models achieve performance competitive with, and in some cases even superior to, SOTA baseline models built on the same base model with full-parameter training, shown in Figure and and in more detail in Table 2 In particular, the best Tina model achieves a >20% performance increase and 43.33% zero-shot Pass @ accuracy on AIME24. Notably, the cost of reproducing the best Tina checkpoint stands at only $9, and of reproducing everything (all experiments, ablations, evaluations, etc.) presented in this paper from scratch $798. We also additionally show FLOPS for Tina in Figure 2 and Appendix B.
Rapid Reasoning Format Adaptation Hypothesis Based on our observations in RL post- training Tina, we hypothesize that LoRA's effectiveness and efficiency stem from rapidly adapting the reasoning format under RL while preserving base model knowledge-a likely more compute-efficient process than the deep knowledge integration of full-parameter training. Partial support comes from studies showing tiny LMs can reason effectively (Hugging Face (025 DeepSeck-Al] 2025). while large LMs can store broader world knowledge (Allen-Zhi and 2025). This distinction suggests reasoning capabilities can be significantly enhanced by focusing on adapting the output format itself, consistent with our hypothesis about LoRA.
2 RELATED WORK
Open-source reasoning replicas Following the release of ol ol-preview (OpenAl 2024), a number of open-source models have emerged to replicate exceed its reasoning capabilities. STILL (Min) et al. 2024) introduced a minimal yet high-quality training recipe designed to elicit reasoning with modest compute, demonstrating that imitation learning from curated traces remains competitive. Sky- Tl (NovaSky Team 2025) further explored scaling using open instruction-tuned checkpoints, while SimpleRI. (Zeng et al. 2025) highlighted the potential of lightweight RL without requiring large-scale
2

---

Published as a conference paper at ICLR 2026
reward models. PRIME CCi et al. 2025) and DeepScaleR (Luo et al. 2025) introduced process supervision and scaling experiments to isolate how reasoning quality evolves with model size and context length. s1 (Muennighoff al., 2025) showed that even strong base models such as Qwen2.5- 32B-Instruct benefit from fine-tuning on only 1 high-quality and long chain-of-thougg data, which is curated to elicit reasoning capabilities. I.1 (Aggarwal and Welleck, 2025) combined prompt engineering with data curation for RL, resulting in models that can efficiently and adaptively control their response length. Meanwhile, OREAL (Lyu al.) 2025) and OpenThinker (Open'Thoughts Team 2025) investigated self-correttion and latent structure emergence through unsupervised and hybrid paradigms. The release of Open Reasoner Zero (Hu et al. 2025) and Open-RS (Dang and Ngo 2025) further emphasized efficient RL-based strategies for reasoning with small models, completing landscape of public alternatives for interpretability and reproducibility.
RL with verifiable rewards Reasoning tasks are well-suited to RI. paradigms, as the correctness or quality of the final output often provides verifiable reward signals (e.g.. the validity of logical deduc- tion). Such signal can effectively guide the model towards learning more robust reasoning strategies. Consequently, various RL approaches have been explored within this domain. Certain methods introduce auxiliary reward models crities to assess reasoning quality, such as ReFT (Luong et al., 2024) and REFINER (Paul et al. 2024). Other techniques employ explicit aale-based verification for self-correction (Wu al. 2024). Some leverage self-play dynamics and exploration, such mutual reasoning (Qi et al. 2024), or utilize inference-aware fine-tuning that optimizes performance under different sampling strategies (Chow al., 2024). Notably, group relative policy optimization (GRPO) has been proposed as a variant of proximal policy optimization (PPO) which removes the need for a separate value network by using a group-based baseline for advantage estimation, improving training efficiency and leading to better reward alignment (Shao et al. 2024), as demonstrated by DeepSeek-RP R1 (DeepSeek-A]] 2025). Subsequently, Dr. GR.GRPO (Liu al. 20256 introduced subtle modification of GRPO addressing its bias to produce long responses.
3 METHOD
Tina is our family of models created by post-training the base model using LoRA during RL (employing a GRPO-style algorithm).. The "Tiny" designation encapsulates dcliberate focus minimalism and efficiency across the entire framework. This encompasses not only the tiny base model architecture and the tiny parameter updates enabled by LoRA, but also extends to tiny overall resource footprint. This minimized footprint is achieved through an efficient training pipeline leveraging accessible open-source resources, and requires only minimal hardware and budget resources (detailed in Section 4).
GRPO formulation Recall the following formulation of GRPO: For each question g. GRPO samples
group G (01. ... ... oG) of outputs from the old policy Ta and optimizes the policy To by maximizing the following objective:
E P(Q)..
Σ
(min (5,4, clip (5;, 1 - 1 c) A,) DxL (Tall
(1)
T
(o Taa (11g)
1-1
where we define 6; molorlo) and A, denotes the advantage computed from group of rewards (ri, (ri,P,,...,rrl rob.
(a)()
A, r; mean (ri, .e..).,
std (r), 12, .... .2,...,,r)) ro))
and
(xl( (1)( - log To(0;1g) 1
(re(()))))
(o,
Note that E and 3 are parameters controlling the clipping range and KL penalty, respectively.
LoRA formulation While most existing open-source models that enable reasoning rely the more expensive ball-parameter training (Min et al.
2024;
NovaSky Team 2025 Zeng et al. 2025
Muennighoff et al. 2025 Aggarwal and Welleck 2025
Cui et al.] 20255 Luo al.) 2025 Lyu
et al. 2025| Open' Owwonopeenuaaaeea Thoughts Team 2025) Hu et al. 2025 Dang and Ngo 2025), we investigate the
of LoRA for parameter-effhcien post-training of reasoning models (Hu al. 2021. Our goal
3

---

Published as a conference paper at ICLR 2026
to assess whether updating only small fraction of parameters still yield strong reasoning capabilities (Han et al. 2024). We follow the standard LoRA setup (Hu et al. 2021) that given frozen pretrained weight matrix Wo e I dxk and trainable low-rank matrices A E IR dxr and B E I rx with < min(d, k), the original forward pass h(x) Wox is modified to be
(xx) Wox + ABx.
In addition to its computational efficiency, LoRA provides modularity: by training only a low-rank decomposition of the parameter updates, it becomes possible to toggle reasoning behavior without maintaining multiple full model copies.
MODEL TRAINING DETAILS
4.1 TRAINING SETUP
We present main training setup follows. Please refer to Appendix A for additional details.
Training datasets & evaluation benchmarks To facilitate meaningful comparisons and enable precise ablations, post-train our Tina models via RL using the setups inherited from publicly available reasoning models. Tina models use open-source training recipes and training datasets from models like STILL-3 (RUCATBoX STILL Team 2025), DeepScaleR (Luo al.) (025), and Open-RS (Dang and Ngo 2025). These models are also used as baselines. We evaluate the reasoning capabilities of our Tina models and the baselines across diverse suite of six challenging benchmarks,
primarily focused on mathematical and scientific reasoning; AIME24/25 (Art of Problem Solving
2024), AMC23 (Art of Problem Solving 2023). MATH500 (Hendrycks et al. 2021 Lightman et
2023
GPQA Diamond (Rein et al. 2024)) and Minerva (Lewkowycz et al.
2022
Table 1: Computational cost breakdown Costs for all tasks in this paper, measured in USD. Our calculation based on the full of experiments shown in Appendix B
Training COST EST.
EVALUATION CoST EST.
TOTAl CoST
Baseline Models Re-Evaluation
$10
S10
Robust Evaluation
$110
$110
Main: Tin=-STTLLL&&-LS---rrveee
$59
$7
S66
Main: S.aa-DeeppcaleR..SS....ee
$84
$10
$94
Main: Tina-Open-RSI
$40
$11
S51
Main: Tina-Open-RS2
$15
Main: Tina-Open-RS3
315
$17
S32
Ablation: OpenThoughts Dataset
$84
$10
Ablation: OpenR1 Dataset
$59
$7
S66
Ablation: Th-Though Dataset
$84
$10
$94
Ablation: LIMR Dataset
$4
$4
$8
Ablation: Dr.RRPO Algorithm
$15
$17
Ablation: Learning Rate
$7
$8
S15
Ablation: LoRA Rank/Alpha
$14
$16
$30
Ablation: Format Reward Only
$15
$17
$32
Ablation: Long Completion Length
$15
$17
Total: All Tasks
$510
$288
$798
Total: Main Tasks
8213
$62
$275
Total: Ckpt. Each Main Task
Total: All Ckpt. Perforrenne Task
$14
$17
$31
Total: Best Ckpt. in Piss-Peffrmaece Task
$8
$1
$9
Computational cost breakdown We use a minimal setup with just two GPUs (NVIDIA L40S GPUs), accessible via commercial cloud platforms at approximate rate of SI USD per GPU hour, including 300 GB storage, based on pricing observed at the time of writing (Cudo Compute). The RL training process for our LoRA models proves be highly efficient, with single RL. step typically
4

---

Published as a conference paper at ICLR 2026
completing within one minute this hardware. Evaluating model checkpoint across our entire suite of six reasoning benchmarks requires approximately L40S GPU hour on average. To ensure cost control, initially established conservative maximum budget of $100 USD for each complete experimental run, encompassing all stages from training to evaluation and miscellaneous tasks. As detailed in Table our actual expenditures were significantly below this ceiling. Notably, all of our experiments required only minimal efforts on hyperparameter search, as detailed in Appendix AA.3
4.2 TRAINING TRICKS
Our training methodology for Tina models is designed to maximize cost-effectiveness and rapidly instill reasoning abilities, aligning with our core hypothesis that LoRA can efficiently adapt a tiny model the structural format of reasoning (detailed in Section ( Towards this goal, the following strategies were employed, utilizing a single, fixed set of hyperparameters across experiments to underscore robustness and minimize tuning overhead:
Concise reasoning with restricted completion length To encourage the model quickly learn the essence of effective reasoning- achieving accurate results through concise and well-structured reasoning paths we restrict the maximum completion length 3,000 tokens during training for all Tina models. This not only guides the model towards more efficient problem-solving expressions but also reduces the computational load per training instance, further enhancing overall training speed and cost-efficiency. An ablation study on longer completion lengths discussed in Section 6.2
Accelerated adaptation with rank-alpha ratio and learning rate scheduling To facilitate rapid assimilation of the new reasoning format, we deviate from common LoRA configurations and learning rate schedules. Specifically:
We set the LoRA alpha be four times the LoRA rank (e.g., rank 32, alpha 128), rather than the conventional two times. This amplified alpha encourages the model to more strongly incorporate the adaptations learned by the LoRA parameters, effectively making it "lean towards" the new reasoning structures reinforced during RL.
We employ an unconventional learning rate schedule. Instead of scheduling the learning rate decay over the standard training horizon, we schedule it over twice the training horizon. This results in a comparatively larger learning rate being applied at each step within the actual training period, promoting faster adaptation of the LoRA parameters the reward signals. These adaptation strategies are chosen to align with our goal of achieving significant reasoning improvements with minimal computational costs, and thus promote efficient learning.
5
EMPIRICAL RESULTS
5.1
SURPRISING EFFECTIVENESS OF EFFICIENT RL REASONING VIA LORA
The results presented in Table 2 demonstrate that significant reasoning performance can be achieved efficiently, yielding models that are competitive with, or outperform, relevant baselines despite the inherent resource constraints of using parameter-efficient tuning
In Table 2 all Tina models exhibit substantial reasoning aptitude, achieving average scores in the range of 48.16% to 50.60%. By default, our reported scores are zero-shot Pass@1 Mean performance. We also conduct robust evaluation experiment in Appendix [B.2] with zero-shot Pass @ 1 Mean @ 10 performance to show the robustness of our approach in this paper. Significantly, nearly all Tina models notably outperform their corresponding bascline average scores, indicating marked improvements instilled by the parameter-efficient RL. The Mana-Open-Re performance observed at 50.60% Furthermore, these strong results were achieved with very limited training durations, ranging from just 19% to 57% of a full training epoch, highlighting the efficiency and rapid adaptation enabled by the Tina approach. These findings strongly support our central
model yielded the highest average
Taales 2 and adopt consistent naming pattern where "Tina-X" denotes that our model is the LoRA counterpart of baseline model X or is trained on dataset X (possibly followed with an extra ablation setup). This can reflect the model origin and serve direct reference the public checkpoints for reproducibility.
5

---

Published as a conference paper at ICLR 2026
Table 2:
Tina model evaluation Performance comparison between Tina models and corresponding parameter-traine SOTA models on six reasoning tasks. The value in the Steps column indicates the training steps of the best model checkpoint within one epoch, while the full model checkpoint evaluation is shown in Appendix B.3 The Baseline column represents the average score achieved by baseline model with full hall-parameter RL (all details of baseline evaluations described in Appendix B.
TINa MODEL
STEPS r$ EPOCI)
ATME24
AIME25
AMC23
MATT1500
GPQA
MINERVA
AvG.
BASELINE
Tina STLLL-3-1. preevvee
53%
36.67
30,00
77.50
84.60
33.33
26.34
48,15
46.36
Tica- DeepScalee-11. S0-Previe
19%
43.33
26.67
67.50
86.20
57.88
28.68
48.38
46.74
Tica- Saen-ppenn---- RS1
20000
80,00
84.00
48.56
Tiea Open R52
87.00
58.60
Tiea Open R53
49.45
hypothesis: robust reasoning capabilities can be effectively and economically cultivated in small language models through the targeted application of LoRA and RL.
Table Tina ablation variants evaluation Performance evaluation of Tina's ablation variants on six reasoning tasks. The value in the Steps column indicates the training steps of the best model checkpoint within one epoch, and the full model checkpoint evaluation shown in Appendix B.3
ABLATION DATASRTS
STEPS (% EPOCH)
AIME24
AIME25
AMC23
MATH500
GPQA
MINERVA
OpenR1 (93.7k)
36.67
30.51
Yina-OpenThoughts (66.1k)
30%
36.67
26.67
72,50
$4.80
41.41
33.09
49.19
Tina Thought (53.3k)
30%
40.00
20.00
80.00
86.00
33.84
26.84
47.78
DeepScaleR (48.3k)
43.33
37.88
25.68
Tina-STILL-3 (33k)
53%
36.67
30.00
77.50
84.60
33.33
26.84
48.16
Tina S1 (18.6k)
34%
43.33
20.00
80.00
84.00
35.35
28.68
48.56
Open 85 (?k)
43.33
26.67
36.36
32.72
Tina-LIMR (1.39k)
58%
46.67
20.00
75.00
83,80
34.85
30.51
48.47
ABLATION ON LEARNING RATE STEPS (% OF EroCH)
AIME24
A1ME25
AMC23
MATH500
GPOA
MINERVA
AVG.
Tina-LTMR-5e-6-1 51-
295
36.67
26.67
75.00
83.60
35.86
29.41
47.87
Tina-LIMR-le-6a4
46.67
34.85
30.51
Tina LLBR--e-7-1r
58%
43.33
16.67
77.50
$4.60
34.85
30.51
47.91
ABIATIOON RANK
(% EPPCC)
A1ME:2
GPQA
MINKRVA
-r-k
29%
20.00
30.00
77.50
$4,20
38.38
31.62
46.95
58%
46.67
20.00
75.00
83.80
34.85
30.51
48.47
Tina- 16-1oRA-rank
43.33
33.33
28.31
Vina-LIMR-8 8-LRR---oo-t-
29%
30.00
26.67
82.50
83,80
33.84
30.51
47.89
Tina-Lime 1at---1-R---------- rank
86%
36.67
20.00
85.00
83.80
31.82
29.04
47.72
ABLATION ON RI. ALGORITIM STEPS (% OF EroCH)
AIME24
A1ME25
AMC23
MATH500
GPQA
MINERVA
AVG.
Tina-Open-853-GHE
36.67
31662
Tina Open R53 BrCxPO
17%
43.33
23.33
80.00
85.00
35.35
30.15
49.53
5.2 ABLATION STUDY: KEY FACTORS
To better understand the factors influencing the performance and efficiency of our Tina models within the proposed low-cost framework, we conduct series of ablation studies. These studies systematically investigate the impact of key design choices and hyperparameters: the underlying training dataset, the learning rate for LoRA updates, the rank of the LoRA adapters, and the specific RL algorithm employed. In each study. typically vary one factor while holding others constant, often based on a high gigh-performing configuration identified in our main experiments or preliminary runs. The results, summarized in Table provide valuable insights of our economical approach.
Impact of training dataset The first section of Table 3 highlights the influence of the dataset used for RL. We compared seven distinct datasets, varying significantly in size (from = 1.4k to *94k samples). Strikingly, the Tina-Open-RS model, trained concise dataset of merely 7k examples, achieved the highest average score (50.60%). This outcome surpasses models trained considerably larger datasets, such Tina-OpenRI (93.7k samples, 49.26% avg).

---

Published as a conference paper at ICLR 2026
This observation strongly supports core "Tiny" premise and reflects the intuition that the quality and diversity the dataset can matter more than the data size.
Impact of learning rate Using the Tina-LIMR configuration as a testbed (second section of Table 3p. we assessed sensitivity to the learning rate. Among the tested values (5x 10 1 1x10% -6 and 5 x 10 "), learning rate of X 10 yielded the optimal average performance (48 (48.47%
for this setup. While performance differences were not drastic, this indicates that learning rate selection remains factor, although effective results were obtained without extensive tuning.
Impact of LoRA rank The third ablation study investigated the impact of LoRA rank, which directly controls the number of trainable parameters. Testing ranks 4, 8, 16, 32, and 64 on the Tina-LIMR setup, we observed considerable robustness. Ranks 8, 16, and 32 all produced strong results, with average scores clustering between 47.89% and 48.922 48.92%. Notably, rank 16 achieved the peak performance (48.92%) in this comparison, slightly outperforming rank 32 (48.47%). Performance decreased slightly at the extremes (rank 4 and 64). This study validates that highly sarameter-efficient configurations (low ranks like 16 or 32) are effective, further enhancing the cost-effecttig coseteffectivssss and minimal overhead of the Tima approach.
Impact of RL algorithm Finally, we compared two RL algorithms, GRPO and Dr.GRPO (Lu et al. 2025b), using the Tina -eee--Opn---- RS3 setup (final section of Table 3. Both algorithms led to similar peak average performance levels (49.45% for GRPO vs. 49.53% for Dr. Gr.GR).). However, Dr.GRPO reached its best checkpoint significantly earlier in the training process (17% of an epoch vs. 57% for GRPO). This suggests potential advantages in sample efficiency for Dr.GRPO in this context with an alternative normalization in loss calculation, offering potentially faster convergence and further reductions in training time and cost.
EMERGING HYPOTHESIS
6.1 RAPID REASONING FORMAT ADAPTATION
To understand why LoRA facilitates both effective and efficient reasoning improvements via RL, we analyze the relationship between training compute and performance, alongside training dynamics.
Less is more for reasoning As shown in Figure 2 proximate training FLOPs reveals stark contrast between full-parameter and LoRA-based training regimes. We show FLOPS because they more invariant than price for cost comparison. First, our LoRA-based Tina models achieve reasoning scores comparable or superior to fully fine-tuned baselines while requiring (in some cases) orders of magnitude fewer training FLOPs. We observe that in Tina models, increased training compute inversely affects performance, in contrast to full nallmmarreeter ones. This highlights a "less compute can yield more performance" phenomenon.
plotting reasoning performance against ap-
Reasoning Performance (6 Tasks)
Tina BS2 Tina- Opa--ppe--S33
Tina- RS1
Tima- DeepScaleR
Tine STILL-3
STIILL- DeepScaleR
t Open-RSa
Open-RS1
Open-RS2
226
Approximate Training FLOPs
Figure 2: Less is more for reasoning Approximate training FLOPs vs. reasoning performance comparison between Tina and baseline models. The calculation is detailed in Appendix A.I
This finding supports our hypothesis regarding how LoRA is able to achieve such high efficiency, which relates the principle of "learn structure/format, maintain knowledge." We posit that LoRA
7

---

Published as a conference paper at ICLR 2026
excels in this scenario because RL for reasoning heavily rewards the model's ability to generate outputs in a specific, verifiable format or structure (e.g.. step-by-step reasoning chains). LoRA appears to be highly adept at learning these structural and stylistic patterns with minimal parameter changes, thus requiring very few FLOPs.. At the same time, because LoRA modifies only a tiny fraction of the weights, it largely preserves the base model's vast pre-trained knowledge. Therefore, LoRA efficiently teaches the model how to format its existing knowledge into effective reasoning traces, rather than potentially imposing costly relearning of concepts or procedures that extensive full-parameter updates might entail. We hypothesize that this focus on structural adaptation allows Tina to achieve high reasoning performance with minimal computational investment.
Tina Mliaa-pen--RS-
Ilna-Open-Ke R53
Tina- Tina-Open-Rst
Completion Length
0.2
Svedlted g
Smeetied DMA
Eesk Nurtsrann Chedlprrt
5000
0.5
Cosine
2000
-.2
0.2
Bud aurtammann theciparr
Teaiig Tamiing
1000
1200
250
500
750
1000
1250
Stap
Stap
Stap
Tina Opp----3--DrGR----
Tina- Rete--ppenS--GReee
Tina- RS3-
Scaled
Dan
Completion Length
3500
0.0
Formar.
0.75
Bcol Receeneeeeneeee Coopees
Traking Tammng
3000
2500
-02
0.25
2000
DasiFartrmanaa Cheqperi
0.00
200
1500
Tuiking Tamknng faee
200
400
Step
Step
Step
Figure 3: Phase transition in LoRA-based RL The "training turning point" in the legend means the step where the format-like metries (e.g., format reward, completion length) start to destabilize. Refer to Appendix for the full set of plots for all Tina models. All raw data is from the Weights & Biases training logs and smoothed via exponential moving average (EMA) with factor 0.1.
Phase transition We use the term "phase transition" to describe a specific, observable training dynamic: sharp, distinct turning point in format-related metrics (e.g., format reward, completion length) that is decoupled from the more gradual evolution of the accuracy reward. As shown in Figure 3 we consistently observe such a training phase transition, or turning point, evident in the format-related metrics (format reward, column 2; completion length, column 3) across most Tina models, thus not a random success during training. Around this transition point (indicated by the green vertical dashed line), the format reward often peaks or destabilizes, while the completion length frequently reaches a maximum before potentially reversing its trend. Notably, this relatively sharp transition observed in format and length metrics does not typically have corresponding distinct turning point in the accuracy reward plots (column l). The accuracy reward often exhibits more gradual fluctuations or slower drift over the training duration, without clear inflection aligned with the format transition.
Another erucial observation is the timing of optimal performance: the best-performing checkpoint, yielding the highest reasoning accuracy on held-out evaluations, consistently occurs just prior to or around this observed phase transition point in the format metrics. This decoupling between the dynamics of accuracy-based and format-based metrics suggests that the LoRA- RL process rapidly optimizes the model's 's ability to adhere the structural elements rewarded by the format score and length constraints. The subsequent transition point may signify where this structural optimization saturates, and becomes unstable. The fact that peak reasoning accuracy is achieved just before this format-driven transition implies that while learning the correct output format is essential and efficiently achieved via LoRA, pushing further on format-centric optimization alone does not necessarily yield better reasoning, and may even be detrimental.
6.2 ABLATION STUDY: COMPLETION LENGTH AND FORMAT REWARD
To further probe the "learn structure/format, maintain knowledge" effect and understand the dynamics of the observed "phase transition," " conducted targeted ablation studies.
8

---

Published as a conference paper at ICLR 2026
Tina Open RS3 long completion
Tina- Open- Open-AES-long.cooplltto
Tina- Open RS3 www.nnSnnnnopppeelttooo
Completion Length
Cosine Scaled
Smostad (EKA SM6
Formast
0.75
Bnd Prttenace
Smoothed CCA 0-221
4000
3000
Ray Caca
Diot Feo.Feeftrrnncc Cheigeer
0.00
Taaiing larking
Step
Step
Step
Figure 4: Phase transition with long completion length Training dynamics for Open- RS33 long-completion with 10k maximum completion length.
Long-completion training One of the training tricks detailed in Section 4.2 the restricted completion length (3k tokens), aiming to foster concise reasoning. To assess whether this restriction significantly influences the learning dynamics or inadvertently caps performance, we conducted an ablation where the maximum completion length was substantially increased to 10k tokens for a Tina model variant. As shown in Figure 4 increasing the completion length did not alter the training dynamics. The model's generated completion lengths during training peaked approximately 4k tokens, remaining well below the 10k limit and comparable to the effective lengths observed with the default 3k setting. Crucially, the "phase transition" effect in format-related metrics persisted, mirroring the patterns observed in Figure 3 The accuracy reward also followed similar trajectory. The persistence of the phase transition indicates that this phenomenon is a robust characteristic of the LoRA-driven format adaptation process, rather than an artifact of restrictive length cap. Furthermore, the final reasoning evaluation scores of this model variant are comparable to those achieved with the 3k limit, i.e., now 350 steps with 50.63 average score v.s. previously 500 steps with 49.45 average score (detailed in Appendix B.3)
Tina-Opena -p-----------oma-tonnn
R53-format-onnl
Tina- Tinna-Oeen-ee 3-3-forsst-oom
0.8
Smeethed 23AA pnd.11
Reasoning Steps
0.9
0.6
-000
0.4
-0.15
0.7
0.2
-0.20
Rol Futt
Kin Bit
0.0
Smoethee E3KA mm0
0.5
Seacthed (36A mm3
Step
Step
Step
Repetition Penaity
0.15
Bax Cate
Tima-Open-RS3--ormate
0.9
Tina-Open-AS3-formattoooo
Ban Data
Completion Lengtn
3500
Tina-Open-Pe R53-tommat---oni
Teg Count
0.88
3000
44
0.7
2500
0.25
0.6
2000
0.5
Smoothhed MoM 2.2.4)
800
Step
Step
Step
Figure 5: Absence of phase transition in format-only training Training dynamics for Lina-Open-RS3 RS3--frmmatooll trained solely on format-related rewards.
Format-only training Our hypothesis posits that LoRA excels at adapting the model to the format of reasoning that rewarded by RL, particularly when this format linked reasoning accuracy. To dissect the roles of format and accuracy rewards, we conducted an ablation study where Tina model variant was trained using only format-based reward components, with the accuracy-based reward entirely removed. Illustrated in Figure 5 the "phase transition" disappeared in this format-only training regime. While the individual format-related reward components and completion length absent.
still evolved, the sharp turning point destabilization signifying the phase transition The disappearance of the phase transition suggests that this dynamic is an emergent property of the interplay between LoRA's rapid adaptation to format requirements and the simultaneous drive to achieve reasoning accuracy. When the accuracy incentive is removed, LoRA may still adapt to the specified format objectives, but this adaptation becomes ineffective from the goal of producing correct reasoning. Specifically, it takes longer training time to reach similar reasoning performance, i.e., now 850 steps with 50.56 average score v.s. previously 500 steps with 49.45 average score. This underscores that while LoRA is highly efficient at learning structural patterns, this capability is harnessed when guided by the objective of accurate problem-solving (detailed in Appendix B.3).

---

Published as a conference paper at ICLR 2026
7 CONCLUSION
We presented Tina models to show that effective reasoning capabilities can be instilled in language models with high cost-efficiency. By combining LoRA with RL, we showed is possible to achieve reasoning performance competitive with larger models that necessitate far more costly aall-parmmeter training. We posited that LoRA's effectiveness stems from its ability to rapidly adapt the structural format of reasoning rewarded by RL, while largely preserving the knowledge embedded within the base model. This adaptation of reasoning pathways appears to be a more compute-efficiet process.
ACKNOWLEDGMENTS
The work was supported in part by the National Science Foundation (NSF) under Award No. 2427856. We also want to express our gratitude to the broader open-source community. This research was made possible by leveraging numerous publicly available resources, including training and evaluation framework, open datasets, accessible pre- pre-trained language models, and the insights shared through technical reports. The computational resources required for the experiments described herein were provided by the Center for Advanced Research Computing (CARC) at USC.
REFERENCES
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. Towards large reasoning models: A survey of reinforced reasoning with large language models, 2025. URL
https://arxiv/ httpx/iww.rggbbb/////0/
09686
Yue Liu, Jiaying Wu, Yufei He, Ruihan Gong, Jun Xia, Liang Li, Hongcheng Gao, Hongyu Chen, Baolong Bi, Jiaheng Zhang, et al. Efficient inference for large reasoning models: A survey, arXiv preprint arXiv:2503.23077 2025a.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng. Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems, 2024. https://// wttaaxxxvvvvooo wwwp:/ogg/abb///aeee ab1/2 09413
Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey -art 2: Surpassing ol ol-preveew through simple distillation, big progress or bitter lesson?, 2024. https:/::://ari ortpai//.rrgg/////////////.b
2411.16489
OpenAl. OpenAl ol system card, 2024. URL/tttps//aaxxxiorrr/////111
DeepSeek-AL. DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
12948
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu Pushing frontiers in open language model post-training, 2025. bttss:/:://aaxi org/abs/ 2411.15122
RUCAIBox STILL Team. STILL-3-I.SB-prevle Enhancing slow thinking abilities of small models through reinforcement learning. 2025. http:tts:///tthhie Cttb..ccm//UAAABo//SSlooo. Thinking_we wi thithh_1 LLMs LLMS
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and lon Stoica. DeepScaleR: Surpassing ol -rreeviww with 1.5b model by scaling rl, 2025. URL httphttts////eetccaat pratnttiiiappjjeeejee com/
10

---

Published as a conference paper at ICLR 2026
Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small Ilms: What works and what doesn't t, 2025. httpsts:: http://axxivv..oo wwwpiiivorrg//e httg//bb/2253.... 16219
Zeyuan Alu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint arXiv:2404.0552 05405, 2024.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey, 2024. URL httttttss///iiiiii http:////xxiiio..//bb.. 220//240.... 14608
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL. https
//gi //github. hoa.uu//c////gggggaaace//////
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. In Proceedings of International Conference on Learning Representations (ICLR), 2025.
NovaSky Team. Sky-TI: Train your own ol preview model within $450, 2025. URL https
//novasky-ai.co thub. wiw.husss.///ossssssssooos.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. SimpleRL- Zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025.
URL https://arxiv. org/ abs/2503. 18892
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding, Process reinforcement through implicit rewards, 2025. URL pattps://arxiin http://wwww.r......a/////o/ absaaas///20 250./5502.0000 01456
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Perey Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL https:
http::/://waxxvorrgrr//// wwt.:/..rggg//ooo/aa/.oo.. hts//bbb/2201.... 19393
Pranjal Aggarwal and Sean Welleck. LI: Controlling how long reasoning model thinks with reinforcement learning, 2025. URI httphttts:///zii.orrrg wwwpiivv..rgabb///55550 04697
Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, Weihan Cao, Jiangning Liu, Hongwei Liu, Junnan Liu, Songyang Zhang, Dahua Lin, and Kai Chen. Exploring the limit of outcome reward for learning mathematical reasoning, 2025. http:::: .rg.
06781
OpenThoughts Team. Open Thoughts, January 2025.
heatttp:://opee-thuuuee
ai
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung- Yeung Shum Xiangyu Zhang. Open- Reasoner-Zero: An open source approach to scaling reinforcement learning on the base model, 2025. URLhhtpps h/tp:::///ihhbbh www.giittu..oooomeoeeee httmm/pppn-eassne--erroor Zero/ -eapeen-Rasone--Zerrr
Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. ReFT: Reasoning with reinforced fine-tuning, 2024. URL htt/htttss//rrxi.orgg/bbss 04867
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West. and Boi Faltings, REFINER: Reasoning feedback on intermediate representations. In Proceedings of European Chapter of the ACL (EACL), pages 1100-1126, 2024.
Zhenyu Wu, Qingkai Zeng, Zhihan Zhang, Zhaoxuan Tan, Chao Shen, and Meng Jiang. Large language models can self-correct with key condition verification. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 12846-12867, 2024.
Zhenting Qi. Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller LLMS stronger problem-solvers, 2024. URL 06195
Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Aleksandra Faust. Inference-aware fine-tuning for of sampling in large language models, 2024. URL https:///rxivv wwwp/ggg////br/.///////////////e
15287
11

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang. Y. K. Li, Y. Wu, and Daya Guo. DeepSeckMath: Pushing the limits of mathemat- ical reasoning in open language models, 2024. URL/httss://arzii.com
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang. Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. 2025b. URL https:
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu. Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models, 2021. URL https
09685
Art of Problem Solving. Aime problems and solutions, February 2024. URL https:/ // artofproblemsolliing com http://gg....//ii/////eeeeppeepe phtplexxxppp//AMMEE/E BBApp/AIM__PPooolle ems Solutiono
Art of Problem Solving. Amc problems and solutions, 2023. URL https://
artoopprobeemoolving. cww/ wwt..//wwwwwweeee.....ee Baapp/AAC_12__rooob__ ems and___d_55. Soluutinn
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL 03874
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In Proceedings of International Conference on Learning Representations (ICLR). 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: graduate-level google-proof Q& benchmark. In Proceedings of Conference Language Modeling (COLM), 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 3843 3857, 2022.
Cudo Compute. Nvidia L40S pricing. URL spu-cloud/nvidia- 140s
Accessed: 2025-04-21.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. NuminaMath, 2024. URL www.tttss//uuggingacce. co/
AI AT-MO/NuminaMrte
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni- MATH: A universal olympiad level mathematic benchmark for large language models, 2024a.
URL https://arxii.... org/ abs/ 2410.441000 079985
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable., 2022.
thub. cww wwtp://wgggiggfcc///aeeeeeeeeeee
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning, 2020. URL http/httts://tt thub. com/ hwtp///gggggaac//ttrttn
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase. and Yuxiong He. Zero: Memory optimization towards training A trillion parameter models. CoRR, abs/ 1bs/1990,02054, 2019. htRLhttp:/// httt:////xxvv org/abs/1910. 02054
12

---

Published as a conference paper at ICLR 2026
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of European Conference Computer Systems (EuroSys), EuroSys '25 page 1279-1297, ACM, March 2025. doi: 10.1 155/3669031.3666609 URL http:::ppdx..oooorrg///.o//o 0tt.6//7........t///.d.......... 11111/.699145369033 3696075
Clémentine Fourrier, Nathan Habib, Hynek Kydlíčck, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for Ilm evaluation, 2023, URL lighteval
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h. h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang. Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 07 2024b. UtI.bttps://zenod
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of Symposium on Operating Systems Principles (SOSP), 2023.
Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi. Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024.
/be////2.../.../.// 14008
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A ADDITIONAL EXPERIMENTAL DETAILS
A.1 TRAINING BUDGET: COST BREAKDOWN
We provide further details how training data amounts, computational cost, time cost, and perfor- mance metrics reported in this paper particularly those presented in figures like Figures and 2 determined and should be interpreted.
Overall comparison (Figure T) For the baseline models included in Figure 1 the approximate training data amounts, computational costs (typically reported as GPU hours or total FLOPs), and training times are sourced from their respective technical reports or publications, leveraging the helpful summary provided in the Open-RS paper (Dang and Ngo 2025). Reasoning performance scores for all models, encompassing both baselines and our Tina models, stem from results presented Tables
6
Also, it is crucial to understand the scope of reported costs:
Epoch vs. Best Checkpoint: Costs cited for Tina and baseline models reflect the resources needed to complete full training epoch or predefined training run, not necessarily the minimal cost to reach the single best-perte perforrifm checkpoint within that run.
Training vs. Evaluation: Reported costs cover training only, omitting the computational expense required for model evaluation across benchmarks since such information is missing from several baseline models.
Particularly, the $9 USD in the abstract represents the estimated cost to train the Tina model up to its best-performing checkpoint and subsequently evaluate that specific checkpoint. For context comparing potential full training runs, the cost to train a Tina model for complete epoch is $14 USD (training only). Including evaluation costs for such a full run would increase the total to approximately $31 USD. We emphasize the $9 representing the efficient path to the best Tina model.
FLOPs estimation (Figure 2 The approximate training FLOPs shown in Figure 2 serve as a
hardware-agnostic measure of computational work. For both Tina and baseline models, these values were estimated based on reported training durations and hardware configurations sourced from technical reports or the Open-RS summary, using standard FLOPs calculation methodologies.
14

---

Published as a conference paper at ICLR 2026
A.2 TRAINING SETUP
Baselines & Datasets. Tina models inherit training recipes (e.g.. hyperparameter and reward design) and training datasets all from public reasoning models. These models are used baselines in the paper. All Tina and baseline models adopt DeepSeek-RLe Beieee----DiEe R1-Distill -Qwen- 1. as their base model checkpoint with default open-source weights.
FTILL-3-1..B-previie (RUCAIBox STILL Team 2025) is slow-thinking reasoning model developed through iterative RL on a curated dataset of 33k reasoning traces. The data originates from mathematics competitions and includes problems from MATH (Hendrycks al. 2021
Lightman et al. 2023). NuminaMathCoT (Ll et al. 2024). and AIME (1983-2023) (Art of
Problem Solving 2024). -Mma-STILL---11 Se-..BB---iiieieee uses the same dataset and reward
pipeline.
DeepScaleRR-1551 1......reeee.. (Luo et al. 2025 focuses on long- mathematical rea- soning via RL, and is trained over approximately 40k problem-ane answer pairs drawn from the AIME (Art of Problem Solving, 2024), AMC (Art of Problem Solving (023))
OMNI-MATH (Gao et al., 2024a), and STILL (RUCAIBoX STILL Team] (025) datasets. Tina-DeepScalek, 10--1.1---PPeieeee
Preview uses this dataset and mirrors the reward design.
Open-RS1/2/3 (Dang and Ngo 2025) are three models from the Open-RS project exploring reasoning performance I.5B models trained via RL. All Open-RS models are trained on small, high-quality datasets further curated from the s1 (Muennighoff et al. (2025) (i.e., Open- S1) and DeepScaleR (Luo et al.) 2025) (i.e., Apen-DeepSScleke datasets. The Tina models (Tina-Open) 2S1/2/3) replicate these setups, using identical data splits and reward scaffolding.
Hyperparameter and Reward Design. We initiated parameter selection by replicating key param- eters from OpenR1 (Hugging Face 2025) and OpenRS (Dang and Ngo 2025). For all experiments presented in this paper, deliberately adopted the default or recommended hyperparameter configu- rations provided in their works. These settings were kept largely fixed across different runs (Table 4). For ablation studies, only the specific factor under investigation (c.g.. learning rate, LoRA rank/alpha, RL algorithm) was varied (Table(5). This approach intentionally circumvents costly hyperparameter search procedures for our specific setup, ensuring negligible tuning overhead and focusing the efficacy of the core LoRA-based RL methodology.
For the main Tina results (Section 5.1), ), only reward designs were adjusted ensure fair comparison of LoRA-trained models with their full-parameter counterparts. Particularly, all the reward designs including Accuracy, Format, Length, Cosine, Tag Count, Reasoning Steps, Repetition Penalty, are defined and implemented by the OpenRI code repository
We show our default choice of hyperparameter in Table
4 and the varied hyperparameter and reward
design in Table for all the LoRA- bassed RL experiments.
15

---

Published as a conference paper at ICLR 2026
Table 4: Common hyperparameter of Tina models.
Tina-STILL-3-1, 50---evie-eeeerrr
LoRA
Tina -ina-DeeSScaleR- SB-Preview
LoRA
-ina-Open-RS(20-15
LoRA
Tina-LIMR- [2}
LoRA
Tina-OpenR1
LoRA
Tina-Opp Sina-OpenThoughts
LoRA
LoRA Modules
query, key, value, dense 32
LoRA Rank
LoRA
128
LoRA Dropout
0.05
Algorithm
GRPO AdamW
Optimizer
Optimizer Momentum
81 82 0.9, 0.999 1e-6
Learning Rate
I.R. Scheduler
Cosine with Min I.R 0.1
Warmup Ratio
Precision
BF16-mixed
Gradient Accumulation Step
Total Train Batch Size
32
Epochs
Hardware
NVIDIA L40S
Max Prompt Length
512
Max Completion Length
3584
Number of Generation
Vilm GPU Memory Utilization
0.4
Vilm Max Model Length
4608
Table 5: Varied hyperparameter and reward design of Tina models where "_" means unchanged from the common settings in Table
Alpha
Drupout
Algorithm
Leaminng
Type
Weights
Fina-STILL-3-- 5T1LL-3-111110 3-1-3---r1iiiie
Accuracy, Length
Tina DeepScaleR 1.58 Previex
Accuracy; Formar
2.1
Tina- Tina-Open-20 Open-253
Conine. Format
2.1
1raa-Open-2S3-DraPPP
DrGRPO
Cosine, Format
Tina-Open-252
Aucuracy,
Open
Accuracy:
Tina 118a-118R
Accuracy: Format
2.1
1eaa-LIMR-SS---111
5
Aecuracy. Fermat
1.1
Tina LIMB Se 7-1r
5e-7
Accuracy, Formar
2.1
Tina fina-118R-60 14-R-64-LRRA
256
Accuracy, Format
Accuracy.
rank
Accuracy:
2.1
Sina-LIKE 11M8-4-Loak
4-LoRA-ranh rank
16
Accuracy, Format
2.1
Accuracy. Cosine.
Tina-OpenRi
Format. Length.
Tag Count.
LLLL
Reasoning Steps,
Repetition Penalty
Accuncy. Cosine.
Format, Length,
Tina-Opentsoughs
Tag Count.
LLLL L11
Ressoning
Repetition Penalty
16

---

Published as a conference paper at ICLR 2026
A.3 TRAINING INFRASTRUCTURE & EVALUATION
Hardware A key element of our low-cost approach was minimizing the hardware footprint. While distributed RL training algorithms like GRPO often benefit from using three more GPUs (e.g,, dedicating one GPU to an inference engine such as vLLM for faster sample generation), we deliberately targeted minimal setup using only two NVIDIA L40S GPUs To enable this, co-located the RL training process and the vLLM on the same two GPUs by constraining vLLM's GPU memory usage. The training itself utilized data parallelism across both GPUs. While running inference and training concurrently on two GPUs might result in a longer wall-clock training time compared setup with dedicated inference GPUs, it significantly reduces the hardware requirement.
Codebase Our implementation builds upon OpenR1 fully open reproduction of DeepSeek- R1 (DeepSeek-Al) 2025) which combines the Accelerate erate (Gugger et al. (002) and Trl (von) Werra et al. 2020) libraries and the DeepSpeed ZeRO optimization (Rajbhandari et al.) 2019).
aims to transparently replicate and extend RL methods used for improving reasoning in language models, particularly focusing on aligning model behavior with reasoning-orientee objectives via verifiable reward signals. Our methodology inherits its scaffolding, training utilities, and reward interfaces.
We evaluate the reasoning capabilities of our Tina models and the baselines across diverse suite of six challenging benchmarks, primarily focused on mathematical and scientific reasoning:
AIME24/25 (Art of Problem Solving 2024) contains 30 high-school- level math problems in algebra, geometry, number theory. and combinatorics from the 2024/2025 American Invitational Mathematics Examination. Each problem demands precise multi-step reasoning.
AMC23 (Art of Problem Solving matics Competition, offering of logic and symbolic manipulation tasks.
2023) includes 40 problems from the 2023 American Mathe-
MATH500 (Hendrycks et al. 202 Lightman et al., 2023) is benchmark comprising 500 competition mathematics problem derived from various sources, covering different difficulty levels and often necessitating multi-step derivation and calculation.
GPQA Diamond (Rein al. 2024), hereafter referred as GPQA, consists of 198 PhD-Ievel science questions across biology, chemistry, and physics. Each question is multiple-choice with subtle distractors.
2022) includes 272 quantitative reasoning problems generally the undergraduate level. The questions span multiple STEM fields, including physics, biology. chemistry, and economics, often requiring mathematical modeling or calculation steps. Includes tasks such as calculating enzyme kinetics from reaction data.
Minerva (Lewkowycz
'Occasionally, NVIDIA RTX 6000 Ada GPUs were used instead, which is reflected in the system configura- tion metadata on Weights& Biases. From our practical experience, these two GPU types are similar in terms of cost and computational performance. For consistency, we report costs and compute metrics based on the L40S.
17

---

Published as a conference paper at ICLR 2026
B ADDITIONAL EMPIRICAL RESULTS
B.1 BASELINE MODELS PERFORMANCE EVALUATION
For existing SOTA reasoning models, we note that performance scores reported in the literature for relevant models often stem from evaluations using disparate frameworks (e.g.. verl (Sheng et al.
2025). lighteval (Fourrier et al., 2023). Im-eval-harres: (Gao al. 2024b)) and inconsistent inference settings (such as different generation hyperparameters or varying numbers of GPUs). These variations can influence reported metrics, creating potential inconsistencies and hindering comparisons between models.
Table 6: Baseline models evaluation (all scores zero-shot Pass@) Mean @ 1).
BASELINE MODKI.
A1ME24
AIME25
AMC23
MATH500
GPQA
MINERVA
AVG.
DeepSeek-HR R1---RR1-- DrnSSek--R--istillee---- -wnd--een.
23.33
16.67
62.50
82.60
31.82
30.15
41.18
I1-Thought-11 5--Prviie
30.00
23.33
72.50
86.80
31.90
30.88
45.90
STILL-3 3-1.BB preview
26.67
26.67
67.50
86.40
34.34
27.57
44.86
PrepScaleR-1.S-PPA
36.67
26.67
77,50
87.80
31.82
31,99
48.74
Open-RS1
26.67
20.00
72,50
83.60
35.35
28.68
44.47
Open-RS2
26.67
13.33
62,50
85.40
34.85
26.84
41.60
Open-RS3
43.33
20.00
67,50
83.00
33.84
28.68
46.06
FastCurl-1:0 1.55-Prreview
26.67
20.00
82.50
83,40
35.88
30.25
46.45
L1 -xxact
30.00
30.00
75.00
85,40
33.33
33.82
47.93
L1-Max
20.00
23.33
67.50
84.60
37.88
33.09
44.40
To mitigate these confounding factors, we performed a comprehensive re-evaluation of key baseline models using a single, consistent methodology throughout this paper. baseline evaluations reported herein utilize the 1ig lighteval framework integrated with the vLLM (Kwon et al. (202) inference engine for efficient generation. For comparability with prior work such as OpenR1, we maintained fixed hardware configuration (two L40S GPUs) and applied standardized set of vLLM inference parameters across all evaluated baseline models. All scores are zero-shot Pass@1 Mean performance. The following is the evaluation command for performance evaluation on reasoning tasks. The MODEL PATH should replaced with either the local path huggingface identifier the model to be evaluated. TASK should be of the six reasoning tasks including aime24, aime25, amc23, math_ 500, gpqa: di and minerva. PATH_ TO TOEE R1 _1__EVALAAEE TTAIPA should be the path the custom evaluate script provided by OpenR1
to combine lighteval and vLLM
MODEL
pretrained-SMODLLPPAN
dtype=bfloat.6,,
data_parallel_sizzez.
max.mooee__length=2276..
gpl.memory.uttll22att:::::00
generaation_paraamttrss--t- secaa_ne__ttoloe htw.e//emwweeeeeeooeeeeeeeeeee 32768, temperature: 0. 6,6, 0.p._p:0..95.
lighteval vllm SMODEL_ARGS "custom/ STASK 0 0"
--custom-taste tasks $PATH_TO_OPENT RELLLLAEEAAAEEEEEE SCRIPT
--use-chat-templaat
18

---

Published as a conference paper at ICLR 2026
B.2 ROBUST EVALUATION EXPERIMENT
We also provide a robust evaluation experiment on the reasoning benchmarks. Specifically, we average the scores of AIME24/25 and AMC23 over 10 independent seeds (i.e., they are zero-shot Pass @ Mean 10 performance) and also add another benchmark, Olympiadbench (He al. 2024). We run this robust experiment on all baseline models and our Tina-DeepScalekt 5B Preview for demonstration. As shown in Tables
with different random
the performance perturbation on average is around 2% and does not affect our conclusion in the paper.
Table 7: Robust baseline models evaluation.
BASBLINE MODEL
A1ME24
AIME25
AMC23
MATH500
GPQA
MINERYA
OLYMPIAD
AYG.
NON-ROBUST AvG.
DeepSook wew.rrlleddddeeee 1.
20.00
21.67
31,50
X2.60
31.82
30.35
53.63
45.77
41.18
TE- Thought-1 Beriggh----S-Peeeeee P.-Prre
32.33
26.67
73.25
86.80
31,90
30.58
54.78
48.09
45.00
STILL-3- 1011..--preet Si-prrview
31.00
27.00
74,25
86.40
34.34
27.57
54.67
47.89
44.86
37.00
25.67
68.25
K7.80
31.82
31.99
56,00
48.36
4K.74
28.67
26.00
75,00
83.60
35.35
28.68
53.33
47.23
64.47
31.35
20.33
70,00
85.40
34.85
26.34
53.90
46.10
41.60
Open:
25.67
83.00
46.06
FastCurl-1.3 P--eeiee Previex
11 Exact
21.00
46.23
111-Kax
21.00
Table 8: Robust evaluation of -ina--eepScaleR-1... 5BPreiiii.........
CHECKPOINT
ReOCH)
AIME24
AIME25
AMC13
MAT11500
MENCaYA
NON-ROBUST Ave.
300
30,00
24.00
74.50
$2,40
39.39
31.25
53.93
47.93
45.65
1000
37.00
24.67
71.50
86.20
37.58
25.68
52.00
48.27
43.38
1500
30.33
23.67
71.25
$4.30
32,83
29.41
50.81
46.23
46.17
24.00
57.00
55.00
18.01
58.50
3500
23,67
22.00
60.75
90,40
31.52
24.36
44.59
41.07
40,94
4000
20,00
23.67
1733
67.38
41.41
27.94
87.10
4390
63.56
2500
23.33
27.00
19.07
06.00
34.85
26.47
29.04
43.40
42.99
19

---

Published as a conference paper at ICLR 2026
B.3 ALL TINA MODELS PERFORMANCE EVALUATION
We present all Tina models' detailed evaluation performance during post-training across six reasoning tasks including AIME24/25, AMC23, MATH500, GPQA and Minerva.
Table 9: Evaluation of -ona-STILLL3--11 50------peee
CHECKPOINT STEPS (3740 STEPS EPOCH)
ATME24
AIME25
AMC23
MATH150
GPQA
MINRRVA
AVG.
500
30.00
13.33
75.00
83.60
35.86
32.35
45.02
36.67
20.00
44.46
1500
26.67
20.00
70.00
83,80
37.37
26.84
44.11
2000
36.67
30.00
77.50
84.60
33.33
26.84
48.16
30,00
35.35
46.54
3000
30.00
20.00
67.50
82.60
30.81
25.74
42.78
3500
30.00
26.67
67.50
82.20
32.32
26.10
44.13
Table 10: Evaluation of Tina Tina-DeepScaleR- 5B- SB-Preview.
CHECKPOINT STEPS (5039 STEPS EPOCH)
AIME24
AIME25
AMC23
MATHS50
GPQA
MINERVA
AYG.
500
30.00
23.33
67,50
82,40
39.39
31.25
45.65
67.50
1500
30.00
20.00
80.00
84.80
32.83
29.41
46. 17
2000
20.00
26.67
57.50
80,60
29.29
24.26
39.72
2500
13.33
16.67
52.50
31.31
18.01
34.47
3000
26.67
16.67
57.50
78.60
28.79
23.16
38.57
23.33
23.33
62.50
80.40
31.82
24.26
40.94
4000
20.00
20.00
70.00
82.00
41.41
27.94
43.56
4500
23.33
20.00
72.50
80.80
34.85
26.47
42.99
44.20
Table 11: Evaluation of Tina-II-Thoughe -..igghhtt1.1B--1111 SB-Previee..
CHECKPOINT STEPS (6660 STEPS PER EPOCH)
AIME24
AIME25
AMC23
MATH150
GPQA
MINERVA
AVG.
33.33
23.33
77.50
85.20
31.31
27.57
46.04
1000
23.33
20.00
80.00
83.20
35.35
29.04
45.15
1500
40.00
20.00
80.00
86.00
33.84
26.84
47.78
26.67
20.00
46.47
2500
30.00
23.33
75.00
85.00
40.40
26.47
46.70
3000
26.67
20.00
67.50
86.80
30.30
26.10
42.90
3500
33.33
16.67
67.50
84.00
40.91
30.88
45.55
4000
30.00
10.00
75.00
84.60
36.87
27.21
43.95
4500
26.67
16.67
72.50
85.40
33.84
25.37
43.41
5000
30.00
23.33
77.50
84.60
37.37
31.62
47.40
20

---

Published as a conference paper at ICLR 2026
Table 12: Evaluation Fina--Oenn-RS RS3.
CHECKPOINT STEPS (875 STEPS PER EroCH)
AIME24
AIME25
AMC23
MATH500
GPQA
MINERVA
AVG.
50
26.67
23.33
75.00
84.20
37.37
29.04
45.94
100
30.00
30,00
65.00
83.00
37.37
29.78
45.86
200
20,00
26.67
70.00
83.80
33.33
27.94
43.62
250
36.67
20.00
65.00
84.60
38.38
28.31
45.49
70.00
30.15
46.03
350
40.00
16.67
77.50
84,40
39,90
27.94
47.74
400
30,00
16.67
70.00
82.80
35.86
31.25
44.43
47.58
500
36.67
23.33
82.50
85,20
37.37
31.62
49.45
550
26.67
16.67
80.00
86.00
35.35
29.78
45.75
46.49
650
20.00
23.33
80.00
85,00
33.33
27.94
44.93
700
33.33
13.33
72.50
85.00
40.40
31,99
46.09
75.00
45.69
800
30.00
23.33
65.00
84,20
38.38
29.04
44.99
850
26.67
26.67
75.00
83. 33.80
31.82
27.94
45.32
Table 13: Evaluation of Fiin--Oee--2 RS2.
CHECKPOINT STEPS (875 STEPS PER EPOCH)
AIME24
AIME25
AMC23
MATH500
GPQA
MINERYA
AvG.
33.33
23.33
77.50
84.20
38.89
29.04
47.72
100
36.67
23.33
72.50
84.20
31.31
28.68
46.12
150
40.00
23.33
72.50
85.80
30.30
30.51
47.07
200
26.67
23.33
70.00
83.80
39.39
29.41
45.43
250
46.67
13.33
72.50
82.60
31.82
30.51
46.24
30.00
26.67
75.00
84.00
29.04
46.34
350
33.33
20.00
75.00
84.80
37.37
28.68
46.53
400
26.67
16.67
70.00
83.20
37.37
27.57
43.58
450
43.33
26.67
77.50
87.00
36.36
32.72
50.60
500
20,00
23.33
67.50
84.20
33.84
29.41
43.05
550
40.00
23.33
72.50
83.60
40.91
30.88
48.54
72.50
84.20
30.88
650
33.33
23.33
$7.50
83.80
34.85
30.51
43.89
700
23.33
26.67
70.00
82.40
33.33
28.68
44.07
750
30.00
23.33
72.50
84.20
38.89
29.04
46.33
800
30.00
26.67
75.00
84.40
32.32
29.41
46.30
850
26.67
23.33
70.00
83.80
35.88
28.68
44.72
Table 14: Evaluation of Fiia-OOeenRRS
CHECKPOINT STEPS (2327 STEPS PER EPOCH)
AIME24
AIME25
AMC23
MATH500
GPQA
MINERVA
AvG.
33.33
20.00
75.00
83,80
31.82
29.78
45.62
600
30.00
30.00
77.50
84,20
34.34
31.62
47.94
800
43.33
20.00
80.00
35.35
28.68
48.56
1000
33.33
20.00
82.50
84.40
35.86
29.78
47.64
1200
36.67
20.00
67,50
84.40
37.88
30.15
46.10
43.75
1600
23.33
13.33
65.00
83,40
35.86
26.84
41.29
1800
26.67
20.00
75.00
84,20
34.34
27.57
44.63
2000
30.00
26.67
72.50
83.00
36.36
27.94
46.08
2200
30.00
23.33
70.00
81.40
30.81
26.47
43.67
2400
30.00
23.33
67.50
81.80
30.30
27.57
43.42
21

---

Published as a conference paper at ICLR 2026
Table 15: Evaluation of Tina-LIMR.
CHECKPOINT STEPS (174 STEPS PER EPOCH)
AIME24
AIME25
AMC23
MATH500
GPQA
MINERVA
AVG.
26.67
85.40
30.51
44.06
100
46.67
20.00
75.00
83.80
34,85
30.51
48.47
150
26.67
20.00
72.50
84.00
37.37
30.15
45.
44.99
Table 16: Evaluation of Mina-OpenRI
CHECKPOINT STEPS (11716 STEPS FER EPOCH)
AIME24
A1ME25
AMC23
MATH500
GPQA
MINERYA
AVG.
30.00
85.20
33.84
30.15
1000
30.00
23.33
72.50
85.60
33.84
26.67
45.32
1500
36.67
26.67
75.00
86,80
39,90
30.51
49.26
2000
26.67
23.33
67.50
83.20
29.80
31,62
43.69
2500
30.00
23.33
72.50
83.80
33.84
26 84
45.05
3000
20.00
34.34
44.13
3500
36.67
23.33
67.50
83.60
31,31
25.74
44.69
Table 17: Evaluation of Tina Tina-Open Opena-oOeThhhhss...
CHECKPOINT STEPS (8259 EPOCH)
A1ME24
AIME25
AMC23
MATH500
GPQA
MINERVA
AVG.
500
33.30
16.67
77.50
84,20
35.86
30.15
46.28
1000
33.33
23.33
80.00
85,20
24.75
32.72
46.56
1500
30.00
23.33
70.00
86.00
37.88
29.04
46.04
2000
30.00
23.33
70.00
84,20
33.33
28.31
44.86
36.67
3000
26.67
23.33
75.00
83.60
34.34
32.72
45.94
3500
20.00
16.67
60.00
84.20
32.32
26. 10
39.88
46.51
4500
30.00
20.00
65.00
85.00
33.84
26. 84
43.45
5000
20.00
33,33
65.00
84.80
40.91
30.88
45.82
Table 18: Evaluation of Tina-Open-RS---rroee
CHECKPOINT STEPS (875 STEPS EPOCH)
A1ME24
AIME25
AMC23
MATH500
GPQA
MINERYA
AVG.
33.33
16.67
75.00
83.80
37.37
26.84
45.50
83.20
150
43.33
23,33
80,00
85,00
35.35
30.15
49.53
200
30.00
23.33
70.00
84.00
39.90
28.68
45.99
45.80
300
36.67
16.67
67.50
84.40
37.88
29.78
45.48
350
26.67
30.00
75.00
84.00
37.88
29.78
47.22
36.67
23.33
72.50
84.40
32.83
27.57
46.22
450
36.67
16.67
72.50
85.60
29.29
27.57
44,72
500
30.00
20.00
72.50
85.60
37.37
29.41
45.81
84.80
47.35
600
33,33
26.67
72.50
83.80
30.30
28.31
45.82
650
26.67
20.00
77.50
82.40
37.88
27.94
45.40
700
36.67
20.00
80.00
83.80
35.35
31.25
47.85
750
30.00
26.67
75.00
84,20
38.89
27.57
47.06
800
20.00
30.00
75.00
82.40
35.86
28.31
45.26
72.50
85.40
36.36
30.15
44.62
22

---

Published as a conference paper at ICLR 2026
Table 19: Evaluation of Tina-Open-RS3-
RS3 Soppen--s--oong-comoe compleeottoooiiionn
CHECKPOINT STEPS (875 STEPS PER EPOCH)
AIME24
AIME25
AMC23
MATH500
GPQA
MINERVA
AVG.
87.00
45.16
100
26,67
26.67
77.50
85.80
31,31
27.21
45.86
150
20.00
30.00
67.50
85.00
34.34
30.15
44,50
45.61
250
26.67
23.33
75.00
83.00
36.87
33.09
46.33
300
33.33
16.67
70.00
86.40
32.32
27.21
44.32
350
43.33
23.33
82.50
83.20
30.51
50.63
400
26.67
23,33
72.50
84,40
35.35
27.57
44,97
450
33.33
16.67
67.50
83.40
33.84
34.19
44.82
84.60
45.43
550
33.33
23.33
70.00
84,20
38.38
29.78
46,50
600
33.33
23.33
67.50
84.80
33.33
27.57
44,98
650
26.67
20.00
75.00
84.20
36.36
30.51
45.46
700
30.00
33,33
70.00
84,20
37.37
28.68
47.26
750
30.00
30.00
72.50
84.60
38.38
27.94
47.24
77.50
86.60
46.32
850
33.33
26.67
72.50
84.00
30.30
31.62
46.40
Table 20: Evaluation of Tina Tina-Open- 0ee---penn-SS- -w3--fomattfoolllyyy
CHECKPOINT STEPS (875 STEPS EPOCH)
AIME24
AIME25
AMC23
MATH500
GPQA
MINERVA
AVG.
50
40,00
20.00
77.50
85.80
32.32
29.04
47.44
100
30.00
70.00
84.20
31,82
27.21
44.98
1.50
30.00
23.33
72.50
84.60
35.35
29.41
45.87
200
30.00
16.67
75.00
84.60
31.82
33.46
45.26
70.00
300
33.33
23.33
70.00
84.80
27.27
29.04
44.63
350
36.67
16.67
67.50
85.60
38.38
27.21
45.34
400
20.00
$7.50
$5.80
33.33
27.21
43.97
450
30.00
26.67
77.50
84.20
29.29
29.41
46.18
500
30.00
16.67
67.50
83.60
32.32
30.88
43,50
600
26.67
20.00
75.00
86.00
34.85
28.31
45.14
650
40.00
23.33
75.00
85.40
36.36
27.94
48.01
48.02
750
33.33
23.33
77.50
86.20
30.81
30.51
46.95
800
30.00
30.00
75.00
84.60
36
33.09
48. 18
50.56
Table 21: Evaluation of Tina-LIMR-Se--6. 5-- with learning rate 5e-6.
CHECKPOINT STEPS (174 STEPS PER EPOCH)
AIME24
AIME25
AMC23
MATH500
GPQA
MINERVA
AVG.
85.40
44.66
100
46.67
20.00
75.00
83,80
34.85
30.51
48.47
150
26.67
20.00
72.50
84.00
37.37
30.15
45.12
200
33.33
30.00
62.50
83.40
30,88
44,99
23

---

Published as a conference paper at ICLR 2026
Table 22: Evaluation
5-- with learning rate 5e-7.
EPOCH)
MATH500
GPQA
AVG.
50
40,00
13.33
72.50
83,00
34.34
29.04
45.37
100
43.33
16.67
77.50
84.60
34.85
30.51
47,91
86.20
46.65
200
33.33
13.33
70.00
83.20
29.29
31.25
43.40
Table 23: Evaluation of Tina-LLM--G4-LoR--rra with LoRA rank 64 and alpha 512.
CHECKPOINT STEPS (174 STEPS PER EPOCH)
AIME24
AIME25
AMC23
MATH500
GPQA
MINERVA
AVG.
84.20
31,62
46.95
100
30.00
23.33
72.50
84,60
32.32
29.78
45.42
150
36.67
20.00
70.00
83.40
31.82
30.88
45.46
Table 24: Evaluation of Tina Liiha-LLMR--BE
with LoRA rank 16 and alpha 64.
CHECKPOINT STEPS (174 STEPS EPOCH)
ATME24
AIME25
AMC23
MATH500
GPQA
MINERVA
AVG.
50
33.33
23.33
62.50
84.20
38,89
31.25
45.58
100
33,33
70.00
83.20
35.35
28.31
48.92
150
26.67
16.67
72.50
83.40
35.35
29.04
43.94
200
36.67
20.00
75.00
83.00
39.39
30.51
47.43
Table 25: Evaluation of -LLAA-rark with LoRA rank 8 and alpha 32.
CHECKPOINT STEPS (174 STEPS PER ErOCH)
AIME24
AIME25
AMC23
MATH500
GPQA
MINERYA
AVG.
30.00
26.67
82.50
83.80
33.84
30.51
47.89
100
26.67
16.67
72.50
84.00
36.87
29.78
44.42
150
53.33
20.00
60.00
83.20
37.37
30,88
47.46
200
23.33
20.00
72.50
85.40
32.83
28.68
43.86
Table 26: Evaluation of Tina-LIMR-4-LORAA with LoRA rank 4 and alpha 16.
CHECKPOINT STEPS (174 STEPS PER EPOCH)
AIME24
AIME25
AMC23
MATH500
GPQA
MINERVA
AVG.
30.00
23.33
65.00
85.00
35.35
29.78
44.74
45.42
150
36.67
20.00
85.00
83.80
31.82
29.0
47.72
200
33.33
23.33
77.50
85.40
35.86
28.31
47.29
24

---

Published as a conference paper at ICLR 2026
C ALL TINA MODELS TRAINING PHASE TRANSITION
We present all Tina models' training phase transitions along the training dynamics. The raw data is from the Weights Biases training logs and smoothed via exponential moving average (EMA) with factor 0.1. Specifically, we observe clear transitions in Tina-DeepScaleR- 5B-Preview, Tina-STILL-3-1- 5B-preview, Tina-Open-Rso Opna-Ope Tina-Open-RS3- as shown in Figures 6
-ina-Open-RS1,
cong-complette completion enion, and. For -na-Ope and Tina- Thooghhss (Figures [0 and [1]. the observation is similar, except the best-performing checkpoint is achieved after the training turning point, rather than before.
Tina-Open-RSet
However, we do not observe such transition in all Tina variants on the LIMR dataset, as shown in Figures which are too brief to extract meaningful information. Also, we do not observe the transition in Tina-Open- -aeen-ppn-SS ------S-------- format-on.. Figure 15 due to the absence of accuracy rewards.
13 and 14 possibly because its small data size leads to training periods
25

---

Published as a conference paper at ICLR 2026
Tina- Diae-DeepScaeR--55B-PPPP-
Data
Raw
Smoothed IEMA a=0.11
0.75
Smoothed am0.1l
Accuracy
0.6
Accuracy
0.4
0.50
0.2
0.25
0.00
0
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Step
Step
Dina-DeepScaleR111BBB
Tina:l/ThhoghhttL.---eeeeee
1.0
1.0
0.8
0.8
Format
Format
0.6
0.6
Rew
Raw Date
0.4
g=0.1)
0.4
g=0.1)
Best- Parforiing Checkpann aint
Best Partorning Checkpoint
0.2
Training Tumnin
0.2
Training Tuming
1000
2000
3000
4000
5000
1000
2000
3000
4000 5000
Step
Step
Mana-DeepScalee-15---PPPP
Tina-l-Thhuggh--15BB-Peee
Completion Length
Completion Length
Raw Data
Raw Data
3000
3000
Best Dent-Performing Checkpsink Paint
Best- Performinn Checkpoint
Training Tuming
2000
2000
1000
1000
0
1000
2000
3000
4000
5000
0
1000
2000
3000
4000 5000
Step
Step
Figure
Transition
Tina-DeepScaleR-11 5B -.r-rriiiew
and
Tina-II- -II---Tght--t-- -1.gghtt1.555 1B5 5---reviie
26

---

Published as a conference paper at ICLR 2026
Tine-STILL-3-1.5B-preeee
Tina Tina-Open-RS RS1
Smoothed (EMA a=0.1)
0.4
Smoothed (EMA a=C. 1.11
Accuracy
Accuracy
0.4
0.2
0.2
Mo k
0.0
0.0
1000
2000
3000
500
1000
2000
2500
Step
Step
Tina-STILL-3-15BBPPPP
Oina-Open-RS1
-0.05
1.0
Raw Dala
-0.10
Best-Perfoming Chackpaint
0.8
Training Turning Point
Format
Len
00.5
0.6
0.4
-0.20
Smoothed IEMA a=0
Dest-Perfoming Checkpoint
0.2
0.25
Trining Tumnnn Point
1000
2000
3000
500
1000
1500
2000
2500
Step
Step
Tina-STILL-3-1.5B-preee
Tina Open-
Completion Length
Completion Length
3000
MeMe ww
3000
M
Rew Data Smoothed
2000
Raw Data
Smoothed
Best-Per Perforrrnn Checkpoint
best-Performing Checkpsint
Training Tuming Point
1000
Tuming
1000
2000
3000
500
1000
1500
2000
2500
Step
Step
Figure 7: Transition in Tina-ST STILL-3-1. 5B 38-previen and Tina-Open-RSi
27

---

Published as a conference paper at ICLR 2026
Tina- Tina-Open-RST
Tina Tina-Open-RR3
0.6
Smoothed (EMA a=0.1)
Cosine Scaled
0.2
Smoothed (EMA a=0.1]
Data
Accuracy
0.4
0.0
WW m
0.2
-.2
0.0
200
600
750
1000
1250
Step
Step
Tina- Opaa0Ope--RS2
Tina-Open-RS3
1.00
Rew Data
Raw Data
0.8
Smoothed (EMA
Format
0.75
Best-Performance Checkpoint
Point
Format
0.6
Best Performance Chackpoint
Training Turning Point
0.50
0.4
0.25
0.2
0.00
0.0
200
400
600
800
200
400
600
800
1000
1200
Step
Step
Tina Open-RS2
Tina-Open-RS3
Completion Length
Completion Length
3000
3000
2000
Raw Deta
2000
Raw Data
Smoothed g-0.1)
Smoothed (FMA
Best-Performance Checkpoint
Pest-Performance Checkpoint
Turning Paint
Tuming
1000
200
400
600
800
250
500
750
1000
1250
Step
Step
Figure 8: Transition in Lina-Open-RS and Tina-Opent OpSn-RS3.
28

---

Published as a conference paper at ICLR 2026
Jin--ppen-RS3-DGGRPP
Data
Cosine Scaled
0.2
Smoothed (EMA a=0.1)
Cosine Scaled
0.2
Smoothed (EMA a=0.1]
0.0
0.0
MM WMMh
-0.2
0.2
200
600
200
400 Step
Step
Tina-Open-RS3-DYGRPO
Rew Data
Raw Data
0.75
Smoithhd
0.75
am0:l]
Format
Best:Performance Checkpoint
Format
Best- Parformance Checkpoint
Point
Training Point
0.50
0.50
0.25
0.25
0.00
0.00
200
400
600
800
200
400
600
Step
Step
Tina-Open-RS3-DrGRPP
Tinn--Open-RS3-oon--oolpeeeeee
Completion Length
Completion Length
3500
4000
3000
My
2500
3000
Raw Deta
Raw Data
2000
Smoothed g-0.1)
g-0.1)
Best-Performance Checkpoint
2000
Pert-Performance Checkpoint
1500
Training Turning Paint
Training Tuming Paint
200
400
600
800
200
400
600
Step
Step
Figure 9: Transition in Tina Sanaa-ppen-aeee - and -ina-Open-ao RS3 -onpee--S---oo---ooo long- complcon
29

---

Published as a conference paper at ICLR 2026
Tina-OpenR1
Tina-OpenRI
0.6
Raw Data Smoothed
Raw Data
Cosine Scaled
Smoothed (EMA
0.2
Accuracy
0.4
0.0
0.2
0.22
0.0
-0.4
0
1000
2000
3000
0
1000
2000
3000
Step
Step
Tina-OpenR1
Tina-OpenRI
1.00
Raw
0.0
Smoothed [EMA 0.1)
Smoothed [EMA a=0.11
Format
0.75
Best-Performing Checkpoint
Pert-PPerfoiing Checkpoint
Training "umming Point
Training Tuming
0.50
Len
0001
0.25
-0.2
0.00
1000
2000
3000
0
1000
2000
3000
Step
Step
Tina-OpenR1
Tina-OpenR1
Reasoning Steps
1.0
Repetition Penalty
-0.05
0.8
-0.10
0.6
-0.15
Minu
Raw
Raw
Smoothed (EMA am0.1) 1]
0.20
Sioot hed a=0 .]
0.4
Best-Purfor Perf-rrrrorinn Checkpoint
Best- BesttPPrferming Checkpoint Triining Tuming
Training Tuming Point
-0.25
1000
2000
3000
1000
2000
3000
Step
Step
Tina-OpenR1
Tina-OpenR1
1.0
Completion Length
Raw Data
2000
Smoothed (EMA
Tag Count
0.8
Best Perforrrro Chieckpoite
Training Turning
1750
0.6
1500
Raw Data
Smooth ed (EMA a=0.1)
0.4
1250
Parforming Chackpaint
Training Tuming Point
0
1000
2000
3000
0
1000
2000
3000
Step
Step
Figure 10: Transition in Tina OpenR1.
30

---

Published as a conference paper at ICLR 2026
Tina-OpenThoughts
Tina Tina-OpenThoughts
0.4
0.6
Raw Data Smoothed
Cosine Scaled
Raw Data
Smoothed (EMA
Accuracy
0.2
0.4
0.0
0.2
0.2
0.0
0.4
0
1000
2000
3000
4000
5000
0
1000
2000
3000
4000
5000
Step
Step
Tinh-OpenThoughts
Tina-OpenThought
1.00
Raw
0.0
Data
Smoothed (EMA az 0.1)
Smoothed (EMA a=0.21
Format
0.75
Best-Performing Checkpoint
Pest-Performing Checkpoint
Training "uuming Point
Trinning Tuming
Len
0.50
-0.1
0.25
0.2
0.00
1000
2000
3000
4000
5000
0
1000
2000
3000
4000
5000
Step
Step
Tina Thouughs
Tina-OpenThoughts
Reasoning Steps
1.0
Repetition Penalty
0.8
-0.1
0.6
Raw
0.2
Raw
Smoothed (EMA a=0 1)
Smoothed IEMA a=0 1I
Best Pestt-Prffomin Checkp.innt
Best-Perfoming Checkpaint
0.4
Training Tuming Point
Training Turning Point
1000
2000 3000
4000
5000
1000
2000
3000
4000
5000
Step
Step
Tina-OpenThoughts
Tina-OpenThoughts
1.0
Raw Data
Completion Length
2000
Smoothed o=0.1]
Tag Count
0.8
Best Persorrrroomn Ch eckpopott
1750
Training Turning
0.6
1500
Raw Data Smoothed (EMA am0.1)
0.4
1250
Parfommrg Chackpaint
Traning Tuming Paint
0
1000
2000
3000
4000
5000
0
1000
2000
3000 4000 5000
Step
Step
Figure 1: Transition in Tina Opha-OhenThoushts..
31

---

Published as a conference paper at ICLR 2026
Tina-LIMR
Tin-LIMM--GLLORArrnn
Tina-LIMR rakk
0.6
Accuracy
Accuracy
0.6
0.4
0.4
0.2
Smoetted JAV swa
0.2
Tine-LMRR-GeLoRkrank
Tine-LMMR-10-LoRA-rre
1.0
0.8
Format,
Format
0.8
0.6
0.41
Smeethed a-0
0.41
Smeothes ICMA
Smoothed
200
200
300
400
100
200
300
50
100
150
200
250
300
Stap
Stap
Step
Tína-LIMR
Tina-LIMR-66-LoRAA-ee
Time-LIMR-1B-LORA--
Complction Longth
3000
Completion Length
3000
2500
2500
2500
Completion
2000
1500
1500
1500
100
200
300
400
100
200
300
100
200
300
Step
Stap
Step
Figure 12: Transition in TiMa-LIMR, Tina-LIMR LIM---I-------- -4----4-----aa rank and 16-L--------------
32

---

Published as a conference paper at ICLR 2026
Tina-LIMR-8LLoRA-rrnl
Tine-LiMR-4LLoRA-renk
0.6
Smoothed (EMA am0.1)
Smoothed (EMA am 0.13
Accuracy
Accuracy
0.4
0.4
Ww
0.2
0.2
0
50
100
150
200
250
300
50
100
150
200
250
300
Step
Step
Tina-LMMR-8-LoRA-ran
Tina-LIMRRA-LoRA-rrad
1.0
1.0
Format
0.8
Format
0.8
0.6
0.6
Row Data
Raw Data
IEMA o=0..)
Smoothed
50
150
200
250
300
100
150
200
250
Step
Step
Tina-LIMR-S-LoRA-ranl
Tina-LIMR-4LLoRA-rank
Completion Length
Completion Length
3000
2500
2500
2000
2000
1500
1500
Smoothed (EMA a=0.11)
Smoothed (EMA a=0.1)
0
100
200
300
0
100
200
300
Step
Step
Figure 13: Transition in Tina-LIMR-8 --L-oL-LA-aaa and Tina LIMR- --
33

---

Published as a conference paper at ICLR 2026
Tina-LiMR-Se--llr
Tina-LiMR-5e-74
0.8
0.6
Accuracy
Accuracy
0.6
0.4
0.4
0.2
Raw Data Smoothed a=0.31
Raw Data
Smoothed (EMA a=0.1)
0.0
100
200
100
200
Step
Step
Tina-LIMR-Se-6-Ir
Tina-LIMR-Se-7-Ir
1.0
1.0
Format
Format
0.8
0.6
0.6
0.4
Smoothed JEMA am0.11
0.4
Smoothee IEMA a=0.11
0
100
200
300
0
50
100
150
200
250
Step
Step
Tina-LIMR-5e-6-Ir
Tina-LIMR-5e:
Completion Length
Completion Length
3000
3000
Data
Smoothed (EMA e=0.
2500
2500
2000
2000
1500
Raw Data
Smoothed a-0.1)
1500
100
200
300
100
200
Step
Step
Figure 14: Transition in Tina-LIMR- L-----IRR--- 5e----e---11 and
34

---

Published as a conference paper at ICLR 2026
Caae-Open-RS-formmtt
Tin--Open-SS-fommatonl
0.8
Data
Smoothed JEKA a-C.11
-0.05
0.6
Format
-0.10
0.4
Len
Wv ~
4//
00.55
0.2
-0.20
Raw Data
0.0
Smoothed (EMA a-0.1)
400
800
200
400
Step
Step
Rina-Open-RS3-fone RS--fomat---ooly
Thna-Open-ASS-ffrmattolll
Rea Reasoning Steps
Repetition Penalty
-0.15
Rew Data
0.9
Smoothed
0.8
W W/
-0.20
0.7
-0.25
Smoothed IEMA a=0.13
0.5
0
200
400
600
800
200
400
600
800
Step
Step
Tina Tina-Open-RS33 -ormaattoo-
Tina-Oper-S33-fomattolly
0.9
Data
Completion Length
3500
Smoothed (EMA e=0
Tag Count
0.8
3000
0.7
2500
0.6
2000
Raw Data
Smoothed (EMA
0.5
200
400
600
800
200
400
600
800
Step
Step
Figure 15: Transition in Tina Tina-Open- totee--R--format-oom
35


Reviews:
[{"review_id": "bOv6HZkqAW", "reviewer": "Reviewer_au9R", "summary": "This paper introduces TINA, demonstrating that applying LoRA during reinforcement learning to a tiny 1.5B model can achieve SOTA-competitive reasoning performance at a tiny fraction of the computational cost.", "questions": "The first citation in this paper appears to be sensitive. It indirectly points to a blog post that is directly related to this work. I am not sure if this potentially violates the double-blind review process.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The authors' research finding that LoRA on small language models can achieve performance on par with full-parameter RL training is highly valuable. I believe this will benefit researchers who are \"compute-scarce\" and those who are new to the field, allowing more people to experiment with and master RL scaling.\n\n- The hypothesis proposed by the authors makes sense. It holds similar viewpoints with the concurrent blog-\"LoRA Without Regret\", suggesting that the amount of information gained during the RL process is less than that in SFT. Therefore, a parameter-efficient method like LoRA can be effectively used to \"organize\" the base model's existing knowledge into the \"structured format\" preferred by a RL reward scalar.", "weaknesses": "Please correct me if my understanding is wrong or biased:-)\n\nOverall, TIna is a very \"timely\" paper. Its primary contribution does not lie in proposing a new methodology, contributing new data or benchmarks, or releasing a powerful SOTA model. Instead, its main contribution comes from \"updating the reader's cognition.\" The authors are loudly proclaiming to the community that \"LoRA + small language models can still achieve significant reasoning gains via RL,\" essentially encouraging everyone to \"Go try LoRA RL training now!\" If I had reviewed this paper earlier this year for a conference like COLM or NeurIPS, I would have given it a clear accept. However, at this current point in time, I feel compelled to raise two additional questions:\n\n- Could the authors provide deeper insights? For example, whether other PEFT methods (e.g., DoRA), or different model families (e.g., Llama, MiMO, or even VLMs), or non-mathematical reasoning tasks (e.g., puzzles, games) are applicable to TINA? Or perhaps some deeper theoretical insights?\n\n- LoRA RL Training is already supported by several existing RL frameworks, such as verl, and they are often scalable, supporting training on larger models and more tasks. Given this, what do the authors believe is the unique, standalone value of the open-sourced code and model weights provided by TINA?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o2oNCIXfnK", "reviewer": "Reviewer_XvES", "summary": "This paper presents a study on using LoRA and RL to improve the reasoning capability of small language models. Through extensive experiments, the authors show us a surprising conclusion that LoRA, compared to full-parameter tuning, is more efficient and effective when using RL to improve the reasoning capability of small language models. To explain this phenomenon, the authors propose the Rapid Reasoning Format Adaptation Hypothesis, suggesting that LoRA enables models to quickly adapt to reasoning formats rewarded by RL while largely preserving the base model’s knowledge. They conduct rigorous ablation studies and detailed training-dynamics analyses to support this claim. Overall, the paper is well-organized, technically sound, and reproducible, offering clear insights into the mechanisms of parameter-efficient RL and setting a valuable precedent for future research.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well written and clearly presents the necessary background, experimental setup, and extensive ablation studies that support the authors’ claims.\n\n2. The experimental setup is clean and effective. The paper adopts a minimalist yet powerful combination of LoRA and GRPO reinforcement learning, eliminating noise from architectural or dataset improvements, which is essential and helps focus on the most important research question the paper aims to answer.\n\n3. The authors propose the Rapid Reasoning Format Adaptation Hypothesis to explain why they observe that RL with LoRA is more effective for reasoning, and rigorous ablation studies support this. This is very beneficial for the entire research community.", "weaknesses": "1. Although the paper evaluates Tina across multiple training recipes (STILL-3-1.5B, DeepScaleR-1.5B, and Open-RS), all experiments ultimately rely on the same underlying base model, DeepSeek-R1-Distill-Qwen-1.5B. This raises concerns about the generality of the claimed findings. It remains unclear whether the observed “rapid reasoning adaptation” and cost-efficiency would still hold for other base models that are not distilled from DeepSeek-R1 or outside the Qwen2.5 family. For instance, would similar trends be observed for models such as Qwen3-1.7B or other architectures of comparable scale? A broader evaluation would strengthen the paper’s generality.\n\n2. The authors attribute LoRA’s effectiveness to its ability to rapidly adapt the reasoning format under RL while preserving the base model’s knowledge. However, using a sufficiently small learning rate in full-parameter fine-tuning could potentially yield similar effects: slow, stable updates that preserve existing knowledge while adapting reasoning behavior. It would be valuable to analyze whether such small-learning-rate full-parameter fine-tuning exhibits the same adaptation dynamics or efficiency gains as LoRA, and to clarify the conceptual and empirical differences between the two.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Na1XUOdrB", "reviewer": "Reviewer_N2DY", "summary": "This paper proposes to use Low-rank adaptation(LORA) finetuning with a  group relative policy optimization (GRPO) approach in order achieve reasoning capacities with tiny models.", "questions": "The paper goes into large efforts to show what can be achieved with small training costs. \nAn alternative to using tiny models could be to uses small models with quantization; How would these compare?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Pursuits an interesting, potentially really important idea\n* Easy to implement\n* Large amount of experiments", "weaknesses": "The Idea is very straightforward (could also be seen as a plus!)\n\nThe paper is missing to report baseline benchmark results of small (but still larger models) for comparison (e.g., you could use something like a deepseek R1 model with ~7B parameters), e.g., in Table 2 and 3.; In the current state it is very hard to judge if the results could lead to any reasonable use case.\n\nMeasuring the training costs in $ is ephemeral and can change very quickly even without hardware changes (for economical reasons, exchange rates, etc...); For a scientific paper, I would recommend reporting FLOPs or at least GPU hours instead or on top.\nReporting the cost breakdown for different ablations and the evaluations to me seems not to be relevant for the main paper; Overall the main paper seems to be a bit stretched out to fill the pages and could be written way more concisely.\n\nWhat am I missing is an in-depth discussion of the advantages and disadvantages of using very small models for reasoning and their place in an agentic AI. While I absolutely see reasons to use small models, reasoning appears to be *the* task, in which at least decently sized models seem to be a good investment; \n\nThe main paper is missing details about the actual training setup (\"Tina models build upon the open-source training recipes and datasets\nof STILL-3 (RUCAIBox STILL Team, 2025), DeepScaleR (Luo et al., 2025), and Open-RS (Dang and Ngo, 2025). \" What does \"build upon\" mean here? Which datasets have been used? Is it a fair comparison if you just collect more datasets for fine-tuning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rCIAiPYrMG", "reviewer": "Reviewer_38rQ", "summary": "This work proposes Tina -- a family of tiny reasoning models that achieve strong performance with extreme cost-efficiency offering a minimalist yet powerful path to efficient reasoning in small LMs. By applying LoRA during RL, Tina boosts reasoning ability at just $9 training cost which is very much cheaper than comparable models.", "questions": "1. Authors, an ideas on extension of the work to tool-calling scenarios and multi-turn rollouts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Novelty: One of the earliest works to demonstrate that GRPO based RL post-training can be used with LoRA\n2. Cost-efficiency: Achieves SOTA or near SOTA reasoning performance at a tiny fraction of typical RL costs\n3. Scalability & accessibility: Tiny models are easier to deploy and experiment with; open-source release ensures reproducibility\n4. Comprehensive validation: Multiple datasets and ablations strengthen claims about effectiveness and generality", "weaknesses": "1. As mentioned in the paper, while the novelty is limited, theis is one of the few works to systematically investigate applying LoRA with GRPO for reasoning and while some tricks on training dynamic have been shared, the technical contribution looks limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)