Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap
Authors: Yifan Sun
Yushan Liang
Zhen Zhang
Xin Liu
Jiaye Teng
Venue: ICLR 2026 Poster (2026)
Keywords: Training Dynamics
Self-Improvement
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper presents a physics-inspired model where the solver-verifier gap drives self-improvement, yielding an exponential capability convergence that accords with empirical observations on various LLMs and datasets.

Abstract: Self-improvement is a significant techniques within the realm of large language model (LLM), aiming to enhance the LLM performance without relying on external data. Despite its significance, generally how LLM performances evolve during the self-improvement process remains underexplored. In this paper, we theoretically model the training dynamics of self-improvement via the concept of solver-verifier gap. This is inspired by the conjecture that the performance enhancement of self-improvement stems from the gap between LLM's solver capability and verifier capability. Based on the theoretical framework, we further show how to model the entire training trajectory. This framework allows quantifying the capability limit of self-improvement by fitting the theoretical model to the experiment results. We validate the effectiveness of the theoretical framework on various LLMs and datasets. Beyond self-improvement, we extend our analysis to investigate how external data influences these dynamics within the framework. Notably, we find that under limited external data regimes, such external data can be utilized at any stage without significantly affecting final performances, which accords with the empirical observations.

Published as a conference paper at ICLR 2026
THEORETICAL MODELING OF LARGE LANGUAGE MODEL SELF-IMPROVEMENT TRAINING DYNAMICS THROUGH SOLVER- VERIFIER GAP
Yifan Sun; Yushan Liang', Zhen Zhang, Xin Liu' & Jiaye Teng!
School of Statistics and Data Science
Shanghai University of Finance and Economics Shanghai, China
(y1 srifan.sun, yushan @stu. sufe. edu. en, Lengflaye@sufe.edu
ABSTRACT
seff-imprrvement is significant techniques within the realm of large language model (LLM), aiming to enhance the LLM performance without relying on external data. Despite its significance, generally how LLM performance evolves during the self-improvement process remains underexplored. model the training dynamics of self-improvement via the concept of solver-verifer gap. This is inspired by the conjecture that the performance enhancement self- improvement stems from the gap between LLM's solver capability and verifier capability. Based on the theoretical framework, we further show how to model the entire training trajectory. This framework allows quantifying the capability limit of self-improvement by fitting the theoretical model to the experiment results. We validate the effectiveness of the theoretical framework on various LLMs and datasets. Beyond self ceff-improvement external data influences these dynamics within the framework. Notably, we find that under limited external data regimes, such external data can be utilized at any stage without significantly affecting final performances, which accords with the empirical observations.
this paper, we theoretically
we extend our analysis to investigate how
INTRODUCTION
Large language models (LLMs) have emerged as one of the most pivotal frontiers in artificial intelligence, propelling the development of diverse applications such as chatbots (Brown et al. 2020)), mathematical reasoning (Wei et al., 2022 Shao et al. (2024), and roboties (Wu et al. 2023). Despite their remarkable success, the training of LLMs typically necessitates massive data. In practice, data collection often confronts significant challenges, and there are even concerns that available data sources could be depleted in the future VVillaoobos al. 2024 Shen al. 2025 Muennighott et al. 2023: Wang et al. 2024a) This data bottleneck motivates researchers to explore alternative training strategies (Gao et al. 2020 Dong al. (024)..
Among these strategies, a growing body of work focuses on training or fine-tuning LLMs using the data they generate, process known self-improvement (Bai al. 2022 Huang al. 2023 Wang let al. 2023) Pang et al. 2024). Self-improvement methodologies initiate with a pre-trained L.M.M, utilize the model to generate new data, and then fine-tune the model with the generated data. Empirical studies have shown that this approach yields promising results across various domains (Zelikman
al. 2022 Wang al. 2023 Tian et al. 2024). However, the theoretical underpinnings of self- improvement remain under-explores (Song et al. 2025 Huang et al. (205)). Specifically, there is lack of theoretical models to explain its mechanisms and insufficient evidence to fully understand the training dynamics involved in this process.
"Equal contribution. Corresponning author

---

Published as a conference paper at ICLR 2026
hedf-Immeovveent
Coww-Ingeovennnt
Salve.4
Selvee Veriffer
Salver Veriliee
Cap
Vertee
YeriferA
Figure 1: Illustration of the theoretical results. The training dynamics potentially empowered by the solver-verifier gap under the theoretical framework. For self-improvement (left), the accuracy exhibits exponential curves towards limit. For cross-improvement (right), adding external data (a different verifier with limited queries) at different times yields similar performance.
In this paper, theoretically model the training dynamics of LLM self- self-immrovemmnt, inspired by the conjecture that self-improvement capability arises from the gap between an LLM's is solver capability and verifier capability [
Specifically, we define these two capabilities as follows:
Solver capability U, (t): The quality of responses directly generated from LLM;
Verifier capability U. (t): The quality of responses generated from LLM and evaluated by LLM.
In practice, differrent metrics can be used to quantify these capabilities. For tasks with ground-trude
labels, the 0-1 1 training loss can serve as direct measure. For tasks without ground truth, uncertainty quantification can be adopted, as it correlates strongly with model capability (Huang al., 2025). Based on the above discussions, we model the training dynamics of LLM self-imppovement using the inspired by potential energy in physics (Rankine [853):
coupled differential equations in Equation |
dU.(() -F((t, dt
dU,(() -BE(t),
dt
where a, B denote the coefficients, denotes the epoch, (t) denotes the solver capability, (t) denotes the verifier capability, and E(t) denotes the capability gap related to (t) - (t). We omit the initial conditions for simplicity. Under such a framework, the resulting dynamics would be
(t(t) R d'e -(a- + U2,00 U.(1) =z k(a- + U2o00
(2)
where a' 3" l denotes the constants with detailed formulations in Section 3.2 and Us Un.oc represent the solver capability and the verifier capability at convergence, respectively. Along the trajectory, both capabilities follow exponential convergence laws according to the framework. Besides, we specifically focus on the solver's ultimate capability U..00 --3 (aU..0 3U..0 + a This result reveals that the solver's ultimate capability relates to both the initial capability U. U..o and the coefficient a, B. Therefore, the ultimate capability might be improved given a larger verifier-solver solver gap at initialization, in accordance with our insights. Detailed derivations are provided in Section 3.2
From the experimental perspective. we find in Section that such theoretical modeling demon- strates strong practical efficacy. Specifically, experimental results across various models and datasets in Figure 3 reveal that the capability dynamics indeed follow an exponential law, as implied by the theoretical framework in Section 3 Besides, we observe in Figure that the verifier capability consistently outperforms the solver throughout the self-improvement process, which might be empiri- cal evidence of the polver-verifier gap's 's crucial role in driving capability improvement. To further investigate the role of the solver-vens ver-verrf gap, we conduct experiments in Section 4.3 on multiple LLMs demonstrating that the gap generally occurs practice. These findings hold across varying sample sizes and even under cross-evaluation scenarios.
We further analyze in Section the application of this framework in cross-improvement potential approach to enhance the ultimate capability of self-improvement. Cross-improvement in this paper refers to the utilization of external data in the verification step (details in Figure 7). Within the
We term this the solver-verifier gap to ensure consistency with our mathematical notation (U, for Solver, U. for Verifier) and our specific Best-of-N NN implementation. We explicitly acknowledge that this concept builds directly upon the generation-verifcall verificatiii gap originally conceptualized by Song et al. (2025). and we adopt this terminology with deep respect for their foundational contribution.
2

---

Published as a conference paper at ICLR 2026
theoretical framework, we contend that cross- roos-improvemmet outperforms self-improvement due the enhanced verification capabilities. We then derive the training dynamics under the cross- poss-improvvemem regimes, with enhanced verification capabilities compared to relf-improvement. This analysis can further assist in answering the question: given the limited amount of external data, how should one allocate these external data during the vrossiimprvvement training process? Our theoretical analyses demonstrate that the timing of using external data is not crucial; rather, the utilization of such data genuinely enhances the training process. Therefore, external data can be incorporated at any stage as desired. Experiments in Section 5.2 on the cross- inorovimprovement validate the theoretical findings. Our main contributions are presented Appendix A.2
RELATED WORK
cefr-lmproeeeet. Self-improvement aims to enhance model performance without relying on
external information (Huang al.) 2023 Wang et al. 2023 Bai et al.) 2022; Pang et al., 2024).
Self-improvement is of significant importance as it enables models to adapt and evolve autonomously.
thereby facilitating their effectiveness across wide range of real-world scenarios, such as reason-
ing (Zelikman et al., [2022] Peng et al., 2024; Huang et al. 2023). alignment (Wang et al. 2023
2024b Ding et | 2024), and planning (Tin et al. 2024). In this paper, we consider a branch of
self-improvement that fine-tunes with the output of LLM (Amini et al. 2024]] [Sessa et al. [024]] Gui
cct al. 2024 Pace et al. 2024 Ouyang et al.) 2022; Rafailov et al. (023). Alternative approaches
to sell-improvement include self-distillation (Buciluà et al. 20066] Hinton et al. 2015 Zhang et al.)
2019) which involves transferring knowledge from a larger, more complex model to a smaller one,
and self-correction (Kumar et al., 2024; Liu et al., (2024) where the model identifies and rectifies its own errors, etc. A body of research has also explored the potential negative consequences of seff-improvement, including degradation issues (Bertrand al. 2024 Gerstgrasser et al. 2024) and
failures on out-of-domain reasoning tasks (Yuan et al. 2025).
Theoretical Understandings on Self- Seff-Immprvement. Theoretical insights into self self-improvement could potentially enhance comprehension, thereby making self-improvement more reliable (Yam- polskiy 2015). Previous works have theoretically studied self-improvement via self-distillation
techniques. providing convergence rates for linear models (Mobahi et al.] 2020: Frei et al.] 2022 |
Das & Sanghavi 2023) Pareek al., 2024), neural networks (Allen (Allen-Zhu & 2023). and general models (dsix-Adsera] 2024). In the realm of LLM, several works have theoretically explored self- improvement with in-context alignment (Wang et al. (202c)), reinforcement learning (Talvitie) 2017]
Choi et al. 2024 Gandhi et al. 2025). meta learning (Kirsch & Schmidhuber) (002)), and diffusion
models (Fu ct al. 2024). Nevertheless, the theoretical understanding of self-improvement in the context of LLM training dynamics remains underexplored.
Most relevant to our work is Song et al. 2025) and /Huang et al. (0025). Song et al. (2025) posits that the key to self-improvement lies in the generation verification gap and further examines the relationship between this gap and pre-training flops. [Hung et al. (2025) further posits that the improvement stems from a sharpening mechanism, in which the verification step sharpens the model performance on the high-quality sequences. Our paper draws inspiration from [Song et al.) (2025): Huang et al. (2025), as we employ the concept of colver-verification sharpening gap in the theoretical analysis, and the training policy in this paper follows Huang et al. (2025). However, different from [Song et al. (2025). Huang et al. (2025), our research primarily centers on developing the self-improvement dynamics based on the solver verificatinn sharpening gap. Additionally, we delve deeper into understanding how cross cross- coossiimpovvemett works within this framework.
Cross- muss-Improvemmnt. Besides seffiimprovement approaches, a branch of papers focuses on enhancing the capabilities of LLM through external data, namely, cross-improvement. One of the most frequently utilized sources of external data is human-annotater data (Ouyang et al., 2022 Lightman et al. 2024
Borchers et al. 2025). Despite its utility, the collection of such data is extremely resource-intensive. Moreover, relying solely on human--nnotatee data restricts the potential for LLM to surpass human performance. Another potential source of external data stems from stronger models (Ho et al. 2023
Chang et al. 2023 Lee et al.) 2024), while access to these stronger models often presents sigmficantt challenges. Our paper also considers the scenario of cross- inoosiimpr that leverages a limited number of tokens from stronger models.
3

---

Published as a conference paper at ICLR 2026
3 THEORETICAL MODELING OF SELF-IMPROVEMENT TRAINING DYNAMICS
This section presents a theoretical framework for sketching training dynamics in LLM self- improvement. We start by introducing necessary notations of seff-improvement in Section We then theoretically model the self-improvement dynamics in Section 3.2
3.1 SOLVER AND VERIFIER
This section introduces the basic notations and definitions, as well as the definitions of solver capability and verifier capability. We start from the basic notations on data and models.
Notations. Let (x, v) denote a prompt-response pair, with yl denoting the k-th token and y denoting the first k tokens. We use L(y) to represent the length of y. Let my vy (ylar) denote the probability that model f generates a response given prompt æ, where / (y/z) can be split with the reto-segressive structure of the response x xy(1/z) IE (y() y ([===111,)) 2)). We denote the best response as y", that is, the ground truth response. Ideally, a model's performance could be measured by its loss relative to this ground truth, for instance, T.)(0) = lly" - هll However, as not all tasks have accessible ground truth, different metric is required. Therefore,
also use the uncertainty metric following Huang et al. (2025) in our framework. We define the uncertainty for a response ú given its prompt and model
its negative log-likelihood: (ū} log x (ū]az) The model's capability is inversely related to the uncertainty: a lower uncertainty signifies a higher capability. Further diseussion is available in Appendix A..3
Solver. The solver is regarded as the model capability to return responses with low uncertainty. Therefore, for each prompt Ti we sample one response (t) to represent the solver solution. That is,
ú: T (-1x).
Note that we only draw one response for each prompt due to calculation efficiency. An alternative solution is to generate multiple responses and use the whole distribution, but the two policies are similar since already take average operations over (i.i.d.) prompts.
Verifier. We use LLM itself as the verifier. For each prompt z4. we first sample N responses based on the LLM output Vi,1, ... Úi,N ny (-]aa)) We then ask the LLM evaluate these responses with a score ((i.s) E [0, 1]. The Best-of-N (BoN) response is then defined as
BoN arg min U (0.,))(),))2))) L(Dr.))
1
U Uy( (zx))
where a denotes the threshold parameter, and we use 1/ (fi()) as a regularizer to discourage those short responses. The BoN solution first eliminates those solutions with small scores s(i,,), and then finds the solution with the best capability. We deploy such a mixed strategy to enhance the computational stability; have large variance. Obviously, the BoN solution merges the L.LM verifier capability and the LLM output. Therefore, the verifier capability can be calculated based on the uncertainty of the BoN solution. We finally remark that our paper uses a slightly different BoN policy compared to Huang al. 2025) where they do not eliminate those responses with low scores, since we want to include more verification capability in the BoN response.
comparison, strategy with only score s(ūa.) might cause Jooo to
Uncertainty Metrics. Based on the above discussion, we use the average uncertainty of LLM response ý to represent the solver capacity and use the average uncertainty of BoN response BBoN to represent the verification capability. Therefore, we define the solver uncertainty Ut(t) and verifier uncertainty U. (t) as
U.(t) A
n Σ log ())(())) (t(t) A
n log ग. T (f)l:))
(5)
where (t) denotes the LLM output. In our framework, uncertainty metrics serve as inverse metrics of capability: a lower uncertainty value (U, or U.) indicates a higher corresponding capability. Note that both Ut(t) and (t) contains randomness, since ūt(t) is randomly generated by LLM, and the score s(-) in goN (t) is also randomly generated by LLM. However, since the prompts x; are independent, the randomness could be controlled when the number of prompts is large.
4

---

Published as a conference paper at ICLR 2026
Self- Iafro-mproveent..... We deploy self-improvement based on the above solver-verifier framework, similar to Huang et al. (2025). Notably, the verifier is slightly different, since we want to include more verification information. Overall, we first generate responses from the LLM. The optimization objective is to minimize the average uncertainty of BoN responses. The loss function Ie(J) for a training step & with function f is defined as the verifier uncertainty (+)+): L.()) & Ut(t) = n Di= log y (t))z.) By minimizing (J), we steer the model to increase the likelihood of generating high-quality responses, improving its solver capability. We summarize one-step self inffoipproeeenttt algorithm in Algorithm
3.2 SELF-IMPROVEMENT DYNAMICS
In this section, we aim to analyze the dynamics of self-impprveeeent. Given the immense complexity of the underlying neural network processes, deriving dynamics from first principles currently intractable. Therefore, study of complex systems. Our techniques are inspired by the concept of potential energy, a widely used concept in physics (Rankine, [853). Following Huang et al. (2025) and (Song al. (2025), , argue that the self-improvement comes from the Capability Gap G(),, defined as the gap between (t) and U, (t), namely,
adopt a phenomenological modeling approach, which is common in the
R 71
Ty (ūx (2;) TT T
G(t) a U,(())U,(t))
log
(6)
BoN (t)
We assume that the change of the solver and verifier capability is driven by a gap potential energy Et(t), expressed where (G) is differentiable and monotonically increasing function for G 2 0, with its minimum f(0) = 0. In this paper, we simply use uncertainty evaluate the response quality, leading the capability gap defined with the uncertainty, as discussed in Section 3.1 We assume that the solver capability and the verifier capability both increase during the process of self-imprrvemmnt,, which is widely observed in the related works (Song et al. 2025j. With the analysis above, following the concept of potential energy, the theoretical framework starts with the following assumptions:
function of the Capability Gap G(t) (Equation O namely E(t) f(G(t)).
U,(()]=o = Us.0, U.(()]e= = Un
dU) = -oF(() dU.(1) = BBE(t),
(8)
dt
dt
where a, are coefficients related the decreasing rate of (t), (t) Equation 7 represents the initial conditions. Since for LLM, the verification capability usually outperforms the solver capability in applications, we assume that Us. > Ur.o- Equation (8) represents the iterative conditions, where we assume that a > B, indicating that the solver capability increases faster than the verifier. Based on the above assumptions, we derive the differential equation governing the Capability Gap:
dG(l) = -))
dt
However, for non-linear function f(G). closed-form solution for above integral not guaranteed. To simplify this problem, we approximate the potential energy function E(t) with linear form, Fi(t) R kG(t) - b, derived from a first-order Taylor expansion. Figure Sngurreiilne illustrates the strong linear relationship between uncertainty gap G and its rate of change dG/dtt with QE method, which provides robust evidence for our approximation. Experiment results on other models are presented in Figure g Based on the Equation (8), we derive Proposition
(a E() Phi-4-mini
Assume ki(a - B) > O, the dynamics of the potential energy E(t), the capability gap G(t). Let the potential energy function f(G be linearly approximated around an initial state Go by its tangent line f(G) z kG - b. where = f' (Go) and = ('o - f(Gn). Capabilities U!) and (!) are governed by the following exponential decay functions:
Proposition 3.1
E(t) kōe -ka-B1
G(t) x de B)t + G
(10)
U.(t) -'c- -3- Us,00+ (t) B'e--kaa- -(aa-)-)) + Uepoo
(11)
where the coefficients are defined as: 8 = U,o0 - Uy k a-3 convergence (t - 00) are given by: Go
and the values at g'' a-z3
_85
h :: Us,00 U.o a' Un, U.. B¹.
5

---

Published as a conference paper at ICLR 2026
Math Train Split
Math Tast Split
GSMBK Train
GSMBK 0.938
R 0.966
R 0.859
R - 0.928
AG/ot
dG//t
-25
-20
25-
-14
10
Figure 2: Verification on linear relationship between the Uncertainty gap G and its rate of change dG/dt on Phi-4-mini with QE method. The scatter points represent empirical data from self- improvement on the Math and GSM8k datasets, while the solid lines show the linear regression fits.
Equation 10 and Equation || demonstrate that uncertainties Us, U,, and their gap G(t) all converge exponentially toward their respective limits. The proof Proposition 3.1 is shown in Appendix B.1 We derive two key corollaries from Proposition 3.1
Corollary 3.1 The partial derivative of the solver's final uncertainty with respect to the initial gap is negative constant: a0,,z.
8
8Go
The proof of Corollary 3.1 is shown in Appendix B.2 Corollary 3.1 shows that a larger initial verifier-solver gap (Go) leads to a lower final solver uncertainty (U 8,0) This aligns with the core intuition that the performance gap is the driver the self-improvement process.
Corollary 3.2
For a given tolerance >> the training time I required to ensure the Capability Gap Intó/c)
is within e of its convergence limit satisfies: >> Ma-0]
The proof of Corollary 3.2 is shown Appendix B.3 Corollary 3.2
provides theoretical guidance for selecting the total number of training epochs 7 in practical application. In practice, we first estimate the key convergence parameters by training for a few initial epochs and fitting the early-stage performance data. These parameters can then be used to determine an appropriate total training time T to balance performance with computational costs.
The dynamics indicate that (i) the gap potential energy E(t) drives the solver capability more strongly; (ii) the change of solver capability and verifier capability slows down as the gap decreases; and (iii) the capability gap might not converge to zero during the training process.
EXPERIMENT
This section conducts experiments on self-improvement to verify the theoretical framework. We first introduce experimental setups in Section 4.1 We then present experiment results of the self- improvement process in Section 4.2 We finally explore the differences between solver capability and verifier capability in Section showing that the solver-verifier gap generally happens in practice.
4.1
SETUP
This section introduces the models, datasets, and key parameters, We consider the following two verification methods: TrueFalse (TF) and Quality Evaluation (QE). We utilize two model families: (a) Phi models: From the Phi family (Abdin al.) (024), we use Phi-4-mini, Phi-3.5-mini, and Phi- 3-mini; (b) Llama models: We use Llama-3.2-3B (Grattafiori et al. (2024) and Llama-3.1-8B. Our experiments mainly focus the models' mathematical problem-soiving capabilities. Accordingly, we employ two representative datasets: GSM8k (Cobbe et al. 2021) and Math (Hendrycks et al. 2021). In addition, we also consider ProntoQA and MBPP datasets with TF method in order to evaluate our framework on QA and code generation problems. Experimental details are provided in Appendix D

---

Published as a conference paper at ICLR 2026
Math Accuracy
Math Uncertainty
GSMBK
GSMBK Uncertainty
Solver
0.90
Solrer Tann
0.45
Solver Thst
Solver Rest
Veriflee Thain
0.85
Vertfer Train
Accuracy
Uncertainty
Veriflor Test
Accuracy
Uncertainty
80
Vertfer Test
0.80
60
0.35
Salver Tran
0.75
Saloer Irain
Salver lest
Savver lest
Verifier
50
Verifer
0.30
0.70
liee
5
10
10
10
Epoch
Epach
Epach
Epoch
Figure 3: Accuracy and uncertainty during the self-improvement of the Phi-4-mini model on the Math and GSM8K datasets using the QE method. The experimental results show that the accuracy increases during self-improvement process while the uncertainty decreases.
4.2 DYNAMICS OF SELF-IMPROVEMENT
This section sketches the dynamics of eeffiimprveemett using Al-generated feedback for supervised fine-tuning (SFT), aiming to verify the theoretical framework proposed in Section 3.2, We test the Phi-4-mini, Phi-3.5-min Phi-3-mini, and Llama-3.2-33 models. We apply Low-Rant Adaptation (LoRA) (Hu et al., (022), which significantly reduces the number of updatable parameters, thereby enhancing SFT efficiency. The chosen hyperparameters are detailed in Appendix D.I
Results. We present the results of self-improvement on the Phi-4-mini model using the QE method, as depicted in Figure 3 The empirical evidence indicates a consistent enhancement in the accuracy of both the solver and the verifier during the self- ceff-improvement process, coupled with concurrent reduction in their respective uncertainties. Furthermore, a narrowing of the gap Gi(t) between the solver and verifier is evident. Results for other models and methods are presented in Appendix D.2
Validation. To validate our theoretical framework, we fit an exponential model to the uncertainty from 10 self-improvemennt epochs. Figure 4 presents the results for the Phi-4-mini model in four different settings (Math / GSM8K datasets train split with QE /E/TF TF metrics), which illustrates the evolution of three key metrics: solver uncertainty, verifier uncertainty, and the uncertainty gap. In Figure the exponential model demonstrates strong fit to the empirical data, with the coefficients of determination (R²²) exceeding 0.9. This empirical evidence validates the exponential law proposed in our theoretical work. Results for other models are presented in Appendix D.2
4.3 VERIFIER OUTPERFORMS SOLVER
In this section, we evaluate the solver and verifier performance of LLM, aiming to validate the utility of the solver-verifier gap used in Section 3 Specifically, the experimental results verify that verifier capability outperforms solver capability consistently. Our evaluation is divided into two settings: self-evaluation and cross-evaluation. Self-evaluation employs the same model for both the solver and verifier roles, whereas cross-ae coos-evvalattion utilizes different models for the solver and verifier.
Self Evaluation. We compare the accuracy and uncertainty of solver and verifier on different models and datasets across different N. Given that the verifier selects one response from N candidates generated by the solver, the verifier's performance is expected to improve as N increases. Detailed results are presented in Appendix [.2]
Cross Evaluation. To better understand the relationship between solver and verifier capabilities, we also perform cross-evaluation, where one model acts as the solver and a different model acts as the verifier. Results presented in Appendix D.2
These experiments indicate that the verifier typically outperforms the solver, revealing a consis- tent positive performance gap between the verifier and the solver across model-task pairs. This performance gap is considered a key driver of self-improvement dynamics.
4.4 IMPACT OF TRAINING STAGES ON SELF IMPROVEMENT
In this section, investigate how upstream training stages influence the self- eelf-improvement dynamics, we conducted a controlled comparative analysis. This section addresses the potential confounder that models different stages might exhibit distinct optimization behaviors. We utilize the EvoLM model
7

---

Published as a conference paper at ICLR 2026
Math(QE)
(Q
MathiTF)
GMMB/TT)
200
Volee andittuty dare 2.11
Nete deartein Sarn S:W
230
200
ees Gerere -UR1
175
175
Bette
Uncertainty
250
Uncrtta by 10me-00---000 2011
Uncortainty inyy
Bi suartanly --1
Boaiterts Syrn 2- Save
Uncertainty
Uncalaate Carve P0r
123
100
120-
Epoch
Epoch
Epoch
Epoch
Figure 4: Exponential trends of model uncertainty during self-improvementt on train split. The results illustrate the uncertainty and gap of the solver and verifier for Phi-4-mini. The scatter points represent the measured data, while the solid lines are the best-fit curves to an exponential model. R² > 0.9 indicates that the evolution of these uncertainties is vell-described by an exponential function.
suite Qi et al. 2025) for this study, which allows us to control for model architecture (Llama-2 1B) while isolating the impact of training stages. We compared three checkpoints representing the full lifecycle:
Standard SFT Baseline (Base Stage): Derived from the raw pre-trained model (IB-160BT) via standard SFT. This represents a model with high latent potential but no domain-specific mid-training.
Mid-train Enhanced Model (Mid-train Stage): Derived from a model that underwent domain- specific Continued Pre-training (CPT) on math data (IB-160BT-8+42BT) before SFT. This represents the Nomain-primme state.
Fully Post-trained Model (Post-train Stage): A model that has completed the full pipeline, including mid-training, SFT and Reinforcement Learning 100Kep16). This represents the converged state.
We present the results in Figure 5 and Figure 9 The results shows clear evolution in training dynamics governed by model training stage: Base model exhibit the highest plasticity, driven by the largest initial gap. Mid-training lowers the initial potential energy and increases initial capability. Post-trained models demonstrate dynamic saturation, validating the asymptotic convergence as the capability gap minimized.
5 DISCUSSIONS ON CROSS-
In this section, we discuss how to model cross- improvement within the framework described in Sec- tion 3.2 The key insight is that external data affects the theoretical framework through the verification capability. We first present the theoretical framework in Section 5.1 under the framework with limited external data. We then conduct experiments in Section 5.2 to validate the theoretical findings.
5.1 THEORETICAL FRAMEWORK OF
In this section, present the theoretical framework of training dynamics of cross-improvement. We follow the notations in Section 3 Besides, assume that we have limited external data with size M. For example, we may acquire M external data in total from better LLM using API queries.
We focus on the allocation of the external data. Specifically, for each epoch t, only M.M prompts could use API queries to get (one) external data, with > = 1 where T denotes the total training epochs. For those chosen prompts, we choose the external data as the BoN response; for those non-chosen prompts, we still choose the BoN data from the N internal responses. Notably, as long as one prompt is chosen to use external data, it will always use the external data.
Cross-Data Effects. In the cross-improvement framework, the use of external data will influence the verifier capability (t), since we use a different definition of BoN which directly relates to the verifier capability. To model the effects, we assume the verifier capability after cross-improvement as
(1(() (1 + m.) - U. -))
(12)
8

---

Published as a conference paper at ICLR 2026
Base-SFT
Midtrain
Post-train
160
Gimo -ONL by
25
70
100
60
20-
50
Loos
80
40
40
20
20
10
10
Epoch
Epoch
Epoch
Figure 5: Self-improvement dynamics across Base, Mid-trained, and Post- stages on train split. Fitted curves (R²) reveal a clear evolutionary trajectory: (1) The Base- model exhibits the highest Solver Rate (a 146.3) and Decay Rate (0.244). (2) The Mid-trained model reduces the initial uncertainty, proving that mid-training optimizes the initialization state. (3) The Post-trained model shows the lowest rate (a 16.0), confirming the theory of saturation near the capability limit.
The parameter y represents the general effect of cross-improvement, while T) represents the ratio of external data used in epoch t. We assume that the effect 7 is time invariant, 7 > when the verifier that adds external data performs better than the verifier without external data; otherwise y 0. Overall, in each step, the cross- dats first boosts the verification capability, then it evolves following the mathematical framework in Section 3.2 We illustrate this procedure in Figure 7
Solutions. We model the above framework on cross cross-improve tross-improvement as
U,(+)l==0 U2,0,
U,(t)) 1-0 Un.0: (+(() (1 + - U. 1),
(13)
(5(t) U,(t - 1),
(14)
(1 = (5(() US(1),
F(t) = (1) b,
(15)
-5(() -oE((),
U5()) = -3E(().
(16)
represents the initial conditions, which are the same as Equation 7 Equation (14 represents the effects of cross-improvement, where the solver capability after inossemmpro (t) remains unchanged, while the verifier capability increases as discussed in Equation (12). The current capability gap (t) is then defined as the gap between the solver capability and the current verifier capability. Equation (16) represents the iterative conditions of cross cross-impronne improvement. Based on the above formulation, we derive an approximate solution of the ultimate uncertainty:
Equation 133
Proposition 5.1 Let the cittmallduta-affecttee solver uncertainty U!(() and verifier uncertainty (t). The ultimate uncertainty vector UTT) can be approximated as
T (1 +T 17 mh)
T -T:3
=(T)=e-Ai
-
-ak(T -1 71)
Tak
-Tab
(17)
0
where U(T) denotes (U,,T UST, 1)T and U0) denotes (Uw,0+ Us 1T. Us(T) is related only to the summation E nh (instead of each Tt at epoch t).
Under Proposition 5.1 we come to the following conclusions: (i) For cross-improvement with
th 1, the approximate solution is around the same; (ii) The cross-improvement with
1= Te outperforms self-improvement with Σ 1/+ = 0 in terms of solver capability, when
7 O. The proof of Proposition 5.1 is shown in Appendix B.4
5.2 EXPERIMENTS
In this section, perform cross- coss-improvemmet experiments using different allocation strategies. Figure [ illusttrates the process of cross vrossimmprovemet. For these experiments, external data are generated by DeepSeek-V V3 which achieves accuracy of 60.47% and 91. 18% Math and GSM8k

---

Published as a conference paper at ICLR 2026
Table I: Solver accuracy with QE method on train split: raw data and relative improvements of strategies average (%). Initial represents the performance of the original model, while Baseline represents the results from inff-improveent.. We deploy three strategies: Early, Uniform, and Late.
The last row shows the difference in verifier accuracy between using all external data at the start (Early strategy at 0) and the baseline's 's initial verifier accuracy (Initial).
Phi-4-mini
Llama-3.2-3B
Strategy
Math(%)
GSMBK(%)
Math %))
GSM8k(%)
Initial
30.31 (1: 0.24)
73.42 (+: 0.33)
36.02 (1: 0.25)
63.10 (== 0.58)
Baseline
45.08 (#: 0.48)
88.53 (#: 0.48)
49.16 (#: 0.90)
88.66 (= 0.73)
Early
46.33 (+: 0.16)
88.54 (+: 0.12)
53.48 (+ 0.41)
88.26 (= 0.57)
Uniform
46.56 (+ 0.19)
88.44 (+ 0.05)
52.74 (+ 0.82)
88.23 (= 0.58)
Late
45.83 0.48)
88.90 (+ 0.21)
51.31 (+ 0.76)
87.19 (- 0.28)
Max-Min
0.73
0.46
2.17
1.07
Avg
46.24
88.63
52.51
87.89
Avg vs Initial
+15.93
+15.21
+16.49
+24.79
Avg vs Baseline
+1.16
+0.10
+3.35
-0.77
Early(! vs Initial (Verifier)
+5.87
+0.96
+0.97
-1.58
dataset respectively. Specifically, we utilize DeepSeek V3 responses for fine-tuning instead of BoN responses. We test three allocation strategies: (a) Early: All external data are introduced in the first epoch. (b) Uniform: An equal amount new external data introduced each epoch. (c) Late: All external data introduced in the eighth epoch.
We present the results on the training data in Table [ We observe that the average solver accuracy for all three strategies is higher than the baseline the MATH dataset. However, Phi-4-mini shows slight performance improvement on GSM8k, while Llama-3.2-3B exhibits drop on GSM8k dataset. This observation indicates that key factor influencing the effectiveness of cross- coos-improvemmnt is indeed whether the external model's verifier capability significantly surpasses that of the original model. In addition, find that the difference in accuracy between the three strategies is slight, which validates the conclusions under Proposition 5.1
6 CONCLUSION
In this paper, we propose a theoretical framework to analyze the training dynamics of self- improvement via the solver-verifier gap. Experimental results on various datasets and models accord well with the theoretical findings. Besides, one may derive the theoretical limits based on the framework, which closely related the model's verification capability. To break the limit, one may apply cross cross-improveed improvement to enhance the model's verification capability. Therefore, we further introduce the corresponding theoretical framework on the dynamics cross-improvement under limited external data regimes, and find that the allocation of external data might have less influence on the final results. Experimental results verify the theoretical findings.
In the end, we provide several future directions: (i) We assume time-invariane property when analyzing the cross cross-improvement, which might be relaxed in future work: (ii) External data can be used to fine-tune self- celf-improved model to further improve the performance of the model; (iii) We use a phenomenological modeling approach in our framework. deeper investigation into the underlying mechanisms that cause the self- seff-improvement dynamics to be well-described by physics-inspired model would be a key future direction. This includes further research into the relationship between the model's key parameters, and 8, and factors such as the LLM's architecture.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This research is theoretical and methodological, focusing on developing a mathematical model for the training dynamics of LLM self-improvement. All experiments benchmarks and did not involve human subjects or sensitive data, thus requiring no Institutional Review Board (IRB) approval. Our work aims to deepen the understanding of the self-improvement mechanism to foster the development of more predictable and reliable models.
conducted on public academic
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide detailed descriptions our theoretical and experiment setup. Our theoretical model is detailed in Section 3 with complete mathematical proofs
for all propositions and corollaries provided in Appendix [ The experiment setup described
in Section [4.] Further implementation details, such as the self-improvement algorithm (Algorithm
and the hyperparameters used to obtain the results (Table 3) are available in Appendix D
ACKNOWLEDGMENTS
We would like to thank the members of the Jiaye's Group for their insightful discussions throughout this project. We are also grateful to the anonymous reviewers for their constructive comments which helped improve the paper. This work is supported by Shanghai Science and Technology Development Funds 24YF2711700 and Fundamental Research Funds for the Central Universities 20241 10586. Xin Liu partially supported by the National Natural Science Foundation of China 12201383 and Innovative Research Team of Shanghai University of Finance and Economics. Yifan Sun would like to express his deepest gratitude to his beloved girlfriend. Jinjin Zheng, for her unwavering companionship and support, which served constant source of strength throughout the inevitable lows this research journey. Yushan Liang would like to extend her special thanks to her parents for their unwavering support.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally your phone. arXiv preprint arXiv:2404. 142199 2024.
Zeyuan Allen-Zhh and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self- distillation in deep learning. The Eleventh International Conference on Learning Representations, 2023. URL https: openrevi ew. net. forum? d=Du 2g
Afra Amini, Tim Vieira, Elliott Ash, and Ryan Cotterell. Variational best-of-n alignment. In NeurlPS 2024 Workshop on Fine- Tuning in Modern Machine Learning: Principles and Scalability, 2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint 2tX.x::22/22.0077. 2022.
Quentin Bertrand, Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining generative models on their own data, In The Twelfth International Conference Learning Representations, 2024.
[d JORA
Enric Boix-Adsera. Towards a theory of model distillation. arXiv preprint arXiv:2403.09053, 2024.
Conrad Borchers, Danielle R Thomas, Jionghao Lin, Ralph Abboud, and Kenneth Koedinger. Augmenting human-annotated training data with large language model generation and distillation in open-response assessment. arXiv preprint arXiv:2501.09126 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
11

---

Published as a conference paper at ICLR 2026
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizzl. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Jonathan Chang, Kianté Brantley, Rajkumar Ramamurthy, Dipendra Misra, and Wen Sun. Learning to generate better than your Ilm. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
Eugene Choi, Arash Ahmadian, Matthicu Geist, Oilvier Pietquin, and Mohammad Gheshlaghi Azar. Self-improving robust preference optimization. arXiv preprint arXiv:2406. 24X1:24006001660,, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint 1rXiv:2110.14:00 2021.
Rudrajit Das and Sujay Sanghavi. Understanding self-distillation in the presence of label noise. In International Conference Machine Learning, pp. ap.71022 7102-7140, PMLR, 2023.
Mucong Ding, Souradip Chakraborty, Vibhu Agrawal, Zora Che, Alec Koppel, Mengdi Wang, Amrit Bedi, and Furong Huang. Sail: Self-improving efficient online alignment of large language models. arXiv preprint arXiv:2406. 15567, 2024.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1107-1128, 2024.
Spencer Frei, Difan Zou, Zixiang Chen, and Quanquan Gu. Self-training converts weak learners to strong learners mixture models. International Conference on Artificial Intelligence and Statistics, pp. 8003-8021, PMLR, 2022.
Shi Fu, Sen Zhang, Yingjie Wang, Xinmei Tian, and Dacheng Tao. Towards theoretical understandings of self-consuming generative models. In International Conference on Machine Learning, pp- 14228-14255. PMLR, 2024.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.013077 2025.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre- pre-trained language models better few-shot learners. arXiv preprint 2xX.:20022.1572:: 2020.
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes. Dhruv Bhandarkar Pai, Andrey Gromov, Dan Roberts, Diyi Yang, David L.. Donoho, and Sanmi Koyejo. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. In First Conference on Language Modeling, 5B2K421-55BB.R......
2024. URL httpp:://ppeneeeee....tt ottp://rwweeeeereeeeoooaaaaaaa/d
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Lin Gui, Cristina Garbacea, and Victor Veitch. BoNBon alignment for large language models and the sweetness of best-of-n sampling. In The Thirth-eighth Annual Conference on Neural Information Processing Systems, 2024. URL /tt///ttttss//ppeeeeieeeeee wwwp:eeewwwwwweeeereeeeeeeeeeeee formm www.mmmmiddhaaaaaa rbx5
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint
2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv 2015.
preprint
12

---

Published as a conference paper at ICLR 2026
Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. (4852-14882, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Audrey Huang, Adam Block, Dylan Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jor- dan T. Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism. The Thirteenth International Conference on Learning Representutions, 2025. URL
bttps::/<pperevieet net/
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang. Hongkun Yu, and Jiawei Han. Large language models can self-improve. In 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, pp. 1051-1068. Association for Computational Linguistics (ACL). 2023.
Louis Kirsch and Jürgen Schmidhuber. Eliminating meta optimization through self-referential meta learning. arXiv preprint arXiv:2212. 14392, 2022.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John Co-Reyes. Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409, /29/7, 2024.
Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael Mahoney, Kurt Keutzer, and Amir Gholami. Llm2llm: Boosting Ilms with novel iterative data enhancement. In Findings of the Association for Computational Linguisties ACL 2024, pp.
2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL / net/ Lttee/Eorumm1d=v8LL0p0..
Dancheng Liu, Amir Nassereldine, Ziming Yang, Chenhui Xu, Yuting Hu, Jiajic Li, Utkarsh Kumar, Changjae Lee, Ruiyang Qin, Yiyu Shi, et al. Large language models have intrinsic self-correction ability. arXiv preprint 2 15673, 2024.
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-50376, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024.
Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self-improvement by reinforcement learning contemplation. In The Twelfth International Conference on Learning Representations, 2024. URL https
//penrevii ew, fow.nett/oorrmm2dd333444444 yÜbrgr.
Divyansh Pareek, Simon Shaolei Du, and Sewoong Oh. Understanding the gains from repeated self- distillation. In The eighth Anmual Conference on Neural Information Processing Systems, 2024. URL U htt.httts://oppereeeee ott.: o=w.aaaa
net.
13

---

Published as a conference paper at ICLR 2026
Xiangyu Peng, Congying Xia, Xinyi Yang, Caiming Xiong, Chien-Sheng Wu, and Chen Xing. Regenesis: Lims can grow into reasoning generalists via self-imprrvemmen. arXiv preprint 1rXi::2410002108, 2024.
Zhenting Qi, Fan Nie, Alexandre Alahi, James Zou, Himabindu Lakkaraju, Yilun Du, Eric Xing. Sham Kakade, and Hanlin Zhang. Evolm: In search of lost language model training dynamics. arXiv preprint urXiv:2506. 16029, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-5337 2023.
William John Macquorn Rankine. On the general law of the transformation of energy. 1853.
Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning Ilms with best-of-n distillation. arXiv preprint arXiv:2407. 242:22627.4.. 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03 2024.
Tao Shen, Didi Zhu, Ziyu Zhao, Chao Wu, and Fei Wu. Will Iims scaling hit the wall? breaking barriers via distributed resources massive edge devices. arXiv preprint vrXiv:2503.02225 2025.
Yuda Song, Hanlin Zhang, Carson Eisenach, Sham M. Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self self- calr-improvemettt capabilities of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL /ettps://openree
net. Fortm?? id=tt
Erik Talvitie. Self-correcting models for model- based reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Ye Tian, Baolin Peng, Linfeng Song. Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu. Toward self-improvemmnt of iims via imagination, searching, and criticizing. Advances Neural Information Processing Systems, 37:52723-52744 2024.
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? limits of Ilm scaling based on human-generated data. In Forty-first International Conference on Machine Learning, 2024.
Ke Wang. Jiahui Zhu, Minjie Ren, Zeming Liu, Shiwei Li, Zongye Zhang, Chenkai Zhang, Xiaoyu Wu, Qiqi Zhan, Qingjie Liu, et al. A survey on data synthesis and augmentation for large language models. arXiv preprint urXiv:2410. 1286,, 2024a.
Xiyao Wang. Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, al. Enhancing visual-language modality alignment in large vision language models via self-improvement. arXiv preprint arXiv:2405. 15973, 2024b.
Yifei Wang, Yuyang Wu, Zeming Wei, Stefanie Jegelka, and Yisen Wang. theoretical understanding of self-correction through in-context alignment. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024c. URL https: http::///ppeeeeeeee.omom foreet//oorummmiioomm
OtvNLTWYWN
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). Association for Computational Linguistics, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of ohothhttht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2444 248877, 2022.
14

---

Published as a conference paper at ICLR 2026
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng. Shuran Song. Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots, 47(8):1087-1102, 2023.
Roman V Yampolskiy. From seed ai technological singularity via recursively self-improving software. arXiv preprini
12, 2015.
Xiangchi Yuan, Chunhui Zhang, Zheyuan Liu, Dachuan Shi, Soroush Vosoughi, and Wenke Lee. Superficial self-improved reasoners benefit from model merging. arXiv preprint arXiv:2503.0210: 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3713-3722, 2019.
15

---

Published as a conference paper at ICLR 2026
Appendix
This part provides the supplementary materials. before. Appendix [ provides omitted derivations of the theoretical framework. We claim the use of LLMs in Appendix Appendix D provvides experiment details and omitted experiment results.
Appendix A provides necessary details omitted
A SUPPLEMENTARY DETAILS
In this section, provide the details omitted before. Appendix A.1 and illustration, while Appendix[A22 summarizes main contributions of our work. In Appendix [A.3] discuss the relationship between uncertainty and capability. Additionally, we further diseuss the difference between our work and prior works, as well as the challenges for self-improvement in Appendix A.4
presents the omitted algorithm
A.1 OMITTED ILLUSTRATION
Figure 6 gives an overview of theoretical framework. Algorithm D presents one-step self- improvement used in our experiments while Figure illlustrates cross- cross-improvemen process.
Self Improvement
Cross Emsprovenent
LUN improves ls casasilly suporisdd
E a)M
LUM imgroves capapiltyy with
-1 - ey (HjR).
Self Inprevmenn bynaniss
0200 - Thll (. 1), U210 - U.f
Intal Condition
Beothe Condition
E2/5) ECAS sove aorrnnt cuailily
by feserr sropoc each
ofte
77,0) U.a.
a0,(0)
(())
Capobiitty
Theanetcca Modeling
01/0) U.g
at-19
(R()
Git) (5 7766)
Coecition
Twesthe Coecition
To Condract
1.,%% Uut
U,00 0200) a500,
Energy AG11)
(*()) - U7Ce 1771)
0.00lim Usp
0L00) Bg06) 6510)
Self Iepeowesent dyeeeses:
AC"10
(n
(700 ole Tlane
The effect introdacing crose-dato at different spach:
(.(() d'c Me Rt Ueaar
US0 U7,0...
in Al 17(3).
f- 400 EL 7A -130
Uiper Paye reprannn inferent uncertandy
Grocc improvonntt surposses seft improveennt salver opobl The timing of using citarel cata ou.da.
Figure 6: Overview of the theoretical framework on reff-improvement and cross-improvement
A.2 CONTRIBUTIONS
In this section, we summarize main contributions of our work. Our main contributions are:
We detail our theoretical framework on the dynamics of self-improvement in Section inspired by the potential energy framework based on the solver-verifier capability gap in Equation 6 Theoretical derivations in Equation (II) imply an exponential law for the model capability, and we further establish the extreme capability based on the framework.
Theoretical framework:
Experiments: We conduct experiments on self- uuff-mprovemmnt to verify the theoretical framework in Section across multiple models, datasets, and verification methods. Our empirical observations indicate that (i) the uncertainty/accuracy during the self-improvement process follows an expo- nential law with extreme capability (Figure 3 as implied by the theoretical framework; (ii) the solver- solver-verifier gap generally happens in practice across different regimes (Section 4.3) validating the utility of the gap the potential energy.
cross-improvemment: We further investigate in Section the tross-improvemmnt under the above framework, where limited external data is provided during the training process. We contend Section [.. that cross- pooos-improvement might improve the verifier capability, thus improving the extreme capability of self-improvemment Empirically, observe in Section 5.2 that if the verifier
16

---

Published as a conference paper at ICLR 2026
Algorithm 1 Self Improvement (One Step)
Require: Prompts set xf: 22," Fn Current model f, Sample Size N, Threshold o
1: J.BoN <- 0, LBBoN <- 0;
2: for each prompt 2; X do
3: C2: + 0:
4: for j .V do
5: Generate response Vi, (-(-1z;);;
6: Ask model evaluate Hi,5 and return a score s(bs,)):
7:
if (0i,)) 2 then Append Whi to Cz.i end if
8:
9:
10:
end for
arg minc., I(bns) 12: Append JPOoN to JIBON
11: J.aNN
T Uy( (x:); (x);:
Append Ug(BBNN) UBO:
14:
end for Uncertainty RESTINT Eucuoo u,
f Adam AdamW(f. Uncertainty);
Ensure: Self-improved model f
Question
Yes
No
Internal Verifier
External data?
Response
External Model
Solver Model
Response
Verifier Model
Response N
Best Response (Verifier Choice)
External Data
Supervised Fine-Tune
Figure 7: A diagram of Cross Improvement
capability is improved by the external data, the performances of LLM are enhanced, and vice versa (Table [ This accords with the thcoretical findings.
A.3 RELATIONSHIP BETWEEN UNCERTAINTY AND CAPABILITY
In this section, we clarify the relationship between uncertainty and capability. Although low uncer- tainty is often correlated with high capability (i.c., accuracy), they not conceptually equivalent. In this work, we use uncertainty instead of accuracy as a proxy metric for model capability. The justifications detailed follows:
Sharpening Mechanism:
Following Huang al. (2025). minimizing the loss (U.) forces the model to sharpen its intrinsic probability distribution onto high-quality responses. A decrease in U, signifies that this distributional shift has occurred and that the model has successfully learned from the verifier's 's output. Thus, U, is the direct measure of the model's capability.
17

---

Published as a conference paper at ICLR 2026
Empirically correlation between accuracy and uncertainty: To validate that U, indicates the model's capability, we calculate the Pearson correlation coefficient (r) between U, and solver accuracy for different models and datasets (Table 2 Accuracy and Uncertainty for most model-dataset pairs are significantly negatively correlated (Irl > 0.8), which proves that reducing solver uncertainty (Ux) translates gains in accuracy.
many real-world scenarios, particularly open-ended genera- tion tasks, ground-truth labels often unavailable or difficult to define. Unlike accuracy, which strictly depends on external gold references, uncertainty is computed from the model's output distribution. This makes uncertainty necessary and universal metric for monitoring capability evolution in label-free regimes where accuracy calculation is infeasible.
A bsence Ground Truth:
The Optimization Objective: In standard SFT, the objective of training is maximize the likelihood of correct responses, which is mathematically equivalent to minimizing their uncertainty (Negative Log-Likelinne Likellioi,hod). In our framework, we regard the output of the verifier as the ground truth for SFT. The loss function at step defined L,(f) & (t) Ei= 1i=1 log T BoN (t)[ which is identical to our definition of verifier uncertainty (t).).
Table 2: Correlation coefficients between Uncertainty and Accuracy.
Dataset
Method
Llama-3.2-3B
Phi-3-mini
Phi-3.5-mini
Phi-4-mini
QE
-0.963***
0.985
0.974
-0.991 ***
GSM8k
TF
-0.966**
0.990
0.967
-0.988
QE
0
0.958
0.177
-0.933**
Math
00967***
0.900
0.167
0.978
ProntoQA
TF
-0.677***
-0.897**
0.764
-0.850 ***
MBPP
TF
-08655
00.951
0..93
-0.6735*
Note: < 0.001, < 0.01, 0.05.
A.4 REVIEW AND PROSPECT
In this section, we discuss the difference between our work and prior works. We also discuss the challenge and perspective of self-improvement.
Different definition:
In previous work (Song et al.) 2025), the solver capability is defined as the accuracy, while the verifier capability is defined the accuracy of responses that the model deems good responses. The calculation of accuracy depends on the golden answer contained the original dataset, which is the most significant difference from definition. In practice, many datasets do not contain gold answers, so our definition could be more widely applied.
Different concerns: Most of the prior works focus finding method that could make self- improvement more efficient. Instead, we pay attention to model the dynamics of self-improvement, which helps understand the progress of self-improvement. In addition, we propose theoretical framework for cross- cross-improvement which could be regarded as a solution to break through the limit of seff-improvement.
Challenge: (i) Under supervised fine-tuning, self-improvement may be misled by incorrect responses. Thus, ensuring that the verifier outputs correct responses is challenge. (ii) The current self- improvement method can only improve the model's capability in similar tasks by optimizing certain task. Finding a method to improve the performance of the model on different tasks challenge.
B THEORETICAL DERIVATION
In this section, we present the detailed theoretical derivations omitted before.
18

---

Published as a conference paper at ICLR 2026
B.1 PROOF OF PROPOSITION 3.1
In this section, we present the detailed derivation of Proposition 3.1 We first derive Equation 10 From Equation (8) and linear approximation S(G) kG b, we have
dG((t
- b).
(18)
dl
The general solution of Equation
(18) is
+ b k
G(t) x Ce Ce- k(a-
(19)
where C is a constant. To find the constant C, we use the initial condition at t 0: G(0)
Ue.0 - Ue, = Go, thus
G(0) Ce
+ k.'
(20)
b
Us,0 C = Uv.0
(21)
k
Let 6 = Un.o Us b F. we have
G(t) de M(a-B)0- +
(22)
k
Then derive Equation From Equation 22 we have
Ett) = kt()) b k(ōe k(a-B)t + b/k) kōe
kōe B))
(23)
Solving the original differential equation for (.(
dt,(t) -aE(t) aa(kée -kakkkt((a-811) k(a-3)1).
(24)
dt
/ dU,(t) = / -akde- k(a -kkokkk -ka- dt
(25)
/ -akôe
M(- -akke-Ma-88 dt aakō c Afje
(-k(a (8)
+ C..
(26)
a6
U,(t) 23
3 c -((a-0)1 + C...
(27)
Using the initial condition U.(0)= = Us. to solve for the constant C.:
ad
ad
U.o a 3 o + C. = Us= a - 33
(28)
Let a' = as a-g and U, = USO - a' Thus, the solution for U,(t) is
U.(l) x d'e- -(a-83+ + U
(29)
The derivation for (t) is analogous, yielding
U.(l)= R kla--88 U2
(30)
where B' = 0-3 and Un.00 = Un.0 Br.
B.2 PROOF OF COROLLARY 3.1
In this section, we present the detailed derivation of Corollary solver uncertainty is given by US U.o a', where a'
3.1 From Proposition 3.1 the final a- and 8 Go b/ Substituting
these into the expression for Us.0 yields U U2.0 a- (oo }). Assuming U, is constant,
have aU.0. 3G0
0. Thus, have a0. 80o
a-
19

---

Published as a conference paper at ICLR 2026
B.3 PROOF OF COROLLARY 3.2
In this section, we present the detailed derivation of Corollary 3.2 The objective of Corollary (.2 is to find the time i such that the capability gap G()) is within a tolerance of its convergence value Goo which is expressed as:
[t Gol <.
(31)
Starting from the expression for G(t) given in Proposition 3.1, have:
G(t) x de
K)a- G
(32)
G(t) Go s de h0--(--8
(33)
Since 8 is defined on the basis of the initial gap, we can assume 8 O. Thus, we have
de k(a-9) <e,
(34)
In(6/c) k(a ()
(35)
B.4 PROOF OF PROPOSITION 5.1
In this section, we present the theoretical derivation of Proposition 5.1 dynamics conditions are formulated as:
According to Section
5.1 the
U,(+)l==0 = Us.0.
U,(t) = U..0
(36)
UŞ(() U.(! - 1),
U}(() (1 mm)) U. - 11),
(37)
(*(() = UT(() - -U((),
E(t) -I) bb b,
(38)
U,(t) US(I) -aE(t,
(t) UZ(() = -BE(t)).
(39)
Denote Ut) vector [U,(t), Ut(t), 11, (t(t) as vector [US(t), Ug(t), 1] with the fol-
lowing relationship holding true:
0
0
1+mm
U+)
0
1
U(t 1).
(40)
0
0
1
With these notations, we can derive the following iteration from Equation (37 to 39
U(t) (I A,)
U( 1),
(41)
1l8k
Bk
3b
Itym
A, =
ok
ak
ab
(42)
147m
0
0
Based on the iteration, U(t) at the end of the T epochs is then:
T
U(T)
(I [I- U((), E=1
(43)
where U()) denotes (U.,. Uso. I]_ The meaning of Un,o, Us.0 is defined in Equation Under the circumstances, the following approximation holds true:
39
T I
A.
R E = ST A. A
II
(44)
1=1
T (1 (T - ΣT
Tßk
- -T3b
A'=
ak - It- 11))
Tak:
-Tab
(45)
0
0
The first approximation is based on matrix Taylor expansion A, (l 1,...; 7' is relatively small.
Ei
RT ||||| 1, given matrix
20

---

Published as a conference paper at ICLR 2026
The second approximation is based the fact that n.(t .,..., ,)) is relatively small quantity, considering that the total epoch a large number. Therefore, the difference matrix between A and A (i * j.ú,j = L, ..... T) will be close to zero, making any two matrices A A (i # j, 1. 1...., 7) approximately commutative.
The third approximation is based the approximation 1479%
7)h, given N small quantity. Equation (44) and (45) indicates that one can derive solution of UT),), which only related to E Therefore, one can come to the conclusion that (i) solver uncertainty at the final epoch 7 is approximately the same for cross-imrrovement with Σ d 1, and (ii) with the elements in matrix A' that y appears in both negatively correlated to the term ? Σi=1 Tt, the final solver uncertainty is also negatively correlated to this term. This indicates that eross-improvement with y 0, E Tht outperforms self-improvement with ? 0, Ei=1 ?e O, in terms of solver capability.
C THE USE OF LARGE LANGUAGE MODELS
We utilize Large Language Models (LLMs) to enhance the clarity, conciseness, and grammatical accuracy of the paper. The model's 's suggestions help refine sentence structure and improve the overall readability of the paper.
D EXPERIMENTAL DETAILS
In this section, detail our experiment hyperparameters and present all results of experiment. In Appendix D.I we detail the hyperparameters chosen in our experiments, We provide all experiment results in Appendix D.2 All of our experiments are run 80G NVIDIA A800 GPUs.
D.1 HYPERPARAMETERS
In this section, we detail the hyperparameters in Table
Table 3: Hyperparameters for SFT
Learning Rate
Weight Decay
Solver Temperature
Veritier Temperature
lc-5
0.01
0.1
Sample Size
Max New Tokens
LoRA Rank
LoRA dropout
16
512
16
0.5
D.2 OMITTED FIGURES
In this section, we will provide experiment details including setup and figures omitted in Section
Verification Methods. We use verification methods TrueFalse and Quality Evaluation in the experiment. TrueFalse (TF): The solver generates N responses, denoted Di,1, ... Di.N. for prompt Jy The verifier then tasked with answering whether each response Di.5 is correct. If the verifier deems a response Di.j correct, its score s(Jt,) ) is set to 1; otherwise, it is set to 0: Quality Evaluation (QE): The solver generates N responses, Vs.1. gi.N. for prompt x;. The verifier then assigns continuous score s(jt,)) between 0 and to each response based on its quality. A score of s(ùr.) 0 indicates completely incorrect answer, while sfr,,) indicates a completely correct answer.
Self Improvement. In this part, we display figures omitted in Section In self-improvement, we employed mini-batch gradient descent with a batch size of 256. For each model-task pair, we conducted training for 10 epochs, saving checkpoint after processing half of the training data. We test the solver accuracy and the verifier accuracy, defined as the accuracy of the response and the Phi-3.5-miin
BoN response, respectively. Figure 10 iiillustrates the results of self-im improvement Phi-3-min and Llama-3.2-33 3.2-3BB model with QE method. We observe that most of model-task pairs have similar results with the Phi-4-mini model except for several pairs. We also present the results with TF
21

---

Published as a conference paper at ICLR 2026
method in Figure When self- self- improving the Phi-3.5-mini and Phi-3-mini models Math data set, we observe that accuracy and uncertainty decrease simultaneously after several training epochs. The reason for this phenomenon might be LLM misleading by incorrect responses, the BoN response may correspond to a incorrect answer with low uncertainty. The solution of this problem could be future direction. Additionally, we note that the verifier accuracy of Llama-3.2-3B decreases as training progresses and is lower than the solver accuracy after 8 epochs of training. One possible reason for this phenomenon is that we use length-regularizee log-likelihood obtain the BoN responses, so the BoN responses tend to be longer responses. Long responses are more likely to contain repeated content, which may reduce accuracy. In addition, we present the self-improvement results on ProntoQA dataset with TF method Figure 12
GSM8K
Validation. this part, we present the validation results of Phi-3.5-mini, Phi-3-mini and Llama-3.2 3B on train split in Figure[14]an observe that the empirical data are strongly fitted to the exponential model for all models. Results of Phi-4-mini, Phi-3.5-mini, Phi-3-mini and Llama-3.2-3B on test split
presented in Figure 15 Experiment results ProntoQA dataset are presented in Figure [6
Self Evaluation. This part we provide the results of self-evaluation for different sample size N. For each model, we evaluate the accuracy and uncertainty of both the solver and the verifier, varying N across the values 2, 4, 8, 16, 32, 64. Figure [8 illustrates the accuracy and uncertainty of Phi-4-mini, Phi-3.5-mini, Phi-3-mini, Llama-3.2 3.2ma- 3B and Llama-3.1-8B on Math and GSM8k with different sample size using TFF and QE respectively, which shows that the verifier outperforms solver in all model-task pairs.
Cross Evaluation, In cross evaluation we set = 16 because, as demonstrated in Figure 18 the accuracy improve slightly for N 16. The results of QE method are presented in Figure [2] while results of TF method are presented in Figure 22 We observe that when a fixed model serves as the solver, the verifier's performance generally surpasses that of the solver, even when a different model is employed as the verifier.
Pass @ K. We investigate the underlying reason for the limit of self-improvement, focusing on how a decrease in the model's response diversity leads to diminishing potential energy, thereby causing the model's capability to plateau. Although the self eeff-improvement paradigm shows great improvement in model capability, it iteratively leads to a performance plateau. To investigate the underlying cause of this saturation, we conduct the Pass @ K experiment on the initial model and the self-improved model. Pass @K K is metric that calculates the proportion of prompts where at least one of the K responses is correct. When K' is large, Pass @K could be used to measure the diversity of the solver. We present the result in Figure 23 We observe that when K is small, Pass @ K increases with the number of epochs of seff-improvement, validating the self-improvement process. However, when K large, we observe a slight decrease in Pass@ K, indicating that the diversity of the solver is reduced through self- seff-mmprovemen. The degradation in diversity is caused by the convergence to a certain response, which is potential reason for the limit of self-improvement, Lack of ProntoQA dataset in experiment as the Pass @K K of the QA problem is close to when K is large.
22

---

Published as a conference paper at ICLR 2026
Train Split
Math Test Split
GSMBK Train Split
GSMBK Tast Split
R² = 0.925
R² 0.857
R - 0.508
R² 0.775
dG 1
dG/ot
(a) Phi-3.5-mini
Math Split
Math
GSMBK Train Split
GSMBK Test Split
R = 0.967
Rf 0.934
R 0,981
0.974
16
dG/ot
dGidt
dGiott
-30
18
-35
-20
20
66
(b) Phi-3-mini
Math Train Split
Math Tast Split
GSMBK Train Split
GSMBK Test Split
0.956
0.900
0.904
0.869
-20
-20
d5/dt
dG/dt
-40
df/oft
-80
100
-320
300
303
102
200
323
320
300
(c)
Figure& 8: Validation of the linear relationship between the Uncertainty gap G' and its rate of change dG/di dt on Phi-3.5-mini, Phi-3-mini and Llama-3.2-3B with QE method. The scatter points represent empirical data from self- improveeeetttt on the Math and GSM8k datasets, while the solid lines show the linear regression fits.
Base- STST-SFT
Midtrain
Post-train
30
puit compect Col0 120-3r
Viitic SActe cocoonttes boundh BHAk
25
120
100
Loss
Las5
20
80
15
60
40
10
20
10
10
5
10
Epoch
Epoch
Epoch
Figure 9: Self-improvement dynamics across Base, Mid-trained, and Post-trained stages test split.
23

---

Published as a conference paper at ICLR 2026
Math Accuracy
Math Uncertainty
GSMBK Accuracy
GSMBK Uncertainty
120
100
Solver
Veriteer Trann
80
Verifer Trin
Solver
Accuracy
0.3
Uncortaincy
80
Verifer Test
Accurag
0.80
Uncertainty
60
Verffer Test
0.2
Salver Thain
0.75
Salver Train
40
Solver Test
40-
Solver Test
Verifler
Vertfler
20
0.1
5
10
5
10
Epoch
Epoch
Epoch
Epoch
(a) Phi-3.5-mini
Math
Math Uncertainty
GSMBK Accuracy
GSMBK Uncertainty
0.40
150
Solver Test
Solver Tral
0.8
150
Solver lest
Solver Tain
Vertter Train
Vertter Train
Accuracy
0.35
Uncertiitty
Verifier leat
Accuracy
Uncrertainty
Verifer lest
100
0.30
Sclver
Solver Train
50-
0.6
Silver Test
Selver Train
50
Vertfir
Verifer
0.25
5
10
10
Epoch
Epoch
Epoch
Epoch
(b) Phi-3-mini
Math Accuracy
Uncertainty
GSMBK Accuracy
GSMBk Uncertainty
0.6
1.0
Solver Train
500
Solver Tain
0.5
Vertfir Tain
Salver Test
0.9
Uncortainty
Varitir Tain
Salver Test
Accuracy
0.4
400
Verifier Test
Accuracy
0.8
300
Veriter
0.7
0.3
200
Solver Train
200
0.6
Solver Train
Solver
0.2
0.5
Verifler
Vedifer Test
0.1
0.4
5 Epoch
Epoch
Epoch
Epoch
(c) Llama-3.2-3B
Figure 10: Accuracy and uncertainty during the aef-immprveemet of the Phi-3.5-mini, Phi-3-mini and Llama-3.2-3B on the Math and GSM8k datasets using the QE method.
24

---

Published as a conference paper at ICLR 2026
Math- Accuracy
Math- Uncertainty
GSMBK Accuracy
Uncertainty
0.45
200
Solver Tist
Salver Train
0.90
100
Solver Ret
Solver Tain
Vertfer Train
0.85
Vertfer Trah
Accuraty
Uncertainty
150
Accuracy
Uncertainty
80
Verffer Test
0.40
0.80
100
0.35
Solver Thain
0.75
Silour Trein
40-
50
0.30
Verflirr Text
0.70
the
Epoch
Epach
Epach
Epoch
(a) Phi-4-mini
Math- Accuracy
Math- Uncertainty
GSMBK- Accuracy
GSMBk Uncertainty
0.4
120
Salver Thain
Salver Thain
100
Verifier Train
Solver Test
0.85
80
Solver Test
Ver firf Fain
Accuracy
0.3
80
Verifer Test
Accuracy
0.80
Uncertainty
60
Verfier Test
0.2
Solver Thain
0.75
Solver Train
40
40
Salver
20
Verifiee
20
Velifier Jest
0.1
5
5
Epoch
Epoch
Epoch
Epoch
(b) Phi-3.5-mini
Math- Accuracy
Math- Uncertainty
GSMBK Accuracy
GSMBK Uncertainty
0.40
Soloer Test
Solver Train
0.8
150
Sober Tran
Snlver Best
Vertter Trai
Vertfer Train
Accuracy
0.35
Uncertainty
100
Veriter Test
Accuracy
0.7
Uncertainty
100
Vertfer Test
0.30
Salver
Solver Train
50
Selver Train
Silver Test
50-
0.25
Verifier
0.5
5 Epoch
10
10
Epoch
Epoch
Epoch
(c) Phi-3-mini
Math- Accuracy
Math Uncertainty
GSMBK Accuracy
GSMBk Uncertainty
0.6
1.0
Solver Irain
500
Solver
Accuracy
0.5
0.4
Uncortainty
500
400
Verifier Test
Verifler Thain
Salver Test
Accuracy
0.8
0.9
Uncerfainty
400
Solvor Test
Ver fiur Test
Verfier Prai
300
0.7
0.3
Solver Train
200
0.6
Solver Train
200
Salver Twel
Solver Ted
0.2
100
0.5
Werifise
100
0.1
0.4
5 Epoch
Epoch
Epoch
Epach
(d) Llama-3.2-3B
Figure 11: Accuracy and uncertainty during the self-im eel--imprvveeett of the Phi-4-mini, Phi-3.5-mini, Phi-3-mini and Llama-3.2 3.2-BB on the Math and GSMSk datasets using the TF method.
25

---

Published as a conference paper at ICLR 2026
Phi-4-mini
Phi-3.5-mini
Phi-3-mini
Llama-3.2-30
Tan
Tran Soheer
Tel Soher
0975
0.975
Solver
2.55
Sare
0.5
0.50
0.959
Tht Verfer
Tex: vatter
0.00
Tex Verder
2.00
Fas Soler
Salvr
0.75
Tex: Verter
0.075
2.50
0.75
0.000
0.053
0.70
0.825
0.025
3.75
0.65
10
140
saterre vater vether
ber saver
360
Srd Silver
250-
Uncertaitty
Uncertainty
100
Unceraiisty
1.50
600-
100
03
400
100
50
200
20
Spoch
Figure 12: Accuracy and uncertainty during the self-improvement of the Phi-4-mini, Phi-3.5-mini,
Phi-3-mini and Llama-3.2-3B 3.2-3B on ProntoQA datset using the TFF method.
Phi-4-mini
Phi-3,5-mini
Phi-3-mini
Llama-3.2-33
0.375
2.45
0.300
0.40
0.25
0.40
0.35
0.203
Tan
Tea ded
86
Saber lactscker
Tra Sebwr Indtsver
Trak SeecSolver
lact Serve
60-
Uncerailty
5c-
Uncertanty
Uncertanty
Uncertainty
4:00
46-
36-
40
103
260
26-
20
16
Epoch
Epech
Epoch
Figure 13: Accuracy and uncertainty during the self-improvement of the Phi-4-mini, Phi-3.5-mini,
Phi-3-mini and Llama-3. 3.2-3B 2-3B on MBPP datset using the TF method.
26

---

Published as a conference paper at ICLR 2026
Math(QE)
(SE)
Math(TF)
(MMRR(TP)
120
122
302
Matker cnisderyy
Wrlter ancrneryy
130
60
Rutker sooumary
Uncertainty
80
Giarlaity Ray Sonu
Uncertainty
Usua only Cara Bhic
Uncertainty
42
Luo dal De
Uncertainty
Bncolatyy Ber
5g
60-
22-
20
20-
26-
20
16-
10
10
Epoch
Epcch
Epoch
Epoch
(a) Phi-3.5-mini
Math(QE)
(SEB
Mathi(F)
GSMBK(TF)
Scaer sOCUIIIr
120
Sanver Liversing
Sawer
120
Lorter seurrdary
310
ceder prompmy
122
200
Vorder Unceedery 1TE
Uecortaints
Uncertainty
40
60
60
32
60
Epoch
Epoch
Epoch
Epoch
(b) Phi-3-mini
MathiQE)
(SEB
MathiTF)
GSMBkITF)
540
Broeitatty 2URL
Salve Ohertarng Cane Sot.
532
500
Vender
Uncertainty
100
Uecendi Dove (Vic.-0) 0911
by beb
Uncertainty
200
E0C-
Car duve SP
Uncertaintyy
200
ouly
Caver (1C)
0:00
222
200
200
2000
100-
0
2
Epoch
Epoch
Epoch
Epoch
(c) 1lama-3.2-3B
Figure 14: Uncertainty curve versus epochs of Phi-3.5-mini, Phi-3-mini and 1lama-3.2-3B models using QE and TF methods Math and GSM8k datasets train split.
27

---

Published as a conference paper at ICLR 2026
Math(QE)
(SMS
MatNTF)
GSMBK(TF)
200
272
Soler decuuciiyy VP4
300
Saler levosiey' Can (1A4010)
172
200
Start lecosiin Cave
Uncertainty
130
Leceudiiy der
(1ra 071
Uncortainty
20
Brumpieyy Car
Cave
155
125
Beundiii Cap PPasti
Unvertaints
00
daay Cay
20
10
Epoch
Epoch
Epoch
Epoch
(a) Phi-4-mini
Math(QE)
(QE)
MathiTF)
GSMBK(TF)
300
craster' 9
Witer tourtelyy the pm
Ber Dnala Co1
Rutter sovetarry
Sarve 20-3
Uncortainty
Uncortainty
Uncortainty
42-
20
16
30
Epech
Epoch
Epoch
Epoch
(b) Phi-3.5-mini
Math(QE)
GSMB
MathiTF)
GSMB//TF)
140
140
320
decttery cares p0
bo Welettrr davere HOUREE
Uncortainny
200
Uecoraiity Cap
Uncertainty
132
Unsertalnty
60
Uiber dity Cae
22
60
Epach
Epoch
Epoch
Epoch
(c) Phi-3-mini
Math(QE)
(QE)
MathiTF)
GSMBKITF)
500
Vedier cenerntiy
sedier comartty
532
Vorter Lnseeiiry
EEC-
400
incertaanty
lecendiiyy Cap (Vrl (a1)
Exeeadiyy Car Cove
422
deeecting Car gPaaste
Uncertainty
Lime /nly Dra (7a0
200
022
200
200
200
222
200
300
200
2
10
20
20
Epoch
Epoch
Epoch
Epoch
(d) 1lamm-3.2-3B
Figure 15: Uncertainty curve versus epochs of Phi-4-mini, Phi-3.5-mini, Phi-3-mini and Llama-3.2. 3B models using QE and TF methods on Math and GSM8k datasets test split.
28

---

Published as a conference paper at ICLR 2026
Phi-4 mini
Phi-3. 3.5 mini
Phi-3-miin
Liama- 3 3-2-38
220
175
200
370
00
250-
922
125
220
220
-22
220
50
30
30
Epoch
Epoch
Epoch
Epach
(a) Train Split
Phi-3-mini
220
175
320-
01C
150
Unsortairra
230
Uncettarry
125
220
1:0
332
-0-
330-
232-
25
Eooch
Epoch
Epoch
(b) Test Split
Figure 16: Uncertainty curve versus epochs of Phi-4-mini, Phi-3.5-mini, Phi-3-mini and Llama-3 -.22- 3B models using TF method on ProntoQA datasets.
Phi-4-mini
PN- 3.5 mini
Phi- 1-i-3minni
Llama-3. 3.2-38
150
325
320
220
20-
12
Spoch
spach
Epoch
tpoch
(a) Train Split
Phi-4-mini
3.5 mini
Phi-3-mini
Liama-
225
200
52
Uncorslittt
Uncorsairyy
42
320
80
32-
20
220
20
36
50-
130-
Epach
Spoch
Esoch
(b) Test Split
Figure 17: Uncertainty curve versus epochs of Phi-4 Phi-4-mini, Phi-3.5-mint Phi-3-mini and Llama-3 3.22- 3B models using TF method on MBPP datasets.
29

---

Published as a conference paper at ICLR 2026
Math Accuracy
Math Uncertainty
GSM8K Accuracy
Uncertainty
0.45
200
0.85
100
Accuracy
0.40
Solver
Uncertainty
150
TF Verffir
TF Solver
OE Solver
Accuracy
0.80
Salver
Uncertainty
80
TF Solver
TF Verffir
QE Solver
Verifier
QE Splver
Verifier
0.35
-a- Verilier
GF Verifir
60
100
0.75
40
0.30
20
40
60
0
20
40
60
20
40
60
0
20
40
60
Sample 5i2e
Sample Size
Sample Size
Sample Size
(a) Phi-4-mini
Math- Accuracy
Math- Uncertainty
GSMBK Accuracy
GSM8K Uncertainty
120
0.34
0,800
Accuracy
Uncertaitty
100
-a- TF Ver
TF Salver
Accuracy
0.775
Uncertainty
TF Vertterr
TF Solver
Solver Varifer
QE Salver
0.30
GE Vertfer
Solver
80
0.750
OE Veritee
Solver
60
Verifler
0.28
0.725
40
0.26
0.700
0
20
40
60
0
20
40
60
20
40
60
0
20
40
60
Sample Size
Sample Size
Sample Size
Sample Size
(b) Phi- miimmi mini
Math Accuracy
Math Uncertainty
GSMBK - Accuracy
GSMBK - Uncertainty
160
150
0.75
Accuracy
0.350
0.325
Verffer
incortaiitt
140
120
TF Vertier
0f Solo
I1 Solver
Aceuracy
0.70
Vertfier
Uncerfainty
100
125
TF Verther
Solver
Silver
0.300
GF Vertter
Solve
100
Vertter
QE Sclver
Of Vertflr
Verifeer
0.60
0.275
80
50
0.250
60
20
40
60
60
20
40
60
0
20
40
60
Sample Size
Sample Size
Sample Size
Sample Size
(c) -iiiiimini
Math- Accuracy
Math- Uncertainty
GSM8K Accuracy
GSMBk Uncertainty
0.9
500
500
Accuracy
0.45
Sover Solver
Uncertaint
400
TF Verifier
OE Solver
TF Solver
Accuracy
0.8
Salver
300
400
TF Solver
TF Veitler
QE Solver
300
Veriter
Solver
Veritter
0.40
Verifier
GF Verifer
200
200
0.35
100
100
0.6
20
40
60
0
20
40
60
20
40
60
0
20
40
60
Sample Size
Sample Size
Sample Size
Sample Size
(d) Llama-3.2-3B
Math Accuracy
Math Uncertainty
GSMBk Accuracy
GSMBK - Uncertainty
0.475
0.450
600
800
Arcuracy
0.425
Salver Solver
Uncertainty
TF Solver
TF Verilin
Accuracy
0.85
Uncerfainty
600
TF Vellfer
Solver
QE Sohrer
Silver
0.400
Veriier
Of Veither
0.80
QE Solver
DE Verifier
400
Verither
0.375
0.350
200
0.75
200
20
40
60
60
40 Sample Size
20
60
0
20
40
60
Sample Size
Sample Size
Sample Size
(c) Llama-3. 3.1--BB..1-BB
Figure 18: Accuracy and uncertainty of Phi-4-mini, Phi-3.5-mini, Phi-3-mini, Llama-3.2 3.2-3B and
Llama-3.1 3.1-BB on Math and GSM8k across different sample size using TF and QE respectively.
30

---

Published as a conference paper at ICLR 2026
Phi-4-mini
Phi-3.5-mini
Phi-3-mini
Liama-3.2-38
0.85
0.55
0.65
9.65
0.90
0.60
2.50
Verfeer
0.80
23
20
40
03
40
50
40
50
300
152
230
Uncertartt
Unceraaity ngy
Uncertarity
150
velver Ver
500
300
25
400
50-
100
200
50
69
69
60
60
Sample
Sample
Sample
Figure 19: Accuracy and uncertainty of Phi-4-mini, Phi-3.5-mini, Phi-3-mini and Llama-3.2-3B on ProntoQA dataset across different sample size using TF method.
Phi-4-mini
Phi-3.5-mini
Phi-3-mini
Llama-3.2-30
0.55
0.50
5.20
5.16
Solver
0.60
0.12
0.24
0.30
0.30
5.08
0.25
0.25
0.30
80
102
250
3200
72-
200
1000
Uncaraino
50
Unceraiinty
60-
Uncartaiity
1.50
400
90
400
20
10
60
60
Sample Size
Sample Size
Sample Size
Sample
Figure 20: Accuracy and uncertainty of Phi-4-mini, Phi-3.5-mini, Phi-3-mini and 1ama-3.2-3B MBPP dataset across different sample size using TF method.
31

---

Published as a conference paper at ICLR 2026
Accuracy (Verifier Solver)
Log L0Uncertainyy Difference)
Llama-3.1-80
0.086
Llama-3.1-8B
2.11
Llama-3.2-3B
0.366
0.350
0.163
2.133
6.138
Llama-32-38
2.61
2.63
2.64
2.43
2A1
Solver
Solver
Phi-3
0.209
0.101
0.106
Phi-3-mini
1.99
1.59
22
Phi-3.5-mini
0.077
0.074
Phi-3.5-minn
Phi-4-mini
0.095
0. 004
0.305
3.00
0.125
Phi-4-mini
2.09
2:11
2.10
2.10
247
Uama- 3. 1-88
Liema Llema-332233 2.38 3.2.38
Phi-3-mini
Phi-3 S-mini
Phi-4-minl
Llama-3 1-80
Llama-3 2:38
Phi-3-minl
Phi 5-mini
Phi-4-mini
Verifier
Verifier
(a)
Accuracy (Verifier Solver)
GSM8k: Log1o(Uncertainty Difference)
Llama-3.1-80
0.222
0.112
0.103
Laam-331-8B
2.91
Llama-3.2-3B
0:270
0.274
0.264
3.62
6.079
Llama-3.2-3B
2.61
2.67
2.64
2.63
2:47
24
Solver
Solver
Phi-3
0.15
Phi-3-mini
Phi-3.5-mini
0.026
0.079
0.087
Phi-3.5-mini
1.64
1.65
1.67
10
Phi-4-mini
0.126
0.129
0.125
9112 Phi-4-mini
0.126
0.05
Phi-4-mini
1.82
1.8
1.83
182
178
-4
Ulmma-3 1-88
Llama- 3:338
Phi-3-mini
Phi-a S-mini
Llama-3 1-88
Llama-3.2-3p 2:38
Phi-3-mini
Phi S-mini
Phi-4-mini
Verifler
Verifier
(b)
Figure 21: (a) Cross-evaluation using QE method with sample size N=16 on Math dataset. (b)
Cross-evaluation using QE method with sample size N=16 GSM8k dataset. For each solver model, sampled responses are verified by different models.
32

---

Published as a conference paper at ICLR 2026
Math: Accuracy (Verifier Solver)
Math: Log 10(nneeraaiyy Difference)
Llama-3.1-80
0.095
0.062
0.081
0.084
0.096
Llama-31-888
2.7L
2.67
2.63
2.70
2.71
Llama-3.2-38
0.257
0.239
0,152
5.105
0.150
Llama-3.2-3
2.63
2.60
2.41
24
Solver
6.12
Solver
Phi-3-mini
0.110
0.009
0:111
Phi-3-mini
1.06
196
Phi-3.5-mini
0.079
0.284
6,082
2.077
0.079
Phi-3.5-mine
2.74
1.73
1.74
1:24
1.74
10
0.00
Phi-4-mini
0.120
0.332
6.130
9.13
6.103
Phi-4-mini
2:12
2:12
2.00
Liama-33 1--8
Liama -ama--... 2.38
Phi-3-mini
Phi-3 S-mini
Phi-4- Phi-4-mini
Llama-3 1-88
Liama 2-38
Phi mni-3-mini
Phi-a 5-min
Phi Phi-4-mini
Verifier
Verifier
(a)
GSMBk: Accuracy (Verifier Solver)
GSMBK: Log10(Uncerteinty Difference)
2.19
Llama-3.1-8
B:322
3.093
6.111
0.105
0.110
Llama-3,1-88
2.90
2.08
2.90
2.90
2.91
3:250
Llama-3.2-38
0.270
0.260
5.225
Llama-3.2-38
2.68
2.67
Solver
solver
Phi-3-mini
0.216
0.227
Phi-3-mini
2.04
2.04
12
Phi-3.5-mini
0.0MS
2.095
0.098
0.092
0.005
2.150
Phi-3.5-mini
187
1.67
1.68
1.M
1.57
10
5.125
Phi-4-mini
-1
Liama 1--8
Liama-33 3.2-38 2.38
Phi-3 S-mini
Phil--mmi 4-mini
Llama-3. 3.10-3.11-88 1-88
Liame 3 2.38
Phi-3 Phi-3-mini 3-mii
Phi- 5-mini
Phi Phi-4-mini
Phi-3-min
Verifier
Verifier
(b)
ProntoQA: Accuracy (Verifier- Solver)
ProntoQA: Logl0(Uncertainty Difference)
Llama-3.1-8B
.0.728
6.115
5.099
0.0%
5.118
Llama-3.1-8B
2.M
2.35
2.95
2.45
2.97
13
Llama-3.2-38
0.144
0.066
Llama-3.2-38
2.50
2.59
2.89
Solver
0.75
Solver
Phi-3-mini
0.130
0.132
0.044
Phi-3-mini
2.35
6,500
Phi-3.5-mini
0.069
0.070
0.066
0.03
0.074
6,25
Phi-3,5-mini
1.87
1.87
1.86
186
1.87
12
5.20
Phi-4-mini
Phi-4-mini
Llama-3.144 3.1-88 1-88
Llama-3.2-32 3.3.38..8 38
Phi-3 Phi-3-mini
Phi-3 S-mini
Phi-4-mini
Phi-4 mini
0.025
Llama-3. 3.1 1-08
Uama-332.38 2.2.8
Phi-3 --mi
Phi-3 Svmini
Phi-4-mini
Verifier
Verifier
(C)
Figure 22: (a) Cross-evaluation using TF method with sample size N=16 on Math dataset. (b) Cross-evaluation using TF method with sample size N=16 on GSM8k dataset. (c) Cross-evaluation using TF method with sample size N=16 on ProntoQA dataset. For each solver model, sampled responses are verified by different models. Figures on the left illustrate the difference of accuracy while figures the right show the 10 logarithms of the uncertainty difference.
33

---

Published as a conference paper at ICLR 2026
Phi-é-min-instruct Math
--stntiinmin Dataset
GSMBK Dataset
GSMBK Dataset
3.00
P
2-6
PasSiKK
0.5
0.03
0.00
0.83
6:75
0.75
0.70
Phi-3-mini-4kkinsteee Math Dataset
GSMBK Dataset
Math Datasse
GSMBk Dataset
1.0
0.0
PassiKK
3.6
PassIK
0.85
0.6
0.5
0.61
(3m)
(3b)
(4a)
(46)
61
Epochs
Epochs
(a) TF
Phi-4-mini-instruct Math Dataset
Phi-3.5 m...mmiiniiiiirrccctt Math Dataset
GSMBk Dataset
GSMBk Dataset
2.7
0.02
0.95
Pess@K
0:00
PassaK
0.93
0.82
0.87
0.80
0.83
24
0.75
0.3
0.75
(1a)
(1b]
12a)
(26)
0.70
Phi-3-mini-4kkinsstruus Math Dataset
Llama-3.2-3BIInstee Math Datacet
GSM8K
GSMBK Dataset
1.00
0.7
Passkk
Passgi
0.2
PasseK
0.80
0.6
0.61
(3g)
13b]
(4a)
(46)
52
64
26
64
10
16
Initial
Epochs
10 Epochs
(b) QE
Figure 23; Pass B K with TF and QE method for different K at 0, and 10. Pass@K K evaluates the diversity of model generation. The efficacy of the self-improvement process is demon- strated by an increase in the Pass@K K metric for small values of K. Conversely, for large values of K, a decrease in Pass @ K is observed. This phenomenon suggests that the diversity of generations decreases during the self-improvement process.
34

---

Published as a conference paper at ICLR 2026
E ADDITIONAL DISCUSSION
Phenomenological results. We acknowledge that neural network optimization is complex. However, our approach is phenomenological akin to how thermodynamics describes complex molecular systems using macroscopic variables. Our core assumption, the linear approximation of potential energy is empirically validated. As shown in Figure 2 and Figure 7, the relationship between the gap G and its rate of change dG/ dt. is strictly linear, with R² > 0.9. Based on linear assumption, exponential convergence law (Eq 10& & Eq 11) fits the training trajectories shown in Figure 4, with R > 0.9. Although deriving this relationship from first principles remains an open challenge, our framework yields valuable insights into the self-improvement mechanism, it provides a precise mathematical characterization of the optimization curve and a rigorous basis for modeling cross-improvement strategies.
Simplified assumptions (linear and time-invariant). We clarify that linearity is inspired by empirical observations, not just a prior assumption, and the time- inverinvriinnce is a local approximation. Figures 2 and 7 plot the dG /tt against G(t), revealing a linear relationship between the potential energy E(t) and gap G(t) across various models.
* Model Complexity and Parameter Estimation. Modeling the coefficients (a, 5) as time- varying functions would significantly increase the complexity of the theoretical framework. Introducing time-varying parameters would result in complex modeling processes and make parameter estimation nearly impossible without introducing further arbitrary assumptions. Given that our constant-coefficiert model already achieves high degree of precision (with R² > 0.9 in fitting training dynamics), suggesting the time-invariant assumption is the most effective approach for capturing the macroscopic dynamics.
TiieeHHomggende of the State. In our framework, the dynamics are driven by the state of the system (the Solver- Solver-Verifier Gap) rather than explicit time. This implies time- homogeneity: any time point t in the training process can be regarded as a start point. The evolution of the system depends on the current capability gap Gtt) and the potential energy E(t). This is empirically supported by Figure 2, which shows a consistent linear relationship between the gap and its rate of change throughout the training process, confirming that the parameters governing this relationship remain stable.
Applicability of Time- Varying Parameters. We acknowledge that the time-invariant assumption is local approximation valid for the standard self-improvement regimes studied in this paper. In specific complex scenarios, such as training across radically different data distributions or using highly dynamic learning rate schedules, the coefficients might drift. We have identified relaxing the time-invariant property as a specific direction (e.g., piece-wise constant) for future work to extend the framework's generality.
Connection between theoretical variables and practical LLM learning signals. The variables in our phenomenological model map directly to concrete, measurable quantities in the LLM training process:
The capability gap G(I): As defined in (6. U,)) This represents the "Quality Delta" between the model's current generation performance U.. and the quality of the verification output U.. In practice, this measured by comparing the average NLL of the model's outputs against the average NLL of the Best-of-N responses selected for training.
The potential energy E(t): E(t) is function of the gap, approximated linearly E()) kG(l) - b. This represents the power which drives self-improvemmen. Equation states du.it) E(t. This means E(t) maps directly to the speed of improvement of the di
model. In a practical training run, high E(t) indicates that the gap large enough to provide strong gradients, leading rapid drop in loss. A low E(t) indicates the model has exhausted the information the current gap, leading to a training plateau.
Therefore, monitoring Ett and G(t) might be equivalent to monitoring the efficiency of the data relative to the model's current state.
35

---

Published as a conference paper at ICLR 2026
Validations of cross-improvement (Equation 12). We justify its validity from both theoretical and empirical perspectives:
Theoretical Rationality: Equation 12 models the capability enhancement as gain factor mechanism. The term (1+77))) acts as a discount factor on the uncertainty. As the ratio of high-quality external data 7/ increases, the verifier's uncertainty U, decreases monotonically, This phenomenological formula captures the intuition that external data reduce the system's ' entropy, with y representing the efficiency of this external information. From a technical standpoint, we explicitly chose this functional form for its mathematical tractability. This specific inverse-linear form allows the coupled dynamics to be approximated by linear matrix operations. If we were to use more complex non-linear scaling laws, the system would become analytically intractable, preventing the derivation of the closed-fone approximate solution in Proposition 5.1. This form strikes necessary balance between capturing physical intuition and maintaining solvability.
Crucially, the validity of this modeling assumption is supported by our experimental results. Based on Eq. we derived Proposition 5.1, which predicts that the final solver capability depends primarily on the total amount of external data, rather than its specific allocation timing. Our experiments in Section 5.2 (Table 1) confirm this conclusion. We observed that different allocation strategies (Early, Uniform, Late) yielded negligible differences in final accuracy. This alignment between the theoretical prediction derived from Eq.12 12 and the empirical outcome strongly validates the reasonableness of the model.
Empirical Validation:
Results and difference of model families. Different model families exhibit distinct exponential trajectories because their inherent properties (such pre-training data quality and architectural maturity) determine the specific values of the decay rate coefficient and the boundary conditions in our derived analytical solution. As discussed in Q1, Llama families exhibit larger solver learning rate a. Mathematically. a larger a maximizes the exponent magnitude steeper decay curve (rapid convergence), as observed in our Llama experiments. Phi families are pre-trained on high-quality synthetic data. This results in a smaller gap and decay constant, leading to flatter decay function. Crucially, the decay function is mathematically coupled to the limit. The theoretical asymptote is defined as Us = U.o -لك (Go b)
I((a - B) resulting in
Phi Family: Starts with a significantly smaller Initial Gap (Go) and lower initial uncertainty (U,.0) due to high-quality upstream synthesis. Plugging these smaller values into the equation mathematcallly dictates a better final capability.
Llama Family: Starts with larger Go and Us.0- Despite the rapid initial descent, the higher starting baseline mathematically leads to convergence at a relatively higher asymptote.
Potential energy as driven force. We justify modeling the training dynamics as Equation 9
Physical Analogy: Potential Difference as the Driving Force. In this paper, we model the training dynamics as a form of dU, Newton's 's Second Law. In Newton's Second Law (F = ma), force is the cause that generates acceleration. Without force, there is no change in the state of motion. Similarly, in our framework, the Potential Energy E(t acts as the force which is the cause that drives the evolution of the model's 's capabilities (U,, U, (inspired by Song et al. (2025)). Just as a physical force dictates how an object moves, the magnitude of this potential energy dictates the intensity of the self- iel--improvement process. This modeling captures the core intuition: the existence of a gap creates a potential difference that physically necessitates and drives the optimization process.
dU, -t),,,, which can be analogous di (t), dt
Empirical Validation.
Despite the simplicity of our model, our results demonstrate that this simple macroscopic model captures the training dynamics with remarkable precision. We fit the theoretical exponential curves derived from our conservative field model to the actual training trajectories. As shown in Figure 4, despite the model's simplicity. it fits the empirical training data (both Math and GSM8k datasets) exceptionally well, with coefficients of determination ( (R22) consistently exceeding 0.9.
36

---

Published as a conference paper at ICLR 2026
Conservative vector field. To statistically verify whether historical information (G(t - 1)) provides any necessary explanatory power for the dynamics, we performed Partial F-test to compare two nested regression models:
Model 1 (Simple Model): AG BiG(() + G(t + (t 1) +
dt
Model 2 (Full Model): di
dG
We evaluated whether adding the historical term G(t 1) significantly reduces the Residual Sum of Squares (RSS). As shown in Table the insignificant F-test result leads us to fail to reject the null hypothesis (82 = 0). This statistically proves that adding historical information does not significantly improve the model's fit. The dynamics are adequately described by the current state alone, providing robust evidence for the memoryless nature of the potential energy.
Table 4: Model Selection via Partial F-test.
Model
Dataset
Split
F-statistic
P-value (F)
Phi-4-mini
Math
Train
1.295
0.2687
Math
Test
1.203
0.2911
GSM8k
Train
0.024
0.8771
GSM8k
Test
9.277
0.0122
Llama-3.2-BB
Math
Train
0.479
0.5114
Math
Test
0.833
0.3917
GSM8k
Train
0.875
0.3808
GSMSk
Test
0.808
0.3985
Phi-3.5-mini
Math
Train
1.807
0,2209
Math
Test
3.218
0.1159
GSM8k
Train
1.945
0.2058
GSM8k
Test
3.140
0.1197
Phi-3-mini
Math
Train
0.000
0.9845
Math
Test
5.300
0.0548
GSMSk
Train
0.528
0.4911
GSM8k
Test
2.163
0.1848
37


Reviews:
[{"review_id": "88s4Sohx4q", "reviewer": "Reviewer_nJpU", "summary": "This paper develops a theoretical framework to explain how LLMs improve themselves without external data. It introduces the solver–verifier gap, where the solver represents generation ability and the verifier represents evaluation ability. Using coupled differential equations inspired by potential energy, the authors model the training dynamics and show that solver and verifier capabilities converge exponentially.", "questions": "See weakness above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. First clear formalization of LLM self-improvement dynamics through solver–verifier interactions.\n2. The differential-equation approach captures exponential convergence behavior that aligns well with empirical trends.\n3. Experiments across datasets and LLMs demonstrate strong fits (R² > 0.9) to the theoretical model, reinforcing its predictive power.\n4. The extension to limited external data regimes adds practical insight into data allocation strategies.", "weaknesses": "1. The model is phenomenological rather than derived from first principles; it lacks a deep mechanistic explanation of why solver-verifier dynamics follow this form.\n2. Assumes linear potential energy and time-invariant coefficients (α, β), which may not generalize to all LLM training settings.\n3. The connection between theoretical variables (E(t), G(t)) and practical LLM learning signals remains abstract.\n4. The paper omits discussion of computational cost, convergence rate sensitivity, and implications for large-scale training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QP4KskS38V", "reviewer": "Reviewer_ZsPe", "summary": "The paper studies the training dynamics during LLM self-improvement training. The paper uses the solver-verifier gap as the key concept for measuring the self-improvement dynamics.  The solver-verifier gap is defined by the log ratio of policy's probability over the best of N generation and the individual generations. The paper proposes to model the generator's capability change and solver's capability change as a differential equation, and thus the change of gap can be measured as the energy function. Experiment results are provided to support the theoretical models. The paper further studies the dynamics of cross-improvement.", "questions": "1. Although some experiments seem to suggest that $E(t)$ can be linear wrt $G(t)$, can the authors show if this is even possible in any toy settings, where we can even have the policy follows some bernoulli distribution parametrized by $p$, and we have some updating dynamics $d p(t)/dt$, can we say something that $E(t)$ is linear wrt $G(t)$?\n\n2. Can the authors explain why does eq 12. a reasoning model?\n\n3. In the final sentence of section 5.2, how did we reach the conclusion that the small difference in accuracy validates prop. 5.1?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper studies the important problem of modeling the LLM self-improvement process, which models a very complex system with simple differential equations. \n\n2. The paper provides experiments to verify the validity of the proposed models.", "weaknesses": "1. I had a difficult time even trying to understand the basic definitons such as solver uncertainty and verifier uncertainty. Are terms like $U_s(t)$ and $U_v(t)$ random variables, since $y_i$ are random variables? If so, how do we even understand these terms and so the gap term? How are they measured in practice?\n\n2. I don't see the necessity of formulating everything as differential equations as the LLM self-improvement update is discrete. \n\n3. The exeperiment setup seems unclear. For example, it is unclear to me what TF and QE methods are. \n\n4. The result of the gap narrows during self-improvement does not seem new to me; it appeared in many previous works already. \n\n5. The paper needs to highlight what are the significance of the current results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xG0R1XMAIE", "reviewer": "Reviewer_Sog4", "summary": "The paper investigates the training dynamics underlying the self-improvement process of large language models, aiming to uncover the mechanisms driving performance evolution without relying on external data. The authors propose a theoretical framework based on the solver–verifier gap, which explains that the essential source of self-improvement lies in the disparity between a model’s generation (solver) and self-evaluation (verifier) capabilities. Inspired by the potential energy framework in physics, the authors formulate a set of coupled differential equations to characterize how the solver and verifier capabilities evolve over time (i.e., training iterations), and derive that their capability growth follows an exponential convergence pattern.", "questions": "Please refer to the relevant points in the Weaknesses section. If the authors can provide clarification and improvements, I would be very happy to raise my score.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper provides an interesting and effective theoretical exposition of LLM self-improvement, accompanied by a relatively rigorous theoretical proof.\n\n2. It conducts diverse empirical studies on mathematical tasks across multiple models, which enhances the credibility of the results.", "weaknesses": "1. The relationship between model capability and output uncertainty is not clearly articulated. The connection between a model’s accuracy on different tasks, its output uncertainty, and its underlying “capability” remains ambiguous. Why can output uncertainty serve as a valid indicator of model capability? How is this metric related to output accuracy? Although the authors provide some explanation in the appendix, the paper lacks further theoretical and empirical justification for the appropriateness of this metric.\n\n2. The theoretical analysis relies on numerous and rather simplified assumptions. In particular, the energy-based assumptions regarding the training process may fail to capture the complex dynamics involved in model optimisation.\n\n3. The experiments are limited to mathematical reasoning tasks (GSM8K, Math) and lack broader validation on code generation or other reasoning benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cVufMNhBYT", "reviewer": "Reviewer_V9jn", "summary": "The paper develops a physics-inspired theoretical framework to explain how large language models (LLMs) improve themselves without external data. It introduces the concept of a solver-verifier gap—the difference between an LLM’s ability to generate solutions (solver) and its ability to evaluate them (verifier)—as the main driver of self-improvement. The authors model training dynamics using coupled differential equations, showing that solver and verifier capabilities evolve exponentially toward limits determined by their initial gap. Experiments on Phi and Llama models across multiple datasets validate these exponential laws and confirm that verifiers consistently outperform solvers. The paper further extends the framework to cross-improvement, where limited external data enhance verifier capability; theoretical and empirical results show that the timing of using such data has minimal impact on final performance. Overall, the study provides a unified quantitative model linking LLM self-improvement behavior, convergence limits, and external data effects", "questions": "How does the base model FLOPs (and families) affect the analysis? \n\nhow do models from different families have different exponential decay functions, which would lead to the modeling of solver and verifier uncertainties (or their gaps) toward asymptotic limits?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is highly original in formalizing LLM self-improvement through a physics-inspired solver-verifier gap framework, transforming an empirical phenomenon into a quantitative model. Its quality is strong, with clear theoretical derivations, well-controlled experiments, and convincing exponential fits that validate the framework across multiple models and datasets. The writing is clear and logically structured, effectively integrating intuition, math, and visualization. In significance, the work provides a foundational step toward understanding autonomous model improvement and offers practical insights for optimizing self- and cross-improvement strategies, making it both conceptually innovative and broadly impactful for future LLM research.", "weaknesses": "Some claims appear to be a bit strong - \"Self-improvement is among the most prominent techniques within the realm of ...\".\n\nI also advocate for using the term \"generation-verification\" gap to respect the originality.\n\nThe time-invariant assumption sounds a bit strong: this assumes that the key parameters governing training dynamics — such as the cross-improvement effect γ, and the solver/verifier rate coefficients α and β remain constant across all epochs. Some more ablations would be helpful to understand their effects.\n\nSome limiting behaviors are worthy of examination - such as how γ α and β would affect the convergence.\n\nIn EvoLM, model developers have found that base models that have undergone mid-training can be more easily adaptable using RL. The work should take into account such factors and see how this affects the self-improvement behaviors. Ideally, a controlled comparison using base, mid, post-trained checkpoints would be an insightful add. Because the field of post-training can heavily depend on upstream training, I am leaning towards advocating for strong accept if this confounder is taken into account.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)