Variational Reasoning for Language Models
Authors: Xiangxin Zhou
Zichen Liu
Haonan Wang
Chao Du
Min Lin
Chongxuan Li
Liang Wang
Tianyu Pang
Venue: ICLR 2026 Poster (2026)
Keywords: Language Models
Variational Reasoning
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a variational reasoning framework that treats thinking traces as latent variables optimized via variational inference, yielding a principled and stable training objective that improves LLM reasoning across diverse benchmarks.

Abstract: We introduce a **variational reasoning** framework for language models that treats thinking traces as latent variables and optimizes them through variational inference. Starting from the evidence lower bound (ELBO), we extend it to a multi-trace objective for tighter bounds and propose a forward-KL formulation that stabilizes the training of the variational posterior. We further show that rejection sampling finetuning and binary-reward RL, including GRPO, can be interpreted as local forward-KL objectives, where *an implicit weighting by model accuracy* naturally arises from the derivation and reveals a previously unnoticed bias toward easier questions. We empirically validate our method on the Qwen 2.5 and Qwen 3 model families across a wide range of reasoning tasks. Overall, our work provides a principled probabilistic perspective that unifies variational inference with RL-style methods and yields stable objectives for improving the reasoning ability of language models.

Published as a conference paper at ICLR 2026
VARIATIONAL REASONING FOR LANGUAGE MODELS
Xiangxin Zho Zichen Liu 14, Haonan Wang 144, Chao Du', Min Lin',
Chongxuan Li Liang Wang2.3 Tianyu Pang
'Sea AI Lab PCAS &CASIA $NUS
'RUC
Code Link
ABSTRACT
We introduce a variational reasoning framework for language models that treats thinking traces as latent variables and optimizes them through variational inference. Starting from the evidence lower bound (ELBO), we extend it to multi-trace ob- jective for tighter bounds and propose forward-KL formulation that stabilizes the training of the variational posterior. We further show that rejection sampling finetun- ing and binary-reward RI., including GRPO, can be interpreted as local forward- KI. objectives, where an implicit weighting by model accuracy naturally arises from the derivation and reveals a previously unnoticed bias toward easier questions. We empirically validate our method on the Qwen 2.5 and Qwen 3 model families across a wide range of reasoning tasks. Overall, our work provides a principled probabilistic perspective that unifies variational inference with RL-style methods and yields stable objectives for improving the reasoning ability of language models.
INTRODUCTION
Reasoning has recently become central focus for large language models (LLMs), driving advances in tasks such as mathematics, coding, and scientific problem solving (Jaech et al., 2024; Comanici et al., 2025; Guo et al., 2025). A common strategy is to let models generate explicit thinking traces before producing final answers. To train such reasoning abilities, two dominant approaches are widely used: supervised finetuning (SFT) (Guha al., 2025; Muennighoff al., 2025) and reinforcement learning (RL) (Yu et al., 2025a; Liu et al., 2025; Zeng et al., 2025), both showing strong empirical success.
Despite this progress, each approach faces limitations. SFT often relies on curated long- wingtthinking traces, which are costly to collect and, as an offline method, may struggle to generalize (Chu et al., 2025) or suffer from catastrophic forgetting (Shenfeld et al., 2025). Recent RL methods typically de- pend on verifiable rewards to mitigate reward hacking, yet training can be unstable and output diversity may collapse (Cheng et al., 2025; Cui et al., 2025b). As a result, correct answers to harder questions become increasingly rare, leading to lower Pass @ K accuracy than base models (Yue et al., 2025a). These challenges motivate the search for a more principled objective for training reasoning models.
To this end, we propose view reasoning through the lens of probabilistic modeling, where thinking traces are treated as latent variables. Variational inference (Kingma & Welling, 2013) provides a natural way to optimize the log-liiel of producing correct answers. This perspective offers several advantages: it replaces the intractable marginalization over thinking traces with tractable lower bounds, enables multi-trace extensions that tighten the objective, and introduces a variational posterior that can sample thinking paths more likely to yield correct answers. In this way, it provides a principled objective for training reasoning models, while remaining compatible with verifiable rewards.
Building on this perspective, we develop a variational reasoning framework for language models in Section 2. The core idea is to decompose reasoning into thinking trace and an answer, leading the maximum log- likelihood estimation (MLE) objective. To make this optimization tractable, we intro- duce an evidence lower bound (ELBO) and extend to an IWAE-style multi-trace trati--traceee formulation (Burda et al., 5), which tightens with more rollouts. To further stabilize the training of the variational posterior, we propose forward-Ke KI. objective that prevents collapse and makes better use of answer hints. Together, these components form a unified training pipeline (as shown in Algorithm 1) that jointly improves the reasoning model and the variational posterior.
Beyond the method itself, our framework also helps interpret existing approaches, described in Section 3. Rejection-sampling finetuning (RFT) (Dong et al., 2023; Touvron et al., 2023) can
EEuual contribution. $I *Correspondence Tianyu Pang.

---

Published as a conference paper at ICLR 2026
be re-expressed as forward-KL optimization weighted by model accuracy, and including GRPO (Shao et al., 2024), admits a similar form. Our analysis shows that this weighting by accuracy arises implicitly and produces systematic bias toward easier questions, not been explicitly recognized before. By placing these methods under a shared probabilistic view, framework provides principled objectives and clarifies the behavior of widely used methods.
RL,
effect that has
We validate our framework on the Qwen2.5 and Qwen3 model families (Yang et al., 2024; Team, 2025b) and observe consistent improvements over strong baselines across diverse reasoning bench- marks, including MATH500, AIME24&25, OlympiadBench, LiveCodeBench, GPQA-Diamond, and MMLU-Pro. Due space constraints, a detailed discussion of related work is deferred to Appendix
2 VARIATIONAL REASONING
Let V* denote the set of all prompt strings over the vocabulary V. Given input question E V*, a reasoning model no (z. y x) generates both a thinking process 2 E V and a predicted answer V". The joint probability can be written as (z y/z) To ( (z =) To (z(xx)). Following standard format template (Guo et al., 2025), these two conditional terms are computed as
Toe (=|æ) no </think>/
<think>]);:
(I)
Tp(y æ2, z) - To (ly, ( <//ss//sswr>) [xr <think>,, </think?, <answer>]). where </think> and </answer> serve as the end-of-sequenee markers for and y. respectively.
We define the marginal distribution (3yla)) = Ex TT(2. y/ Σx ( x) (2 (x), where the notation Pa (y(()) highlights that this distribution is induced by me. This is different from the non-thinking probability wo ()) which does not marginalize over possible thinking traces.
2.1 EVIDENCE LOWER BOUND
Let V2 C V denote the oracle set (possibly infinite) of correct answers to the question æ. The marginal probability that Te generates a correct answer Po (x) Eyey, Po(y)æ)) Maximizing this probability gives the maximum log liglikklihhodd estimation (MLE) objective: max, log Po (x) However, this MLE objective intractable because computing Pe (g)(z) requires summing over all possible thinking traces z. To make learning feasible, we apply variational inference (Kingma & Welling, 2013) derive an evidence lower bound (ELBO):
log Po(D (z) log Σ,
Te To lx, (=)x)=
To (J' ar, (z)
log Bgg (xo(
(0 x y¹)
2 (vy(v') [Egn(=le.r [log No To xx 2)] DrL. (4s (z (|
CELBO( Va. y2: To. To. 9o)
In this expression, Te (1 [x 2) denotes the probability of producing correct answer given the question æ and particular thinking trace z. The distribution go(2|æ; [ y²) is the variational posterior, which conditions not only the question æ but also auxiliary answer hint y':
(=(2,, y²) = g ([l,/////eaata </think>] [x, <hint>, y </hint>, <think>l).
(3)
Here, <hint> and </hint> are shown as example delimiters; in experiments, we ablate different special tokens to wrap the hint y' and concatenate it after Conditioning on y encourages the vari- ational posterior to generate thinking traces that are more likely to yield correct answers. A simple yet effective design choice is to let y' come directly from the oracle set, that is, supp(as(y") C Va- In practice, y may be a rephrasing of a reference answer or any correct expression sampled from Vi
We can further show (detailed in Appendix A. that maximizing the ELBO objective w.r.t. do in Eq. (2) is equivalent to minimizing the reverse KL divergence between qo (2 x, y²) and Po [z. V=)
LELBO x Ve, z/: Na. ) = log (2) (x) DxL (g0(2/2 y/ P (=) Ve)).
(4)
the true posterior. Compared with the prior distribution To( (zz), this posterior distribution re-weights thinking traces by To () ax, 2), thus favoring that are more likely to produce correct answers. According to Eq. (4), we know that the optimal solution
Here Po(z|æ (z(z,,,10
Po σ))
for maxy, CELBO Va. y'; Te, qo) is: Vy' go(z'), ), there is o (z v') Po(z
We will omit special tokens such as </think> and < answer> in the formulas without ambiguity.
2

---

Published as a conference paper at ICLR 2026
2.2 EXTENSION TO IWAB-STYLE LOWER BOUND
In reinforcement learning (RL), it is now common practice to perform parallel rollouts of multiple thinking traces and answers y for given question æ (Shao al., 2024). This naturally motivates us to extend the single-trace BLBO in Eq. (4) to an inportance-weigheel autoencoder (IWAE) style bound (Burda al., 2015). By leveraging multiple K traces, this approach yields strictly tighter lower bound. Specifically, we obtain the following /WAE-style lower bound for log (V lx):
1 K
K
To (zk Vala))
( (z, Va. b/e: To. 90) = (a log
(5)
2o (z x y²)
These IWAE-style bounds satisfy LELBO S CK+1
ELBO < log Po (x)) for any K E N+, which means the bound becomes tighter as K increases (the proof is similar to that of Burda al. (2015)). The single-trace ELBO objective in Eq. (4) corresponds to the special case of K = i.,. LELBO = Li.O
Gradient estimation. We now derive the gradient of Lo Ve, ;/: no. 2o) w.r.t. the model parameters 8 (see Appendix A.2 for the gradient w.r.t. the variational parameters Φ, i... VLL
V VoL (x. y1: no. qo) E EXLX (=(
Pa Vo log no (2k, (Zk V2(
(6)
Pk
(zk Y'e xx) Pk. g x. y²)
where
Pk-
and
Σz=1 Pj
Estimating Pk- The weight Pk: in Eq. (6) can be decomposed as Pk ga(zx (=. To To zk))
no (x(()))))
ga(=x "x is the likelihood ratio of the thinking trace 21: and the second term, To jr, zx), is the probability producing a correct answer given æ and zą. reasoning models, single trace zạ may contain thousands of tokens. Directly computing the likelihood ratio over such long sequences often leads to high variance, a phenomenon also reported in concurrent studies (Cetin
where the first term, mp(2x|=))
<o(aa (æ) 1/|xal
al., 2025; Zheng et al., 2025). To mitigate this issue, use the geometric mean (6 (2.9") as surrogate for the likelihood ratio of zg. This per-token normalization reduces variance at the cost of introducing some bias, effectively spreading the ratio evenly across the thinking tokens.
As for computing To (Va,, lx, 2x), we consider two unbiased estimators: (i) likelihood -baeed estimator is (2 2) = 1//zl Ey-et()))) [T =)]. where Dal is cardinality of 3'2 and ((2)))) is the uni- form distribution on V'; (ii) accuracy-based estimator is No (3 aar, 2) = E By (u xx.x) [1 (y E Va)], where 1-() is the indicator function. When [/ = 1, i.e., there is a unique correct answer expression y" "Zou et al. (2025) show that the likelihood based estimator has lower variance (in fact, zero) compared to the accuracy- based one. We now extend this comparison to general cases when [)'= > 1:
Theorem 1. (Proof in Appendix A.3) For /> > 1, the worst-case variances of the seelihood-based estimator and the accuracy-basa estimator over all possible no (under fixed To ( x, 2)) are
max Varuae (Diz 1) To 2)²; max Varace F002,,.., (1 <0 ar, 2)) (7) To
By
Therefore, the accuracy-based estimator has lower worst-case variance, i.e., max Varaee S maX Varma, whenever the model accuracy (conditional on z) satisfies To T |x, z) 2 D.ll
Note that for many practical questions, the space of correct answers can be quite flexible, so typically Da > 1. In this regime, the accuracy-based estimator enjoys much lower worst-case variance. Based on this insight, in our experiments we estimate the weight Pk as
Pk ( x (x).
(m.
1/1=xl
E.y- [1 Vw)],
(
where the expectation E (yn.s(w) [I E V2)] is approximated by sampling multiple candidate answers for each thinking trace zk. similar to the implementation in Qi et al. (2025).
Estimating Vo log To (Z (z). When evaluating VoCK in Eq. (6), we need the gradient Vo log ( a (x) = V log (z xx) + V log тe (1/ze x. 22). The first term, Vo log Te ( straightforward to calculate. For the second term, Vo log To (V ar, 2x), we also adopt an accuracy- Synra(plm.e (E)))) Vo log (o) Symr (ylm. [[ve)=))
In practice, the expectations w.r.t. To ( an, 2x) are approximated using the same samples drawn to estimate pf*t.
based estimator: Vo log (o)) 2x)
3

---

Published as a conference paper at ICLR 2026
Algorithm 1 Training pipeline of variational reasoning
Inputs: An initial reasoning model Te, (z, y/æz), variational posterior [z y'). question-ansent answer dataset {æt yz E X, where yr E x is one of the reference answers corresponding to æ Inputs: Rollout numbers K and M, training rounds T, steps per round So and Sp. optimizer O The trained model parameters Or and variational parameters ØT
Outputs:
1: Initialize qdo (z yy) Sg copy the
To, (|2, </tt///hinii [x, <h <h y </hint>, <//intt
2: Construct 3/2 (or its subset by rephrasing y and rule-based/model-basee verifier 1l E V+e
3: for t to T do
## Updating variational parameters oe with V
torward in Eq. (9); initializing oe t Φc-1
LM
4: for 1 to S.p do
5: Sample a training batch of questions B C
6:
for all questions
B do ## Collecting 21:M and compute weights wm for each m
7:
Rollout 21:M ~ ( y ~ (o(y') = 2((2²)))
8:
Compute Wm EyrTT (y)=,=m) [1 V&) and üm
Wim
>
105
Update oe + O.step Ø EeEB Em= Wm Vo, log qo. (zm (c. y²)
9:
## Updating model parameters 0g with VoLK in Eq. (6); initializing 0 + 02-1
10:
for 1 to Sg do
11:
Sample a training batch of questions C
12:
for all questions E B do
## Collecting 21: and compute weights iig for each k
13:
Rollout 21:K ~ 90, (z xz, y'), y go(())))))))))) = ()22) ## Estimate px by Eq. (8)
14:
Compute Pi: To, (Ex 3
(x,
1/|pkl
Eyyno, (wl (y E V=) and 5E = Σ
Ex-ray (wlm.y)) ((VEV.) oo (1)1
15:
Compute Ve, log Ta, (Jese x, zx)
Payno (vle ( [1
Update 0₂ + O.step (r, [8]
EEEB ΣK I Vo. (log To, (2 + log No. (J x 2x))
17:
return Of and ΦT
2.3 OPTIMIZING THE VARIATIONAL POSTERIOR VIA FORWARD KL DIVERGENCE
While Eq. (5) provides IWAE-style bounds that yield tighter optimization of the MLE objective w.r.t. the model parameters 9 (through V.LELBO), our pilot experiments show unexpected behavior for the optimization of the variational parameters (through VeLELBO or V & Recall from Eq. (4) that the ELBO objective minimizes the reverse KL divergence DRI. (10(2)z) pz y ( x; )))) where both the expectation and Monte Carlo samples are taken under go y )"). In practice however, the policy model (z(=)) ) is often already well-trained due to pretraining of base LLMss (Liu et al., 2025), while the variational posterior qo (z x. yy) may struggle to effectively use hints y' from correct answers without collapsing into shortcut reasoning (e.g., directly leaking answer tokens into the thinking trace). To address this imbalance, we propose to optimize qo (z(2) æ y²) using the forward KL divergence DDLL
(Po (Po Va (o (z|2, 3/)),, whose gradient w.r.t. ó can be written as:?
M
VOlDKL (Pollar, (oo y')) (z
Σ
Wm V log (2m 1 3/)
CM Forward+
where W'
Wm
and Wm To(Vll zmn) E [1 Va)).
Σj=1
M
wy
This approximation, Vo follows a derivation similar to Bomschhen & Bengio (2015) (see Appendix A.4), with the sample size M necessarily equal to used in VeC Unlike IWAE, this objective is an approximation rather than a lower bound. Optimizing Eq. (9) can be viewed as weighted supervised finetuning (SFT) for (2 x. y²). where training data is sampled from (z(2)))
We summarize the overall training pipeline of our variational reasoning method in Algorithm 1. In our experiments, we train for only single round (T 1), leaving multi-round training as an interesting direction for future work.
We assume that (3))) ) is fixed prior distrihution and does involve gradients.
4

---

Published as a conference paper at ICLR 2026
3 CONNECTION TO OTHER METHODS
In prior work, the thinking trace & and the final answer y are often treated together as the full output in the formulations (Shao et al., 2024; Guo et al., 2025; Liu et al., 2025; Wu et al., 2025). By explicitly decomposing the output into a thinking process z and an answer y. as we have done above, we can gain fresh perspectives on how our formulation relates other mainstream methods.
Connection to rejection sampling finetuning (RFT). RFT methods (Dong et al., 2023; Touvron al., 2023) generate multiple candidate outputs for each input æ using a reference model Tee and then select the candidate with the highest reward. The reference model may be a strong teacher model or identical to the learner тe (i.e., Jee mg with sg denoting stop-gree coo-gradient). Formally, the gradient of RFT training objective,
only on the learning of the thinking trace 2, can be written as: (n) (n [1 E V+e) log Te (z y |=)]
V V Ce.RTT )T) _
V (= (e) [Trer ( ar, z) (log To (z]=)) + log To mol a, x))1
only win.
(10)
B (=|w) [Fier (J x 2) log To (z(2)a))) [m)]
xe (z)
Pret(D'n (z)) Vo DxL. (Prer(z/ar, Vim (=(2))))
Here (r)((()))))) = Ewey. (a denotes the model accuracy on instruction a, and the true posterior of the reference model is Prer(z x. = Tad (m))) As seen, the RFT objective Par(Val))
can be viewed as maximizing a forward KL divergence weighted by (r))('a with the optimal solution T (=) = P Prer(2|aa = Vn). In practice, this weighting downplays hard questions with small Prer()(m (z)), biasing training toward easier ones. In contrast, our formulation in Eq. (9) treats all questions more evenly, ensuring that the objective remains attentive to difficult cases.
Connection to binary reward RL. In the case of RL training with 0-1 binary reward, the training objective can be written in a form similar to Eq. (10), focusing only on the thinking trace z:
V 9 RI (I. ka)
B Bro()) (3) E Vn.)]
only w.r.t
V ell
1 (=|e))
(TT
58
(J kc, z) log Ta (z|=)]
(11)
Po Ov' xx)
VolDKL (z| Va (=(2))))
Thus, the local gradient for (=(=) in binary-reward RL is equivalent to minimizing the forward
KL divergence between (2(2)))) and the true posterior Pj (2|1x,
=o (2. (#)(=) with the
update further weighted by the model accuracy P () x). Interestingly, the RL training objective
itself Lai RI. (æx mo) = (x (F x. [1 (y )'m) (o() whereas the MLE objective in Eq. (2) for our variational reasoning framework instead maximizes log (=))).
Furthermore, in Group Relative Policy Optimization (GRPO) (Shao et al., 2024), one of the most widely used RL training objectives, the reward is normalized by the standard deviation of rewards within a group of rollouts. Under 0-1 binary reward, each rollout reward follows a Bernoulli distri- bution with mean Pa() |æz) and standard deviation (Po() (z) ) (1 Po xx)). The gradient of the GRPO objective can therefore be derived as (we omit the min and clip operations for brevity)
V V Lo To)
only Wet.
P (Va
VODDA (P (z|2z, Vo (z(x)))
(12)
V
p (2))
which follows from Vo VoLo- x To) = V RL (I. TO)/ PoO'==
P z)). Thus,
the local gradient for To (z) ()) in GRPO is still equivalent to minimizing the forward KL divergence
between To (2 ar) and the true posterior P (2 [z|x, ).). Note that the per-instruction weight becomes Pa (z)), which increases monotonically with model accuracy and therefore
/ (P= also emphasizes easier questions with higher (z))) In Appendix A.5, we extend these analyses and derive gradients for more general RL reward shaping, including cases with a format reward.
EXPERIMENTS
Datasets. We train on the Despoke-Stratos 17k dataset curated by Li et al. (2025a), which combines math problems from Numina-Math (Li et al., 2024) with code problems from APPS (Hendrycks al., 2021a) and TACO (Li et al., 2023). The dataset contains 16,710 samples, each paired with
5

---

Published as a conference paper at ICLR 2026
Table 1: Performance of models trained from Qwen3-4B-Base All models are trained on Bespoke- Stratos- except for General--Raasone--B. The best and second-best results are highlighted using bold text and underlined text, respectively.
MATH500 AIME24 AIME25
AMC23 OlympiadBench
Method
Avg
Avg@2
Avg@32
Avg @32
Avg@32
Avg@2
Qwen3-4B-Ban
45.30
4.79
5.73
27.73
23.37
21.38
71.70
19.06
16.77
55.00
45.18
41.54
84.70
27.29
24.17
70.16
50.45
51.35
Burs-PB-GML-4B
87.30
33.54
26.77
74.06
54.45
55.23
Ours-PB-Acc-4B
88.30
31.67
27.29
75.63
55.71
55.72
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@1
Method
Avg
@
Avg@8
Avg@8
Avg@8
Qwen3-4B-B
29.10
18.54
5,46
1.32
36.89
18.26
General-Reasonee
40.97
61.40
17.90
2.85
61.36
36.90
Bespoke-Stratos-4B8
44.95
71.22
19.54
3.25
63.03
40.40
Durs-PB-GMML-41
45.52
79.53
31.25
6.20
63.52
45.60
Jurs-PB-Acc-4B
45.33
80.29
33.68
5.79
65.53
46.12
Table 2: Performance of models trained from (wen3-8B-Basee trained on Despoke-Stratos-11
MATH500
AIME24
AIME25
AMC23
OlympiadBench
Method
Avg
Avg@2
Avg@32
Avg@32
Avg @32
Avg@2
Qwen3-8BB -eene-B-BBase
65.20
11.46
10.10
45.00
34.72
33.30
-trooke-trrtts-833
89.70
39.58
28.85
78.91
55.64
58.54
Ours-PB-GMIL-BI
91.60
44.06
31.67
83.59
58.23
61.83
Ours-PB-Acc-SB
91.80
45.63
31.98
85.47
58.98
62.77
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@
Method
Avg
Avg @
Avg@8
Avg@8
Avg@8
Qwen3-8B-Bass
35,42
41.14
13.65
1.42
45.62
27.45
Bespoke-Stree Stratts---ttt
53.03
81.53
36.89
7.11
68.74
49.46
52.72
87.36
45.51
13.82
70.76
54.03
Ours-PB-Acc-SB 53.66
86.47
49.33
13.21
70.76
54.69
long- tangtttinkiig trace generated by DeepSeek-RL (Guo al., 2025) QwQ-32B-Preee -eervvew (Team, 2024). To ensure fair evaluation, all training data is strictly separated from the test sets.
Baselines. We compare against broad set of baselines for rigorous evaluation. For Qwen2.5-Instruet models, we include Bespoke-Stratee (Labs, 2025) and RLT (Cetin et al., 2025) at the 7B and 32B scales, where RLT trains teachers via RL to improve distillation. For Qwen3-Base models, pare with Despoke-Stratos-4BB. GRPO (Shao et al., 2024) and model-based verifier. Models marked are trained by us; others official releases. All Bespoke-Streeiee -ttratos models are distilled on Bespoke- 17kk This ensures fair comparison, as all methods (except
and Geeneral-Reasoner--BB (Ma et al., 2025), the latter trained with
follow the same training recipes and datasets.
Evaluation. We assess our models on broad set of challenging benchmarks: MATH500 (Hendrycks al., 2021b), subset of competition math curated by Lightman et al. (2024); A1ME24&25 (MAA, 2025); AMC23 (MAA, 2023); OlympiadBench (He et al., 2024); LiveCodeBench (Jain et al., 2025),, with Easy, Medium, and Hard subsets (LCB-E, LCB-M, LCB-H) for fine- ane-grained coding evaluation; GPQA-Diamond (Rein et al., 2024) (GPQA-D), graduate-level natural science questions; and MML.--Po (Wang al., 2024), a diverse multiple-choice benchmark. Among these, GPQA-D and MMLU-Pro out-of-distribution (OOD) relative to our training data.
Decoding and average accuracy. Following Hochlehnert et al. (2025), we sample responses with Cemperature-a ..7 and report average accuracy over & responses per question (Avg @ k). To reduce randomness and ensure fair comparison, we use larger & for smaller datasets: Av for AIME24, AIME25, and AMC23; Avg@s 8 for IPQA-Diamond, LCB-E, LCB-M, and LCB-H: Avg@2 for MATH500 and OlynpiadBench; and Avg@ for the large MMLU-Pra (12k+ questions). All evaluations conducted with SkyThought (Team, 2025a), with additional details in Appendix D.

---

Published as a conference paper at ICLR 2026
Table 3: Performance of models trained from Qwen2.5-32B -nn-strut trained on Despoke-Stratoss 17k. The best and second-best results are highlighted using bold text and underlined text, respectively.
MATH500
AIME24
AIME25
AMC23
OlympiadBench
Method
Avg
Avg@2
Avg@32
Avg@32
Avg@32
Avg@2
Qwen2.5-32B-Instnn
80.70
15.83
12.08
61.95
46.96
43.51
Respoke-Stratos-255
92.60
55.42
46.88
92.19
64.68
70.34
RLT-32B
93.50
56.77
47.19
91.48
63.21
70.43
Ours- Muu--AA-GML-32B
93.20
56.56
48.13
93.98
64.24
71.22
Ours-PA-Ace-32BB
93.50
58.85
50.31
92.97
64.39
72.01
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@1
Method
Avg
Avg@8
Avg@8
Avg@8
Avg@8
Qwen2.5-32B Instruct
46.28
79.88
40.60
9.76
59.19
47.14
Despoke-Stratos-33
57.57
94.78
73.54
30.48
75.22
66.32
RLT-32B
59.09
93,20
72.15
29.78
74.88
65.82
Ours- Bur--PA-GML-32B
60.92
95.19
72.21
35,57
75.57
67.89
Ours-PA-AAce32B
60.73
94.78
73.18
31.81
75.55
67.21
0.8
Bespake-Stratos-ee Stratts-ttttt
0.8
Despoko-Stratosee
BespokeeStratos-Bi 88t
Gradlent Norm
Ours-4B
Ours-88
Gradlent Norm
Ours-88
0.6
0.6
2.5
2.0
0.4
1.5m
0.4
0.5
0.2
Bespoke Belppok--ttratss-pp Stratos-4B
1.0
0.2
2.0
Ours-4B
0.5
0.5
Steps
Steps
Steps
Figure 1: Training loss and gradient norm of different methods during Qwen3-Base model training.
4.1 TRAINING DETAILS FOR VARIATIONAL REASONING
We conduct experiments on D Q 2-en2..533B--nsttteeee (Yang et al., 2024), Qwen3-4B-Ban Basse, and Qwen3-88 -Bse (Team, 2025b). Following Algorithm 1, we first train an initial reasoning model Te, on Bespoke- Stratts- 17k using the recipe from Labs (2025), and then a variational posterior do with the forward KL divergence (Eq, (9)) on the same dataset. These models are later used to compute the weights Pk. For Te,, we adopt the prompt template from Labs (2025); for qø- test two alternative templates ("-PA" and "-PB", see Appendix E). Both Te, and qs finetuned independently from the same base model without weight sharing (Appendix C.I).
Next, the trained 4e to generate 8 responses (thinking traces and final answers) per training sample. For each response, the weight 75 in Eq. (6) is computed from Jø- Te, and, when using the accuracy-based estimator, math/code verifiers from SkyThought. To estimate To (3'a |ac, 2) in Pk. compare three options: a naive likelihood method ("-L"), an accuracy-based method ("-Ace", Sec- tion 2.2), and a geometric mean of token-level probabilities ("-GML"), as detailed in Appendix C.2.
We train the final reasoning model my following Eq. (6) under two data settings. 17K: the full Bespoke- Stratos- 7k dataset. To enhance efficiency. we create a mixed dataset containing, for each original sample, the qo generateted response with the highest Pa and the original sample itself. 1K: fixed 1,000- sample subset uniformly drawn from the full dataset, where all ge gennerrede responses per sample are used for weighted SFT with 7g. The same IK subset is reused across related experiments. Main results are reported with 17K, while ablations use both 17K and 1K configurations (Appendix C.3).
4.2
MAIN RESULTS
We evaluate our method across four model variants: Qwen3-4B/8B--Bn (Tables and 2) and Qwen2.557B/322--Inte Instruct (Tables 3 and 6). Extended results provided in Appendix F.I.
Variational reasoning performance. All methods substantially improve the reasoning ability of the base model, but our approach consistently achieves the best results. As shown in Tables and 2, variational reasoning yields substantial improvements in math, code, and other general domains compared to the base model (c.g., 160% improvement in math and over 152% in other domains). also surpasses all baselines in average accuracy (e.g., over 8.5% higher than the strong baseline -trookk--ttato-4BB3 that uses the same training data, and over 1 14% in other domains).
7

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study on the effect of conditioning the proposal distribution y'.
MATH500
AIME24
AIME25
AMC23
OlympiadBench
Method
Avg
Avg@2
Avg@32
Avg@32
Avg@32
Avg@2
Qwen3-4B-Ban
45.30
4.79
5.73
27.73
23.37
21.38
Ours-4B
88.30
31.67
27.29
75.63
55.71
55.72
w/o y'
81.20
23.44
23.96
65.70
46.59
48.18
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@1 @
Method
Avg
Avg@8
Avg@8
Avg@8
Avg@8
Owen3-4B-Ban 4B-Base
29.10
18.54
5.46
1.32
36.89
18.26
Ours-4B
45.33
80.29
33.68
5.79
65.53
46.12
w/o y
40.53
67.93
16.63
2.44
61.49
37.80
MATH500
AIME24
AIME25
AMC23
100
100
130
Fass8 (96
60-
40
40
40-
20
Despebe-Sratoo--1
20
Ours-48
32
16
GPQA-Diamond
LiveCodeBench-Eaal
LiveCodeBench-Meddie
LiveCodeeBncch-Hard
100
100
100
50
Passgk (%)
40
60
20
20-
MATH500
AIME24
AIME25
AMC23
130
Passak (%)
30
40
40-
20
Bespetaa-Streto--tt
Ours-88
GPQA-Diamond
LiveCodeBench-Eass
LiveCodeBench-Medlum
LiveCodeBench-Hardd
Passak (%c
20
80
120
30-
40
40
20
10:
Figure 2: Pass @ comparison of baselines versus our method based on Qwen3-4B/8B-DD -asc..
Notably, GPQA-Diamond and MML.UPro be considered out-of-distributtin sets, training data only cover math and code, whereas they are in-domain for Bnneeral-Reasoner--e Despite this, our method significantly outperforms General-Reasone on these benchmarks, suggesting that the reasoning improvements from variational reasoning generalize effectively.
Additionally, our method demonstrates robustness across different prompt templates. Performance remains consistent between Prompt Template A and B (denoted as "-PA" and "-PB" in Table 6), with both outperforming baselines. Across four model scales, the accuracy-basee estimator ("-Acc") and the geometric mean of token likelihood estimator ("-GML")) exhibit similar performance, though the accuracy-haid variant shows a slight advantage in math-related benchmarks.
Pass K analysis. We report Pass@k K results of experiments based on Owen33-4/8BB-Be for different values of K. Figure reveals two key trends: (1) Our method's advantage increases with larger K complex benchmarks (e.g., LiveCodeBBnnchHHardd, and (2) Performance gaps diminish on simpler tasks (e.g.. LiveCodeBench- Easy) and multiple-choice questions (e.g., This aligns with expectations, as simpler tasks offer limited room for improvement, and multiple- choice formats inherently allow high Pass @ K with sufficiently large K. These results underscore the strong potential of variational reasoning in tackling complex tasks.
8

---

Published as conference paper at ICLR 2026
Table 5: Ablation study on effects of different To (3 x mx. estimators. Experiments are done in data 1k setting. Ace: accuracy; GML:: geometric mean of token likelihood; L: naive likelihood.
MATH500
AIME24
AIME25 AMC23
OlympiadBench Avg@2
Method
Avg
Avg@2
Avg@32
Avg@32
Avg@32
Owen2.5-7B-Instuee
75.60
10.94
7.40
51.10
39.91
36.99
Bespoke-Stratce -trattttttttotKKKKK
77.20
16.25
13.96
53.75
40.88
40.41
Ours-Acc-7B-IK
81.30
19.69
18.44
61.64
45.99
45,41
Ours- Murs-GML-7B-IK
81.30
19.27
18.33
62.50
45.48
45.38
Ours-L-7B-IK
79.90
17.81
14.17
59.53
43.62
43.01
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@)
Method
Avg
Avg@8
Avg@8
Avg@8
Avg@8
Bwen2.5-7B-Instruee
29.99
62.50
18.20
3.35
48.20
32.45
Poopokk-Stratos7777 7B-IK
37.94
60.37
13.59
1.22
56.07
33.84
Ours-Acc-7B-IK
41.16
68.13
21.42
1.42
60.94
38.62
Murs-GML-7B-IK
41.35
68.41
23.30
2.74
61.31
39,42
Ours-L-7B-IK
39.90
66.42
19.90
1.93
58.61
37.35
AIME24
AIME25
AMC23
LiveCodeBencc--Medid
20
62
16
Accuracy
16-
14
56
Owen2 1on22.----nntturn
10
54
36
Bespoke Strtts
Ours 7B-1K
8
Figure 3: Effects of scaling up the number of thinking traces (K in Algorithm 1) sampled from variational posterior qæ the performance of the final reasoning model mg.
Training dynamics. We monitor training loss and gradient norms during training for Qwen3-4B/8B- base models (see Figure 1). Compared Bespoke- Bespoke-Stratos-455 ABrat's-4B/BB. our method yields lower average training loss and fewer gradient norm spikes, indicating greater training stability. We attribute this stability the To qo ratio Pt- Specifically, for reasoning trace zg. the weight Pa large when the trace is both high-qualit (high (Da (x. zx)) and aligned with the reasoning policy (high likelihood ratio Ty (z x)) x)/ go 2x [2 y²)). This adaptive weighting promotes stable and effective training.
4.3 ABLATION STUDIES
As mentioned in Section 4.1, we conduct ablation studies on both the 17k and 1K data settings to better analyze variational reasoning. Additional ablations are provided in Appendix F.2.
Scaling the number of thinking traces Zg. We investigate the effect of increasing the number of traces sampled from the variational posterior qa (i.e., in Algorithm 1) on the performance the reasoning model Ty. Experiments
conducted under the 1k data setting and faithful to Algorithm 1. We scale K exponentially from to 32, adjusting the batch size to keep optimization steps consistent. Results in Figure suggest that increasing K further enhance model performance. This implies practical trade-off between training computational cost and reasoning accuracy when selecting K.
Conditioning on y'. We ablate the necessity of conditioning on y in data 17k setting. The variant w/o y samples thinking traces z by the initial reasoning model instead of the variational posterior. Results (Table 4) show that removing y' the condition negatively affects the performance.
Different no n) le Jæc 23.) estimators. We ablate different estimators for тo ( zx) used in the weight Pk in data 1 setting (Table 5). We find estimators based accuracy geometric mean of token likelihood outperform the naive likelihood by a large margin, validating our analysis in Sec. 2.2.
CONCLUSION
We introduced variational reasoning framework principled and stable objective for training reasoning models, while clarifying biases in existing SFT/RFT and RL methods. Beyond consistent gains strong baselines diverse reasoning tasks, our analysis offers probabilistic perspective for interpreting current approaches. A natural future direction is extending training beyond a single round (7 > in Algorithm 1) and exploring richer posterior design for the answer hint y'.

---

Published as a conference paper at ICLR 2026
REFERENCES
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J.
Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research, 24(210):1 2023, URL http: / /224//22///// / 2224/2222222 22-0364 html.
// //jmr...o/ org/ http//apprss///eee
Jörg Bornschein and Yoshua Bengio. Reweighted wake-sleep. In International Conference on Learning Representations (ICLR), 2015.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509.00573 2015.
Edoardo Cetin, Tianyu Zhao, and Yujin Tang, Reinforcement learning teachers of test time scaling. arXiv preprint arXiv:2506.08338 2025.
Haolin Chen, Yihao Feng, Zuxin Liu, Weiran Yao, Akshara Prabhakar, Shelby Heinecke, Ricky Ho, Phil Mui, Silvio Savarese, Caiming Xiong, et al. Language models hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding. arXiv preprint arXiv:241 004411..4282. 2024.
Yang Chen, Zhuolin Yang. Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemotronn nemotron: Advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505. 2025.
Daixuan Cheng. Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 2025,
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, r1 generalizes: A comparative study of foundation model post-training. arXiv preprint urXiv:2501.17161 2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06200 2025.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint 200:22500.01456 2025a.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617 2025b.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint 230400304 2023.
Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arXiv preprint arXiv:2410.15115. 2024.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Racof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu. Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. arXiv preprint urXiv:2506.04177 047887 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting agi with olympiad-lewe bilingual multimodal scientific problems, 2024.
10

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding chal- lenge competence with APPS. In Thirty-fithh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021a. URL https: https//ppprevvieeeee merr/o d=sD93G0ZH3
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. arXiv preprint arXiv:2504.071 07086, 2025.
Matthew Douglas Hoffman, Du Phan, david dohan, Sholto Douglas, Tuan Anh Le, Aaron T Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A. Saurous. Training tfain-of-thhouhh via vatenl-variable inference. In Thirty- Conference on Neural Information Processing Systems, 2023. URL https:://// https::/ htpp//ppeeeeiieeeeeeeeeeeee net/ peorummidmall 7ptaa11cpp.o.....
Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL / //wp////ee/.....e///
net Sett/ooruniid-OUs ca4
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung- Yeung Shum Xiangyu Zhang. Open- reasoner-zero: An open source approach to scaling reinforcement learning on the base model. https: /tttss//gtas htg.ttb..bbb....g.g.g hhub. com/ Open Reasoner- Zero/ ZererOpen- Reaeennn-aann----e Zero, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free eval- uation of large language models for code. The Thirteenth International Conference on Learning Representations, 2025. URL https: : opwnreviewwneeeee//e/pe net forumr www.hhhucccchcccccccccc
Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, Tianyu Pang, and Wenhu Chen. Verltool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint arXiv:2509.01055, 2025.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training ilms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.0955 00.60096...66...... 2025.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 131.61131 2013.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Liammin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Bespoke Labs. Bespoke-stratos:
The unreasonable effectiveness of reasoning distilla- www.ss/ttwurrasoob---fffeeeeeeie
tion. www.bespokelabs.ak www.bepokclas..ii/o//boessp wwt.a///bbssookeeeeeeee flog//bspooe-strttoss distillation, 2025. Accessed: 2025-01-22
11

---

Published as a conference paper at ICLR 2026
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review. arXiv preprint 1805.
1805.0090,, 00909, 2018
Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G Patil, Matei Zaharia, et al. Lims can easily learn reason from demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.07374 2025a.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. Codeilo: Condensing reasoning patterns via code input-output prediction. arXiv preprint arXiv:2502.03316, 2025b.
Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312, 14852, 2023.
Hunter Lightman, Vineet Kosaraju. Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL hwtpt:ttt//ppeeeeeeeeeeeetetetee /penreviee.
net/ forum?? ddv8100066000
Jiawei Liu and Lingming Zhang. Code-rl: Reproducing rl for code with reliable rewards. https:
//github. com/ com/ganler/ ganler/ code- r1, 2025.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https: opwnreview.eeeeee/ee net forum?ide 6kkgRiiqYY7
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol -previeww with 1 1.5b model by scaling rl, 2025. Notion Blog.
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing Ilm reasoning across all domains. arXiv preprint arXiv:2505.14652 2025.
Yecheng Jason Ma. William Liang, Guanzhi Wang. De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar, Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.1299 12931, 2023.
MAA. American mathematics competitions amc. https:/ .aeaaarrgg/2200 2023.
MAA. American invitational mathematics examination aime. https: /maa. org/, 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501.1939) 2025.
Penghui Qi, Zichen Liu, Tianyu Pang. Chao Du Wee Sun Lee, and Min Lin. Optimizing anytime reasoning via budget relative policy optimization. arXiv preprint 2rXiv:2505..3438, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-533 -37741, 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: graduate-level google-prod q&a benchmark. In First Conference Language Modeling, 2024.
12

---

Published as a conference paper at ICLR 2026
Yangjun Ruan, Neil Band, Chris J Maddison, and Tatsunori Hashimoto. Reasoning to learn from latent thoughts. arXiv preprint arXiv:2503. 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv22402033300 2024.
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. Rl's razor: Why online reinforcement learning forgets less. arXiv preprint arXiv:2509.04255 2025.
Yunhao Tang, Sid Wang, and Rémi Munos. Learning to chain-of-thought with jensen's 's evidence lower bound. arXiv preprint (xXiv:2503.19611 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501. 2025.
NovaSky Team. Sky-tl: Train your own ol preview model within $450. https//novassy- higgitubbioo/postskkkyt.. 2025a. Accessed: 2025-01-09.
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown. Hugging Face, 2024.
Qwen Team. Qwen3, April 2025b. URL https: http///wwllllmmmmm 1m. g itttb.i.ooo. htt.ooooolllg/////////eo
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.00288 2023.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275. 2022.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprini arXiv:2504.20073 2025.
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing 11m reasoning via reinforcement learning on open software evolution. arXiv preprint arXIv:2502.18449, 2025.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503. 12.500..1600 10460, 2025.
Yongliang WU, Yizhou Zhou, Zhou Ziheng. Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, -iunn Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508.0562. 2025.
Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system reasoning in Ilms: Learning how to think with meta Mhain-of-thougeee arXiv preprint arXiv:2501.0468) 2025.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Il reasoning with rule-based reinforcement learning, arXiv preprint arXiv:2502.14766. 2025.
Yifei Xu, Tusher Chakraborty, Srinagesh Sharma, Leonardo Nunes, Emre Kıcıman, Songwu Lu, and Ranveer Chandra. Direct reasoning optimization: Lims can reward and refine their own reasoning for open-ended tasks. arXiv preprint arXiv:2506. 13351, 2025.
13

---

Published as a conference paper at ICLR 2026
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An. Sim- pletir: End-to-end reinforeement learning for multi-turn tool-integrated reasoning. arXiv preprint N0XYv:2500.02479, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mci Li, Mingieng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv: 2111v:2412.15115 2024.
John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. arXiv preprint arXiv:2504.21798, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale, arXiv preprint arXiv:2503. /4476, 2025a.
Tianyu Yu, Bo Ji, Shouli Wang. Shu Yao, Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, et al. Ripr: Extrapolating rlvr to general domains without verifiers. arXiv preprint arXiv:2506. 18254, 2025b.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in imms beyond the base model? arXiv preprint arXiv:2504.13877 2025a.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.051 2025b.
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https: h/tt:s///kutttee -npp not ion. site/ simpleri wew.leerrrrrrroonoeooooaooaoooee 2025. Notion Blog.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071 2025.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL httt::tp/aaaiiiee o abs /2...2203.11.. 13372.
Han Zhong, Yutong Yin, Shenao Zhang, Xiaojun Xu, Yuanxin Liu, Yifei Zuo, Zhihan Liu, Boyi Liu, Sirui Zheng, Hongyi Guo, et al. Brite: Bootstrapping reinforced thinking process to enhance language model reasoning. arXiv preprint arXiv:2501.18858 2025.
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint arXiv:2505.21493, 2025.
14

---

Published as a conference paper at ICLR 2026
A DETAILED DERIVATIONS
In this section, we provide detailed derivations of the conclusions presented in the main text, along with some additional results.
A.1 DERIVATION FOR EQ. (4)
The ELBO objective induced from Eq. (2) can be rewritten as
(a Vz, y/: na, (o)
- (g y [log Te (D'n æ =)] (1o (²)
= (x) mnyy") [log no (z. [x] + 7/ (2 y'))
(13)
log (3 (z) Dxt. (9a (z| x.
where H(-) is entropy function and Pa(z æ. )/m) T=
is the true posterior distribution.
P=(
A.2 DERIVATION FOR EQ. (6)
Given the IWAE-style lower bound ELBO (x, Ver yo; To, 96) in Eq. (5), we can derive its gradient w.r.t. model parameters 0 as:
V K
Pa, To, 4o)
ne (z (
V Vo
log K
4s (2x lan yr)
k=1
Vemp(E Nalla))
E -E
gol (r,3r)
Te k=
(14)
go(=x|ar
K
=
Pk Vo log No Zk Von (x))
Pk
TØ Veel xx) 2o (zN x., y')
where
and Ps:
Σド=1
K
Pj
Using the notations of and Pe. we can further derive the gradient w.r.t. Φ as:
& CK (x; Va- 3/ nT: qp)
K
T Pa (x) gol 24 (x, y')
= V xa
log
K
K
E -
log
1
Σ
Pk
V v log (0 v))
Σ
Pa v log 90(2z/
K
k=
k=l
k=
K
1
-pk
log
Σ
Pk
V log Vo
K
A.3 PROOF OF THEOREM 1
As to the computation of To (2 (2,)),)) 2), there are two unbiased estimators:
Likelihood-based estimator: T0()a 2) Dal By))(()))) (x)
(15)
Accuracy-based estimator: (0) 2) EE (v( [1 V.),
where [Pmm is the cardinal (number of elements) of V.,, ((()))) is the uniform distribution on V... and I (.) is the indicator function. When Dal i.e., there is unique correct answer expression
15

---

Published as a conference paper at ICLR 2026
y" Zhou et al. (2025) show that the likelihood- baaseeddd estimator has lower variance (in fact, zero)
compared to the accuracy-banae based one. We now extend this comparison to general cases when [>/z >
Varike Val Var yar()) [To (y) x =)];
(16)
Varace = ne (3' x, =) (1 - no () x z))
Note that the variance Varae of awccracy--baaa -baaed estimator independent of [)/= and the model distribution over different elements in y Assuming that in the worst case where only one element y² E De has non-zero probability under ne, i.c., To x. = ( xx x), we have
Vir worst A
max Vanike = (13all - 1) me (3/ xz, 2)²
(17)
Var worst
max Varace = To () x x) (1 nee (2/ [x, z))
Here we slightly abuse the notation of maX since the maximization is taken w.r.t. all Ta under fixed value TT (2,, z). As seen, worst Var holds when the model accuracy (condition on a, 2) satisfies
1
(18)
which almost always holds for [Va] >> 1.
A.4 DERIVATION FOR EQ. (9)
Now we derive the gradient of the forward KL divergence DxL Dia v\) w.r.t. di
(x) y²))
VoBro(=le)))) [log go (2 x y')l
Tp 2) ar 2) Po [x)
(ro
log 2o (2|22 yy)
(19)
M
E ExLa (=(((()
wm V log Zm lx, y²)
A V Toroaaddd
Wm
where
im
and
Wm x0
f= Wj
A.5 CONNECTION TO MORE GENERAL RL REWARD SHAPING
In the literature on reinforcement learning with verifiable rewards (RLVR), various strategies for reward shaping have been proposed, many of which can be expressed as
a if y Yz:
R( a1 9) = 3 if y $ Yo A y Viormati
(20)
otherwise,
where a, 9, are hyperparameters, Vtormat is the set of answers that correctly follow required format (e.g., \boxed }) and is typically independent of Apparently, Vie Vormat holds for any Then we can derive the gradient of training objective under the general reward shaping R(æ, y) as
V RL T&))
V DE no (z|m) E ( m.=) [[xx, 3)
E Exo (=)a) B (
R(x y)
(V, (oggo log ma (ylæ, x))]
only w.I.t.
1E (z|m) (ro())) R y)
V loggm(zz(zz)
(21)
Ne (x)a=)
(x)(() I((a - 3) no Oz |z, 2) (3 - 7) To (Viamaal /2, 2)) Vo lolo [0(2)=))
(a- P) Vo | (2|2, V/e (20(2))))
+ +B8 -7) Pe Pa (z) VoDKLL (P [e, Viormt ) To (x(xx)))
16

---

Published as a conference paper at ICLR 2026
where Po (Yoomall (z) is the probability that the output answers follow the required format. is easy to know that the optimal solution for Eq. (21) can be written as:
(a- (a--8) p (z, aal (B - .)).P P/5 (z, Vivemnal 2r)
Te (z|
(22)
(a (a- 5) P (D's x + (3 - 7)- P (Niormat (z)
Remark. When 3 = 7/, i... there is format reward, the optimization problem in Eq. (21) degrades to Eq. (11). When a > 3 > y, the model To (2|2z) will tend to hack reward function on hard problems (i.e., low (0 (z)) that can easily follow format (i.e., high Pa (z)), where To æ:) will seek modes of P æ, )oormn)). Besides, there may be an intuition that setting / < 0 could alleviate reward hacking, however, shown in Eq. (21), the optimization only depends on the relative values of a - 3 and |
Now we show that it straightforward debias Po 2)) and (D (x) in Eq. (21). Specifically, we can rewrite the reward function as (note that reward functions are equivalent up to any constant):
R(x, y) (a - 3) 1 ( Vom) (B -7) 1 1y Vivema).
(23)
Then the debiased version of reward function is (a 1a-38 3) R² y) = P (1/z)a) xx)
(3-7) 1(y E Va) + P () (x)
1 E Mormaa)
(24)
where in practice Po (J' xx) and P (Jeoomalal (z) can be approximated by the ratio of correct answers (i.e., model accuracy) and the ratio of correct format for each batch of RL rollouts (larger rollout number could lead to more accurate estimation). After using the debiased reward function R (ar, y). the optimal solution Tे (2|1:) becomes
(a 3)
x Vo)
+ +(B --) 1 22, Vioomma)
(=(=))
(25)
-
A.6 SPECIAL CASES EQ. (6)
Special case I: (0 x, y') = ( (z). In this case, we can simplify the gradient estimation as
V C (x, Vzi ne, x
E Ex Bnngooo WR Vo log
(26)
Wk Lj=1")
where
Wk
and we To 2k).
which can be regarded as normalized version of VeriFree (Zhou et al., 2025).
Special case II: K 1. In this case LILBO LELBO and we can simplify the gradient estimation as
V LLLBO (x:, Va. y': no, 90) Ez (x|2x)))) Vo log To (z,
[10g n0 ( V lolog Vo (27)
CELBO Va. 3 ne. qo) E Ez-go
A.7 MORE DERIVATIONS FOR EQ. (11)
Now investigate the gradient of binary reward RL w.nt. To(y)
V VoLuiRL (x:, )o)
VgEn, o E (r m. [1 E Vr)] LL ( m (yl >)
[1 E V=) (Va loa (olg)) log
only w.L.
ET (x)m,)
[ Va) V log To (y)la,2))
(28)
No (e,)))
(x(() | (1)
where the optimal solution is straightforward that Vz, there is TT [x, 2) 1. However, this optimal solution is usually unachievable, since it requires the model to return 100% correct answers independent of the thinking process
17

---

Published as a conference paper at ICLR 2026
A.8 APPLYING VARIATIONAL POSTERIOR FOR RL
Recall that the objective function of binary reward RL is defined as
Db-RL (z, TT)) _ (x [1 Vx)
(29)
E (=|2)) (T) (r,, =]]
Suppose the data points are drawn from a behavior policy (=(x)), the RL objective can be reformu- lated using an importance sampling correction term as follows:
To (2(2)))
Coi-RL( Te) = (z(()()))
q(z[<) (oVVal
(30)
Then, natural question arises: what is the optimal behavior policy q(z(<)) that minimizes the variance of estimating Csi.N((Z To)? Specifically, we can compute
To(2 (==(ar)
Po(z a. Voo)
Varg (=/a()
(J l00, (2, =)
Var (()())
((2(2r) a:))
Po(z vwo q(2(10)
= Po Varg(=|=)
(31)
1
((2
which is equivalent minimizing
Po(z|ar)>>) g(((z))
min Σ
s.t. Σg(2/0) = ((2()))))))))))))))()))) 2 O.
(32)
Using calculus of variations with Lagrange multiplier X, we obtain
Σ (2(22)) -1) -
8
((2(x)
(= (r) = Pa(z x. Vvr) and =
(33)
<z(2 To (J (z2, 22)
(z x. )eee = arg min Varg(=(a)) (x(()))
Therefore, we show that optimizing the variational posterior ds approximate the true posterior Pz(z|æ, 2x) in Eq. (9) naturally yields an (approximately) optimal behavior policy for RL, one that minimizes the variance of the objective estimator. practice, the trained variational posterior ds can thus be employed as the behavior policy to reduce variance, which is fully compatible with actor-critic frameworks that incorporate advantage estimation with baselines.
A.9 CONNECTION REINFORCEMENT LEARNING TEACHERS
In the derivation of our method, the ELBO objective in Eq. (4) minimizes reverse KL divergence DKL (9s (z x y') I (a Va)). As analyzed in Section 2.3, we propose to optimize go (o(2/z, x. y') using the forward KL divergence (Po Pa 1l (z|Ø, y')), which shares the same optimal solution.
Alternatively, can also optimize the reverse KL divergence by policy gradient method follows:
V CELBO
DeL (20(2|0, y')
Volly [log To (z, Via x)) log go (2| xx)
log
log (z|2, 3/) log qs v') log (x)2,,)))) (34)
1o 2) log (= y') (=(2))))
V log qs
reward
18

---

Published as a conference paper at ICLR 2026
More concisely, the reverse KL divergence can be alternatively minimized via reinforcement learning using log To ()/a (z. 2) - log (o(2)2.))) as the reward function.
(=(()))
This derivation establishes a connection to Reinforcement I Learning Teachers (RLTs) (Cetin et al., 2025), who focus on training reasoning LLMs act as teachers for distilling new students. Their approach introduces RLTs optimized specifically for effective student distillation. RLTs are trained by GRPO using dense rewards obtained by feeding each explanation the student and evaluating its understanding of the solution.
Specifically, the dense reward in RLT combines two components: one measuring the student's likelihood of reaching the correct solution (analogous to log To (V z)), and another regularizing the teacher's explanation to remain coherent from the student's perspective given only its prior
knowledge and the question (analogous to log
go (=(aa,)))
Mi (x(2
While RLT employs intuitively designed reward, our work provides rigorous theoretical justification from a variational inference perspective. Furthermore, we enhance the method with a tighter IWAE- style lower bound and an accuracy-based estimator, as detailed in Section 2.2.
B RELATED WORK
SFT and RL methods for reasoning. Reasoning has emerged as a central capability of LLMs, driving advances in domains such as mathematics, programming, and scientific discovery (Jaech et al., 2024; Comanici et al., 2025; Team et al., 2025). Among the approaches developed to strengthen these abilities, SFT and RL have become the two dominant paradigms (Uesato al., 2022; Rafailov et al., 2023; Guha et al., 2025; Hu et al., 2025; Hochlehnert et al., 2025). Building on the DeepSeek-R2 framework (Shao et al., 2024; Guo et al., 2025), a range of new RI. algorithms have been proposed, including Dr. GRPO (Liu et al., 2025), DAPO (Yu et al., 2025a), REINFORCE++ (Hu, 2025), VinePPO (Kazemnejad et al., 2024), and VAPO (Yue et al., 2025b). In parallel, extensive empirical studies have explored the design space of RL for reasoning (Zeng et al., 2025; Team et al., 2025), focusing on dimensions such as curriculum learning (Wen et al., 2025; Luo et al., 2025) and reward design (Gao al., 2024; Cui et al., 2025a; Ma et al., 2023; Qi et al., 2025). While early progress has centered on mathematical reasoning, recent work has extended RL-based methods to code and software engineering tasks (Liu & Zhang, 2025; Xie al., 2025; Wei al., 2025; Yang et al., 2025; Chen et al., 2025; Li et al., 2025b), as well as to agentic problem-solving scenarios (Wang et al., 2025; Jin al., 2025; Jiang et al., 2025; Xue et al., 2025).
Decomposing thinking and answering processes. Traditional studies on LLM reasoning ability often treat model responses holistically. In contrast, recent line of research explicitly decomposes the LLM response into a thinking trace and final answer y. given a question æ (Chen et al., 2024; Xiang al., 2025: Zhou et al., 2025; Zhong et al., 2025). This decomposition offers several novel and useful perspectives.
Zhou et al. (2025) propose VeriFree, which directly optimizes Po az: = (x)(())) [To at, 2)] using policy gradient. Their algorithm simultaneously optimizes Ta(2 (x) via policy gradient methods with To(y)a x;, 2) as a reward, and performs weighted SFT on mo (y x 2). This approach demonstrates strong performance in general domains where aule-bsed verifiers are typically unavailable. Subse- quent works (Yu et al., 2025b; Xu et al., 2025) further improve upon this by reshaping the reward, e.g.. intuitively replacing the product of token probabilities with the mean when computing To ar 2).
Chen et al. (2024) introduce LaTRO. formulating reasoning as sampling from latent distribution {(2(2) (z) and optimizing log Po (o(y))) via variational manner. Their derived lower bound is:
log (o(y)a x) Eg(=)=) [log Te (e/lla æz, =)] DRL (g(z(æ)) || TT (=(=x²))
They set the proposal distribution q( (x) to T2(2 [x). resulting in reinforcement learning algorithm where log Te (y x, 2) serves as the reward. Tang et al. (2025) and Ruan et al. (2025) tighten this
bound using ideas similar to IWAE (Burda al., 2015).
more natural choice for the variational distribution is the true posterior (a xx, y), though it is intractable. Hoffman et al. (2023) use MCMC to sample from the posterior, while Hu et al. (2024) employ GFlowNets (Bengio et al., 2023) to fine-tune an LLM to approximate it. Both methods use an EM-like algorithm optimize the ELBO of log (y)la)))
19

---

Published as a conference paper at ICLR 2026
Our approach uses forward KL divergence to train variational posterior and derives novel objective based on a tighter IWAE-style bound. Additionally, we propose an accuracy-bart based estimator for ( xz, 2g), instead of the likelihood- estimator used in Zhou et al. (2025). We also build con- nections to other mainstream finetuning algorithms that enhance reasoning, such as RFT and GRPO.
Reinforcement learning probabilistic inference. Previous works have also explored connections between reinforcement learning and probabilistic inference. Notably, Levine (2018) discuss RL and control from probabílistic inference perspective. Their approach begins from reinforcement learning standpoint, where the goal is to search for an optimal policy, and constructs a probabilistic graphical model (PGM) in which the posterior distribution over actions corresponds an optimal policy. This is achieved by defining observations in the PGM based on rewards, and the resulting inference problem is then solved via variational inference. This formulation differs from our approach.
In contrast, our method starts from the objective of maximizing log Po()a [x)) treats the reasoning process discrete latent variable 2, and optimizes the ELBO to train the reasoning model. We subsequently draw connections between our framework and other RL algorithms (Section 3). From an application perspective, our work focuses on the language domain and explicitly models reasoning traces as discrete latent variables, which further distinguishes it from the aforementioned work.
C TRAINING DETAILS
In this section, detail the training procedure used in method. Our framework builds on LLaMA-Factory (Zheng et al., 2024). By default, SFT averages token-level cross-entropy over all valid tokens in batch. However, as shown in Section 2, our variables and y are defined at the sentence level. To align with this, we modify the objective: instead of normalizing by the number of valid tokens, we sum the loss over all tokens and divide by constant equal to the average response length in the training set (precomputed offline). This change parallels the difference between GRPO (Shao et al., 2024) and Dr. GRPO (Liu al., 2025), thus we name this slight modification as Dr. SFT. Both our models and Despoke-SStratos-4BAA further extend the framework with weighted SFT, as the original LLaMA-Facton does not support weighting. This feature is essential for parts of our method that require weighted training. All experiments are conducted on NVIDIA H100 GPUs.
are trained with this modified objective. We
C.1 TRAINING THE INITIAL. REASONING MODEL AND VARIATIONAL. POSTERIOR
For training, we use the following settings:
1. Initial reasoning model Tho (following Labs (2025)):
Muttchssiz=:98 cutoff_len=1 6384
Optimizer: Adam (Loshchilov & Hutter, 2019) with adam_betal=0.99
adam_beta 999, adam_epsi 0e weigyt_decay=0
Learning rate schedule: cosine with warmup_ratio=0.1
learning_rate-1. 0e-5, max ma__gaaaa_ooomm_111____1
Training for epochs
Precision: bfloat16
Baselines (Bespoke-ttraoos-7B/220 and Bespoke-Strrtoo-4BBBBB)) are trained with the same setup
2. Variational posterior 96:
batch_size-1
Optimizer: AdamW with adam betal=0. adam beeeaet---- 95, weight_decaye 0e-4 warmup_ratee rttp_rrttii000555
adam www.meeeeioon.0eeeee
Training for 10 epochs
All other hyperparameters follow those used for noo
20

---

Published as a conference paper at ICLR 2026
C.2 DETAILS OF SAMPLING FROM VARIATIONAL POSTERIOR
Using the trained variational posterior 9s. sample reasoning traces (including final answers) for each question in Pispoke-Stratoss 17k with vLLM (Kwon et al., 2023), using temperature ceeratuueeO... 64, and dtype=bf foofloaa..... 6. After obtaining the sampled reasoning traces, we compute the importance weight pk for each question-thinkin-annse triplet using the pre-trained initial reasoning model Tea and the variational posterior qs- This is done in forward mode by evaluating the log-iikelihodss under both models, without requiring backpropagation, which is efficient.
top_p=1. 0, top_k=-1, wax_ttkenss-e
To estimate the term Te () æz, z) used in Pk. we adopt an uccuracy-based estimator. Specifically, for each question and each sampled thinking trace, we use Th generate 8 answers under the same sampling configuration as above. The correctness of these answers is evaluated using the math/code verifiers from 3kyThought, and the average accuracy is taken as the estimate of wo (Yin x, z). Experiments utilizing the accuracy-based estimator are labeled as "-Acc".
Additionally, we employ the geometric mean of token-level probabilities under TT (V x, 2) as an alternative and intuitive estimator. This approach mitigates the inherent length bias present in the strict definition of TT x; 2), which computes the product token probabilities and consequently assigns excessively small values to longer reasoning traces. This estimator provides an intuitive approximation without requiring an external verifier. Experiments utilizing the estimator based on geometric mean are labeled as *-GML", while those with the naive estimator labeled as "-I".
This process results a weighted dataset where each sample consists a question- guestion-thinking. thinking-answnn answer triplet along with its corresponding weight Pk. which will be utilized in subsequent training stages.
C.3 DETAILS TRAINING FINAL REASONING MODEL
To train the final reasoning model To, adopt the following procedure. For the 17k data setting, we select, for each question, the reasoning trace with the highest importance weight Pk: among the samples sampled from the variational posterior. In experiments using the accuracy-based estimator, we pair the selected reasoning trace with a randomly chosen verified answer generated by the initial reasoning model. For other estimators, the original answer from the dataset is retained. The resulting synthetic data is then mixed with the original Bespoke- Strrtts-trtt--tttt 1 17 dataset.
We maintain the same training configuration as used for the initial reasoning model, with one exception: the batch size is increased to batch_st _s ze=1 92. This adjustment ensures that the total number of optimization steps remains consistent with baseline models (e.g., Bespoke-Streee -trattssttttt) as the mixed dataset is twice the size of the original.
For the 1k data setting, the baseline model (e.g., Bespoke-Stratoo--B-Ik)) is trained with the follow- ing configuration: We adopt battch_sti ze=1 and optimizer (Loshchilov& Hutter, 2019) with parameters addam
We use the AdamW 9, adam_beta b= 95, adam epsilpppee 0e-8, and we weight httggghhtdeeeleeeeeeeee 0e-4. A cosine learning rate schedule is waaampppprttiit rati 0. 1, alongside learning rate 1. 0e-5 and gradient clipping at max max_grad aar__norm-1__1_m 0. Training is conducted for 5 epochs.
applicd with warmup
In our method under the 1k setting, we do not combine with the original dataset. Instead, we use all reasoning traces, weighted by Pk. which is faithful to Algorithm 1. To match the number of optimization steps in the baseline, we proportionally adjust the batch size while keeping all other hyperparameters unchanged.
D DETAILS OF EVALUATION
We conduct all evaluations using SkyThought, specifically at commit Cd1 F1 Team (2025b) suggest avoiding greedy decoding for models with long thinking traces. Thus, responses are sam- pled from the models using temperature-01 and 1op_p=1.0 1.0.. O. A generous token budget of max max_tooens==3911 is allocated to accommodate lengthy outputs.
hhttps://gi wtt..ccm/Nooaakyy.....mm /l/Kky/hought
21

---

Published as a conference paper at ICLR 2026
To maximize reproducibility, we perform inference using dtype=float accepting po- tential decrease in speed for improved consistency. For model parallelism, we configure tensor parallel size=4 for 4B/7B/8B models and patssoo_prraaeelle
for the 32B models. We choose vLLM (Kwon et al., 2023) as the inference backend. For models based Qwen2.5,
v1 1 7 0, while for Qwen3-based models, Although we anticipate that these version differences have negligible impact on evaluation accuracy, we document them here ensure full reproducibility.
vl 11==0=0.. B
E PROMPT TEMPLATES
In Section 2, we abstractly introduced how we define the prompt patterns used in the reasoning model Te and variational posterior Qş. In this section, we provide details of the prompt templates used in practice, as shown below.
Prompt template A (PA) for variational posterior
Your role an assistant involves providing precise and accurate solutions before providing detailed explanations with your full work showing your systematic thinking process leading to each solution. Your explanations should show how you engaged in comprehensive cycle of analysis, summarizing, exploration, reassessment, reilection, backtracing, and iteration to develop well-consicered thinking process. Please structure your response into two main sections: Solution and Explanation. In the Solution section, present your well-thought solution that accurately answers the question. The solution should remain logical, accurate, concise expression style and detail necessary step needed to reach the conclusion, formatted as follows: snrgginoof_sluutionn
(final formatted, precise, and clear solution) <jené_of_solutiani/ In the Explanation section, comprehensively detail your reasoning process using the specified format: <lbbegih_o__explaaatiiooo
(explanation with steps separated Each step should shew
with " 'An' detaile led considerations leading to your solutions such analisying questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps.
Prompt template B (PB) for variational posterior 4e
Your role assistant involves reconst the internal reasoning process that connects a provided question its correct answer. Your task is to methodically reverse-engineer the legical steps, demonstrating a full cycle of analysis, summarization, idea generation, verification, error correction, and iterative refinement. Please structure your response into distinct parts: solution and Thought. In the Solution section, present the given correct answer in a precise and clear format: [[begin_of_solutinnn
(provided correct solution)
<l In the Thought section, articulate the the (detailed thought precess
step-by-step cognitive journey that leads the solution. speci fied format: <|begin_of_thoughn with steps separated by Vin\\n' < <|end_ too_ghhughh |>. Each step should reflect analytical breakcowns, synthesis of key points, generation of logical pathways, validation of each step's accuracy, refinement of any missteps, and reassessment of previous conclusions. The focus is solely on depicting the internal, structured thinking that arrives at the provided soluticn.
22

---

Published as a conference paper at ICLR 2026
Prompt template for reasoning model w
Your role as an assistant invol ves theraughly exploring questions through a systematic long thinking process before providing the final preci se and accurate solutions. This requi res engagi ng in comprehensive cycle of analysis, summarizing, exploration, reassesement, reflection, backtracing, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution. In the Thought section, detail your reasoning process using the specified format: [bbogin_of_choughtt (thought with steps separated with 'MaX\n') < [ Each step should include detailed considerations such as analyzing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previcus steps. In the Solution section, based on varieus attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The solution should remain logical, accurate, concise expression style and detail necessary step needed reach the conclusion, formatted as follows: bppggin_o_ssolution clear solution) <lend_ of of of_solutionn>>> Now, try to solve the following question through the above guidelines:
(final formatted, precise, and
23

---

Published as a conference paper at ICLR 2026
Table 6: Performance of models trained from Qwen2.5-7B-Instruee trained on Pespoke-Stratos 17k. The best and second-best results are highlighted using bold text and underlined text, respectively.
MATH500
AIME24
AIME25
AMC23
OlympiadBench Avg@2
Method
Avg
Avg@2
Avg@32
Avg@32
Avg@32
Qwen2.5-7B-Insteee
75.60
10.94
7.40
51.10
39,91
36.99
Bespoke-Stratos-7B
82.20
19.58
19.48
63.28
45.03
45.91
RLT-7B
84.30
22.81
19.48
64.84
46.43
47.57
Mus-PA-GMM--7B
85.30
24.17
20.42
68.20
46.88
48,99
83.40
22.50
20.83
65.39
47.55
47.94
Durs-PB-GML-78
84.00
22.08
20.42
66.80
46.29
47.92
Ours-PB-Acc-777
84.80
23.96
19.69
65.00
48.15
48.32
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@)
Method
Avg
Avg@8
Avg@8
Avg@8
Avg@8
Qwen2.5-7B-Instre
29.99
62.50
18.20
3,35
48.2
32.45
Bespoke-Stran Strrkkss-ttttt-775
39.02
69.30
23.06
2.95
60.59
38.98
RLT-7B
41.60
72.32
25.06
3.66
61.28
40.78
Ours- PA-GML-7B
43.62
74.52
27.79
4.78
60.88
42.32
Ours-PA-Acc-7BB
43.56
74.45
29.43
4.37
60.72
42.51
Ours-PB-Gin 7B-GML-77
41.60
74.73
28.46
4.57
61.14
42.10
Mus--PBAcce-77
41.67
75.07
29.49
4.68
61.28
42.44
F EXTENDED RESULTS
F.1 EXTENDED MAIN RESULTS
Due to space constraints in the main paper, we present extended evaluation results in this section.
We report the evaluation results for models fine-tuned from Qwen2.5-7B- -reen.5-7B-Inttrucc on the Bespoke- Stratos-7B dataset in Table 6. All variants of our method outperform all baselines in terms of average accuracy, demonstrating the superiority of the variational reasoning approach. Notably, the two prompt templates used for the variational posterior de yield similar results, indicating that our method robust to the choice of template. We attribute this robustness to the fact that the posterior is obtained by fine-tuning the model qs. rather than through prompt engineering alone, thereby reducing the sensitivity to specific prompt formulations.
Additionally, we plot the distributions of the thinking token length versus the log-likelihood ratio me(=x|=))
log gu(>>] (x.))) and the answer token length versus the log-likelihood of the answer log me (Y'. x, z&) in Figure 4. The results reveal strong correlations between these variables, indicating the presence of length biases. This observation further justifies the use of estimators based accuracy or the geometric mean of token likelihood, rather than the naive likelihood.
Another noteworthy observation is that our evaluation results for General-Reasonn Real--Ressoo--BB (see Table 1) differ from those reported by Ma et al. (2025) in their original paper, despite using their officially released checkpoints and provided prompt template. This discrepancy be attributed to several fac- tors: (1) different evaluation frameworks: we employ SkyThought whereas they utilize simple-evals': temperature
and max_ max_tokens==89p 12, while they primarily employ greedy decoding (i.e., temperature=0) except for AIME24 and AIME25, along with more constrained token budget of max_tokers=8192
(2) different sampling configurations:
To facilitate a fair comparison, we provide results on the common benchmarks that we and Ma et al. (2025) both utilize, comparing our model accuracy with their officially reported values (see Table 7). As demonstrated, our method continues to outperform (where indicates their officially reported accuracy).
by a significant margin
hhttps://gi thub. www..ccoooope........ /s1 /simple e-eval
24

---

Published as a conference paper at ICLR 2026
Thinking Length vs. Log
Answer Length vs. Log Likellhood Answer
500
0-
Likelihood of Answer
Log Likellhood Ratio
-500
100
-500
-1000
-700
-1500
-250
2000
-300
2000
4000 6000 8000 10000 12000 14000 16000 Thinking Length
400
6:00
800
1000
1200
Length
Figure 4: Density maps of the thinking token length versus the log- log-likelihood ratio log qa( a.yy" (left), and the answer token length versus the log- likelihhodd of the answer log No (Y'n x z1) (right).
mo (za (m)
Table 7: Performance of models trained from Qwen3-4B-BBl
All models are trained on Bespoke- Here, denotes accuracy values officially reported by Ma et al. (2025), rather than results obtained through our own evaluation. The best and second-best results are highlighted using bold text and underlined text, respectively.
Stratos- 17k except for
Method
MATH500
AIME24
AIME25
AMC23
Qwen3-4B-Bal Base
45,30
4.79
5.73
27.73
71.70
19.06
16.77
55,00
80.6
20.0
15.4
60.0
Pespoke-Stratos-ee
84.70
27.29
24.17
70.16
Ours-PB-GMI
87.30
33.54
26.77
74.06
Ours-PB-Acc-4B
88.30
31.67
27.29
75.63
Method
OlympiadBench GPQA-D MMLU-Pro
Qwen3-4B-BB Base
23.37
29.10
36.89
45.18
40.97
61.36
General- RBa
47.7
42.9
62.8
Despoke-Stratos-e
50.45
44.95
63.03
Durs-PB-GML-4B
54.45
45.52
65.52
PB-ce-4
55.71
45.33
65.53
F.2 EXTENDED ABLATION STUDIES
This section presents extended ablation studies analyzing: the impact of different training data sources and different ways of data usage (Appendix F.3); the effect of data overlap between variational posterior training and reasoning model training (Appendix F.4); the comparison between Dr. SFT and naive SET (Appendix F.4); and the influence of cutoff length during training (Appendix F.6).
F.3 EFFECTS OF DIFFERENT FINAL DATA SOURCES AND WAYS OF DATA USAGE
In our main experiments, we prioritize training efficiency by using the 17k data setting, selecting the variational reasoning trace 2ς with the highest weight P& and mixing with the original Bespoke- Stratos- - 17k data (which results in double data sizes), rather than using all cight traces sampled from the variational posterior. To evaluate the impact of this simplification, we conduct ablation studies under the 1k data setting. We compare variants that either mix do not mix with the original data, and that use either single-best best trace selection ("-S") or weighted multiple traces ("-M")..
Results are shown in Table 8. The best performance is achieved by the variant that uses weighted multiple reasoning traces without mixing with the original data. This suggests that, when com- putational cost is not constraint, the optimal approach is utilize all reasoning traces from the
25

---

Published as a conference paper at ICLR 2026
Table Ablation study on the effects of different final data sources (only sampled from variational posterior vs. mixed) and different ways to use samples from variational posterior to train reasoning models (single best reasoning trace selection ("-S"" vs. weighted multiple reasoning traces ("-M")). This ablation is done in data 1k setting. The best and second-best results are highlighted using bold text and underlined text, respectively.
MATH500
AIME24
AIME25
AMC23
OlympiadBench Avg@2
Method
Avg
Avg@2
Avg@32
Avg@32 32
Avg@32
Qwen2.5-7B-Instruee
75.60
10.94
7,40
51.10
39.91
36.99
Respoke-Stratos-T7e
77.20
16.25
13.96
53.75
40.88
40.41
Ours-M-7B-IK
79.80
18.65
16.98
60.55
44.81
44.16
w/o Mix
81.30
19.69
18.44
61.64
45.99
45,41
Burs-S--B-111
81.10
19.90
17.08
60.39
43.92
44.48
w/o Mix
80.40
17.92
17.81
59.06
43.18
43.67
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@)
Method
Avg
Avg@8 8
Avg@8
Avg@8
Avg@8
Qwen2.5-7B -
29.99
62.50
18.20
3.35
48.2
32.45
Pispoke-Stratos-777
37.94
60.37
13.59
1.22
56.07
33.84
Ours-M-7B-IK
42.80
65.73
18.99
1.93
60.49
37.99
w/o Mix
41.16
68.13
21.42
1.42
60.94
38.62
Ours-S-7B-IK
40.34
64.84
18.20
2.13
59.54
37.01
w/o Mix
39.96
65.04
18.27
1.32
59.87
36.89
variational posterior, weighted by Pk. for training the final reasoning model. Another interesting observation is that for the single-trace method, data mixing improves performance, whereas for the weighted multi-trace method, mixing slightly degrades performance. This may indicate that the weighted ensemble of variational traces already provides sufficient information, making the original data redundant this scenario.
F.4 EFFECTS OF DATA OVERLAP
In the 17k data setting, the variational posterior 9a is trained on all 17k samples and generates thinking traces for the same of 17k samples, which are subsequently used to train the final reasoning model Ty. In other words, both the variational posterior and the final reasoning model are trained on the same set of question-answer pairs
An interesting question is how our method performs in the absence of data overlap. To investigate this, we design experimental settings: the first is the same 1k data setting introduced earlier; the second is constructed by splitting Pespoke-Stratoss 17k into two non-overlapppig subsets: one contains 15.710 samples (approximately 16k) and the other contains Ik samples. In the latter setting. we train the variational posterior 9s and the initial reasoning model Ta, the 16k subset. We then use the trained (a to sample thinking traces for the 1 subset, and employ both is and me, to compute Pk. Finally, the final reasoning model my is trained on the 1k subset using weighted multiple reasoning traces. This setting is referred to as "w/o Overlap".
The results are presented in Table 9, Both the overlap and non-overlap variants exhibit similar performance in terms of average accuracy, and both outperform the baseline, Bespoke Bespoke-Stratos- Stratts- -B- IKi. This suggests that the trained variational posterior generalizes reasonably well and can be applicd to broader scenarios.
F.5 COMPARING DR. SET WITH NAIVE SFT
As detailed in Appendix C, we employ a slightly modified objective function, Dr. SFT, where the loss is defined the sum of all valid token losses normalized by constant, rather than the mean loss across valid tokens in the batch. We conduct an ablation study comparing this Dr. SFT approach against naive SFT when training the final reasoning model mg.
26

---

Published as a conference paper at ICLR 2026
Table 9: Ablation study on the effects of data overlap between variational posterior training and reasoning model training. This ablation is done in data 1k setting. The best and second-bes best results highlighted using bold text and underlined text, respectively.
MATH500
AIME24
AIME25
AMC22
OlympiadBench Avg@2
Method
Avg
Avg@2
Avg@32
Avg@32
Avg@32
1inn2.5-7B-Instruue
75.60
10.94
7.40
51.10
39.91
36.99
Bespoke-Sttatoe Stakkk--ttttt--BBB 7B-1Kt
77.20
16.25
13.96
53.75
40.88
40.41
Ours-7B-1K
79.80
18.65
16.98
60.55
44.81
44.16
w/o Mix
81.30
19.69
18.44
61.64
45.99
45.41
w/o Overlap
80.60
20.83
18.44
61.17
44.81
45.17
w/o Mix w/o Overlap
82.00
18.75
16.88
63.52
47.11
45.65
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@)
Method
Avg
Avg@8 8
Avg@8
Avg@8
Avg@8
Qwen2.5-7B-Insteee
29.99
62.50
18.20
3.35
48.2
32.45
Bespoke-Stratee Strcc-Sttttts-BBttt -
37.94
60.37
13.59
1.22
56.07
33.84
Ours-7B-1
42.80
65.73
18.99
1.93
60.49
37.99
w/o Mix
41.16
68.13
21.42
1,42
60.94
38.62
w/o Overlap
38.19
66.00
19.84
1.12
60.65
37.16
w/o Mix w/o Overlap
39.65
68.82
19.96
2.44
60.36
38.25
The results are presented in Table 10. Both variants demonstrate comparable performance, with less than 2% difference in average accuracy, and both outperform the baseline. This allows to conclude that the primary performance improvement stems from the variational reasoning mechanism rather than from this minor modification to the objective function.
F.6 ABLATION STUDY THE EFFECTS OF CUTOFF LENGTH WIEN TRAINING
In our main experiments (17k data setting), we use cutoff length of cutoff Lerff_ee==111 0am==1388 6384. To investigate the impact of this hyperparameter, we conduct an ablation study comparing two variants: one using the default cutoff saafff__een1688__ (denoted "-Len16k") and another with cutoff_len=327 (denoted "-Len32k").
The results are presented in Table 11. Both variants exhibit similar performance. We further analyze the average completion token lengths on several evaluation benchmarks (see Table 12). All methods produce significantly longer reasoning traces compared to the Qwen2.5-7B-Innsee generating slightly longer thinking traces. Notably, the 16k and 32k cutoff variants result in similar generation lengths during inference. This indicates that increasing the cutoff length beyond 16k has minimal effect on the model's output. Therefore, we can confidently use the 16k setting for better training efficiency without sacrificing performance.
with our methods
27

---

Published as a conference paper at ICLR 2026
Table 10: Ablation study comparing Dr. SFT and Naive SFT. This ablation is done in data 17k setting. The best and second-best results are highlighted using bold text and underlined text, respectively.
MATH500
AIME24
AIME25
AMC23 OlympiadBench Avg@2
Method
Avg
Avg@2
Avg Av
wyg@32
32
Avg@32
@wen2.5-7B-Instteen
75.60
10.94
7.40
51.10
39.91
36.99
Despoke-Stratos-7B
82.20
19.58
19.48
63.28
45.03
45.91
RLT-7B
84.30
22.81
19.48
64.84
46.43
47.57
Ours-PA-GML-7B
85.30
24.17
20.42
68.20
46.88
48.99
w/ naive SFT
84.00
22.60
20.10
65.31
49.11
48.23
Ours-PA-Ace-
83.40
22.50
20.83
65.39
47.55
47.94
w/ naive SFT
84.10
23.02
21.04
67.66
46.96
48.56
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@1
Method
Avg
Avg@8
Avg@8
Avg@8
Avg@8
Qwen2.5-7B BB--trnntttt
29.99
62.50
18.20
3.35
48.2
32.45
Despoke-Stratos-75
39.02
69.30
23.06
2.95
60.59
38.98
RLT-7B
41.60
72.32
25.06
3,66
61.28
40.78
Burs-PA-GM--7B
43.62
74.52
27.79
4.78
60.88
42.32
w/ naive SFT
42.49
74.73
25.85
3.66
61.41
41.63
43.56
74.45
29.43
4.37
60.72
42.51
w/ naive SFT
42.11
75.69
27.79
5.39
61.22
42.44
Table 11: Ablation study on effects of cutoff length used training.
This ablation is done in data 17k setting. Lenl6k: cutoff [__ee=116333 6384; Len32k: cutoff _ff__en=32768 The best and second-best results are highlighted using bold text and underlined text, respectively.
MATH500
AIME24 AIME25 AMC23
OlympiadBench Avg@2
Method
Avg
Avg@2
Avg@32
Avg@32
Avg@32
Been2.5-7B-Instruue
75.60
10.94
7.40
51.10
39.91
36.99
Bespoke-Stratos--BB
82.20
19.58
19.48
63.28
45.03
45.91
RLT-7B
84.30
22.81
19.48
64.84
46.43
47.57
85.30
24.17
20.42
68.20
46.88
48.99
Our--PA-GML-7B-LLnnB3
84.10
22.08
20.94
66.80
48.37
48.46
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@1
Method
Avg
Avg @
Avg@8
Avg@8 @8
Avg@8
Qwen2.5-7BBInnttnu
29.99
62.50
18.20
3,35
48.2
32.45
Despoke-Stratos-1
39.02
69.30
23.06
2.95
60.59
38.98
RLT-7B
41.60
72.32
25.06
3.66
61.28
40.78
Ours-PA-GMI-7733
43.62
74.52
27.79
4.78
60.88
42.32
Dun-PPA-GMMI-7B-Lee222
42.49
74.93
28.58
4.37
61.64
42.40
Table 12: Average completion token length of models trained from Qwen2.5-7B-Insteee
MATH500
AIME24
ATME25
AMC23
MMLU-Pro
Method
Avg@2
@3g@32
Avg@32
Avg@32
Avg@)
Qwen2.5-7B-Insttee
564
1270
1027
849
531
Stspoke-Stratos7BB
5801
18413
15769
10921
3889
RLT-7B
5508
18143
15769
10986
3942
Durs-PA-GML-7B
5677
18299
16471
11338
3924
Ours-PA PA-GMLL7B--Lan
5688
18170
16747
11531
3965
Ours-PA-Ace- PA-Acc-7B
5688
18170
16747
11531
3965
5803
18479
16615
11080
4052
Ours-PB-Ace-78
5787
18651
16696
11591
3974
28

---

Published as a conference paper at ICLR 2026
F.7 EFFECTS OF WEIGHTS FOR TRAINING THE REASONING MODEL
Our framework employs weighted supervised fine-tuning (SFT) for the reasoning model, as formalized Eq. 6. The weight Pk 7u( im.y"] T () x 2x) is derived from an IWAE-style evidence lower bound (ELBO). To analyze the contribution of the weighting scheme, we conduct an ablation study with variants: (I) uniform weighting (pk 1), which reduces the method standard SFT on traces sampled from the variational posterior; and (2) correctness-only weighting, which uses only the final answer probability ( 22, 2x) and omits the likelihood ratio. Results in Table 13 show that both components of the full weighting scheme contribute positively to performance. These findings align with our theoretical derivation and validate the design of the objective.
Se (a:)
Table 13: Ablation study on the effects of different SFT weights for training the reasoning model.
The best and scond-best results are highlighted using bold text and underlined text, respectively.
MATH500
AIME24
A1ME25
AMC23
OlympiadBench
Method
Avg
Avg@2
Avg@32
Av
Avg@32
Avg@2
Qwen3-4B-Bas -Bae
45.30
4.79
5.73
27.73
23.37
21.38
General-Reasoneed
71.70
19.06
16.77
55.00
45.18
41.54
Despoke-Stratos-4e
84.70
27.29
24.17
70.16
50.45
51.35
Duss-PB-Ace-4B
88.30
31.67
27.29
75.63
55.71
55.72
w/ Pk
87.00
31.04
26.04
72.89
52.52
53.90
w/ Ph = To Je x 2x)
88.20
31.45
26.56
73.43
54.45
54.82
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro
Method
Avg
Avg@8
Avg@8
Avg@8
Avg@8
Avg@)
Qwen3-4B-Base
29.10
18.54
5.46
1.32
36.89
18.26
General- Beneral-Reasoner-44B
40.97
61.40
17.90
2.85
61.36
36.90
Despoke-Stratos-e
44.95
71.22
19.54
3.25
63.03
40.40
Ours-PB- -B-s-BB-Ac-BB
45.33
80.29
33.68
5.79
65.53
46.12
w/ Pk w/ Pk = To (2,
45.07
78.09
28.82
5.38
64.24
44.32
45.07
78.43
30.27
5.48
64.41
44.73
F.8 EFFECTS OF NOISY HINTS
As sanity check, we conduct an experiment to confirm a basic expectation: sampling reasoning traces from the well-trained variational posterior should depend critically on the quality of the conditioning hint y'. We generate noisy hints by using an LLM (Qwen3-4BB 1eer-----ruuut-2507 the original hints, introducing errors, and and use these noisy hints for sampling reasoning traces using the variational posterior. The results, shown in Table 14, confirm that providing noisy hints significantly degrades the quality of the sampled reasoning traces. This outcome validates our basic assumption, it demonstrates the sensitivity of the posterior its conditioning input, which is consistent with its theoretical role.
rewrite
F.9 EFFECTS OF THE VERIFIER'S 'S ACCURACY
To estimate qo (2 x y') for the weight Pk: we have two options: fikelihoodd-bsee estimator (and its variant with geometric mean modification) and an accuracy-based estimator. The likelihood- approach does not require verifier but relies on a reference answer, whereas the accuracy-base estimator depends verifier.
To examine the robustness of our method to verifier accuracy, we conduct an ablation study using a simulated, highly inaccurate "dummy verifier" by setting all (z |æt y²) values to 0.5. As shown in Table 15, while an inaccurate verifier can degrade performance, our method still outperforms the baselines. We attribute this robustness to the fact that the variational posterior is conditioned reasoning hint, which maintains a high overall correctness. Nevertheless, these results confirm that a more accurate verifier still preferable in practice.
29

---

Published as a conference paper at ICLR 2026
Table 14: Ablation study on the effects of the quality of hints used for sampling from the variational posterior. The best and second-bed -bst results are highlighted using bold text and underlined text. respectively.
MATH500
AIME24
AIME25
AMC23
OlympiadBench Avg@2
Method
Avg
Avg@2
Avg@3 Av
@2
Av
Qwen3-4B-Bal
45.30
4.79
5.73
27.73
23.37
21.38
General-Reasoneer
71.70
19.06
16.77
55.00
45.18
41.54
Bespoke Bespoke-Stratos-45 -trratos-BBtt
84.70
27.29
24.17
70.16
50.45
51.35
Ours-PB-Acc-4B
88.30
31.67
27.29
75.63
55.71
55.72
w/ noisy hints
86.10
27.29
24.27
70.00
50.52
51.64
GPQA-D
LCR-E
LCB-M
LCB-H
MMLU-Pro Avg@)
Method
Avg
Avg @8
Avg@8
Avg@8
Avg@8
Qwen3-4B-BB
29.10
18.54
5.46
1.32
36.89
18.26
40.97
61.40
17.90
2.85
61.36
36.90
Despoke-Stratoss
44.95
71.22
19.54
3.25
63.03
40.40
Ours-PB-AA MBssPBBAcc-BB
45.33
80.29
33.68
5.79
65,53
46.12
w/ noisy hints
44.82
74.24
26.21
4.98
64.18
42.88
F.10 EFFECTS GEOMETRIC MEAN MODIFICATION ON LIKELIHOOD RATIO
As derived in Section 2.2, the weight Pk used for training the reasoning model in Eq. (6) can be [ (m) decomposed as PI: go (..l(")))" Ty (Va zt.,, where the first term. g.(=)l.,y))
ma(=al)) is the likelihood
ratio the thinking trace Zk.
The standard likelihood ratio is unbounded, which can lead to high variance, and exhibits clear length bias as demonstrated in Figure (left). To address this, we heuristically introduce a geometric mean modification to the likelihood ratio, defined in Eq. 8. While this modification introduces estimation bias, we evaluate its practical utility by comparing our method against a variant that removes this modification. Results presented in Table 16 indicate that the geometric mean is indeed beneficial. The principled variant (strictly derived from variational inference) performs slightly worse than our modified version, yet still surpasses all baseline methods, confirming the overall robustness of the framework.
F.11 COMPARISON AGAINST BASELINES WITH MATCHED COMPUTATIONAL BUDGET
Our method involves training both a variational posterior and a reasoning model, and also includes ad- ditional sampling and forward passes compute token probabilities, which incurs extra computational cost. To further ensure fair comparison, we scaled the compute of the baseline, Bespoke-Stratos Stratos-4B. to exceed the total GPU hours used by our framework. This is achieved by increasing both its training epochs and batch size. Consequently, the baseline's total number of training tokens also largely exceeds that of method. The results, presented in Table 17, indicate that under similar computational budget, our approach achieves better performance. This demonstrates the practical value of variational reasoning.
30

---

Published as a conference paper at ICLR 2026
Table 15; Ablation study on the effects of the verifier's accuracy. The best and second becond-best results are highlighted using bold text and underlined text, respectively.
MATH500 AIME24
AIME25
AMC23 OlympiadBench
Method
Avg
Avg@2
Avg@32
Avg@ 32
Avg@32
Avg@2
Qwen3-4B-Ban
45.30
4.79
5.73
27.73
23.37
21.38
Ceneral-Reasonere
71.70
19.06
16.77
55.00
45.18
41.54
Bespoke-Stratos-4B1
84.70
27.29
24.17
70.16
50.45
51.35
Ours-PB-Acce4B
88.30
31.67
27.29
75.63
55.71
55.72
w/ dummy verifier
88.00
31.25
26.45
74.21
53.64
54.71
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@)
Method
Avg
Avg @8
Avg@8
Avg@8
Avg@8
Qwen3-4B-BB
29.10
18.54
5.46
1.32
36.89
18.26
Rearrner-Ba
40.97
61.40
17.90
2.85
61.36
36.90
Despoke-Stratosee
44.95
71.22
19.54
3.25
63.03
40.40
Ours-PB-Acc-4B
45.33
80.29
33.68
5.79
65.53
46.12
w/ dummy verifier
45.26
79.67
31.12
4.98
64.90
45.18
Table 16: Ablation study on the effects of geometric mean modification of the likelihood ratio. The best and second-bes bestt results are highlighted using bold text and underlined text, respectively.
MATH500 AIME24 AIME25 AMC23 OlympiadBench
Method
Avg
Avg@2
Avg@32
Avg@1 @32
Avg@32
Avg@2
Qwen3-4B-Base
45.30
4.79
5.73
27.73
23.37
21.38
71.70
19.06
16.77
55.00
45.18
41.54
Sespoke-Stratos-4p1
84.70
27.29
24.17
70.16
50.45
51.35
Ours-PB-Acc-4B
88.30
31.67
27.29
75.63
55.71
55.72
w/o geometric mean
88.00
32.60
27.60
74.14
52.30
54.93
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@1
Method
Avg
Avg@8
Av
Avg@8
Avg@8
Qwen3-4B-Base
29.10
18.54
5.46
1.32
36.89
18.26
General-Reasoner-411
40.97
61.40
17.90
2.85
61.36
36.90
Despoke-Stratos-te
44.95
71.22
19.54
3.25
63,03
40.40
Ours-PB-Acc-4B
45.33
80.29
33.68
5.79
65.53
46.12
w/o geometric mean
45.20
78.57
31.18
5.99
65.22
45.23
Table 17: Performance comparison under a similar computational budget. results are highlighted using bold text and underlined text, respectively.
The best and second-best
MATH500
AIME24
A1ME25
AMC23
OlympiadBench Avg@2
Method
Avg
Avg@2
Avg@32
Av
Avg@32
Qwen3-4B-Bas
45.30
4.79
5.73
27.73
23.37
21.38
Sypokk-Stratos-43
84.70
27.29
24.17
70.16
50.45
51.35
w/ scaled compute
86.20
29.79
25.52
72.81
52.30
53.32
Ours-PB-GMIL-..
87.30
33.54
26.77
74.06
54.45
55.23
Ours-PB-Acc-4B
88.30
31.67
27.29
75.63
55.71
55.72
GPQA-D
LCB-E
LCB-M
LCB-H
MMLU-Pro Avg@1
Method
Avg
Avg @
Avg@8
Avg@8
Avg@8
Qwen3-4B-Base
29.10
18.54
5,46
1.32
36.89
18.26
Bespoke-Stratos-Pe
44.95
71.22
19.54
3.25
63.03
40.40
w/ scaled compute
45.01
75.61
27.91
4.57
63,99
43.42
45.52
79.53
31.25
6.20
65.52
45.60
Ours-PB-Acc-49
45.33
80.29
33.68
5.79
65.53
46.12
31

---

Published as a conference paper at ICLR 2026
G EXAMPLES OF GENERATED RESPONSES
In this section, provide examples of responses from the variational posterior 9s and the final reasoning model Ty trained based on Qwen3-4B-Ban The provided example of the variational posterior 96 is based the variant with prompt template B, and the question-answe pair is one of the samples from the Bespoke-Stratos- model wa based on one of the samples MATH500. See the provided examples below.
dataset. The provided example of the final reasoning
Example of variational posterior 4
Prompt:
<lim_stare Psysten
Your role as an assistant invol ves reconst the internal reasoning process that connects a provided question to its correct answer. Your task is to methodically reverse-engineer the legical steps, demenstrating full cycle of analysis, summarization, idea generation, verification, error correction, and iterative refinement. Please structure your response into two distinct parts: Solution and Thought. In the Solution section, present the given correct answer in precise and clear format:
<|bogin_of_solution)) (provided correct solution) <lend_of_solutione
In the Thought section, articulate the step-by-step cognitive journey that leads to the solution. Use the speci fied format: < (detailed thought precess with steps separated by 'n'n') should reflect analytical breakaowns, synthesis key points, generation of logical pathways, validation of each step's accuracy, refinement of any missteps, and reassessment of previous conclusions. The focus is solely depicting the internal, structured thinking that arrives the provided solution.<lim end|> <lim_stards him_start/>user
thooughhtttt Each step
Return your final response within \bexed(). A figure is equiangular parallelogram if and only if it is a $\textbf( (A) (rectaaacag lequad \textbfl (B) \text (regular /qquya
polygon) /equad \textbīl ) Atext (rhombus) 'qquad \textbfl (D) \textdf( (B)) ) \text <
To determine the correct answer, we need to analyze the definitions and properties of the given options in relaticn to equiangular parallelogram.
1. *DDfinitions and Propertics**:
A ..parallelogramx, is a quadrilateral with both pairs of opposite sides parallel.
Exguuangularr means all angles are equal. In quadrilateral, degrees (since the sum of angles in quadrilateral is 360 degrees, and 360/4 90).
this means each angle
A **ectangge*** is a parallelogram with all angles equal to 90 degrees.
A r*guul polygon** (for quacrilaterale
square, which
both equiangular and squilateral.
A **rhombss** is parallelogram with all sides equal, but not necessarily equiangular.
A **quuare** is special rhombus, being both equiangular and equilateral.
both rectangle and
A **trapczoid** has at least one pair of parallel sides, but not necessarily two pairs.
2. +sanayysis
32

---

to satisfy the "if Published as a conference paper at ICLR 2026
An equianqular parallelogram must have all angles equal to 90 degrees, making it rectangle.
The "if and only if" condition requires that the converse also true: rectangle is an equiangular parallelogram.
A requiar polygon (quadrilateral) is a square, which is tco specific since the problem does not mention sides peing equal. A rhombus does not require all angles to be equal, necessarily equiangular.
it is not
A square is a special case of a rectangle but requires beth equiangular and equilateral properties, which are not both specified here.
A trapezoid does not necessarily have pairs of parallel sides, hence not a parallelogram.
3. *CConcussion*::
The correct answer is restangle, it satisfies both the equiangular condition and the properties parallelogram.
(boxed[A]
<lend of of_sutiionnn
[bbegin_of_thoughhh
Response:
Okay, 1et try figure out this problem. The questien is: "A figure an equiangular parallelogram if and only if it is a..." with options given. The answer choices are rectangle, regular polygon, rhombus, square, trapezoid. Hmm. So, first, let's s break down the terms here.
An equiangular parallelogram. Let's a recall definitions.
parallelogram is a quadrilateral with both pairs of opposite sides parallel. Equiangular means that all the angles are equal. In quadrilateral, 1s all angles degrees because the sum of angles in quadrilateral degrees. So, 360 divided by 4 is 90, So, an equiangular quadrilateral has four right angles. wait, but a parallelogran by definition ready has opposite angles equal. So, if parallelogram equiangular, that forces all angles degrees, right? So, that would make it a rectangle. Because rectangle is a parallelogram with all angles equal to 90 degrees.
equal, each angle must be 90
90
So, then, equiangular parallelogran is rectangle. Then the "if and only if" part. So, the statement is saying that figure an equiangular parallelogram exactly when the converse would also have to be true. That is, if it's à [answer), then it's an equiangular parallelogram, and vice versa.
a [answer]. So,
So, let's check the options. Option reetangle, then it's parallelogram with all angles equal (to 90 degrees), so it's an equiangular parallelogram. Conversely, if it's s an equiangular parallelogram, then as we saw, all angles are 90 degrees,
rectangle. If figure
it's restangle. Sc, that and only if" condition.
But wait, let's check the other options just to be thorough. Option B is regular polygon. A regular polygon is polygen that is both equiangular and equilateral. In the quadrilateral, regular polygon would be a square. So, square is a regular quadrilateral. But a parallclogram that is equiangular is rectangle, which isn't necessarily a square unless it's also equilateral (all sides equal). So, regular polygon is too specific here. Moreover, the question is abcut.
33

---

Published as a conference paper at ICLR 2026
parallelogram, which is quadrilateral, but "regular pelygen" could refer to any regular polygon, not necessarily quadrilateral, but in the answer choices, it's listed as square,
option. But since regular polygen (quadrilaterau) but the question is about an equiangular parallelogram, which is a rectangle. So, unless rectangles are considered regular pelygens, but rectangles so, regular polygon is not the correct answer here. 3o, is out.
not regular polygons unless they are squares.
Option C is rhombus. A rhombus is parallelogram with all sides equal. A rhombus doesn't have to be equiangular. In fact, a rhombus have angles of, say, 60 and 120 degrees. So, rhombus is not necessarily equiangular. So, an equiangular parallelegrem is a rectangle, not necessarily a rhembus. So, C is out.
Option is square. A square is parallelogram that is beth equiangular (all angles 90 degrees) and equilatens (all sides equal). So, the problem here is that an equiangular parallelogram is rectangle, which is only square if the sides are equal. nut the question doesn't specify that the sides are equal. So, an equiangular parallelogram
rectangle, but rectangle is square only if it's equilateral. Since the problem is only specifyine fying equiangular (all angles equal) and parallelogram, the figure is a rectangle, which is a type of parallelogram but not necessarily square, Therefore, the "if and only if" would require that being square is equivalent being equiangular parallelogram. But that's s not true because a square a special case. So, D is not correct.
Option is trapezoid. trapezeid is quadrilateral with least one pair of parallel sices. However, a parallelogram has two pairs of parallel sides, so a parallelogram is special case of trapezoid (depending on the definition; in some definiticns, trapezoids have exactly one pair). But even so, trapezcid isn't necessarily parallelogram. Moreover, an equianqular trapezoid would have all angles equal, but a trapezoid, which is not necessarily a parallelogram, having all angles equal doesn't guarantce that the sides are parallel. For example, an iscsceles trapezoid
be oquiangular (with two pairs of equal angles), but that's s different case. However, the question specifics an equiangular parallelogram. So, if it's a parallelogram, then it's already a quadrilateral with two pairs of parallel sides, and if it's equiangular, all angles are 90 degrees, making it rectangle. trapezoid isn't required to have two pairs parailel sices, so an equiangular trapezoid isn't necessarily a parallelogram. Therefore, is out.
So, the only remaining option is rectangle. let just verify again. it's rectangle, then it's parallelogram with angles equal, so equiangular. Conversely, if it's an equiangular parallelogram, all angles are 90 degrees, hence a rectangle. The "if and only if" holds here. So, the answer should be A.
Wait, but hold on. The problem says "a figure is an equiangular parallelogram if and only if it is a [answer]", So, this is a biconditional statement. So, the statement and the statement "it is [answer] must be equivalent. Therefore, the [answer] must be such that every equiangular parallelogran [answer] is an equiangular parallelogram.
[answer], and every
But as thought earlier, rectangle is equiangular parallelogram. However, the reverse is every rectangle equiangular parallelogram? Well, yes, because a rectangle by
34

---

Published as a conference paper at ICLR 2026
definition has four right angles, se it's equiangular. Se, the equivalence holds. Therefore, the if and only if is satisfied.
But wait, the answer options include square. So, maybe there's a confusion here. Let me think again. The question is asking for the figure that is an equiangular parallelogram it and only if it is [answer] So, the figure is parallelogram that is equiangular, and this happens exactly when it's [answer]. So, since we concluded that this [answer] is rectangle, then A is correct.
But let me check if any of the other options could alse satisfy the biconditional. For example, square. So, is square equiangular parallelogram? Yes, because a square parallelogram with all sides equal and all angles equal (90 degrees). However, the problem states "if and only if". Se, the figure is equiangular parallelogram exactly when it is square. But that's not true. Recause rectangle equiangular parallelogram, but a square is a special case of rectangle. so, if the statement was "if and only if it is square", then it would be too restrictive, because there are rectangles that squares. Therefore, the "if and only is" with square would not hold. similarly, regular polygon is too broad or specific. So, the correct answer is rectangle.
alsc
equiangular parallelograms which
not
Therefore, the answer should says "equiangular parallelogram". Wait, in some contexts, a parallelogram being equiangular is the same as being rectangle, so equiangular parallelogram = rectangle. Then the questien is asking, "An equiangular parallelogram is [answer] if and enly if it a..." Hmm, perhaps the woraing confusing. Let me parse the sentence again.
But wait, the original problem
"A figure is equiangular parallelogram if and only if it is [answer]*
So, the figure is defined as an equiangular parallelogram, and this is equivalent (if and only if) it is [answer] Se, the definition of equiangular parallclogram is "it is [answer]" " Therefore, the [answer] must be the name given to the figure that is exactly the equiangular parallelogram. Since we've ve concluded that an equiangular parallelogram is a rectangle, then the definition would be "a restangle is equiangular parallelegram", But the
options have rectangle if and only if is symnetric. So, saying "a figure is an equiangular parallelogram if and only if it is a rectangle" is correct. So, that would be answer A.
A. However, in Legie, the
But then why is square an option? Maybe made a mistake here. Let's consider that perhaps the question from country where parallelogram definitions differ. For example, in some definitions, parallelogram is quadrilateral with beth pairs of opposite sides parallel, and in these cases, parallelogram is only a rectangle is it has right angles. But standard definitions, a rectangle is a parallelogram with four right angles, so that's correct.
Alternativel vely, maybe there is fifferent interpretation. wait, the problem is the "if and only if" part. So, we take "equiangular parallelogram" as the definition, then the "if and only if" it is [answer]. So, the [answer] must be co-extensive with the equiangular parallelogram. In other words, the set equiangular parallelograms is the same as the set of [answer]. So,
35

---

Published as a conference paper at ICLR 2026
since we know that an equiangular parallelogram is a rectangle, but rectangle is not necessarily square (since rectangles can have sides of dirferent lengths), then the set of equiangular parallelograms exactly the set of rectangles. Theretore, the answer is rectangle.
But maybe the problem is trying to trick into thinking that square the answer because it's special case. But since problem doesn't mention sides being equal, just angles being equal, it's a rectangle. So, the answer should be A.
Alternatively, perhaps there's different approach. Let's recall that in a parallelogram, opposite angles are equal. equiangular, then all angles are equal. So, in a parallclogram, this forces all angles to be 90 segrees. Therefore, it's rectangle. Therefore, equiangular parallelogram is another for rectangle. So, if the question is asking for the term that describes such figure, the answer is rectangle. So, A.
it's
But maybe the options are labeled differently. Iet me check the options again:
A) rectangle
B) regular polygon
C) rhombus
D) square
E) trapezoid
Yes. So, given that, the answer is A. But 1et me verify ence more. Suppose someone thinks that "equiangular" refers both angles and sides. But no, equiangular just means all angles Equilateral would mean all sides are equal. So, the problem only menticning angles, So, a parallelogram that is equiangular must have all angles 90 degrees, hence a rectangle. Therefere, the answer is A.
equal.
think that's a solid, So, think the answer A, rectangle.
**Final Answer** (boxed(A)
Example of reasoning model n7
Prompt:
slim_start| |>systen
Your role as an assistant involves theroughly exploring questions through systematic long thinking process before providing the final precise and accurate solutions. This requires engaging in comprehensi cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop moll-considéreen thinking process. Please structure your response into two main sections: Thought and Solution. In the Thought section, detail your reasoning process using the specified format: <lbegin_off ohaggin_o__thouhht// (thought with steps separated with 'Annn') <f Each step should include detailed considcrations such as analisying questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previcus steps.
36

---

Published as a conference paper at ICLR 2026
In the Solution section, based on various attempts, explerations, and reflections from the Thought section, systematically present the final solution that you deem correct. The solution shculd remain Logicat, accurate, concise expression style and detail necessary step needed to reach the conclusion, formatted as follows: <lbegin_oo oi_souuiionn>>>> (final formatted, precise, and clear solution] send_of_solution] Now, try to solve the following question through the above guidelines: < [li_start/>user
If (x) = (x--2) what is the value of $f (-2) +f Express your answer common fraction < im_end] <lim_start start Passistant
Response:
Okay, I need to find the value of f(-2) + f(-1) + [() for the function fx) (3x 2) (x 2). Hmm, let me start by recalling how to evaluate a function at specific point. For each x-value given, substitute it into the function and simplify. Let me break this dow
step by step.
First, let's compute f--2). Substituting x = -2 into the function:
f(-2) [3* (-2) (]/[(-2) 2] - [(-6) 2]/(1/[]] (-4) - 2. wait, that seems scraightforwat
The numerator becomes and the denominator becomes -4, and a negative divided by negative is positive 2. Okay, so f(-2) is 2.
Next, let's find ((-1). substituting
-1:
f(-1) - [3+ (-1) - 2]/[(-1) 2] = ((-3) - 21/1-3] (-5) 5/3. Emm, negative divided by negative is positive, so 5/3. That makes sense.
Now, moving on to (0) Substitute x 0:
(0) = 13+0 21/10 - 21 = 10 21/1-21 = (-2) = 1. Okay, (() is 1.
Now, adding these three results together: 5/3 1. Let me convert the whole numbers to fractions to make addition easier. is 6/3, and 1 is 3/3. So adding them up:
6/3 3/3
(6 3) 14/3.
Wait, let me double-check each calculation to make sure I didn't make any mistakes. Starting with f(-2):
Numerator: (-2) -6, minus is -8, Denominator: -2 -4.
So -8/-4 is indeed 2, Correct.
For f(-1):
Numerator: 3+ (-1) -3, minus is -5, Denominator: -1 -3.
So -5/-3 is 5/3. Correct.
For f())
Numerator: 3+0 = 0, minus is -2. Denominator: 0 2 -2. So -2/-2 is 1. Correct.
37

---

Published as a conference paper at ICLR 2026
Adding them up: 2 + 5/3 + 1. Converting 2 and 1 to thirds: 2 is 6/3, 1 is 3/3. So 6/3 5/3 + 3/3 14/3. That seems right.
Alternatively, maybe can acd them in decimal form verify. is 2.0, 5/3 is approximately 1.666.... and 1 is 1.0. Adding those: 2.0 + 1.666... 1.0 = 4.666..., which is 14/3 (since 14 divided by 3 is approximately 4. 4.666...) So that checks out.
don't think I made any errors here. Each substitution was done correctly, and the arithmetic adds up. Therefore, the final answer should be 14/3.
oend_of_thought>
find / f(-2) + f(-1) (())) for V (x). (xrac(3x-2)
\)
1. #Calculate Ar #(-2) Vi..
Al
f(-2) (3 211-2 21 \frac1-6 2)(-)
(-rac(-8) 2 \]
2. #Calcullte \ E(-1) :)*.
VI
f(-1) = (-11) 2)11 - 2) = \frac[-3 - ()(3)
(-rac(-51(-31 = (5)
3. Calculltte A( (0) ****
MI
((C) (3rac () - 2
4. +um the results:** \[
\frac(5) 1 (frac(6)33 Afrac(5) (3) + \frac (3)
(fra \
The final answer is \(boxed (1)) (3))
<end of solution>
38


Reviews:
[{"review_id": "5gLiulcWim", "reviewer": "Reviewer_xKMW", "summary": "The paper proposes a variational inference framework for training LLM reasoning model by treating thinking traces as latent variables and maximizing evidence lower bound. Key innovations include an IWAE-style multi-trace objective that tightens the bound with multiple samples, as well as optimization via forward KL divergence to avoid mode collapse. Authors show existing methods like rejection sampling finetuning and GRPO can be treated as local forward KL objectives with implicit reweighting. Experiments on Qwen models show consistent improvements over baselines on math, coding and general reasoning benchmarks.", "questions": "See questions in weakness part", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Authors address the important topic of injecting reasoning into LLMs by providing a probabilistic perspective. Theorems and proofs are clear to understand. There are some parts of the method that require additional discussion (see weakness), but it is overall a well-polished work.\n* The paper provides a novel perspective on RFT/GRPO and can have impact on future works.\n* Experiments are well conducted on serveral benchmarks spanning different aspects of reasoning. Results show consistent gains across multiple model scales.", "weaknesses": "* The mode collapse issue mentioned in section 2.3 brings up an important research question for introducing latent reasoning in LLMs but may require additional elaboration. The proposed forward KL is claimed to prevent mode collapse, but weighted SFT still concentrate probability on high-accuracy traces. My understanding is that the model will still collapse to the \"correct\" modes while sacrificing diversity. I wonder if the authors can provide more insights and experiment results on this part.\n* Authors mention that the method currently is applied to only single iteration, while standard variational EMs alternate until close to convergence. Is there any reason why T>1 wasn't specifically tested? Will we observe certain collapse or divergence, as commonly observed in experiments with EM algorithms? Additional results can improve the credibility especially given that the paper's claim focuses on mode collapse/ diverse reasoning. \n* The paper is missing discussion on computational overhead from the method as compared to other baselines, especially with multiple rollouts and multiple answer samples per trace. \n* Could the authors provide some insights on the scaling properties of the method, especially with different model sizes and different K?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ib0YOscz6S", "reviewer": "Reviewer_7MH7", "summary": "This paper proposes a framework for training reasoning models by treating thinking traces as latent variables and optimizing the log-likelihood of correct answers using variational inference (VI). The authors derive an IWAE-style multi-trace objective (ELBO) to update the reasoning model $\\pi_{\\theta}$ and a forward-KL objective (Eq. 9) to stabilize the training of the variational posterior $q_{\\phi}$.\n\nA key theoretical contribution is showing that existing methods like Rejection Sampling Finetuning (RFT) and GRPO can be interpreted as local forward-KL objectives, revealing an implicit bias toward easier questions. Empirically, the method demonstrates strong performance, outperforming baselines like Bespoke-Stratos on several reasoning benchmarks.", "questions": "The author mentions directly ELBO formulation with reverse KL will have mode collapse. How bad is it? Do you have small scale observations?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel and Insightful Theoretical Framework: The paper's primary strength is its principled unification of several reasoning-as-inference methods (SFT, RL) under a single probabilistic lens. Framing reasoning as a latent variable problem is elegant.\n\n2. Valuable Unification and Bias Discovery: Showing that RFT and GRPO can be derived as forward-KL minimizations (Eq. 10 & 11) is a significant insight. The discovery of the implicit accuracy-weighting, and its resulting bias toward easier questions, is a valuable, non-obvious finding for the community that could influence future method design.\n\n3. Strong Empirical Performance: The method achieves impressive and consistent gains across multiple model scales (4B to 32B). It surpasses strong baselines, for instance, by over 8.5% on average accuracy compared to Bespoke-Stratos-4B on the same training data (Table 1), demonstrating its effectiveness.", "weaknesses": "1. Extreme Methodological and Computational Complexity: The proposed training pipeline (Algorithm 1) is substantially more complex than the baselines. As detailed in Appendix C, the full process requires (at minimum) three separate, full finetuning runs ($\\pi_{\\theta_0}$, $q_{\\phi}$, and $\\pi_{\\theta}$) plus a massive offline data generation and re-weighting pass. This re-weighting step (for the \"-Acc\" estimator) is itself a huge computational burden, requiring lots of verifications. \n\n2. The final algorithm essentially boils down to two complex, weighted SFT steps. This raises the critical question: is the VI formalism necessary? The argument for its necessity is weakened by the authors' own admission that they must modify the \"principled\" importance weight (Eq. 8, using a geometric mean) to mitigate high variance, thereby introducing bias and moving away from the pure ELBO derivation. The paper is missing the important ablation: a comparison against a simpler weighted SFT (e.g., using only verifier accuracy as a weight) to prove that the complex $\\pi_{\\theta} / q_{\\phi}$ importance-weighting is truly beneficial.\n\n3. While Appendix C provides training details, the paper presents its strong empirical results without a clear accounting of the total compute. The performance gains are achieved with what is likely an orders-of-magnitude increase in total compute over the SFT baseline. Without a compute-matched baseline (e.g., SFT on more data or for more epochs), it is impossible to assess the actual efficiency of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wjH5bHec4A", "reviewer": "Reviewer_b5vB", "summary": "The paper reframes reasoning in language models as a latent variable problem, where the intermediate “thinking trace” is treated as a hidden variable optimized through variational inference. Starting from the ELBO, the authors derive a tighter multi-trace objective and introduce a forward-KL formulation that stabilizes training of the variational posterior. This yields a unified probabilistic view that connects supervised finetuning, rejection sampling finetuning, and binary-reward RL as approximate inference objectives under the same framework. Empirically, the method consistently improves reasoning performance across math, code, and general benchmarks on Qwen models, showing both higher stability and generalization. Overall, it offers a principled bridge between likelihood-based reasoning and RL-style optimization.", "questions": "How does the variational posterior q_{\\phi} scale with model size? Is there evidence that a separate learned posterior remains necessary for very large models, or can it be amortized within the base model?\n\nHow sensitive is the proposed method to the quality or accuracy of the “hint” used in the variational posterior? Would the framework still hold when the hint is noisy, partially correct, or unavailable?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper offers a clean and principled unification of reasoning-oriented training methods for LLMs under a single probabilistic framework. By treating reasoning traces as latent variables and optimizing an IWAE-style ELBO, it grounds both RL and supervised finetuning as approximate inference procedures, clarifying their implicit assumptions. The forward-KL formulation is a well-motivated and technically strong addition that improves stability without sacrificing generality. The paper is clearly written and empirically supported with competitive results across diverse reasoning domains. Overall, it provides both conceptual clarity and practical value, advancing the theoretical understanding of reasoning optimization in large models. \n\nThere is a nice connection to older works of RL for non-LLM domains: see https://arxiv.org/pdf/1805.00909. They also use latent optimization via ELBO.", "weaknesses": "I believe the authors haven't done a full literature search on prior RL works, such as https://arxiv.org/pdf/1805.00909, https://openreview.net/pdf?id=a147pIS2Co that discuss a similar idea in the non-LLM setting. Could the authors explain the differences to these works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BPRVsL7rAs", "reviewer": "Reviewer_nKtU", "summary": "This paper applies variational inference to enhance language models' chain-of-thought reasoning by treating reasoning traces as latent variables and optimizing an ELBO-based objective. The authors further extend the objective to an IWAE-style multi-trace formulation for a tighter bound and introduce a stabilized training procedure using a forward-KL term. Experiments on math, coding, and general reasoning benchmarks show that this variational reasoning framework consistently outperforms SFT, RFT, and RL baselines, while also generalizing well to out-of-distribution tasks.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Leverages a principled variational inference framework to improve reasoning by treating thought traces as latent variables, providing a clean probabilistic formulation.\n- Uses a forward-KL based objective to address instability and collapse issues in training the variational posterior, leading to more stable optimization.\n- Algorithmic pipeline is clearly presented and appears reproducible, with explicit sampling, weighting, and update procedures.\n- Strong empirical evaluation across diverse reasoning tasks (math, coding, and general benchmarks), including out-of-distribution settings.\n- Comprehensive ablations that assess key design choices, such as the effect of the answer hint, different estimators for correctness probability, and varying the number of sampled traces.", "weaknesses": "- The forward-KL update is largely heuristic and does not come with formal guarantees on bias or convergence. While it stabilizes training in practice, the theoretical trade-offs and conditions under which the approximation holds are not fully analyzed.\n\n- The geometric-mean modification of the importance ratio reduces variance for long reasoning traces but may introduce systematic bias. The paper does not characterize the direction or magnitude of this bias, nor discuss potential failure modes.\n\n- The method relies on external verifiers to estimate correctness probabilities. The overall stability and performance depend on verifier quality; imperfect or incomplete verifiers may mislabel valid/invalid reasoning paths and influence learning dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EG9ArH18Co", "reviewer": "Reviewer_xe5w", "summary": "This paper proposes a framework for fine-tuning a reasoning LLM to generate better “thinking traces”. Specifically, this paper first trains an auxiliary LLM to use the answer as a hint to generate better thinking traces, then trains the original LLM to generate similar high-quality thinking traces without seeing any hint. The idea is put in a variational framework: the thinking traces are formulated as latent variables and the auxiliary LLM and original LLM are alternately optimized for ELBO. The auxiliary LLM is viewed as a variational posterior, and a forward-KL objective is used to optimize it to avoid collapsing to shortcut reasoning. A tighter IWAE‑style multi‑trace bound is used, and practical tricks are proposed for it to work. Besides, The authors show that RFT and binary‑reward RL including GRPO are local forward‑KL updates implicitly weighted by model accuracy, explaining a bias towards easier questions. On math/code/general‑knowledge benchmarks with Qwen 2.5/3 models, the method outperforms strong baselines and trains more stably.", "questions": "* Have you tried to use the variational posterior to generate high-quality thinking traces and use them to do simple SFT on the original LLM? How crucial are the VI/ELBO components in that case?\n* Is it possible to extend this method to scenarios where a rule-based verifier doesn't exist?\n* As a lightweight qualitative check, it would be interesting to compare the auxiliary model’s traces when conditioned on a correct vs. incorrect answer hint.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "* The proposed method is intuitive at a high level: to train an auxiliary LLM which can make use of a hint to generate better thinking trace, and use this model to improve the original one.\n* Many techniques are proposed to make the method work in practice, including the tighter IWAE-style lower bound, accuracy-based estimator for lower worst-case variances, forward-KL to avoid the variational posterior collapsing to trivial solution, etc.\n* The connection between RFT/binary reward RL and forward-KL is revealed, which also reveals their bias toward easier questions.", "weaknesses": "* The paper states that \"our work provides a principled probabilistic perspective that unifies variational inference with RL-style methods\". However, tricks like using geometric mean as a surrogate for the likelihood ratio and using forward-KL to optimize the variational posterior are not compatible with VI as far as I can see. I appreciate the method for its practical impact and clarity of intent, but I think connecting it to VI to make it \"principled\" may be overstating without additional justification or ablations.\n* The presentation is difficult to follow on a first read. Algorithm 1 greatly clarifies the pipeline, but it appears after substantial technical detail. Reordering the exposition (high‑level pipeline, objective, implementation details) could make the paper more accessible to newcomers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)