Variational Reasoning for Language Models

## Abstract

Abstract We introduce a variational reasoning framework for language models that treats thinking traces as latent variables and optimizes them through variational inference. Starting from the evidence lower bound (ELBO), we extend it to a multi-trace objective for tighter bounds and propose a forward-KL formulation that stabilizes the training of the variational posterior. We further show that rejection sampling finetuning and binary-reward RL, including GRPO, can be interpreted as local forward-KL objectives, where an implicit weighting by model accuracy naturally arises from the derivation and reveals a previously unnoticed bias toward easier questions. We empirically validate our method on the Qwen 2.5 and Qwen 3 model families across a wide range of reasoning tasks. Overall, our work provides a principled probabilistic perspective that unifies variational inference with RL-style methods and yields stable objectives for improving the reasoning ability of language models.

## 1 Introduction

Reasoning has recently become a central focus for large language models (LLMs), driving advances in tasks such as mathematics, coding, and scientific problem solving (Jaech et al., 2024; Comanici et al., 2025; Guo et al., 2025). A common strategy is to let models generate explicit thinking traces before producing final answers. To train such reasoning abilities, two dominant approaches are widely used: supervised finetuning (SFT) (Guha et al., 2025; Muennighoff et al., 2025) and reinforcement learning (RL) (Yu et al., 2025a; Liu et al., 2025; Zeng et al., 2025), both showing strong empirical success.

Despite this progress, each approach faces limitations. SFT often relies on curated long-thinking traces, which are costly to collect and, as an offline method, may struggle to generalize (Chu et al., 2025) or suffer from catastrophic forgetting (Shenfeld et al., 2025). Recent RL methods typically depend on verifiable rewards to mitigate reward hacking, yet training can be unstable and output diversity may collapse (Cheng et al., 2025; Cui et al., 2025b). As a result, correct answers to harder questions become increasingly rare, leading to lower Pass@K accuracy than base models (Yue et al., 2025a). These challenges motivate the search for a more principled objective for training reasoning models.

To this end, we propose to view reasoning through the lens of probabilistic modeling, where thinking traces are treated as *latent variables*. Variational inference (Kingma & Welling, 2013) provides a natural way to optimize the log-likelihood of producing correct answers. This perspective offers several advantages: it replaces the intractable marginalization over thinking traces with tractable lower bounds, enables multi-trace extensions that tighten the objective, and introduces a variational posterior that can sample thinking paths more likely to yield correct answers. In this way, it provides a principled objective for training reasoning models, while remaining compatible with verifiable rewards.

Building on this perspective, we develop a variational reasoning framework for language models in Section [2](https://arxiv.org/html/2509.22637v2#S2). The core idea is to decompose reasoning into a thinking trace and an answer, leading to the *maximum log-likelihood estimation (MLE)* objective. To make this optimization tractable, we introduce an evidence lower bound (ELBO) and extend it to an IWAE-style multi-trace formulation (Burda et al., 2015), which tightens with more rollouts. To further stabilize the training of the variational posterior, we propose a forward-KL objective that prevents collapse and makes better use of answer hints. Together, these components form a unified training pipeline (as shown in Algorithm [1](https://arxiv.org/html/2509.22637v2#alg1)) that jointly improves the reasoning model and the variational posterior.

Beyond the method itself, our framework also helps interpret existing approaches, as described in Section [3](https://arxiv.org/html/2509.22637v2#S3). Rejection-sampling finetuning (RFT) (Dong et al., 2023; Touvron et al., 2023) can be re-expressed as forward-KL optimization weighted by model accuracy, and binary-reward RL, including GRPO (Shao et al., 2024), admits a similar form. Our analysis shows that this weighting by accuracy arises implicitly and produces a systematic bias toward easier questions, an effect that has not been explicitly recognized before. By placing these methods under a shared probabilistic view, our framework provides principled objectives and clarifies the behavior of widely used methods.

We validate our framework on the Qwen2.5 and Qwen3 model families (Yang et al., 2024; Team, 2025b) and observe consistent improvements over strong baselines across diverse reasoning benchmarks, including MATH500, AIME24&25, OlympiadBench, LiveCodeBench, GPQA-Diamond, and MMLU-Pro. Due to space constraints, a detailed discussion of related work is deferred to Appendix [B](https://arxiv.org/html/2509.22637v2#A2).

## 2 Variational reasoning

Let $\mathcal{V}^{*}$ denote the set of all prompt strings over the vocabulary $\mathcal{V}$. Given an input question ${\bm{x}}\in\mathcal{V}^{*}$, a reasoning model $\pi_{\theta}({\bm{z}},{\bm{y}}|{\bm{x}})$ generates both a thinking process ${\bm{z}}\in\mathcal{V}^{*}$ and a predicted answer ${\bm{y}}\in\mathcal{V}^{*}$. The joint probability can be written as $\pi_{\theta}({\bm{z}},{\bm{y}}|{\bm{x}})=\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\cdot\pi_{\theta}({\bm{z}}|{\bm{x}})$. Following a standard format template (Guo et al., 2025), these two conditional terms are computed as(^1^11We will omit special tokens such as </think> and </answer> in the formulas without ambiguity.)

$$ $\begin{split}\pi_{\theta}\left({\bm{z}}|{\bm{x}}\right)&=\pi_{\theta}\big(\left[{\bm{z}},\texttt{</think>}\right]\mid\left[{\bm{x}},\texttt{<think>}\right]\big)\textrm{;}\\ \pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})&=\pi_{\theta}\big(\left[{\bm{y}},\texttt{</answer>}\right]\mid\left[{\bm{x}},\texttt{<think>},{\bm{z}},\texttt{</think>},\texttt{<answer>}\right]\big)\textrm{,}\end{split}$ (1) $$

where </think> and </answer> serve as the end-of-sequence markers for ${\bm{z}}$ and ${\bm{y}}$, respectively. We define the marginal distribution $P_{\theta}({\bm{y}}|{\bm{x}})=\sum_{{\bm{z}}}\pi_{\theta}({\bm{z}},{\bm{y}}|{\bm{x}})=\sum_{{\bm{z}}}\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}})$, where the notation $P_{\theta}({\bm{y}}|{\bm{x}})$ highlights that this distribution is *induced* by $\pi_{\theta}$. This is different from the non-thinking probability $\pi_{\theta}({\bm{y}}|{\bm{x}})$, which does not marginalize over possible thinking traces.

### 2.1 Evidence lower bound

Let $\mathcal{Y}_{{\bm{x}}}\subset\mathcal{V}^{*}$ denote the oracle set (possibly infinite) of correct answers to the question ${\bm{x}}$. The marginal probability that $\pi_{\theta}$ generates a correct answer is $P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})=\sum_{{\bm{y}}\in\mathcal{Y}_{{\bm{x}}}}P_{\theta}({\bm{y}}|{\bm{x}})$. Maximizing this probability gives the *maximum log-likelihood estimation (MLE)* objective: $\max_{\theta}\log P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$. However, this MLE objective is intractable because computing $P_{\theta}({\bm{y}}|{\bm{x}})$ requires summing over all possible thinking traces ${\bm{z}}$. To make learning feasible, we apply variational inference (Kingma & Welling, 2013) to derive an *evidence lower bound (ELBO)*:

$$ $\begin{split}\log P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})&=\log\sum\nolimits_{{\bm{z}}}\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}})\\ &=\log\mathbb{E}_{q_{\phi}({\bm{y}}^{\prime})}\mathbb{E}_{q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\frac{\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}})}{q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\right]\\ &\geq\mathbb{E}_{q_{\phi}({\bm{y}}^{\prime})}\underbrace{\left[\mathbb{E}_{q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\log\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\right]-\mathbb{D}_{\textrm{KL}}\left(q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})||\pi_{\theta}({\bm{z}}|{\bm{x}})\right)\right]}_{\definecolor{tcbcolback}{rgb}{0.9296875,0.9296875,0.9296875}\hbox to75.32pt{\vbox to9.69pt{\pgfpicture\makeatletter\hbox{\thinspace\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{0.25,0.25,0.25}\pgfsys@color@gray@fill{0.25}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{3.0pt}\pgfsys@lineto{0.0pt}{6.69281pt}\pgfsys@curveto{0.0pt}{8.34969pt}{1.34312pt}{9.69281pt}{3.0pt}{9.69281pt}\pgfsys@lineto{72.32011pt}{9.69281pt}\pgfsys@curveto{73.97699pt}{9.69281pt}{75.32011pt}{8.34969pt}{75.32011pt}{6.69281pt}\pgfsys@lineto{75.32011pt}{3.0pt}\pgfsys@curveto{75.32011pt}{1.34312pt}{73.97699pt}{0.0pt}{72.32011pt}{0.0pt}\pgfsys@lineto{3.0pt}{0.0pt}\pgfsys@curveto{1.34312pt}{0.0pt}{0.0pt}{1.34312pt}{0.0pt}{3.0pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{0.9296875,0.9296875,0.9296875}\pgfsys@color@rgb@fill{0.9296875}{0.9296875}{0.9296875}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{3.0pt}\pgfsys@lineto{0.0pt}{6.69281pt}\pgfsys@curveto{0.0pt}{8.34969pt}{1.34312pt}{9.69281pt}{3.0pt}{9.69281pt}\pgfsys@lineto{72.32011pt}{9.69281pt}\pgfsys@curveto{73.97699pt}{9.69281pt}{75.32011pt}{8.34969pt}{75.32011pt}{6.69281pt}\pgfsys@lineto{75.32011pt}{3.0pt}\pgfsys@curveto{75.32011pt}{1.34312pt}{73.97699pt}{0.0pt}{72.32011pt}{0.0pt}\pgfsys@lineto{3.0pt}{0.0pt}\pgfsys@curveto{1.34312pt}{0.0pt}{0.0pt}{1.34312pt}{0.0pt}{3.0pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{2.0pt}{2.95277pt}\pgfsys@invoke{ }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\ignorespaces{$\mathcal{L}_{\textrm{ELBO}}({\bm{x}},\mathcal{Y}_{{\bm{x}}},{\bm{y}}^{\prime};\pi_{\theta},q_{\phi})$}}}}}\pgfsys@invoke{ }\pgfsys@endscope}\pgfsys@invoke{ }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{ }\pgfsys@endscope\hss}}\endpgfpicture}}}\textrm{.}\end{split}$ (2) $$

In this expression, $\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})$ denotes the probability of producing a correct answer given the question ${\bm{x}}$ and a particular thinking trace ${\bm{z}}$. The distribution $q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})$ is the variational posterior, which conditions not only on the question ${\bm{x}}$ but also on an auxiliary answer hint ${\bm{y}}^{\prime}$:

$$ $q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})=q_{\phi}\big([{\bm{z}},\texttt{</think>}]\mid[{\bm{x}},\texttt{<hint>},{\bm{y}}^{\prime},\texttt{</hint>},\texttt{<think>}]\big)\textrm{.}$ (3) $$

Here, <hint> and </hint> are shown as example delimiters; in experiments, we ablate different special tokens to wrap the hint ${\bm{y}}^{\prime}$ and concatenate it after ${\bm{x}}$. Conditioning on ${\bm{y}}^{\prime}$ encourages the variational posterior to generate thinking traces ${\bm{z}}$ that are more likely to yield correct answers. A simple yet effective design choice is to let ${\bm{y}}^{\prime}$ come directly from the oracle set, that is, $\operatorname{supp}[q_{\phi}({\bm{y}}^{\prime})]\subset\mathcal{Y}_{{\bm{x}}}$. In practice, ${\bm{y}}^{\prime}$ may be a rephrasing of a reference answer or any correct expression sampled from $\mathcal{Y}_{{\bm{x}}}$.

We can further show (detailed in Appendix [A.1](https://arxiv.org/html/2509.22637v2#A1.SS1)) that maximizing the ELBO objective w.r.t. $q_{\phi}$ in Eq. ([2](https://arxiv.org/html/2509.22637v2#S2.E2)) is equivalent to minimizing the *reverse* KL divergence between $q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})$ and $P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})$:

$$ $\mathcal{L}_{\textrm{ELBO}}({\bm{x}},\mathcal{Y}_{{\bm{x}}},{\bm{y}}^{\prime};\pi_{\theta},q_{\phi})=\log P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})-{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbb{D}_{\textrm{KL}}\left(q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})||P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})\right)}\textrm{.}$ (4) $$

Here $P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})=\frac{\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}})}{P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}$ is the true posterior. Compared with the prior distribution $\pi_{\theta}({\bm{z}}|{\bm{x}})$, this posterior distribution re-weights thinking traces by $\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})$, thus favoring ${\bm{z}}$ that are more likely to produce correct answers. According to Eq. ([4](https://arxiv.org/html/2509.22637v2#S2.E4)), we know that the optimal solution for $\max_{q_{\phi}}\mathcal{L}_{\textrm{ELBO}}({\bm{x}},\mathcal{Y}_{{\bm{x}}},{\bm{y}}^{\prime};\pi_{\theta},q_{\phi})$ is: $\forall{\bm{y}}^{\prime}\sim q_{\phi}({\bm{y}}^{\prime})$, there is $q_{\phi}^{*}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})=P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})$.

### 2.2 Extension to IWAE-style lower bound

In reinforcement learning (RL), it is now common practice to perform parallel rollouts of multiple thinking traces ${\bm{z}}$ and answers ${\bm{y}}$ for a given question ${\bm{x}}$ (Shao et al., 2024). This naturally motivates us to extend the single-trace ELBO in Eq. ([4](https://arxiv.org/html/2509.22637v2#S2.E4)) to an importance-weighted autoencoder (IWAE) style bound (Burda et al., 2015). By leveraging multiple $K$ traces, this approach yields a strictly tighter lower bound. Specifically, we obtain the following *IWAE-style lower bound* for $\log P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$:

$$ $\mathcal{L}_{\textrm{ELBO}}^{K}({\bm{x}},\mathcal{Y}_{{\bm{x}}},{\bm{y}}^{\prime};\pi_{\theta},q_{\phi})=\mathbb{E}_{{\bm{z}}_{1:K}\sim q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\log\frac{1}{K}\sum_{k=1}^{K}\frac{\pi_{\theta}({\bm{z}}_{k},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}{q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})}\right]\textrm{.}$ (5) $$

These IWAE-style bounds satisfy $\mathcal{L}_{\textrm{ELBO}}^{K}\leq\mathcal{L}_{\textrm{ELBO}}^{K+1}\leq\log P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$ for any $K\in\mathbb{N}^{+}$, which means the bound becomes tighter as $K$ increases (the proof is similar to that of Burda et al. (2015)). The single-trace ELBO objective in Eq. ([4](https://arxiv.org/html/2509.22637v2#S2.E4)) corresponds to the special case of $K=1$, i.e., $\mathcal{L}_{\textrm{ELBO}}=\mathcal{L}_{\textrm{ELBO}}^{1}$.

Gradient estimation. We now derive the gradient of $\mathcal{L}_{\textrm{ELBO}}^{K}({\bm{x}},\mathcal{Y}_{{\bm{x}}},{\bm{y}}^{\prime};\pi_{\theta},q_{\phi})$ w.r.t. the model parameters $\theta$ (see Appendix [A.2](https://arxiv.org/html/2509.22637v2#A1.SS2) for the gradient w.r.t. the variational parameters $\phi$, i.e., $\nabla_{\phi}\mathcal{L}_{\textrm{ELBO}}^{K}$):

$$ $\begin{split}&\nabla_{\theta}\mathcal{L}_{\textrm{ELBO}}^{K}({\bm{x}},\mathcal{Y}_{{\bm{x}}},{\bm{y}}^{\prime};\pi_{\theta},q_{\phi})=\mathbb{E}_{{\bm{z}}_{1:K}\sim q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\sum_{k=1}^{K}\widetilde{\rho}_{k}\nabla_{\theta}\log\pi_{\theta}({\bm{z}}_{k},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})\right]\textrm{,}\\ &\textrm{where}\quad\widetilde{\rho}_{k}=\frac{\rho_{k}}{\sum_{j=1}^{K}\rho_{j}}\quad\text{and}\quad\rho_{k}=\frac{\pi_{\theta}({\bm{z}}_{k},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}{q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})}\textrm{.}\end{split}$ (6) $$

Estimating $\rho_{k}$. The weight $\rho_{k}$ in Eq. ([6](https://arxiv.org/html/2509.22637v2#S2.E6)) can be decomposed as $\rho_{k}=\frac{\pi_{\theta}({\bm{z}}_{k}|{\bm{x}})}{q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})}\cdot\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}}_{k})$, where the first term, $\frac{\pi_{\theta}({\bm{z}}_{k}|{\bm{x}})}{q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})}$, is the *likelihood ratio of the thinking trace* ${\bm{z}}_{k}$, and the second term, $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}}_{k})$, is the probability of producing a correct answer given ${\bm{x}}$ and ${\bm{z}}_{k}$. In reasoning models, a single trace ${\bm{z}}_{k}$ may contain thousands of tokens. Directly computing the likelihood ratio over such long sequences often leads to high variance, a phenomenon also reported in concurrent studies (Cetin et al., 2025; Zheng et al., 2025). To mitigate this issue, we use the geometric mean $\big(\tfrac{\pi_{\theta}({\bm{z}}_{k}|{\bm{x}})}{q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})}\big)^{1/|{\bm{z}}_{k}|}$ as a surrogate for the likelihood ratio of ${\bm{z}}_{k}$. This per-token normalization reduces variance at the cost of introducing some bias, effectively spreading the ratio evenly across the thinking tokens.

Figure: Algorithm 1 Training pipeline of variational reasoning

As for computing $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}}_{k})$, we consider two unbiased estimators: (i) likelihood-based estimator is $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})=|\mathcal{Y}_{\bm{x}}|\cdot\mathbb{E}_{{\bm{y}}\sim\mathcal{U}(\mathcal{Y}_{\bm{x}})}\left[\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\right]$, where $|\mathcal{Y}_{\bm{x}}|$ is cardinality of $\mathcal{Y}_{\bm{x}}$ and $\mathcal{U}(\mathcal{Y}_{\bm{x}})$ is the uniform distribution on $\mathcal{Y}_{\bm{x}}$; (ii) accuracy-based estimator is $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})=\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\right]$, where $\mathds{1}(\cdot)$ is the indicator function. When $|\mathcal{Y}_{\bm{x}}|=1$, i.e., there is a unique correct answer expression ${\bm{y}}^{*}$, Zhou et al. (2025) show that the likelihood-based estimator has lower variance (in fact, zero) compared to the accuracy-based one. We now extend this comparison to general cases when $|\mathcal{Y}_{\bm{x}}|>1$:

###### Theorem 1 .

(Proof in Appendix [A.3](https://arxiv.org/html/2509.22637v2#A1.SS3)) For $|\mathcal{Y}_{\bm{x}}|>1$, the worst-case variances of the likelihood-based estimator and the accuracy-based estimator over all possible $\pi_{\theta}$ (under fixed $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})$) are

$$ $\max_{\pi_{\theta}}\textrm{Var}_{\textrm{like}}=\left(|\mathcal{Y}_{\bm{x}}|-1\right)\cdot\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})^{2}\textnormal{;}\quad\max_{\pi_{\theta}}\textrm{Var}_{\textrm{acc}}=\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})\cdot\left(1-\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})\right)\textrm{.}$ (7) $$

Therefore, the accuracy-based estimator has lower worst-case variance, i.e., $\max_{\pi_{\theta}}\textrm{Var}_{\textrm{acc}}\leq\max_{\pi_{\theta}}\textrm{Var}_{\textrm{like}}$, whenever the model accuracy (conditional on ${\bm{x}},{\bm{z}}$) satisfies $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})\geq\frac{1}{|\mathcal{Y}_{\bm{x}}|}$.

Note that for many practical questions, the space of correct answers can be quite flexible, so typically $|\mathcal{Y}_{\bm{x}}|\gg 1$. In this regime, the accuracy-based estimator enjoys much lower worst-case variance. Based on this insight, in our experiments we estimate the weight $\rho_{k}$ as

$$ $\rho_{k}^{\textrm{est}}=\big(\tfrac{\pi_{\theta}({\bm{z}}_{k}|{\bm{x}})}{q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})}\big)^{1/|{\bm{z}}_{k}|}\cdot\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}}_{k})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\right]\textrm{,}$ (8) $$

where the expectation $\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}}_{k})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\right]$ is approximated by sampling multiple candidate answers for each thinking trace ${\bm{z}}_{k}$, similar to the implementation in Qi et al. (2025).

Estimating $\nabla_{\theta}\log\pi_{\theta}({\bm{z}}_{k},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$. When evaluating $\nabla_{\theta}\mathcal{L}_{\textrm{ELBO}}^{K}$ in Eq. ([6](https://arxiv.org/html/2509.22637v2#S2.E6)), we need the gradient $\nabla_{\theta}\log\pi_{\theta}({\bm{z}}_{k},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})=\nabla_{\theta}\log\pi_{\theta}({\bm{z}}_{k}|{\bm{x}})+\nabla_{\theta}\log\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}}_{k})$. The first term, $\nabla_{\theta}\log\pi_{\theta}({\bm{z}}_{k}|{\bm{x}})$, is straightforward to calculate. For the second term, $\nabla_{\theta}\log\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}}_{k})$, we also adopt an *accuracy-based estimator*: $\nabla_{\theta}\log\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}}_{k})=\frac{\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}}_{k})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}}_{k})\right]}{\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}}_{k})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\right]}$. In practice, the expectations w.r.t. $\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}}_{k})$ are approximated using the same samples drawn to estimate $\rho_{k}^{\mathrm{est}}$.

###### Theorem 1 .

### 2.3 Optimizing the variational posterior via forward KL divergence

While Eq. ([5](https://arxiv.org/html/2509.22637v2#S2.E5)) provides IWAE-style bounds that yield tighter optimization of the MLE objective w.r.t. the model parameters $\theta$ (through $\nabla_{\theta}\mathcal{L}_{\mathrm{ELBO}}^{K}$), our pilot experiments show unexpected behavior for the optimization of the variational parameters $\phi$ (through $\nabla_{\phi}\mathcal{L}_{\mathrm{ELBO}}$ or $\nabla_{\phi}\mathcal{L}_{\mathrm{ELBO}}^{K}$). Recall from Eq. ([4](https://arxiv.org/html/2509.22637v2#S2.E4)) that the ELBO objective minimizes the *reverse* KL divergence $\mathbb{D}_{\textrm{KL}}\left(q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})||P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})\right)$, where both the expectation and Monte Carlo samples are taken under $q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})$. In practice, however, the policy model $\pi_{\theta}({\bm{z}}|{\bm{x}})$ is often already well-trained due to pretraining of base LLMs (Liu et al., 2025), while the variational posterior $q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})$ may *struggle to effectively use hints ${\bm{y}}^{\prime}$ from correct answers without collapsing into shortcut reasoning* (e.g., directly leaking answer tokens into the thinking trace). To address this imbalance, we propose to optimize $q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})$ using the *forward* KL divergence $\mathbb{D}_{\textrm{KL}}(P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})||q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime}))$, whose gradient w.r.t. $\phi$ can be written as:(^2^22We assume that $q_{\phi}({\bm{y}}^{\prime})$ is a fixed prior distribution and does not involve gradients.)

$$ $\begin{split}\!\!\!&\nabla_{\phi}\mathbb{D}_{\textrm{KL}}(P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})||q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime}))\simeq\mathbb{E}_{{\bm{z}}_{1:M}\sim\pi_{\theta}({\bm{z}}|{\bm{x}})}\!\!\left[\sum_{m=1}^{M}\!\widetilde{w}_{m}\nabla_{\phi}\log q_{\phi}({\bm{z}}_{m}|{\bm{x}},{\bm{y}}^{\prime})\right]\triangleq\nabla_{\phi}\mathcal{L}_{\textrm{forward}}^{M}\textrm{,}\!\!\!\!\!\!\!\!\!\!\\ \!\!\!&\textrm{where}\quad\widetilde{w}_{m}=\frac{w_{m}}{\sum_{j=1}^{M}w_{j}}\quad\text{and}\quad w_{m}=\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}}_{m})=\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}}_{m})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\right]\textrm{.}\!\!\!\!\!\!\!\!\!\!\end{split}$ (9) $$

This approximation, $\nabla_{\phi}\mathcal{L}_{\textrm{forward}}^{M}$, follows a derivation similar to Bornschein & Bengio (2015) (see Appendix [A.4](https://arxiv.org/html/2509.22637v2#A1.SS4)), with the sample size $M$ not necessarily equal to $K$ used in $\nabla_{\theta}\mathcal{L}_{\mathrm{ELBO}}^{K}$. Unlike IWAE, this objective is an approximation rather than a lower bound. Optimizing Eq. ([9](https://arxiv.org/html/2509.22637v2#S2.E9)) can be viewed as a weighted supervised finetuning (SFT) for $q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})$, where training data is sampled from $\pi_{\theta}({\bm{z}}|{\bm{x}})$.

We summarize the overall training pipeline of our variational reasoning method in Algorithm [1](https://arxiv.org/html/2509.22637v2#alg1). In our experiments, we train for only a single round ($T=1$), leaving multi-round training as an interesting direction for future work.

## 3 Connection to other methods

In prior work, the thinking trace ${\bm{z}}$ and the final answer ${\bm{y}}$ are often treated together as the full output in the formulations (Shao et al., 2024; Guo et al., 2025; Liu et al., 2025; Wu et al., 2025). By explicitly decomposing the output into a thinking process ${\bm{z}}$ and an answer ${\bm{y}}$, as we have done above, we can gain fresh perspectives on how our formulation relates to other mainstream methods.

Connection to rejection sampling finetuning (RFT). RFT methods (Dong et al., 2023; Touvron et al., 2023) generate multiple candidate outputs for each input ${\bm{x}}$ using a reference model $\pi_{\textrm{ref}}$, and then select the candidate with the highest reward. The reference model may be a strong teacher model or identical to the learner $\pi_{\theta}$ (i.e., $\pi_{\textrm{ref}}=\pi_{\theta}^{\mathrm{sg}}$, with $\mathrm{sg}$ denoting stop-gradient). Formally, the gradient of RFT training objective, *focusing only on the learning of the thinking trace ${\bm{z}}$*, can be written as:

$$ $\begin{split}\nabla_{\theta}\mathcal{L}_{\textrm{RFT}}({\bm{x}},\pi_{\theta})\triangleq{}&\nabla_{\theta}\mathbb{E}_{\pi_{\textrm{ref}}({\bm{z}}|{\bm{x}})}\mathbb{E}_{\pi_{\textrm{ref}}({\bm{y}}|{\bm{x}},{\bm{z}})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\cdot\log\pi_{\theta}({\bm{z}},{\bm{y}}|{\bm{x}})\right]\\ ={}&\nabla_{\theta}\mathbb{E}_{\pi_{\textrm{ref}}({\bm{z}}|{\bm{x}})}\left[\pi_{\textrm{ref}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\cdot\left(\log\pi_{\theta}({\bm{z}}|{\bm{x}}){\color[rgb]{0.66796875,0.66796875,0.66796875}\definecolor[named]{pgfstrokecolor}{rgb}{0.66796875,0.66796875,0.66796875}{}+\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\right)\right]\\ \xRightarrow[\scriptstyle\pi_{\theta}({\bm{z}}|{\bm{x}})]{\textrm{only w.r.t.}}{}&\nabla_{\theta}\mathbb{E}_{\pi_{\textrm{ref}}({\bm{z}}|{\bm{x}})}\left[\pi_{\textrm{ref}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\cdot\log\pi_{\theta}({\bm{z}}|{\bm{x}})\right]\\ ={}&-{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}P_{\textrm{ref}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}\cdot\nabla_{\theta}\mathbb{D}_{\textrm{KL}}(P_{\textrm{ref}}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})||\pi_{\theta}({\bm{z}}|{\bm{x}}))\textrm{.}\end{split}$ (10) $$

Here $P_{\textrm{ref}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})=\sum_{{\bm{y}}\in\mathcal{Y}_{{\bm{x}}}}P_{\textrm{ref}}({\bm{y}}|{\bm{x}})$ denotes the *model accuracy* on instruction ${\bm{x}}$, and the true posterior of the reference model is $P_{\textrm{ref}}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})=\frac{\pi_{\textrm{ref}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\pi_{\textrm{ref}}({\bm{z}}|{\bm{x}})}{P_{\textrm{ref}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}$. As seen, the RFT objective can be viewed as maximizing a forward KL divergence weighted by $P_{\textrm{ref}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$, with the optimal solution $\pi_{\theta}^{*}({\bm{z}}|{\bm{x}})=P_{\textrm{ref}}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})$. In practice, this weighting downplays hard questions with small $P_{\textrm{ref}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$, biasing training toward easier ones. In contrast, our formulation in Eq. ([9](https://arxiv.org/html/2509.22637v2#S2.E9)) treats all questions more evenly, ensuring that the objective remains attentive to difficult cases.

Connection to binary reward RL. In the case of RL training with a 0–1 binary reward, the training objective can be written in a form similar to Eq. ([10](https://arxiv.org/html/2509.22637v2#S3.E10)), focusing only on the thinking trace ${\bm{z}}$:

$$ $\begin{split}\nabla_{\theta}\mathcal{L}_{\textrm{bi-RL}}({\bm{x}},\pi_{\theta})\triangleq{}&\nabla_{\theta}\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\mathbb{E}_{\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\right]\\ \xRightarrow[\scriptstyle\pi_{\theta}({\bm{z}}|{\bm{x}})]{\textrm{only w.r.t.}}{}&\nabla_{\theta}\mathbb{E}_{\pi_{\theta}^{\mathrm{sg}}({\bm{z}}|{\bm{x}})}\left[\pi_{\theta}^{\mathrm{sg}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\cdot\log\pi_{\theta}({\bm{z}}|{\bm{x}})\right]\\ ={}&-{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}P_{\theta}^{\mathrm{sg}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}\cdot\nabla_{\theta}\mathbb{D}_{\textrm{KL}}(P_{\theta}^{\mathrm{sg}}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})||\pi_{\theta}({\bm{z}}|{\bm{x}}))\textrm{.}\end{split}$ (11) $$

Thus, the local gradient for $\pi_{\theta}({\bm{z}}|{\bm{x}})$ in binary-reward RL is equivalent to minimizing the forward KL divergence between $\pi_{\theta}({\bm{z}}|{\bm{x}})$ and the true posterior $P_{\theta}^{\mathrm{sg}}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})=\frac{\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}})}{P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}$, with the update further weighted by the model accuracy $P_{\theta}^{\mathrm{sg}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$. Interestingly, the RL training objective itself is $\mathcal{L}_{\textrm{bi-RL}}({\bm{x}},\pi_{\theta})=\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\mathbb{E}_{\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\right]=P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$, whereas the MLE objective in Eq. ([2](https://arxiv.org/html/2509.22637v2#S2.E2)) for our variational reasoning framework instead maximizes $\log P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$.

Furthermore, in Group Relative Policy Optimization (GRPO) (Shao et al., 2024), one of the most widely used RL training objectives, the reward is normalized by the standard deviation of rewards within a group of rollouts. Under 0–1 binary reward, each rollout reward follows a Bernoulli distribution with mean $P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$ and standard deviation $\sqrt{P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})\cdot(1-P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}}))}$. The gradient of the GRPO objective can therefore be derived as (we omit the $\min$ and clip operations for brevity)

$$ $\nabla_{\theta}\mathcal{L}_{\textrm{bi-GRPO}}({\bm{x}},\pi_{\theta})\xRightarrow[\scriptstyle\pi_{\theta}({\bm{z}}|{\bm{x}})]{\textrm{only w.r.t.}}-{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\sqrt{\frac{P_{\theta}^{\mathrm{sg}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}{1-P_{\theta}^{\mathrm{sg}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}}}\cdot\nabla_{\theta}\mathbb{D}_{\textrm{KL}}(P_{\theta}^{\mathrm{sg}}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})||\pi_{\theta}({\bm{z}}|{\bm{x}}))\textrm{,}$ (12) $$

which follows from $\nabla_{\theta}\mathcal{L}_{\textrm{bi-GRPO}}({\bm{x}},\pi_{\theta})=\nabla_{\theta}\mathcal{L}_{\textrm{bi-RL}}({\bm{x}},\pi_{\theta})/\sqrt{P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})\cdot(1-P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}}))}$. Thus, the local gradient for $\pi_{\theta}({\bm{z}}|{\bm{x}})$ in GRPO is still equivalent to minimizing the forward KL divergence between $\pi_{\theta}({\bm{z}}|{\bm{x}})$ and the true posterior $P_{\theta}^{\mathrm{sg}}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})$. Note that the per-instruction weight becomes $\sqrt{P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})/(1-P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}}))}$, which increases monotonically with model accuracy and therefore also emphasizes easier questions with higher $P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$. In Appendix [A.5](https://arxiv.org/html/2509.22637v2#A1.SS5), we extend these analyses and derive gradients for more general RL reward shaping, including cases with a *format reward*.

## 4 Experiments

Datasets. We train on the Bespoke-Stratos-17k dataset curated by Li et al. (2025a), which combines math problems from Numina-Math (Li et al., 2024) with code problems from APPS (Hendrycks et al., 2021a) and TACO (Li et al., 2023). The dataset contains 16,710 samples, each paired with a long-thinking trace generated by DeepSeek-R1 (Guo et al., 2025) or QwQ-32B-Preview (Team, 2024). To ensure fair evaluation, all training data is strictly separated from the test sets.

**Table 1: Performance of models trained from Qwen3-4B-Base. All models are trained on Bespoke-Stratos-17k except for General-Reasoner-4B. The best and second-best results are highlighted using bold text and underlined text, respectively.**
| Method | MATH500 | AIME24 | AIME25 | AMC23 | OlympiadBench | Avg |
| --- | --- | --- | --- | --- | --- | --- |
| Avg@2 | Avg@32 | Avg@32 | Avg@32 | Avg@2 |  |  |
| Qwen3-4B-Base | 45.30 | 4.79 | 5.73 | 27.73 | 23.37 | 21.38 |
| General-Reasoner-4B | 71.70 | 19.06 | 16.77 | 55.00 | 45.18 | 41.54 |
| Bespoke-Stratos-4B^† | 84.70 | 27.29 | 24.17 | 70.16 | 50.45 | 51.35 |
| Ours-PB-GML-4B | 87.30 | 33.54 | 26.77 | 74.06 | 54.45 | 55.23 |
| Ours-PB-Acc-4B | 88.30 | 31.67 | 27.29 | 75.63 | 55.71 | 55.72 |
| Method | GPQA-D | LCB-E | LCB-M | LCB-H | MMLU-Pro | Avg |
| Avg@8 | Avg@8 | Avg@8 | Avg@8 | Avg@1 |  |  |
| Qwen3-4B-Base | 29.10 | 18.54 | 5.46 | 1.32 | 36.89 | 18.26 |
| General-Reasoner-4B | 40.97 | 61.40 | 17.90 | 2.85 | 61.36 | 36.90 |
| Bespoke-Stratos-4B^† | 44.95 | 71.22 | 19.54 | 3.25 | 63.03 | 40.40 |
| Ours-PB-GML-4B | 45.52 | 79.53 | 31.25 | 6.20 | 65.52 | 45.60 |
| Ours-PB-Acc-4B | 45.33 | 80.29 | 33.68 | 5.79 | 65.53 | 46.12 |

**Table 2: Performance of models trained from Qwen3-8B-Base trained on Bespoke-Stratos-17k.**
| Method | MATH500 | AIME24 | AIME25 | AMC23 | OlympiadBench | Avg |
| --- | --- | --- | --- | --- | --- | --- |
| Avg@2 | Avg@32 | Avg@32 | Avg@32 | Avg@2 |  |  |
| Qwen3-8B-Base | 65.20 | 11.46 | 10.10 | 45.00 | 34.72 | 33.30 |
| Bespoke-Stratos-8B^† | 89.70 | 39.58 | 28.85 | 78.91 | 55.64 | 58.54 |
| Ours-PB-GML-8B | 91.60 | 44.06 | 31.67 | 83.59 | 58.23 | 61.83 |
| Ours-PB-Acc-8B | 91.80 | 45.63 | 31.98 | 85.47 | 58.98 | 62.77 |
| Method | GPQA-D | LCB-E | LCB-M | LCB-H | MMLU-Pro | Avg |
| Avg@8 | Avg@8 | Avg@8 | Avg@8 | Avg@1 |  |  |
| Qwen3-8B-Base | 35.42 | 41.14 | 13.65 | 1.42 | 45.62 | 27.45 |
| Bespoke-Stratos-8B^† | 53.03 | 81.53 | 36.89 | 7.11 | 68.74 | 49.46 |
| Ours-PB-GML-8B | 52.72 | 87.36 | 45.51 | 13.82 | 70.76 | 54.03 |
| Ours-PB-Acc-8B | 53.66 | 86.47 | 49.33 | 13.21 | 70.76 | 54.69 |

Baselines. We compare against a broad set of baselines for rigorous evaluation. For Qwen2.5-Instruct models, we include Bespoke-Stratos (Labs, 2025) and RLT (Cetin et al., 2025) at the 7B and 32B scales, where RLT trains teachers via RL to improve distillation. For Qwen3-Base models, we compare with Bespoke-Stratos-4B/8B^† and General-Reasoner-4B (Ma et al., 2025), the latter trained with GRPO (Shao et al., 2024) and a model-based verifier. Models marked ^† are trained by us; others are official releases. All Bespoke-Stratos models are distilled on Bespoke-Stratos-17k. This ensures fair comparison, as all methods (except General-Reasoner) follow the same training recipes and datasets.

Evaluation. We assess our models on a broad set of challenging benchmarks: MATH500 (Hendrycks et al., 2021b), a subset of competition math curated by Lightman et al. (2024); AIME24&25 (MAA, 2025); AMC23 (MAA, 2023); OlympiadBench (He et al., 2024); LiveCodeBench (Jain et al., 2025), with Easy, Medium, and Hard subsets (LCB-E, LCB-M, LCB-H) for fine-grained coding evaluation; GPQA-Diamond (Rein et al., 2024) (GPQA-D), graduate-level natural science questions; and MMLU-Pro (Wang et al., 2024), a diverse multiple-choice benchmark. Among these, GPQA-D and MMLU-Pro are out-of-distribution (OOD) relative to our training data.

Decoding and average accuracy. Following Hochlehnert et al. (2025), we sample responses with temperature=0.7 and report average accuracy over $k$ responses per question (Avg@$k$). To reduce randomness and ensure fair comparison, we use larger $k$ for smaller datasets: Avg@32 for AIME24, AIME25, and AMC23; Avg@8 for GPQA-Diamond, LCB-E, LCB-M, and LCB-H; Avg@2 for MATH500 and OlympiadBench; and Avg@1 for the large MMLU-Pro (12k+ questions). All evaluations are conducted with SkyThought (Team, 2025a), with additional details in Appendix [D](https://arxiv.org/html/2509.22637v2#A4).

### 4.1 Training details for variational reasoning

We conduct experiments on Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct (Yang et al., 2024), Qwen3-4B-Base, and Qwen3-8B-Base (Team, 2025b). Following Algorithm [1](https://arxiv.org/html/2509.22637v2#alg1), we first train an *initial* reasoning model $\pi_{\theta_{0}}$ on Bespoke-Stratos-17k using the recipe from Labs (2025), and then a variational posterior $q_{\phi}$ with the forward KL divergence (Eq. ([9](https://arxiv.org/html/2509.22637v2#S2.E9))) on the same dataset. These models are later used to compute the weights $\widetilde{\rho}_{k}$. For $\pi_{\theta_{0}}$, we adopt the prompt template from Labs (2025); for $q_{\phi}$, we test two alternative templates (“-PA” and “-PB”, see Appendix [E](https://arxiv.org/html/2509.22637v2#A5)). Both $\pi_{\theta_{0}}$ and $q_{\phi}$ are finetuned independently from the same base model without weight sharing (Appendix [C.1](https://arxiv.org/html/2509.22637v2#A3.SS1)).

**Table 3: Performance of models trained from Qwen2.5-32B-Instruct trained on Bespoke-Stratos-17k. The best and second-best results are highlighted using bold text and underlined text, respectively.**
| Method | MATH500 | AIME24 | AIME25 | AMC23 | OlympiadBench | Avg |
| --- | --- | --- | --- | --- | --- | --- |
| Avg@2 | Avg@32 | Avg@32 | Avg@32 | Avg@2 |  |  |
| Qwen2.5-32B-Instruct | 80.70 | 15.83 | 12.08 | 61.95 | 46.96 | 43.51 |
| Bespoke-Stratos-32B | 92.60 | 55.42 | 46.88 | 92.19 | 64.68 | 70.34 |
| RLT-32B | 93.50 | 56.77 | 47.19 | 91.48 | 63.21 | 70.43 |
| Ours-PA-GML-32B | 93.20 | 56.56 | 48.13 | 93.98 | 64.24 | 71.22 |
| Ours-PA-Acc-32B | 93.50 | 58.85 | 50.31 | 92.97 | 64.39 | 72.01 |
| Method | GPQA-D | LCB-E | LCB-M | LCB-H | MMLU-Pro | Avg |
| Avg@8 | Avg@8 | Avg@8 | Avg@8 | Avg@1 |  |  |
| Qwen2.5-32B-Instruct | 46.28 | 79.88 | 40.60 | 9.76 | 59.19 | 47.14 |
| Bespoke-Stratos-32B | 57.57 | 94.78 | 73.54 | 30.48 | 75.22 | 66.32 |
| RLT-32B | 59.09 | 93.20 | 72.15 | 29.78 | 74.88 | 65.82 |
| Ours-PA-GML-32B | 60.92 | 95.19 | 72.21 | 35.57 | 75.57 | 67.89 |
| Ours-PA-Acc-32B | 60.73 | 94.78 | 73.18 | 31.81 | 75.55 | 67.21 |

Figure: Figure 1: Training loss and gradient norm of different methods during Qwen3-Base model training.
Refer to caption: x1.png

Next, we use the trained $q_{\phi}$ to generate 8 responses (thinking traces and final answers) per training sample. For each response, the weight $\widetilde{\rho}_{k}$ in Eq. ([6](https://arxiv.org/html/2509.22637v2#S2.E6)) is computed from $q_{\phi}$, $\pi_{\theta_{0}}$, and, when using the accuracy-based estimator, math/code verifiers from SkyThought. To estimate $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})$ in $\widetilde{\rho}_{k}$, we compare three options: a naive likelihood method (“-L”), an accuracy-based method (“-Acc”, Section [2.2](https://arxiv.org/html/2509.22637v2#S2.SS2)), and a geometric mean of token-level probabilities (“-GML”), as detailed in Appendix [C.2](https://arxiv.org/html/2509.22637v2#A3.SS2).

We train the final reasoning model $\pi_{\theta}$ following Eq. ([6](https://arxiv.org/html/2509.22637v2#S2.E6)) under two data settings. 17K: the full Bespoke-Stratos-17k dataset. To enhance efficiency, we create a mixed dataset containing, for each original sample, the $q_{\phi}$-generated response with the highest $\widetilde{\rho}_{k}$ and the original sample itself. 1K: a fixed 1,000-sample subset uniformly drawn from the full dataset, where all 8 $q_{\phi}$-generated responses per sample are used for weighted SFT with $\widetilde{\rho}_{k}$. The same 1K subset is reused across related experiments. Main results are reported with 17K, while ablations use both 17K and 1K configurations (Appendix [C.3](https://arxiv.org/html/2509.22637v2#A3.SS3)).

**Table 4: Ablation study on the effect of conditioning the proposal distribution on ${\bm{y}}^{\prime}$.**
| Method | MATH500 | AIME24 | AIME25 | AMC23 | OlympiadBench | Avg |
| --- | --- | --- | --- | --- | --- | --- |
| Avg@2 | Avg@32 | Avg@32 | Avg@32 | Avg@2 |  |  |
| Qwen3-4B-Base | 45.30 | 4.79 | 5.73 | 27.73 | 23.37 | 21.38 |
| Ours-4B | 88.30 | 31.67 | 27.29 | 75.63 | 55.71 | 55.72 |
| w/o ${\bm{y}}^{\prime}$ | 81.20 | 23.44 | 23.96 | 65.70 | 46.59 | 48.18 |
| Method | GPQA-D | LCB-E | LCB-M | LCB-H | MMLU-Pro | Avg |
| Avg@8 | Avg@8 | Avg@8 | Avg@8 | Avg@1 |  |  |
| Qwen3-4B-Base | 29.10 | 18.54 | 5.46 | 1.32 | 36.89 | 18.26 |
| Ours-4B | 45.33 | 80.29 | 33.68 | 5.79 | 65.53 | 46.12 |
| w/o ${\bm{y}}^{\prime}$ | 40.53 | 67.93 | 16.63 | 2.44 | 61.49 | 37.80 |

Figure: Figure 2: Pass@K comparison of baselines versus our method based on Qwen3-4B/8B-Base.
Refer to caption: x2.png

### 4.2 Main results

We evaluate our method across four model variants: Qwen3-4B/8B-Base (Tables [1](https://arxiv.org/html/2509.22637v2#S4.T1) and [2](https://arxiv.org/html/2509.22637v2#S4.T2)) and Qwen2.5-7B/32B-Instruct (Tables [3](https://arxiv.org/html/2509.22637v2#S4.T3) and [6](https://arxiv.org/html/2509.22637v2#A6.T6)). Extended results are provided in Appendix [F.1](https://arxiv.org/html/2509.22637v2#A6.SS1).

Variational reasoning performance. All methods substantially improve the reasoning ability of the base model, but our approach consistently achieves the best results. As shown in Tables [1](https://arxiv.org/html/2509.22637v2#S4.T1) and [2](https://arxiv.org/html/2509.22637v2#S4.T2), variational reasoning yields substantial improvements in math, code, and other general domains compared to the base model (e.g., over 160% improvement in math and over 152% in other domains). It also surpasses all baselines in average accuracy (e.g., over 8.5% higher than the strong baseline Bespoke-Stratos-4B^† that uses the same training data, and over 14% in other domains).

Notably, GPQA-Diamond and MMLU-Pro can be considered out-of-distribution test sets, as our training data only cover math and code, whereas they are in-domain for General-Reasoner-4B. Despite this, our method significantly outperforms General-Reasoner on these benchmarks, suggesting that the reasoning improvements from variational reasoning generalize effectively.

Additionally, our method demonstrates robustness across different prompt templates. Performance remains consistent between Prompt Template A and B (denoted as “-PA” and “-PB” in Table [6](https://arxiv.org/html/2509.22637v2#A6.T6)), with both outperforming baselines. Across four model scales, the accuracy-based estimator (“-Acc”) and the geometric mean of token likelihood estimator (“-GML”) exhibit similar performance, though the accuracy-based variant shows a slight advantage in math-related benchmarks.

**Table 5: Ablation study on effects of different $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}}_{k})$ estimators. Experiments are done in data 1k setting. Acc: accuracy; GML: geometric mean of token likelihood; L: naive likelihood.**
| Method | MATH500 | AIME24 | AIME25 | AMC23 | OlympiadBench | Avg |
| --- | --- | --- | --- | --- | --- | --- |
| Avg@2 | Avg@32 | Avg@32 | Avg@32 | Avg@2 |  |  |
| Qwen2.5-7B-Instruct | 75.60 | 10.94 | 7.40 | 51.10 | 39.91 | 36.99 |
| Bespoke-Stratos-7B-1K^† | 77.20 | 16.25 | 13.96 | 53.75 | 40.88 | 40.41 |
| Ours-Acc-7B-1K | 81.30 | 19.69 | 18.44 | 61.64 | 45.99 | 45.41 |
| Ours-GML-7B-1K | 81.30 | 19.27 | 18.33 | 62.50 | 45.48 | 45.38 |
| Ours-L-7B-1K | 79.90 | 17.81 | 14.17 | 59.53 | 43.62 | 43.01 |
| Method | GPQA-D | LCB-E | LCB-M | LCB-H | MMLU-Pro | Avg |
| Avg@8 | Avg@8 | Avg@8 | Avg@8 | Avg@1 |  |  |
| Qwen2.5-7B-Instruct | 29.99 | 62.50 | 18.20 | 3.35 | 48.20 | 32.45 |
| Bespoke-Stratos-7B-1K^† | 37.94 | 60.37 | 13.59 | 1.22 | 56.07 | 33.84 |
| Ours-Acc-7B-1K | 41.16 | 68.13 | 21.42 | 1.42 | 60.94 | 38.62 |
| Ours-GML-7B-1K | 41.35 | 68.41 | 23.30 | 2.74 | 61.31 | 39.42 |
| Ours-L-7B-1K | 39.90 | 66.42 | 19.90 | 1.93 | 58.61 | 37.35 |

Figure: Figure 3: Effects of scaling up the number of thinking traces ($K$ in Algorithm [1](https://arxiv.org/html/2509.22637v2#alg1)) sampled from variational posterior $q_{\phi}$ on the performance of the final reasoning model $\pi_{\theta}$.
Refer to caption: x3.png

Pass@K analysis. We report Pass@K results of experiments based on Qwen3-4B/8B-Base for different values of K. Figure [2](https://arxiv.org/html/2509.22637v2#S4.F2) reveals two key trends: (1) Our method’s advantage increases with larger K on complex benchmarks (e.g., LiveCodeBench-Hard), and (2) Performance gaps diminish on simpler tasks (e.g., LiveCodeBench-Easy) and multiple-choice questions (e.g., GPQA-Diamond). This aligns with expectations, as simpler tasks offer limited room for improvement, and multiple-choice formats inherently allow high Pass@K with sufficiently large K. These results underscore the strong potential of variational reasoning in tackling complex tasks.

Training dynamics. We monitor training loss and gradient norms during training for Qwen3-4B/8B-base models (see Figure [1](https://arxiv.org/html/2509.22637v2#S4.F1)). Compared to Bespoke-Stratos-4B/8B^†, our method yields lower average training loss and fewer gradient norm spikes, indicating greater training stability. We attribute this stability to the $\pi_{\theta}/q_{\phi}$ ratio in $\widetilde{\rho}_{k}$.
Specifically, for a reasoning trace ${\bm{z}}_{k}$, the weight $\widetilde{\rho}_{k}$ is large when the trace is both high-quality (high $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}}_{k})$) and aligned with the reasoning policy (high likelihood ratio ${\pi_{\theta}({\bm{z}}_{k}|{\bm{x}})}/{q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})}$). This adaptive weighting promotes stable and effective training.

### 4.3 Ablation Studies

As mentioned in Section [4.1](https://arxiv.org/html/2509.22637v2#S4.SS1), we conduct ablation studies on both the 17k and 1k data settings to better analyze variational reasoning. Additional ablations are provided in Appendix [F.2](https://arxiv.org/html/2509.22637v2#A6.SS2).

Scaling the number of thinking traces ${\bm{z}}_{k}$. We investigate the effect of increasing the number of traces ${\bm{z}}$ sampled from the variational posterior $q_{\phi}$ (i.e., $K$ in Algorithm [1](https://arxiv.org/html/2509.22637v2#alg1)) on the performance of the reasoning model $\pi_{\theta}$. Experiments are conducted under the 1k data setting and faithful to Algorithm [1](https://arxiv.org/html/2509.22637v2#alg1). We scale $K$ exponentially from $1$ to $32$, adjusting the batch size to keep optimization steps consistent. Results in Figure [3](https://arxiv.org/html/2509.22637v2#S4.F3) suggest that increasing $K$ can further enhance model performance. This implies a practical trade-off between training computational cost and reasoning accuracy when selecting $K$.

Conditioning on ${\bm{y}}^{\prime}$. We ablate the necessity of conditioning on ${\bm{y}}^{\prime}$ in data 17k setting. The variant w/o ${\bm{y}}^{\prime}$ samples thinking traces ${\bm{z}}$ by the initial reasoning model instead of the variational posterior. Results (Table [4](https://arxiv.org/html/2509.22637v2#S4.T4)) show that removing ${\bm{y}}^{\prime}$ as the condition negatively affects the performance.

Different $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}}_{k})$ estimators. We ablate different estimators for $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}}_{k})$ used in the weight $\widetilde{\rho}_{k}$ in data 1k setting (Table [5](https://arxiv.org/html/2509.22637v2#S4.T5)).
We find estimators based on accuracy or geometric mean of token likelihood outperform the naive likelihood by a large margin, validating our analysis in Sec. [2.2](https://arxiv.org/html/2509.22637v2#S2.SS2).

## 5 Conclusion

We introduced a variational reasoning framework as a principled and stable objective for training reasoning models, while clarifying biases in existing SFT/RFT and RL methods. Beyond consistent gains over strong baselines on diverse reasoning tasks, our analysis offers a probabilistic perspective for interpreting current approaches. A natural future direction is extending training beyond a single round ($T>1$ in Algorithm [1](https://arxiv.org/html/2509.22637v2#alg1)) and exploring richer posterior design for the answer hint ${\bm{y}}^{\prime}$.

## References

- Bengio et al. (2023)
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Bengio.
Gflownet foundations.
*Journal of Machine Learning Research*, 24(210):1–55, 2023.
URL [http://jmlr.org/papers/v24/22-0364.html](http://jmlr.org/papers/v24/22-0364.html).
- Bornschein & Bengio (2015)
Jörg Bornschein and Yoshua Bengio.
Reweighted wake-sleep.
In *International Conference on Learning Representations (ICLR)*, 2015.
- Burda et al. (2015)
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov.
Importance weighted autoencoders.
*arXiv preprint arXiv:1509.00519*, 2015.
- Cetin et al. (2025)
Edoardo Cetin, Tianyu Zhao, and Yujin Tang.
Reinforcement learning teachers of test time scaling.
*arXiv preprint arXiv:2506.08388*, 2025.
- Chen et al. (2024)
Haolin Chen, Yihao Feng, Zuxin Liu, Weiran Yao, Akshara Prabhakar, Shelby Heinecke, Ricky Ho, Phil Mui, Silvio Savarese, Caiming Xiong, et al.
Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding.
*arXiv preprint arXiv:2411.04282*, 2024.
- Chen et al. (2025)
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping.
Acereason-nemotron: Advancing math and code reasoning through reinforcement learning.
*arXiv preprint arXiv:2505.16400*, 2025.
- Cheng et al. (2025)
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei.
Reasoning with exploration: An entropy perspective.
*arXiv preprint arXiv:2506.14758*, 2025.
- Chu et al. (2025)
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma.
Sft memorizes, rl generalizes: A comparative study of foundation model post-training.
*arXiv preprint arXiv:2501.17161*, 2025.
- Comanici et al. (2025)
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025.
- Cui et al. (2025a)
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al.
Process reinforcement through implicit rewards.
*arXiv preprint arXiv:2502.01456*, 2025a.
- Cui et al. (2025b)
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al.
The entropy mechanism of reinforcement learning for reasoning language models.
*arXiv preprint arXiv:2505.22617*, 2025b.
- Dong et al. (2023)
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang.
Raft: Reward ranked finetuning for generative foundation model alignment.
*arXiv preprint arXiv:2304.06767*, 2023.
- Gao et al. (2024)
Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu.
On designing effective rl reward at training time for llm reasoning.
*arXiv preprint arXiv:2410.15115*, 2024.
- Guha et al. (2025)
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al.
Openthoughts: Data recipes for reasoning models.
*arXiv preprint arXiv:2506.04178*, 2025.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- He et al. (2024)
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun.
Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024.
- Hendrycks et al. (2021a)
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt.
Measuring coding challenge competence with APPS.
In *Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)*, 2021a.
URL [https://openreview.net/forum?id=sD93GOzH3i5](https://openreview.net/forum?id=sD93GOzH3i5).
- Hendrycks et al. (2021b)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the math dataset.
*NeurIPS*, 2021b.
- Hochlehnert et al. (2025)
Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge.
A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility.
*arXiv preprint arXiv:2504.07086*, 2025.
- Hoffman et al. (2023)
Matthew Douglas Hoffman, Du Phan, david dohan, Sholto Douglas, Tuan Anh Le, Aaron T Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A. Saurous.
Training chain-of-thought via latent-variable inference.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=a147pIS2Co](https://openreview.net/forum?id=a147pIS2Co).
- Hu et al. (2024)
Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin.
Amortizing intractable inference in large language models.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=Ouj6p4ca60](https://openreview.net/forum?id=Ouj6p4ca60).
- Hu (2025)
Jian Hu.
Reinforce++: A simple and efficient approach for aligning large language models.
*arXiv preprint arXiv:2501.03262*, 2025.
- Hu et al. (2025)
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung-Yeung Shum Xiangyu Zhang.
Open-reasoner-zero: An open source approach to scaling reinforcement learning on the base model.
[https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero), 2025.
- Jaech et al. (2024)
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
Openai o1 system card.
*arXiv preprint arXiv:2412.16720*, 2024.
- Jain et al. (2025)
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica.
Livecodebench: Holistic and contamination free evaluation of large language models for code.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=chfJJYC3iL](https://openreview.net/forum?id=chfJJYC3iL).
- Jiang et al. (2025)
Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, Tianyu Pang, and Wenhu Chen.
Verltool: Towards holistic agentic reinforcement learning with tool use.
*arXiv preprint arXiv:2509.01055*, 2025.
- Jin et al. (2025)
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han.
Search-r1: Training llms to reason and leverage search engines with reinforcement learning.
*arXiv preprint arXiv:2503.09516*, 2025.
- Kazemnejad et al. (2024)
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux.
Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment.
*arXiv preprint arXiv:2410.01679*, 2024.
- Kingma & Welling (2013)
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
*arXiv preprint arXiv:1312.6114*, 2013.
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*, 2023.
- Labs (2025)
Bespoke Labs.
Bespoke-stratos: The unreasonable effectiveness of reasoning distillation.
www.bespokelabs.ai/blog/bespoke-stratos-the-unreasonable-effectiveness-of-reasoning-distillation, 2025.
Accessed: 2025-01-22.
- Li et al. (2025a)
Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G Patil, Matei Zaharia, et al.
Llms can easily learn to reason from demonstrations structure, not content, is what matters!
*arXiv preprint arXiv:2502.07374*, 2025a.
- Li et al. (2024)
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al.
Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.
*Hugging Face repository*, 13:9, 2024.
- Li et al. (2025b)
Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He.
Codei/o: Condensing reasoning patterns via code input-output prediction.
*arXiv preprint arXiv:2502.07316*, 2025b.
- Li et al. (2023)
Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li.
Taco: Topics in algorithmic code generation dataset.
*arXiv preprint arXiv:2312.14852*, 2023.
- Lightman et al. (2024)
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=v8L0pN6EOi](https://openreview.net/forum?id=v8L0pN6EOi).
- Liu & Zhang (2025)
Jiawei Liu and Lingming Zhang.
Code-r1: Reproducing r1 for code with reliable rewards.
[https://github.com/ganler/code-r1](https://github.com/ganler/code-r1), 2025.
- Liu et al. (2025)
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin.
Understanding r1-zero-like training: A critical perspective.
*arXiv preprint arXiv:2503.20783*, 2025.
- Loshchilov & Hutter (2019)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
In *International Conference on Learning Representations*, 2019.
URL [https://openreview.net/forum?id=Bkg6RiCqY7](https://openreview.net/forum?id=Bkg6RiCqY7).
- Luo et al. (2025)
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica.
Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025.
Notion Blog.
- Ma et al. (2025)
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen.
General-reasoner: Advancing llm reasoning across all domains.
*arXiv preprint arXiv:2505.14652*, 2025.
- Ma et al. (2023)
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar.
Eureka: Human-level reward design via coding large language models.
*arXiv preprint arXiv:2310.12931*, 2023.
- MAA (2023)
MAA.
American mathematics competitions - amc.
[https://maa.org/](https://maa.org/), 2023.
- MAA (2025)
MAA.
American invitational mathematics examination - aime.
[https://maa.org/](https://maa.org/), 2025.
- Muennighoff et al. (2025)
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto.
s1: Simple test-time scaling.
*arXiv preprint arXiv:2501.19393*, 2025.
- Qi et al. (2025)
Penghui Qi, Zichen Liu, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin.
Optimizing anytime reasoning via budget relative policy optimization.
*arXiv preprint arXiv:2505.13438*, 2025.
- Rafailov et al. (2023)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
*Advances in Neural Information Processing Systems*, 36:53728–53741, 2023.
- Rein et al. (2024)
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman.
Gpqa: A graduate-level google-proof q&a benchmark.
In *First Conference on Language Modeling*, 2024.
- Ruan et al. (2025)
Yangjun Ruan, Neil Band, Chris J Maddison, and Tatsunori Hashimoto.
Reasoning to learn from latent thoughts.
*arXiv preprint arXiv:2503.18866*, 2025.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Shenfeld et al. (2025)
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal.
Rl’s razor: Why online reinforcement learning forgets less.
*arXiv preprint arXiv:2509.04259*, 2025.
- Tang et al. (2025)
Yunhao Tang, Sid Wang, and Rémi Munos.
Learning to chain-of-thought with jensen’s evidence lower bound.
*arXiv preprint arXiv:2503.19618*, 2025.
- Team et al. (2025)
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al.
Kimi k1. 5: Scaling reinforcement learning with llms.
*arXiv preprint arXiv:2501.12599*, 2025.
- Team (2025a)
NovaSky Team.
Sky-t1: Train your own o1 preview model within $450.
https://novasky-ai.github.io/posts/sky-t1, 2025a.
Accessed: 2025-01-09.
- Team (2024)
Qwen Team.
Qwq: Reflect deeply on the boundaries of the unknown.
*Hugging Face*, 2024.
- Team (2025b)
Qwen Team.
Qwen3, April 2025b.
URL [https://qwenlm.github.io/blog/qwen3/](https://qwenlm.github.io/blog/qwen3/).
- Touvron et al. (2023)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
*arXiv preprint arXiv:2307.09288*, 2023.
- Uesato et al. (2022)
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins.
Solving math word problems with process-and outcome-based feedback.
*arXiv preprint arXiv:2211.14275*, 2022.
- Wang et al. (2024)
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al.
Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.
In *The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2024.
- Wang et al. (2025)
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al.
Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.
*arXiv preprint arXiv:2504.20073*, 2025.
- Wei et al. (2025)
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang.
Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution.
*arXiv preprint arXiv:2502.18449*, 2025.
- Wen et al. (2025)
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al.
Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond.
*arXiv preprint arXiv:2503.10460*, 2025.
- Wu et al. (2025)
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang.
On the generalization of sft: A reinforcement learning perspective with reward rectification.
*arXiv preprint arXiv:2508.05629*, 2025.
- Xiang et al. (2025)
Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al.
Towards system 2 reasoning in llms: Learning how to think with meta chain-of-thought.
*arXiv preprint arXiv:2501.04682*, 2025.
- Xie et al. (2025)
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo.
Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning.
*arXiv preprint arXiv:2502.14768*, 2025.
- Xu et al. (2025)
Yifei Xu, Tusher Chakraborty, Srinagesh Sharma, Leonardo Nunes, Emre Kıcıman, Songwu Lu, and Ranveer Chandra.
Direct reasoning optimization: Llms can reward and refine their own reasoning for open-ended tasks.
*arXiv preprint arXiv:2506.13351*, 2025.
- Xue et al. (2025)
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An.
Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning.
*arXiv preprint arXiv:2509.02479*, 2025.
- Yang et al. (2024)
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report.
*arXiv preprint arXiv:2412.15115*, 2024.
- Yang et al. (2025)
John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang.
Swe-smith: Scaling data for software engineering agents.
*arXiv preprint arXiv:2504.21798*, 2025.
- Yu et al. (2025a)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al.
Dapo: An open-source llm reinforcement learning system at scale.
*arXiv preprint arXiv:2503.14476*, 2025a.
- Yu et al. (2025b)
Tianyu Yu, Bo Ji, Shouli Wang, Shu Yao, Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, et al.
Rlpr: Extrapolating rlvr to general domains without verifiers.
*arXiv preprint arXiv:2506.18254*, 2025b.
- Yue et al. (2025a)
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang.
Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?
*arXiv preprint arXiv:2504.13837*, 2025a.
- Yue et al. (2025b)
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, et al.
Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.
*arXiv preprint arXiv:2504.05118*, 2025b.
- Zeng et al. (2025)
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He.
7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient.
[https://hkust-nlp.notion.site/simplerl-reason](https://hkust-nlp.notion.site/simplerl-reason), 2025.
Notion Blog.
- Zheng et al. (2025)
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al.
Group sequence policy optimization.
*arXiv preprint arXiv:2507.18071*, 2025.
- Zheng et al. (2024)
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma.
Llamafactory: Unified efficient fine-tuning of 100+ language models.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)*, Bangkok, Thailand, 2024. Association for Computational Linguistics.
URL [http://arxiv.org/abs/2403.13372](http://arxiv.org/abs/2403.13372).
- Zhong et al. (2025)
Han Zhong, Yutong Yin, Shenao Zhang, Xiaojun Xu, Yuanxin Liu, Yifei Zuo, Zhihan Liu, Boyi Liu, Sirui Zheng, Hongyi Guo, et al.
Brite: Bootstrapping reinforced thinking process to enhance language model reasoning.
*arXiv preprint arXiv:2501.18858*, 2025.
- Zhou et al. (2025)
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du.
Reinforcing general reasoning without verifiers.
*arXiv preprint arXiv:2505.21493*, 2025.

## Appendix A Detailed derivations

In this section, we provide detailed derivations of the conclusions presented in the main text, along with some additional results.

### A.1 Derivation for Eq. ( 4 )

The ELBO objective induced from Eq. ([2](https://arxiv.org/html/2509.22637v2#S2.E2)) can be rewritten as

$$ $\begin{split}\mathcal{L}_{\textrm{ELBO}}({\bm{x}},\mathcal{Y}_{{\bm{x}}},{\bm{y}}^{\prime};\pi_{\theta},q_{\phi})&\triangleq\mathbb{E}_{q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\log\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\right]-\textrm{KL}\left(q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})||\pi_{\theta}({\bm{z}}|{\bm{x}})\right)\\ &=\mathbb{E}_{q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\log\pi_{\theta}({\bm{z}},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})\right]+\mathcal{H}\left(q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})\right)\\ &=\log P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})-{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathbb{D}_{\textrm{KL}}\left(q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})||P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})\right)}\textrm{,}\end{split}$ (13) $$

where $\mathcal{H}(\cdot)$ is entropy function and $P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})=\frac{\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\pi_{\theta}({\bm{z}}|{\bm{x}})}{P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}$ is the true posterior distribution.

### A.2 Derivation for Eq. ( 6 )

Given the IWAE-style lower bound $\mathcal{L}_{\textrm{ELBO}}^{K}({\bm{x}},\mathcal{Y}_{{\bm{x}}},{\bm{y}}^{\prime};\pi_{\theta},q_{\phi})$ in Eq. ([5](https://arxiv.org/html/2509.22637v2#S2.E5)), we can derive its gradient w.r.t. model parameters $\theta$ as:

$$ $\begin{split}&\nabla_{\theta}\mathcal{L}_{\textrm{ELBO}}^{K}({\bm{x}},\mathcal{Y}_{{\bm{x}}},{\bm{y}}^{\prime};\pi_{\theta},q_{\phi})\\ ={}&\nabla_{\theta}\mathbb{E}_{{\bm{z}}_{1:K}\sim q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\log\frac{1}{K}\sum_{k=1}^{K}\frac{\pi_{\theta}({\bm{z}}_{k},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}{q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})}\right]\\ ={}&\mathbb{E}_{{\bm{z}}_{1:K}\sim q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\frac{\frac{1}{K}\sum_{k=1}^{K}\frac{\nabla_{\theta}\pi_{\theta}({\bm{z}}_{k},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}{q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})}}{\frac{1}{K}\sum_{k=1}^{K}\frac{\pi_{\theta}({\bm{z}}_{k},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}{q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})}}\right]\\ ={}&\mathbb{E}_{{\bm{z}}_{1:K}\sim q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\sum_{k=1}^{K}\widetilde{\rho}_{k}\nabla_{\theta}\log\pi_{\theta}({\bm{z}}_{k},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})\right]\\ &\textrm{where}\quad\widetilde{\rho}_{k}=\frac{\rho_{k}}{\sum_{j=1}^{K}\rho_{j}}\quad\text{and}\quad\rho_{k}=\frac{\pi_{\theta}({\bm{z}}_{k},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}{q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})}\textrm{.}\end{split}$ (14) $$

Using the notations of $\rho_{k}$ and $\widetilde{\rho}_{k}$, we can further derive the gradient w.r.t. $\phi$ as:

$$ $\begin{split}\!\!\!\!\!&\nabla_{\phi}\mathcal{L}_{\textrm{ELBO}}^{K}({\bm{x}},\mathcal{Y}_{{\bm{x}}},{\bm{y}}^{\prime};\pi_{\theta},q_{\phi})\!\!\\ \!\!\!\!\!={}&\nabla_{\phi}\mathbb{E}_{{\bm{z}}_{1:K}\sim q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\log\frac{1}{K}\sum_{k=1}^{K}\frac{\pi_{\theta}({\bm{z}}_{k},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}{q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})}\right]\!\!\\ \!\!\!\!\!={}&\mathbb{E}_{{\bm{z}}_{1:K}\sim q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\left(\log\frac{1}{K}\sum_{k=1}^{K}\rho_{k}\right)\cdot\sum_{k=1}^{K}\nabla_{\phi}\log q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})-\sum_{k=1}^{K}\widetilde{\rho}_{k}\nabla_{\phi}\log q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})\right]\!\!\\ \!\!\!\!\!={}&\mathbb{E}_{{\bm{z}}_{1:K}\sim q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\sum_{k=1}^{K}\left(-\widetilde{\rho}_{k}+\log\frac{1}{K}\sum_{k=1}^{K}\rho_{k}\right)\cdot\nabla_{\phi}\log q_{\phi}({\bm{z}}_{k}|{\bm{x}},{\bm{y}}^{\prime})\right]\textrm{.}\!\!\end{split}$ $$

### A.3 Proof of Theorem 1

As to the computation of $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})$, there are two unbiased estimators:

$$ $\begin{split}\textbf{Likelihood-based estimator:}&\quad\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})=|\mathcal{Y}_{\bm{x}}|\cdot\mathbb{E}_{{\bm{y}}\sim\mathcal{U}(\mathcal{Y}_{\bm{x}})}\left[\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\right]\textrm{;}\\ \textbf{Accuracy-based estimator:}&\quad\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})=\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\right]\textrm{,}\end{split}$ (15) $$

where $|\mathcal{Y}_{\bm{x}}|$ is the cardinal (number of elements) of $\mathcal{Y}_{\bm{x}}$, $\mathcal{U}(\mathcal{Y}_{\bm{x}})$ is the uniform distribution on $\mathcal{Y}_{\bm{x}}$, and $\mathds{1}(\cdot)$ is the indicator function. When $|\mathcal{Y}_{\bm{x}}|=1$, i.e., there is a unique correct answer expression ${\bm{y}}^{*}$, Zhou et al. (2025) show that the likelihood-based estimator has lower variance (in fact, zero) compared to the accuracy-based one. We now extend this comparison to general cases when $|\mathcal{Y}_{\bm{x}}|>1$:

$$ $\begin{split}\textrm{Var}_{\textrm{like}}&=|\mathcal{Y}_{\bm{x}}|^{2}\cdot\textrm{Var}_{{\bm{y}}\sim\mathcal{U}(\mathcal{Y}_{\bm{x}})}\left[\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\right]\textrm{;}\\ \textrm{Var}_{\textrm{acc}}&=\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})\cdot\left(1-\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})\right)\textrm{.}\end{split}$ (16) $$

Note that the variance $\textrm{Var}_{\textrm{acc}}$ of accuracy-based estimator is independent of $|\mathcal{Y}_{\bm{x}}|$ and the model distribution $\pi_{\theta}$ over different elements in $\mathcal{Y}_{\bm{x}}$. Assuming that in the worst case where only one element ${\bm{y}}^{*}\in\mathcal{Y}_{\bm{x}}$ has non-zero probability under $\pi_{\theta}$, i.e., $\pi_{\theta}({\bm{y}}^{*}|{\bm{x}},{\bm{z}})=\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})$, we have

$$ $\begin{split}\textrm{Var}_{\textrm{like}}^{\textrm{worst}}&\triangleq\max_{\pi_{\theta}}\textrm{Var}_{\textrm{like}}=\left(|\mathcal{Y}_{\bm{x}}|-1\right)\cdot\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})^{2}\textrm{;}\\ \textrm{Var}_{\textrm{acc}}^{\textrm{worst}}&\triangleq\max_{\pi_{\theta}}\textrm{Var}_{\textrm{acc}}=\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})\cdot\left(1-\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})\right)\textrm{.}\end{split}$ (17) $$

Here we slightly abuse the notation of $\max_{\pi_{\theta}}$, since the maximization is taken w.r.t. all $\pi_{\theta}$ *under fixed value $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})$*. As seen, $\textrm{Var}_{\textrm{like}}^{\textrm{worst}}\geq\textrm{Var}_{\textrm{acc}}^{\textrm{worst}}$ holds when the model accuracy (condition on ${\bm{x}},{\bm{z}}$) satisfies

$$ $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}})\geq\frac{1}{|\mathcal{Y}_{x}|}\textrm{,}$ (18) $$

which almost always holds for $|\mathcal{Y}_{x}|\gg 1$.∎

### A.4 Derivation for Eq. ( 9 )

Now we derive the gradient of the forward KL divergence $\mathbb{D}_{\textrm{KL}}(P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})||q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime}))$ w.r.t. $\phi$:

$$ $\begin{split}&\nabla_{\phi}\mathbb{D}_{\textrm{KL}}(P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})||q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime}))\\ ={}&-\nabla_{\phi}\mathbb{E}_{P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})}\left[\log q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})\right]\\ ={}&-\nabla_{\phi}\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\left[\frac{\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})}{P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}\log q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})\right]\\ \simeq{}&\mathbb{E}_{{\bm{z}}_{1:M}\sim\pi_{\theta}({\bm{z}}|{\bm{x}})}\left[\sum_{m=1}^{M}\widetilde{w}_{m}\nabla_{\phi}\log q_{\phi}({\bm{z}}_{m}|{\bm{x}},{\bm{y}}^{\prime})\right]\triangleq\nabla_{\phi}\mathcal{L}_{\textrm{forward}}^{M}\textrm{,}\\ &\textrm{where}\quad\widetilde{w}_{m}=\frac{w_{m}}{\sum_{j=1}^{M}w_{j}}\quad\text{and}\quad w_{m}=\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}}_{m})\textrm{.}\end{split}$ (19) $$

### A.5 Connection to more general RL reward shaping

In the literature on reinforcement learning with verifiable rewards (RLVR), various strategies for reward shaping have been proposed, many of which can be expressed as

$$ $\mathcal{R}({\bm{x}},{\bm{y}})=\begin{cases}\alpha&\textrm{if }{\bm{y}}\in\mathcal{Y}_{{\bm{x}}}\textrm{;}\\ \beta&\textrm{if }{\bm{y}}\notin\mathcal{Y}_{{\bm{x}}}\land{\bm{y}}\in\mathcal{Y}_{\textrm{format}}\textrm{;}\\ \gamma&\textrm{otherwise,}\end{cases}$ (20) $$

where $\alpha,\beta,\gamma$ are hyperparameters, $\mathcal{Y}_{\textrm{format}}$ is the set of answers that correctly follow required format (e.g., \boxed{}) and is typically independent of ${\bm{x}}$. Apparently, $\mathcal{Y}_{{\bm{x}}}\subset\mathcal{Y}_{\textrm{format}}$ holds for any ${\bm{x}}$. Then we can derive the gradient of training objective under the general reward shaping $\mathcal{R}({\bm{x}},{\bm{y}})$ as

$$ $\begin{split}&\nabla_{\theta}\mathcal{L}_{\textrm{general-RL}}({\bm{x}},\pi_{\theta})\\ \triangleq{}&\nabla_{\theta}\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\mathbb{E}_{\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\left[\mathcal{R}({\bm{x}},{\bm{y}})\right]\\ ={}&\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\mathbb{E}_{\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\left[\mathcal{R}({\bm{x}},{\bm{y}})\cdot\left(\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}}){\color[rgb]{0.66796875,0.66796875,0.66796875}\definecolor[named]{pgfstrokecolor}{rgb}{0.66796875,0.66796875,0.66796875}+\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\right)\right]\\ \xRightarrow[\scriptstyle\pi_{\theta}({\bm{z}}|{\bm{x}})]{\textrm{only w.r.t.}}{}&\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\mathbb{E}_{\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\left[\mathcal{R}({\bm{x}},{\bm{y}})\cdot\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})\right]\\ ={}&\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\left[\left((\alpha-\beta)\cdot\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})+(\beta-\gamma)\cdot\pi_{\theta}(\mathcal{Y}_{\textrm{format}}|{\bm{x}},{\bm{z}})\right)\cdot\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})\right]\\ ={}&{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}(\alpha-\beta)\cdot P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}\cdot\nabla_{\theta}\mathbb{D}_{\textrm{KL}}(P_{\theta}^{\mathrm{sg}}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})||\pi_{\theta}({\bm{z}}|{\bm{x}}))\\ &\quad+{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}(\beta-\gamma)\cdot P_{\theta}(\mathcal{Y}_{\textrm{format}}|{\bm{x}})}\cdot\nabla_{\theta}\mathbb{D}_{\textrm{KL}}(P_{\theta}^{\mathrm{sg}}({\bm{z}}|{\bm{x}},\mathcal{Y}_{\textrm{format}})||\pi_{\theta}({\bm{z}}|{\bm{x}}))\textrm{,}\end{split}$ (21) $$

where $P_{\theta}(\mathcal{Y}_{\textrm{format}}|{\bm{x}})$ is the probability that the output answers follow the required format. It is easy to know that the optimal solution for Eq. ([21](https://arxiv.org/html/2509.22637v2#A1.E21)) can be written as:

$$ $\pi_{\theta}^{*}({\bm{z}}|{\bm{x}})=\frac{(\alpha-\beta)\cdot P_{\theta}^{\mathrm{sg}}({\bm{z}},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})+(\beta-\gamma)\cdot P_{\theta}^{\mathrm{sg}}({\bm{z}},\mathcal{Y}_{\textrm{format}}|{\bm{x}})}{(\alpha-\beta)\cdot P_{\theta}^{\mathrm{sg}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})+(\beta-\gamma)\cdot P_{\theta}^{\mathrm{sg}}(\mathcal{Y}_{\textrm{format}}|{\bm{x}})}\textrm{.}$ (22) $$

Remark. When $\beta=\gamma$, i.e., there is no format reward, the optimization problem in Eq. ([21](https://arxiv.org/html/2509.22637v2#A1.E21)) degrades to Eq. ([11](https://arxiv.org/html/2509.22637v2#S3.E11)). When $\alpha>\beta>\gamma$, the model $\pi_{\theta}({\bm{z}}|{\bm{x}})$ will tend to hack reward function on hard problems (i.e., low $P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$) that can easily follow format (i.e., high $P_{\theta}(\mathcal{Y}_{\textrm{format}}|{\bm{x}})$), where $\pi_{\theta}({\bm{z}}|{\bm{x}})$ will seek modes of $P_{\theta}^{\mathrm{sg}}({\bm{z}}|{\bm{x}},\mathcal{Y}_{\textrm{format}})$. Besides, there may be an intuition that setting $\beta<0$ could alleviate reward hacking, however, as shown in Eq. ([21](https://arxiv.org/html/2509.22637v2#A1.E21)), the optimization only depends on the relative values of $\alpha-\beta$ and $\beta-\gamma$.

Now we show that it is straightforward to debias $P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$ and $P_{\theta}(\mathcal{Y}_{\textrm{format}}|{\bm{x}})$ in Eq. ([21](https://arxiv.org/html/2509.22637v2#A1.E21)). Specifically, we can rewrite the reward function as (note that reward functions are equivalent up to any constant):

$$ $\mathcal{R}({\bm{x}},{\bm{y}})=(\alpha-\beta)\cdot\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})+(\beta-\gamma)\cdot\mathds{1}({\bm{y}}\in\mathcal{Y}_{\textrm{format}})\textrm{.}$ (23) $$

Then the debiased version of reward function is

$$ $\mathcal{R}^{\ddagger}({\bm{x}},{\bm{y}})=\frac{(\alpha-\beta)}{P_{\theta}^{\mathrm{sg}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}\cdot\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})+\frac{(\beta-\gamma)}{P_{\theta}^{\mathrm{sg}}(\mathcal{Y}_{\textrm{format}}|{\bm{x}})}\cdot\mathds{1}({\bm{y}}\in\mathcal{Y}_{\textrm{format}})\textrm{,}$ (24) $$

where in practice $P_{\theta}^{\mathrm{sg}}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})$ and $P_{\theta}^{\mathrm{sg}}(\mathcal{Y}_{\textrm{format}}|{\bm{x}})$ can be approximated by the ratio of correct answers (i.e., model accuracy) and the ratio of correct format for each batch of RL rollouts (larger rollout number could lead to more accurate estimation). After using the debiased reward function $\mathcal{R}^{\ddagger}({\bm{x}},{\bm{y}})$, the optimal solution of $\pi_{\theta}^{*}({\bm{z}}|{\bm{x}})$ becomes

$$ $\pi_{\theta}^{*}({\bm{z}}|{\bm{x}})=\frac{(\alpha-\beta)\cdot P_{\theta}^{\mathrm{sg}}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})+(\beta-\gamma)\cdot P_{\theta}^{\mathrm{sg}}({\bm{z}}|{\bm{x}},\mathcal{Y}_{\textrm{format}})}{\alpha-\gamma}\textrm{.}$ (25) $$

### A.6 Special cases in Eq. ( 6 )

Special case I: $q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})=\pi_{\theta}^{\mathrm{sg}}({\bm{z}}|{\bm{x}})$. In this case, we can simplify the gradient estimation as

$$ $\begin{split}&\nabla_{\theta}\mathcal{L}_{\textrm{ELBO}}^{K}({\bm{x}},\mathcal{Y}_{{\bm{x}}};\pi_{\theta},\pi_{\theta}^{\mathrm{sg}})=\mathbb{E}_{{\bm{z}}_{1:K}\sim\pi_{\theta}({\bm{z}}|{\bm{x}})}\left[\sum_{k=1}^{K}\widetilde{w}_{k}\nabla_{\theta}\log\pi_{\theta}({\bm{z}}_{k},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})\right]\\ &\textrm{where}\quad\widetilde{w}_{k}=\frac{w_{k}}{\sum_{j=1}^{K}w_{j}}\quad\text{and}\quad w_{k}=\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}}_{k})\textrm{,}\end{split}$ (26) $$

which can be regarded as a normalized version of VeriFree (Zhou et al., 2025).

Special case II: $K=1$. In this case $\mathcal{L}_{\textrm{ELBO}}^{1}=\mathcal{L}_{\textrm{ELBO}}$ and we can simplify the gradient estimation as

$$ $\begin{split}\nabla_{\theta}\mathcal{L}_{\textrm{ELBO}}({\bm{x}},\mathcal{Y}_{{\bm{x}}},{\bm{y}}^{\prime};\pi_{\theta},q_{\phi})&=\mathbb{E}_{{\bm{z}}\sim q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\nabla_{\theta}\log\pi_{\theta}({\bm{z}},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})\right]\textrm{;}\\ \nabla_{\phi}\mathcal{L}_{\textrm{ELBO}}({\bm{x}},\mathcal{Y}_{{\bm{x}}},{\bm{y}}^{\prime};\pi_{\theta},q_{\phi})&=\mathbb{E}_{{\bm{z}}\sim q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\left[\left(\log\frac{\pi_{\theta}({\bm{z}},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})}{q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\right)\cdot\nabla_{\phi}\log q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})\right]\textrm{.}\end{split}$ (27) $$

### A.7 More derivations for Eq. ( 11 )

Now we investigate the gradient of binary reward RL w.r.t. $\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})$:

$$ $\begin{split}\nabla_{\theta}\mathcal{L}_{\textrm{bi-RL}}({\bm{x}},\pi_{\theta})\triangleq{}&\nabla_{\theta}\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\mathbb{E}_{\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\right]\\ ={}&\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\mathbb{E}_{\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\cdot\left({\color[rgb]{0.66796875,0.66796875,0.66796875}\definecolor[named]{pgfstrokecolor}{rgb}{0.66796875,0.66796875,0.66796875}\nabla_{\theta}\log\pi_{\theta}({\bm{z}}|{\bm{x}})+}\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\right)\right]\\ \xRightarrow[\scriptstyle\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})]{\textrm{only w.r.t.}}{}&\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\mathbb{E}_{\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\cdot\nabla_{\theta}\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\right]\\ ={}&\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\left[\nabla_{\theta}\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\right]\textrm{,}\\ \end{split}$ (28) $$

where the optimal solution is straightforward that $\forall{\bm{z}}$, there is $\pi_{\theta}^{*}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})=1$. However, this optimal solution is usually unachievable, since it requires the model to return $100\%$ correct answers independent of the thinking process ${\bm{z}}$.

### A.8 Applying variational posterior for RL

Recall that the objective function of binary reward RL is defined as

$$ $\begin{split}\mathcal{L}_{\textrm{bi-RL}}({\bm{x}},\pi_{\theta})&\triangleq\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\mathbb{E}_{\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})}\left[\mathds{1}({\bm{y}}\in\mathcal{Y}_{{\bm{x}}})\right]\\ &=\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}\left[\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\right]\textrm{.}\end{split}$ (29) $$

Suppose the data points are drawn from a behavior policy $q({\bm{z}}|{\bm{x}})$, the RL objective can be reformulated using an importance sampling correction term as follows:

$$ $\mathcal{L}_{\textrm{bi-RL}}({\bm{x}},\pi_{\theta})=\mathbb{E}_{q({\bm{z}}|{\bm{x}})}\left[\frac{\pi_{\theta}({\bm{z}}|{\bm{x}})}{q({\bm{z}}|{\bm{x}})}\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\right]\textrm{.}$ (30) $$

Then, a natural question arises: *what is the optimal behavior policy $q({\bm{z}}|{\bm{x}})$ that minimizes the variance of estimating $\mathcal{L}_{\textrm{bi-RL}}({\bm{x}},\pi_{\theta})$?* Specifically, we can compute

$$ $\begin{split}\textrm{Var}_{q({\bm{z}}|{\bm{x}})}\left[\frac{\pi_{\theta}({\bm{z}}|{\bm{x}})}{q({\bm{z}}|{\bm{x}})}\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\right]&=\textrm{Var}_{q({\bm{z}}|{\bm{x}})}\left[\frac{P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})}{q({\bm{z}}|{\bm{x}})}P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})\right]\\ &=P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})^{2}\cdot\textrm{Var}_{q({\bm{z}}|{\bm{x}})}\left[\frac{P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})}{q({\bm{z}}|{\bm{x}})}\right]\\ &=P_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}})^{2}\cdot\left(\sum_{{\bm{z}}}\frac{P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})^{2}}{q({\bm{z}}|{\bm{x}})}-1\right)\textrm{,}\end{split}$ (31) $$

which is equivalent to minimizing

$$ $\min_{q({\bm{z}}|{\bm{x}})}\sum_{{\bm{z}}}\frac{P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})^{2}}{q({\bm{z}}|{\bm{x}})}\quad\textrm{s.t.}\quad\sum_{{\bm{z}}}q({\bm{z}}|{\bm{x}})=1\textrm{, }q({\bm{z}}|{\bm{x}})\geq 0\textrm{.}$ (32) $$

Using calculus of variations with a Lagrange multiplier $\lambda$, we obtain

$$ $\begin{split}&\delta\left[\sum_{{\bm{z}}}\frac{P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})^{2}}{q({\bm{z}}|{\bm{x}})}+\lambda\left(\sum_{{\bm{z}}}q({\bm{z}}|{\bm{x}})-1\right)\right]=0\\ \Rightarrow{}&q^{*}({\bm{z}}|{\bm{x}})=P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})\textrm{ and }\lambda=1\\ \Rightarrow{}&P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})=\arg\min_{q({\bm{z}}|{\bm{x}})}\textrm{Var}_{q({\bm{z}}|{\bm{x}})}\left[\frac{\pi_{\theta}({\bm{z}}|{\bm{x}})}{q({\bm{z}}|{\bm{x}})}\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})\right]\textrm{.}\end{split}$ (33) $$

Therefore, we show that optimizing the variational posterior $q_{\phi}$ to approximate the true posterior $P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})$ in Eq. ([9](https://arxiv.org/html/2509.22637v2#S2.E9)) naturally yields an (approximately) optimal behavior policy for RL, one that minimizes the variance of the objective estimator. In practice, the trained variational posterior$q_{\phi}$ can thus be employed as the behavior policy to reduce variance, which is fully compatible with actor-critic frameworks that incorporate advantage estimation with baselines.

### A.9 Connection to Reinforcement Learning Teachers

In the derivation of our method, the ELBO objective in Eq. ([4](https://arxiv.org/html/2509.22637v2#S2.E4)) minimizes *reverse* KL divergence $\mathbb{D}_{\textrm{KL}}\left(q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})||P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})\right)$.
As analyzed in Section [2.3](https://arxiv.org/html/2509.22637v2#S2.SS3), we propose to optimize $q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})$ using the *forward* KL divergence $\mathbb{D}_{\textrm{KL}}(P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})||q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime}))$, which shares the same optimal solution.

Alternatively, we can also optimize the reverse KL divergence by policy gradient method as follows:

$$ $\begin{split}&\nabla_{\phi}\mathcal{L}_{\textrm{ELBO}}\\ ={}&\nabla_{\phi}\mathbb{D}_{\textrm{KL}}\left(q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})||P_{\theta}({\bm{z}}|{\bm{x}},\mathcal{Y}_{{\bm{x}}})\right)\\ ={}&\nabla_{\phi}\mathbb{E}_{q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}[\log\pi_{\theta}({\bm{z}},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})-\log q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})]\\ ={}&\mathbb{E}_{q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\Big[\log\pi_{\theta}({\bm{z}},\mathcal{Y}_{{\bm{x}}}|{\bm{x}})\nabla_{\phi}\log q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})-\log q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})\nabla_{\phi}\log q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})\Big]\\ ={}&\mathbb{E}_{q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}\Bigg[\Bigg(\underbrace{\log\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})-\log\frac{q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}{\pi_{\theta}({\bm{z}}|{\bm{x}})}}_{\text{reward}}\Bigg)\nabla_{\phi}\log q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})\Bigg].\end{split}$ (34) $$

More concisely, the reverse KL divergence can be alternatively minimized via reinforcement learning using $\log\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})-\log\frac{q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}{\pi_{\theta}({\bm{z}}|{\bm{x}})}$ as the reward function.

This derivation establishes a connection to Reinforcement Learning Teachers (RLTs) (Cetin et al., 2025), who focus on training reasoning LLMs to act as teachers for distilling new students. Their approach introduces RLTs optimized specifically for effective student distillation. RLTs are trained by GRPO using dense rewards obtained by feeding each explanation to the student and evaluating its understanding of the solution.

Specifically, the dense reward in RLT combines two components: one measuring the student’s likelihood of reaching the correct solution (analogous to $\log\pi_{\theta}(\mathcal{Y}_{{\bm{x}}}|{\bm{x}},{\bm{z}})$), and another regularizing the teacher’s explanation to remain coherent from the student’s perspective given only its prior knowledge and the question (analogous to $-\log\frac{q_{\phi}({\bm{z}}|{\bm{x}},{\bm{y}}^{\prime})}{\pi_{\theta}({\bm{z}}|{\bm{x}})}$).

While RLT employs an intuitively designed reward, our work provides rigorous theoretical justification from a variational inference perspective. Furthermore, we enhance the method with a tighter IWAE-style lower bound and an accuracy-based estimator, as detailed in Section [2.2](https://arxiv.org/html/2509.22637v2#S2.SS2).

## Appendix B Related work

SFT and RL methods for reasoning.
Reasoning has emerged as a central capability of LLMs, driving advances in domains such as mathematics, programming, and scientific discovery (Jaech et al., 2024; Comanici et al., 2025; Team et al., 2025). Among the approaches developed to strengthen these abilities, SFT and RL have become the two dominant paradigms (Uesato et al., 2022; Rafailov et al., 2023; Guha et al., 2025; Hu et al., 2025; Hochlehnert et al., 2025). Building on the DeepSeek-R1 framework (Shao et al., 2024; Guo et al., 2025), a range of new RL algorithms have been proposed, including Dr. GRPO (Liu et al., 2025), DAPO (Yu et al., 2025a), REINFORCE++ (Hu, 2025), VinePPO (Kazemnejad et al., 2024), and VAPO (Yue et al., 2025b). In parallel, extensive empirical studies have explored the design space of RL for reasoning (Zeng et al., 2025; Team et al., 2025), focusing on dimensions such as curriculum learning (Wen et al., 2025; Luo et al., 2025) and reward design (Gao et al., 2024; Cui et al., 2025a; Ma et al., 2023; Qi et al., 2025). While early progress has centered on mathematical reasoning, recent work has extended RL-based methods to code and software engineering tasks (Liu & Zhang, 2025; Xie et al., 2025; Wei et al., 2025; Yang et al., 2025; Chen et al., 2025; Li et al., 2025b), as well as to agentic problem-solving scenarios (Wang et al., 2025; Jin et al., 2025; Jiang et al., 2025; Xue et al., 2025).

Decomposing thinking and answering processes.
Traditional studies on LLM reasoning ability often treat model responses holistically. In contrast, a recent line of research explicitly decomposes the LLM response into a thinking trace ${\bm{z}}$ and a final answer ${\bm{y}}$, given a question ${\bm{x}}$ (Chen et al., 2024; Xiang et al., 2025; Zhou et al., 2025; Zhong et al., 2025). This decomposition offers several novel and useful perspectives.

Zhou et al. (2025) propose VeriFree, which directly optimizes $P_{\theta}({\bm{y}}|{\bm{x}})=\mathbb{E}_{\pi_{\theta}({\bm{z}}|{\bm{x}})}[\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})]$ using policy gradient. Their algorithm simultaneously optimizes $\pi_{\theta}({\bm{z}}|{\bm{x}})$ via policy gradient methods with $\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})$ as a reward, and performs weighted SFT on $\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})$. This approach demonstrates strong performance in general domains where rule-based verifiers are typically unavailable. Subsequent works (Yu et al., 2025b; Xu et al., 2025) further improve upon this by reshaping the reward, e.g., intuitively replacing the product of token probabilities with the mean when computing $\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})$.

Chen et al. (2024) introduce LaTRO, formulating reasoning as sampling from a latent distribution $q({\bm{z}}|{\bm{x}})$ and optimizing $\log P_{\theta}({\bm{y}}|{\bm{x}})$ via a variational manner. Their derived lower bound is:

$$ $\log P_{\theta}({\bm{y}}|{\bm{x}})\geq\mathbb{E}_{q({\bm{z}}|{\bm{x}})}\left[\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})\right]-\mathbb{D}_{\textrm{KL}}\left(q({\bm{z}}|{\bm{x}})\parallel\pi_{\theta}({\bm{z}}|{\bm{x}})\right).$ $$

They set the proposal distribution $q({\bm{z}}|{\bm{x}})$ to $\pi_{\theta}({\bm{z}}|{\bm{x}})$, resulting in a reinforcement learning algorithm where $\log\pi_{\theta}({\bm{y}}|{\bm{x}},{\bm{z}})$ serves as the reward. Tang et al. (2025) and Ruan et al. (2025) tighten this bound using ideas similar to IWAE (Burda et al., 2015).

A more natural choice for the variational distribution is the true posterior $P_{\theta}({\bm{z}}|{\bm{x}},{\bm{y}})$, though it is intractable. Hoffman et al. (2023) use MCMC to sample from the posterior, while Hu et al. (2024) employ GFlowNets (Bengio et al., 2023) to fine-tune an LLM to approximate it. Both methods use an EM-like algorithm to optimize the ELBO of $\log P_{\theta}({\bm{y}}|{\bm{x}})$.

Our approach uses forward KL divergence to train a variational posterior and derives a novel objective based on a tighter IWAE-style bound. Additionally, we propose an accuracy-based estimator for $\pi_{\theta}(\mathcal{Y}_{\bm{x}}|{\bm{x}},{\bm{z}}_{k})$, instead of the likelihood-based estimator used in Zhou et al. (2025). We also build connections to other mainstream finetuning algorithms that enhance reasoning, such as RFT and GRPO.

## Appendix C Training details

In this section, we detail the training procedure used in our method. Our framework builds on LLaMA-Factory (Zheng et al., 2024). By default, SFT averages token-level cross-entropy over all valid tokens in a batch. However, as shown in Section [2](https://arxiv.org/html/2509.22637v2#S2), our variables ${\bm{z}}$ and ${\bm{y}}$ are defined at the sentence level. To align with this, we modify the objective: instead of normalizing by the number of valid tokens, we sum the loss over all tokens and divide by a constant equal to the average response length in the training set (precomputed offline). This change parallels the difference between GRPO (Shao et al., 2024) and Dr. GRPO (Liu et al., 2025), thus we name this slight modification as Dr. SFT. Both our models and Bespoke-Stratos-4B/8B^† are trained with this modified objective. We further extend the framework with weighted SFT, as the original LLaMA-Factory does not support weighting. This feature is essential for parts of our method that require weighted training. All experiments are conducted on NVIDIA H100 GPUs.

### C.1 Training the initial reasoning model and variational posterior

For training, we use the following settings:

- 1.
Initial reasoning model $\pi_{\theta_{0}}$ (following Labs (2025)):
•
batch_size=96, cutoff_len=16384
•
Optimizer: AdamW (Loshchilov & Hutter, 2019) with adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1.0e-8, weight_decay=0
•
Learning rate schedule: cosine with warmup_ratio=0.1
•
learning_rate=1.0e-5, max_grad_norm=1.0
•
Training for 3 epochs
•
Precision: bfloat16
•
Baselines (Bespoke-Stratos-7B/32B and Bespoke-Stratos-4B/8B^†) are trained with the same setup
- 2.
Variational posterior $q_{\phi}$:
•
batch_size=16
•
Optimizer: AdamW with adam_beta1=0.9, adam_beta2=0.95, adam_epsilon=1.0e-8, weight_decay=1.0e-4
•
warmup_ratio=0.05
•
Training for 10 epochs
•
All other hyperparameters follow those used for $\pi_{\theta_{0}}$

### C.2 Details of sampling from variational posterior

Using the trained variational posterior $q_{\phi}$, we sample 8 reasoning traces (including final answers) for each question in Bespoke-Stratos-17k with vLLM (Kwon et al., 2023), using temperature=0.7, top_p=1.0, top_k=-1, max_tokens=32764, and dtype=bfloat16. After obtaining the sampled reasoning traces, we compute the importance weight $\widetilde{\rho}_{k}$ for each question-thinking-answer triplet using the pre-trained initial reasoning model $\pi_{\theta_{0}}$ and the variational posterior $q_{\phi}$. This is done in forward mode by evaluating the log-likelihoods under both models, without requiring backpropagation, which is efficient.

To estimate the term $\pi_{\theta}(\mathcal{Y}_{\bm{x}}\mid{\bm{x}},{\bm{z}})$ used in $\widetilde{\rho}_{k}$, we adopt an *accuracy-based estimator*. Specifically, for each question and each sampled thinking trace, we use $\pi_{\theta}$ to generate 8 answers under the same sampling configuration as above. The correctness of these answers is evaluated using the math/code verifiers from SkyThought, and the average accuracy is taken as the estimate of $\pi_{\theta}(\mathcal{Y}_{\bm{x}}\mid{\bm{x}},{\bm{z}})$. Experiments utilizing the accuracy-based estimator are labeled as “-Acc”.

Additionally, we employ the geometric mean of token-level probabilities under $\pi_{\theta}(\mathcal{Y}_{\bm{x}}\mid{\bm{x}},{\bm{z}})$ as an alternative and intuitive estimator. This approach mitigates the inherent length bias present in the strict definition of $\pi_{\theta}(\mathcal{Y}_{\bm{x}}\mid{\bm{x}},{\bm{z}})$, which computes the product of token probabilities and consequently assigns excessively small values to longer reasoning traces. This estimator provides an intuitive approximation without requiring an external verifier. Experiments utilizing the estimator based on geometric mean are labeled as “-GML”, while those with the naive estimator are labeled as “-L”.

This process results in a weighted dataset where each sample consists of a question-thinking-answer triplet along with its corresponding weight $\widetilde{\rho}_{k}$, which will be utilized in subsequent training stages.

### C.3 Details of training final reasoning model

To train the final reasoning model $\pi_{\theta}$, we adopt the following procedure. For the 17k data setting, we select, for each question, the reasoning trace with the highest importance weight $\widetilde{\rho}_{k}$ among the 8 samples sampled from the variational posterior. In experiments using the accuracy-based estimator, we pair the selected reasoning trace with a randomly chosen verified answer generated by the initial reasoning model. For other estimators, the original answer from the dataset is retained. The resulting synthetic data is then mixed with the original Bespoke-Stratos-17k dataset.

We maintain the same training configuration as used for the initial reasoning model, with one exception: the batch size is increased to batch_size=192. This adjustment ensures that the total number of optimization steps remains consistent with baseline models (e.g., Bespoke-Stratos-32B), as the mixed dataset is twice the size of the original.

For the 1k data setting, the baseline model (e.g., Bespoke-Stratos-7B-1K^†) is trained with the following configuration: We adopt batch_size=16 and cutoff_len=32768. We use the AdamW optimizer (Loshchilov & Hutter, 2019) with parameters adam_beta1=0.9, adam_beta2=0.95, adam_epsilon=1.0e-8, and weight_decay=1.0e-4. A cosine learning rate schedule is applied with warmup_ratio=0.1, alongside a learning rate of 1.0e-5 and gradient clipping at max_grad_norm=1.0. Training is conducted for 5 epochs.

In our method under the 1k setting, we do not combine with the original dataset. Instead, we use all 8 reasoning traces, weighted by $\widetilde{\rho}_{k}$, which is faithful to Algorithm [1](https://arxiv.org/html/2509.22637v2#alg1). To match the number of optimization steps in the baseline, we proportionally adjust the batch size while keeping all other hyperparameters unchanged.

## Appendix D Details of evaluation

We conduct all evaluations using SkyThought, specifically at commit 0d190f1.(^3^33[https://github.com/NovaSky-AI/SkyThought](https://github.com/NovaSky-AI/SkyThought)) Team (2025b) suggest avoiding greedy decoding for models with long thinking traces.
Thus, responses are sampled from the models using temperature=0.7 and top_p=1.0.
A generous token budget of max_tokens=38912 is allocated to accommodate lengthy outputs.

To maximize reproducibility, we perform inference using dtype=float32, accepting a potential decrease in speed for improved consistency. For model parallelism, we configure tensor_parallel_size=4 for 4B/7B/8B models and tensor_parallel_size=8 for the 32B models. We choose vLLM (Kwon et al., 2023) as the inference backend.
For models based on Qwen2.5, we use vllm==0.7.0, while for Qwen3-based models, we use vllm==0.8.4.
Although we anticipate that these version differences have negligible impact on evaluation accuracy, we document them here to ensure full reproducibility.

## Appendix E Prompt templates

In Section [2](https://arxiv.org/html/2509.22637v2#S2), we abstractly introduced how we define the prompt patterns used in the reasoning model $\pi_{\theta}$ and variational posterior $q_{\phi}$.
In this section, we provide details of the prompt templates used in practice, as shown below.

## Appendix F Extended results

### F.1 Extended main results

Due to space constraints in the main paper, we present extended evaluation results in this section.

We report the evaluation results for models fine-tuned from Qwen2.5-7B-Instruct on the Bespoke-Stratos-7B dataset in Table [6](https://arxiv.org/html/2509.22637v2#A6.T6). All variants of our method outperform all baselines in terms of average accuracy, demonstrating the superiority of the variational reasoning approach. Notably, the two prompt templates used for the variational posterior $q_{\phi}$ yield similar results, indicating that our method is robust to the choice of template. We attribute this robustness to the fact that the posterior is obtained by fine-tuning the model $q_{\phi}$, rather than through prompt engineering alone, thereby reducing the sensitivity to specific prompt formulations.

**Table 6: Performance of models trained from Qwen2.5-7B-Instruct trained on Bespoke-Stratos-17k. The best and second-best results are highlighted using bold text and underlined text, respectively.**
| Method | MATH500 | AIME24 | AIME25 | AMC23 | OlympiadBench | Avg |
| --- | --- | --- | --- | --- | --- | --- |
| Avg@2 | Avg@32 | Avg@32 | Avg@32 | Avg@2 |  |  |
| Qwen2.5-7B-Instruct | 75.60 | 10.94 | 7.40 | 51.10 | 39.91 | 36.99 |
| Bespoke-Stratos-7B | 82.20 | 19.58 | 19.48 | 63.28 | 45.03 | 45.91 |
| RLT-7B | 84.30 | 22.81 | 19.48 | 64.84 | 46.43 | 47.57 |
| Ours-PA-GML-7B | 85.30 | 24.17 | 20.42 | 68.20 | 46.88 | 48.99 |
| Ours-PA-Acc-7B | 83.40 | 22.50 | 20.83 | 65.39 | 47.55 | 47.94 |
| Ours-PB-GML-7B | 84.00 | 22.08 | 20.42 | 66.80 | 46.29 | 47.92 |
| Ours-PB-Acc-7B | 84.80 | 23.96 | 19.69 | 65.00 | 48.15 | 48.32 |
| Method | GPQA-D | LCB-E | LCB-M | LCB-H | MMLU-Pro | Avg |
| Avg@8 | Avg@8 | Avg@8 | Avg@8 | Avg@1 |  |  |
| Qwen2.5-7B-Instruct | 29.99 | 62.50 | 18.20 | 3.35 | 48.2 | 32.45 |
| Bespoke-Stratos-7B | 39.02 | 69.30 | 23.06 | 2.95 | 60.59 | 38.98 |
| RLT-7B | 41.60 | 72.32 | 25.06 | 3.66 | 61.28 | 40.78 |
| Ours-PA-GML-7B | 43.62 | 74.52 | 27.79 | 4.78 | 60.88 | 42.32 |
| Ours-PA-Acc-7B | 43.56 | 74.45 | 29.43 | 4.37 | 60.72 | 42.51 |
| Ours-PB-GML-7B | 41.60 | 74.73 | 28.46 | 4.57 | 61.14 | 42.10 |
| Ours-PB-Acc-7B | 41.67 | 75.07 | 29.49 | 4.68 | 61.28 | 42.44 |

Additionally, we plot the distributions of the thinking token length versus the log-likelihood ratio $\log\frac{\pi_{\theta}({\bm{z}}_{k}\mid{\bm{x}})}{q_{\phi}({\bm{z}}_{k}\mid{\bm{x}},{\bm{y}}^{\prime})}$ and the answer token length versus the log-likelihood of the answer $\log\pi_{\theta}(\mathcal{Y}_{\bm{x}}\mid{\bm{x}},{\bm{z}}_{k})$ in Figure [4](https://arxiv.org/html/2509.22637v2#A6.F4). The results reveal strong correlations between these variables, indicating the presence of length biases. This observation further justifies the use of estimators based on accuracy or the geometric mean of token likelihood, rather than the naive likelihood.

Figure: Figure 4: Density maps of the thinking token length versus the log-likelihood ratio $\log\frac{\pi_{\theta}({\bm{z}}_{k}\mid{\bm{x}})}{q_{\phi}({\bm{z}}_{k}\mid{\bm{x}},{\bm{y}}^{\prime})}$ (left), and the answer token length versus the log-likelihood of the answer $\log\pi_{\theta}(\mathcal{Y}_{\bm{x}}\mid{\bm{x}},{\bm{z}}_{k})$ (right).
Refer to caption: x4.png

Another noteworthy observation is that our evaluation results for General-Reasoner-4B (see Table [1](https://arxiv.org/html/2509.22637v2#S4.T1)) differ from those reported by Ma et al. (2025) in their original paper, despite using their officially released checkpoints and provided prompt template. This discrepancy can be attributed to several factors: (1) different evaluation frameworks: we employ SkyThought whereas they utilize simple-evals(^4^44[https://github.com/openai/simple-evals](https://github.com/openai/simple-evals)); (2) different sampling configurations: we use temperature=0.7 and max_tokens=38912, while they primarily employ greedy decoding (i.e., temperature=0) except for AIME24 and AIME25, along with a more constrained token budget of max_tokens=8192.

To facilitate a fair comparison, we provide results on the common benchmarks that we and Ma et al. (2025) both utilize, comparing our model accuracy with their officially reported values (see Table [7](https://arxiv.org/html/2509.22637v2#A6.T7)). As demonstrated, our method continues to outperform General-Reasoner-4B^‡ by a significant margin (where ^‡ indicates their officially reported accuracy).

**Table 7: Performance of models trained from Qwen3-4B-Base. All models are trained on Bespoke-Stratos-17k except for General-Reasoner-4B. Here, ^‡ denotes accuracy values officially reported by Ma et al. (2025), rather than results obtained through our own evaluation. The best and second-best results are highlighted using bold text and underlined text, respectively.**
| Method | MATH500 | AIME24 | AIME25 | AMC23 |
| --- | --- | --- | --- | --- |
| Qwen3-4B-Base | 45.30 | 4.79 | 5.73 | 27.73 |
| General-Reasoner-4B | 71.70 | 19.06 | 16.77 | 55.00 |
| General-Reasoner-4B^‡ | 80.6 | 20.0 | 15.4 | 60.0 |
| Bespoke-Stratos-4B^† | 84.70 | 27.29 | 24.17 | 70.16 |
| Ours-PB-GML-4B | 87.30 | 33.54 | 26.77 | 74.06 |
| Ours-PB-Acc-4B | 88.30 | 31.67 | 27.29 | 75.63 |
| Method | OlympiadBench | GPQA-D | MMLU-Pro |  |
| Qwen3-4B-Base | 23.37 | 29.10 | 36.89 |  |
| General-Reasoner-4B | 45.18 | 40.97 | 61.36 |  |
| General-Reasoner-4B^‡ | 47.7 | 42.9 | 62.8 |  |
| Bespoke-Stratos-4B^† | 50.45 | 44.95 | 63.03 |  |
| Ours-PB-GML-4B | 54.45 | 45.52 | 65.52 |  |
| Ours-PB-Acc-4B | 55.71 | 45.33 | 65.53 |  |

### F.2 Extended ablation studies

This section presents extended ablation studies analyzing: the impact of different training data sources and different ways of data usage (Appendix [F.3](https://arxiv.org/html/2509.22637v2#A6.SS3)); the effect of data overlap between variational posterior training and reasoning model training (Appendix [F.4](https://arxiv.org/html/2509.22637v2#A6.SS4)); the comparison between Dr. SFT and naive SFT (Appendix [F.4](https://arxiv.org/html/2509.22637v2#A6.SS4)); and the influence of cutoff length during training (Appendix [F.6](https://arxiv.org/html/2509.22637v2#A6.SS6)).

### F.3 Effects of different final data sources and ways of data usage

In our main experiments, we prioritize training efficiency by using the 17k data setting, selecting the variational reasoning trace ${\bm{z}}_{k}$ with the highest weight $\widetilde{\rho}_{k}$ and mixing it with the original Bespoke-Stratos-17k data (which results in double data sizes), rather than using all eight traces sampled from the variational posterior. To evaluate the impact of this simplification, we conduct ablation studies under the 1k data setting. We compare variants that either mix or do not mix with the original data, and that use either single-best trace selection (“-S”) or weighted multiple traces (“-M”).

Results are shown in Table [8](https://arxiv.org/html/2509.22637v2#A6.T8). The best performance is achieved by the variant that uses weighted multiple reasoning traces without mixing with the original data. This suggests that, when computational cost is not a constraint, the optimal approach is to utilize all reasoning traces from the variational posterior, weighted by $\widetilde{\rho}_{k}$, for training the final reasoning model. Another interesting observation is that for the single-trace method, data mixing improves performance, whereas for the weighted multi-trace method, mixing slightly degrades performance. This may indicate that the weighted ensemble of variational traces already provides sufficient information, making the original data redundant in this scenario.

**Table 8: Ablation study on the effects of different final data sources (only sampled from variational posterior vs. mixed) and different ways to use samples from variational posterior to train reasoning models (single best reasoning trace selection (“-S”) vs. weighted multiple reasoning traces (“-M”)). This ablation is done in data 1k setting. The best and second-best results are highlighted using bold text and underlined text, respectively.**
| Method | MATH500 | AIME24 | AIME25 | AMC23 | OlympiadBench | Avg |
| --- | --- | --- | --- | --- | --- | --- |
| Avg@2 | Avg@32 | Avg@32 | Avg@32 | Avg@2 |  |  |
| Qwen2.5-7B-Instruct | 75.60 | 10.94 | 7.40 | 51.10 | 39.91 | 36.99 |
| Bespoke-Stratos-7B-1K^† | 77.20 | 16.25 | 13.96 | 53.75 | 40.88 | 40.41 |
| Ours-M-7B-1K | 79.80 | 18.65 | 16.98 | 60.55 | 44.81 | 44.16 |
| w/o Mix | 81.30 | 19.69 | 18.44 | 61.64 | 45.99 | 45.41 |
| Ours-S-7B-1K | 81.10 | 19.90 | 17.08 | 60.39 | 43.92 | 44.48 |
| w/o Mix | 80.40 | 17.92 | 17.81 | 59.06 | 43.18 | 43.67 |
| Method | GPQA-D | LCB-E | LCB-M | LCB-H | MMLU-Pro | Avg |
| Avg@8 | Avg@8 | Avg@8 | Avg@8 | Avg@1 |  |  |
| Qwen2.5-7B-Instruct | 29.99 | 62.50 | 18.20 | 3.35 | 48.2 | 32.45 |
| Bespoke-Stratos-7B-1K^† | 37.94 | 60.37 | 13.59 | 1.22 | 56.07 | 33.84 |
| Ours-M-7B-1K | 42.80 | 65.73 | 18.99 | 1.93 | 60.49 | 37.99 |
| w/o Mix | 41.16 | 68.13 | 21.42 | 1.42 | 60.94 | 38.62 |
| Ours-S-7B-1K | 40.34 | 64.84 | 18.20 | 2.13 | 59.54 | 37.01 |
| w/o Mix | 39.96 | 65.04 | 18.27 | 1.32 | 59.87 | 36.89 |

### F.4 Effects of data overlap

In the 17k data setting, the variational posterior $q_{\phi}$ is trained on all 17k samples and generates thinking traces for the same set of 17k samples, which are subsequently used to train the final reasoning model $\pi_{\theta}$. In other words, both the variational posterior and the final reasoning model are trained on the same set of question–answer pairs.

An interesting question is how our method performs in the absence of data overlap. To investigate this, we design two experimental settings: the first is the same 1k data setting introduced earlier; the second is constructed by splitting Bespoke-Stratos-17k into two non-overlapping subsets: one contains 15,710 samples (approximately 16k) and the other contains 1k samples. In the latter setting, we train the variational posterior $q_{\phi}$ and the initial reasoning model $\pi_{\theta_{0}}$ on the 16k subset. We then use the trained $q_{\phi}$ to sample thinking traces for the 1k subset, and employ both $q_{\phi}$ and $\pi_{\theta_{0}}$ to compute $\widetilde{\rho}_{k}$. Finally, the final reasoning model $\pi_{\theta}$ is trained on the 1k subset using weighted multiple reasoning traces. This setting is referred to as “w/o Overlap”.

The results are presented in Table [9](https://arxiv.org/html/2509.22637v2#A6.T9). Both the overlap and non-overlap variants exhibit similar performance in terms of average accuracy, and both outperform the baseline, Bespoke-Stratos-7B-1K^†. This suggests that the trained variational posterior generalizes reasonably well and can be applied to broader scenarios.

**Table 9: Ablation study on the effects of data overlap between variational posterior training and reasoning model training. This ablation is done in data 1k setting. The best and second-best results are highlighted using bold text and underlined text, respectively.**
| Method | MATH500 | AIME24 | AIME25 | AMC23 | OlympiadBench | Avg |
| --- | --- | --- | --- | --- | --- | --- |
| Avg@2 | Avg@32 | Avg@32 | Avg@32 | Avg@2 |  |  |
| Qwen2.5-7B-Instruct | 75.60 | 10.94 | 7.40 | 51.10 | 39.91 | 36.99 |
| Bespoke-Stratos-7B-1K^† | 77.20 | 16.25 | 13.96 | 53.75 | 40.88 | 40.41 |
| Ours-7B-1K | 79.80 | 18.65 | 16.98 | 60.55 | 44.81 | 44.16 |
| w/o Mix | 81.30 | 19.69 | 18.44 | 61.64 | 45.99 | 45.41 |
| w/o Overlap | 80.60 | 20.83 | 18.44 | 61.17 | 44.81 | 45.17 |
| w/o Mix w/o Overlap | 82.00 | 18.75 | 16.88 | 63.52 | 47.11 | 45.65 |
| Method | GPQA-D | LCB-E | LCB-M | LCB-H | MMLU-Pro | Avg |
| Avg@8 | Avg@8 | Avg@8 | Avg@8 | Avg@1 |  |  |
| Qwen2.5-7B-Instruct | 29.99 | 62.50 | 18.20 | 3.35 | 48.2 | 32.45 |
| Bespoke-Stratos-7B-1K^† | 37.94 | 60.37 | 13.59 | 1.22 | 56.07 | 33.84 |
| Ours-7B-1K | 42.80 | 65.73 | 18.99 | 1.93 | 60.49 | 37.99 |
| w/o Mix | 41.16 | 68.13 | 21.42 | 1.42 | 60.94 | 38.62 |
| w/o Overlap | 38.19 | 66.00 | 19.84 | 1.12 | 60.65 | 37.16 |
| w/o Mix w/o Overlap | 39.65 | 68.82 | 19.96 | 2.44 | 60.36 | 38.25 |

### F.5 Comparing Dr. SFT with naive SFT

As detailed in Appendix [C](https://arxiv.org/html/2509.22637v2#A3), we employ a slightly modified objective function, Dr. SFT, where the loss is defined as the sum of all valid token losses normalized by a constant, rather than the mean loss across valid tokens in the batch. We conduct an ablation study comparing this Dr. SFT approach against naive SFT when training the final reasoning model $\pi_{\theta}$.

The results are presented in Table [10](https://arxiv.org/html/2509.22637v2#A6.T10). Both variants demonstrate comparable performance, with less than 2% difference in average accuracy, and both outperform the baseline. This allows us to conclude that the primary performance improvement stems from the variational reasoning mechanism rather than from this minor modification to the objective function.

### F.6 Ablation study on the effects of cutoff length when training

In our main experiments (17k data setting), we use a cutoff length of cutoff_len=16384. To investigate the impact of this hyperparameter, we conduct an ablation study comparing two variants: one using the default cutoff_len=16384 (denoted as “-Len16k”) and another with cutoff_len=32768 (denoted as “-Len32k”).

The results are presented in Table [11](https://arxiv.org/html/2509.22637v2#A6.T11). Both variants exhibit similar performance. We further analyze the average completion token lengths on several evaluation benchmarks (see Table [12](https://arxiv.org/html/2509.22637v2#A6.T12)). All methods produce significantly longer reasoning traces compared to the Qwen2.5-7B-Instruct, with our methods generating slightly longer thinking traces. Notably, the 16k and 32k cutoff variants result in similar generation lengths during inference. This indicates that increasing the cutoff length beyond 16k has a minimal effect on the model’s output. Therefore, we can confidently use the 16k setting for better training efficiency without sacrificing performance.

**Table 10: Ablation study comparing Dr. SFT and Naive SFT. This ablation is done in data 17k setting. The best and second-best results are highlighted using bold text and underlined text, respectively.**
| Method | MATH500 | AIME24 | AIME25 | AMC23 | OlympiadBench | Avg |
| --- | --- | --- | --- | --- | --- | --- |
| Avg@2 | Avg@32 | Avg@32 | Avg@32 | Avg@2 |  |  |
| Qwen2.5-7B-Instruct | 75.60 | 10.94 | 7.40 | 51.10 | 39.91 | 36.99 |
| Bespoke-Stratos-7B | 82.20 | 19.58 | 19.48 | 63.28 | 45.03 | 45.91 |
| RLT-7B | 84.30 | 22.81 | 19.48 | 64.84 | 46.43 | 47.57 |
| Ours-PA-GML-7B | 85.30 | 24.17 | 20.42 | 68.20 | 46.88 | 48.99 |
| w/ naive SFT | 84.00 | 22.60 | 20.10 | 65.31 | 49.11 | 48.23 |
| Ours-PA-Acc-7B | 83.40 | 22.50 | 20.83 | 65.39 | 47.55 | 47.94 |
| w/ naive SFT | 84.10 | 23.02 | 21.04 | 67.66 | 46.96 | 48.56 |
| Method | GPQA-D | LCB-E | LCB-M | LCB-H | MMLU-Pro | Avg |
| Avg@8 | Avg@8 | Avg@8 | Avg@8 | Avg@1 |  |  |
| Qwen2.5-7B-Instruct | 29.99 | 62.50 | 18.20 | 3.35 | 48.2 | 32.45 |
| Bespoke-Stratos-7B | 39.02 | 69.30 | 23.06 | 2.95 | 60.59 | 38.98 |
| RLT-7B | 41.60 | 72.32 | 25.06 | 3.66 | 61.28 | 40.78 |
| Ours-PA-GML-7B | 43.62 | 74.52 | 27.79 | 4.78 | 60.88 | 42.32 |
| w/ naive SFT | 42.49 | 74.73 | 25.85 | 3.66 | 61.41 | 41.63 |
| Ours-PA-Acc-7B | 43.56 | 74.45 | 29.43 | 4.37 | 60.72 | 42.51 |
| w/ naive SFT | 42.11 | 75.69 | 27.79 | 5.39 | 61.22 | 42.44 |

**Table 11: Ablation study on effects of cutoff length used in training. This ablation is done in data 17k setting. Len16k: cutoff_len=16384; Len32k: cutoff_len=32768. The best and second-best results are highlighted using bold text and underlined text, respectively.**
| Method | MATH500 | AIME24 | AIME25 | AMC23 | OlympiadBench | Avg |
| --- | --- | --- | --- | --- | --- | --- |
| Avg@2 | Avg@32 | Avg@32 | Avg@32 | Avg@2 |  |  |
| Qwen2.5-7B-Instruct | 75.60 | 10.94 | 7.40 | 51.10 | 39.91 | 36.99 |
| Bespoke-Stratos-7B | 82.20 | 19.58 | 19.48 | 63.28 | 45.03 | 45.91 |
| RLT-7B | 84.30 | 22.81 | 19.48 | 64.84 | 46.43 | 47.57 |
| Ours-PA-GML-7B-Len16k | 85.30 | 24.17 | 20.42 | 68.20 | 46.88 | 48.99 |
| Ours-PA-GML-7B-Len32k | 84.10 | 22.08 | 20.94 | 66.80 | 48.37 | 48.46 |
| Method | GPQA-D | LCB-E | LCB-M | LCB-H | MMLU-Pro | Avg |
| Avg@8 | Avg@8 | Avg@8 | Avg@8 | Avg@1 |  |  |
| Qwen2.5-7B-Instruct | 29.99 | 62.50 | 18.20 | 3.35 | 48.2 | 32.45 |
| Bespoke-Stratos-7B | 39.02 | 69.30 | 23.06 | 2.95 | 60.59 | 38.98 |
| RLT-7B | 41.60 | 72.32 | 25.06 | 3.66 | 61.28 | 40.78 |
| Ours-PA-GML-7B-Len16k | 43.62 | 74.52 | 27.79 | 4.78 | 60.88 | 42.32 |
| Ours-PA-GML-7B-Len32k | 42.49 | 74.93 | 28.58 | 4.37 | 61.64 | 42.40 |

**Table 12: Average completion token length of models trained from Qwen2.5-7B-Instruct.**
| Method | MATH500 | AIME24 | AIME25 | AMC23 | MMLU-Pro |
| --- | --- | --- | --- | --- | --- |
| Avg@2 | Avg@32 | Avg@32 | Avg@32 | Avg@1 |  |
| Qwen2.5-7B-Instruct | 564 | 1270 | 1027 | 849 | 531 |
| Bespoke-Stratos-7B | 5801 | 18413 | 15769 | 10921 | 3889 |
| RLT-7B | 5508 | 18143 | 15769 | 10986 | 3942 |
| Ours-PA-GML-7B | 5677 | 18299 | 16471 | 11338 | 3924 |
| Ours-PA-GML-7B-Len32k | 5688 | 18170 | 16747 | 11531 | 3965 |
| Ours-PA-Acc-7B | 5688 | 18170 | 16747 | 11531 | 3965 |
| Ours-PB-GML-7B | 5803 | 18479 | 16615 | 11080 | 4052 |
| Ours-PB-Acc-7B | 5787 | 18651 | 16696 | 11591 | 3974 |

## Appendix G Examples of generated responses

In this section, we provide examples of responses from the variational posterior $q_{\phi}$ and the final reasoning model $\pi_{\theta}$ trained based on Qwen3-4B-Base. The provided example of the variational posterior $q_{\phi}$ is based on the variant with prompt template B, and the question-answer pair is one of the samples from the Bespoke-Stratos-17k dataset. The provided example of the final reasoning model $\pi_{\theta}$ is based on one of the samples in MATH500.
See the provided examples below.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: See questions in weakness part
soundness: 2
strengths: * Authors address the important topic of injecting reasoning into LLMs by providing a probabilistic perspective. Theorems and proofs are clear to understand. There are some parts of the method that require additional discussion (see weakness), but it is overall a well-polished work.
* The paper provides a novel perspective on RFT/GRPO and can have impact on future works.
* Experiments are well conducted on serveral benchmarks spanning different aspects of reasoning. Results show consistent gains across multiple model scales.
summary: The paper proposes a variational inference framework for training LLM reasoning model by treating thinking traces as latent variables and maximizing evidence lower bound. Key innovations include an IWAE-style multi-trace objective that tightens the bound with multiple samples, as well as optimization via forward KL divergence to avoid mode collapse. Authors show existing methods like rejection sampling finetuning and GRPO can be treated as local forward KL objectives with implicit reweighting. Experiments on Qwen models show consistent improvements over baselines on math, coding and general reasoning benchmarks.
weaknesses: * The mode collapse issue mentioned in section 2.3 brings up an important research question for introducing latent reasoning in LLMs but may require additional elaboration. The proposed forward KL is claimed to prevent mode collapse, but weighted SFT still concentrate probability on high-accuracy traces. My understanding is that the model will still collapse to the "correct" modes while sacrificing diversity. I wonder if the authors can provide more insights and experiment results on this part.
* Authors mention that the method currently is applied to only single iteration, while standard variational EMs alternate until close to convergence. Is there any reason why T>1 wasn't specifically tested? Will we observe certain collapse or divergence, as commonly observed in experiments with EM algorithms? Additional results can improve the credibility especially given that the paper's claim focuses on mode collapse/ diverse reasoning. 
* The paper is missing discussion on computational overhead from the method as compared to other baselines, especially with multiple rollouts and multiple answer samples per trace. 
* Could the authors provide some insights on the scaling properties of the method, especially with different model sizes and different K?

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: The author mentions directly ELBO formulation with reverse KL will have mode collapse. How bad is it? Do you have small scale observations?
soundness: 3
strengths: 1. Novel and Insightful Theoretical Framework: The paper's primary strength is its principled unification of several reasoning-as-inference methods (SFT, RL) under a single probabilistic lens. Framing reasoning as a latent variable problem is elegant.

2. Valuable Unification and Bias Discovery: Showing that RFT and GRPO can be derived as forward-KL minimizations (Eq. 10 & 11) is a significant insight. The discovery of the implicit accuracy-weighting, and its resulting bias toward easier questions, is a valuable, non-obvious finding for the community that could influence future method design.

3. Strong Empirical Performance: The method achieves impressive and consistent gains across multiple model scales (4B to 32B). It surpasses strong baselines, for instance, by over 8.5% on average accuracy compared to Bespoke-Stratos-4B on the same training data (Table 1), demonstrating its effectiveness.
summary: This paper proposes a framework for training reasoning models by treating thinking traces as latent variables and optimizing the log-likelihood of correct answers using variational inference (VI). The authors derive an IWAE-style multi-trace objective (ELBO) to update the reasoning model $\pi_{\theta}$ and a forward-KL objective (Eq. 9) to stabilize the training of the variational posterior $q_{\phi}$.

A key theoretical contribution is showing that existing methods like Rejection Sampling Finetuning (RFT) and GRPO can be interpreted as local forward-KL objectives, revealing an implicit bias toward easier questions. Empirically, the method demonstrates strong performance, outperforming baselines like Bespoke-Stratos on several reasoning benchmarks.
weaknesses: 1. Extreme Methodological and Computational Complexity: The proposed training pipeline (Algorithm 1) is substantially more complex than the baselines. As detailed in Appendix C, the full process requires (at minimum) three separate, full finetuning runs ($\pi_{\theta_0}$, $q_{\phi}$, and $\pi_{\theta}$) plus a massive offline data generation and re-weighting pass. This re-weighting step (for the "-Acc" estimator) is itself a huge computational burden, requiring lots of verifications. 

2. The final algorithm essentially boils down to two complex, weighted SFT steps. This raises the critical question: is the VI formalism necessary? The argument for its necessity is weakened by the authors' own admission that they must modify the "principled" importance weight (Eq. 8, using a geometric mean) to mitigate high variance, thereby introducing bias and moving away from the pure ELBO derivation. The paper is missing the important ablation: a comparison against a simpler weighted SFT (e.g., using only verifier accuracy as a weight) to prove that the complex $\pi_{\theta} / q_{\phi}$ importance-weighting is truly beneficial.

3. While Appendix C provides training details, the paper presents its strong empirical results without a clear accounting of the total compute. The performance gains are achieved with what is likely an orders-of-magnitude increase in total compute over the SFT baseline. Without a compute-matched baseline (e.g., SFT on more data or for more epochs), it is impossible to assess the actual efficiency of the method.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: How does the variational posterior q_{\phi} scale with model size? Is there evidence that a separate learned posterior remains necessary for very large models, or can it be amortized within the base model?

How sensitive is the proposed method to the quality or accuracy of the “hint” used in the variational posterior? Would the framework still hold when the hint is noisy, partially correct, or unavailable?
soundness: 3
strengths: The paper offers a clean and principled unification of reasoning-oriented training methods for LLMs under a single probabilistic framework. By treating reasoning traces as latent variables and optimizing an IWAE-style ELBO, it grounds both RL and supervised finetuning as approximate inference procedures, clarifying their implicit assumptions. The forward-KL formulation is a well-motivated and technically strong addition that improves stability without sacrificing generality. The paper is clearly written and empirically supported with competitive results across diverse reasoning domains. Overall, it provides both conceptual clarity and practical value, advancing the theoretical understanding of reasoning optimization in large models. 

There is a nice connection to older works of RL for non-LLM domains: see https://arxiv.org/pdf/1805.00909. They also use latent optimization via ELBO.
summary: The paper reframes reasoning in language models as a latent variable problem, where the intermediate “thinking trace” is treated as a hidden variable optimized through variational inference. Starting from the ELBO, the authors derive a tighter multi-trace objective and introduce a forward-KL formulation that stabilizes training of the variational posterior. This yields a unified probabilistic view that connects supervised finetuning, rejection sampling finetuning, and binary-reward RL as approximate inference objectives under the same framework. Empirically, the method consistently improves reasoning performance across math, code, and general benchmarks on Qwen models, showing both higher stability and generalization. Overall, it offers a principled bridge between likelihood-based reasoning and RL-style optimization.
weaknesses: I believe the authors haven't done a full literature search on prior RL works, such as https://arxiv.org/pdf/1805.00909, https://openreview.net/pdf?id=a147pIS2Co that discuss a similar idea in the non-LLM setting. Could the authors explain the differences to these works.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Please refer to the weaknesses section.
soundness: 3
strengths: - Leverages a principled variational inference framework to improve reasoning by treating thought traces as latent variables, providing a clean probabilistic formulation.
- Uses a forward-KL based objective to address instability and collapse issues in training the variational posterior, leading to more stable optimization.
- Algorithmic pipeline is clearly presented and appears reproducible, with explicit sampling, weighting, and update procedures.
- Strong empirical evaluation across diverse reasoning tasks (math, coding, and general benchmarks), including out-of-distribution settings.
- Comprehensive ablations that assess key design choices, such as the effect of the answer hint, different estimators for correctness probability, and varying the number of sampled traces.
summary: This paper applies variational inference to enhance language models' chain-of-thought reasoning by treating reasoning traces as latent variables and optimizing an ELBO-based objective. The authors further extend the objective to an IWAE-style multi-trace formulation for a tighter bound and introduce a stabilized training procedure using a forward-KL term. Experiments on math, coding, and general reasoning benchmarks show that this variational reasoning framework consistently outperforms SFT, RFT, and RL baselines, while also generalizing well to out-of-distribution tasks.
weaknesses: - The forward-KL update is largely heuristic and does not come with formal guarantees on bias or convergence. While it stabilizes training in practice, the theoretical trade-offs and conditions under which the approximation holds are not fully analyzed.

- The geometric-mean modification of the importance ratio reduces variance for long reasoning traces but may introduce systematic bias. The paper does not characterize the direction or magnitude of this bias, nor discuss potential failure modes.

- The method relies on external verifiers to estimate correctness probabilities. The overall stability and performance depend on verifier quality; imperfect or incomplete verifiers may mislabel valid/invalid reasoning paths and influence learning dynamics.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 1
questions: * Have you tried to use the variational posterior to generate high-quality thinking traces and use them to do simple SFT on the original LLM? How crucial are the VI/ELBO components in that case?
* Is it possible to extend this method to scenarios where a rule-based verifier doesn't exist?
* As a lightweight qualitative check, it would be interesting to compare the auxiliary model’s traces when conditioned on a correct vs. incorrect answer hint.
soundness: 2
strengths: * The proposed method is intuitive at a high level: to train an auxiliary LLM which can make use of a hint to generate better thinking trace, and use this model to improve the original one.
* Many techniques are proposed to make the method work in practice, including the tighter IWAE-style lower bound, accuracy-based estimator for lower worst-case variances, forward-KL to avoid the variational posterior collapsing to trivial solution, etc.
* The connection between RFT/binary reward RL and forward-KL is revealed, which also reveals their bias toward easier questions.
summary: This paper proposes a framework for fine-tuning a reasoning LLM to generate better “thinking traces”. Specifically, this paper first trains an auxiliary LLM to use the answer as a hint to generate better thinking traces, then trains the original LLM to generate similar high-quality thinking traces without seeing any hint. The idea is put in a variational framework: the thinking traces are formulated as latent variables and the auxiliary LLM and original LLM are alternately optimized for ELBO. The auxiliary LLM is viewed as a variational posterior, and a forward-KL objective is used to optimize it to avoid collapsing to shortcut reasoning. A tighter IWAE‑style multi‑trace bound is used, and practical tricks are proposed for it to work. Besides, The authors show that RFT and binary‑reward RL including GRPO are local forward‑KL updates implicitly weighted by model accuracy, explaining a bias towards easier questions. On math/code/general‑knowledge benchmarks with Qwen 2.5/3 models, the method outperforms strong baselines and trains more stably.
weaknesses: * The paper states that "our work provides a principled probabilistic perspective that unifies variational inference with RL-style methods". However, tricks like using geometric mean as a surrogate for the likelihood ratio and using forward-KL to optimize the variational posterior are not compatible with VI as far as I can see. I appreciate the method for its practical impact and clarity of intent, but I think connecting it to VI to make it "principled" may be overstating without additional justification or ablations.
* The presentation is difficult to follow on a first read. Algorithm 1 greatly clarifies the pipeline, but it appears after substantial technical detail. Reordering the exposition (high‑level pipeline, objective, implementation details) could make the paper more accessible to newcomers.


Decision: Accept (Poster)