Beyond Verifiable Rewards: Scaling Reinforcement Learning in Language Models to Unverifiable Data
Authors: Yunhao Tang
Sid Wang
Lovish Madaan
Remi Munos
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
LLMs
Reasoning
Unverifiable
Verifiable
Primary area: reinforcement_learning
TL;DR: Scaling RL-based reasoning to unverifiable data

Abstract: We propose to scale RL to unverifiable data with a novel algorithm JEPO (Jensen's Evidence lower bound for Policy Optimization). While most prior effort on scaling RL for LLMs focuses on verifiable data where ground truth answers are typically short-form and can be matched easily, we investigate the case where such assumptions are less valid (e.g., when answers are long-form such as mathematical proofs). To scale RL training to unverifiable data with contemporary training constraints, we propose JEPO. JEPO applies Jensen's evidence lower bound, a pragmatic simplification of the evidence lower bound which views chain-of-thought as a latent variable in the generative process. We show that on verifiable datasets (math), JEPO is as effective as RL with verifiable reward; on semi-verifiable and unverifiable datasets (numina and numina-proof), JEPO improves on soft-match based evaluations compared to RL with verifiable reward which can only leverage a subset of the data source as well as test set likelihood evaluations.

Beyond Verifiable Rewards: Scaling Reinforcement Learning in Language Models to Unverifiable Data
Yunhao Tang
Sid Wang Meta Platforms, Inc.
Lovish Madaan Meta Platforms, Inc.
Rémi Munos Meta Platforms, Inc.
Abstract
We propose to scale RL to unverifiable data with a novel algorithm JEPO (Jensen's s Evidence lower bound for Policy Optimization). While most prior effort on scaling RL for LLMs focuses on verifiable data where ground truth answers are typi- cally short-form and can be matched easily, we investigate the case where such assumptions are less valid (e.g., when answers are long-form such as mathematical proofs). To scale RI training to unverifiable data with contemporary training constraints, we propose JHPO. JEPO applies Jensen's evidence lower bound, a pragmatic simplification of the evidence lower bound which views chain-of-thougle latent variable in the generative process. We show that on verifiable datasets (math), JEPO as effective as RL with verifiable reward; on semi-verifiale and unverifiable datasets (numina and mumina-proof] JEPO improves soft-match based evaluations compared to RL with verifiable reward which can only leverage subset of the data source as well as test set likelihood evaluations.
Introduction
Reinforcement learning from verifiable reward (RLVR) has proved effective at endowing language models with capabilities beyond canonical pre-training and supervised fine-tuning [1, 3, 4, 5, 6]. At its core, reinforcement learning (RL) allows for the training of thain-of-thouglee at scale, which elicit significant performance improvements especially for reasoning intensive tasks [7, 8]. In the case of mathematical reasoning, it encourages hep-by-step solutions that lead up to a final answer [9, 10], whose correctness can be verified to produce reward signal for RL training.
However, a main limitation of current RLVR is the data source: verifiable rewards are mostly derived from datasets where ground truth answers are short-form and can be checked in relatively easy ways [4, 6]. For example, most answers to popular benchmarks MATH [11] and AIME [12] are integers and can be verified using simple string match. Yet, for many data sources, the ground truth answers unverifiable by current standards. For example, for long-form data with the answer being the whole proof, its inherent correctness is hard to assess without human raters [13].
less verifiable
The boundary between verifiable and unverfiable data, though often blurry in practice, can be made actionable: define data unverifiable, if its ground truth answer cannot be verified with reasonably simple automatic procedure.
of interest to scale RL to such data sources, for a few notable practical reasons: (1) some data have inherently long answers which cannot be cast into short-form answers a straightforward way; (2) data sources with long-form answers exist in abundance, and it is sub-optimal not to leverage such data for training. In this work, seek to tackle the problem of scaling RL to these unverifiable data sources.
We propose JEPO (short for Jensen's Evidence lower bound for Policy Optimization), a novel RL algorithm that can equally post-train models verifiable unverifiable data. The design of the algorithm is inspired by a latent variable view of chain-of-thour make of Jensen's 's evidence lower bound, novel pragmatic simplification of the full evidence lower bound [16, 17], named after Jensen's inequality [18]. Optimizing a simplified objective
[14, 15]. Different from prior work,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

forgoes the complication of training expensive auxiliary models, making JEPO more suitable for contemporary large-scale training [19, 20].
The final algorithm consists of an hybrid online RL and SFT-like loss. When viewed as an alternative online RL, the lower bound method does not require any external verifiable reward, forgoing the need for the ground truth to be easily verifiable. In sum, our technical contributions are as follows:
Algorithm Followed by brief background on latent variable modeling, we derive the Jensen's s evidence lower bound in Section 3, the stochastic optimization algorithm. In Section we show how its multi-sample extension [21] tightens the theoretical bound and alludes to better performance in practice. While JEPO works similarly as regular online RL baselines, we detail few important aspects of its implementations in Appendix B.2.
Theoretical Connections We offer an intriguing graphical model view of a few important al- gorithms connecting probabilistic inference with chain-of-thour of-tthugghht optimization. See details in Appendix A. We further discuss the practical trade-offs and connections between JEPO and online RL algorithms. Due space limits, such results are presented Appendix D.
Empirical Results Finally in Section 6, Section 7, and Section 8, we show that for verifiable data, JEPO is competitive compared to online RL with verifiable rewards. For semi-verifiable and unverifiable data, JEPO has performance advantage over online RL, SFT baselines. As a by-product, we also showcase the utility of generating chain-of-thouu for long-form proofs, an observation that interesting in its own right.
2
Reinforcement learning for language models
language model can be understood as a policy Ty in the context of reinforcement learning. Given a prompt x, the policy generates a response y, which then gets assessed by a human user. Usually, the objective is to optimize Te such that certain reward function r(z, y) that captures human preference maximized [22, 23]. Formally, consider the maximization problem
max E (-))-o))))) [r(a,, v)] /IKL. (To(-(a)), <re((-1=))
with KL. regularization that encourages Ty to stay close to the reference policy. The reward r(x, y) captures the human preference of response y in response to prompt and can take various forms: for example, can be extracted from human annotations [22, 24, 23], generated atuomatic feedback such as code execution [25]. We focus on specialized setting where the reward is derived from access to certain ground truth the problem.
2.1
RL from ground truth feedback
We focus applications where the prompt typically specifies question and there is an example of an desirable ground truth a". Such a formulation is applicable to mathematical reasoning [11, 26, 10] where x is question and a" is the ground truth answer. When the correctness of model generated answer a can be easily verified against the ground truth a", a verifiable reward r is available by matching a* against the answer a. As another example, when a" a long-form proof, such a reward is not immediately available and such cases are considered less verifiable. In broader context, RLVR also includes code applications where the reward is computed via unit tests [25, 27]. We do not consider such use cases in this work.
2.2 Chain-of-thought
For aforementioned applications where the model required to reason about the question æ and generate an answer a, getting the model to generate chain-of-thoughtt steps e leading up to the final conclusion [7, 8]. Henceforth, we can decompose the generation and an answer a. The generative process for the response y ~ n (x:) is made more concrete as e ~ T0(- ~ no(-|xr (1, c). Given a prompt æ, the intuitive role of ohain-of-thought is such that it makes the marginal likelihood of the ground truth answer a" higher. As such, we can interpret chain-of-thought as a latent variable and formulate the optimization of chain-of-tee ofain-hf-thought as latent variable modeling [15, 14].
sequence of reasoning
= (c. a) into chain-of-thouge

---

3 A Jensen's lower bound for chain-of-thought as latent variable modeling
We start with the initial motivation to increase the marginal likelihood of the ground truth answer a" (i.e,, the evidence) given the generative process with chain-of-thought
max log To(a" [z).
(2)
Directly optimizing the log likelihood is not tractable because its gradient cannot be estimated via samples in an unbiased way (see, e.g., discussion on this in the probabilistic inference literature [17]).
As the main contribution of this work, we propose a tractable lower bound objective by directly applying the Jensen inequality lower bound the log likelihood
log mo (a" (r)) log E (-vollll) (To [xr,c) 2 Bewxo(-a)) (log (o [x, c)),
Co(o,a*)
where we exchange the order of the concave log function and expectation E [-]. There are conditions under which the lower bound Co(æ, a²) is tight. For example, if all chain of thoughts с in the support of no( (-)) induce the same probability of predicting the ground truth answer To(a" 1x c), i.e., (a" [x, c) Fo(a"lz, c'). Vc, c' supp (=+()|2)). In practice when the optimization is approx- result, there might be a gap between the lower
imate, such conditions are not likely hold. bound and log (x)) and we will examine its empirical impact in practice.
The gap between the marginal log likelihood and the lower bound can be expressed as the KL divergence between and the posterior distribution [17]
log (a "0" x)) Lo(o, = KL p")()))
where p** (c|2,, a²) = ma(a" (c.c)
The posterior defines a distribution over chain-of- thought, and effectively denotes how likely is the chain-of-thought given that the ground truth answer is a = a and the prompt is æ. For experienced readers, this lower bound is closely related to the evidence lower bound [28, 17], which will elaborate more below.
E. ".,')a
3.1 Stochastic gradient estimate
The lower bound permits stochastic gradient estimates. Concretely, given samples from the current policy e ~ Te(-(r), we can construct an estimate of Ce a²) as
log (a xr, c) Va log wa (clær - Va log mo c).
The gradient has two terms: 91 is a REINFORCE gradient estimate with log mo (a" (a., as the reward function for sampled chain-of-thouge [29]. The second gradient ø2 is reminiscent of supervised learning loss that encourages the model predict ground truth answer a" given sampled chain-of-thought c.
In practice, we can add control variate to the REINFORCE gradient estimate reduce variance. One option is to learn prompt-answer dependent function [30]; another sample-based alternative is to generate n i.i.d. chain-of-thoughts in parallel o; To (-()), and construct leave-one-out control variates n Ejzs log To(a (y) 32]. The overall gradient estimate is the average over
samples:
(log (a vx) Vo Polog logmo(G(z))
+
Vo log (oggo(a"
(5)
n
n2
1=1
Note the control variates uy do not introduce any bias to the gradient estimate since they are statistically independent from Va log Te(c(lz).
Connections to supervised fine-tuning In the very special case where there is no chain-of ofin--of-hhee oh-thought, the gradient estimate reduces to just SFT part Va log na (a"|x)) which effectively the supervised fine-tuning loss from prompt to answer a". Here, the key difference is that the loss To(a" [rc, ci) further conditions on the -hain-of-thoughts e,s whose distribution changes over time and introduce more non- stationarity to the optimization process.
3

---

Tightening the objective via multi-sample Jensen's lower bound
If the Jensen's lower bound is loose, will induce sizable discrepancy from the true objective of interest. We need strategies to tighten the lower bound for policy optimization.
similarly simple yet a tighter lower bound alternative, is an extension the multi-sample case [21]. Indeed, consider the n-sample Jensen's 's lower bound
C (x,a") = ((a0)=) log Σ To(a" lz, ci)
(6)
Importantly, the log function is outside of the n-sample average to tighten the bound. It is straightfor- ward to verify that 2/" (x, a²) recovers the Jensen's 's lower bound as defined before in Equ (3). As
shown in [21], the lower bound becomes tighter n increases (/²² (ar,a*) a²) C. c(x+1) (x, a²) for any
2 O. As - oc, the bound approaches the marginal likelihood ((n) (xr, a") + log (a" ()), the ultimate objective of interest, under certain regularity conditions on Tg-
To maximize the multi-sample lower bound c)²) (ax a²) with gradient ascent, we can construct the REINFORCE stochastic gradient estimate as follows,
log
Vg log Vg log n To (a [s, (1,
Σ v (a(a" (z,
n
9 (n)
Empirically, the first term gi tends to have high variance as n increases [33], since the objective log 1 n Σ== To [az e,) correlates updates to all n samples. Akin to before, we can introduce the leave-one-out control variate without incurring any bias for variance reduction [31, 32]
(n)
n
log VL
To To(a" (2, (i)
0
Vo (og
where ū; = log n-T En To(a" "0"" C,). Note that the second term 92, though can be estimated via random samples, is unlike a regular SFT loss since is the log average of multiple probabilities, in- stead of the average of log probabilities. As n co, since log Σ no(a") [ac, c,) - log xo (a*(2;)
we see that at least conceptually 32 can be understood as directly maximizing the marginal likeli- hood, where the average over probabilities effectively marginalize the chain-of-thought conditional distribution. As we will show in Section 6, multi-sample lower bound generally improves the single sample Jensen's lower bound. This means that tightened lower bound improve training objectives both in theory and in practice Due to space limit, we detail important technical details of the practical implementation Appendix
Connections to related algorithms and prior work
The JEPO algorithms bear close connections to number of algorithmic alternatives, which we discuss in Appendix D due to space limit. See Algorithm for the pseudocode of the full algorithm.
Training with unverifiable data natural way to generalize RL training to unverifiable data to make use of LLM feedback, e.g., L..MMax-judge uses LLM to assess the quality of the generated response [34, 35, 36]. However, despite its conceptual simplicity. MLM-as-uudge might not produce reliable assessment for domain-specine
long-form data [10, 13]. When optimizing against judge scores, it is also more likely to over-optimize [37]. As a result, this work we apply LLM LLM-as-judee ju-jee only for short-form evaluations and not for training.
Closely related to our work the concurrent VR-CLI (verifiable reward with completion likelihood improvement) [38] where they apply log probs of golden generations as reward. Using our terminol- ogy, their approach resembles the first part the gradient Eqn (5) of the Jensen's 's evidence lower bound. Without a SFT-like component, their update does not optimize for the marginal likelihood only partially. JEPO also applies the multi-sample technique to tighten the lower bound, achieving better empirical performance, which we will demonstrate in Section 6.

---

0.54
training set accuracy
test sestet accuracy
0.52
training set reward
0.775
0.725
0.50
0.710
0.725
0.48
0.700
0:46
0675
0.650
0.10
Iennents bound
0.025
Oeitre
0625
Hillossemppe jarnants know bound
-0.6
-0.6
-02
(L. Tead
(L, Rear)
Average per token lower bound
Figure 1: Short-form answer experiments with MATH. We compare three baselines: online RI with access to the oracle Sympy-hased hased reward and JEPO. In the left plot, we monitor the reward the training dataset. Online RL obtains the best training time trade-off. followed by salli- lower bound and the single-sample lower bound: In the middle plot, we monitor the evaluation on a test set during training. Multi-sample lower bound and online RL obtains similar performance: In the right plot, we graph training reward against the lower bound objectives, averaged over training tokens. The two signals bear positive correlations overall and multi-sample lower bound yields better correlations.
Likelihood- scoring Prior work showcased the utility of Likelihood- scoring in filtering of chain-of of- ofothhught [39,40] 40J. The algorithms mostly proceed in an iterative fashion akin to expectation- maximization [41], which theory can also maximize the evidence of the desirable final answers. Complementary to such work, since we extend the training process to fully online RI. settings, we forgo the need of variational posteriors which allows for training on unverifiable data scale. To understand the limitations, compare reward-free JEPO and RL with verifiable reward in Appendix D.4 where we highlight a genuine trade-off.
Chain-of- thooght as latent variable modeling The idea of casting optimizing chain of of-thuught as latent variable modeling is not new. Previously, [14] proposed an algorithm motivated by maximizing ELBBO to tackle reasoning problems. Such algorithm also draws close connections to prior work [42, 43, 44, 45] all of which resemble hybrid offline-online RI. training loop, where they alternate between sampling and filtering via reward. They also have interpretation as EM algorithmic variants [41].
Despite the appeal of a full ELBO formulation, it is rarely implemented in practice due to the requirement of learning the posterior distribution. Indeed, despite the formulation of [14] they ended up approximating the posterior with MCMC, which effectively made use of an explicit reward to filter samples. This also marks key difference from our work we do not apply any explicit reward scoring throughout our algorithmic design and practical implementation. In addition, [15] has proposed a more systemic hierarchical latent variable modeling view of shain-of-though Similar to our motive, [46] optimized an ELBO inspired objective for prompt selection, where they did not resort to an external reward. We discuss additional connections beteen JEPO and full ELBO approach to chain-of-thouee tf-thougttt Appendix D.I and Appendix D.3.
Evidence lower bound and RL The connections between evidence lower bound and RL has been extensively studied in both the variational inference [47, 17] and RL community [48, 49]. In the RL literature, much of the variational inference view has been used to better interpret and improve existing algorithms with much focus on the goal-conditional problems, where single reward assigned at the end of a trajectory. Such setting is quite akin to the RL.FF case, where a sequence terminates with a single reward [50, 51, 52]. Our formulation also naturally incorporates the tighter multi-sample lower bound [21, 33] as special cases, which has seen little adoption in prior RL literature. In Appendix D.2, we discuss how JEPO relates to a variance reduced policy gradient algorithm akin to RLOO, defined follows
Vo log To(c)) (mo(a"les) wx) VoTo(a"les),
n
where W. - Ejei Te Te(a"le;) is the leave-one-out baseline akin similar constructs in JEPO.
5

---

log10 test set ground truth NLL
Ordine Rl syros score Mazi-ssmple soKey acore
0.00
training set set reward
0.2
0.0
Multi-sample bound n sample
Multi sample
Multisample bound n=1
-0.4
0.60
10
12
14
(1, Hret)
Epochs
(a) Ablation with number of samples n for JEPO
(h) Ablation with mumina test set NI.I
Figure 2: Figure (a): Ablation number samples n for multi-sample lower bounds. As we increase the number of samples, the multi-sample lower bound seems to further improve the training-time efficiency. This corroborates the theoretical insight that as n increases, the multi-sample lower bound objectives become tighter. Figure (b): Test proxy NLL evaluation for training the numina dataset. We evaluate the proxy NLL of the trained models the numina set, approximated with n = samples lower bound defined Eqn (8). Both JEPO and the combined algorithm sees improvement the NLL (lower the better), while online RI. does not improve on test set NLL. This hints at different solutions found by the online RL and JEPO algorithms, despite similar improvement trend in the sampling based evaluations.
6 Experiments with verifiable data
We start by comparing JEPO against RI baselines on verifiable data. We focus on the mathematical reasoning dataset MATH [11] where the prompt asks the model mathematical question with short form answer a". We mainly focus on the two algorithmic variants proposed in this work: the JEPO defined through the gradient estimate in Eqn (4) as well its multi-sample variant Eqn (7). As a strong baseline, we consider the online policy gradient RL algorithm which applies Sympy [53] to automatically match the answers. The RL algorithm applies leave-one-out for variance reduction, as is commonly practiced [54, 2]. Our main experiments are based on the 8B and 70B model from the Llama 3 model family [55]. All algorithmic variants apply identical aaper-parameters which we detail in Appendix B.
We highlight again that the RL baseline is at advantage in this setting, since the reward is fairly accurate and is itself being used as evaluation signals too [56]. We do not compare with other baselines developed in prior work (e.g., g.. [14]) as they can be interpreted as variants of online RL algorithms with certain low-level implementation differences.
6.1 Comparion on MATH
when there is an answer match and r = 0 otherwise. Note that JEPO does not require access to such reward, but we monitor the reward scores during training. Figure left plot shows the training performance all basclines. For the x-axis, we use the KL divergence KL(TTe, Teet) caleulated over the training set. Following the practice in [37], adopt the KL divergence as a certain measure of the optimization budget that the algorithm has consumed. Note that here all experiments are run with the same regularization coefficient / 10 since it achieves a good trade-off for all algorithmic variants over all.
During RL training, we use a reward
Training performance Figure left plot shows that online RL achieves good KI-performance trade-off on the training set. This is probably not big surprise since online RI. optimizes for the very same objective that we monitor here. In the meantime, JEPO enjoys reasonable performance: as the policy deviates from the reference policy, the reward performance improves despite not explicitly training for it (in theory JEPO optimizes for a hard string match rather than Sympy match). (2) the multi-sample JEPO obtains noticeably better performance than the one-sample lower bound baseline, despite using the same n generations per update.
Evaluation Figure 1 middle plot shows the evaluation performance on an held-out test set. We note that the reward on the training set higher than the test set, because the model has been SFT'ed on on the training prompts. For evaluation, observe that the multi-sample lower bound method obtains

---

Test unveriliable set accuracy
0373
Test set combined accuracy
0.48
0.22
0.46
6.M0
0.00
6.42
Sympy
0.14
0.47
0.49
04455
1:30
0.10
Val campla Bound
Hull sandle lcvar board
Epochs
Epochs
Epochs
Figure 3: Evaluation comparison of training 70B models semi-verifiable Numina dataset. We show evaluation results during the course of training. Left plot shows combined accuracy on the unverifiable subset (about 40%) of the test set; middle plot shows the combined accuracy on the full test set; right plot shows the Sympy score on the full test set. While JEPO progresses more slowly on the Sympy scores compared to online RL, it gains the combined accuracy; the combined algorithm seems achieve the best of both worlds.
similar performance as online RL, despite being outperformed during training. We conjecture that this is because online RL tends to overfit the training prompts more significantly, producing high training reward that does not transfer as well to the evaluation time. This shows that even without training on the reward signal explicitly, JEPO can obtain similar evaluation performance RI..
Statistical correlation between objectives Figure right plot graphs the training time reward against the lower bound objectives. If we consider the training reward as ground truth objective to optimize for, we see that the multi-sample lower bound displays stronger correlations between the surrogate objective and the ground truth. Due to space limit, we show a comprehensive set of ablations in Appendix C.I.
Experiments with semi-verifiale data
We now consider semi-verifialle data where a good proportion of the dataset contains answers which are not easily verifiable. We focus a post-processed Numina dataset [57] where prompts are mathematical questions and ground truth answers are partly verifiable. For instance, one example of the ground truth answer is the whole expression: Vr E R, x2 + (a 1)x + 1 2 0. Given model generated answer, it is hard to verify whether it is equivalent to the above expression without case-specific parsing. See Appendix B for details on how we post-process the dataset.
RL baseline and reward For the RL baseline. apply the Sympy reward as introduced in the previous section. Because the dataset contains answers which are hard to verify, the reward is effectively only applicable to a subset of the data. The default training examples. We estimated at least 40% of such examples cannot be verified by the automatic scorer. We consider online RL with such reward as a baseline, as it has access to highly specialized verifiable reward but only applicable to subset of the data.
contains about 22k
Combining JEPO and RL baseline We also compare with an algorithm that combines the loss function of JEPO and RL baseline with the Sympy reward. When sample n generations from single prompt, and if none of the generation obtains positive score (note this does not mean that the example is necessarily unverifiable), apply the JEPO loss; otherwise, apply the baseline RL loss. This allows for a dynamic combination of two losses, and still leverages the whole dataset.
Evaluation We have an held-out Numina test set consisting of 640 examples. The test set contains both verifiable and unverifiable examples, which we evaluate in two ways: (1) Sympy reward I "sympy(e (a, a²) {0. 1), which generally underestimates the true accuracy when ground truth is semi-verifiable; (2) Sympy combined with LLueees-judge Trombined (a, a a²), which combines two (a, a²) + (u, a (I (reams (a.aa) )=0-- The LLM-as-judge
sources of scores (combiined(a, a") ;= r Tsympy
score Vilm (a, a²) also binary: it is based on a 5-time majority voted decision of a prompted 70B instruction-tunee model [55]. Though imperfect, we observe that LLM-as-judge reasonably mitigates some false negatives caused by rigid Sympy scoring. Importantly, we reiterate that we not train on such combined scores they are used for evaluations only.
7

---

7.1 Comparison on Numina
Unless otherwise stated, we will experiment with the multi-sample algorithm given its performance gains in Section 6. Below, Figure shows the evaluation performance comparing the RL baseline, JEPO and their combined algorithm. Since the Numina dataset is more challenging, we experiment throughout with 70B models.
Sympy scoring evaluation Figure 3 right plot shows the evaluation accuracy using the Sympy score. Overall, all algorithms make steady progress as the training progresses. However, since online RL baseline trains with the same reward signal, it achieves slight acceleration compared to JEPO. The combined algorithm achieves a similar rate of progress with the Sympy scores on the test set.
Due to the abundance of symbolic expressions as ground truth in the Numina dataset, here the Sympy reward a much more specialized scoring method than e.g., string match compared to the MATH case. This partly explains why the online RL bascline is competitive, it trains the same signal.
Combined scoring evaluation Figure 3 left plot and middle plot shows the combined accuracy which alleviates some false negatives due the Sympy scoring, As seen from the overall metries, the accuracy increases by about 25% compared to the Sympy scores. The left plot shows the performance on the unverifiable test subset (40% of the test set) while the middle plot shows the full set. We observe that both JEPO and the combined algorithm achieves faster rate of progress and asymptotes to slightly better performance than the online RL baseline with this combined metric, especially on the unverifiable subset. Interestingly, note that by training on verifiable rewards, online RL can also make progress on the unverifiable test set.
Though the Sympy scoring is quite specialized, it is only applicable to subset of the full Numina training set. Meanwhile, JEPO can leverage the full dataset, despite with less specialized signals. The combined algorithm seems to achieve the best of both worlds.
7.2 Ablation study
We carry out additional ablations to better understand the performance difference.
Assessment with set negative likelihood: lower the better We further evaluate the proxy negative log likelihood (NLL) that the trained model produces on test set, computed via the n-sample lower bound [21, 40]
n
log To (a Jx, .i) ot) ]
(LLX)-NLL(TD) = -E(e) (((e)))))
log
12 Σ
Figure 2(b) shows such proxy NLL during training, where we see different pattern for the online RL baseline and JEPO. For JEPO, the proxy NLL decreases over time. We expect such result because JEPO optimizes for the same objective on the training set, and before overfitting, we expect improvement on the test set. Meanwhile, maybe surprisingly, online RL does not make progress on the test set NL... The combined algorithm is between the two extremes.
There maybe good reasons for online RL not to make progress test set NLL. Particularly, for each ground truth answer in the dataset a", the Sympy scorer defines a sizable collection of correct answers A {a T'smmyy (a, a²) 1} whose aggregate probability (AArr) increases under online RL (evidenced by test set accuracy improvement in Figure 3 right plot). In other words, online RL might not improve the proxy NIL of a particular a" inside A. The above observation means that the policy found by online RI. and JEPO can produce different answers to the same question. This is related to the model calibration issue for RL post-training in general [20].
Comparison with SET baseline golden chain-of-thought To assess another option to improve semi- verifiable performance, we carry out another comparison against a SFT baseline, which trains on the golden chain-of of-in-of-thought dataset [57]. We observe performance improvements across evaluation metrics as well, though generally underperforming RI. See Appendix B for full results.
8 Experiments with unverifiable data
Finally, we experiment with unverifiable data, where the full dataset has long-form ground truth and cannot be easily verified with hard-coded programs. We consider post-processed Numina-proof, extracted from the original Numina dataset where all ground truth answers are full proof. The proof
8

---

381 LORE
0.06
SFT (100)
-0.12
-0.08
log10 test set NLL
log10 test set NLL
agerrodioo
-0.34
-0.32
-0.16
0.34
-0.18
-0.38
0.30
-0.22
-0222
-0.22
10
200
Epochs
Epochs
(a) Test set NLL on numina numina-proof
(b) Ablation with algorithmic components
Figure 4: Figure (a): Test set proxy NLL evaluation for training on the unverifiable Numina-prows datasct. For JEPO outperforms the SFT baseline at the same data budget (measured in epochs) and achieves asymptotically better test performance. Figure (b): Comparison of different baselines various algorithmic variants, with the 70B model. We observe that the supervised component the JEPO loss plays a key role at learning efficiency and achieving good asymptotic performance.
numina-prool test NLL, across
often contains multiple sentences of paragraphs, without final short-form answer as in MATH the verifiable subset of Numina.
Baselines and evaluation Since the ground truth is long-form and cannot be verified easily, we do not have RL baseline with verifiable reward. Instead, SFT on the raw dataset (æ,, a²) is reasonable baseline. Through a few ablations, we also compare with methods akin to VR-CLI [38]. which corresponds the REINFORCE part of the single-sample lower bound gradient in Eqn (5). We evaluate NLL on the test set, akin to the ablations in Section 7. We do not carry out sampling based evaluations as long-form answers are hard to assess for frontier models [13].
8.1 Comparison on Numina-proof
As main experiments, compare JEPO with SFT. Note that always started with instruction-tuned models [55] and the SFT baseline can be understood as a continued SFT. We show the curve after an initial transient phase where the test set NLL drops significantly for all runs, which can be attributed to that the modes learn to format answer correctly. Figure 4(a) shows the test set NLL comparison between SFT and JEPO, with both 8B and 70B models. At both scales, JEPO outperforms SFT with test set NL.. at the same training data epoch. Also, JEPO is asymptotically better than SFT.
8.2 Ablation study on the importance of JEPO supervised loss
To better understand the role of different algorithmic components, we compare with additioanl baselines: recall that JEPO update contains two parts: REINFORCE component, whose single- sample variant is akin to VR-CLI [38]; and a supervised loss component. We compare with a variant where the supervised loss is down-weighted (s 0.01) and another where it removed (Saup 0).
Figure 4(b) shows the comparison on the test set NLL. We see that by downweighting the supervised loss, JEPO makes much less progress the test NLL given the same training epochs. Specifically, when the supervised loss removed (8aup 0), test NLL also seems to plateau at worse level. Interestingly, this contrasts the observation in MATH experiments (Section 6) where small values of Saup works better. We speculate that the key difference is that nature of the chain-of-thouse for MATH. the chain-of chain-of-thought details solution steps and final answer be readily inferred. For long-form data, the chain-of-thought is high-level outline, and it still takes extra effort to produce the full answer (e.g., proof), hence the importance of the supervised loss.
differs:
Limitations and future work
Possible directions for future research include studying the impact that various loss components (e.g., the REINFORCE and the supervised loss) have on overfitting; more organic ways to combine verifiable rewards and JEPO losses; ways to scale such methods, more general purpose data (e.g.. in the form of meta-thought [1, 58] or to pre-training [40].

---

References
[1] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
[2] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint urXiv:2402.03360 2024.
[3] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T, ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:241 15124, 2024.
[4] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
[5] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12555 2025.
[6] Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829, 2025.
[7] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by ratio- nale generation: Learning to solve and explain algebraic word problems. arXiv preprint 1rXiv:1705.041 2017.
[8] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thougl! prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[10] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.200055 2023.
[11] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.038 033744 2021.
[12] Art of Problem Solving (AoPS). Aime problem set 1983-2024, 1983.
[13] Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislay Balunović, Nikola Jovanović, and Martin Vechev. Proof or bluff? evaluating Ilms 2025 usa math olympiad. arXiv preprint arXiv:2503.211 /934,, 2025.
[14] Matthew Douglas Hoffman, Du Phan, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous. Training oh thought via variable inference. Advances in Neural Information Processing Systems, 36, 2024.
[15] Xinyang Hu, Fengzhuo Zhang, Siyu Chen, and Zhuoran Yang. Unveiling the statistical founda- tions of chain-of-thouge prompting methods. arXiv preprint 2.Xi::2408.14111 2024.
[16] David M Blei and Michael I Jordan. Variational inference for dirichlet process mixtures. 2006.
[17] David M Blei, Alp Kucukelbir, and Jon McAuliffe. Variational inference: review for statisticians. Journal of the American statistical Association,
2017.
10

---

[18] Johan Ludwig William Valdemar Jensen. Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acta mathematica, 30(1):175-193, 1906.
[19] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020.
[20] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087774 2023.
[21] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint 150,00019,, 2015.
[22] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[23] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 25-27730-27744, 2022.
[24] Daniel M Ziegler, Nisan Stiennon, Jeffiry Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 099999,, 2019.
[25] Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code lims in execution feedback with reinforcement learning. arXiv preprint arXiv:2410.02089, 2024.
[26] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint vrXiv:2211.14271 2022.
[27] Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing I1m reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449 2025.
[28] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[29] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-29) 1933.
[30] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[31] Andriy Mnih and Danilo Rezende. Variational inference for monte carlo objectives. In International Conference on Machine Learning. pages 2196. PMLR, 2016.
[32] Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get baseline for free! 2019.
[33] Tom Rainforth, Adam Kosiorek, Tuan Anh Le, Chris Maddison, Maximilian Igl, Frank Wood, and Yee Whye Teh. Tighter variational bounds are not necessarily better. In International Conference Machine Learning, pages 4277-4285. PMLR, 2018.
[34] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling rein- forcement learning from human feedback with feedback. arXiv preprint 230:2230900026: 2023.
11

---

[35] Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024.
[36] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2025.
[37] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[38] Alexander Gurung and Mirella Lapata. Learning reason for long-form story generation. arXiv preprint arXiv:2503.22822 228288, 2025,
[39] Eric 7elikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024.
[40] Yangjun Ruan, Neil Band, Chris J Maddison, and Tatsunori Hashimoto. Reasoning to learn from latent thoughts. arXiv preprint arXiv:2503.18866 2025.
[41] Todd K Moon. The expectation-maxxiizatioo algorithm. IEEE Signal processing magazine, 13(6))47-60, 1996.
[42] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488 2022.
[43] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling, arXiv preprint arXiv:2308. 00892388 2023.
[44] Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint 23 12.06885, 2023.
[45] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2.08..0 2023.
[46] Alessandro Sordoni, Eric Yuan, Mare-Alexande Côté, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. Joint prompt optimization of stacked I1ms using variational inference. Advances in Neural Information Processing Systems, 36, 2024.
[47] Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In Interna- lional conference on machine learning, pages 324-333. PMLR, 2016
[48] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00900 2018.
[49] Brendan O'Donoghue, Ian Osband, and Catalin Ionescu. Making sense of reinforcement learning and probabilistic inference. arXiv preprint arXiv:2001.00805, 2020.
[50] Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[51] Ben Eysenbach. Xinyang Geng, Sergey Levine, and Russ R Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. Advances in neural information processing systems, 33: 4783-144795 2020.
[52] Yunhao Tang and Alp Kucukelbir. Hindsight expectation maximization for gonl-condi reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 2863-2871. PMLR, 2021.
12

---

[53] Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, Amit Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rath- nayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Sco- patz. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, January 2017.
[54] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 12248-12267, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[55] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint ar arXiv:2407.21783, 2024.
[56] Albert S Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse, and Aaditya K Singh. Harp: A challenging human-anmann annotated math reasoning benchmark. arXiv preprint arXiv:2412.08819, 2024.
[57] Jia Li, Edward Beeching. Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository. 2024.
[58] Violet Xiang, Charlic Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system 2 reasoning in lmms: Learning how to think with meta chain-of-though. arXiv preprint arXiv:2501.0444 2025.
[59] Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1, 2016.
[60] Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https:/ //tg///ggggggaeeeeeeeooeeo co Coo/AI-M0//MmmnaaMtttiee 1. 5] (https://githon com/ wwwpagiingaae..o/AAA-.....
project- numina/ann aimo- pwwenooproress---e progress-prize//bl... http://////nni/eer/r/////a/aa/// www.seteetaaaaaaeettaaaaee.... 2024.
[61] Nathan Grinsztajn, Yannis Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Olivier Pietquin, et al. Averaging lig-likelihoods in direct alignment. arXiv preprint arXiv:2406. 19188, 2024.
Mohammad Gheshlaghi Azar, Florian Strub, Bill Wu,
[62] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Openai baselines. https://github. com/ http://enaaaaaeeeeeeeeeee
2017.
[63] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017.
[64] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783 2025.
[65] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
13

---

6ef
(a) Probabilistic inference
(b) CoT with full ELBO
(c) CoT with Jensen's s hound (d) CoT with Jensen's hound with KL regularization
Figure 5: Graphical models for various algorithmic formulations discussed in this work. Solid lines represent generative models and dashed lines represent inference models. Circles represent random variables and squares represent parameters. Shading indicates that the random variable is observed, and is used for providing feedback simplified notation for the binary optimality variable
for learning process. For CoT optimization, (a=a*) from the random variable a.
A Review of the graphical model perspective
We make a more extended discussion about the graphical model shown in Figure 5.
Probabilistic inference with learnable prior Figure 5(a) shows the generic structure for proba- bilistic inference with learnable prior, with latent variable 2 and observable Here, A controls both the prior and observation generation:
po() po(-Ic).
The inference parameter & denotes a the posterior inference distribution (z(o)) that seeks to approx- (c))p((ale)
imate the true posterior pa(zlo) Ee po(c') (olee) Together, they form ELBO that lower bounds the marginal log likelihood [17]
loggo(=1o) + log go (o(2|o)
log (o)) 2 (x
(=(=)
(o)
The right hand side (o) can be optimized via stochastic gradient descent on the joint variable (0, (). 'he lower bound is tight when the inference distribution is exactly the posterior qa (=(2))) po(2)o). A learnable prior refers to the fact that the prior distribution over latent Po(2) depends on too, while in much of the prior literature is is kept constant [59, 17J. For the transition from generic probabilistic inference to our use case, learnable prior is also fundamentally important.
Chain-of-the of-thoghhtt with full ELBO Figure 5(b) shows direct mapping of the probabilistic infer- ence structure to the case of optimizing chain-of-thought. Here, the chain-of-thought e the latent variable and the ground truth answer is the observable. more precise mathematically definition would be to consider yet another binary optimality variable O = 1 (a=ar) } that determines whether the random variable answer a optimal. Here, we directly replace it with a" for notational simplicity.
we further introduce a general conditional dependency on the prompt x, we arrive at the lower bound defined in Eqn (3)
go(c)o, a*)
log Ta (a" (x:) 2 (eogo(--aao log To(a" (xx log
To
(o,o(1,
thahught with Jensen's lower bound In Figure 5(c), we replace the variational posterior Is by the prior distribution itself тy. discussed in the main paper, this looses the lower bound but make the optimization objective much simpler. See detailed derivations in Section 3. We see there there appears to be duplicated arrow that goes from such duplication to distinguish between the inference distribution (dashed arrow) and the generative distribution (solid arrow); in this particular case, we deliberately make the two distributions identical.
the latent variable c. We make
Jensen's 's lower bound with regularization Finally, Figure 5(d) presents the graphical model for case where a KL regularization is added Jensen's 's lower bound (see Lemma D.6 for formal statements). In this case, the generative prior distribution is computed from the reference policy Tre parameterized by l which kept fixed during training, while the rest of the distributions are still parameterized by 0.
14

---

Algorithm JEPO: chain-of-thought optimization with Jensen's 's lower bound (or its multi-sample
extension)
1: INPUT policy R
2: while f = 0, 1, 2... do
3: (i) For each sampled prompt 2, collect n generations (ye) and extract their corresponding
chain-of-thoughtss (c)] TT(-(z)).
4: (ii) Evaluate TT(a" [r.)) with a forward pass;
calculate gradients
Ve log ne(c)) Vo (a xz, c.) with backprop.
5: (iii) Update with n-sample average of gradient estimate Eqn (4) or its multi-sample variant
Eqn (7)-
6: end while
B Hyper-parameters and experimental settings
We experimented with the Llama 3 model of size 8B and 70B. All experiments are conducted with identical
settings: we always apply batch size of B = 64 prompts per update, and sample n distinct generations per prompt. All training and evaluation sampling are conducted at temperature of r and with top-p-
We train the MATH training set with 7500 examples and evaluate the test set with 2500 examples. A supervised fine-tuning on the training set is conducted to warm up the RL training, hence the gap between training and test set accuracy.
For both training and evaluation, we provide system instructions that ask the model to generate a response with step-by-step solution, followed by final conclusion phrased as the final answer is followed by the answer predicted by the model. This consistent with the prompt structure discussed for Llama models [55,66].
All experiments are conducted with an entropy regularization coefficient > ablated in the main paper.
which we have
B.1 Dataset post-processing
We use unverifiable proofs data from Numina 1.5 [60] for our experiments. We clean and filter the questions and their corresponding solutions using some simple regex heuristies. For example, we replace leading blanks, markdown headings like ##,, prefixes like "Problem:" and "Solution", letter-digit combinations like "A1" "G5"/ "ROU", and trailing dots and blanks. After cleaning, we have 58088 proofs from the Numina dataset.
B.2 Important technical details of JEPO implementations
We detail the implementation details of the JELB-RL algorithm. We highlight a few key technical elements in the implementation, which we have found be important in getting the best performance.
Formatting penalty We find it useful to have an additional RL loss with the reward as Treg (I. y)
-p y does not follow the formatting requirement (that the identifier phrase the final answer in y) and zero otherwise. We find that this generally helps stabilize the training process. This is especially useful for small models (8B) where under temperature sampling, can often not follow instructions strictly. For large models (70B), we also found that its formatting might be inconsistent after multi-epoch training. We find a value p = 10 suffices while smaller values tend make the training less stable due to weaker penalties.
Per-sequence log probabilities During the log-ave-ery operation that defines the lower bound in Eqn (3), it is important apply the per-sequence log probs without average over the sequence length. Concretely, the bound is calculated as follows
1 n
log
M Σ j=1
To(aãlæ, cy. ozze)
71
[<||a-
where |a"| denotes the sequence length of the ground truth a". It is important not to average the sequence level log probs Ex<(ar ne To(a Cj, a²,) with a factor of 1/|a*| as suggested in other
contexts [61, 2], can modify the objective landscape significantly especially when |a"| is large.
15

---

janeen' Inser hrund =1=
0.80
training set accuracy
-5
-neens Inwer hound =
nsen's lower bound -1- 0.31
Jeline RI 18- D01)
0.70
0.03)
Jriine RL D.3)
0.65
log (L(TI, πref)
Figure 6: Ablation of regularization coefficient B. As & increases, all algorithmic variants obtain better efficiency the training performance- KL divergence trade-off. However. strong regularization also prevents policy from deviating much from the reference policy, preventing bigger training improvements.
Advantage normalization Both the baseline RL and the JEPO apply advantage post- post-processing. following common practice [30, 62]. For example, in the multi-sample JEPO, the advantage for the i-th generation
lc, ci) U.
A, = log
no
Y2
j=1
further normalization applied to the advantage A, = clip(A, //p/As//sdd (A, such that the outcome A, is applied in the actual update. Advantage normalization is especially important for JEPO because its raw advantage takes wider range of numerical values.
Weighted supervised learning loss We also introduce weighting coefficient for the supervised loss Soup useful for ablations. We find that small values 0 ~ 10-2 tends to work for short-answer applications (e.g., MATH) while a large value 1 important for semi long-fore forg-form data (e.g., numina and numina-proof).
KL-regularization In our early investigation, we found it useful to have a KL regularization at very small coefficient 3 10-3 The regularization helps prevent formatting collapse, and also prevents the policy from drifting too much in case the updates are noisy [24, 23].
Put together, given n samples, the JEPO update
(A. A!r0)) V log log + Brup V log
Ta (xx (o,, - VV&KLL (Te(-1æ), )er((-())
71
where Are is the normalized advantage for the formatting penalty. The normalization makes it such that the ultimate update optimizes for lower bound more akin to the weighted lower bound [63] though the underlying algorithmic motivations differ.
The lower bound loss is applied only generations with correct format, otherwise, the loss is masked out. Also, we find that the sequence level normalization with a factor of 1/ (lcg + (a*l) or 1/|c] does not make significant difference [2, 64].
C Additional ablations
We now provide ablation results on a few important dimensions of the algorithm.
C.1 Ablations for verifiable data
We detail additional ablations for the verifiable data.
16

---

0.70
test set set accuracy
0.65
0.60
Online RL (88)
Multisemple Jensen's bound (9B)
Online RL (70B)
0.55
semtle Jensen's bound (700)
0.50
log KL(TT, (TT, πref)
Figure 7: Ablation of model size (SB vs. 70B). We find that the multi-sample JEPO is fairly competitive against the online RL algorithm the 70B scale. Both algorithu traces out a similar KL-performance trade-off, with salple JEPO obtaining a slightly better performance given a similar compute budget as online RL.
Multi-sample ablation on sample size n We ablate on the number of sample n used for con- structing per gradient update. In theory, increases, the multi-sample lower bound becomes tighter and asymptotically approaches marginal likelihood objective (which is equivalent to RL objective). We vary the sample size 21, 4, 8) and compare the performance. Figure 2(a) shows that as n increases, the algorithm becomes more KL-efficient: with a fixed budget on KL, the model obtains better performance. Intriguingly, reward over-optimization [37]- as the optimization progresses, the training reward drops slightly (for blue curve). We can interpret this the result of the fact that JEPO does not optimize for the same indicator matching function as the reward we monitor.
also observe training performance akin to
Regularization ablation We investigate the impact of the regularization coefficient B E (0, 10-3 10-2 10-1). Figure 6 shows the training performance of the single-sample lower bound vs. online RL. One observation is that as 3 increases, the trade-off efficiency for both algorithms improves however, in general the algorithm also makes less deviation from the reference policy, hence leading to less improvement for a fixed training steps.
Scaling up model size against the online RL in the 70B case. Figure 7 shows that the JEPO obtains competitive performance against online RL in terms of the KL-performance trade-off. With roughly the same amount of compute budget, we find that the JEPO seems drift further from the reference policy, hence extending the trade-off curve to a performance of 70% test set accuracy, which outperforms online RL modestly.
Since the multi-sample JEPO appears more competitive, we compare
Supervised loss We find that low value of Sup generally works better for the JEPO algorithms. The speculation is that when Suu is large, the supervised loss encourages the policy to place weights on the ground truth despite that the chain-of-thougeee has low quality. This leads to overfitting the training set, in more severe way than online RL.
C.2 Ablations on semi-verifiable data
Comparison with SFT baseline on golden Bhein-of-thought See ablation results in Figure A few observations are in order: (1) SFT generally is not as good as the RL jobs, but it improves over time we train more; (2) There is initial drop in performance, which can be explained by the fact that the golden chain-of-thouu the starting model has been post-tran post-trained with [55]. Through SFT, the model needs unlearn the step-by-step format and learns the more freeform hybrid format in the golden bfain-of-thoughe data; (3) Asymptotically, SFT performs lower than RL runs.
does not conform to the familiar "step-by-step" that
D JEPO's connections with alternative algorithmic variants
We discuss detailed connections between JEPO and alternative algorithmic formulations below.
17

---

lower bounds the marginal log likelihood Co Ca,o(z,a") log To (a" (2) and it tight if and only if the inference distribution equals the posterior distribution go (c(2,, a²) = pdd (clor, a"). Since BLBOO is function of both policy parameter 0 and optimize both with stochastic gradient estimates: given Test unverifiable set accuracy
Test set combined accuuracy
Sympy accuracy
0.35
0.175
0.30
0.40
0.100
0.13
0.325
0.13
0.00
tyng soar senple hum lacan
Oatire
0.075
syrop
muman guane
625
Malining qit loaad
0,000
Epochs
Epochs
Epochs
Figure 8: Additional comparison against SFT baseline which trains golden chain-of-thought data from the numina dataset. We show that the SFT baseline also improves upon various metries, despite generally underperforming algorithms.
D.1 Evidence lower bound
The evidence lower bounds (ELBO) [16, 28, 21J control for the tightness of the lower bound with an inference distribution qa (o(c|ær, a²) which defines distribution over chain-of-thoughts. The lower bound takes the following form
La,alz,a") a²) Ee log (a(aaaaalz - log 2o(c|r,a") a') To(c)))
where the expectation under с ~ (.(-lx,a")
inference distribution parameter Φ, chain-of-thought sample 4o 4 a"),
90 v log To (a Ir, + Vo log Teg
96 V. loggs(clz, a²) log (x, log Is (0(c) a*)
no (e(c)))
V log (c(2,a)))"
Juxtaposing the form of the gradient here and the gradient to the Jensen's 's lower bound defined in Eqn (4), we observe that the inference distribution gradient Is bears resemblance to the REINFORCE gradient; while the policy distribution gradient ge bears resemblance to the SFT gradient. In fact, we can show that under the special parameterization go (c/æ, a*) == <a(clz), the two gradients are exactly equivalent. Such an observation stated formally below.
Lemma D.1. (Jensen's lower bound as a special case of ELBO) When (o(c),, == ne (c(c)), ELBO is equivalent to the Jensen's lower bound Le.s a²) La(x,a") a*) gradient esimtates are equivalent to the Jensen's 's lower bound's stochastic gradient estimates.
Proof. The proof follows from the fact that when qo Te. we have
Io V log <<(c)z) log To |z, c) Ve log Te(c)))
Adding this gradient to go. simple manipulation shows that the aggregate gradient is equivalent to the gradient of the lower bound defined in Eqn (4).
By introducing an inference distribution fs. ELBO is much expressive than the Jensen's 's lower bound and allows for tighter approximation to the marginal log likelihood. However, this also introduces additional complexity of having to learn the approximate posterior distribution. In our applications large size can be a major computational overhead. In practice, [14] approximates the posterior via few steps of MCMC and forgoes learning with such a distribution altogether. We take different approach with a similar motivation: by tightening the lower bound with multiple samples, also avoid the need for explicit posterior.
of interest, training posterior model
18

---

D.2 Reinforcement learning
We show that there is close connection between the lower bound formulation and the expected reward (return) maximization objective in RL [65]. Concretely, we will see how the lower bound objectives are closely related to a conditional expectation trick that produces a RI. policy gradient estimate with lower variance. First, we show that (up to log transform) RI. optimizes for the same target as the lower bound objectives, given the indicator reward.
Lemma D.2. (RL. optimality is equivalent to maximum likelihood optimality) When r(r, y) = {a=aa)> }. the optimal policy to the RI. objective is equivalent to the optimal policy of the maximum likelihood objective Eqn (2).
Proof. The conclusion follows from the fact that EE (a=a = mTa (a" (aa[==] (z). Hence the two objectives differ by a log operation and yield the same optimal solution.
Assuming access to n i.i.d. trajectories (y: )= ~ TT(-|x), we start with the classic RL policy gradient with leave-one-out baseline (i.e., RLOO [54])
1 a Σ n
Ve (o()))))))) (alla)-(((eea) (I(a.aa)) - w.).
Ivanilla-pg
where w/ = - E 11 (a,=a-) is the leave-one-out baseline. Now, we present a new policy gradient estimate of the RL objective with guaranteed variance reduction, which is also feasible to implement with sample-basee based learning.
Definition D.3 (A variance-reduced RI policy gradient estimate). Given n trajectories (y.) from single prompt we define rearredue
n
Vo logog ro(c)) (To(a")a) - w.) - Vamg(a"les),
(10)
Ejtt where W; = n jætt To (a" (es) is the leave-one-out baseline akin to similar constructs in the lower bound case.
We show that the seriance-reduced policy gradient estimate is closely related to the classic gradient estimate via the conditional expectation trick.
Lemma D.4. (Conditional expectation) Under the same assumption as Lemma D.2 and denoting ~ (-(-)c) as the sampling process ag TT(-Ic,), it holds that Ovar-reuued--gg is conditional expectation Ivanilla-pg
gvar-reduced-pg En (1l) Ivanillapp (c))
(11)
We note that without the leave-one-out baselines W;, W; the conclusion Eqn (11) is straightforward as both estimates Eqn (10) and Eqn (9) become plain averages of i.i.d. terms. Now, using Lemma D.4, we immediately see that the new gradient estimate yields smaller variance.
Theorem D.5. (Variance reduction) Under the same assumption as Lemma D.2, we have guaranteed variance reduction
(I))) (-1x)) 19var-reduccd-ppll S V (w)) [avanilla-pa]
(12)
The proof is provided in Appendix E. Putting Ivarreduced-pg from Eqn (10) and the gradient estimate of the Jensen's lower bound (Eqn (4)) bideeby-side, we identify intriguing similarities. Both gradient estimates employ two terms that update either the chain-of ohain-of-thought component To (-(ar) or the answer component n (=|2,,, c), with the only subtle difference being the extra log-transform needed for obtaining the Jensen lower bound. This alludes to the fact that the lower bound gradient has intrinsic built-in variance reduction. We provide additional discussions of few practical trade-offs in using the variance-reduced estimate Ivar-reducod-pg in Appendix E.
19

---

D.3 Optimizing Jensen's lower bound with regularization is optimizing special ELBO
When optimzing the lower bound objectives, we also apply the KL regularization motivated from the regularized RL formulation (Eqn (I)). Though this combination seems ad-hoc, we will see that optimizing such an hybrid objective is in fact equivalent to maximizing special ELBO.
Incorporating the regularization into the lower bound formulation, have aggregate objective
(a,, a²) (TLLL(TT, (oo Teef)-
(13)
If we refine the regularization little more: instead of the generation level regularization, we apply (a(a))) then the resulting
regularization at the chain-of-thought: KT ( == (=11a(-1 log Tur(cla)
aggregate objective can be interpreted in a more coherent way, as an ELBO to a concrete generative process.
Lemma D.6. (Regularized lower bound as an ELBO to a special generative process) Assume a generative process [- a (-la, c) that defines marginal distribution Pre. (alar == E. (c)(( () no (a" (x,c). Then the regularized objective Lo a²) - (TL,(TT Teat is a lower bound to the log likelihood log Pr (alsx).
Proof. Applying the same derivation the regular ELBO, log likelihood log Pao (a/x) is lower bounded as
max (o log To(a"((z, log log Tc(c
20(c|2r a*)
[(a) (o)T log (o (xx_,,,,)))
Teet (c/2z)
Lo a²) (To Tret,,
where inequality (a) is due to choosing 9s my and the last equality is by definition. Hence the proof is complete.
Note that the aggregate objective Eqn (13) can also be optimized via stochastic gradient ascent. We just need add an additional term associated with the KL regularization, the original gradient estimate to Lo (z, a²) defined in Equ (4). An example of such a gradient estimate usually takes the following form
To [c(c)))
log
V log To(c))), e~ -
Though our lower bound interpretation (Lemma D.6) is under regularization only on the chain-of- thought, in practice, we still apply the full generation level regularization following common practice [22, 24, 23].
D.4 Practical trade-offs compared to RL
As discussed earlier, JEPO does not require an external verifiable reward, as it can be understood adopting the indicator reward r(x, y) 1l [a J. In practice, this can be instantiated string match float (answer == answer..r However, such reward function will likely induce false negatives, as semantically equivalent generations might be vastly different strings. In practice, more lenient match is typically applied to better balance the false negative. For example, for math problems [11, 56]. usually programmtic checks are implemented to check for equivalence of two short expressions, such that e.g., pi and 3. 141459966 might be considered equivalent.
striet
More formally, consider a general reward function r(z, y) match(a, a²) calculated as binary match between a and a". We can rewrite the RL objective as Ej [mmatch(a, a")]. In order to adapt the formulation in this work to the lower bound case, we need work explicitly with the equiva- lent set AA == {a match(a, a²) = 1}. We will need to calculate quantities such as the probability To (o(A, c) == E.E To(a|x, c). which reduces to ne (a" [x. e) in case we use exact match. Com- puting such probabilities is expensive since we need to enumerate all a E A if inverting the match function feasible at all. As result, the lower bound formulation cannot be adapted generic match function or reward function.
20

---

In summary, when good (verifiable) reward is available (Sympy vs. string for certain math datasets, see Section 7), online RL is an advantage. There are also cases where good rewards are not easy to come by, and JEPO is a decent default strategy. An example is where the ground truth answer takes rather long form, in which case string match or programmatic check will produce too much false negatives, see Section 8.
E Variance-reduced RL gradient estimate
We provide more discussion on the variance-reducee RL gradient estimate.
E.1 Proof of variance reduction
Recall that denote (y:)== as the of generations and (c) be the set of Chain-of-thoughts generated from prompt z. We drop the dependency on prompt wherever the context is clear.
Proof of Theorem 12. A direct computation shows that
V ( @vanillappe =B( ~m @vanilla-pe Ivar-rducce--p Ivar-reduced-pg E (uo) [Avanilla-pe]]
(14)
(g. - (m)(-a))))) [llaaanila-p Ivar- [| ((o-l) (ovur-
where the cross-term vanishes due Eqn (11). From this, Eqn (12) follows immediately.
Proof of Lemma D.4. We begin by Ivanilla-pe
computing the conditional
expectation
Ba-xo((le)
(c))==) which yields
Bawxo(-le)
Vo log (y)) (a,==r) | (c)) + Bawzo(-ec
Vo log (v()))
7
n
(15)
i=1
where the notation To(-)c) to indicate that each answer a, mo(:le;) is ii.d. sampled from its corresponding chain- Shain-of--thught. Expanding the first term I, we have
1 Vo (((lc) + Volog [e=aa) To(alc))
(a) n
(16)
(omo((a + V log xo(()) ne (a(a" (c:))
(6)
where (a) is by definition of the expectation and a E A denotes dummy answer variable; (b) is due the definition of the indcator function. Now recalling the definition of w, as leave-one-out baseline to simplify term II:
72
Bawnol-le) Vo log (16) ws (c)
(awso((lo)
V log no(ys) (a,=ar) I (c) (17)
n
n(n
Note we can explicitly compute each summand on the right hand side of Eqn (17) as product of two conditional expectations, thanks to the fact that when + j:
Ea-to(-le) Ve log To (y)) [aay=a") (c)) (a) (Baa,oa((e))) Ve Vo log (a(ccc))) Ve logne(())) T0(a")c))
(18)
(6) V log No (c:) TT(a" (a*le;),
where (a) is due to the definition of the indicator function; (b) is based the zero-mean property of score functions. Plugging Eqn (18) into the right hand side of Eqn (17), we have
11
Vo log (c()) n
Σ mo
1 n
Vo log (og)((()) Wi,
(19)
j
where we used the definition of w; from Eqn (10). Lastly, we combine Egn (16) and Eqn (19) and obtain
I+ Vono(a"le)) Vo log (o(s)) (To(a*)cz) wis) redruedu
(20)
Thus have concluded the proof of Lemma D.4.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have made accurate claims in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitation of the proposed algorithms.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
22

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The proof is provided in full in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our results are reproducible given detailed description of the algorithm and pseudocode.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
23

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
We do not open source data and code, though we describe algorithmic details.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
oonooicyoooooooooooolooooomoynle for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. wttpcccccppcccccctccccccccccccct hwwbblic/guidess/eeeeeeeee httpc/gggideee/eeeuuuueeeeeeeeee www.eeuuubmssssooooiooooooo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided details on hyper-parameter choice, experimental design and baseline comparison the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
24

---

7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our experimental results include error bars that reflect statistical significance of the results.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: We did not provide details of computational resources required to execute the experiment, as this highly depends on the specifics of the implementation.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt///nurip.... httpppppppppppppiptttttiiaieaaaa
25

---

Answer: [Yes]
Justification:
We confirm the conforming to ethics guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is algorithmic and theoretical in nature and does not produce immediate societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work is algorithmic and theoretical and hence does not impose risks for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
26

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
We do not release assets and hence no need for the accompanying licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do provide additional assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
27

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
There is no human study nor crowd sourcing in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We make use of no LLM in this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer LLM policy (https: : htt////eurrppp.eee httparrpppscc//onneeeenee////220 for what should or should not be described.
28


Reviews:
[{"review_id": "oY3Jh9TYFU", "reviewer": "Reviewer_SjMC", "summary": "The paper introduces a new rl scheme for training on datasets where prompts and ground-truth answers are available, but the correctness of generated solutions is difficult to verify. The method aims to maximize the likelihood of the ground truth given the prompt, treating the cot reasoning as a latent variable. A Jensen lower bound is used as a tractable surrogate for the likelihood. Several techniques and theoretical analyses are proposed to tighten the bound and reduce variance. Comprehensive experiments are conducted to validate the approach.", "questions": "- **Entropy dynamics**: In JEPO, how does the entropy behave during training? In standard verifiable RL training, entropy typically decreases and then stabilizes. Does JEPO follow a similar pattern, or is the behavior different? Would adding an explicit entropy loss to JEPO be beneficial?\n\n- **Answer granularity**: In the verifiable reward setting, is the correct answer $a^*$ just the final boxed answer (e.g., an integral value in AIME problems), or does it include the full solution reasoning block? How do you extract $c_i$ from the generation? Do you simply parse the content inside `<think>` tokens?\n\n- **Robustness to incorrect labels**: How robust is JEPO to incorrect ground-truth labels $a^*$? For example, in [1], it was shown that RLVR is relatively robust to such label noise. Have you observed similar robustness in JEPO?\n\n[1] Shao, Rulin, et al. *Spurious rewards: Rethinking training signals in RLVR*. arXiv preprint arXiv:2506.10947 (2025).", "limitations": "No obvious limitations are observed.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n- The problem of extending RL from verifiable to unverifiable settings is important and timely.\n- The proposed method is theoretically sound. The techniques introduced to tighten the lower bound and control variance are solid.\n- The experimental evaluation is thorough, covering verifiable, semi-verifiable, and unverifiable settings, and scaling across models from 7B to 70B parameters. The method demonstrates strong generalization and scalability.\n- The ablations on sample count $n$ and the sft coefficient are insightful.\n\n**Weaknesses:**\n\n- I do not observe any obvious weaknesses in the paper.", "was_revised": true, "final_justification": "The method is interesting and statistically sound. I like the paper and would be glad to see it accepted.", "rebuttal": "Thank you very much for your time and efforts in reviewing our manuscript. We will make sure to incorporate your feedback into our later revisions.\n\nThank you very much for your constructive feedback and very positive view towards our paper!\n\n> Entropy dynamics: In JEPO, how does the entropy behave during training? In standard verifiable RL training, entropy typically decreases and then stabilizes. Does JEPO follow a similar pattern, or is the behavior different? Would adding an explicit entropy loss to JEPO be beneficial?\n\nIn JEPO, entropy dynamics behaves rather similarly as regular RL training. This might be because JEPO ultimately looks to find CoTs which maximize the marginal likelihood of the ground truth answer, in a similar spirit as regular RL that finds CoTs that make the ground truth answer generation more likely.\n\nWe speculate that adding entropy to JEPO would potentially have a similar effect as adding entropy to regular RL. It might help avoid premature convergence in some cases but tuning the entropy coefficient itself involves some hyper-parameter tuning to make things stable.\n\n> Answer granularity: In the verifiable reward setting, is the correct answer  just the final boxed answer (e.g., an integral value in AIME problems), or does it include the full solution reasoning block? How do you extract  from the generation? Do you simply parse the content inside <think> tokens?\n\nThank you for raising this point! We have actually discussed the specifics of this important technical detail in Appendix B. We will make it more clear in the revision.\n\nDuring SFT we have taught the model to carry out reasoning, followed by the final answer is, followed by the ground truth answer (e.g., an integer value). In other words, the overall string output from the model should be\n\nReasoning + `the final answer is` + answer\n\nWe extract the string before the marker `the final answer is` as the reasoning CoT $c_i$, and the string after as the final answer $a$. If the model does not comply with this formatting, we penalize the model output with a constant penalty. We find that with this, the model can quickly learn to format the full response correctly.\n\nAnother idea is indeed as the reviewer suggests, we can use a special thinking token as a delimiter, rather than a marker string. This will potentially make the formatting more stable and the parsing more convenient.\n\n> Robustness to incorrect labels: How robust is JEPO to incorrect ground-truth labels ? For example, in [1], it was shown that RLVR is relatively robust to such label noise. Have you observed similar robustness in JEPO?\n\nWe have not thoroughly investigated the robustness of JEPO performance to ground truth labels. Though based on the conceptual connection between JEPO and regular RLVR, we would expect some robustness to transfer given the evidence in [1]. It would be a valuable follow up ablation to carry out, to better understand whether JEPO scales better in the presence of data noise."}, {"review_id": "8mOmk2Tb3W", "reviewer": "Reviewer_kgcS", "summary": "The paper considers the problem setting of unverifiable data, where it's difficult to judge the ground truth correctness of arbitrary answers (e.g., long-form non-formalized mathematical proofs). Noting that RL is difficult to use in such situations, the authors propose JEPO, which maximizes a lower bound to the log likelihood of a (assumed available) ground truth answer. Critically, this differs from directly supervised finetuning on the answer because the authors specifically consider the setting with chain-of-thought, so the log likelihood involves an expectation over chain-of-thoughts. The authors also extend this to a multi-sample bound. The authors evaluate on math datasets, ranging from settings where the answer is verifiable to unverifiable, comparing JEPO with varying levels of samples against an RL baseline.", "questions": "See questions above as well. Below are some more clarifying questions.\n\nLine 87 - regarding the motivation, is this really what we want? Note that this does not directly enforce correct chains of thoughts. The objective could be satisfied by the LM always outputting the right answer, for any arbitrary chain of thought, including wrong chains of thoughts. I understand that it's hard to enforce correctness on the chain of thought without something more expensive like process reward models, but I'd be interested in seeing more analysis around whether this is a problem, and if not, why not.\n\n$g_2^{(n)}$ is itself a lower bound right? Seems worth discussing and investigating more. Also, how does this compare to the unbiased $g_2$ term in Eq. 4?\n\nI only took a brief look at VR-CLI but it seems to me more different from your work than your writing suggests? Maybe worth more explanation.\n\nLines 334-335 - where in Section 6 do you show the different $\\beta_{sup}$?\n\nOverall, I am on the fence about this work. The paper explores interesting directions and there are many good ideas in it. But unfortunately I think the evaluations are not strong, and the weaknesses above are a bit too much, leading me to borderline reject instead of borderline accept. But I would consider increasing my score if the authors convincingly address my points above.", "limitations": "There is much more the authors could discuss on limitations (e.g., my points above).\n\nThe authors claim that \"Our work is algorithmic and theoretical in nature and does not produce immediate societal impacts\", but I disagree with this. If there is any chance the work may have impact on e.g., improving the mathematical ability of frontier models, this could have significant societal impacts. Any capability advances come with possible harm in the hands of bad actors. While I agree that this work in particular is unlikely to have negative (or positive) societal impacts, I wouldn't say that chance is 0.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nProblem setting is interesting and worth exploring; improvements could lead to advances in important areas such as proof generation. \n\nOverall ideas are generally good, and authors consider and explore several worthwhile directions. \n\nCore theory seems mostly correct and reasonable to me. \n\nPaper is not too hard to understand.\n\nWeaknesses:\n\nLine 70 - should be \"a desirable\".\n\nLines 78-80 - not grammatically correct. Maybe an easy fix for this is \"getting\" -> \"we get\".\n\nEq 4 is not strictly correct; the expectation is omitted.\n\nLine 122 - should be \"introduces\".\n\nLine 127 - I recommend \"yet a tighter\" -> \"yet tighter\" and removing the comma \"alternative, is\" -> \"alternative is\".\n\nIt is OK to refer to previous papers for derivations (as is done several times throughout), but it would be better if at least the Appendix includes more details, rather than a single reference, even if it is as simple as copying math and changing the notation to match the rest of the paper. The less time and effort the reader has to expend to understand the work, the better.\n\nLine 133 - what are these regularity conditions? Should be stated, at least in the Appendix.\n\nFirst term of Eq. 7 I believe is missing a 1/n.\n\nLine 146 - \"practice .\" has an extra space.\n\nLine 160 reads a bit awkwardly, maybe \"their update only partially optimizes for the...\" would be better.\n\nContrary to the checklist point 7, I don't see any error bars, confidence intervals or tests of statistical significance for the experiments. Are the authors referring to the lightly shaded dots (which I presume are from a separate seed)? Those are not error bars. \n\nI would suggest moving the SFT comparison from the Appendix to the main paper (as an additional line on the existing plots). Also, instead of just saying in line 235 that there are a bunch of ablations in the Appendix, it would be good to briefly outline what those ablations are (e.g., comparison vs. SFT, regularization, model size, etc.). \n\nLine 203 - hyperparameters like batch size being constant is good for fair comparison, but not all hyperparameters should be constant. Things like learning rate should be tuned (at least using a coarse search) for each method. \n\nLines 206-208 - I don't think this is good reasoning for not comparing against other work. Different RL methods can have dramatically different results, and different RL algorithms have much more than \"low-level implementation differences\". Even if there are only \"low-level implementation differences\", these can still have material impacts on performance. What RL algorithm is used in the paper? \n\nLines 227-230 - on Figure 1, the RL frontier is clearly to the left of the JEPO frontier, except for very high KL divs. For very high KL divs, you might be running into the limits of model capacity, which may explain why performance is similar. In any case, the point of building these plots and frontiers is to look at the whole frontier, and overall the RL frontier looks much better.\n\nFigure 2 - it's interesting and good to verify but perhaps unsurprising that RL, which is based on a sympy scorer and not exact match, would have lower NLL than your objective which includes in it an essentially SFT component on the (train set) answers. Later on I see that this is discussed, but the organization is kind of awkward - the figure appears way before the discussion, and the discussion of Figure 2b comes after discussing Figure 3. I get there are space constraints and this is space efficient, but it's probably worth trying to reorganize things or maybe even use in-line figures so the figures and paper content flow better.\n\nLines 260-261 - it seems important to me to quantify how many false negatives are mitigated. If checking the entire dataset is too much work, you could at least randomly sample a subset and say what proportion of false negatives were corrected. Another thing that would be important to check is how many false positives are added. LLM-as-judge might also incorrectly think some good-looking or close-enough answers are correct. The LLM-as-judge evaluation might possibly be biased in favor of JEPO over RL given that the authors' previous NLL results imply that RL is finding different, possibly less standard solutions - these seem like the kinds of solutions an LLM-as-judge would be more likely to miss. Furthermore, if JEPO outputs things that look more like the test set solutions, but are subtly wrong, the LLM-as-judge may catch fewer wrong things from JEPO relative to RL. To check this isn't a concern, I think it's pretty important to have some quantification of the false positive and false negative rates across RL answers and JEPO answers on some subset of data.\n\nLine 288 - seems like it should be \"with test set\" instead of \"with set\".\n\nLines 335-338 - I have a simpler alternate explanation. Here is evaluating NLL only, whereas previous evaluation was based on sympy and LLM-as-judge. It's not really surprising that NLL is more impacted by the supervised learning component than metrics like sympy scoring which are not as directly related. \n\nThere should probably be a conclusion section, even if a short one.\n\n(Not sure where to put this, so putting here, though only sort of a weakness): originality of the paper I'm saying is fair; there's novelty in the application of ideas to this problem setting, but I won't give a higher score since I think it's close enough conceptually to existing work on e.g. ELBO, variational inference, IWAE, etc. (this is also a reason why a lot of the proofs in the paper are simply references to existing works)", "was_revised": true, "final_justification": "While the discussion with the authors has improved my evaluation of the paper slightly, I am not sure it's enough to push me from borderline reject to borderline accept. I am still very much on the fence about this work (as it stands). If the AC/meta-reviewer wants to accept this paper, I would consider that a reasonable decision (though still personally weakly disagree with that).", "rebuttal": "Thank you very much for your time and efforts in reviewing our manuscript. We will make sure to incorporate your feedback into our later revisions.\n\n> Line 70… Line 160 reads a bit awkwardly...\n\nThank you very much for the extremely detailed feedback on typos and grammar improvements, we will make sure to revise in our next manuscript.\n\n> Contrary to the checklist point 7, I don't see any error bars..\n\nIndeed, lightly shaded dots are results from multiple runs with the same configuration. In our own experience, we have found in some cases, it is a bit more informative to display the full distribution over different runs rather than standard errors or confidence intervals. We’d be happy to provide additional information such as error bars and confidence intervals if the reviewer finds them useful.\n\n> I would suggest moving the SFT comparison from the Appendix to the main paper...\n\nWe are definitely happy to move the SFT ablation discussion and more details into the main paper as the reviewer suggests. The current formatting in the paper is primarily due to space limitations, but if we are allowed an extra page in camera ready, additional results can be presented in the main paper.\n\n> Line 203 - Things like learning rate should be tuned (at least using a coarse search) for each method.\n\nIt is true that hyper-parameters like learning rate can be ablated in this case. In our early investigation, we have found the evaluation performance to be reasonably robust against learning rate variations, though the training curve gets impacted slightly (e.g., within a stable interval of learning rate, increasing the learning rate leads to faster growth in training reward across the board). We can provide additional information in the revision.\n\nBesides learning rate, we have indeed included other hyper-parameters for comparison, which we find to be central to RLVR methods in general, such as the number of samples per prompt $n$ and the regularization coefficient $\\beta$.\n\n> Lines 206-208 - I don't think this is good reasoning for not comparing against other work... What RL algorithm is used in the paper?\n\nWe agree that some technical details can make a material difference in practice. Maybe a more precise way to state the rationale here, is that some algorithmic variations are orthogonal to our algorithmic proposals in this work, and can be readily combined with our method in a complementary way. For such cases, we argue it might offer less value to carry out a comparison.\n\nThe algorithm we compare against is a simple version of RLOO (Reinforce with leave-one-out [1]. We argue it is the backbone for most existing RL algorithms such as GRPO [2] and its variants. Indeed, when on-policy (which is the experimental setup we adopt), GRPO and its variants will fall back to RLOO. RLOO also shares the most algorithmic structure with our method (e.g., leave-one-out control variate), which we find a comparison might be the most insightful.\n\n[1] Ahmadian et al, 2024, Back to Basics: Revisiting REINFORCE...\n\n[2] Shao et al, Pushing the limits of mathematical reasoning...\n\n> Lines 227-230 - on Figure 1, the RL frontier is clearly to the left of the JEPO frontier, except for very high KL divs..\n\nWe acknowledge that RL traces out a better KL-reward frontier in this case. But we have emphasized a few points in the paper: (1) It is the training plot where RL is better, and on evaluation plot (mid plot), JEPO is as competitive; (2) RL is at an advantage here because it has access to the reward function to plot against too, unlike JEPO, which does not assume access to the reward. \n\nThe key message is that Figure 1 shows a case where RLVR should work the best (i.e., short-form data with a good verifiable reward). Section 7-8 shows the results on semi-verifiable and unverifiable domains, where JEPO methods shine. This is a practical trade-off we seek to highlight in the paper.\n\n> Figure 2 - it's interesting and good to verify but perhaps unsurprising that RL ...\n\nThank you for finding the results interesting. Indeed, the results are meant to display a potential misalignment between likelihood based approach and Sympy based approach. It is indeed a limitation from the space constraints that causes our sub-optimal arrangement of texts and figures in the paper, we will make improvements in the revision.\n\n> Lines 260-261...  I think it's pretty important to have some quantification of the false positive and false negative rates across RL answers and JEPO answers on some subset of data.\n\nWe acknowledge that LLM-as-judge might have its limitations. In our early investigation of LLM-as-judge, we did find that the judge model can alleviate more false positives than the false positives it introduces. As evidence of this, we have found that training against LLM-as-judge signals helps improve evaluation performance to a level slightly better than training against just limited Sympy signals. \n\nWith some manual check, we find that Sympy correlates with human 80% of the time while judge correlates with 90%. We find that the judge’s improvement on Sympy’s false negative overwhelms its potential loss on false positive, resulting in an accuracy that’s higher than just using judge. Hence we use a combined score for eval. We are happy to provide more detailed LLM-as-judge calibration results in revision.\n\n> Lines 335-338 - I have a simpler alternate explanation...\n\nWe agree with the reviewer’s interpretation, this is indeed the conclusion that we seek to find more empirical evidence for, since it is not obvious by default.\n\n> originality of the paper I'm saying is fair\n\nDespite the strong connections to prior work such as ELBO, VAE, IWAE, we believe that there are many technical contributions on top which distinguish our work: (1) JEPO is the first implementation of such ideas “as they are developed mathematically”. Though prior work borrows high level ideas, the implementations turn out to be very much similar to RLVR, see discussion in 170-175; (2) JEPO has unique technical contributions specific to LLM, instrumental to actually making the system “work”, e.g., how to resolve certain technical constraints specific to LLM. Indeed, it is not trivial to apply VAE ideas to LLM out of the box.\n\nWhile JEPO is not groundbreaking in its mathematical foundation (which is the same as VAE), we argue that lots of important work in space is not either. Rather, we feel technical solidity should take more priority over novelty in the space of ideas.\n\n> Line 87 - regarding the motivation, is this really what we want? ... but I'd be interested in seeing more analysis around whether this is a problem.\n\nWe believe JEPO does not resolve the fundamental CoT correctness issue arising in conventional outcome based RLVR scenarios. Indeed as the reviewer suggested, we cannot do better unless a process supervision signal is available.\n\nOur speculation is that by leveraging potentially more unverifiable and semi-verifiable data, JEPO can benefit from a larger effective training set, compared to RLVR with a more rigorous reward function. In doing so, the model might overfit less to the final ground truth answer, and hence reducing the effect of “bad CoTs leading up to correct answer”. We do not have a systemic evaluation of this, rather we only evaluate the test set performance as a proxy metric.\n\n>  is itself a lower bound right? Seems worth discussing and investigating more. Also, how does this compare to the unbiased term in Eq. 4?\n\nNote that $g_2^{(n)}$ is itself not a lower bound, but rather a partial gradient to the $n$-sample lower bound (see Eqn 7).\n\nEqn 4 shows $g_2$ for a single sample lower bound, which can be seen as a special case to the $n$-sample lower bound with $n=1$. \n\nComparing the two: these are gradients to different lower bounds. In general, the bound is tighter as $n$ increases (see discussion near line 131), but the variance of the gradient might increase as well, causing a trade-off in principle. Figure 2 shows the empirical comparison that indicates that increasing $n$ in general seems to be helpful, though there is also the practical constraint of the algorithm being more expensive in $n$ due to generations. We will highlight more such discussions in the revision.\n\n> I only took a brief look at VR-CLI but it seems to me more different from your work than your writing suggests?\n\nWe leaned on the more conservative side and discussed some similarities between VR-CLI and our approach. To be fair, VR-CLI is on a high-level similar to using likelihood as a reward, which bears intuitively resemblance to the first part of the gradient $g_1, g_1^{(n)}$ in this case. Yet, many technical details might differ: VR-CLI is not theoretically grounded to improve the likelihood objective, as we have discussed in this work with the lower bound formulation, which requires transformation of individual log likelihood.\n\n> Lines 334-335 - where in Section 6 do you show the different \n\nThis was a discussion of early investigations we have: we found that when $\\beta_\\text{sup}$ is large, it tends to produce worse results with short-form answer dataset like MATH. We speculate this is because the force of SFT loss is too large with large values of $\\beta_\\text{sup}$, and causes the training to overfit more (after all, it’s simple to predict just a few tokens worth of answers, compared to long-form answers). We will provide more detailed results in the revision.\n\n> There is much more the authors could discuss on limitations.\n\nWe are happy to discuss the limitations as the reviewer pointed out, given more space in the camera ready revisions.\n\n> The authors claim that \"Our work is algorithmic and theoretical in nature and does not produce immediate societal impacts\"\n\nWe agree that if the algorithmic improvements reach certain points, it can generate sizable social impact. We are happy to provide more discussions on this in the checklist and in the revision."}, {"review_id": "KJ6lBA6G2p", "reviewer": "Reviewer_roE2", "summary": "The authors propose a policy optimization algorithm that targets reinforcement learning with unverifiable rewards. The algorithm, namely Jensen's Evidence lower bound for Policy Optimization (JEPO), leverages Jensen's evidence lower bound, which is based on a latent variable perspective of the LLM's chain-of-thought. The algorithm itself also presents a much simpler objective and makes RL more scalable. The authors provide both theoretical justification and empricial evidence to demonstrate the effectiveness of the algorithm on MATH and Numina.", "questions": "1. In the introduction (line 28), the authors define data as unverifiable if the ground truth answer to the problem cannot be verified with reasonable automated effort. If I understand correctly, this definition still assumes the availability of the answer. Is that correct? If so, do we also care about the quality and the uniqueness of the answer?\n2. In Section 5, the authors indicate the difficulty of using LLM-as-a-Judge as a way to tackle unverifiable data. Do the authors have concrete examples of this, even when the (unverfiable) ground truth answer is available?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed algorithm is shown to be espeically effective in the scenarios of math problems with hard-to- or non-verifiable ground truth.\n2. The core policy optimization algorithm is built upon Jensen's lower bound and is theoretically sound.\n3. The paper is clearly written and well-structured. It follows standard notation and is easy to understand.\n\nWeaknesses:\n\n1. The theoretical section of the paper is robust, but the experiments presented are somewhat lacking.\n    - The main experiments are based on MATH and Numina only, which only assesses in-distribution problems. This is important as we do not always have in-distribution data for training.\n    - In the Numina benchmark, roughly 40% of training examples cannot be scored by SymPy. As the authors note, the baseline RL agent therefore optimizes only a subset of the data, and JEPO is evaluated under conditions where a large fraction of examples have no ground-truth verification. This leaves open whether the method scales when the majority of data are truly unverifiable, or when tasks fall outside the narrow domain of mathematical proofs.\n    - In the core MATH benchmark the authors state \"We do not compare with other baselines developed in prior work.\" This leaves open the possibility that apparent gains might come from experimental setup rather than the proposed JEPO algorithm.\n2. Some minor points:\n    - For MATH evaluation, it might be more ideal to consider MATH-500, which are less likely to be contaminated.\n    - SymPy is known to have many parsing issues specifically for MATH [2], so the reward may not be entirely accurate either.\n    - The authors use Llama 3 70B as the LLM-as-a-Judge for evaluation. The authors also mention the drawback of using LLM-as-a-Judge for domain-specific and long-form data. While Llama 3 70B is a strong model, it may be a less competitive option compared to some of the leading reasoning models available today, especially on this particular task of verifying proofs.\n    - For the semi- and unverifiable splits, success is measured by a composite score that falls back on majority voting from Llama 3 70B when automatic SymPy checks fail. Because of the above reasoning, it makes it hard to know whether gains are real reasoning improvements or artifacts of that judge's potentially deviated preferences.\n\nReferences\n\n[1] Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., ... & Cobbe, K. (2023, May). Let's verify step by step. In The Twelfth International Conference on Learning Representations.\n\n[2] Kydlicek, H., Lozovskaya, A., Habib, N., & Fourrier, C. (2025, February 14). Fixing Open LLM Leaderboard with Math‑Verify. Hugging Face Blog.", "was_revised": true, "final_justification": "I decided to keep my original score of this paper after the rebuttal. My major concern in the original review is that the paper only demonstrate limited evaluation in in-distribution/domain settings, which is rarely the case the real world.\n\nSpecifically, the authors have only demonstrated evaluation on the MATH and NuminaMath test sets. While this approach is ideal for experiments, questions about whether it truly transfers to more challenging and mainstream benchmarks like the AIME remain.\n\nThe authors have provided sufficient clarifications in the rebuttal; however, they have not included experimental results to address my concerns about the practicality of the method.", "rebuttal": "Thank you very much for your time and efforts in reviewing our manuscript. We will make sure to incorporate your feedback into our later revisions.\n\nThank you very much for your constructive feedback and very positive view towards our paper!\n\n> The main experiments are based on MATH and Numina only, which only assesses in-distribution problems. This is important as we do not always have in-distribution data for training.\n\nWe acknowledge this is a limitation of this work, that the training and testing sets are mostly in distribution. That said, we think this setting offers a more narrow yet more controlled investigation into performance gains, and maybe rule away slightly confounding factors that impact how out of distribution generalization works.\n\nWe would hope that investigations into out of distribution generalization can be pursued on its own right in another follow up work, or in industry labs where such considerations are more practically meaningful. Nevertheless, we hope this work offers some limited insights.\n\n> In the Numina benchmark, roughly 40% of training examples cannot be scored by SymPy. As the authors note, the baseline RL agent therefore optimizes only a subset of the data, and JEPO is evaluated under conditions where a large fraction of examples have no ground-truth verification. This leaves open whether the method scales when the majority of data are truly unverifiable, or when tasks fall outside the narrow domain of mathematical proofs.\n\nThank you for the comments on this. We have indeed manually checked that among the *non-proof* Numina dataset, there is *still* a significant portion (about 40%) of the examples where the answers are in between short-form (e.g., a single number of a single expression) and very long-form (e.g., a full proof). These answers are usually a bit elaborate and cannot be checked by the Sympy scorer we implemented for the experiments.\n\nIn Sec 7, when training on the semi-verifiable Numina dataset, we actually excluded the proof Numina sub-set, and only trained on the non-proof section of the dataset. If we understand your comment correctly, this would mean that our approach can scale to cases where a good portion of the data is unverifiable (40%), though not necessarily “majority of the data”.\nWe think it is a meaningful ablation to understand whether JEPO works in the presence of a majority of unverifiable data. That said, in practice, data collection works jointly with algorithmic design, and if the in-the-wild Numina dataset contains 40% unverifiable data, it might be less interesting for a case where most data is unverifiable. Would the reviewer has in mind some dataset like this?\n\n> In the core MATH benchmark the authors state \"We do not compare with other baselines developed in prior work.\" This leaves open the possibility that apparent gains might come from experimental setup rather than the proposed JEPO algorithm.\n\nThank you for pointing this out. Maybe more precisely, what we meant is that: (1) We compare with a RL algorithmic baseline in a way that minimizes the discrepancy between experimental setups; (2) There are other RL algorithmic variants, with changes complementary to our JEPO algorithm. \n\nBy (1), we practically mean that the only difference between JEPO vs. RL baseline, is the loss function. They share other settings such as hyper-parameters, training-generation loop, leave-one-out control variates construct etc. This rules out confounding factors as the reviewer worries about as much as possible, leading us to conclude more confidently that the improvements or trade-offs are due to loss functions. \n\nIt is also important to note that since we do on-policy training, our algorithm recovers state-of-the-art algorithms like GRPO (when GRPO is on-policy, its importance ratios).\n\nBy (2), we mean that there are changes such as different ways of doing group aggregation (e.g., difference between Dr. GRPO vs. GRPO), which are complementary to the designs of our algorithm. That is, in principle, such algorithmic changes can be combined with our approach, for which we find an ablation offers less insight.\n\n> For MATH evaluation, it might be more ideal to consider MATH-500, which are less likely to be contaminated.\n\nThank you for the suggestion on this. We can try to carry out additional evaluation on this if the reviewer finds it useful.\n\n> SymPy is known to have many parsing issues specifically for MATH [2], so the reward may not be entirely accurate either.\n\nThank you very much for the suggestion. We have some additional fixes inside the “Sympy” parser that actually removes some of the false positives and negatives suggested in the paper you shared, but still there might be some inaccuracies remaining. We will make sure to discuss this point in the revision.\n\n> The authors use Llama 3 70B as the LLM-as-a-Judge for evaluation. The authors also mention the drawback of using LLM-as-a-Judge for domain-specific and long-form data. While Llama 3 70B is a strong model, it may be a less competitive option compared to some of the leading reasoning models available today, especially on this particular task of verifying proofs.\n\nWe agree with the reviewer’s comment on this. We used LLM-as-judge mainly to assess semi-long-form data, so data in Numina which is not as long-form as proof, but is more elaborate than a single mathematical expression that can be verified by Sympy. We find LLM-as-judge to work reasonably well in this case, even with Llama 70B, though indeed more powerful models can potentially provide more accurate assessments.\n\nTo provide some calibration results using LLM-as-judge for the Llama 70B model: with manual checks, we find that Sympy correlates with human 80% while LLM-as-judge can improve it to 90%. But potentially more competitive models can offer even higher correlation with some tweaks or tuning, which we leave to further investigations.\n\nWe want to emphasize that we did not use LLM-as-judge to verify proof in the paper. The main reason is due to the negative result shared in [1], where the authors shared discrepancies between LLM-as-judge from frontier models and human annotators. Though the Numina proof dataset might be distributionally quite different from the Math Olympiad as studied in [1], we feel less confident in using LLM-as-judge score for proof verifications.\n\n[1] Petrov et al, 2025, Proof or Bluff: Evaluating LLMs on 2025 USA Math Olympiad\n\n> For the semi- and unverifiable splits, success is measured by a composite score that falls back on majority voting from Llama 3 70B when automatic SymPy checks fail. Because of the above reasoning, it makes it hard to know whether gains are real reasoning improvements or artifacts of that judge's potentially deviated preferences.\n\nWe acknowledge that LLM-as-judge might potentially have its deviated preferences, e.g., to certain formatting. In our early investigation of LLM-as-judge, we do find that the judge model can alleviate more false positives than the false positives it introduces compared to Sympy checker. As evidence of this, we have found that training against LLM-as-judge signals helps improve evaluation performance to a level slightly better than training against just limited Sympy signals.\n\nOur early manual check of LLM-as-judge found that it is quite accurate for assessing medium-form data in the semi-verifiable dataset (note that they are generally not as long as full proof, but rather elaborate short-form answers). We can provide a more accurate evaluation of LLM-as-judge against human annotations in the revision, in case the reviewer finds it useful.\n\n> In the introduction (line 28), the authors define data as unverifiable if the ground truth answer to the problem cannot be verified with reasonable automated effort. If I understand correctly, this definition still assumes the availability of the answer. Is that correct? If so, do we also care about the quality and the uniqueness of the answer?\n\nThat is correct, we still assume access to a ground truth answer.\n\nRegarding quality: we think it is definitely important to guarantee that the answers are of high quality. Indeed, this will directly feed into how well the model learns from the training set and impacts its ability to generalize. We feel this argument applies for both regular RLVR and JEPO.\n\nRegarding uniqueness: we believe this is quite a subtle question and whose impact might warrant more careful investigations. In RLVR, the verifier or reward function judges if an answer is equivalent to the ground truth answer, and so it kind of assumes that the ground truth might be non-unique (otherwise we’d use a string match reward). It might be interesting to investigate further how the diversity or uniqueness of the answers, impact RLVR and JEPO in more practical scenarios.\n\n> In Section 5, the authors indicate the difficulty of using LLM-as-a-Judge as a way to tackle unverifiable data. Do the authors have concrete examples of this, even when the (unverfiable) ground truth answer is available?\n\nWe meant to indicate that LLM-as-judge can be used for assessing the equivalence of model generated answer and an available *ground truth answer*, hence replacing the role of a Sympy checker in more unverifiable domains. That is, our discussion always assumes access to a ground truth reward.\n\nIn case a ground truth reward is not available, it feels more akin to the conventional RLHF settings where a reward model trained from human preference or a judge model directly assesses the quality of responses. Along this line, [1] is an example where the LLM is directly prompted as a judge to deliver an evaluation score (without access to ground truth).\n\n[1] Lee et al, 2024, RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback"}, {"review_id": "78y8Y2QCWm", "reviewer": "Reviewer_aCix", "summary": "The paper introduces JEPO (Jensen’s Evidence lower bound for Policy Optimization), an RL algorithm that dispenses with explicit, automatically checkable rewards and instead maximizes a Jensen-tightened lower bound on the log-likelihood of ground-truth answers, treating the model’s chain-of-thought as a latent variable. Compared with standard online RL from verifiable rewards (e.g., Sympy-matched answers in math datasets), JEPO requires no external reward signal, works with both verifiable and unverifiable long-form answers, and combines a REINFORCE–style term with a supervised log-probability term. Experiments on three settings—verifiable (MATH), semi-verifiable (Numina), and unverifiable (Numina-Proof)—show that single-sample JEPO matches, and its multi-sample variant often surpasses, traditional RL on evaluation metrics, while being able to exploit the entire dataset when only a subset is verifiable.", "questions": "see weakness", "limitations": "see weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- Conceptual novelty – Re-framing chain-of-thought optimization as latent-variable modeling and applying Jensen’s inequality yields a simple, principled objective that avoids auxiliary posterior models and scales naturally to unverifiable data.\n\n- Practical relevance – Many real-world tasks have long, hard-to-verify answers; JEPO demonstrates competitive or better performance on such datasets, highlighting genuine utility beyond synthetic short-answer benchmarks.\n\n- Thorough empirical study – The authors test across three verification regimes (verifiable, semi-verifiable, unverifiable) and two model sizes (8 B, 70 B), include ablations on sample count, supervised-loss weight, and KL regularization, and compare with combined JEPO + RL baselines, providing a detailed performance picture.\n\n\nWeaknesses:\n\n- Evaluation of unverifiable tasks relies on proxy metrics only. For Numina-Proof (Sec. 7 & App. C.1), the authors switch to a test-set negative-log-likelihood surrogate. While JEPO shows lower NLL than RL, the paper never validates that lower NLL correlates with human-judged proof quality, leaving the headline claim (“improves reasoning on unverifiable data”) empirically under-supported. A small human study or at least an LLM-as-judge sanity check would have strengthened the evidence.\n- Loose theoretical guarantees – While the Jensen lower bound is tractable, it is strictly looser than the full ELBO; the paper provides limited analysis of how the gap affects convergence or final policy optimality, leaving theoretical performance bounds unclear.\n- Reward-free versus reward-rich trade-off – In verifiable settings JEPO lags behind standard RL during training and relies on KL regularization to maintain stability, suggesting efficiency drawbacks when high-quality rewards are available; the paper could discuss when one should prefer JEPO in practice.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time and efforts in reviewing our manuscript. We will make sure to incorporate your feedback into our later revisions.\n\nThank you very much for your constructive feedback and very positive view towards our paper!\n\n> Evaluation of unverifiable tasks relies on proxy metrics only. For Numina-Proof (Sec. 7 & App. C.1), the authors switch to a test-set negative-log-likelihood surrogate. While JEPO shows lower NLL than RL, the paper never validates that lower NLL correlates with human-judged proof quality, leaving the headline claim (“improves reasoning on unverifiable data”) empirically under-supported. A small human study or at least an LLM-as-judge sanity check would have strengthened the evidence.\n\nThank you for raising the suggestion. Indeed we agree that the result can be strengthened further by correlating NLL with human-judged proof quality. We did not yet have the capacity to carry out the human study by the time of the publication, so tried to resort to test set likelihood as a proxy. With negative results of using automated proof judge [1], we feel that LLM-as-judge for the proof data might not deliver very reliable eval results either.\n\nWe did have some manual qualitative assessments of the trained model: while the baseline outputs a proof directly, JEPO trained model outputs a high-level proof strategy followed by the detailed proof. This might corroborate with why the test set NLL is better, due to the learned CoT strategy before the detailed proof.\n\nDespite the limitations, we would still like to argue that the comparison is apple-to-apple, with a reasonable baseline.\n\n[1] Petrov et al, 2025, Proof or Bluff: Evaluating LLMs on 2025 USA Math Olympiad\n\n> Loose theoretical guarantees – While the Jensen lower bound is tractable, it is strictly looser than the full ELBO; the paper provides limited analysis of how the gap affects convergence or final policy optimality, leaving theoretical performance bounds unclear.\n\nWe acknowledge the theoretical looseness with the bound. With theory, we can seek to close the bound by increasing $n$, but fundamentally this is what we could do with such a methodological design.\n\nWe have extensively assessed the empirical impact of such a theoretical loose bound in experiments. Fig 1 showed that despite with a lower bound objective, we can optimize the target reward function; Fig 2 showed that increasing $n$ to $n=4$ correlates with the reward objective better than $n=1$, which corroborates with the performance ablations of Figure 2 across different $n$s.\n\nThe high-level message is that while the lower bound objective might be a loose objective, its looseness can be mitigated largely in practice by choosing a reasonable value of $n$. In our ablations, $n=4$ seems to suffice for the math related domains. And we leave to practitioners to understand the impact of this hyper-parameter in their specific settings.\n\n> Reward-free versus reward-rich trade-off – In verifiable settings JEPO lags behind standard RL during training and relies on KL regularization to maintain stability, suggesting efficiency drawbacks when high-quality rewards are available; the paper could discuss when one should prefer JEPO in practice.\n\nIndeed JEPO relies on KL regularization slightly more than RLVR in our ablations, but we feel that’s not a strong constraint. More broadly, we find that on domains where data is short-form and the reward is quite accurate, RLVR retains its advantage compared to JEPO (as we have shown in Fig 1). That said, JEPO is still quite competitive in this regime.\n\nYet in semi-verifiable or non-verifiable domains, JEPO has an advantage due to its more flexible learning objective (without a reliance on good reward). In practical situations where semi-verifiable or non-verifiable data are abundant, JEPO might be worth a try for practitioners."}]

Author rebuttal:

Meta-review: This paper addresses the problem of scaling reinforcement learning for LLMs beyond verifiable data, where correctness cannot be automatically checked (e.g., mathematical proofs). The authors propose JEPO (Jensen’s Evidence lower bound for Policy Optimization), which treats chain-of-thought as a latent variable and maximizes a Jensen-tightened lower bound on the log-likelihood of ground-truth answers. This enables RL training without explicit reward signals, covering verifiable, semi-verifiable, and unverifiable domains. Experimental results are also reported.

Reviewers agreed that the problem is timely and found the method theoretically sound and practically relevant. Strengths highlighted include the conceptual novelty of reframing RL as latent-variable modeling, clear applicability to unverifiable reasoning tasks, and comprehensive ablations across different datasets and model sizes. The paper suffers from some presentation issues (typos, figure placement, lack of clarity in some derivations), limited evaluation breadth (primarily math-based tasks, broader OOD tests or stronger verification baselines), and reliance on proxy metrics like LLM-as-a-judge for unverifiable data. 

Overall, I recommend acceptance. The contribution is technically solid and addresses an important gap in reinforcement learning for language models, especially in unverifiable domains where existing RL methods fall short. While presentation and empirical breadth can be improved, the rebuttal clarified key concerns and the main results remain sound. Please incorporate the final suggestions of the reviewers in the final version as promised.
Decision: Accept (poster)