Title: Beyond Verifiable Rewards: Scaling Reinforcement Learning in Language Models to Unverifiable Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

LLMs

Reasoning

Unverifiable

Verifiable
Abstract: We propose to scale RL to unverifiable data with a novel algorithm JEPO (Jensen's Evidence lower bound for Policy Optimization). While most prior effort on scaling RL for LLMs focuses on verifiable data where ground truth answers are typically short-form and can be matched easily, we investigate the case where such assumptions are less valid (e.g., when answers are long-form such as mathematical proofs). To scale RL training to unverifiable data with contemporary training constraints, we propose JEPO. JEPO applies Jensen's evidence lower bound, a pragmatic simplification of the evidence lower bound which views chain-of-thought as a latent variable in the generative process. We show that on verifiable datasets (math), JEPO is as effective as RL with verifiable reward; on semi-verifiable and unverifiable datasets (numina and numina-proof), JEPO improves on soft-match based evaluations compared to RL with verifiable reward which can only leverage a subset of the data source as well as test set likelihood evaluations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces JEPO (Jensen’s Evidence lower bound for Policy Optimization), an RL algorithm that dispenses with explicit, automatically checkable rewards and instead maximizes a Jensen-tightened lower bound on the log-likelihood of ground-truth answers, treating the model’s chain-of-thought as a latent variable. Compared with standard online RL from verifiable rewards (e.g., Sympy-matched answers in math datasets), JEPO requires no external reward signal, works with both verifiable and unverifiable long-form answers, and combines a REINFORCE–style term with a supervised log-probability term. Experiments on three settings—verifiable (MATH), semi-verifiable (Numina), and unverifiable (Numina-Proof)—show that single-sample JEPO matches, and its multi-sample variant often surpasses, traditional RL on evaluation metrics, while being able to exploit the entire dataset when only a subset is verifiable.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Conceptual novelty – Re-framing chain-of-thought optimization as latent-variable modeling and applying Jensen’s inequality yields a simple, principled objective that avoids auxiliary posterior models and scales naturally to unverifiable data.\n\n- Practical relevance – Many real-world tasks have long, hard-to-verify answers; JEPO demonstrates competitive or better performance on such datasets, highlighting genuine utility beyond synthetic short-answer benchmarks.\n\n- Thorough empirical study – The authors test across three verification regimes (verifiable, semi-verifiable, unverifiable) and two model sizes (8 B, 70 B), include ablations on sample count, supervised-loss weight, and KL regularization, and compare with combined JEPO + RL baselines, providing a detailed performance picture.\n\n\nWeaknesses:\n\n- Evaluation of unverifiable tasks relies on proxy metrics only. For Numina-Proof (Sec. 7 & App. C.1), the authors switch to a test-set negative-log-likelihood surrogate. While JEPO shows lower NLL than RL, the paper never validates that lower NLL correlates with human-judged proof quality, leaving the headline claim (“improves reasoning on unverifiable data”) empirically under-supported. A small human study or at least an LLM-as-judge sanity check would have strengthened the evidence.\n- Loose theoretical guarantees – While the Jensen lower bound is tractable, it is strictly looser than the full ELBO; the paper provides limited analysis of how the gap affects convergence or final policy optimality, leaving theoretical performance bounds unclear.\n- Reward-free versus reward-rich trade-off – In verifiable settings JEPO lags behind standard RL during training and relies on KL regularization to maintain stability, suggesting efficiency drawbacks when high-quality rewards are available; the paper could discuss when one should prefer JEPO in practice.\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nsee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a policy optimization algorithm that targets reinforcement learning with unverifiable rewards. The algorithm, namely Jensen's Evidence lower bound for Policy Optimization (JEPO), leverages Jensen's evidence lower bound, which is based on a latent variable perspective of the LLM's chain-of-thought. The algorithm itself also presents a much simpler objective and makes RL more scalable. The authors provide both theoretical justification and empricial evidence to demonstrate the effectiveness of the algorithm on MATH and Numina.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The proposed algorithm is shown to be espeically effective in the scenarios of math problems with hard-to- or non-verifiable ground truth.\n2. The core policy optimization algorithm is built upon Jensen's lower bound and is theoretically sound.\n3. The paper is clearly written and well-structured. It follows standard notation and is easy to understand.\n\nWeaknesses:\n\n1. The theoretical section of the paper is robust, but the experiments presented are somewhat lacking.\n    - The main experiments are based on MATH and Numina only, which only assesses in-distribution problems. This is important as we do not always have in-distribution data for training.\n    - In the Numina benchmark, roughly 40% of training examples cannot be scored by SymPy. As the authors note, the baseline RL agent therefore optimizes only a subset of the data, and JEPO is evaluated under conditions where a large fraction of examples have no ground-truth verification. This leaves open whether the method scales when the majority of data are truly unverifiable, or when tasks fall outside the narrow domain of mathematical proofs.\n    - In the core MATH benchmark the authors state \"We do not compare with other baselines developed in prior work.\" This leaves open the possibility that apparent gains might come from experimental setup rather than the proposed JEPO algorithm.\n2. Some minor points:\n    - For MATH evaluation, it might be more ideal to consider MATH-500, which are less likely to be contaminated.\n    - SymPy is known to have many parsing issues specifically for MATH [2], so the reward may not be entirely accurate either.\n    - The authors use Llama 3 70B as the LLM-as-a-Judge for evaluation. The authors also mention the drawback of using LLM-as-a-Judge for domain-specific and long-form data. While Llama 3 70B is a strong model, it may be a less competitive option compared to some of the leading reasoning models available today, especially on this particular task of verifying proofs.\n    - For the semi- and unverifiable splits, success is measured by a composite score that falls back on majority voting from Llama 3 70B when automatic SymPy checks fail. Because of the above reasoning, it makes it hard to know whether gains are real reasoning improvements or artifacts of that judge's potentially deviated preferences.\n\nReferences\n\n[1] Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., ... & Cobbe, K. (2023, May). Let's verify step by step. In The Twelfth International Conference on Learning Representations.\n\n[2] Kydlicek, H., Lozovskaya, A., Habib, N., & Fourrier, C. (2025, February 14). Fixing Open LLM Leaderboard with Math‑Verify. Hugging Face Blog.\n\nQUESTIONS:\n1. In the introduction (line 28), the authors define data as unverifiable if the ground truth answer to the problem cannot be verified with reasonable automated effort. If I understand correctly, this definition still assumes the availability of the answer. Is that correct? If so, do we also care about the quality and the uniqueness of the answer?\n2. In Section 5, the authors indicate the difficulty of using LLM-as-a-Judge as a way to tackle unverifiable data. Do the authors have concrete examples of this, even when the (unverfiable) ground truth answer is available?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem setting of unverifiable data, where it's difficult to judge the ground truth correctness of arbitrary answers (e.g., long-form non-formalized mathematical proofs). Noting that RL is difficult to use in such situations, the authors propose JEPO, which maximizes a lower bound to the log likelihood of a (assumed available) ground truth answer. Critically, this differs from directly supervised finetuning on the answer because the authors specifically consider the setting with chain-of-thought, so the log likelihood involves an expectation over chain-of-thoughts. The authors also extend this to a multi-sample bound. The authors evaluate on math datasets, ranging from settings where the answer is verifiable to unverifiable, comparing JEPO with varying levels of samples against an RL baseline.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nProblem setting is interesting and worth exploring; improvements could lead to advances in important areas such as proof generation. \n\nOverall ideas are generally good, and authors consider and explore several worthwhile directions. \n\nCore theory seems mostly correct and reasonable to me. \n\nPaper is not too hard to understand.\n\nWeaknesses:\n\nLine 70 - should be \"a desirable\".\n\nLines 78-80 - not grammatically correct. Maybe an easy fix for this is \"getting\" -> \"we get\".\n\nEq 4 is not strictly correct; the expectation is omitted.\n\nLine 122 - should be \"introduces\".\n\nLine 127 - I recommend \"yet a tighter\" -> \"yet tighter\" and removing the comma \"alternative, is\" -> \"alternative is\".\n\nIt is OK to refer to previous papers for derivations (as is done several times throughout), but it would be better if at least the Appendix includes more details, rather than a single reference, even if it is as simple as copying math and changing the notation to match the rest of the paper. The less time and effort the reader has to expend to understand the work, the better.\n\nLine 133 - what are these regularity conditions? Should be stated, at least in the Appendix.\n\nFirst term of Eq. 7 I believe is missing a 1/n.\n\nLine 146 - \"practice .\" has an extra space.\n\nLine 160 reads a bit awkwardly, maybe \"their update only partially optimizes for the...\" would be better.\n\nContrary to the checklist point 7, I don't see any error bars, confidence intervals or tests of statistical significance for the experiments. Are the authors referring to the lightly shaded dots (which I presume are from a separate seed)? Those are not error bars. \n\nI would suggest moving the SFT comparison from the Appendix to the main paper (as an additional line on the existing plots). Also, instead of just saying in line 235 that there are a bunch of ablations in the Appendix, it would be good to briefly outline what those ablations are (e.g., comparison vs. SFT, regularization, model size, etc.). \n\nLine 203 - hyperparameters like batch size being constant is good for fair comparison, but not all hyperparameters should be constant. Things like learning rate should be tuned (at least using a coarse search) for each method. \n\nLines 206-208 - I don't think this is good reasoning for not comparing against other work. Different RL methods can have dramatically different results, and different RL algorithms have much more than \"low-level implementation differences\". Even if there are only \"low-level implementation differences\", these can still have material impacts on performance. What RL algorithm is used in the paper? \n\nLines 227-230 - on Figure 1, the RL frontier is clearly to the left of the JEPO frontier, except for very high KL divs. For very high KL divs, you might be running into the limits of model capacity, which may explain why performance is similar. In any case, the point of building these plots and frontiers is to look at the whole frontier, and overall the RL frontier looks much better.\n\nFigure 2 - it's interesting and good to verify but perhaps unsurprising that RL, which is based on a sympy scorer and not exact match, would have lower NLL than your objective which includes in it an essentially SFT component on the (train set) answers. Later on I see that this is discussed, but the organization is kind of awkward - the figure appears way before the discussion, and the discussion of Figure 2b comes after discussing Figure 3. I get there are space constraints and this is space efficient, but it's probably worth trying to reorganize things or maybe even use in-line figures so the figures and paper content flow better.\n\nLines 260-261 - it seems important to me to quantify how many false negatives are mitigated. If checking the entire dataset is too much work, you could at least randomly sample a subset and say what proportion of false negatives were corrected. Another thing that would be important to check is how many false positives are added. LLM-as-judge might also incorrectly think some good-looking or close-enough answers are correct. The LLM-as-judge evaluation might possibly be biased in favor of JEPO over RL given that the authors' previous NLL results imply that RL is finding different, possibly less standard solutions - these seem like the kinds of solutions an LLM-as-judge would be more likely to miss. Furthermore, if JEPO outputs things that look more like the test set solutions, but are subtly wrong, the LLM-as-judge may catch fewer wrong things from JEPO relative to RL. To check this isn't a concern, I think it's pretty important to have some quantification of the false positive and false negative rates across RL answers and JEPO answers on some subset of data.\n\nLine 288 - seems like it should be \"with test set\" instead of \"with set\".\n\nLines 335-338 - I have a simpler alternate explanation. Here is evaluating NLL only, whereas previous evaluation was based on sympy and LLM-as-judge. It's not really surprising that NLL is more impacted by the supervised learning component than metrics like sympy scoring which are not as directly related. \n\nThere should probably be a conclusion section, even if a short one.\n\n(Not sure where to put this, so putting here, though only sort of a weakness): originality of the paper I'm saying is fair; there's novelty in the application of ideas to this problem setting, but I won't give a higher score since I think it's close enough conceptually to existing work on e.g. ELBO, variational inference, IWAE, etc. (this is also a reason why a lot of the proofs in the paper are simply references to existing works)\n\nQUESTIONS:\nSee questions above as well. Below are some more clarifying questions.\n\nLine 87 - regarding the motivation, is this really what we want? Note that this does not directly enforce correct chains of thoughts. The objective could be satisfied by the LM always outputting the right answer, for any arbitrary chain of thought, including wrong chains of thoughts. I understand that it's hard to enforce correctness on the chain of thought without something more expensive like process reward models, but I'd be interested in seeing more analysis around whether this is a problem, and if not, why not.\n\n$g_2^{(n)}$ is itself a lower bound right? Seems worth discussing and investigating more. Also, how does this compare to the unbiased $g_2$ term in Eq. 4?\n\nI only took a brief look at VR-CLI but it seems to me more different from your work than your writing suggests? Maybe worth more explanation.\n\nLines 334-335 - where in Section 6 do you show the different $\\beta_{sup}$?\n\nOverall, I am on the fence about this work. The paper explores interesting directions and there are many good ideas in it. But unfortunately I think the evaluations are not strong, and the weaknesses above are a bit too much, leading me to borderline reject instead of borderline accept. But I would consider increasing my score if the authors convincingly address my points above.\n\nLIMITATIONS:\nThere is much more the authors could discuss on limitations (e.g., my points above).\n\nThe authors claim that \"Our work is algorithmic and theoretical in nature and does not produce immediate societal impacts\", but I disagree with this. If there is any chance the work may have impact on e.g., improving the mathematical ability of frontier models, this could have significant societal impacts. Any capability advances come with possible harm in the hands of bad actors. While I agree that this work in particular is unlikely to have negative (or positive) societal impacts, I wouldn't say that chance is 0.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new rl scheme for training on datasets where prompts and ground-truth answers are available, but the correctness of generated solutions is difficult to verify. The method aims to maximize the likelihood of the ground truth given the prompt, treating the cot reasoning as a latent variable. A Jensen lower bound is used as a tractable surrogate for the likelihood. Several techniques and theoretical analyses are proposed to tighten the bound and reduce variance. Comprehensive experiments are conducted to validate the approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The problem of extending RL from verifiable to unverifiable settings is important and timely.\n- The proposed method is theoretically sound. The techniques introduced to tighten the lower bound and control variance are solid.\n- The experimental evaluation is thorough, covering verifiable, semi-verifiable, and unverifiable settings, and scaling across models from 7B to 70B parameters. The method demonstrates strong generalization and scalability.\n- The ablations on sample count $n$ and the sft coefficient are insightful.\n\n**Weaknesses:**\n\n- I do not observe any obvious weaknesses in the paper.\n\nQUESTIONS:\n- **Entropy dynamics**: In JEPO, how does the entropy behave during training? In standard verifiable RL training, entropy typically decreases and then stabilizes. Does JEPO follow a similar pattern, or is the behavior different? Would adding an explicit entropy loss to JEPO be beneficial?\n\n- **Answer granularity**: In the verifiable reward setting, is the correct answer $a^*$ just the final boxed answer (e.g., an integral value in AIME problems), or does it include the full solution reasoning block? How do you extract $c_i$ from the generation? Do you simply parse the content inside `<think>` tokens?\n\n- **Robustness to incorrect labels**: How robust is JEPO to incorrect ground-truth labels $a^*$? For example, in [1], it was shown that RLVR is relatively robust to such label noise. Have you observed similar robustness in JEPO?\n\n[1] Shao, Rulin, et al. *Spurious rewards: Rethinking training signals in RLVR*. arXiv preprint arXiv:2506.10947 (2025).\n\nLIMITATIONS:\nNo obvious limitations are observed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-05"}