Learning to Rank Chain-of-Thought: Using a Small Model

## Abstract

Abstract Large Language Models (LLMs) struggle with reliable mathematical reasoning, and current verification methods are often computationally expensive. This paper introduces the Energy Outcome Reward Model (EORM), a highly efficient, lightweight post-hoc verifier designed to address this challenge. EORM uses an energy-based framework to rank Chain-of-Thought (CoT) solutions, learning to distinguish correct from incorrect reasoning using only simple outcome labels, thus eliminating the need for expensive annotations. With only 55M parameters, over 127 times smaller than typical reward models, EORM boosts the accuracy of Llama 3 8B to 90.7% on GSM8k and 63.7% on MATH. This performance is achieved by efficiently selecting the optimal reasoning path from a pool of candidates, allowing it to match or exceed the accuracy of far more resource-intensive Best-of-N sampling techniques. Crucially, our experiments show that EORM generalizes effectively to out-of-distribution problems and unseen models, indicating it learns fundamental principles of valid reasoning. This robustness, combined with its efficiency, establishes EORM as a practical tool for deploying more dependable LLMs in complex, real-world applications. Our code is available: https://github.com/ericjiang18/EnergyORM/tree/main

## 1 Introduction

Mathematical reasoning remains a critical and challenging domain for Large Language Models (LLMs), demanding a high degree of logical consistency and multi-step inferential accuracy that often eludes current architectures (Guo et al., 2025; Tong et al., 2024). While Chain-of-Thought (CoT) (Wei et al., 2022; Kojima et al., 2022) has significantly advanced the ability of LLMs to articulate intermediate reasoning steps, thereby improving performance on complex tasks (Wang et al., 2023; Huang et al., 2022; Yao et al., 2022), it offers no intrinsic guarantee of correctness. This leads to a fundamental challenge: among the multiple Chain-of-Thought produced by the LLMs, how can we reliably identify the single most accurate one? This selection problem, widely referred to as the Best-of-N challenge, has become a key bottleneck in advancing the reliability of mathematical reasoning with LLMs (Tong et al., 2024; Xiong et al., 2024; Press et al., 2022).

To solve the Best-of-N problem, researchers have tried many approaches, and currently, existing approaches fall into two categories: (1) majority voting and (2) post-training with reinforcement learning (RL) to train a reward model for answer selection. Majority voting (Toh et al., 2025; Wang et al., 2023) is the simplest method, it chooses the answer most frequently generated by the model. While this method is easy to implement, it tends to reflect the model’s biases rather than identifying the truly correct answer. The RL-based approaches are more sophisticated, with three prominent variants: (1) Preference Optimization (PO), (2) Outcome Reward Models (ORM), and (3) Process Reward Models (PRM). Preference Optimization (Pang et al., 2024; Zhang et al., 2024) uses pairwise comparisons between correct and incorrect answers to train a reward model. Although it can improve answer quality, it requires training a new model for each task and struggles with scenarios involving multiple complex outputs. Outcome Reward Models (Lyu et al., 2025; Hosseini et al., 2024; Cobbe et al., 2021b) are similar but allow supervision from multiple positive and negative examples, making them more flexible than pairwise-only methods. Process Reward Models (She et al., 2025; Wang et al., 2025) go one step further by assigning rewards to each step in the reasoning process. However, this requires extensive annotation of every Chain-of-Thought (CoT) trace, making data preparation labor-intensive. We provide a more in-depth comparison of the limitations of the different types of reward model in [Table˜1](https://arxiv.org/html/2505.14999v3#S1.T1).

This motivates the need for a more efficient and generalizable alternative. Our approach begins with the observation that model outputs are sequences of tokens, which allows us to tokenize these responses and analyze their statistical patterns. Each output can be interpreted as a sample from an underlying probability distribution. Leveraging the Universal Approximation Theorem (Liu et al., 2025; Kratsios, 2021), we recognize that a sufficiently expressive neural network can, in principle, learn to distinguish between these distributions. Building on this insight, we propose training a lightweight neural network model to classify responses as good or bad, effectively select the best answer using Best-of-N, acting like a “LLM-as-a-Judge”. Crucially, because this model learns from token-level patterns rather than task-specific features, it is capable of generalizing across outputs from different base LLMs. To further enhance our neural network ability to capture complex distributions and support robust generalization, we adopt an Energy-Based Model (EBM) architecture (Deng et al., 2020b; Du & Mordatch, 2020). EBMs have demonstrated strong performance in related tasks, utilizing their uniqueness of using Energy Probabilistic Landscape, and in our experiments, our EBM-based judge consistently outperforms both majority voting and traditional Outcome Reward Models in identifying the best answer.

Table: Table 1: EORM’s feature benefit compared to other reasoning baselines. We compare EORM with ORM (Outcome Reward Model), PRM (Process Reward Model), and PL (Preference Learning) to demonstrate the feature benefit of our method over these baselines.

In this paper, we introduce the Energy Outcome Reward Model (EORM), an efficient post-hoc verifier for CoT outputs. EORM leverages principles from Energy-Based Models (EBMs) (Du & Mordatch, 2019; Liu et al., 2020) to effectively rerank and select the best candidate solutions. Our methodology trains a lightweight model to assign a scalar energy score to each candidate, learning to distinguish correct from incorrect reasoning using only simple binary outcome labels. Critically, this approach allows us to drastically reduce the model size from the typical 7B to 8B parameters of reward models to just 55M, achieving a greater than 127x reduction in parameter count while maintaining high performance.

Our main contributions are threefold:

- •
A Novel Small and Efficient Energy Reward Model for CoT Verification. We propose Eorm, a lightweight and efficient verifier that assigns a scalar energy score to each CoT solution. By training on simple binary outcome labels (correct/incorrect), Eorm eliminates the need for costly step-by-step annotations or preference pairs required by Process and Preference-based Reward Models.
- •
State-of-the-Art Performance on Math Reasoning. We demonstrate that Eorm significantly boosts the performance of various open-source LLMs on challenging mathematical reasoning benchmarks, including GSM8k and MATH. Our approach consistently achieves superior accuracy by effectively reranking candidate solutions, often matching or exceeding the performance of more computationally intensive methods.
- •
Demonstrated Robust Generalization. We empirically validate that Eorm generalizes effectively to both out-of-distribution datasets and unseen LLM architectures. This highlights its ability to learn underlying principles of logical reasoning rather than overfitting to the stylistic patterns of the models in its training set.

The remainder of this paper is organized as follows. We first review related work in [Section˜2](https://arxiv.org/html/2505.14999v3#S2). Next, we detail the Eorm architecture and training methodology in [Section˜3](https://arxiv.org/html/2505.14999v3#S3). We then present our comprehensive experimental results and analysis in [Section˜4](https://arxiv.org/html/2505.14999v3#S4) and abaltions in [Section˜5](https://arxiv.org/html/2505.14999v3#S5). Finally, we conclude the paper in [Section˜6](https://arxiv.org/html/2505.14999v3#S6).

Figure: Figure 1: An overview of flow chart of EORM. In the EORM process, the model tokenizes the question-answer pair, then computes an energy score using an Energy-Based Model (EBM). The Bradley-Terry loss serves as the objective for reward-based fine-tuning. During deployment, the trained energy reward model computes energy scores for classification tasks.
Refer to caption: figures/flow_new.png

## 2 Related Work

Our work intersects with advancements in Chain-of-Thought (CoT) reasoning, the verification and reranking of Large Language Model (LLM) outputs, and Energy-Based Models (EBMs). CoT reasoning (Wei et al., 2022; Kojima et al., 2022) has significantly improved multi-step reasoning in LLMs (Bai et al., 2023; Touvron et al., 2023; OpenAI, 2023; Guo et al., 2025), with further refinements like Least-to-Most prompting (Zhou et al., 2022) and Tree-of-Thoughts (Yao et al., 2023; Zhang et al., 2023). However, the fallibility of CoT outputs (Tong et al., 2024), where a single error can invalidate a solution, necessitates robust verification. Common approaches like self-consistency (Wang et al., 2023) improve reliability but incur substantial computational costs by sampling numerous solutions (Wu et al., 2024). To address this, various reranking and verification techniques have emerged, including training separate verifier models (Cobbe et al., 2023; Khalifa et al., 2023; Li et al., 2023) or adopting learning-to-rank perspectives (Liu, 2009; Deng et al., 2020a). These methods, while effective, often introduce their own complexities or computational demands. Our approach, EORM, draws inspiration from EBMs (Du & Mordatch, 2019), which assign scalar energy scores to configurations and are well-suited for ranking (Grathwohl et al., 2019; Liu, 2009). Specifically, we leverage the insight that classifier logits can be interpreted as negative energies (Grathwohl et al., 2019; Liu et al., 2020). By training a lightweight EBM with a pairwise Bradley-Terry loss (Liu, 2009) on outcome labels, EORM efficiently reranks CoT candidates, offering a distinct, streamlined post-hoc verification mechanism that complements retrieval-augmented methods (Rakin et al., 2024; Schick et al., 2023; Yao et al., 2022; Press et al., 2022; Khattab et al., 2022) and other specialized verifiers like Chain-of-Actions (Pan et al., 2025) or Search-in-the-Chain (Xu et al., 2023). For a more detailed discussion of related literature, please refer to [Appendix˜B](https://arxiv.org/html/2505.14999v3#A2).

## 3 Methodology

In this section, we introduce the architectural design and technical foundations of our proposed Energy Outcome Reward Model (Eorm), a lightweight verifier for Chain-of-Thought (CoT) reasoning. We begin by outlining the principles of Energy-Based Models (EBMs), which form the foundation of our approach. We then detail the specific architecture and training objective of Eorm, showing how it learns to rank CoT candidates effectively using only simple outcome-based supervision.

### 3.1 Preliminaries: Energy-Based Models for Ranking

Evaluating CoT reasoning requires a flexible mechanism that can distinguish high-quality reasoning from incorrect or incoherent paths. Instead of treating this as a classification task, we model it as a preference-ranking problem using Energy-Based Models (EBMs).

Energy-Based Models (EBMs) provide a flexible approach to modeling distributions by assigning a scalar energy $E_{\theta}(y)$ to each configuration $y$ from a space $\mathcal{Y}$. This energy function is parameterized by $\theta$. Lower energy typically corresponds to more desirable or probable configurations. In this work, $\mathcal{Y}$ is the space of possible Chain-of-Thought text sequences $y$, and $\theta$ represents the learnable parameters of our Eorm model.

Given an energy function $E_{\theta}(y)$, the corresponding Boltzmann (or Gibbs) distribution $p_{\theta}(y)$ assigns a probability to each configuration $y\in\mathcal{Y}$ as:

$$ $p_{\theta}(y)\;=\;\frac{\exp\bigl(-E_{\theta}(y)\bigr)}{Z_{\theta}},$ (1) $$

where $Z_{\theta}$ is the partition function, a normalization constant that ensures $p_{\theta}(y)$ sums to unity:

$$ $Z_{\theta}\;=\;\sum_{y^{\prime}\in\mathcal{Y}}\exp\bigl(-E_{\theta}(y^{\prime})\bigr)\quad(\text{for discrete }\mathcal{Y}).$ (2) $$

A key challenge in working with EBMs is that computing the partition function $Z_{\theta}$ is often computationally intractable. However, for tasks involving ranking or selecting the best candidate from a finite set $\mathcal{Y}_{\mathrm{cand}}\subset\mathcal{Y}$, the explicit computation of $Z_{\theta}$ is unnecessary. Since $Z_{\theta}$ is constant for all $y\in\mathcal{Y}_{\mathrm{cand}}$, comparing probabilities $p_{\theta}(y)$ is equivalent to comparing the unnormalized scores $\exp(-E_{\theta}(y))$, which in turn relates directly to comparing the energies $E_{\theta}(y)$. This equivalence implies that energy minimization provides a direct mechanism for identifying the most probable (preferred) solution within a candidate set, without needing to compute $Z_{\theta}$:

$$ $y^{*}\;=\;\arg\min_{y\in\mathcal{Y}_{\mathrm{cand}}}E_{\theta}(y)\;=\;\arg\max_{y\in\mathcal{Y}_{\mathrm{cand}}}p_{\theta}(y).$ (3) $$

This makes EBMs well-suited for our task of reranking CoT solutions based on learned preferences encoded in the energy function $E_{\theta}(y)$.

### 3.2 Eorm : Architecture and Training Objective

Building on the EBM framework, Eorm is designed as a lightweight yet powerful verifier for CoT reasoning. The core idea is to train a small neural network to learn an energy function $E_{\theta}(y)$ that maps any given CoT solution $y$ to a scalar energy score. By design, this function assigns low energy to correct reasoning paths and high energy to incorrect ones, enabling effective reranking of candidate solutions.

#### Architecture.

To implement the energy function $E_{\theta}(y)$, Eorm uses a lightweight Transformer encoder. A given CoT solution $y$ is first tokenized, and a special classification token (e.g., [CLS]) is prepended. The sequence is then passed through the Transformer encoder. The final hidden state corresponding to the [CLS] token, denoted $\mathbf{h}_{\text{CLS}}$, serves as a holistic representation of the entire reasoning path. This representation is fed into a simple energy head—a Multi-Layer Perceptron (MLP) with Layer Normalization—which projects it to the final scalar energy value:

$$ $E_{\theta}(y)\;=\;\text{MLP}\!\Bigl(\text{LayerNorm}\bigl(\mathbf{h}_{\text{CLS}}\bigr)\Bigr)\,\in\,\mathbb{R}.$ (4) $$

This scalar output $E_{\theta}(y)$ represents the energy assigned to the sequence $y$. Lower values indicate higher assessed quality or correctness.

#### Training Objective.

Eorm is trained to distinguish between correct and incorrect solutions using a simple yet effective pairwise ranking loss. This approach only requires binary outcome labels for each CoT, avoiding the need for expensive, fine-grained annotations required by process-based reward models. For a given problem, we collect a set of generated CoT solutions $\mathcal{Y}_{n}$ and partition it into a subset of correct solutions, $\mathcal{Y}_{+}$, and a subset of incorrect solutions, $\mathcal{Y}_{-}$.

$$ $\mathcal{Y}_{+}\;=\;\bigl\{\,y\in\mathcal{Y}_{n}\,\mid\,l(y)=1\bigr\},\quad\mathcal{Y}_{-}\;=\;\bigl\{\,y\in\mathcal{Y}_{n}\,\mid\,l(y)=0\bigr\}.$ (5) $$

The model’s parameters $\theta$ are then optimized to ensure that for any pair of solutions $(y_{+},y_{-})$ where $y_{+}\in\mathcal{Y}_{+}$ and $y_{-}\in\mathcal{Y}_{-}$, the energy of the correct solution is lower than that of the incorrect one. We formalize this using the Bradley-Terry loss, which is summed over all possible pairs within the group:

$$ $\mathcal{L}(\theta;\mathcal{Y}_{n})\;=\;\frac{1}{|\mathcal{Y}_{+}||\mathcal{Y}_{-}|}\sum_{y_{+}\in\mathcal{Y}_{+}}\sum_{y_{-}\in\mathcal{Y}_{-}}\log\Bigl(1+\exp\bigl(E_{\theta}(y_{+})-E_{\theta}(y_{-})\bigr)\Bigr).$ (6) $$

Minimizing this loss encourages a clear separation in the energy landscape, pushing the model to assign consistently lower energy scores to correct solutions. This pairwise formulation provides a strong learning signal, particularly beneficial for CoT ranking where subtle differences can exist between multiple flawed or correct reasoning paths. During training, the model parameters $\theta$ are updated by iterating through the dataset and minimizing this loss via gradient-based optimization. For a more detailed theoretical analysis, please refer to Appendix [C](https://arxiv.org/html/2505.14999v3#A3).

Table: Table 2: Performance comparison on math reasoning benchmarks. We evaluate EORM on GSM8K and MATH using five LLM structures (Mistral, DeepSeekMath, LLaMA3, Qwen2.5, LLaMA2), measuring accuracy. EORM achieves the highest performance across models, with best accuracy values highlighted in bold.

## 4 Experiments

Figure: Figure 2: A comparison of the parameter sizes between a standard reward model and our EORM Model. A typical reward model has approximately 7 billion parameters, while EORM has only 55 million, demonstrating a size reduction of over 127 times and highlighting EORM’s efficiency.
Refer to caption: figures/size.png

In this section, we empirically evaluate our proposed Energy Outcome Reward Model (Eorm). A key advantage of Eorm, as highlighted in our introduction, is its exceptional efficiency; with only 55M parameters, it is over 127 times smaller than typical 7B reward models, as shown in [Figure˜2](https://arxiv.org/html/2505.14999v3#S4.F2). Our evaluation is structured into three main parts. (1) First, in [Section˜4.1](https://arxiv.org/html/2505.14999v3#S4.SS1), we assess its performance on in-distribution mathematical reasoning tasks. (2) Second, in [Section˜4.2](https://arxiv.org/html/2505.14999v3#S4.SS2), we test its robustness on out-of-distribution benchmarks. (3) Finally, in [Section˜4.3](https://arxiv.org/html/2505.14999v3#S4.SS3), we analyze its ability to generalize to outputs from unseen LLM architectures. For our experiments, we use the GSM8k (Cobbe et al., 2021a) and MATH (Hendrycks et al., 2021) datasets for in-distribution tasks, and AIME 2024, AMC, and AGIEval’s SAT Math and Gaokao Math subsets (Zhong et al., 2023) for out-of-distribution evaluation. We applied EORM to several open-source Large Language Models (LLMs): Mistral-7B-v0.1 (Jiang et al., 2023), DeepSeekMath-7B (Shao et al., 2024), Llama 3 8B (Grattafiori et al., 2024), Qwen 2.5 7B (Yang et al., 2024), and Llama 2 7B (Touvron et al., 2023).

EORM was trained using a pairwise Bradley-Terry loss function (Liu, 2009). The training dataset was constructed from Chain-of-Thought (CoT) solutions for problems in the in-domain GSM8k and MATH training splits. These solutions were generated using all five aforementioned LLMs. For each training problem, we generated $n=256$ CoT candidates with a temperature of $0.7$ and a sampling probability of $0.9$, ensuring all attempts were included, regardless of their correctness. Each training instance comprised the original question, a generated solution, and a label indicating whether the solution was correct ($y^{+}$) or incorrect ($y^{-}$). EORM learned from these pairs to assign lower energy scores to preferred solutions by interpreting classifier logits as negative energies (Grathwohl et al., 2019; Liu et al., 2020). The GPT-2 tokenizer (Radford et al., 2019) was employed for the energy-based tokenizer.

For evaluation, we generated Chain-of-Thought (CoT) candidates across all models using a temperature of $0.7$ and a sampling probability of $0.9$. Specifically, $n=256$ candidates were generated for each in-distribution problem, while $n=64$ candidates were generated for each out-of-distribution problem. Subsequently, EORM was used post-hoc to select the candidate with the lowest energy. The final answer accuracy is reported in the two subsections that follow.

Figure: Figure 3: EORM performance with varying samples per question. We conduct experiments to show how the number of samples influences the problem-solving rate, using accuracy as the metric. The results indicate that model performance improves as the number of samples increases.
Refer to caption: figures/orm_gsm8k.png

### 4.1 In Distribution Learning

Our primary evaluation focused on EORM’s performance within the domains it was trained on, namely the GSM8k and MATH test sets. The quantitative outcomes, detailed in [Table˜2](https://arxiv.org/html/2505.14999v3#S3.T2), consistently show that employing EORM as a post-hoc reranker leads to substantial improvements in final answer accuracy compared to the baseline performance of the underlying LLMs generating the candidate solutions. By effectively identifying and selecting the most plausible reasoning path from the generated set, EORM significantly enhances problem-solving capabilities. For instance, when integrated with Llama 3 8B and reranking $n=256$ candidate solutions, EORM achieved high accuracy levels of 90.7% on GSM8k and 63.7% on MATH, demonstrating its ability to leverage multiple reasoning attempts effectively. The relationship between the number of candidate samples ($n$) and the resulting accuracy improvement is visualized in [Figure˜3](https://arxiv.org/html/2505.14999v3#S4.F3), illustrating how performance generally scales with more samples but strong gains are often achievable with moderate sampling budgets. Beyond quantitative accuracy, qualitative analyses provide deeper insights into EORM’s function.

### 4.2 Out of Distribution Learning

A critical aspect of evaluating any reasoning model is its ability to generalize to new, unseen problems and potentially different reasoning styles. To rigorously test this, we assessed the performance of EORM (trained exclusively on GSM8k and MATH data) on a suite of OOD benchmarks known for their difficulty: AIME 2024, AMC, and AGIEval’s SAT Math and Gaokao Math subsets (Zhong et al., 2023). These tasks often require more complex mathematical insights or different problem-solving strategies than those predominant in the training datasets. For this phase, evaluations primarily utilized Llama-3 8B and Qwen-2.5 7B as the base models, generating $n=64$ candidate solutions for each OOD problem instance. The results, documented in [Table˜5](https://arxiv.org/html/2505.14999v3#S4.T5), demonstrate EORM’s strong generalization capabilities. When applied to the Llama-3 base model, EORM generally yielded superior performance compared to alternative baseline reranking methods like TTRL (Zuo et al., 2025) and MathWizard (Luo et al., 2025) across the majority of the evaluated OOD datasets under the same experimental setup. Impressively, EORM achieved significant scores on highly challenging benchmarks such as AIME 2024 (reaching 10.0%) and the AGIE Gaokao Math problems (achieving 70.3%). Additionally the Qwen-2.5 base model also find similar finding, and provides further evidence that EORM is not merely overfitting to patterns specific to GSM8k and MATH. Instead, it appears to learn more fundamental, transferable principles related to the structure and validity of mathematical reasoning, allowing it to effectively discern higher-quality solutions even when faced with novel problem types and domains.

Table: Table 3: Comparison of EORM with other reasoning methods. We evaluate EORM against two reasoning baselines, TTRL and MathWizard, using accuracy as the metric across four mathematical datasets. Each method uses 64 samples per question for a fair comparison. EORM consistently achieves the highest performance, with the best Accuracy values highlighted in bold.

### 4.3 Generalization Capabilities

To further assess Eorm’s robustness, we analyze its generalization capabilities when encountering outputs from LLMs not seen during training. This is crucial for real-world applicability where new models are constantly emerging. We test two scenarios: generalization to unseen models on in-distribution tasks, and a more challenging test of generalization to unseen models on out-of-distribution tasks.

#### Generalization to Unseen Models on In-Distribution Tasks.

We employ a leave-one-out training strategy where a variant of Eorm, termed “Eorm Generalize,” is trained on CoT data from four of the five LLMs and tested on the held-out model. The results, detailed in [Table˜4](https://arxiv.org/html/2505.14999v3#S4.T4), show that this variant consistently improves accuracy over the base performance. For instance, when Llama 3 8B is the held-out model, its base accuracy on GSM8k is 42.9%. “Eorm Generalize” elevates this to 76.6%, while the standard Eorm (trained on all models) reaches 90.7%. The significant uplift from “Eorm Generalize” demonstrates that our model learns generalizable features of correct reasoning, rather than merely overfitting to the stylistic patterns of the models in its training set. The performance gap between the standard Eorm and “Eorm Generalize” is expected, as the latter lacks exposure to the target LLM’s specific error patterns and stylistic nuances. However, the crucial finding is that “Eorm Generalize” still provides a substantial boost, indicating that it captures fundamental principles of logical consistency and procedural correctness. This ability to transcend the idiosyncrasies of individual LLMs confirms that Eorm is learning the underlying structure of valid mathematical arguments.

Table: Table 4: Performance on in-distribution benchmarks (GSM8k, MATH) when EORM is trained without exposure to the target model’s outputs. EORM Generalize is trained on CoT data from four LLMs and tested on the fifth (target) LLM. EORM (from Table [2](https://arxiv.org/html/2505.14999v3#S3.T2)) is trained on data from all five LLMs. Base performance of the target LLM is also provided. Accuracy is reported.

#### Generalization to Unseen Models and Tasks (Out-of-Distribution).

We test a more demanding scenario where the “Eorm Generalize” variant is evaluated on challenging OOD benchmarks using CoT solutions from the held-out LLM. This setup assesses Eorm’s ability to transfer learned principles to both novel tasks and unfamiliar model outputs simultaneously. The results in [Table˜5](https://arxiv.org/html/2505.14999v3#S4.T5) show that even under this compounded challenge, our model provides tangible improvements. For example, with Llama 3 8B as the held-out model, “Eorm Generalize” raises the average OOD accuracy from a base of 37.2% to 40.5%. This suggests Eorm internalizes fundamental and transferable heuristics about the structure and coherence of valid reasoning that apply broadly. As anticipated, the standard Eorm—which benefited from seeing the target model’s in-distribution outputs during training—achieves superior OOD performance. Nevertheless, the fact that “Eorm Generalize” still delivers tangible improvements in this dual-unseen scenario is a strong testament to its robustness. This finding highlights that the energy functions learned by Eorm are not merely memorizing solution patterns but are instead capturing more abstract, transferable markers of high-quality reasoning applicable across diverse problem domains.

Table: Table 5: Comparison of EORM with against baseline for out-of distribution problem and answer. We gray shaded the generalized result.

## 5 Ablation Studies

To validate the contributions of key components within the EORM framework, we conducted a series of ablation studies, with results summarized in Figure [4](https://arxiv.org/html/2505.14999v3#S5.F4). We investigated two primary aspects: the choice of model architecture and the impact of the tokenizer. We first assessed the importance of the Transformer architecture by comparing our standard EORM against a simpler baseline where the Transformer encoder was replaced with a standard Multi-Layer Perceptron (MLP) verifier. As shown in Figure [4](https://arxiv.org/html/2505.14999v3#S5.F4) (left), the Transformer-based EORM significantly outperforms the MLP variant, achieving accuracies of 90.7% on GSM8k and 63.7% on MATH, compared to the MLP’s 82.6% and 52.1%, respectively. This highlights that the Transformer’s ability to model sequential dependencies and long-range relationships is crucial for effectively capturing the logical structure inherent in Chain-of-Thought reasoning paths. Next, we examined EORM’s sensitivity to the tokenizer by training it with two different tokenizers: a universal GPT-2 tokenizer and the native tokenizer from the Llama 3 model family. The results, presented in Figure [4](https://arxiv.org/html/2505.14999v3#S5.F4) (right), reveal that the choice of tokenizer has a negligible impact on performance. The model achieves nearly identical scores on both GSM8k (86.6% vs. 86.8%) and MATH (61.8% for both). This robustness suggests that EORM learns fundamental, transferable features of valid reasoning that are not dependent on specific tokenization schemes, underscoring its generalizability.

Figure: Figure 4: Ablation studies on key components of EORM. (Left) Performance comparison between the Transformer-based EORM and a simpler MLP verifier on the GSM8k and MATH benchmarks. (Right) Impact of using a universal (GPT-2) versus a native (Llama 3) tokenizer. The results highlight the critical role of the Transformer architecture and demonstrate the model’s robustness to the choice of tokenizer.
Refer to caption: figures/ablation.png

## 6 Conclusion

We introduced EORM, a lightweight energy-based model for post-hoc verification of Chain-of-Thought reasoning. Trained solely on outcome labels, EORM efficiently reranks candidate solutions to significantly boost mathematical reasoning performance across various benchmarks. Its effectiveness, combined with a parameter count over 127x smaller than typical reward models, presents a practical path toward more dependable and accessible LLMs for complex reasoning.

## References

- Bai et al. (2023)
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.
Qwen technical report.
*arXiv preprint arXiv:2309.16609*, 2023.
- Cobbe et al. (2021a)
Karl Cobbe, Jacob Hilton, and John Schulman.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021a.
- Cobbe et al. (2021b)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
Training verifiers to solve math word problems, 2021b.
- Cobbe et al. (2023)
Karl Cobbe, Gregory Druck, Anand Kirubarajan, , et al.
Improving mathematical reasoning with process supervision.
2023.
- Deng et al. (2020a)
Yunfu Deng, Qian Liu, Xiaodong He, Zihan Chen, and Hongchao Dai.
Residual energy-based models for text generation.
In *International Conference on Learning Representations (ICLR)*, 2020a.
- Deng et al. (2020b)
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc’Aurelio Ranzato.
Residual energy-based models for text generation, 2020b.
- Du & Mordatch (2019)
Yilun Du and Igor Mordatch.
Implicit generation and modeling with energy-based models.
*Advances in Neural Information Processing Systems*, 32, 2019.
- Du & Mordatch (2020)
Yilun Du and Igor Mordatch.
Implicit generation and generalization in energy-based models, 2020.
- Grathwohl et al. (2019)
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky.
Your classifier is secretly an energy based model and you should treat it like one.
*arXiv preprint arXiv:1912.03263*, 2019.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Hendrycks et al. (2021)
Dan Hendrycks, Colin Burns, Spencer Chen, , et al.
Measuring mathematical problem solving with the MATH dataset.
In *Advances in Neural Information Processing Systems (NeurIPS)*, volume 34, pp. 3512–3524, 2021.
- Hosseini et al. (2024)
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal.
V-star: Training verifiers for self-taught reasoners, 2024.
- Huang et al. (2022)
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch.
Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.
In *International Conference on Machine Learning*, pp. 9118–9147. PMLR, 2022.
- Huang et al. (2024)
Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen.
Key-point-driven data synthesis with its enhancement on mathematical reasoning, 2024.
- Jacovi et al. (2024)
Alon Jacovi, Swaroop Mishra, Giovanni Napolitano, , et al.
Reveal: A benchmark for step-level reasoning verification.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL)*, pp. 10042–10063, Bangkok, Thailand, 2024.
- Jiang et al. (2023)
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.
Mistral 7b, 2023.
- Jiang et al. (2024)
Eric Hanchen Jiang, Zhi Zhang, Dinghuai Zhang, Andrew Lizarraga, Chenheng Xu, Yasi Zhang, Siyan Zhao, Zhengjie Xu, Peiyu Yu, Yuer Tang, Deqian Kong, and Ying Nian Wu.
Dodt: Enhanced online decision transformer learning through dreamer’s actor-critic trajectory forecasting, 2024.
URL [https://arxiv.org/abs/2410.11359](https://arxiv.org/abs/2410.11359).
- Karakida et al. (2016)
Ryo Karakida, Masato Okada, and Shun-ichi Amari.
Dynamical analysis of contrastive divergence learning: Restricted boltzmann machines with gaussian visible units.
*Neural Networks*, 79:78–87, 2016.
- Khalifa et al. (2023)
Nadir Khalifa, Abdullah Alnasrallah, Joke Demuynck, , et al.
GRACE: Discriminator-guided chain-of-thought reasoning.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pp. 11710–11725, Singapore, 2023.
- Khattab et al. (2022)
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia.
Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive NLP.
*arXiv preprint arXiv:2212.14024*, 2022.
- Kojima et al. (2022)
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.
Large language models are zero-shot reasoners.
*Advances in neural information processing systems*, 35:22199–22213, 2022.
- Kratsios (2021)
Anastasis Kratsios.
The universal approximation property: Characterization, construction, representation, and existence.
*Annals of Mathematics and Artificial Intelligence*, 89(5–6):435–469, January 2021.
ISSN 1573-7470.
URL [http://dx.doi.org/10.1007/s10472-020-09723-1](http://dx.doi.org/10.1007/s10472-020-09723-1).
- LeCun et al. (2006)
Yann LeCun, Sumit Chopra, and Raia Hadsell.
A tutorial on energy-based learning.
In *Predicting Structured Data*, pp. 1–59. MIT Press, 2006.
- Li et al. (2024a)
Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng.
Common 7b language models already possess strong math capabilities, 2024a.
- Li et al. (2024b)
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al.
Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.
*Hugging Face repository*, 13(9):9, 2024b.
- Li et al. (2023)
Shuohang Li, Liangming Pan, Jinpeng Zhang, , et al.
Diverse: Step-aware verifier for chain-of-thought reasoning.
*arXiv preprint arXiv:2306.04637*, 2023.
- Lin et al. (2025)
Zongyu Lin, Yao Tang, Xingcheng Yao, Da Yin, Ziniu Hu, Yizhou Sun, and Kai-Wei Chang.
Qlass: Boosting language agent inference via q-guided stepwise search.
*arXiv preprint arXiv:2502.02584*, 2025.
- Liu et al. (2025)
Hude Liu, Jerry Yao-Chieh Hu, Zhao Song, and Han Liu.
Attention mechanism, max-affine partition, and universal approximation, 2025.
URL [https://arxiv.org/abs/2504.19901](https://arxiv.org/abs/2504.19901).
- Liu (2009)
Tie-Yan Liu.
Learning to rank for information retrieval.
In *Foundations and Trends® in Information Retrieval*, volume 3, pp. 225–331. Now Publishers Inc., 2009.
- Liu et al. (2020)
Weitang Liu, Xiaoyun Wang, John D Owens, and Yixuan Li.
Energy-based out-of-distribution detection.
*Advances in Neural Information Processing Systems*, 33:21464–21475, 2020.
- Liu et al. (2023)
Xiang Liu, Baolin Peng, Yichong Zhang, , et al.
MMIQC: Multi-modal in-context learning for math reasoning.
*arXiv preprint arXiv:2407.13690*, 2023.
- Luo et al. (2025)
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-Guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang.
Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Lyu et al. (2025)
Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al.
Exploring the limit of outcome reward for learning mathematical reasoning.
*arXiv preprint arXiv:2502.06781*, 2025.
- OpenAI (2023)
OpenAI.
Gpt-4 technical report, 2023.
- Pan et al. (2025)
Zhenyu Pan, Haozheng Luo, Manling Li, and Han Liu.
Chain-of-action: Faithful and multimodal question answering through large language models.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Pang et al. (2024)
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston.
Iterative reasoning preference optimization.
In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), *Advances in Neural Information Processing Systems*, volume 37, pp. 116617–116637. Curran Associates, Inc., 2024.
- Press et al. (2022)
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis.
Measuring and narrowing the compositionality gap in language models.
*arXiv preprint arXiv:2210.03350*, 2022.
- Radford et al. (2019)
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.
Language models are unsupervised multitask learners.
*OpenAI blog*, 1(8):9, 2019.
- Rakin et al. (2024)
Salman Rakin, Md AR Shibly, Zahin M Hossain, Zeeshan Khan, and Md Mostofa Akbar.
Leveraging the domain adaptation of retrieval augmented generation models for question answering and reducing hallucination.
*arXiv preprint arXiv:2410.17783*, 2024.
- Schick et al. (2023)
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
Toolformer: Language models can teach themselves to use tools.
*arXiv preprint arXiv:2302.04761*, 2023.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- She et al. (2025)
Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang, and Shujian Huang.
R-prm: Reasoning-driven process reward modeling, 2025.
- Song et al. (2021)
Lin Song, Kuan Zhao, Yilun Du, and Jun Qi.
Trainable energy-based models for solvable lattice models.
*arXiv preprint arXiv:2101.03288*, 2021.
- Tang et al. (2024)
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei.
Mathscale: Scaling instruction tuning for mathematical reasoning, 2024.
- Team (2024)
Llama Team.
The llama 3 herd of models, 2024.
URL [https://arxiv.org/abs/2407.21783](https://arxiv.org/abs/2407.21783).
- Toh et al. (2025)
Vernon Toh, Deepanway Ghosal, and Soujanya Poria.
Not all votes count! translated program for verification improves self-consistency of language models for math reasoning.
In *2nd AI for Math Workshop @ ICML 2025*, 2025.
- Tong et al. (2024)
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He.
DART-math: Difficulty-aware rejection tuning for mathematical problem-solving.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
URL [https://openreview.net/forum?id=zLU21oQjD5](https://openreview.net/forum?id=zLU21oQjD5).
- Touvron et al. (2023)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
*arXiv preprint arXiv:2307.09288*, 2023.
- Vaswani et al. (2017)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In *Advances in neural information processing systems*, pp. 5998–6008, 2017.
- Wang et al. (2024)
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui.
Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 9426–9439, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.510.
- Wang et al. (2025)
Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al.
Visualprm: An effective process reward model for multimodal reasoning.
*arXiv preprint arXiv:2503.10291*, 2025.
- Wang et al. (2023)
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.
Self-consistency improves chain of thought reasoning in language models.
In *The Eleventh International Conference on Learning Representations*, 2023.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou.
Chain of thought prompting elicits reasoning in large language models.
In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), *Advances in Neural Information Processing Systems*, 2022.
- Wu et al. (2024)
Haoran Wu, Avishkar Balika Singh, McKay Andrus, Bohyung Yang, Abdelrahman Mourad, , et al.
Mitigating misleading chain-of-thought reasoning with selective filtering.
*arXiv preprint arXiv:2403.19167*, 2024.
- Xiong et al. (2024)
Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri.
Large language models can learn temporal reasoning.
*arXiv preprint arXiv:2401.06853*, 2024.
- Xu et al. (2023)
Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-seng Chua.
Search-in-the-chain: Towards the accurate, credible and traceable content generation for complex knowledge-intensive tasks.
*arXiv preprint arXiv:2304.14732*, 2023.
- Yang et al. (2024)
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al.
Qwen2. 5 technical report.
*arXiv preprint arXiv:2412.15115*, 2024.
- Yao et al. (2022)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
*arXiv preprint arXiv:2210.03629*, 2022.
- Yao et al. (2023)
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan.
Tree of thoughts: Deliberate problem solving with large language models.
*arXiv preprint arXiv:2305.10601*, 2023.
- Yu et al. (2024)
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.
Metamath: Bootstrap your own mathematical questions for large language models.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Yue et al. (2024)
Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen.
MAmmoTH2: Scaling instructions from the web.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
- Zhang et al. (2024)
Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin.
Chain of preference optimization: Improving chain-of-thought reasoning in llms.
*Advances in Neural Information Processing Systems*, 37:333–356, 2024.
- Zhang et al. (2023)
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola.
Automatic chain of thought prompting in large language models.
In *The Eleventh International Conference on Learning Representations (ICLR 2023)*, 2023.
- Zhong et al. (2023)
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan.
Agieval: A human-centric benchmark for evaluating foundation models.
*arXiv preprint arXiv:2304.06364*, 2023.
- Zhou et al. (2022)
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al.
Least-to-most prompting enables complex reasoning in large language models.
*arXiv preprint arXiv:2205.10625*, 2022.
- Zhou et al. (2024)
Kun Zhou, Beichen Zhang, jiapeng wang, Zhipeng Chen, Xin Zhao, Jing Sha, Zhichao Sheng, Shijin Wang, and Ji-Rong Wen.
Jiuzhang3.0: Efficiently improving mathematical reasoning by training small data synthesis models.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
- Zuo et al. (2025)
Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou.
Ttrl: Test-time reinforcement learning, 2025.
URL [https://arxiv.org/abs/2504.16084](https://arxiv.org/abs/2504.16084).

## Appendix A Algorithm

Figure: Algorithm 1 Training Eorm using Pairwise Bradley-Terry Loss

## Appendix B Extended Related Work

This section provides a more detailed discussion of research areas related to our work on the Energy Outcome Reward Model (EORM), covering Chain-of-Thought reasoning, techniques for verifying and reranking LLM outputs, and Energy-Based Models.

### B.1 Chain-of-Thought and Multi-Step Reasoning

Large Language Models (LLMs) (Bai et al., 2023; Touvron et al., 2023; OpenAI, 2023; Guo et al., 2025) have demonstrated remarkable capabilities, particularly when prompted to generate step-by-step solutions. Chain-of-Thought (CoT) prompting (Wei et al., 2022; Kojima et al., 2022) has become a cornerstone technique for eliciting complex multi-step reasoning from LLMs. This approach encourages models to "think aloud," breaking down problems into manageable intermediate steps. Various refinements to CoT have been proposed, such as Least-to-Most prompting (Zhou et al., 2022), which guides the model through progressively more complex steps, and Tree-of-Thoughts (ToT) (Yao et al., 2023; Zhang et al., 2023), which explores multiple reasoning paths in a structured manner. Retrieval-augmented CoT methods (Pan et al., 2025; Rakin et al., 2024; Schick et al., 2023; Yao et al., 2022; Press et al., 2022; Khattab et al., 2022) further enhance reasoning by allowing models to consult external knowledge sources or tools.
Despite these advancements, CoT outputs are not infallible; a single incorrect step can derail the entire reasoning process (Tong et al., 2024). This has led to the development of post-hoc processing techniques. Among the most prominent is self-consistency (Wang et al., 2023), which samples multiple CoT outputs and selects the final answer via majority vote. While effective, self-consistency and similar ensemble methods often require generating a large number of candidate solutions, leading to substantial computational overhead (Wu et al., 2024). Our work seeks to mitigate this cost by providing a more efficient mechanism for identifying high-quality CoT solutions.

### B.2 Reranking and Verification of LLM Outputs

Given the variability in the quality of CoT paths, a subsequent verification or reranking stage is often crucial for improving the final answer accuracy (Wu et al., 2024; Jacovi et al., 2024; Li et al., 2023; Lin et al., 2025). Researchers have explored various strategies for this purpose. Some approaches focus on training separate verifier models to score the correctness of generated solutions or even individual reasoning steps (Cobbe et al., 2023; Khalifa et al., 2023; Jiang et al., 2024). For example, DI-VeRSe (Li et al., 2023) trains a model to perform step-aware verification. Other methods adopt a learning-to-rank perspective (Liu, 2009; Deng et al., 2020a), training models to compare and order candidate solutions based on their perceived quality using pairwise or listwise objectives.
While these verification and reranking techniques can be effective, many existing methods either rely on large, specialized verifier models that add significant computational load or employ complex, computationally expensive decoding strategies. The goal of EORM is to provide a lightweight yet powerful verifier that can efficiently identify correct CoT solutions without imposing such heavy overhead. Techniques like Chain-of-Actions (Pan et al., 2025) and Search-in-the-Chain (Xu et al., 2023) also introduce verification but often focus on different aspects or involve more intricate integration with the generation process. EORM distinguishes itself by its post-hoc applicability and its foundation in energy-based principles for efficient scoring.

### B.3 Energy-Based Models (EBMs)

Energy-Based Models (EBMs) provide a flexible and powerful framework for modeling complex data distributions by assigning a scalar "energy" value to each input configuration (LeCun et al., 2006; Du & Mordatch, 2019). Lower energy values typically correspond to more plausible or desirable configurations. EBMs have found successful applications in diverse domains, including computer vision (Grathwohl et al., 2019), generative modeling (Song et al., 2021; Du & Mordatch, 2019), out-of-distribution detection (Liu et al., 2020), and ranking tasks (Grathwohl et al., 2019; Liu, 2009).
A key characteristic of EBMs is that they do not necessarily require explicit normalization of the probability distribution (i.e., computation of the partition function), which is often intractable for high-dimensional or complex output spaces (Karakida et al., 2016; lecun2006tutorial). This property makes EBMs particularly well-suited for ranking scenarios, where the primary goal is to compare the relative "goodness" of different candidates rather than to compute their absolute probabilities. This is directly applicable to CoT reranking, as we aim to identify the best solution among several alternatives.
A pivotal insight, leveraged by our work, is that the logits produced by standard discriminative classifiers can be interpreted as negative unnormalized log-probabilities, or effectively, as negative energies (Grathwohl et al., 2019). By adopting this perspective, an EBM can be trained using objectives similar to those used for classifiers, without the need for complex sampling procedures often associated with training generative EBMs. EORM builds on this by using a pairwise Bradley-Terry loss (Liu, 2009), a common technique in learning-to-rank, to train the energy function to distinguish between correct and incorrect CoT solutions. Our work thus extends the application of EBM principles to the domain of multi-step reasoning in LLMs, offering an efficient and theoretically grounded alternative to existing verification methods.

## Appendix C Theoretical Details

This appendix furnishes comprehensive definitions, rigorous proofs, and supplementary remarks that underpin the theoretical concepts and analyses presented throughout the main text of the paper.

### C.1 Preliminaries: Foundations of Energy-Based Models

We begin by establishing the fundamental concepts of Energy-Based Models (EBMs) that form the basis of our proposed Eorm.

###### Definition C.1 (Energy Function) .

An *energy function* $E_{\theta}:\mathcal{Y}\to\mathbb{R}$ is a function, parameterized by $\theta$, that assigns a scalar value $E_{\theta}(y)$ (its energy) to each possible configuration or input $y$ from a space $\mathcal{Y}$. By convention, lower energy values typically correspond to more desirable or probable configurations. In the Eorm framework, $\mathcal{Y}$ denotes the space of possible Chain-of-Thought (CoT) text sequences, which are characterized by their variable length and complex linguistic and logical structures.

###### Definition C.2 (Boltzmann Distribution) .

Given an energy function $E_{\theta}(y)$, the corresponding *Boltzmann (or Gibbs) distribution* $p_{\theta}(y)$ assigns a probability (or probability density for continuous $\mathcal{Y}$) to each configuration $y\in\mathcal{Y}$ as:

$$ $p_{\theta}(y)\;=\;\frac{\exp\bigl(-E_{\theta}(y)\bigr)}{Z_{\theta}},$ (7) $$

where $Z_{\theta}$ is the partition function.

###### Definition C.3 (Partition Function) .

The *partition function* $Z_{\theta}$ serves as a normalization constant, ensuring that the Boltzmann distribution $p_{\theta}(y)$ integrates (or sums, for discrete $\mathcal{Y}$) to unity over the entire space $\mathcal{Y}$:

$$ $Z_{\theta}\;=\;\int_{y^{\prime}\in\mathcal{Y}}\exp\bigl(-E_{\theta}(y^{\prime})\bigr)\,dy^{\prime}\quad(\text{or }\sum_{y^{\prime}\in\mathcal{Y}}\exp\bigl(-E_{\theta}(y^{\prime})\bigr)\text{ for discrete }\mathcal{Y}).$ (8) $$

The computational intractability of $Z_{\theta}$ for complex, high-dimensional spaces, such as those involving text sequences, is a principal motivation for developing methods like Eorm that can operate effectively without its explicit calculation.

A key property of EBMs relevant to ranking tasks is the equivalence between energy minimization and probability maximization.

###### Theorem C.1 (Energy–Probability Equivalence for Ranking) .

For any finite candidate set $\mathcal{Y}_{\mathrm{cand}}\subset\mathcal{Y}$, the configuration $y^{*}$ that minimizes the energy function $E_{\theta}(y)$ over this set is also the configuration that maximizes the Boltzmann probability $p_{\theta}(y)$:

$$ $y^{*}\;=\;\arg\min_{y\in\mathcal{Y}_{\mathrm{cand}}}E_{\theta}(y)\;=\;\arg\max_{y\in\mathcal{Y}_{\mathrm{cand}}}p_{\theta}(y).$ (9) $$

###### Proof of Theorem ˜ C.1 (Energy–Probability Equivalence for Ranking).

Consider a finite candidate set $\mathcal{Y}_{\mathrm{cand}}$ and the Boltzmann probability $p_{\theta}(y)=\frac{\exp\bigl(-E_{\theta}(y)\bigr)}{Z_{\theta}}$ for $y\in\mathcal{Y}_{\mathrm{cand}}$. The partition function $Z_{\theta}$, whether defined globally over $\mathcal{Y}$ or locally over $\mathcal{Y}_{\mathrm{cand}}$ (if considering probabilities conditional on this set), is a positive constant with respect to the specific choice of $y$ from $\mathcal{Y}_{\mathrm{cand}}$. Consequently, maximizing $p_{\theta}(y)$ over $y\in\mathcal{Y}_{\mathrm{cand}}$ is equivalent to maximizing its numerator, $\exp\bigl(-E_{\theta}(y)\bigr)$:

$$ $\arg\max_{y\in\mathcal{Y}_{\mathrm{cand}}}p_{\theta}(y)=\arg\max_{y\in\mathcal{Y}_{\mathrm{cand}}}\frac{\exp\bigl(-E_{\theta}(y)\bigr)}{Z_{\theta}}=\arg\max_{y\in\mathcal{Y}_{\mathrm{cand}}}\exp\bigl(-E_{\theta}(y)\bigr).$ $$

The function $f(x)=\exp(-x)$ is strictly monotonically decreasing, as its derivative, $f^{\prime}(x)=-\exp(-x)$, is always negative for $x\in\mathbb{R}$. A property of strictly monotonically decreasing functions is that maximizing $f(x)$ is equivalent to minimizing $x$. Applying this, maximizing $\exp\bigl(-E_{\theta}(y)\bigr)$ is equivalent to minimizing its argument, $E_{\theta}(y)$:

$$ $\arg\max_{y\in\mathcal{Y}_{\mathrm{cand}}}\exp\bigl(-E_{\theta}(y)\bigr)=\arg\min_{y\in\mathcal{Y}_{\mathrm{cand}}}E_{\theta}(y).$ $$

Combining these steps, we conclude that $\arg\max_{y\in\mathcal{Y}_{\mathrm{cand}}}p_{\theta}(y)=\arg\min_{y\in\mathcal{Y}_{\mathrm{cand}}}E_{\theta}(y)$. This equivalence is fundamental to using energy functions for ranking and selection tasks.
∎

###### Definition C.1 (Energy Function) .

###### Definition C.2 (Boltzmann Distribution) .

###### Definition C.3 (Partition Function) .

###### Theorem C.1 (Energy–Probability Equivalence for Ranking) .

###### Proof of Theorem ˜ C.1 (Energy–Probability Equivalence for Ranking).

### C.2 Details for Eorm Architecture and Training Objective

The Eorm model architecture employs a Transformer encoder (Vaswani et al., 2017) to process input CoT sequences. Each input sequence $y$, typically a concatenation of a question and a candidate CoT solution, is first tokenized. A special classification token (e.g., [CLS]) is prepended to the tokenized sequence; its final hidden state representation is conventionally used in Transformer models to capture a summary of the entire sequence.
Let $\mathbf{h}_{\text{CLS}}$ be the final hidden state vector from the Transformer encoder corresponding to this [CLS] token. This vector is then passed through a small Multi-Layer Perceptron (MLP) head, which includes Layer Normalization (ba2016layer) for improved training stability. The MLP projects $\mathbf{h}_{\text{CLS}}$ to a single scalar value, which Eorm interprets as the energy $E_{\theta}(y)$ of the input sequence:

$$ $E_{\theta}(y)\;=\;\text{MLP}\!\Bigl(\text{LayerNorm}\bigl(\mathbf{h}_{\text{CLS}}\bigr)\Bigr)\,\in\,\mathbb{R}.$ (10) $$

The model is trained such that lower energy values $E_{\theta}(y)$ correspond to higher-quality (i.e., correct) CoT solutions.

For training Eorm, the data is organized into groups. Each group $\mathcal{Y}_{n}$ pertains to a single problem instance $n$ and comprises multiple candidate CoT solutions $\{y_{1},\dots,y_{k}\}$ generated for that problem. Each candidate $y_{i}$ is accompanied by a binary label $l(y_{i})\in\{0,1\}$, where $l(y_{i})=1$ indicates a correct solution and $l(y_{i})=0$ indicates an incorrect one. Within each group $\mathcal{Y}_{n}$, we delineate two subsets: $\mathcal{Y}_{+}=\{y\in\mathcal{Y}_{n}\mid l(y)=1\}$ (correct solutions) and $\mathcal{Y}_{-}=\{y\in\mathcal{Y}_{n}\mid l(y)=0\}$ (incorrect solutions).
The optimization of Eorm’s parameters $\theta$ is driven by the pairwise Bradley-Terry loss (bradley1952rank). For a single group $\mathcal{Y}_{n}$ (assuming it is non-degenerate, i.e., $|\mathcal{Y}_{+}|>0$ and $|\mathcal{Y}_{-}|>0$), the loss is defined as:

$$ $\mathcal{L}(\theta;\mathcal{Y}_{n})\;=\;\frac{1}{|\mathcal{Y}_{+}||\mathcal{Y}_{-}|}\sum_{y_{+}\in\mathcal{Y}_{+}}\sum_{y_{-}\in\mathcal{Y}_{-}}\log\Bigl(1+\exp\bigl(E_{\theta}(y_{+})-E_{\theta}(y_{-})\bigr)\Bigr).$ (11) $$

This loss function penalizes instances where a correct solution $y_{+}$ is assigned a higher (or not sufficiently lower) energy than an incorrect solution $y_{-}$. The training procedure, outlined in Algorithm [2](https://arxiv.org/html/2505.14999v3#alg2), aims to minimize this loss across all training groups.

### C.3 Interpreting Classifier Logits as Energies: Connection to Eorm

A pivotal insight that connects discriminative machine learning models to EBMs is the interpretation of classifier outputs (logits) as quantities related to energy (Grathwohl et al., 2019; Liu et al., 2020). This subsection elaborates on this connection and situates Eorm’s methodology within this context.

###### Definition C.4 (Classifier Logits and Softmax Probability) .

For a $K$-class discriminative classifier, let $f_{\theta}(y)=[f_{1}(y),\dots,f_{K}(y)]\in\mathbb{R}^{K}$ denote the vector of output scores (logits) for an input $y$, where $\theta$ represents the classifier’s parameters. The conditional probability $P(k|y;\theta)$ of $y$ belonging to class $k$ is commonly computed using the softmax function:

$$ $P(k|y;\theta)=\frac{\exp(f_{k}(y))}{\sum_{j=1}^{K}\exp(f_{j}(y))}.$ (12) $$

###### Proposition C.1 (Implicit Energy Functions from Classifier Logits) .

A $K$-class classifier, as described by its logits $f_{\theta}(y)$ and softmax probabilities $P(k|y;\theta)$ ([Definition˜C.4](https://arxiv.org/html/2505.14999v3#A3.Thmdefinitioninner4)), can be understood as implicitly defining $K$ class-conditional energy functions $E_{k}(y;\theta)=-f_{k}(y)$ for each class $k$.

- 1.
Using these energy functions, the conditional probability $P(k|y;\theta)$ can be expressed in an explicitly energy-based form:
$P(k|y;\theta)=\frac{\exp(-E_{k}(y;\theta))}{\sum_{j=1}^{K}\exp(-E_{j}(y;\theta))}.$
(13)
- 2.
Furthermore, a joint probability distribution $P(y,k;\theta)$ over inputs and classes can be consistently defined within an EBM framework. If we posit a joint energy $E(y,k;\theta)=E_{k}(y;\theta)+E_{0}(y;\theta)$, where $E_{0}(y;\theta)$ is an energy function associated with the input $y$ itself (representing the marginal energy of $y$), then:
$P(y,k;\theta)=\frac{\exp\bigl(-(E_{k}(y;\theta)+E_{0}(y;\theta))\bigr)}{Z^{\prime}_{\theta}},$
(14)
where $Z^{\prime}_{\theta}=\sum_{y^{\prime}\in\mathcal{Y}}\sum_{l=1}^{K}\exp\bigl(-(E_{l}(y^{\prime};\theta)+E_{0}(y^{\prime};\theta))\bigr)$ is the global partition function. This joint distribution correctly recovers the classifier’s original conditional probabilities $P(k|y;\theta)$.

###### Proof of Proposition ˜ C.1 .

:
Part 1: Derivation of the energy-based form for $P(k|y;\theta)$.

We begin with the standard softmax definition for $P(k|y;\theta)$:

$$ $P(k|y;\theta)=\frac{\exp(f_{k}(y))}{\sum_{j=1}^{K}\exp(f_{j}(y))}.$ $$

By defining the class-conditional energy $E_{k}(y;\theta)=-f_{k}(y)$, it follows that the logit $f_{k}(y)=-E_{k}(y;\theta)$.
Substituting $f_{k}(y)=-E_{k}(y;\theta)$ into the numerator yields $\exp(-E_{k}(y;\theta))$.
Similarly, substituting into each term of the sum in the denominator yields $\sum_{j=1}^{K}\exp(-E_{j}(y;\theta))$.
Thus, the conditional probability $P(k|y;\theta)$ becomes:

$$ $P(k|y;\theta)=\frac{\exp(-E_{k}(y;\theta))}{\sum_{j=1}^{K}\exp(-E_{j}(y;\theta))},$ $$

which is [Eq.˜13](https://arxiv.org/html/2505.14999v3#A3.E13). The denominator, $\sum_{j=1}^{K}\exp(-E_{j}(y;\theta))$, serves as a local (input-dependent) partition function, often denoted $Z(y;\theta)$, for the conditional distribution $P(\cdot|y;\theta)$.

Part 2: Consistent definition of the joint probability $P(y,k;\theta)$.

Let us define a joint energy function for the pair $(y,k)$ as $E(y,k;\theta)=E_{k}(y;\theta)+E_{0}(y;\theta)$. Here, $E_{k}(y;\theta)=-f_{k}(y)$ are the class-conditional energies derived from classifier logits, and $E_{0}(y;\theta)$ represents an energy associated with the input $y$, effectively capturing the energy of $y$ under its marginal distribution.
The joint probability $P(y,k;\theta)$ is then given by the Boltzmann distribution corresponding to this joint energy:

$$ $P(y,k;\theta)=\frac{\exp(-E(y,k;\theta))}{Z^{\prime}_{\theta}}=\frac{\exp\bigl(-(E_{k}(y;\theta)+E_{0}(y;\theta))\bigr)}{Z^{\prime}_{\theta}},$ $$

where $Z^{\prime}_{\theta}=\sum_{y^{\prime}\in\mathcal{Y}}\sum_{l=1}^{K}\exp\bigl(-(E_{l}(y^{\prime};\theta)+E_{0}(y^{\prime};\theta))\bigr)$ is the global partition function, summing over all possible inputs $y^{\prime}$ and all classes $l$.

To confirm consistency, we derive the conditional probability $P(k|y;\theta)$ from this joint distribution using the fundamental relation $P(k|y;\theta)=\frac{P(y,k;\theta)}{P(y;\theta)}$.
First, the marginal probability of $y$, $P(y;\theta)$, is obtained by summing (or marginalizing) the joint probability $P(y,l;\theta)$ over all classes $l$:

$$ $\displaystyle P(y;\theta)$ $\displaystyle=\sum_{l=1}^{K}P(y,l;\theta)$ $\displaystyle=\sum_{l=1}^{K}\frac{\exp\bigl(-(E_{l}(y;\theta)+E_{0}(y;\theta))\bigr)}{Z^{\prime}_{\theta}}$ $\displaystyle=\frac{\exp(-E_{0}(y;\theta))}{Z^{\prime}_{\theta}}\sum_{l=1}^{K}\exp(-E_{l}(y;\theta)).$ $$

Now, we compute the conditional probability $P(k|y;\theta)$:

$$ $\displaystyle P(k|y;\theta)$ $\displaystyle=\frac{P(y,k;\theta)}{P(y;\theta)}$ $\displaystyle=\frac{\frac{\exp\bigl(-(E_{k}(y;\theta)+E_{0}(y;\theta))\bigr)}{Z^{\prime}_{\theta}}}{\frac{\exp(-E_{0}(y;\theta))}{Z^{\prime}_{\theta}}\sum_{l=1}^{K}\exp(-E_{l}(y;\theta))}$ $\displaystyle=\frac{\exp\bigl(-(E_{k}(y;\theta)+E_{0}(y;\theta))\bigr)}{\exp(-E_{0}(y;\theta))\sum_{l=1}^{K}\exp(-E_{l}(y;\theta))}$ $\displaystyle=\frac{\exp(-E_{k}(y;\theta))\exp(-E_{0}(y;\theta))}{\exp(-E_{0}(y;\theta))\sum_{l=1}^{K}\exp(-E_{l}(y;\theta))}$ $\displaystyle=\frac{\exp(-E_{k}(y;\theta))}{\sum_{l=1}^{K}\exp(-E_{l}(y;\theta))}.$ $$

This resulting expression for $P(k|y;\theta)$ is identical to the energy-based form derived in Part 1 ([Eq.˜13](https://arxiv.org/html/2505.14999v3#A3.E13)) and, consequently, consistent with the original softmax formulation of the classifier ([Definition˜C.4](https://arxiv.org/html/2505.14999v3#A3.Thmdefinitioninner4)), given the definition $E_{j}(y;\theta)=-f_{j}(y)$.
This demonstrates that a joint energy definition of the form $E(y,k;\theta)=E_{k}(y;\theta)+E_{0}(y;\theta)$ allows a standard classifier to be seamlessly integrated into a joint EBM framework. The specific choice of $E_{0}(y;\theta)$ influences the modeled marginal $P(y;\theta)$, but the conditional $P(k|y;\theta)$ remains determined by the classifier’s logits. For example, if one aims to have $P(y,k)=P(k|y)P(y)$, then $E(y,k)$ can be set to $-\log P(k|y)-\log P(y)$, which implies a specific form for $E_{0}(y;\theta)$ related to $-\log P(y)$ and the local partition function $Z(y;\theta)$. However, the proposition’s assertion of consistency holds for a general $E_{0}(y;\theta)$ representing the energy contribution of $y$.
∎

###### Definition C.4 (Classifier Logits and Softmax Probability) .

###### Proposition C.1 (Implicit Energy Functions from Classifier Logits) .

###### Proof of Proposition ˜ C.1 .

### C.4 Conceptual Analysis of the Learned Energy Landscape

The energy function $E_{\theta}(y)$ learned by Eorm effectively defines an "energy landscape" across the high-dimensional, discrete space $\mathcal{Y}$ of CoT solutions. The characteristics of this landscape are paramount to Eorm’s ability to discriminate effectively between high-quality, correct reasoning paths and flawed or incorrect ones. While Eorm does not employ $E_{\theta}(y)$ for generative sampling (e.g., to create novel CoTs by navigating this landscape), an understanding of the intended structure of this landscape, as sculpted by the training objective, offers valuable insights into its operational mechanism.

###### Definition C.5 (Optimal Energy Separation (Idealized Goal)) .

Consider a specific problem or question $q$. Let $\mathcal{Y}_{C,q}\subseteq\mathcal{Y}$ denote the set of all correct CoT solutions for $q$, and $\mathcal{Y}_{I,q}\subseteq\mathcal{Y}$ denote the set of all incorrect CoT solutions for $q$. An ideally structured energy landscape, from the perspective of discriminating solutions for question $q$, would satisfy the condition:

$$ $\sup_{y_{c}\in\mathcal{Y}_{C,q}}E_{\theta}(y_{c})<\inf_{y_{i}\in\mathcal{Y}_{I,q}}E_{\theta}(y_{i}).$ (15) $$

This inequality implies the existence of an energy threshold $\tau_{q}$ such that all correct solutions for question $q$ possess an energy $E_{\theta}(y_{c})<\tau_{q}$, while all incorrect solutions have an energy $E_{\theta}(y_{i})>\tau_{q}$. Achieving such perfect global separation for all possible CoTs is a highly stringent condition and unlikely to be fully realized in practice. However, the training process of Eorm is designed to approximate this separation, particularly for the types of candidate solutions encountered in the training data.

###### Definition C.5 (Optimal Energy Separation (Idealized Goal)) .

#### Role of the Pairwise Bradley-Terry Loss in Shaping the Landscape.

The pairwise Bradley-Terry loss, as defined in [Eq.˜11](https://arxiv.org/html/2505.14999v3#A3.E11), is instrumental in sculpting the desired energy landscape. For each pair of solutions $(y_{+},y_{-})$ from the same problem context, where $y_{+}$ is correct and $y_{-}$ is incorrect, the loss term is $\mathcal{L}_{pair}(y_{+},y_{-})=\log(1+\exp(E_{\theta}(y_{+})-E_{\theta}(y_{-})))$.
To understand its effect, let us define the "energy margin" for a correctly ordered pair as $\delta(y_{+},y_{-})=E_{\theta}(y_{-})-E_{\theta}(y_{+})$. The loss term can then be expressed as $\log(1+\exp(-\delta(y_{+},y_{-})))$. Minimizing this loss is equivalent to maximizing the margin $\delta(y_{+},y_{-})$, thereby actively pushing $E_{\theta}(y_{+})$ to be lower than $E_{\theta}(y_{-})$.
The gradient of this loss with respect to the energies (see components in [Eq.˜18](https://arxiv.org/html/2505.14999v3#A3.E18)) illustrates this dynamic:
The term $\sigma(E_{\theta}(y_{+})-E_{\theta}(y_{-}))$ acts as a dynamic weighting factor.

- •
If $E_{\theta}(y_{+})\geq E_{\theta}(y_{-})$ (i.e., the pair is misordered or has no margin, $\delta(y_{+},y_{-})\leq 0$), the sigmoid term $\sigma(E_{\theta}(y_{+})-E_{\theta}(y_{-}))$ is $\geq 0.5$. This results in a relatively strong gradient signal that pushes to decrease $E_{\theta}(y_{+})$ and increase $E_{\theta}(y_{-})$, effectively trying to correct the order and increase the margin.
- •
If $E_{\theta}(y_{+})\ll E_{\theta}(y_{-})$ (i.e., the pair is well-ordered with a large positive margin $\delta(y_{+},y_{-})\gg 0$), the sigmoid term approaches 0. The gradient signal becomes weak, as the desired ordering is already satisfied.

This adaptive mechanism concentrates the learning effort on problematic pairs, carving out low-energy regions for correct solutions relative to incorrect ones. The loss enforces relative energy orderings rather than absolute energy targets, which can make the training more robust, for example, to imbalances in the number of correct versus incorrect examples per problem.

### C.5 Theoretical Analysis of the Eorm Training Objective

This section details the mathematical properties of the Eorm training objective, including definitions of relevant functions and formal derivations.

###### Definition C.6 (Sigmoid Function) .

The *sigmoid function* $\sigma:\mathbb{R}\to(0,1)$ is defined as:

$$ $\sigma(z)=\frac{1}{1+\exp(-z)}.$ (16) $$

###### Definition C.7 (Softplus Function) .

The *softplus function* $\text{softplus}:\mathbb{R}\to\mathbb{R}_{>0}$ is defined as:

$$ $\text{softplus}(z)=\log(1+\exp(z)).$ (17) $$

A useful property is that its derivative is the sigmoid function: $\frac{d}{dz}\text{softplus}(z)=\sigma(z)$.

###### Theorem C.2 (Gradient of Pairwise Bradley-Terry Loss) .

Let $\mathcal{L}(\theta;\mathcal{Y}_{n})$ be the pairwise Bradley-Terry loss for a group $\mathcal{Y}_{n}$, as given by [Eq.˜11](https://arxiv.org/html/2505.14999v3#A3.E11). Assuming the energy function $E_{\theta}(y)$ is differentiable with respect to its parameters $\theta$, the gradient of this loss is:

$$ $\nabla_{\!\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})\;=\;\frac{1}{|\mathcal{Y}_{+}||\mathcal{Y}_{-}|}\sum_{y_{+}\in\mathcal{Y}_{+}}\sum_{y_{-}\in\mathcal{Y}_{-}}\sigma\Bigl(E_{\theta}(y_{+})-E_{\theta}(y_{-})\Bigr)\Bigl(\nabla_{\!\theta}E_{\theta}(y_{+})\;-\;\nabla_{\!\theta}E_{\theta}(y_{-})\Bigr),$ (18) $$

where $\sigma(\cdot)$ is the sigmoid function ([Definition˜C.6](https://arxiv.org/html/2505.14999v3#A3.Thmdefinitioninner6)).

###### Proof of Theorem ˜ C.2 .

The pairwise Bradley-Terry loss for a non-degenerate group $\mathcal{Y}_{n}$ (where $\mathcal{Y}_{+}$ and $\mathcal{Y}_{-}$ are non-empty) is:

$$ $\mathcal{L}(\theta;\mathcal{Y}_{n})\;=\;\frac{1}{|\mathcal{Y}_{+}||\mathcal{Y}_{-}|}\sum_{y_{+}\in\mathcal{Y}_{+}}\sum_{y_{-}\in\mathcal{Y}_{-}}\log\Bigl(1+\exp\bigl(E_{\theta}(y_{+})-E_{\theta}(y_{-})\bigr)\Bigr).$ $$

Let $L_{pair}(y_{+},y_{-};\theta)=\log\bigl(1+\exp\bigl(E_{\theta}(y_{+})-E_{\theta}(y_{-})\bigr)\bigr)$ denote the loss contribution from a single pair $(y_{+},y_{-})$. Using the definition of the softplus function ([Definition˜C.7](https://arxiv.org/html/2505.14999v3#A3.Thmdefinitioninner7)), we can write $L_{pair}(y_{+},y_{-};\theta)=\text{softplus}(E_{\theta}(y_{+})-E_{\theta}(y_{-}))$.
Due to the linearity of the gradient operator, we have:

$$ $\nabla_{\!\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})\;=\;\frac{1}{|\mathcal{Y}_{+}||\mathcal{Y}_{-}|}\sum_{y_{+}\in\mathcal{Y}_{+}}\sum_{y_{-}\in\mathcal{Y}_{-}}\nabla_{\!\theta}L_{pair}(y_{+},y_{-};\theta).$ $$

To compute $\nabla_{\!\theta}L_{pair}(y_{+},y_{-};\theta)$, we apply the chain rule, noting that $\frac{d}{dz}\text{softplus}(z)=\sigma(z)$:

$$ $\displaystyle\nabla_{\!\theta}L_{pair}(y_{+},y_{-};\theta)$ $\displaystyle=\nabla_{\!\theta}\text{softplus}\bigl(E_{\theta}(y_{+})-E_{\theta}(y_{-})\bigr)$ $\displaystyle=\left.\frac{d}{dz}\text{softplus}(z)\right|_{z=E_{\theta}(y_{+})-E_{\theta}(y_{-})}\cdot\nabla_{\!\theta}\bigl(E_{\theta}(y_{+})-E_{\theta}(y_{-})\bigr)$ $\displaystyle=\sigma\bigl(E_{\theta}(y_{+})-E_{\theta}(y_{-})\bigr)\cdot\bigl(\nabla_{\!\theta}E_{\theta}(y_{+})\;-\;\nabla_{\!\theta}E_{\theta}(y_{-})\bigr).$ $$

Substituting this expression back into the sum for $\nabla_{\!\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})$ yields the statement of the theorem, [Eq.˜18](https://arxiv.org/html/2505.14999v3#A3.E18).
∎

###### Remark C.1 (Pairwise Bradley-Terry Loss as Negative Log-Likelihood) .

The pairwise Bradley-Terry loss function used in Eorm has a strong theoretical grounding as the negative log-likelihood (NLL) of observed preferences under the Bradley-Terry probabilistic model for pairwise comparisons (bradley1952rank).
This model assumes that each item $y$ possesses an underlying positive "strength" or "score," denoted $\pi_{y}$. The probability that item $y_{i}$ is preferred over item $y_{j}$ is then given by $P(y_{i}\text{ preferred over }y_{j})=\frac{\pi_{i}}{\pi_{i}+\pi_{j}}$.
In an energy-based formulation, we can define the strength of a solution $y$ as inversely related to its energy: $\pi_{y}=\exp(-E_{\theta}(y))$. A lower energy $E_{\theta}(y)$ thus corresponds to a higher strength $\pi_{y}$.
Under this definition, the probability that a correct solution $y_{+}$ is preferred over an incorrect solution $y_{-}$ (which implies $E_{\theta}(y_{+})$ should be less than $E_{\theta}(y_{-})$) can be modeled as:

$$ $\displaystyle P(y_{+}\text{ preferred over }y_{-}|\theta)$ $\displaystyle=\frac{\pi_{y_{+}}}{\pi_{y_{+}}+\pi_{y_{-}}}$ $\displaystyle=\frac{\exp(-E_{\theta}(y_{+}))}{\exp(-E_{\theta}(y_{+}))+\exp(-E_{\theta}(y_{-}))}.$ $$

Dividing both the numerator and the denominator by $\exp(-E_{\theta}(y_{+}))$ yields:

$$ $\displaystyle P(y_{+}\text{ preferred over }y_{-}|\theta)$ $\displaystyle=\frac{1}{1+\exp(-E_{\theta}(y_{-})+E_{\theta}(y_{+}))}$ $\displaystyle=\frac{1}{1+\exp\bigl(-(E_{\theta}(y_{-})-E_{\theta}(y_{+}))\bigr)}$ $\displaystyle=\sigma\bigl(E_{\theta}(y_{-})-E_{\theta}(y_{+})\bigr),\quad(\text{using the definition of }\sigma(z)\text{ from \lx@cref{creftypecap~refnum}{def:sigmoid}}).$ $$

The negative log-likelihood (NLL) of observing this single preference $y_{+}>y_{-}$ is:

$$ $\displaystyle\text{NLL}(y_{+}>y_{-}|\theta)$ $\displaystyle=-\log P(y_{+}\text{ preferred over }y_{-}|\theta)$ $\displaystyle=-\log\sigma\bigl(E_{\theta}(y_{-})-E_{\theta}(y_{+})\bigr).$ $$

Using the identity $-\log\sigma(x)=\log(1+e^{-x})=\text{softplus}(-x)$, we have:

$$ $\displaystyle\text{NLL}(y_{+}>y_{-}|\theta)$ $\displaystyle=\log\left(1+\exp\bigl(-\left(E_{\theta}(y_{-})-E_{\theta}(y_{+})\right)\bigr)\right)$ $\displaystyle=\log\left(1+\exp\bigl(E_{\theta}(y_{+})-E_{\theta}(y_{-})\bigr)\right).$ $$

This expression is precisely the loss contribution from a single pair $(y_{+},y_{-})$ as defined in [Eq.˜11](https://arxiv.org/html/2505.14999v3#A3.E11). This term is also recognizable as the logistic loss or binary cross-entropy for the event of $y_{+}$ being preferred over $y_{-}$, given the "logit" of this preference is $E_{\theta}(y_{-})-E_{\theta}(y_{+})$.
Therefore, minimizing the average pairwise Bradley-Terry loss is equivalent to performing Maximum Likelihood Estimation (MLE) for the parameters $\theta$ under this probabilistic preference model, where preferences are determined by energy differences. This equivalence provides a robust theoretical justification for the chosen loss function.

###### Proposition C.2 (Unbiased Gradient Estimation from Non-Degenerate Groups) .

Let $\mathcal{G}$ represent the true underlying distribution of all training groups $n$. The loss for group $n$, $\mathcal{L}(\theta;\mathcal{Y}_{n})$, is defined such that $\mathcal{L}(\theta;\mathcal{Y}_{n})=0$ if group $n$ is degenerate (i.e., it lacks either positive examples, $|\mathcal{Y}_{+}|=0$, or negative examples, $|\mathcal{Y}_{-}|=0$). Let $\mathcal{G}^{\dagger}$ be the conditional distribution of groups $n$ from $\mathcal{G}$, given that they are non-degenerate.
In stochastic gradient descent (SGD), if we sample a group $n\sim\mathcal{G}^{\dagger}$ (i.e., we exclusively process non-degenerate groups) and compute its gradient $\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})$ using [Eq.˜11](https://arxiv.org/html/2505.14999v3#A3.E11), this gradient serves as an unbiased estimate of the expected gradient over *non-degenerate* groups, $\mathbb{E}_{n\sim\mathcal{G}^{\dagger}}[\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})]$. Furthermore, this expectation is proportional to the true expected gradient over *all* groups, $\mathbb{E}_{n\sim\mathcal{G}}[\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})]$.

###### Proof of Proposition ˜ C.2 .

Let $\mathbb{E}_{\mathcal{G}}[\cdot]$ denote the expectation with respect to the distribution $\mathcal{G}$ over all groups, and $\mathbb{E}_{\mathcal{G}^{\dagger}}[\cdot]$ denote the expectation with respect to the conditional distribution $\mathcal{G}^{\dagger}$ over non-degenerate groups. Let $\mathcal{S}_{\text{all}}$ be the set of indices for all possible training groups, $\mathcal{S}_{\text{nd}}$ be the set of indices for non-degenerate groups, and $\mathcal{S}_{\text{d}}$ be the set of indices for degenerate groups, such that $\mathcal{S}_{\text{all}}=\mathcal{S}_{\text{nd}}\cup\mathcal{S}_{\text{d}}$ and $\mathcal{S}_{\text{nd}}\cap\mathcal{S}_{\text{d}}=\emptyset$.

The true expected gradient over all groups is $\mathbb{E}_{\mathcal{G}}[\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})]$. This can be expanded as:

$$ $\displaystyle\mathbb{E}_{\mathcal{G}}[\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})]$ $\displaystyle=\sum_{i\in\mathcal{S}_{\text{all}}}P_{\mathcal{G}}(i)\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{i})$ $\displaystyle=\sum_{i\in\mathcal{S}_{\text{nd}}}P_{\mathcal{G}}(i)\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{i})+\sum_{i\in\mathcal{S}_{\text{d}}}P_{\mathcal{G}}(i)\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{i}).$ $$

By our definition, for any degenerate group $i\in\mathcal{S}_{\text{d}}$, the loss $\mathcal{L}(\theta;\mathcal{Y}_{i})=0$. Consequently, its gradient $\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{i})=\mathbf{0}$ for $i\in\mathcal{S}_{\text{d}}$.
Thus, the sum simplifies to:

$$ $\mathbb{E}_{\mathcal{G}}[\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})]=\sum_{i\in\mathcal{S}_{\text{nd}}}P_{\mathcal{G}}(i)\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{i}).$ $$

Let $P(\mathcal{S}_{\text{nd}})=\sum_{i\in\mathcal{S}_{\text{nd}}}P_{\mathcal{G}}(i)$ be the total probability of sampling a non-degenerate group. We assume $P(\mathcal{S}_{\text{nd}})>0$ (otherwise, no training would occur).
The probability of sampling a specific non-degenerate group $i\in\mathcal{S}_{\text{nd}}$ under the conditional distribution $\mathcal{G}^{\dagger}$ (i.e., given that the sampled group is non-degenerate) is $P_{\mathcal{G}^{\dagger}}(i)=P_{\mathcal{G}}(i|i\in\mathcal{S}_{\text{nd}})=\frac{P_{\mathcal{G}}(i)}{P(\mathcal{S}_{\text{nd}})}$.
The expected gradient when sampling exclusively from non-degenerate groups is:

$$ $\displaystyle\mathbb{E}_{\mathcal{G}^{\dagger}}[\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})]$ $\displaystyle=\sum_{i\in\mathcal{S}_{\text{nd}}}P_{\mathcal{G}^{\dagger}}(i)\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{i})$ $\displaystyle=\sum_{i\in\mathcal{S}_{\text{nd}}}\frac{P_{\mathcal{G}}(i)}{P(\mathcal{S}_{\text{nd}})}\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{i})$ $\displaystyle=\frac{1}{P(\mathcal{S}_{\text{nd}})}\sum_{i\in\mathcal{S}_{\text{nd}}}P_{\mathcal{G}}(i)\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{i}).$ $$

Comparing the two expectations, we find:

$$ $\mathbb{E}_{\mathcal{G}}[\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})]=P(\mathcal{S}_{\text{nd}})\cdot\mathbb{E}_{\mathcal{G}^{\dagger}}[\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})].$ $$

Since $P(\mathcal{S}_{\text{nd}})$ is a positive constant (typically close to 1 if degenerate groups are infrequent), the gradient $\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})$ for $n\sim\mathcal{G}^{\dagger}$ (as computed in LABEL:alg:ebm_training_bradley_terry_appendix) is an unbiased estimate of $\mathbb{E}_{\mathcal{G}^{\dagger}}[\nabla_{\theta}\mathcal{L}(\theta;\mathcal{Y}_{n})]$. Furthermore, this expected gradient over non-degenerate groups is directly proportional to the true expected gradient over all groups. In the context of SGD, this means that updating parameters using gradients from only non-degenerate groups will, on average, follow a direction proportional to the true full-batch gradient direction. This justifies the common practice of skipping degenerate groups during training, as it maintains an optimization path towards minimizing the true expected loss, with the effective learning rate scaled by $P(\mathcal{S}_{\text{nd}})$.
∎

###### Definition C.6 (Sigmoid Function) .

###### Definition C.7 (Softplus Function) .

###### Theorem C.2 (Gradient of Pairwise Bradley-Terry Loss) .

###### Proof of Theorem ˜ C.2 .

###### Remark C.1 (Pairwise Bradley-Terry Loss as Negative Log-Likelihood) .

###### Proposition C.2 (Unbiased Gradient Estimation from Non-Degenerate Groups) .

###### Proof of Proposition ˜ C.2 .

## Appendix D Implementation Details

This section provides key code snippets illustrating the implementation of the EORM model architecture and the training objective used in this work.

### D.1 EORM Model Architecture ( TransEBM )

The EORM model is implemented using PyTorch, leveraging a standard Transformer encoder architecture. The core components are defined in the TransEBM class, shown in Listing LABEL:lst:trans_ebm. The model takes token IDs and an attention mask as input and outputs a single scalar energy value for the sequence, derived from the final hidden state of the prepended [CLS] token. The variable representing the model’s hidden dimension is referred to as dim_model.

Figure: Listing 1: PyTorch implementation of the TransEBM model.

### D.2 Pairwise Bradley-Terry Loss Function

The model is trained using a pairwise Bradley-Terry objective, implemented as shown in Listing LABEL:lst:bt_loss. This function takes the energy scores assigned by the model to a group of candidate solutions and their corresponding binary labels (1 for correct, 0 for incorrect). It computes the average loss over all pairs of correct and incorrect candidates within the group, encouraging lower energy for correct solutions.

Figure: Listing 2: PyTorch implementation of the Pairwise Bradley-Terry loss.

### D.3 Hyperparameter Settings

**Table 6: Key hyperparameters for Eorm model architecture and training.**
| Category | Hyperparameter | Value |
| --- | --- | --- |
| Eorm Model Architecture | Embedding Dimension | 4096 |
| Transformer Encoder Layers | 2 |  |
| Attention Heads | 4 |  |
| Dropout Rate | 0.2 |  |
| Feed-forward Dimension | $4\times\text{d_model}$ |  |
| Activation Function | GELU |  |
| Normalization Style | Pre-LN (NormFirst) |  |
| Tokenizer Configuration | Base Tokenizer | GPT-2 |
| Max Sequence Length | 4096 |  |
| CLS ID | BOS token of tokenizer |  |
| PAD ID | EOS token |  |
| Training Configuration | Optimizer | AdamW |
| Learning Rate | $1\times 10^{-4}$ |  |
| Weight Decay | 0.01 |  |
| LR Scheduler | Cosine with Warmup |  |
| Warmup Ratio | 0.2 (of total steps) |  |
| Number of Epochs | 50 |  |
| Batch Size (groups) | 1 |  |
| Gradient Clipping Norm | 1.0 |  |
| FP16 (AMP) | CUDA |  |
| Training/Validation Split | 80% / 20% |  |
| DataLoader | 2 |  |

The training and evaluation of the Eorm model were conducted using a specific set of hyperparameters, primarily configured via command-line arguments with defaults specified in the training script. Key architectural parameters for the Eorm model (a Transformer-based EBM) and crucial settings for the optimization process are detailed in [Table˜6](https://arxiv.org/html/2505.14999v3#A4.T6). The AdamW optimizer was used in conjunction with a cosine learning rate schedule including a warmup phase. Training was performed with Automatic Mixed Precision (AMP) if the ‘–fp16‘ flag was enabled and a CUDA-compatible GPU was available. The tokenizer specified by default was ‘gpt2‘, with its vocabulary size determining the model’s embedding layer dimensions. A fixed random seed (42) was used for shuffling and creating the 80/20 train-validation split from the combined training data.

## Appendix E Dataset Details

This section outlines the dataset utilized for training and evaluating our Eorm model. We describe the data format, present a comparison with other notable mathematical reasoning datasets, and specify the overall size of our training corpus. The core of our dataset comprises (question, Chain-of-Thought solution, correctness label) triplets. These CoT solutions were generated by employing a suite of Large Language Models to solve problems sourced from established mathematical reasoning benchmarks, principally the training splits of GSM8k (Cobbe et al., 2021a) and MATH (Hendrycks et al., 2021), as elaborated in the main experimental setup ([Section˜4](https://arxiv.org/html/2505.14999v3#S4)).

### E.1 Data Format

Each instance in our dataset is represented as a JSON object, with multiple instances typically stored in a JSON Lines (‘.jsonl‘) file. Each JSON object contains the original mathematical problem, a generated CoT solution, and a binary label indicating whether the provided solution correctly answers the problem. An illustrative example of a single data instance is presented in [Section˜E.1](https://arxiv.org/html/2505.14999v3#A5.SS1).

### E.2 Comparison Between Other Dataset

**Table 7: Comparison of the Eorm training data scale and problem synthesis approach with other mathematical reasoning datasets. Dataset sizes are approximate, in thousands (k) of samples. "Synthesis Agent" refers to the primary method or model used for generating the core problems in the dataset.**
| Dataset | Synthesis Agent | Dataset Size (k) |
| --- | --- | --- |
| WizardMath (Luo et al., 2023) | GPT-4 | 96 |
| MetaMathQA (Yu et al., 2024) | GPT-3.5 | 395 |
| MMIQC (Liu et al., 2024a) | GPT-4+GPT-3.5+Human | 2294 |
| Orca-Math (Mitra et al., 2024) | GPT-4 | 200 |
| Xwin-Math-V1.1 (Li et al., 2024) | GPT-4 | 1440 |
| KPMath-Plus (Huang et al., 2024) | GPT-4 | 1576 |
| MathScaleQA (Tang et al., 2024) | GPT-3.5+Human | 2021 |
| DART-Math | DeepSeekMath-7B-RL | 591 |
| EORM | None | 14958 |

To position our data generation efforts for training Eorm, [Table˜7](https://arxiv.org/html/2505.14999v3#A5.T7) provides a comparative overview against several other prominent datasets used in mathematical reasoning research. The table highlights the scale (approximate number of samples) and the primary "Synthesis Agent" responsible for creating the core problems or question-answer pairs in those datasets. Many existing datasets, as indicated, leverage powerful large language models (LLMs) like GPT-4 or GPT-3.5, sometimes augmented with human effort, to synthesize novel mathematical problems. In contrast, our approach for the Eorm training data (labeled as "Eorm" in the table) does not involve the synthesis of new problems. Instead, we focus on generating a very large corpus of Chain-of-Thought (CoT) solutions by prompting various existing LLMs with problems from established, open benchmarks (such as GSM8k and MATH). Each generated CoT solution is then labeled for correctness based on its final answer. This strategy, reflected by "None" under "Synthesis Agent" for our Eorm data, results in a distinct type of dataset geared towards learning to verify reasoning processes rather than generating new problem instances. The substantial size of our collected data, approximately 14.96 million (14958k) (question, CoT solution, label) triplets, is intended to provide a diverse and comprehensive basis for training a robust Eorm verifier capable of understanding a wide range of reasoning styles and error patterns.

## Appendix F Hardware Resources

The computational experiments presented in this paper, encompassing both the generation of Chain-of-Thought (CoT) solutions and the training of our Energy Outcome Reward Model (Eorm), utilized high-performance Graphics Processing Units (GPUs). For the initial phase of generating the CoT candidate solutions from various Large Language Models (LLMs), a range of NVIDIA GPUs was employed. This included access to NVIDIA A800 (40GB), NVIDIA A100 (with both 40GB and 80GB variants), NVIDIA RTX A6000 Ada (48GB), and NVIDIA H100 GPUs (80GB). The availability of this diverse hardware allowed for extensive data generation across multiple LLM architectures. The training of the Eorm model itself was conducted on a more focused set of high-end accelerators. Specifically, we utilized configurations consisting of 2 to 4 NVIDIA H100 (80GB) GPUs. With a setup of 4 NVIDIA H100 (80GB) GPUs, the complete training process for the Eorm model took approximately 36 hours.

## Appendix G Ethics and Societal Impact

This research focuses on improving the correctness and efficiency of mathematical reasoning in LLMs, which can yield positive societal impacts such as accelerating scientific discovery, enhancing educational tools, and reducing the energy consumption of large-scale AI computations by minimizing flawed outputs. However, we recognize that a powerful and efficient reasoning verifier is a dual-use technology. In malicious hands, it could be used to automate the selection of the most effective or deceptive generated content, such as refining disinformation or generating plausible-but-malicious code. Furthermore, the performance of EORM is intrinsically tied to the dataset used for its training; any biases present in the problem sets (e.g., cultural context in word problems) could lead to the model favoring certain reasoning styles or performing inequitably across different problem domains. Our work is intended purely for beneficial applications, and we advocate for establishing strong ethical guidelines for the development and use of verifier and reward models in advanced LLM systems.

## Appendix H The Use of Large Language Models (LLMs)

Large Language Models (LLMs) are central to this research in two primary ways. First, they are the object of study; our EORM framework is designed to evaluate and improve the reasoning outputs of base models such as Llama 3, Mistral, and DeepSeekMath. Second, these same LLMs were used as a critical tool to generate the large-scale dataset of Chain-of-Thought solutions (both correct and incorrect) that EORM was trained on. For the preparation of this manuscript, our use of LLMs was strictly limited to polishing language and generating figures. All underlying research and intellectual content, including the EORM framework, its theoretical foundations, the experimental design, and the analysis of results, was completed entirely by the authors.

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 6
presentation: 3
questions: 1.  **"ORM" Baseline:** The comparison in Figure 3 against "ORM" is a key result. Could you please clarify what this "ORM" baseline model is? What is its architecture, and what loss function is it trained with?

2.  **Training Data Scaling:** The model was trained on a very large dataset of ~15 million samples. How does EORM's performance scale with the *number* of training examples? Is this massive dataset essential for the 55M model, or can it achieve strong (e.g., 90% of final) performance with, say, 1M or 5M examples?

3.  **Generalization Test Setup:** The "EORM Generalize" (leave-one-out) results in Table 4 are very strong. Did you also experiment with a more practical "general-purpose verifier" scenario, such as training on a *subset* of models (e.g., Llama 2, Mistral) and testing on a completely *disjoint* set (e.g., Llama 3, Qwen 2.5)?

4.  **MLP Ablation Details:** For the "MLP Verifier" ablation, how was the variable-length CoT sequence (token embeddings) fed into the MLP? Was it truncated, or was some form of pooling (e.g., average pooling) used?
soundness: 3
strengths: * **Exceptional Efficiency:** The most significant strength is the model's size. At only **55M parameters**, EORM is over 127 times smaller than standard 7B-8B parameter reward models. This makes it an incredibly practical and lightweight tool that can be cheaply deployed for inference alongside a generator LLM.

* **Low-Cost Supervision:** The model is trained *only* on binary outcome labels (correct/incorrect). This is a massive practical advantage over Process Reward Models (PRMs), which require costly and labor-intensive step-by-step human annotations, and is simpler than collecting preference pairs for Preference Optimization (PO).

* **Strong Empirical Performance:** Despite its small size, EORM achieves outstanding results. It significantly boosts the performance of all base LLMs it is applied to. For instance, it improves Llama 3 8B's accuracy from 76.6% to 90.7% on GSM8k and from 28.9% to 63.7% on MATH. It consistently outperforms both majority voting and a baseline ORM in reranking.

* **Demonstrated Generalization (Robustness):** The paper provides a very thorough validation of its generalization claims, which is a critical aspect for a practical verifier.
    * **OOD Tasks:** When trained only on GSM8k and MATH, EORM generalizes well to more difficult, unseen datasets like AIME 2024 and Gaokao Math.
    * **Unseen Models:** The leave-one-out experiments are highly convincing. A variant trained on 4/5 models (e.g., excluding Llama 3) can still dramatically improve the held-out model's accuracy (e.g., Llama 3 GSM8k base 42.9% -> 76.6%). This strongly supports the claim that EORM learns fundamental principles of valid reasoning rather than just overfitting to the stylistic quirks of its training models.

* **Clarity and Soundness:** The paper is exceptionally well-written. The problem is clearly motivated and contrasted with existing work. The methodological choice of using an EBM for ranking (which avoids computing the partition function $Z_{\theta}$) and training with a Bradley-Terry loss is elegant, well-justified, and theoretically sound.
summary: This paper addresses the challenge of reliably and efficiently verifying mathematical reasoning in Large Language Models (LLMs). The authors introduce the **Energy Outcome Reward Model (EORM)**, a novel and highly efficient post-hoc verifier for ranking Chain-of-Thought (CoT) solutions.

The core of the method is an **Energy-Based Model (EBM)** implemented as a lightweight (55M parameter) Transformer. This model is trained to assign a scalar "energy" score to any given CoT solution, where correct reasoning paths are assigned lower energy than incorrect ones. A key advantage of this approach is its training data requirement: EORM learns effectively using only simple **binary outcome labels** (i.e., whether the final answer is correct or incorrect), completely avoiding the need for expensive step-by-step annotations (like PRMs) or preference pairs (like POs).

The paper's main contributions are:
1.  A novel, lightweight (55M) and efficient verifier architecture (EORM) based on EBM principles, which is over 127 times smaller than typical 7B reward models.
2.  State-of-the-art reranking performance on mathematical benchmarks. By selecting the best from a pool of candidates, EORM boosts the accuracy of Llama 3 8B to **90.7% on GSM8k** and **63.7% on MATH**.
3.  Strong empirical evidence of **robust generalization**. The model, trained on GSM8k and MATH, generalizes effectively to out-of-distribution (OOD) datasets (like AIME and Gaokao Math) and, critically, to outputs from LLM architectures not seen during training.
weaknesses: * **Confusing "ORM" Baseline:** In Figure 3, the paper compares EORM against "ORM". This is confusing because the paper's own model is an "Energy **Outcome Reward Model**" (EORM). The paper also mentions "traditional Outcome Reward Models", but the specific architecture and loss function (e.g., standard classification cross-entropy?) of this "ORM" baseline are not defined. This makes the comparison in Figure 3 difficult to interpret.

* **Massive Training Data Requirement:** The model's success appears to be supported by a very large dataset of approximately 15 million (14958k) (question, CoT, label) triplets. The "low-cost supervision" (outcome labels) is thus traded for "high-volume data generation." While this is a *far better* trade-off than PRMs (which need high-cost supervision), the computational cost of generating and auto-labeling 15 million samples is still significant. The paper would be strengthened by a study of performance vs. training data scale.

* **Weak Ablation Baseline (MLP):** The ablation study in Figure 4 (left) compares the Transformer-based EORM to an "MLP Verifier". This is a weak baseline, as a simple MLP cannot effectively process variable-length, sequential token data. The conclusion that the "Transformer's ability... is crucial" is almost certainly correct, but a stronger baseline (e.g., a GRU/RNN, or an MLP on-top-of-average-pooled-embeddings) would make this point more definitively.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - Could the authors clarify how the baselines in Table 2 were selected and evaluated?
Specifically, were all models re-evaluated under the same data splits, sampling settings, and inference conditions as EORM, or were the reported numbers taken from prior work? A detailed explanation would help assess the fairness and validity of the comparison, especially given that several baselines use reinforcement learning or instruction-tuning procedures. (see W1)

- Since EORM is trained on final correctness, one could simply select candidates by checking answer accuracy when ground-truths are available. In what scenarios does EORM offer an advantage over such direct answer-based selection? Process reward models (PRM) provide additional step-wise signals that typical labeling cannot capture, so it would be helpful to clarify how EORM’s practical value compares when such signals are or are not available. (see W2)
soundness: 3
strengths: [S1] Methodological Modification. The paper provides a clear and focused description of its methodological modification of the energy-based modeling framework for ranking Chain-of-Thought outputs. The adaptation of EBM to reasoning verification is reasonable and technically sound.

[S2] Efficiency and Scalability. The proposed model is lightweight, using only 55M parameters compared to multi-billion-parameter reward models, which demonstrates strong potential for efficient and scalable deployment as a post-hoc verifier.
summary: The authors propose the Energy Outcome Reward Model (EORM), a lightweight post-hoc verifier for Chain-of-Thought (CoT) reasoning. The approach employs an energy-based framework that assigns scalar energy scores to CoT outputs, allowing the model to rank reasoning chains and select the most plausible ones. In contrast to process- or preference-based reward models, EORM uses only binary outcome labels, reducing annotation costs. The model is significantly smaller than typical reward models while maintaining strong performance and generalization across different base models and out-of-distribution reasoning tasks.
weaknesses: [W1] Unclear and Potentially Unfair Comparison Setup (Table 2).
The main quantitative comparison in Table 2 does not clearly specify how the baselines were selected or evaluated. The listed models, including WizardMath, DART-Math, and MetaMath, incorporate different forms of instruction tuning or reinforcement learning (for example, RLHF, RLEIF, or preference optimization), but the paper does not clarify whether they were re-evaluated under the same experimental setup, dataset splits, or sampling conditions as EORM.
As a result, it is difficult to determine whether the reported performance gains stem from the proposed reranking mechanism or from differences in training data, or evaluation methodology. A fair comparison would require verifying that all models were tested on identical candidate sets and reasoning samples, or at least discussing the limitations of using heterogeneous pre-trained baselines.
In addition, given that GRPO and related reinforcement learning frameworks have recently become standard approaches for improving LLM reasoning, the paper should include a discussion or comparison with such methods to position EORM more clearly within the current landscape.

[W2] Questionable Necessity of EORM Given Its Supervision on Correctness Labels.
EORM is trained using binary correctness labels (correct or incorrect) for each Chain-of-Thought (CoT) output. These labels require access to ground-truth answers, which are often unavailable in realistic reasoning settings.
While outcome-level supervision is cheaper than step-level annotation, this advantage only holds for structured math benchmarks such as GSM8K and MATH where answers are explicitly known. In broader domains such as commonsense or scientific reasoning, correctness labels are difficult or costly to obtain, limiting the method’s general applicability.
Moreover, if correctness labels already exist, it is unclear why a separate verifier model is needed to approximate an evaluation signal that can be computed directly. The paper would benefit from clarifying whether EORM is intended for use on unlabeled reasoning data and how it could generalize without direct correctness supervision.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. I suspect that stronger reasoning models, like Qwen 2.5 7B, often "know" that they cannot answer correctly and express it in its CoT explicitly. This means that an outcome reward model may not be useful, and it's sufficient to check if the model didn't respond "I don't know the correct answer". Do you observe such phenomena? How often such models respond with a number which is incorrect, as opposed of not giving any number?

2. When describing the Bradley-Terry loss you cite (Liu, 2009). However, I couldn't find Bradley-Terry loss defined there. Could you point to the exact place where this loss is defined?

3. You write that majority voting "tends to reflect the model’s biases rather than identifying the truly correct answer". What does it mean?

4. In Table 1, you indicate that EORM "Uses Internal Indicators as Reward". What does it mean?

5. In line 209 you write that the CLS tokens is prepended. Shouldn't it be appended?

6. What is "sampling probability" in line 265?
soundness: 1
strengths: * The paper tackles an interesting and practically important problem: how to approximately verify the CoT responses from LLMs.
* The EORM model trained by the authors is indeed lightweight compared to typical ORMs, which has practical implications.
* EORM is trained using an original approach with a special pairwise loss.
* The authors show that EORM generalizes (to a degree) between different base LLMs as well as between datasets.
summary: The introduces EORM (Energy Outcome Reward Model) -- a small model assessing the correctness of CoT-style LLM responses to mathematical queries. EORM is parametrized as a small (55M parameters) transformer with an MLP head, which is lightweight compared to typical outcome reward models trained by others (e.g., LLMs with 7B parameters). EORM receives a complete CoT on the input, returns a real-valued score representing the correctness assessment. EORM is trained via a special pairwise loss which promotes ranking correct responses higher than the incorrect ones.
weaknesses: In general I like the idea of training a lightweight ORM, and using an special loss for that. However, a significant weakness of the paper is its poor experimental setup which makes assessing the performance of EORM difficult.

1. In Table 2, you present a large set of results comparing EORM with other methods / base models. However, these results are not aligned on generation budgets, so comparing them is not meaningful. For instance, it may turn out that if some of the considered base models we sample 256 times and perform simple majority voting, the results are better than with EORM (which also samples 256 times). The same remark applies to Table 3. Also, the results change dramatically with different generation budgets, whereas the authors use a fixed budget of 256 (or 64 for OOD benchmarks) samples.

2. EORM should be compared with existing, open-source ORMs, and perhaps also with PRMs. In Figure 3 there appears some ORM, but it is not specified what exactly is this model. Also, the comparison between different verifiers could be expressed in terms of the AUC metric which would provide more direct comparison than downstream-task accuracy.

3. EORM uses non-standard loss, which is one of more important "selling points" of the paper. But it would be important to see if this way of training is better that, say, using simple binary cross-entropy loss, or contrastive learning approach. 

4. The authors train EORM on just two math datasets (GSM8K, MATH). It would strengthen the paper if more benchmarks would be considered (especially that these two are likely leaked into pre-training of the mainstream LLMs.) For instance, what about training on older AIME problems and evaluating on AIME 2024/2025?

5. The effectiveness of any ORM is conditioned on the fact that the correct CoT is often present among the generated samples. Therefore, it would be important to see for different models and datasets how often it is the case.

6. Some related works seem to be missing. For instance, [1] has a very similar theme of training a lightweight CoT ranker for math problems, so it would be good to include a comparison with this work.

**Minor**
* In line 389-390 you say Llama 3 8B gets 42.9% on GSM8K, which is inconsistent with Table 2 and 4 (76.6%).
* Figure 3 uses a strange scale on x axis: looks logarithmic, but not exactly: why x = 1 and x = 196 is included?
* In Figure 4, to make it more readable, datasets should be represented as points on x axis, not colors.

[1] Lightweight Latent Verifiers for Efficient Meta-Generation Strategies (https://arxiv.org/abs/2504.16760)

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. The proposed Energy Outcome Reward Model (EORM) introduces a ranking-based energy formulation for verifying reasoning traces. Could the authors clarify how this approach conceptually differs from prior reward modeling frameworks (e.g., PRM, ORM) beyond model size and supervision type?

2. The paper highlights strong cross-model generalization. Could the authors discuss whether this robustness arises from the model architecture, the training objective, or specific properties of the reasoning data distribution?

3. While EORM focuses on outcome-based supervision, have the authors considered hybridizing it with *process supervision* signals (e.g., step-level correctness annotations) to improve interpretability or reasoning faithfulness?

4. Could the authors provide variance or confidence intervals for key metrics (e.g., reasoning accuracy, ranking precision) across multiple runs to demonstrate the statistical robustness of EORM’s reported improvements?

5. How sensitive is EORM to noisy or imperfect correctness labels during training, and did the authors evaluate its stability under controlled label corruption experiments?

6. The paper claims strong cross-model generalization (e.g., from Llama to Mistral and Qwen). Were the reasoning traces normalized or token-aligned across models before training, and how does this preprocessing affect ranking performance?
soundness: 3
strengths: - **Strong empirical results:** State-of-the-art accuracy on GSM8k and MATH, with solid OOD generalization.  
- **Efficiency:** 55M-parameter verifier outperforms 7B reward models, highlighting excellent cost–performance tradeoffs.  
- **Clarity:** Methodology, loss function, and data preparation are explicitly described, enabling replication.  
- **Robust ablations:** Demonstrate architecture sensitivity and tokenizer invariance, strengthening the argument for generalizability.  
- **Practical impact:** Provides a scalable and plug-in verifier for reasoning tasks that could complement or replace Best-of-N sampling.
summary: This paper proposes the **Energy Outcome Reward Model (EORM)** — a lightweight, energy-based verifier for Chain-of-Thought (CoT) reasoning.  
EORM reframes the *Best-of-N selection* problem as a ranking task: it assigns a scalar “energy” score to each reasoning trace, preferring lower energies for correct reasoning.  
By training with only binary outcome labels, EORM avoids costly preference or process supervision.  
Despite having only **55M parameters** (≈127× smaller than typical reward models), it significantly improves reasoning accuracy on GSM8k and MATH, reaching **90.7%** and **63.7%** respectively when paired with Llama 3 8B.  
Notably, EORM generalizes well across unseen models and out-of-distribution tasks (e.g., AIME 2024, AMC), suggesting it learns transferable reasoning principles.  
The contribution is practical, technically sound, and experimentally convincing, though conceptually incremental relative to prior verification frameworks.
weaknesses: - **Limited theoretical depth:** The approach is primarily empirical; the energy formulation does not introduce new learning theory.  
- **Incremental conceptual novelty:** Extends known EBM and reward-modeling ideas rather than introducing new reasoning paradigms.  
- **Evaluation scope:** Focused on math reasoning; broader reasoning domains (commonsense, logical entailment) are not tested.  
- **No human interpretability study:** While effective, it is unclear what the model learns as a “signal of correctness.”  
- **Formatting issue:**  Figure 2 is disproportionately large relative to the information it presents. The visual content is minimal and could be conveyed more effectively as a compact table. The current formatting distracts from the narrative and gives the impression of filler space rather than substantive insight. Figure 5 can also be more compact. I would also recommend using PDF for your figure.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. For the ablation of the architecture, how is the input feed to the MLP (instead of the transformer encoder) verifier? That is how different output lengths are handled?
2. How does the ORM in Figure 3 differ from EORM?
3. What parts are really important for EORM? That is:
    + What makes it distinct from ORM trained as a small transformer model instead of a head on top of a large model?
    + How does the Bradley-Terry loss affect the performance?
soundness: 2
strengths: + The proposed method is evaluated against a large collection of fine-tuned models (Mistral, Llama2, DeepSeekMath, Llama 3, and Qwen 2.5) with parameters ranging around 7B.
+ The work shows how to create a relatively small and powerful reward model that beats majority voting and fine-tuned models
summary: The work shows that a relatively small (~55 M parameters) energy-based transformer encoder can be used as an efficient reward model for math-oriented tasks.
It provides a comparison with an established majority voting baseline, along with fine-tuned models and methods such as test-time RL.
The results show that there seems to be a strong case for the EBM reward model, showing its ability to generalize beyond the distribution of training data and model completions.
However, the work seems to underexplore several important aspects:
+ importance of Bradley-Terry loss on logits vs classification approach
+ comparison to reward models created by tuning a reward head on top of an autoregressive transformer (or maybe this is the ORM type used in Figure 3)
weaknesses: Presentation:

- Figure 1: Using an incorrect solution (70) to the "daps baps" problem as an example of something that an EORM thinks is correct and marking it in green can be misleading.
Note that the correct answer is 40, as 4 * 10 daps = 7*10 yaps and 5*14 yaps = 70 yaps = 3*14 baps = 42 baps
- line 188-189 "where Zθ is the partition function, a normalization constant that ensures pθ (y) sums to unity:" -> where Zθ is a normalization constant that ensures pθ (y) sums to unity:
- 264-265 "For each training problem, we generated n = 256 CoT candidates with a temperature of 0.7 and a sampling probability of 0.9, ensuring all attempts were included, regardless of their correctness." - this is a little bit unclear way of saying that authors use nucleus sampling with temperature 0.7 and top-p=0.9
- 360-361 "Additionally, the Qwen-2.5 base model also finds similar findings."
- Table 3: lines 374-377: "Comparison of EORM with other reasoning methods. We evaluate EORM against two reasoning baselines, TTRL and MathWizard, using accuracy as the metric across four mathematical datasets.": However, only TTRL (Zuo et al., 2025), DART-MATH (Prop2Diff) (Tong et al., 2024), and EORM are presented
- line 907 has a citation error for Layer Norm (? instead of citation)

Other:  
- A proper description of ORM in Figure 3 would be beneficial (how exactly does it differ from EORM)
- A proper description of how the input feeds to the MLP verifier (instead of the transformer encoder) in the ablation section would be beneficial
- minor: work does not evaluate recent "thinking" models