Logic-Regularized Verifier Elicits Reasoning from LLMs
Authors: Xinyu Wang, Changzhi Sun, Lian Cheng, Yuanbin Wu, Dell Zhang, Xuelong Li, Xiaoling Wang
Accepted at: acl25

Logic-Regularized Verifier Elicits Reasoning from LLMs

Xinyu Wang*1,2#, Changzhi Sun*2, Lian Cheng1,2#, Yuanbin Wu1,
Dell Zhang2, Xiaoling Wang1†, and Xuelong Li2†

1Department of Computer Science and Technology, East China Normal University
2Institute of Artificial Intelligence (TeleAI), China Telecom
{xinyu_wang@stu, ybwu@cs, x1wang@cs}.ecnu.edu.cn
{czsun}@chinatelecom.cn
{dell.z, xuelong_li}@ieee.org

Abstract

Verifiers are crucial components for enhancing modern LLMs’ reasoning capability. Typical verifiers require resource-intensive supervised dataset construction, which is costly and faces limitations in data diversity. In this paper, we propose LoVER, an unsupervised verifier regularized by logical rules. LoVER treats the verifier as a binary latent variable, utilizing internal activations and enforcing three logical constraints on multiple reasoning paths: negation consistency, intra-group consistency, and inter-group consistency (grouped by the final answer). By incorporating logical rules as priors, LoVER can leverage unlabeled examples and is directly compatible with any off-the-shelf LLMs. Experiments on 10 datasets demonstrate that LoVER significantly outperforms unsupervised baselines, achieving performance comparable to the supervised verifier (reaching its 95% level on average). The source code is publicly available at https://github.com/wangxinyufighting/llm-lover.

1 Introduction

Verifiers guide LLMs by providing feedback to optimize their parameters (RL scaling) or outputs (inference scaling), which greatly enhances models’ reasoning capabilities (Ouyang et al., 2022; Snell et al., 2024). Verifiers are usually trained with supervised learning (Cobbe et al., 2021; Yu et al., 2024), where they learn to classify reasoning outputs as true or false based on labeled data. It presents two challenges: 1) The verifier relies heavily on labeled data for training, which can be expensive to collect (particularly in specialized or complex domains). For example, annotating a single Olympiad-level problem typically takes a significant amount of time, and adding fine-grained step-level process annotations (Lightman et al., 2023) further increases the workload; 2) Relying on expert annotations may result in a lack of diversity in the solutions (Basile et al., 2021; Xu et al., 2024a), as annotators may favor familiar reasoning methods while overlooking equally valid but less intuitive ones. For example, when annotating geometric problems, annotators may prefer the standard coordinate method and down-vote the less obvious geometric observation. While one can improve the supervision process in various aspects (Yang et al., 2019) (e.g., more experts with diverse mathematical backgrounds and education experiences), LLMs themselves already compact large amounts of knowledge and abilities to sample diverse generations (Minaee et al., 2024; Xu et al., 2024b), a natural question is whether we could build verifiers without the supervision process?

To address these challenges, recent research has focused on unsupervised verifiers to uncover the intrinsic reasoning capabilities of LLMs. Typical works include: 1) CoT-Decoding (Wang and Zhou, 2024), which proposes a heuristic rule-based verifier by observing the probabilities of the outputs of LLMs. It selects the correct reasoning path based on the probability difference between the top and secondary tokens in the answer span. In experiments, we observed that CoT-Decoding is sensitive to the backbone LLM choice. For example, when using 11lama-7b on the GSM8K dataset, CoT-Decoding is 4.8% lower than the majority voting strategy. 2) CCS (Burns et al., 2023) introduces an unsupervised verifier, which is essentially a linear probe optimized through logical consistency loss. Unfortunately, CCS can only address Yes-No questions and struggles to scale to general reasoning tasks. A practical verifier should have fewer limitations on its target problems, enabling it to handle a broader range of reasoning scenarios and provide more flexibility in real-world applications

In this paper, we propose a principled framework
<table>
  <tr>
    <th>Verifier</th>
    <th>Paradigm</th>
    <th>Prior</th>
    <th>Annotation</th>
    <th>Input</th>
    <th>Model</th>
    <th>Scenario</th>
  </tr>
  <tr>
    <td>Wang and Zhou (2024)</td>
    <td>Unsup.</td>
    <td>Heuristics</td>
    <td>-</td>
    <td>Probabilities</td>
    <td>-</td>
    <td>General</td>
  </tr>
  <tr>
    <td>Burns et al. (2023)</td>
    <td>Unsup.</td>
    <td>Logic rule</td>
    <td>-</td>
    <td>Hidden states</td>
    <td>Linear</td>
    <td>Yes-No</td>
  </tr>
  <tr>
    <td>Cobbe et al. (2021)</td>
    <td>Sup.</td>
    <td>-</td>
    <td>Outcome-based</td>
    <td>Text</td>
    <td>Decoder-only</td>
    <td>General</td>
  </tr>
  <tr>
    <td>Lightman et al. (2023)</td>
    <td>Sup.</td>
    <td>-</td>
    <td>Process-based</td>
    <td>Text</td>
    <td>Decoder-only</td>
    <td>General</td>
  </tr>
  <tr>
    <td>LoVER</td>
    <td>Unsup.</td>
    <td>Logic rule</td>
    <td>-</td>
    <td>Hidden states</td>
    <td>MLP</td>
    <td>General</td>
  </tr>
</table>

Table 1: Comparison between existing verifiers. Paradigm: The verifier is trained using a supervised (Sup.) or unsupervised (Unsup.) learning paradigm. Prior: The prior knowledge used in the verifier. Annotation: The type of annotation data. “Outcome-based” is solution-level annotation. “Process-based” is step-level annotation. Input: The input data type of the verifier. Model: The model architecture. Scenario: Reasoning scenarios suitable for the verifier. “General” typically refers to reasoning problems that have a correct answer. “Yes-No” indicates that the answer to the question is either Yes or No.

LoVER, an unsupervised probabilistic verifier regularized by logical rules. For each reasoning path, we search for the implicit, internal “beliefs” or “knowledge” learned by the LLM to infer the truth value of the reasoning. LoVER begins by generating contrastive assertions through incorporating text templates. It then takes the internal activations of these assertions from the LLM as inputs and produces a binary latent variable to indicate the truth value. Furthermore, LoVER incorporates three logical constraints including negation consistency, intra-group consistency, and inter-group consistency (with multiple reasoning paths grouped by the final answer). To bridge the gap between discrete logical rules and continuous neural networks, we propose corresponding soft probabilistic objectives that support differentiable training. Our contributions are summarized as follows:

• We propose LoVER, a scalable and principled framework for verifying the truth value of reasoning paths, leveraging intrinsic knowledge learned by the LLM and regularized by logical rules. Additionally, LoVER is fully compatible with any off-the-shelf LLMs.
• To combine discrete logical rules with neural networks, we propose soft probabilistic objectives that enable LoVER to be trained end-to-end, improving its scalability and performance.
• Our extensive experiments across diverse datasets, including mathematical reasoning, common sense reasoning, and various backbones, demonstrate the effectiveness of the proposed method.

2 Approach

In this section, we present the proposed LoVER, an unsupervised verifier designed to reason over the internal activations of LLMs.

Task Definition Given an LLM and an input question \( q \), we first generate \( N \) complete solutions \( \{ s_i \}_{i=1}^N \), with each \( s_i \) representing a CoT path (Sec. 2.1). We then select the best solution based on a learned verifier. For each solution \( s_i \), we define \( x_i = q \oplus s_i, x_i^+ = x_i \oplus \mathrm{T}^+, x_i^- = x_i \oplus \mathrm{T}^- \), where \( \oplus \) denotes the text concatenation, and \( \mathrm{T}^+,\mathrm{T}^- \) are text templates. Given \( N \) reasoning paths, we group them into \( M \) sets (\( M \leq N \)) based on the final answer (extracted through rules from the answer token). \( \mathcal{A} \) represents the index set from 1 to \( N \), and \( \mathcal{A}_k \) denotes the \( k \)-th group, with \( \mathcal{A} = \cup_{k=1}^M \mathcal{A}_k \).

The verifier models a probabilistic distribution \( p_\theta(z|x) \), where \( x \in \cup_{i=1}^N \{ x_i^+, x_i^- \} \) and \( z \in \{0, 1\} \) is a binary latent variable indicating whether the natural language statement \( x \) is valid. In this paper, bold letters indicate variables.

Inspired by CoT-Decoding (Wang and Zhou, 2024) and CCS (Burns et al., 2023), to find the correct answer, we first augment each reasoning path to derive its correct and incorrect assertions, and then treat the truth values of the assertions as binary latent variables. On the one hand, we leverage the internal activations of the LLM as input, enabling better utilization of the model’s intrinsic knowledge. On the other hand, the logical constraints provide implicit supervision signals to update the verifier, significantly reducing the need for human supervision.

Next, we first introduce LLM decoding strategy (Sec. 2.1) and how to obtain contrastive assertions (Sec. 2.2). Then we detail the latent verifier model (Sec. 2.3) and describe the logical constraints imposed on the latent variables (Sec. 2.4). Finally, we present the training and inference procedure (Sec. 2.5). Fig. 1 shows an overview of our method.

2.1 LLM Decoding Strategy

Given an input question \( q \) and a typical decode-only LLM, there are various strategies to decode \( N \) solutions, such as beam search, nucleus sampling, and others. In this work, we follow the
Figure 1: An illustration of our proposed LoVER. For any question \( q \), we create \( x_i \) by combining \( q \) with the \( i \)-th solution from \( N \) solutions. We form \( x_i^+ \) and \( x_i^- \) by adding “This is a true/false answer.” to \( x_i \), respectively. Choosing the correct solution involves determining which assertion, \( x_i^+ \) or \( x_i^- \), is correct. The hidden states of LLMs are used to represent \( x_i^+ \) and \( x_i^- \), which are then input into LoVER to predict the correctness probability of each assertion. We extract the final answer from each solution and group assertions with identical answers together. These assertions follow three natural logical constraints that guide LoVER’s unsupervised training. **Negation Consistency** ensures that only one of \( x_i^+ \) or \( x_i^- \) is correct. **Intra-group Consistency** requires that assertions in the same group have equal correctness probabilities. **Inter-group Consistency** ensures that only one group’s \( x^+ \) assertion is correct across all groups.

CoT-Decoding (Wang and Zhou, 2024). Specifically, we keep the top \( N \) tokens with the highest probabilities at decoding step 0, and then continue with greedy decoding for each token, ultimately producing \( N \) solutions. Compared to other strategies, this method is more likely to produce a natural CoT reasoning path and does not rely on complex prompt engineering (Wang and Zhou, 2024). In the experiments, we also study the impact of different decoding strategies(Table 4).

2.2 Contrastive Assertions

For each \( x_i = q \oplus s_i \), we construct each contrastive assertions by appending the text templates T\(^+\) and T\(^-\). Formally, this is denoted as \( x_i^+ = x_i \oplus \mathrm{T}^+ \) and \( x_i^- = x_i \oplus \mathrm{T}^- \). In this paper, we adopt T\(^+\) = “This is a true answer.” and T\(^-\) = “This is a false answer.”. Importantly, rather than directly considering each reasoning path \( x_i \), we introduce contrastive assertions \( x_i^+ \) and \( x_i^- \), which help elicit the internal “beliefs” or “knowledge” learned by the model (Burns et al., 2023).

2.3 Latent Verifier Model

For each natural language assertion \( x \in \bigcup_{i=1}^N \{x_i^+, x_i^-\} \), we first compute the feature vector of \( x \), denoted as \( \phi(x) \),\footnote{The default is the hidden representation of the last token in the middle layer, and we also explore other options. For details, please refer to Table 5.} then pass it through a randomly initialized MLP, and finally map it to a probability value using the sigmoid function. Formally, we define the probabilistic distribution of verifier \( p_\theta(z|x) \) where \( z \in \{0, 1\} \) is a binary latent variable indicating whether the natural language statement \( x \) is valid. For simplicity, we use \( p_\theta(z) \) to represent \( p_\theta(z = 1|x) \):

\[
p_\theta(z) = p_\theta(z = 1|x) = \mathrm{Sigmoid}(\mathrm{MLP}(\phi(x))).
\]

Importantly, LoVER does not modify the weights of the LLM and it does not use labels.

2.4 Logical Constraints

After introducing the binary latent variables \( \bigcup_{i=1}^N \{z^+, z^-\} \), we observe that certain natural logical consistencies between them should be satisfied. Let us look at three such logical consistency requirements.

**Negation Consistency** Given the contrastive assertions \( x_i^+ \) and \( x_i^- \), their corresponding binary latent variables \( z_i^+ \) and \( z_i^- \) should satisfy negation consistency:

\[
z_i^+ = 1 - z_i^-,\ i \in \mathcal{A}.
\]

To this end, we relax the logic with soft probability (Chen et al., 2022a; Burns et al., 2023) for differentiability in training and regularization of binary latent variables. Inspired by CCS (Burns et al.,
2023), we aim for the contrastive assertions \( x_i^+ \) and \( x_i^- \) to satisfy the following: 1).the sum of their probabilities equals 1 (probability normalization); 2).their probabilities differ significantly (the law of excluded middle).

\[
\mathcal{L}_{\text{sum}} = \sum_{i=1}^{N} \left[ p_\theta(z_i^+) + p_\theta(z_i^-) - 1 \right]^2,
\]
\[
\mathcal{L}_{\text{diff}} = \sum_{i=1}^{N} \min \left\{ p_\theta(z_i^+), p_\theta(z_i^-) \right\}^2,
\]
\[
\mathcal{L}_{\text{nega}} = \mathcal{L}_{\text{sum}} + \mathcal{L}_{\text{diff}}.
\]

Note that both losses are necessary; using either one alone leads to a degenerate solution (Burns et al., 2023).

Intra-group Consistency For each group \( \mathcal{A}_k \) of reasoning paths, they share the same answer, though their reasoning processes may differ. Overall, we expect the corresponding binary latent variables to satisfy intra-group consistency, i.e.,
\[
z_i^+ = z_j^+, z_i^- = z_j^-, \quad i \in \mathcal{A}_k, j \in \mathcal{A}_k.
\]
To achieve this goal, we use a simple squared loss:
\[
\mathcal{L}_{\text{intra}}^+ = \sum_{k=1}^{M} \sum_{i \in \mathcal{A}_k, j \in \mathcal{A}_k} \left[ p_\theta(z_i^+) - p_\theta(z_j^+) \right]^2,
\]
\[
\mathcal{L}_{\text{intra}}^- = \sum_{k=1}^{M} \sum_{i \in \mathcal{A}_k, j \in \mathcal{A}_k} \left[ p_\theta(z_i^-) - p_\theta(z_j^-) \right]^2,
\]
\[
\mathcal{L}_{\text{intra}} = \mathcal{L}_{\text{intra}}^+ + \mathcal{L}_{\text{intra}}^-.
\]

Inter-group Consistency Among the \( N \) reasoning paths, there are \( M \) distinct answers. We assume that the LLM’s capabilities are sufficiently strong to ensure the presence of a correct answer. We examine the GSM8k dataset and find that when \( N = 10 \), the P@10 accuracy of qwen-2.5 can reach 91.43%. This confirms the validity of the above assumption. Specifically, for each group \( \mathcal{A}_k \), we randomly select an \( a_k \) and hope that its corresponding binary latent variable satisfies:
\[
\sum_{k=1}^{M} z_{a_k}^+ = 1, \quad a_k \in \mathcal{A}_k.
\]
(1)

To achieve this inter-group consistency, we propose a soft probability solution.
\[
\mathcal{L}_{\text{inter}}^{\text{sum}} = \left[ \sum_{k=1}^{M} p_\theta(z_{a_k}^+) - 1 \right]^2.
\]

However, in the experiments, we observe that if only the loss \( \mathcal{L}_{\text{inter}}^{\text{sum}} \) is used, the \( M \) probabilities \( p_\theta(z_{a_k}^+) \) tend to become uniform. To address this issue, we propose an entropy regularization. We introduce a probability distribution \( \hat{p} \) which defines over the \( M \) variables \( \{z_{a_k}^+\}_{k=1}^{M} \).
\[
\mathcal{L}_{\text{inter}}^h = \mathcal{H} \left[ \hat{p}(\cdot) \right], \quad \hat{p}(z_{a_k}^+) = \frac{p_\theta(z_{a_k}^+)}{\sum_{i=1}^{M} p_\theta(z_i^+)},
\]
\[
\mathcal{L}_{\text{inter}} = \mathcal{L}_{\text{inter}}^{\text{sum}} + \mathcal{L}_{\text{inter}}^h,
\]
where \( \mathcal{H} \) denote the entropy function. In addition, we also explore a soft logic-based solution, which is encapsulated in Appendix A.

2.5 Training and Inference

Training The final loss function is the sum of three losses mentioned above, which is defined as:
\[
\mathcal{L} = \mathcal{L}_{\text{nega}} + \mathcal{L}_{\text{intra}} + \mathcal{L}_{\text{inter}}.
\]

LOVER is structured as a MLP with 2 hidden layers, and we select ReLU as the activation function. We use AdamW (Loshchilov and Hutter, 2019) as optimizer (weight_decay = 0.01), and set learning rate to \( 1 \times e^{-5} \).

Inference Given an input question \( q \), we first decode \( N \) candidate solutions (Sec. 2.1) to obtain \( \{x_i\}_{i=1}^{N} \). For each \( x_i \), we generate contrastive assertions \( x_i^+ \) and \( x_i^- \) of \( s_i \) (Sec. 2.2) and compute corresponding probability \( p_\theta(z_i^+) \) and \( p_\theta(z_i^-) \) based on the latent verifier model (Sec. 2.3). Both \( p_\theta(z_i^+) \) and \( 1 - p_\theta(z_i^-) \) should represent the probability that the \( x_i \) is correct. We consequently take the average of these (Burns et al., 2023):
\[
p_\theta(z_i) = \frac{1}{2} \left[ p_\theta(z_i^+) + (1 - p_\theta(z_i^-)) \right].
\]
Then we group them into \( M \) sets based in the final answer (extracted through rules from answer token). For each group \( \mathcal{A}_k \), we compute the group score \( g_k \) using two strategies: max and sum. The max strategy computes the group score by selecting the maximum \( p_\theta \) within the group: \( g_k = \max_{i \in \mathcal{A}_k} p_\theta(z_i) \). The sum strategy computes the group score by summing up all \( p_\theta \) within the group: \( g_k = \sum_{i \in \mathcal{A}_k} p_\theta(z_i) \). Finally, we select the answer with the highest group score \( g_k \) among the \( M \) groups. Appendix C provides PyTorch-style pseudocode for the inference procedure.
<table>
  <tr>
    <th rowspan="2">LLM</th>
    <th rowspan="2">Dataset → Verifier ↓</th>
    <th colspan="2">Mathematics</th>
    <th colspan="2">Open-Domain Knowl.</th>
  </tr>
  <tr>
    <th>GSM8K</th>
    <th>iGSM</th>
    <th>HotpotQA</th>
    <th>MMLU-P</th>
  </tr>
  <tr>
    <td rowspan="6">mistral-7b</td>
    <td>Supervised</td>
    <td>61.18</td>
    <td>44.50</td>
    <td>82.96</td>
    <td>41.93</td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>16.15</td>
    <td>19.00</td>
    <td>70.74</td>
    <td>37.65</td>
  </tr>
  <tr>
    <td>Majority Voting</td>
    <td>43.82</td>
    <td>34.50</td>
    <td>76.86</td>
    <td>41.07</td>
  </tr>
  <tr>
    <td>CoT-Decoding (max)</td>
    <td>36.62</td>
    <td>12.75</td>
    <td>69.87</td>
    <td>38.04</td>
  </tr>
  <tr>
    <td>CoT-Decoding (sum)</td>
    <td>47.76</td>
    <td>36.75</td>
    <td>76.86</td>
    <td>41.43</td>
  </tr>
  <tr>
    <td>LoVER (max)</td>
    <td>50.41</td>
    <td>28.75</td>
    <td>74.24</td>
    <td>40.36</td>
  </tr>
  <tr>
    <td>LoVER (sum)</td>
    <td><b>53.14</b></td>
    <td><b>41.25</b></td>
    <td><b>77.95</b></td>
    <td><b>41.82</b></td>
  </tr>
  <tr>
    <td rowspan="6">llama-8b</td>
    <td>Supervised</td>
    <td>83.24</td>
    <td>43.75</td>
    <td>82.96</td>
    <td>55.86</td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>36.09</td>
    <td>35.00</td>
    <td>77.29</td>
    <td>45.67</td>
  </tr>
  <tr>
    <td>Majority Voting</td>
    <td>75.96</td>
    <td>39.00</td>
    <td>81.00</td>
    <td>50.58</td>
  </tr>
  <tr>
    <td>CoT-Decoding (max)</td>
    <td>38.51</td>
    <td>23.25</td>
    <td>77.95</td>
    <td>45.92</td>
  </tr>
  <tr>
    <td>CoT-Decoding (sum)</td>
    <td>71.11</td>
    <td>40.50</td>
    <td>78.16</td>
    <td>54.61</td>
  </tr>
  <tr>
    <td>LoVER (max)</td>
    <td>70.60</td>
    <td>39.00</td>
    <td>81.04</td>
    <td>52.65</td>
  </tr>
  <tr>
    <td>LoVER (sum)</td>
    <td><b>79.97</b></td>
    <td><b>42.50</b></td>
    <td><b>83.80</b></td>
    <td><b>55.61</b></td>
  </tr>
  <tr>
    <td rowspan="6">qwen-7b</td>
    <td>Supervised</td>
    <td>92.11</td>
    <td>54.25</td>
    <td>83.62</td>
    <td>61.13</td>
  </tr>
  <tr>
    <td>Greedy</td>
    <td>61.71</td>
    <td>46.50</td>
    <td>78.82</td>
    <td>54.93</td>
  </tr>
  <tr>
    <td>Majority Voting</td>
    <td>89.46</td>
    <td>51.00</td>
    <td><b>83.41</b></td>
    <td>57.00</td>
  </tr>
  <tr>
    <td>CoT-Decoding (max)</td>
    <td>63.84</td>
    <td>31.50</td>
    <td>81.66</td>
    <td>54.22</td>
  </tr>
  <tr>
    <td>CoT-Decoding (sum)</td>
    <td>89.76</td>
    <td>51.00</td>
    <td>82.09</td>
    <td>58.81</td>
  </tr>
  <tr>
    <td>LoVER (max)</td>
    <td>83.47</td>
    <td>48.25</td>
    <td>81.41</td>
    <td>59.28</td>
  </tr>
  <tr>
    <td>LoVER (sum)</td>
    <td><b>91.43</b></td>
    <td><b>52.00</b></td>
    <td>82.75</td>
    <td><b>61.24</b></td>
  </tr>
</table>

Table 2: The overall experimental results of LoVER and other baselines on the four datasets. Accuracy is utilized to measure the performance. The best results of each setting are in bold. MMLU-P stands for "MMLU-Pro" dataset.

3 Experiments

Datasets We conduct experiments on datasets of both mathematical and open-domain knowledge reasoning. For mathematical reasoning, we use the Grade-school math problems, GSM8K (Cobbe et al., 2021) and more challenging iGSM dataset (Ye et al., 2024). For open-domain knowledge reasoning, we use HotpotQA(Yang et al., 2018) and MMLU-Pro(Wang et al., 2024). Furthermore, to evaluate the out-of-distribution (OOD) generalization of LoVER, we employ Boolean Expressions, Web of Lies, Object Counting, Navigate, Multi-Step Arithmetic and Causal Judgement from BIG-Bench Hard(Suzgun et al., 2022). The details of datasets are provided in Appendix B.

Evaluation We evaluate accuracy by strictly matching the final answer from the response with the ground truth answer.

Baselines. We test three open-source LLMs: llama-8b, mistral-7b and qwen with different scales, ranging from 0.5B, 1.5B, 3B, 7B, and 32B.

2 We compare LoVER against following methods:
• “Greedy” decoding selects the most probable token at each step.
• “Majority Voting” decodes multiple outputs and select the optimal answer by voting (Lewkowycz et al., 2022; Wang et al.). 3
• “CoT-Decoding” (Wang and Zhou, 2024) selects correct reasoning paths based on answer confidence (probability disparity between the top and secondary tokens in answer spans).
• “Supervised” is the supervised LoVER. It is trained using gold label data, with the training objective being the standard binary cross-entropy loss. Theoretically, this is the ceiling of LoVER.

3.1 Main Results

LoVER effectively enhances reasoning abilities

2Specific versions are llama-3.1-8b-instruct (Grattafiori et al., 2024), mistral-7b-instruct-v0.3 (Jiang et al., 2023), qwen2.5-instruct (Qwen et al., 2025).
3The default decoding strategy is described in Sec. 2.1.
across models and reasoning types. As shown in Table 2, LoVER (sum) achieves the highest accuracy in all scenarios. LoVER (max) outperforms CoT-Decoding in 40% of cases and matches the average accuracy of Majority Voting.

LoVER (sum) achieves an average absolute accuracy gain of 3.3% over Majority Voting. Unlike Majority Voting, which relies solely on the frequency of answers, LoVER not only considers answer counts but also leverages the internal knowledge of LLMs. Driven by logical constraints, LoVER can more effectively utilize the LLMs’ internal knowledge to score the correctness of assertions. Thus, LoVER represents an optimized and weighted voting method. Compared to CoT-Decoding, LoVER shows an average absolute accuracy gain of 2.9%. LoVER focuses on the correctness of the solution itself, rather than emphasizing the format of the solution as in CoT-Decoding. CoT-Decoding aims to elicit reasoning paths with CoT processes, leading to significant accuracy gains on weaker models (those unable to autonomously generate CoT solutions without CoT prompting) but limited improvements on stronger models. As a result, compared to CoT-Decoding, LoVER is less affected by the underlying capabilities of the LLM. LoVER (sum) consistently outperforms LoVER (max), demonstrating the effectiveness of the sum strategy and highlighting the importance of the frequency of answers. CoT-Decoding (max) achieves an average accuracy similar to Greedy, indicating that relying solely on the probability with answer tokens is insufficient.

3.2 Ablation Studies

3.2.1 The effect of different logic constraints

Incorporating logical constraints can significantly enhance LoVER’s performance. Table 3 reveals that the exclusion of the \( \mathcal{L}_{\text{inter}} \) led to the most significant drop in reasoning accuracy, indicating its crucial role in enhancing model performance. Without \( \mathcal{L}_{\text{inter}} \), LoVER tends to optimize towards assigning a score of 1 to all \( x_i^+ \) and 0 to all \( x_i^- \).

In this scenario, LoVER loses the ability to discern the correctness of assertions. Removing \( \mathcal{L}_{\text{nega}} \) also results in a noticeable decrease in accuracy. Without \( \mathcal{L}_{\text{nega}} \), the premise that each assertion has only one correctness label cannot be satisfied. As a result, LoVER tends to optimize towards assigning identical scores to both \( x_i^+ \) and \( x_i^- \). The removal of \( \mathcal{L}_{\text{intra}} \) has a minimal impact on reasoning accuracy. \( \mathcal{L}_{\text{intra}} \) enforces consistency in correctness probabilities for solutions with the same final answer. However, solution correctness depends not only on the final answer but also on the problem-solving process, which may contain errors even if the answer is correct. Enforcing the consistency of correctness probabilities solely based on the same final answer may have limitations.

Logical constraints have a more substantial impact on LoVER (max) than on LoVER (sum). Insufficient constraints hinder LoVER (max)’s ability to accurately assess assertion validity, while LoVER (sum) mitigates this by incorporating answer frequency, reducing sensitivity to constraint variations.

<table>
  <tr>
    <th>Loss</th>
    <th colspan="2">GSM8K</th>
    <th colspan="2">MMLU-Pro</th>
  </tr>
  <tr>
    <th></th>
    <th>sum</th>
    <th>max</th>
    <th>sum</th>
    <th>max</th>
  </tr>
  <tr>
    <td>LoVER</td>
    <td><b>53.14</b></td>
    <td><b>50.41</b></td>
    <td><b>41.82</b></td>
    <td><b>40.36</b></td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{\text{nega}} \)</td>
    <td>51.78</td>
    <td>34.79</td>
    <td>40.07</td>
    <td>37.37</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{\text{intra}} \)</td>
    <td>52.76</td>
    <td>44.20</td>
    <td>41.75</td>
    <td>40.00</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{\text{inter}} \)</td>
    <td>49.96</td>
    <td>38.13</td>
    <td>38.15</td>
    <td>35.44</td>
  </tr>
</table>

Table 3: Accuracy of LoVER on GSM8K and MMLU-Pro using different setting of logic constraints over mistral-7b.

<table>
  <tr>
    <th>Decoding Strategies</th>
    <th>Nat</th>
    <th>Temp</th>
    <th>Beam</th>
  </tr>
  <tr>
    <td>Majority Voting</td>
    <td>43.82</td>
    <td>27.07</td>
    <td>24.79</td>
  </tr>
  <tr>
    <td>CoT-Decoding</td>
    <td>47.76</td>
    <td>28.05</td>
    <td>24.94</td>
  </tr>
  <tr>
    <td>LoVER</td>
    <td><b>53.14</b></td>
    <td><b>31.31</b></td>
    <td><b>28.20</b></td>
  </tr>
</table>

Table 4: Accuracy of LoVER on GSM8K test set using different decoding strategies over mistral-7b. Nat stands for natural CoT decoding, which is the default decoding strategy in this paper. Temp stands for Temperature sampling (temperature = 0.7). Beam stands for Beam Search sampling.

LoVER consistently achieves the best performance across all decoding strategies, delivering an average absolute accuracy improvement of 5.7% compared to Majority Voting. In contrast, CoT-Decoding only achieves only a 1.69% improvement over Majority Voting. The performance of
CoT-Decoding tends to rely more heavily on the format of LLMs’ output rather than its correctness. If the decoding strategy fails to elicit outputs in a specific format (e.g., those containing a CoT process), the effectiveness of CoT-Decoding is compromised. In contrast, LoVER analyzes the correctness of LLMs’ outputs and is therefore less affected by changes in sampling strategies compared to CoT-Decoding.

3.2.3 The effect of hidden states from different layers

Hidden States of middle layer optimize LoVER’s Performance. Previous research shows that hidden state from the middle to deeper layers contain richer knowledge compared to those in the shallower layers. Table 5 shows that LoVER achieves optimal performance when utilizing hidden states from the 15th or 20th layers, which aligns with the conclusions of prior research. The selection of hidden states’ layer has a significantly greater impact on LoVER (max) compared to LoVER (sum). For instance, on GSM8K, the standard deviation of accuracy for LoVER (max) across different layers is 7.1, whereas it is only 0.6 for LoVER (sum). This is because the knowledge contained in the hidden states directly influences LoVER (max)’s judgment on the correctness of assertions. LoVER (sum) incorporates the frequency of answers, which mitigates the influence of layer selection.

<table>
  <tr>
    <th rowspan="2">Layer</th>
    <th colspan="2">GSM8K</th>
    <th colspan="2">MMLU-Pro</th>
  </tr>
  <tr>
    <th>sum</th>
    <th>max</th>
    <th>sum</th>
    <th>max</th>
  </tr>
  <tr>
    <td>5</td>
    <td>51.82</td>
    <td>36.26</td>
    <td>41.32</td>
    <td>38.78</td>
  </tr>
  <tr>
    <td>15</td>
    <td>51.91</td>
    <td>37.28</td>
    <td>41.34</td>
    <td>39.46</td>
  </tr>
  <tr>
    <td>20</td>
    <td><b>53.14</b></td>
    <td><b>50.41</b></td>
    <td><b>41.82</b></td>
    <td><b>40.36</b></td>
  </tr>
  <tr>
    <td>25</td>
    <td>52.41</td>
    <td>32.16</td>
    <td>41.25</td>
    <td>38.07</td>
  </tr>
  <tr>
    <td>32</td>
    <td>52.22</td>
    <td>38.80</td>
    <td>41.37</td>
    <td>38.10</td>
  </tr>
</table>

Table 5: Accuracy of LoVER on GSM8K test set and MMLU-Pro using hidden states from different mistral-7b layers.

<table>
  <tr>
    <th>max_op</th>
    <th>Majority Voting</th>
    <th>LoVER</th>
  </tr>
  <tr>
    <td>2</td>
    <td>55.0</td>
    <td>67.0 (+21.8%)</td>
  </tr>
  <tr>
    <td>4</td>
    <td>40.0</td>
    <td>47.0 (+17.5%)</td>
  </tr>
  <tr>
    <td>8</td>
    <td>25.0</td>
    <td>30.0 (+20.0%)</td>
  </tr>
  <tr>
    <td>16</td>
    <td>18.0</td>
    <td>21.0 (+16.7%)</td>
  </tr>
</table>

Table 6: Accuracy of LoVER on iGSM with different difficulty over mistral-7b. A higher max_op indicates greater difficulty. The values in parentheses indicate the percentage accuracy gain over the baseline.

the baseline across varying levels of problem difficulty, with no significant decline in performance gains as the difficulty increases. This is attributed to LoVER’s effective utilization of logical rules to harness the internal knowledge of LLMs, suggests that LoVER exhibits greater robustness in handling complex reasoning problems.

3.2.5 The effect of numbers of solutions per question

![An accuracy comparison of LoVER and baselines across different numbers of solutions on GSM8K over mistral-7b.](page_1012_1042_377_246.png)

Figure 2: An accuracy comparison of LoVER and baselines across different numbers of solutions on GSM8K over mistral-7b.

LoVER maintains high reasoning accuracy regardless of N. The reasoning accuracy of all methods increases as the number of solutions grows shown in Table 2. Across different numbers of solutions, LoVER consistently outperform Majority Voting, achieving an average absolute accuracy gain of 6.4%. As the number of solutions increases (> 5), LoVER surpasses CoT-Decoding, delivering an average absolute accuracy gain of 3.9%. Compared to CoT-Decoding, LoVER extracts richer information from hidden states, enabling more effective utilization of the diverse solutions sampled during the decoding process.

3.2.6 The effect of model scales

LoVER enhances reasoning accuracy across model scales. Figure 3 shows that LoVER en-
hances reasoning accuracy across different model scales over the Qwen-2.5 family. LoVER enables a 7B-parameter LLM to achieve reasoning accuracy comparable to that of a 32B-parameter LLM. LoVER achieves an average accuracy gains of 1.7% across five models with varying parameter sizes and consistently outperforms the baselines. In contrast, CoT-Decoding and Majority Voting exhibit comparable performance.

3.3 OOD Generalization

<table>
  <tr>
    <th>Datasets</th>
    <th>Voting</th>
    <th>CoT-D</th>
    <th>LoVER</th>
  </tr>
  <tr>
    <td>Boolean Exp</td>
    <td>61.6</td>
    <td><b>72.4</b></td>
    <td><b>72.4</b></td>
  </tr>
  <tr>
    <td>Web of Lies</td>
    <td>25.6</td>
    <td>39.6</td>
    <td><b>44.8</b></td>
  </tr>
  <tr>
    <td>Object Cnt</td>
    <td>46.0</td>
    <td>48.0</td>
    <td><b>48.4</b></td>
  </tr>
  <tr>
    <td>Navigate</td>
    <td>49.6</td>
    <td>57.2</td>
    <td><b>57.6</b></td>
  </tr>
  <tr>
    <td>Arithmetic</td>
    <td>12.0</td>
    <td>11.6</td>
    <td><b>12.4</b></td>
  </tr>
  <tr>
    <td>Causal Jud</td>
    <td>45.9</td>
    <td><b>60.4</b></td>
    <td><b>60.4</b></td>
  </tr>
</table>

Table 7: Reasoning accuracy of LoVER on six OOD datasets over mstral-7b. Voting stands for "Majority Voting"; CoT-D stands for "CoT-Decoding". Boolean Exp stands for "Boolean Expression" dataset; Object Cnt stands for "Object Counting" dataset; Arithmetic stands for "Multi-Step Arithmetic Two" dataset; Causal Jud stands for "Causal Judgment" dataset.

The improvements brought by LoVER can be transferred to OOD problems. We select six distinct datasets from BIG-Bench Hard as OOD datasets. As shown in table 7, LoVER achieves higher or equal accuracy than CoT-Decoding across six all datasets, and outperforms Majority Voting on all datasets, with an average accuracy improvement of 2.5%. This demonstrates the strong OOD generalization capability of LoVER. By leveraging the latent knowledge and logical constraints, LoVER learns patterns for evaluating assertion correctness, independent of the domain-specific context of the assertions.

4 Related Work

LLM Reasoning Existing research on LLM reasoning can be roughly divided into two categories: extrinsic reasoning and intrinsic reasoning. Extrinsic reasoning mainly involves complex prompt engineering (Wei et al., 2022; Yao et al., 2024), verifier based on outcome or processes (Cobbe et al., 2021; Lightman et al., 2023), and customized search algorithms (e.g., A*, MCTS) (Zhuang et al.; Wan et al., 2024). Research in this domain focuses on various reasoning tasks such as mathematical reasoning (Cobbe et al., 2021; Hendrycks et al.), logical reasoning (Liu et al., 2020), common-sense reasoning (Yang et al., 2018), and more (Liu et al., 2024; Yuan et al., 2024; Chen et al., 2022b). Intrinsic reasoning seeks to explore the model’s internal knowledge, primarily through the observation and manipulation of its hidden layers or output probabilities. The probe method involves using auxiliary classifiers or probes to analyze and interpret the internal representations learned by the model, offering insights into its understanding and reasoning processes (Belinkov, 2022; Alain and Bengio, 2016). Unlike supervised probes, CCS (Burns et al., 2023) learns a linear classifier to uncover latent knowledge in an unsupervised manner, while CoT-Decoding (Wang and Zhou, 2024) assesses the truth value of candidate solutions based on answer confidence. However, the main drawbacks are that CoT-Decoding is essentially an expert-curated heuristic rule, and CCS is limited to Yes-No questions, lacking scalability. LoVER belongs to the second category and is applicable to general reasoning tasks that do not require supervised data. It can be seen as an unsupervised probe (verifier) guided by logic rule.

Neural Logical Reasoning Neural logic integrates neural networks with logical reasoning to enhance model’s interpretability, consistency, and reasoning capabilities. One paradigm involves learning logical operators such as AND, OR, and NOT as differentiable neural modules, guided by self-supervised logic regularization (Shi et al., 2020). Prior studies have demonstrated its effectiveness in proof generation (Sun et al., 2021), fact checking (Chen et al., 2022a), NLI (Li et al., 2019) and recommender systems (Chen et al., 2021). Another standard method is based on the variational EM
framework (Ru et al., 2021; Qu and Tang, 2019; Zhou et al., 2020). LoVER draws inspiration from both lines of work. We represent the output of the verifier as binary latent variables, which are regularized with soft logic.

5 Conclusion

We propose LoVER, an unsupervised verifier regularized by logical rules for enhancing LLMs’ reasoning capability. We design three logical rules to guide LoVER in effectively leveraging unlabeled data, achieving performance comparable to supervised methods. LoVER is compatible with any white-box LLMs and adaptable to diverse reasoning tasks. Experiments show LoVER significantly enhances the reasoning accuracy of LLMs while demonstrating strong OOD generalization capabilities.

Limitations

LoVER relies on the hidden states of LLMs, which inherently restricts its applicability to white-box LLMs. This dependency prevents LoVER from being directly utilized in black-box LLMs scenarios. Although LoVER does not rely on extracting final answers from responses and can be applied to scenarios where responses lack explicit conclusions, we do not conduct experiments to explore this aspect in this paper.

Acknowledgments

The authors wish to thank the reviewers for their helpful comments and suggestions. This work was supported by NSFC grant(No.62136002 and 62477014), Ministry of Education Research Joint Fund Project(8091B042239), and Fundamental Research Funds for the Central Universities.

References

Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644.

Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silvia Paun, Barbara Plank, Massimo Poesio, Alexandra Uma, et al. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st workshop on benchmarking: past, present and future, pages 15–21. Association for Computational Linguistics.

Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219.

Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations.

Hanxiong Chen, Shaoyun Shi, Yunqi Li, and Yongfeng Zhang. 2021. Neural collaborative reasoning. In Proceedings of the Web Conference 2021, pages 1516–1527.

Jiangjie Chen, Qiaoben Bao, Changzhi Sun, Xinbo Zhang, Jiaze Chen, Hao Zhou, Yanghua Xiao, and Lei Li. 2022a. Loren: Logic-regularized reasoning for interpretable fact verification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10482–10491.

Jiangjie Chen, Rui Xu, Ziquan Fu, Wei Shi, Zhongqiao Li, Xinbo Zhang, Changzhi Sun, Lei Li, Yanghua Xiao, and Hao Zhou. 2022b. E-kar: A benchmark for rationalizing natural language analogical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3941–3955.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Aaron Grattafori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019. A logic-driven framework for consistency of neural models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3924–3935.

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.

Jian Liu, Leyang Cui, Hannmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124.

Yufang Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, and Aimin Zhou. 2024. Investigating and mitigating object hallucinations in pretrained vision-language (clip) models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18288–18301.

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.

Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meyssam Chenaghlou, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. 2024. arXiv preprint arXiv:2402.06196.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Meng Qu and Jian Tang. 2019. Probabilistic logic neural networks for reasoning. Advances in neural information processing systems, 32.

Qwen., An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115.

Dongyu Ru, Changzhi Sun, Jiangtao Feng, Lin Qiu, Hao Zhou, Weinan Zhang, Yong Yu, and Lei Li. 2021. Learning logic rules for document-level relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1239–1250.

Shaoyun Shi, Hanxiong Chen, Weizhi Ma, Jiaxin Mao, Min Zhang, and Yongfeng Zhang. 2020. Neural logic reasoning. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1365–1374.

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.

Changzhi Sun, Xinbo Zhang, Jiangjie Chen, Chun Gan, Yuanbin Wu, Jiaze Chen, Hao Zhou, and Lei Li. 2021. Probabilistic graph reasoning for natural proof generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3140–3151.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.

Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2024. Alphazero-like tree-search can guide large language model decoding and training. In Forty-first International Conference on Machine Learning.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.

Xuezhi Wang and Denny Zhou. 2024. Chain-of-thought reasoning without prompting. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyuan Jiang, et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Jin Xu, Mariët Theune, and Daniel Braun. 2024a. Leveraging annotator disagreement for text classification. In Proceedings of the 7th International Conference on Natural Language and Speech Processing (IC-NLSP 2024), pages 1–10.

Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024b. A survey on knowledge
distillation of large language models. arXiv preprint arXiv:2402.13116.

Yinfei Yang, Oshin Agarwal, Chris Tar, Byron C Wallace, and Ani Nenkova. 2019. Predicting annotation difficulty to improve task routing and model performance for biomedical information extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1471–1480.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36.

Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. 2024. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. In The Thirteenth International Conference on Learning Representations.

Fei Yu, Anningzhe Gao, and Benyou Wang. 2024. Ovm, outcome-supervised value models for planning in mathematical reasoning. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 858–875.

Siyu Yuan, Jiangjie Chen, Changzhi Sun, Jiaqing Liang, Yanghua Xiao, and Deqing Yang. 2024. Analogykb: Unlocking analogical reasoning of language models with a million-scale knowledge base. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1249–1265.

Wangchunshu Zhou, Jinyi Hu, Hanlin Zhang, Xiaodan Liang, Maosong Sun, Chenyan Xiong, and Jian Tang. 2020. Towards interpretable natural language understanding with explanations as latent variables. Advances in Neural Information Processing Systems, 33:6803–6814.

Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang. Toolchain*: Efficient action space navigation in large language models with a* search. In The Twelfth International Conference on Learning Representations.
A  Logic-based inter-group Consistency

The basic idea is to transform Eq. 1 into a logical expression and then apply product t-norms to relax the logic (Chen et al., 2022a; Li et al., 2019).

\[
r = (z_1^+ \land \neg z_2^+ \land \cdots \land \neg z_M^+) \\
\lor (\neg z_1^+ \land z_2^+ \land \cdots \land \neg z_M^+) \\
\vdots \\
\lor (\neg z_1^+ \cdots \land z_M^+),
\]
\( \mathcal{L}_{\text{inter}}^{\text{logic}} = \text{t-norms}(r) \).

The accuracy comparison of LoVER using using different kinds of \( \mathcal{L}_{\text{inter}} \) are provided in Table 8. LoVER using default \( \mathcal{L}_{\text{inter}} \) is better.

<table>
  <tr>
    <th></th>
    <th>GSM8K</th>
    <th>sum</th>
    <th>max</th>
  </tr>
  <tr>
    <td>default \( \mathcal{L}_{\text{inter}} \)</td>
    <td>53.14</td>
    <td>50.41</td>
    <td></td>
  </tr>
  <tr>
    <td>logic-based \( \mathcal{L}_{\text{inter}} \)</td>
    <td>52.99</td>
    <td></td>
    <td>46.39</td>
  </tr>
</table>

Table 8: Reasoning accuracy of LoVER on GSM8K test set over mstral-7b using different kinds of \( \mathcal{L}_{\text{inter}} \).

B  Datasets

B.1 HotpotQA

HotpotQA is a Wikipedia-based question-answering dataset. In the full wiki setting, HotpotQA consists of 90,447 training samples, 7,405 validation samples, and 7,405 test samples. A key feature of HotpotQA is that these questions require finding and reasoning over multiple supporting documents to answer.

Since the test set does not provide standard answers, we use the validation set as the test set. To make the extraction of final answers from response easier and more accurate, we only selected questions with "yes" or "no" as answers for our experiments. This resulted in 5,481 training samples and 458 test samples. Table 9 provides examples of questions with "yes" and "no" answer. We used the entire test set and a randomly selected subset of 3,000 training samples for the experiments.

In our experiments, we only utilized the questions and did not incorporate the related supporting texts.

B.2 MMLU pro

MMLU-Pro is an advanced benchmark for assessing language models on more extensive and challenging tasks. It consists of over 12,000 questions, each furnished with ten possible answers, spanning 14 distinct domains. These domains include Biology, Business, Chemistry, Computer Science, Economics, Engineering, Health, History, Law, Math, Philosophy, Physics, Psychology, and Others.

MMLU-Pro provides over 12,000 test samples and 70 validation samples, with no training set included. Therefore, we randomly selected one-third (4 out of 14) of the domains to serve as the test set, while the remaining domains were used for training and validation. With the random seed set to 0, we obtained data from the following four domains as the test set: Computer Science, Physics, History, and Biology. The statistics of test set is shown in Tabel 10.

<table>
  <tr>
    <th>Datasets</th>
    <th>#Data</th>
  </tr>
  <tr>
    <td>Computer Science</td>
    <td>410</td>
  </tr>
  <tr>
    <td>Physics</td>
    <td>1299</td>
  </tr>
  <tr>
    <td>History</td>
    <td>381</td>
  </tr>
  <tr>
    <td>Biology</td>
    <td>717</td>
  </tr>
  <tr>
    <td>All</td>
    <td>2807</td>
  </tr>
</table>

Table 10: The statistics of the four test datasets from MMLU-Pro.

B.3 BIG-Bench Hard

We randomly selected six datasets from BIG-Bench Hard as out-of-distribution (OOD) datasets, with the specific dataset names and corresponding statistics summarized in Table 11.

B.4 iGSM

We construct a more challenging mathematics dataset called iGSM. Following iGSM(Ye et al., 2024), we control problems difficulty by setting different numbers of operations (\( \textbf{max\_op} \)) in the solutions. We utilized the iGSM synthetic data

HotpotQA

Question: Were Scott Derrickson and Ed Wood of the same nationality?
Answer: yes

Question: Were Pavel Urysohn and Leonid Levin known for the same type of work?
Answer: no

Table 9: Examples of HotpotQA dataset with "yes" and "no" answer.
<table>
  <tr>
    <th>Datasets</th>
    <th>#Data</th>
  </tr>
  <tr>
    <td>Boolean Expression</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Web of Lies</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Object Counting</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Navigate</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Multi-Step Arithmetic Two</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Causal Judgment</td>
    <td>187</td>
  </tr>
  <tr>
    <td>All</td>
    <td>1437</td>
  </tr>
</table>

Table 11: The statistics of the six OOD datasets from BIG-Bench Hard.

generator\footnote{https://github.com/facebookresearch/iGSM} to construct the dataset by setting random\_seed to 42, max\_edge to 12, perm\_level to 5, detail\_level to 0, and max\_op to 2, 4, 8, and 16, respectively. The detailed data statistics are presented in Table 12.

<table>
  <tr>
    <th>max_op</th>
    <th>test</th>
    <th>training</th>
    <th>#Data</th>
  </tr>
  <tr>
    <td>2</td>
    <td>100</td>
    <td>-</td>
    <td></td>
  </tr>
  <tr>
    <td>4</td>
    <td>100</td>
    <td>-</td>
    <td></td>
  </tr>
  <tr>
    <td>8</td>
    <td>100</td>
    <td>1500</td>
    <td></td>
  </tr>
  <tr>
    <td>16</td>
    <td>100</td>
    <td>-</td>
    <td></td>
  </tr>
  <tr>
    <td>All</td>
    <td>400</td>
    <td>1500</td>
    <td></td>
  </tr>
</table>

Table 12: The statistics of the iGSM test set in our experiments.

Table 14 provides two examples of iGSM where max\_op is 2 and 16 respectively.

iGSM: max\_op=2

Question: The number of each Goldfish’s Proximal Phalanx equals 8. The number of each Swordfish’s Metacarpal I equals the sum of each Goldfish’s Bone, each Goldfish’s Proximal Phalanx and each Eel’s Bone. The number of each Mahi Mahi’s Radial Carpal equals 3. The number of each Goldfish’s Metacarpal IV equals 12. How many Radial Carpal does Mahi Mahi have?

Solution: Define Mahi Mahi’s Radial Carpal as n; so n = 3.
Answer: 3

Table 13: Examples of iGSM dataset with different max\_op.

iGSM: max\_op=16

Question: The number of each Spinal Cord’s Transitional Epithelial Cells equals the difference of each Cerebellum’s Hepatocytes and each Albatross’s Spinal Cord. The number of each Cerebellum’s Hepatocytes equals each Parrot’s Organs. The number of each Albatross’s Spinal Cord equals each Cerebellum’s Hepatocytes. The number of each Spinal Cord’s Hepatocytes equals the sum of each Cerebellum’s Hepatocytes, each Albatross’s Spinal Cord and each Parrot’s Cerebellum. The number of each Parrot’s Cerebellum equals 6. How many Cells does Albatross have?

Solution: Define Parrot’s Cerebellum as c; so c = 6. Define Parrot’s Organs as i; so i = c = 6. Define Cerebellum’s Hepatocytes as X; so X = i = 6. Define Albatross’s Spinal Cord as u; so u = X = 6. Define Spinal Cord’s Hepatocytes as S; b = X + u = 6 + 6 = 12; so S = b + c = 12 + 6 = 18. Define Spinal Cord’s Transitional Epithelial Cells as F; so F = X - u = 6 - 6 = 0. Define Spinal Cord’s Cells as W; so W = S + F = 18 + 0 = 18. Define Albatross’s Cells as o; so o = u * W = 6 * 18 = 16..
Answer: 16

Table 14: Examples of iGSM dataset with different max\_op.

B.5 GSM8K

GSM8K consists of 8,792 high quality grade school math problems, with 7,473 in the training set and 1,319 in the test set. GSM8K is designed to evaluate the mathematical reasoning capabilities of models. In our experiments, we select 7,000 samples randomly from the training set for training, 473 samples from the training set as the validation set, and the entire test set for testing.
C  Pseudocode (PyTorch-like)

"""Step 1: Sample n responses from LLM"""
def sample_responses(llvm, prompt, n):
    responses = []
    for _ in range(n):
        response = model.sample(prompt)
        responses.append(response)
    return responses

"""Step 2: Create contrastive assertions"""
def create_assertions(question, responses):
    pos_assertions = []
    neg_assertions = []
    for response in responses:
        qa_pair = f"Q:{question} A:{response}"
        pos_assertion= qa_pair+"this is a true answer."
        neg_assertion= qa_pair+"this is a false answer."
        pos_assertions.append(positive_assertion)
        neg_assertions.append(negative_assertion)
    return pos_assertions, neg_assertions

"""Step 3: Get hidden states of assertions"""
def get_hidden_states(llvm, assertions, layer_idx):
    representations = []
    for assertion in assertions:
        outputs = llvm(assertion, output_hidden_states)
        hidden_state = outputs.hidden_states[layer_idx]
        hidden_state = hidden_state[:, -1, :]
        representations.append(hidden_state)
    return representations

"""Step 4: Compute scores using verifier"""
def compute_scores(verifier, pos_reps, neg_reps):
    pos_scores = verifier(pos_reps)
    neg_scores = verifier(neg_reps)
    final_scores = 0.5 * (pos_scores+(1-neg_scores))
    return final_scores

"""Step 5a: Select final answer by Max strategy"""
def select_by_max(responses, scores):
    max_idx = argmax(scores)
    best_response = responses[max_idx]
    return extract_final_answer(best_response)

"""Step 5b: Select final answer by Sum strategy"""
def select_by_sum(responses, scores):
    answer_scores = {}
    for response, score in zip(responses, scores):
        final_answer = extract_final_answer(response)
        answer_scores[final_answer] += score
    best_answer = find_key_by_max_value(answer_scores)
    return best_answer

if __name__ == "__main__":
    # Step 1: Generate responses
    prompt = construct_prompt(q)
    responses = sample_responses(llvm, prompt, n)

    # Step 2: Create contrastive assertions
    pos_asserts, neg_asserts=create_assertions(q, responses)

    # Step 3: Get hidden representations
    layer = 20
    pos_reps = get_hidden_states(llvm, pos_asserts, layer)
    neg_reps = get_hidden_states(llvm, neg_asserts, layer)

    # Step 4: Compute verification scores
    scores = compute_scores(verifier, pos_reps, neg_reps)

    # Step 5: Select final answer based on strategy
    final_answer_max=select_by_max(responses, scores)
    final_answer_sum=select_by_sum(responses, scores)

Reviewer paper summaries: The paper introduces LOVER, an unsupervised verifier designed to enhance the reasoning capabilities of LLM. Unlike traditional supervised verifiers, LOVER uses latent knowledge from LLMs along with logical constraints (negation, intra-group, and inter-group consistency) to identify the correctness of reasoning paths without needing labeled data. The core idea is to model the truthfulness of assertions through binary latent variables guided by soft logic-based regularization. Experiments conducted on mathematical and open-domain reasoning tasks demonstrate that LOVER outperforms unsupervised baselines (e.g., CoT-Decoding) and achieves results comparable (approximately 95%) to supervised methods, as well as showcasing strong out-of-distribution (OOD) generalization.
Summary of strengths: 1. The paper is well-written and clearly structured, making it easy for readers to follow 
2. The method is well-motivated and innovative. The authors build upon existing unsupervised approaches, specifically identifying limitations in the baseline method, CoT-Decoding, through pilot experiments. They observe that CoT-Decoding's performance heavily depends on the choice of backbone LLM, which restricts its applicability and robustness. Motivated by these insights, the authors propose a new unsupervised probabilistic verifier, which leverages logical rules as regularization constraints.
3. The three logical constraints are well-defined and reasonable.
4. The experiments are comprehensive and the analysis is insightful. The authors also perform an extensive series of ablation studies to validate the design choices behind their proposed logical constraints, providing a deeper understanding to their approach. Overall, the results are promising and convincingly demonstrate the benefits of their method.
Summary of weaknesses: 1. **Clarify Computation Efficiency**: While data efficiency is mentioned in the Appendix, further discussing inference-time compute of LOVER to its performance could further emphasize practical benefits. For example, whether applying majority voting on LOVER’s results could increase the accuracy?
2. **Experiments with Larger Models**: The authors could consider extending the evaluation of LOVER to larger models, beyond the current largest size of 8B. Although a verifier might benefit more smaller models, it would be interesting to explore how a larger LOVER architecture (e.g., using an MLP with more layers or neurons) could potentially enhance even larger-scale models (e.g., 70B). Such experiments would clarify LOVER's potential effectiveness in large-scale deployments.
Comments, suggestions and typos: Line 97 “intrisic” -> “intrinsic”
Line 362, “hightest” -> “highest”
Overall assessment scores: 4
Metareview score: 3.5
Metareview: This paper presents LOVER, an unsupervised verifier designed to improve the reasoning abilities of large language models (LLMs). LOVER uses logical constraints—negation consistency, intra-group consistency, and inter-group consistency—as soft probabilistic objectives to guide training without labeled data. It leverages LLM internal activations and contrastive assertions built from reasoning paths to train a binary latent variable model. Compatible with various LLMs, LOVER shows strong performance and outperforms unsupervised baselines across various datasets.
Summary of reasons to publish: 1. The paper is well-written and structured
2. The paper extends CCS elegantly to allow signals beyond yes/no questions for the verifier training, the method is novel and reasonable
3. Extensive experiments show the effectiveness of the method