Score Consistency Meets Preference Alignment: Dual-Consistency for Partial Reward Modeling
Authors: Bin Xie, Bingbing Xu, Yige Yuan, Shengmao Zhu, Huawei Shen
Accepted at: acl25

From Outcomes to Processes:
Guiding PRM Learning from ORM for Inference-Time Alignment

Bin Xie1,2, Bingbing Xu1*, Yige Yuan1,2, Shengmao Zhu1,2, Huawei Shen1,2,
1State Key Laboratory of AI Safety, Institute of Computing Technology, CAS
2University of Chinese Academy of Sciences
{xiebin23s@ict.ac.cn, xubingbing@ict.ac.cn}

Abstract
Inference-time alignment methods have gained significant attention for their efficiency and effectiveness in aligning large language models (LLMs) with human preferences. However, existing dominant approaches using reward-guided search (RGS) primarily rely on outcome reward models (ORMs), which suffer from a critical granularity mismatch: ORM s are designed to provide outcome rewards for complete responses, while RGS methods rely on process rewards to guide the policy, leading to inconsistent scoring and suboptimal alignment. To address this challenge, we introduce process reward models (PRMs) into RGS and argue that an ideal PRM should satisfy two objectives: Score Consistency, ensuring coherent evaluation across partial and complete responses, and Preference Consistency, aligning partial sequence assessments with human preferences. Based on these, we propose SP-PRM, a novel dual-consistency framework integrating score consistency-based and preference consistency-based partial evaluation modules without relying on human annotation. Extensive experiments on dialogue, summarization, and reasoning tasks demonstrate that SP-PRM substantially enhances existing RGS methods, achieving a 3.6%–10.3% improvement in GPT-4 evaluation scores across all tasks. Code is publicly available at this link.

1 Introduction
Large language models (LLMs), trained on extensive text corpora, demonstrate strong performance across a range of natural language processing tasks (Achiam et al., 2023; Touvron et al., 2023; Liu et al., 2024a). However, they often exhibit misalignment with human preferences (Gehman et al., 2020; Ouyang et al., 2022; Bai et al., 2022; Deshpande et al., 2023). Post-training alignment methods, such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), incur substantial computational costs and typically require retraining. Inference-time alignment emerges as a promising alternative, enabling flexible adaptation to diverse objectives with minimal computational overhead (Wang et al., 2024; Ji et al., 2024).

Reward-guided search (RGS) has emerged as a dominant inference-time alignment framework. Best-of-N (Stiennon et al., 2020), a representative approach, generates N candidate responses and selects the optimal one using a reward model (RM). Although effective for improving text quality (Nakano et al., 2021; Touvron et al., 2023), increasing N introduces prohibitive inference latency and memory costs (Sun et al., 2024). Recent work explores process rewards during generation, such as token-, chunk-, or sentence-level rewards. For example, ARGs (Khanov et al., 2024) computes token-wise rewards and integrates them into logits to determine the next token. Other methods (Zhou et al., 2024; Li et al., 2024) extend this idea to other segments, using direct RM scores or log-probability differences between tuned and untuned

![Top: Inaccurate rewards of partial sequences resulting in misaligned responses via reward-guided search method. Bottom: Existing ORM s lack score consistency.](page_1012_370_388_312.png)

Figure 1: (Top) Inaccurate rewards of partial sequences resulting in misaligned responses via reward-guided search method. (Bottom) Existing ORM s lack score consistency.
Figure 2: Empirical Analysis of Reward Model Behaviors: (a) Existing ORMs maintain strong correlation with human preferences; (b) Score consistency may impair semantic understanding; (c) Existing ORMs exhibit length-dependent evaluation confidence.

language models.

However, the RMs employed in the aforementioned methods are outcome reward models (ORMs), which are specifically trained and designed to evaluate the quality of complete responses. In practice, RGS methods rely on process rewards to guide the policy. While ORMs can technically accept partial sequences as input to derive process rewards, this approach results in a potential granularity mismatch problem (Xu et al., 2024). Specifically, as illustrated in the top panel of Figure 1, ORMs produce inaccurate rewards when evaluating partial sequences, leading to suboptimal token selections and ultimately resulting in misaligned responses.

To combat the above challenge, we introduce a process reward model (PRM) in RGS and propose that an ideal PRM should satisfy the following two objectives: (1) Score Consistency, which requires the PRM to assign consistent scores between complete and partial sequences (i.e., complete sequences with high scores should have correspondingly high-scoring partial subsequences, and vice versa). We demonstrate that this property enables RGS methods to generate optimal outputs, while empirical experiments reveal that original ORMs lack this property (see the bottom panel of Figure 1). (2) Preference Consistency, which requires the PRM to align with human preferences when evaluating partial sequences. Since responses that only satisfy score consistency may contain segments misaligned with human preferences (see Figure 2b), this could compromise their semantic understanding capabilities and lead to biases toward specific patterns While score consistency drives the PRM to optimize for better complete responses, preference consistency preserves semantic understanding capability, thereby yielding high-quality outputs.

To achieve these objectives, we propose SP-PRM, a novel dual-consistency framework that induces a PRM from an ORM. It comprises two core modules: score consistency-based partial evaluation and preference consistency-based partial evaluation. Specifically, the score consistency module addresses the granularity mismatch inherent in ORMs by deconstructing complete responses into partial sequences and implementing reward modeling based on the Bradley-Terry model. This enables the RM to predict cumulative future rewards from intermediate states, effectively capturing long-term dependencies. The preference consistency module aligns PRM rewards for partial sequences with human preferences. As Figure 2a illustrated, strong RMs show high human-preference consistency (approximated using GPT-4 and DeepSeek-V3). Leveraging this insight, we employ an RM as a reference model to compute partial-sequence entropy, reweighting their their contribution to the training process. This prioritizes sequences that better reflect human preferences, thereby enhancing alignment. Built upon these two modules, SP-PRM derives its guidance from the ORM without human annotation, while simultaneously anticipating long-term alignment from partial contexts and maintaining human preference consistency, thus preventing local pattern overfitting and partial-complete response inconsistencies.

We conduct extensive evaluations on three tasks, including dialogue generation, text summarization, and complex reasoning, and apply our approach to model architectures ranging from 1B to 3B parameters. The results demonstrate that SP-PRM substantially enhances existing RGS methods, achieving a 3.6%–10.3% improvement in GPT-4 evaluation scores across all tasks.
Figure 3: Overview of the SP-PRM Framework: Score Consistency-based and Preference Consistency-based partial Evaluation.

2 Preliminaries

In this section, we review the process of reward modeling and the general reward-guided search framework.

2.1 Reward Modeling

We typically train a reward model on a preference dataset \( \mathcal{D} \). Each sample in \( \mathcal{D} \) is represented as a triplet \((x, y^w, y^l)\), where \(x\) is the prompt, and \(y^w\) and \(y^l\) represent two distinct responses. Compared to \(y^l\), the response \(y^w\) is more aligned with human preferences. Following the Bradley and Terry (1952) and Ouyang et al. (2022), the loss function is defined as:

\[
\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y^w, y^l) \sim \mathcal{D}} \log \left( \sigma(r_\theta(x, y^w) - r_\theta(x, y^l)) \right)
\]

where \( \sigma \) is the sigmoid function.

2.2 Reward-guided Search

Reward-guided search is a popular framework in inference-time alignment. Given a prompt \(x\), at each step, the language model \( \pi_\theta \) generates \(N\) candidate segments (tokens, chunks, sentences, or responses). The reward model then selects the top-\(k\) segments from these candidates. These selected segments are merged with the already generated prefix sequences. After repeating this process for multiple steps, \(k\) responses are ultimately yielded as the final generation results (see Algorithm 1 for details). However, the RMs employed in the aforementioned methods are outcome reward models (ORMs), which are specifically designed to evaluate the quality of complete responses. In practice, RGS methods rely on process rewards to guide the policy. This granularity mismatch leads to inconsistent scoring between partial and complete sequences.

3 Analysis and Motivation

In this section, we theoretically analyze the requirements that RGS imposes on reward models and experimentally verify whether existing ORMs satisfy these requirements.

3.1 Score Consistency Enables LMs to Generate Optimal Results via RGS

We start by defining score consistency and demonstrating that the RM possessing this property can effectively guide the generation of optimal responses, regardless of the granularity of the generation.

**Score Consistency:** A reward model \(r\) satisfies score consistency if and only if for any two sequences \(y^1\) and \(y^2\) (assume \(|y^1| = |y^2| = T\), if not, pad shorter sequences to the same length \(T\)), \( \forall t \in \{1, \ldots, T\} \), the following holds:

\[
r(x, y^1) \geq r(x, y^2) \Rightarrow r(x, y^1_{<t}) \geq r(x, y^2_{<t}).
\]

**Theorem 1.** *Given a prompt \(x\), if there exists an optimal response \(y^*\), which refers to a response achieving the highest score under \(r\), and \(r\) satisfies score consistency, it can guide the LM policy \(\pi\) to generate \(y^*\), regardless of generation granularity.*

*Proof.* We provide a detailed proof of token-level generation, the detailed chunk-level proof is in Appendix A. Analogously, the sentence- and response-level guidance yield identical results under score consistency.

The optimal response \(y^* = (y^*_1, \ldots, y^*_T)\) under \(r\) satisfies \(r(x, y^*) \geq r(x, y)\) for all \(y\). By *score consistency*:

\[
r(x, y^*_{<t}) \geq r(x, y_{<t}) \quad \forall t \leq \max(|y^*|, |y|).
\]

For sequences of different lengths, shorter sequences are padded to equal.

**Token-level generation:** Let \(y^*_{<t}\) be the prefix of \(t-1\) optimal tokens already chosen (for \(t=1\), \(y^*_{<1}\) is empty). At step \(t\), RGS chooses:

\[
\hat{y}_t = \arg\max_{y_t \in \mathcal{V}} r(x, y^*_{<t} \oplus y_t).
\]

The definition of score consistency is for any prefix length \(k-1\), which implies that if \(y^*\) is optimal
globally, any prefix of \( y^* \) is also optimal among all prefixes of the same length. Thus, for the current step, considering prefixes of length \( t \) that start with \( y_{<t}^* \):

\[
r(x, y_{<t}^* \oplus y_t^*) \geq r(x, y_{<t}^* \oplus y_t), \quad \forall y_t \in \mathcal{V}.
\]

This is because \( y_{<t}^* \oplus y_t^* \) is \( y_{<t+1}^* \), the optimal prefix of length \( t \). Therefore, \( \hat{y}_t = y_t^* \). By induction, the token-level RGS recovers \( y^* \).

Theorem 1 shows that score consistency can guide an LLM to what the reward model deems optimal. However, it may not align human preferences.

3.2 Observations

To evaluate whether existing ORMs satisfy score consistency (SC) and to analyze potential limitations of SC, we introduce the Agreement Rate (AR) metric. This metric measures the extent to which two evaluation metrics concur on the order relationship (i.e., which response is preferred) when assessing sample pairs within a preference dataset. Here, we provide a detailed definition of \( \mathrm{AR}_{\mathrm{RM-SC}}(t) \), \( \mathrm{AR}_{\mathrm{SC-HP}}(t) \) and \( \mathrm{AR}_{\mathrm{RM-HP}}(t) \) given in Appendix B.

\[
\mathrm{AR}_{\mathrm{RM-SC}}(t) = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\left[ r(x, y_{<t}^w) > r(x, y_{<t}^{y'}) \right]
\]

where \( \mathbb{I}[\cdot] \) denotes the indicator function, \( N \) is the number of evaluation samples, \( r(\cdot) \) is the reward model score, \( t \) is the prefix length.

Specifically, given a preference dataset \( \mathcal{D} = \{(x, y^w, y')\}_{i=1}^N \), where \( y^w \) is preferred over \( y' \), SC requires that for any prefix length \( t \), \( y_{<t}^w \) should also be preferred over \( y_{<t}' \). Thus, for a given reward model, sample \( (x, y^w, y') \), and prefix length \( t \), if the RM evaluation satisfies the SC requirement, the RM is considered consistent with SC for that instance. \( \mathrm{AR}_{\mathrm{RM-SC}} \) measures the proportion of cases where the RM and SC are consistent across all pairs. Ideally, if the RM fully satisfies SC, the agreement rate \( \mathrm{AR}_{\mathrm{RM-SC}} \) should be 100%.

We conducted experiments on the HH-RLHF dataset. In these experiments, we formed pairwise combinations of three distinct evaluation criteria. The results are presented in the bottom of Fig.1 and Fig.2.

Observation 1: Existing ORMs Lack Score Consistency. Fig. 1 reveals that ORMs achieve limited agreement with score consistency requirements—only 57% at 5 tokens, improving marginally to 60% at 50 tokens (\( \mathrm{AR}_{\mathrm{RM-SC}} \ll 100\% \)). This significant gap suggests potential myopic decoding decisions.

Observation 2: ORMs Maintain Strong Correlation with Human Preferences. Despite lacking score consistency, ORMs demonstrate robust agreement with human preferences (\( \mathrm{AR}_{\mathrm{RM-HP}} > 65\% \) across all prefix lengths, Fig. 2a). This indicates RMs’ potential as effective proxies for semantic understanding in reward modeling.

Observation 3: Score Consistency May Impair Semantic Understanding. Fig. 2b shows consistently low agreement rates (<45% at 5 tokens, <65% at 50 tokens) between human preferences and score consistency requirements. Given that human preferences reflect semantic understanding capability, this suggests that strict consistency optimization might compromise the RM’s semantic comprehension abilities.

Observation 4: ORMs Exhibit Length Dependent Evaluation Confidence. To analyze ORMs’ discriminative ability under partial observability, we introduce reward gap \( \Delta_r = |r(x, y_{<t}^w) - r(x, y_{<t}')| \). Larger gaps indicate higher RM confidence and lower evaluation difficulty. Fig. 2c shows \( \Delta_r \) increases with prefix length, with model capacity significantly affecting the rate of confidence gain—UltraRM-13B achieves 63% of maximum \( \Delta_r \) at t=15 tokens, while DeBERTa requires 35 tokens for comparable performance.

4 Methodology

Based on the theoretical analysis and experimental observations in Section 3, We propose SP-PRM, a novel dual-consistency framework comprising two core modules: score consistency-based partial evaluation and preference-based partial evaluation. Fig. 3 illustrates the overall framework.

4.1 Score Consistency Partial Evaluation

In this section, we construct dataset \( \mathcal{D}_{\text{partial}} = \{(x, y_{<t}^w, y_{<t}')\}_{i=1}^N \) by extracting incomplete sequences from preference dataset \( \mathcal{D} \), then perform reward modeling based on the Bradley-Terry model, which enhances the score consistency of the reward model.
4.1.1 Partial Sequence Dataset Construction
We propose two truncation approaches for constructing incomplete sequences from the preference dataset \( \mathcal{D} \), balancing training objective alignment with sample utilization efficiency.

Token-Level Truncation (TLT). We generate partial sequences at each token position to maintain strict score consistency:

\[
\mathcal{D}_{\text{partial}}^{\text{TLT}} = \bigcup_{t=1}^{T} \left\{ \left( x, y_{<t}^w, y_{<t}^l \right) \right\}
\]

where \( y_{<t}^w \) represents the \( t \)-token prefix of the preferred response. This comprehensive approach scales linearly with average response length. Such expansion either demands substantial computational resources or restricts sampling to under 5% of the original data, risking overfitting.

Stochastic Sampling Truncation (SST). To address the limitations, we develop an adaptive truncation strategy that optimizes sample utilization while mitigating overfitting. For each \( (y^w, y^l) \) pair:

1. Compute maximum valid length \( T = \max(|y^w|, |y^l|) \)
2. Sample \( k \) once from uniform distribution \( k \sim U(0, 2K) \)
3. Sample \( k \) times from uniform distribution \( t_i \sim U(1, T) \) to obtain \( t_1, \ldots, t_k \)
4. Generate partial pairs:

\[
\mathcal{D}_{\text{partial}}^{\text{SST}} = \bigcup_{i=1}^{k} \left\{ \left( x, y_{<t_i}^w, y_{<t_i}^l \right) \right\}
\]

This approach yields a dataset size dependent solely on hyperparameter \( K \), significantly improving sample utilization and mitigating overfitting.

4.1.2 Reward Modeling for Score Consistency
We train the reward model \( r_\theta \) using the partial sequence dataset \( \mathcal{D}_{\text{partial}} \). Following Eq. 1, we define the loss function as:

\[
\mathcal{L}_{\text{SC}} = -\mathbb{E}_{(x, y_{<t}^w, y_{<t}^l) \sim \mathcal{D}}
\log \left( \sigma(r_\theta(x, y_{<t}^w) - r_\theta(x, y_{<t}^l)) \right)
\] (2)

By minimizing this loss function, we obtain the reward model \( r_\theta^{\text{SC}} \) constrained by score consistency.

4.2 Preference-based Partial Evaluation
The empirical analysis in Fig.2b demonstrates that optimizing solely for score consistency can degrade the semantic capabilities of the RM. However, Fig.2a shows that large reward models typically trained on datasets with full responses exhibit higher agreement rates with human preferences. Therefore, we introduce such a reward model as a reference reward model \( r_\phi \) to constrain the optimization of \( r_\theta \), maintaining human preference alignment while optimizing for score consistency. Specifically, when the evaluations of score consistency and \( r_\phi \) for the sample \( (x, y_{<t}^w, y_{<t}^l) \) align, we consider it to represent a good balance between human preference and score consistency, retaining the sample. Otherwise, the sample is removed from \( \mathcal{D}_{\text{partial}} \). Notably, we also assign different sample weights based on RM’s confidence in its evaluation results, as detailed below.

For an incomplete sequence \( y_{<t} \), longer prefixes typically contain richer semantic information, which reduces the evaluation difficulty for the reward model, corresponding to higher confidence. We hypothesize that this is due to the reduced uncertainty in future tokens, which is often measured by Shannon entropy. Section 3.2 shows that longer sequences lead to a greater reward gap. Incorporating these insights, we use \( r_\phi \) to calculate the entropy of the reward gap for the sample \( (x, y_{<t}^w, y_{<t}^l) \), thereby obtaining the confidence in the evaluation. Specifically, we first normalize RM’s scores for \( (x, y_{<t}^w) \) and \( (x, y_{<t}^l) \) into a probability distribution and calculate their Shannon entropy:

\[
p_t^w = \sigma(|r_\phi(x, y_{<t}^w) - r_\phi(x, y_{<t}^l)|)
p_t^l = 1 - p_t^w
H_t = -(p_t^w \log p_t^w + p_t^l \log p_t^l)
\]

Higher entropy corresponds to a smaller reward gap, which typically occurs for shorter prefixes, leading to lower confidence and thus lower weights, and vice versa. Specifically, for samples violating score consistency, we remove them from \( \mathcal{D}_{\text{partial}} \), assigning them a weight of zero. Formally,

\[
w_t = \begin{cases}
1/H_t & \text{if } r_\phi(x, y_{<t}^w) > r_\phi(x, y_{<t}^l) \\
0 & \text{otherwise}
\end{cases}
\]

The final reward model \( r_\theta(x, y) \) is trained using a modified Bradley-Terry objective that integrates partial and complete sequence scoring:
<table>
  <tr>
    <th rowspan="2">Model (→)</th>
    <th colspan="5">Llama-3-2B-Instruct</th>
    <th colspan="5">Llama-3-8B-Instruct</th>
  </tr>
  <tr>
    <th>Method (↓)</th>
    <th>Reward (↑)</th>
    <th>Div. (↑)</th>
    <th>Coh. (↑)</th>
    <th>Win-tie (↑)</th>
    <th>Reward (↑)</th>
    <th>Div. (↑)</th>
    <th>Coh. (↑)</th>
    <th>Win-tie (↑)</th>
  </tr>
  <tr>
    <td>Base</td>
    <td></td>
    <td>2.35 (± 0.15)</td>
    <td>0.76</td>
    <td>0.60</td>
    <td>50.00</td>
    <td>2.61 (± 0.23)</td>
    <td>0.77</td>
    <td>0.63</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>ARGS-G</td>
    <td></td>
    <td>2.51 (± 0.13)</td>
    <td>0.73</td>
    <td>0.60</td>
    <td>56.33</td>
    <td>2.72 (± 0.22)</td>
    <td>0.73</td>
    <td>0.61</td>
    <td>52.67</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td></td>
    <td>2.60 (± 0.24)</td>
    <td>0.79</td>
    <td><b>0.61</b></td>
    <td>57.00</td>
    <td>2.85 (± 0.25)</td>
    <td>0.80</td>
    <td><b>0.62</b></td>
    <td>55.33</td>
  </tr>
  <tr>
    <td>+Ablation</td>
    <td></td>
    <td>2.37 (± 0.22)</td>
    <td>0.70</td>
    <td>0.58</td>
    <td>46.00</td>
    <td>2.62 (± 0.23)</td>
    <td>0.71</td>
    <td>0.60</td>
    <td>47.67</td>
  </tr>
  <tr>
    <td>TBS</td>
    <td></td>
    <td>2.65 (± 0.20)</td>
    <td>0.86</td>
    <td>0.57</td>
    <td>61.67</td>
    <td>3.08 (± 0.27)</td>
    <td>0.82</td>
    <td>0.61</td>
    <td>59.00</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td></td>
    <td><b>2.86 (± 0.19)</b></td>
    <td><b>0.88</b></td>
    <td>0.59</td>
    <td><b>62.33</b></td>
    <td><b>3.12 (± 0.21)</b></td>
    <td><b>0.87</b></td>
    <td>0.61</td>
    <td><b>61.33</b></td>
  </tr>
  <tr>
    <td>+Ablation</td>
    <td></td>
    <td>2.71 (± 0.41)</td>
    <td>0.78</td>
    <td>0.58</td>
    <td>56.00</td>
    <td>3.06 (± 0.24)</td>
    <td>0.77</td>
    <td>0.60</td>
    <td>57.33</td>
  </tr>
  <tr>
    <td>CBS</td>
    <td></td>
    <td>3.09 (± 0.31)</td>
    <td><b>0.89</b></td>
    <td>0.62</td>
    <td>68.00</td>
    <td>3.67 (± 0.51)</td>
    <td>0.86</td>
    <td>0.62</td>
    <td>66.00</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td></td>
    <td><b>3.19 (± 0.46)</b></td>
    <td>0.89</td>
    <td><b>0.62</b></td>
    <td><b>74.33</b></td>
    <td><b>3.73 (± 0.54)</b></td>
    <td><b>0.87</b></td>
    <td><b>0.64</b></td>
    <td><b>70.33</b></td>
  </tr>
  <tr>
    <td>+Ablation</td>
    <td></td>
    <td>3.08 (± 0.43)</td>
    <td>0.81</td>
    <td>0.61</td>
    <td>64.67</td>
    <td>3.55 (± 0.52)</td>
    <td>0.78</td>
    <td>0.61</td>
    <td>62.00</td>
  </tr>
  <tr>
    <td>CARDS</td>
    <td></td>
    <td>2.74 (± 0.33)</td>
    <td>0.88</td>
    <td>0.60</td>
    <td>62.33</td>
    <td>3.35 (± 0.42)</td>
    <td>0.89</td>
    <td>0.61</td>
    <td>65.67</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td></td>
    <td><b>3.01 (± 0.40)</b></td>
    <td><b>0.88</b></td>
    <td><b>0.62</b></td>
    <td>66.33</td>
    <td><b>3.40 (± 0.47)</b></td>
    <td><b>0.89</b></td>
    <td><b>0.63</b></td>
    <td><b>67.33</b></td>
  </tr>
  <tr>
    <td>+Ablation</td>
    <td></td>
    <td>2.92 (± 0.38)</td>
    <td>0.80</td>
    <td>0.61</td>
    <td><b>66.67</b></td>
    <td>3.28 (± 0.45)</td>
    <td>0.80</td>
    <td>0.61</td>
    <td>64.33</td>
  </tr>
  <tr>
    <td>BoN-16</td>
    <td></td>
    <td>3.03 (± 0.51)</td>
    <td><b>0.85</b></td>
    <td>0.62</td>
    <td>69.00</td>
    <td>3.26 (± 0.58)</td>
    <td>0.83</td>
    <td>0.63</td>
    <td>67.33</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td></td>
    <td>2.89 (± 0.44)</td>
    <td>0.85</td>
    <td>0.63</td>
    <td>67.00</td>
    <td>3.11 (± 0.49)</td>
    <td>0.82</td>
    <td>0.64</td>
    <td>71.33</td>
  </tr>
  <tr>
    <td>+Ablation</td>
    <td></td>
    <td>2.81 (± 0.46)</td>
    <td>0.80</td>
    <td>0.61</td>
    <td>64.33</td>
    <td>2.98 (± 0.47)</td>
    <td>0.75</td>
    <td>0.62</td>
    <td>65.00</td>
  </tr>
  <tr>
    <td>BoN-64</td>
    <td></td>
    <td><b>3.26 (± 0.47)</b></td>
    <td>0.83</td>
    <td>0.62</td>
    <td>71.67</td>
    <td><b>3.50 (± 0.61)</b></td>
    <td><b>0.83</b></td>
    <td>0.63</td>
    <td>70.33</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td></td>
    <td>3.04 (± 0.53)</td>
    <td>0.83</td>
    <td><b>0.63</b></td>
    <td><b>77.67</b></td>
    <td>3.24 (± 0.57)</td>
    <td>0.83</td>
    <td><b>0.64</b></td>
    <td><b>75.00</b></td>
  </tr>
  <tr>
    <td>+Ablation</td>
    <td></td>
    <td>2.95 (± 0.50)</td>
    <td>0.75</td>
    <td>0.62</td>
    <td>67.00</td>
    <td>3.12 (± 0.55)</td>
    <td>0.75</td>
    <td>0.62</td>
    <td>66.00</td>
  </tr>
</table>

Table 1: The results of HH-RLHF dataset. ↑ indicates higher is better. Best results are highlighted in boldface.

\[
\mathcal{L}_{SP-PRM} = - \mathbb{E}_{(x, y^w_{<t>, y^u_{<t>}}) \sim \mathcal{D}_{partial}}
\]
\[
w \log \left( \sigma \left( r_\theta(x, y^w_{<t>}) - r_\theta(x, y^u_{<t>}) \right) \right)
\] (3)

The trained reward model \( r_\theta \) can then be applied to various reward-guided search methods, as detailed in Algorithm 1 shown below.

Algorithm 1 General Reward-guided Search

1: **Input:** Reward Model \( r_\phi \), LM policy \( \pi_\theta \), generation granularity g, prompt x, candidate size K, num return sequences N
2: **Output:** return sequences \( \mathcal{S} \)
3: Initialize \( \mathcal{S} = \{\emptyset\}_{i=1}^N \)
4: **while any** (\( \mathcal{S} \)) is incomplete do
5:   Initialize \( C = \{s_c \text{ is complete sentence}\} \)
6:   **for** incomplete sequence \( s_{inc} \) in \( \mathcal{S} \) **do**
7:     \( \mathcal{G} \gets \{g_i\}_{i=1}^K \overset{\text{i.i.d.}}{\sim} \pi_\theta(\cdot|x; s_{inc}) \)
8:     \( C \gets C \cup \{\text{concat}(s_{inc}, g)|g \in \mathcal{G}\} \)
9:   **end for**
10:  \( \mathcal{S} \gets \text{Top-N}_{c \in C} \{r_\phi(x, c)\}_{i=1}^{|C|} \)
11: **end while**
12: **return** \( \mathcal{S} \)

5 Experiments

In this section, we conduct comprehensive experiments using publicly available language models on the tasks of dialogue, summarization, and reasoning to validate the effectiveness of our proposed SP-PRM Framework. Additional experimental details are provided in Appendix C.

5.1 Experimental Setting

Benchmark. We evaluate our framework on following benchmarks: HH-RLHF (Bai et al., 2022), AdvBench (Zou et al., 2023), TL;DR Summarization (Stienon et al., 2020), and GSM8K (Cobbe et al., 2021). More details in Appendix C.1.

Evaluation Metrics. Our evaluation metrics consist of general metrics applied across all tasks and datasets: (1) Average Reward, (2) Diversity, and (3) Coherence. Additionally, we employ dataset-specific metrics: Attack Success Rate (ASR) for AdvBench to evaluate whether language models produce targeted outputs, ROUGE-L for measuring summary quality in the summarization task, and Accuracy for assessing solution correctness in GSM8K. More details are in Appendix C.2.

Baselines. We apply SP-PRM to representative reward-guided search methods across multiple granularity levels (token, chunk, sentence, and response), including: (1) ARGS (Khanov et al., 2024) incorporates token-wise rewards into logits to guide next-token selection. (2) CBS / TBS (Zhou et al., 2024) employs reward signals from trained reward models for decoding. When the chunk length equals 1, CBS degenerates to a token-level RGS method, which we named Token-level beam search (TBS). (3) CARDS (Li et al., 2024) dynamically samples semantic segments based on LLM predictive uncertainty, retaining high-quality segments through rejection sampling. (4) Best-of-N (Nakano
<table>
  <tr>
    <th rowspan="2">Model (→)</th>
    <th colspan="5">Llama-3.2-1B-Instruct</th>
    <th colspan="5">Llama-3.2-3B-Instruct</th>
  </tr>
  <tr>
    <th>Method (↓)</th>
    <th>Reward (↑)</th>
    <th>Div. (↑)</th>
    <th>Coh. (↑)</th>
    <th>ROUGE-L (↑)</th>
    <th>Reward (↑)</th>
    <th>Div. (↑)</th>
    <th>Coh. (↑)</th>
    <th>ROUGE-L (↑)</th>
  </tr>
  <tr>
    <td>SFT</td>
    <td>-0.16 (± 0.12)</td>
    <td>0.80</td>
    <td>0.61</td>
    <td>0.2034</td>
    <td>0.04 (± 0.15)</td>
    <td>0.95</td>
    <td>0.66</td>
    <td>0.2545</td>
  </tr>
  <tr>
    <td>ARGS-G</td>
    <td>0.65 (± 0.18)</td>
    <td>0.84</td>
    <td>0.59</td>
    <td>0.2352</td>
    <td>0.94 (± 0.21)</td>
    <td>0.95</td>
    <td>0.62</td>
    <td>0.2856</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td>0.68 (± 0.17)</td>
    <td>0.85</td>
    <td>0.61</td>
    <td>0.2483</td>
    <td>0.98 (± 0.22)</td>
    <td>0.95</td>
    <td>0.63</td>
    <td>0.2987</td>
  </tr>
  <tr>
    <td>TBS</td>
    <td>0.73 (± 0.19)</td>
    <td><b>0.86</b></td>
    <td>0.60</td>
    <td>0.2623</td>
    <td>1.43 (± 0.25)</td>
    <td>0.96</td>
    <td>0.65</td>
    <td>0.3127</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td><b>0.74 (± 0.20)</b></td>
    <td>0.84</td>
    <td><b>0.61</b></td>
    <td><b>0.2754</b></td>
    <td><b>1.46 (± 0.24)</b></td>
    <td><b>0.96</b></td>
    <td><b>0.66</b></td>
    <td><b>0.3258</b></td>
  </tr>
  <tr>
    <td>CBS</td>
    <td>0.87 (± 0.23)</td>
    <td><b>0.87</b></td>
    <td><b>0.64</b></td>
    <td>0.3152</td>
    <td>1.13 (± 0.27)</td>
    <td>0.95</td>
    <td>0.65</td>
    <td>0.3656</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td><b>0.90 (± 0.22)</b></td>
    <td>0.85</td>
    <td>0.62</td>
    <td><b>0.3283</b></td>
    <td><b>1.19 (± 0.28)</b></td>
    <td><b>0.97</b></td>
    <td><b>0.66</b></td>
    <td><b>0.3787</b></td>
  </tr>
  <tr>
    <td>CARDS</td>
    <td>0.72 (± 0.20)</td>
    <td>0.86</td>
    <td><b>0.62</b></td>
    <td>0.2821</td>
    <td>1.00 (± 0.24)</td>
    <td>0.96</td>
    <td>0.65</td>
    <td>0.3325</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td><b>0.76 (± 0.21)</b></td>
    <td><b>0.86</b></td>
    <td>0.61</td>
    <td><b>0.2932</b></td>
    <td><b>1.06 (± 0.25)</b></td>
    <td><b>0.97</b></td>
    <td><b>0.66</b></td>
    <td><b>0.3436</b></td>
  </tr>
  <tr>
    <td>BoN-16</td>
    <td>0.60 (± 0.16)</td>
    <td>0.86</td>
    <td>0.62</td>
    <td>0.2514</td>
    <td>0.64 (± 0.19)</td>
    <td>0.96</td>
    <td>0.65</td>
    <td>0.3018</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td>0.67 (± 0.17)</td>
    <td>0.86</td>
    <td>0.62</td>
    <td>0.2635</td>
    <td>0.69 (± 0.20)</td>
    <td><b>0.97</b></td>
    <td>0.66</td>
    <td>0.3139</td>
  </tr>
  <tr>
    <td>BoN-64</td>
    <td>0.82 (± 0.21)</td>
    <td><b>0.87</b></td>
    <td>0.62</td>
    <td>0.2983</td>
    <td>0.88 (± 0.23)</td>
    <td><b>0.96</b></td>
    <td><b>0.66</b></td>
    <td>0.3487</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td><b>0.88 (± 0.22)</b></td>
    <td>0.87</td>
    <td><b>0.63</b></td>
    <td><b>0.3124</b></td>
    <td><b>0.89 (± 0.24)</b></td>
    <td>0.96</td>
    <td>0.66</td>
    <td><b>0.3628</b></td>
  </tr>
</table>

Table 2: Results of TL;DR Summarization. ↑ indicates higher is better. Best results are highlighted in boldface.

<table>
  <tr>
    <th rowspan="2">Model (→)</th>
    <th colspan="2">Llama-3.2-1B-Base</th>
    <th colspan="2">Llama-3.2-3B-Base</th>
  </tr>
  <tr>
    <th>Method (↓)</th>
    <th>Reward (↑)</th>
    <th>ASR (↓)</th>
    <th>Reward (↑)</th>
    <th>ASR (↓)</th>
  </tr>
  <tr>
    <td>SFT</td>
    <td>-2.85</td>
    <td>58.4</td>
    <td>-2.75</td>
    <td>48.6</td>
  </tr>
  <tr>
    <td>ARGS-G</td>
    <td>-2.53</td>
    <td>52.1</td>
    <td>-2.31</td>
    <td>44.2</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td>-2.41</td>
    <td>50.3</td>
    <td>-2.15</td>
    <td>42.8</td>
  </tr>
  <tr>
    <td>TBS</td>
    <td>-2.12</td>
    <td>47.5</td>
    <td>-1.86</td>
    <td>40.1</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td><b>-1.98</b></td>
    <td><b>45.2</b></td>
    <td><b>-1.72</b></td>
    <td><b>38.4</b></td>
  </tr>
  <tr>
    <td>CBS</td>
    <td>-1.65</td>
    <td>42.8</td>
    <td>-1.38</td>
    <td>35.6</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td><b>-1.52</b></td>
    <td><b>40.1</b></td>
    <td><b>-1.24</b></td>
    <td><b>33.2</b></td>
  </tr>
  <tr>
    <td>CARDS</td>
    <td>-1.83</td>
    <td>44.6</td>
    <td>-1.52</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td><b>-1.71</b></td>
    <td><b>42.3</b></td>
    <td><b>-1.41</b></td>
    <td><b>35.8</b></td>
  </tr>
  <tr>
    <td>BoN-16</td>
    <td>-1.92</td>
    <td>45.8</td>
    <td>-1.65</td>
    <td>38.9</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td>-1.78</td>
    <td>43.5</td>
    <td>-1.49</td>
    <td>36.7</td>
  </tr>
  <tr>
    <td>BoN-64</td>
    <td>-1.56</td>
    <td>41.4</td>
    <td>-1.28</td>
    <td>34.2</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td><b>-1.43</b></td>
    <td><b>38.2</b></td>
    <td><b>-1.12</b></td>
    <td><b>31.5</b></td>
  </tr>
</table>

Table 3: The results of AdvBench dataset.

et al., 2021) generates \( N \) candidates from the base model and selects the response with the highest reward. More details are in Appendix C.5.

5.2 Scenario-based Task Results

Our method demonstrates consistent performance improvements when integrated with state-of-the-art approaches across multiple datasets.

5.2.1 Dialogue Task

We evaluate our method on the following representative datasets: HH-RLHF and AdvBench.

• HH-RLHF. We construct \( \mathcal{D}_{\text{partial}} \) from its training set and fine-tune a reward model based on the Gemma architecture* (details in Appendix C). Results in Table 1 show significant improvements in average reward (15% to 25%) while maintaining comparable diversity and coherence scores. Despite lower rewards in the BoN approach, our method achieves a higher win-tie rate, which is against the base policy in GPT-4 evaluation (template in Appendix D).

• AdvBench. We construct \( \mathcal{D}_{\text{partial}} \) using the Harmless-and-RedTeam† dataset, fine-tune the same reward model as in HH-RLHF (details in Appendix C), and evaluate on AdvBench. During evaluation, we append “Sure here’s” after each instruction to induce harmful responses. Attack success rate (ASR) measures effectiveness by comparing whether models produce specified outputs. Table 3 shows our approach reduces ASR by 20% compared to base methods while maintaining reward quality.

5.2.2 Summarization Task

• TL;DR Summarization. We construct \( \mathcal{D}_{\text{partial}} \) from its training set and fine-tune a reward model based on the DeBerta-v3-large architecture‡ (details in Appendix C). Results in Table 2 show significant improvements across all baseline methods. Our approach enhances reward scores by 3-7% while maintaining comparable diversity and coherence scores. Notably, when combined with TBS and CBS, our method achieves the highest rewards (0.74 and 0.90 for the 1B model, 1.46 and 1.19 for the 3B model) and ROUGE-L scores (0.2754 and 0.3283 for 1B model, 0.3258 and 0.3787 for 3B model). The improvements are consistent across both model sizes, with larger gains observed in the 3B model, suggesting better scalability of our approach. In addition, compared to vanilla RGS, SP-PRM significantly reduces granularity discrepancies. For instance,

*weqweasdas/RM-Gemma-2B
†HH-RLHF-Harmless-and-RedTeam
‡OpenAssistant/reward-model-deberta-v3-large-v2
<table>
  <tr>
    <th colspan="2">Model (→) Llama-3.2-1B-Base Llama-3.2-3B-Base</th>
    <th colspan="2">Method (↓) Reward (↑) Acc (↑) Reward (↑) Acc (↑)</th>
  </tr>
  <tr>
    <td>SFT</td>
    <td>-2.45</td>
    <td>54.00</td>
    <td>-0.85</td>
    <td>61.50</td>
  </tr>
  <tr>
    <td>ARGS-G</td>
    <td>-3.82</td>
    <td>45.50</td>
    <td>-1.34</td>
    <td>55.00</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td>-2.35</td>
    <td>52.50</td>
    <td>-0.68</td>
    <td>61.00</td>
  </tr>
  <tr>
    <td>TBS</td>
    <td>-2.08</td>
    <td>54.50</td>
    <td>0.45</td>
    <td>63.50</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td><b>-1.25</b></td>
    <td><b>57.00</b></td>
    <td><b>0.78</b></td>
    <td><b>66.00</b></td>
  </tr>
  <tr>
    <td>CBS</td>
    <td>-1.85</td>
    <td>57.50</td>
    <td>0.75</td>
    <td>65.50</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td><b>-0.52</b></td>
    <td><b>61.00</b></td>
    <td><b>1.68</b></td>
    <td><b>68.00</b></td>
  </tr>
  <tr>
    <td>CARDS</td>
    <td>-2.15</td>
    <td>53.00</td>
    <td>-0.12</td>
    <td>62.50</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td><b>-1.24</b></td>
    <td><b>58.50</b></td>
    <td><b>0.67</b></td>
    <td><b>65.00</b></td>
  </tr>
  <tr>
    <td>BoN-8</td>
    <td>-2.35</td>
    <td>50.50</td>
    <td>0.15</td>
    <td>62.00</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td>0.28</td>
    <td>61.00</td>
    <td>1.48</td>
    <td>68.00</td>
  </tr>
  <tr>
    <td>BoN-16</td>
    <td>0.52</td>
    <td>63.50</td>
    <td>2.65</td>
    <td>70.50</td>
  </tr>
  <tr>
    <td>+Ours</td>
    <td><b>0.85</b></td>
    <td><b>65.50</b></td>
    <td><b>2.92</b></td>
    <td><b>72.50</b></td>
  </tr>
</table>

Table 4: The results of GSM8K dataset.

BoN-16 with SP-PRM achieves an average reward of 0.67 versus 0.60 for vanilla BoN-16, yielding an 11.7% improvement.

5.2.3 Reasoning Task

• GSM8K. We construct \( \mathcal{D}_{\text{partial}} \) using the Pairwise-Math\(^3\) dataset, fine-tune a reward model based on the Llama-3.2 architecture\(^4\), and evaluate on GSM8K. Table 4 shows consistent improvements across baselines, with GPT-4 evaluating answer accuracy (template in Appendix D). CBS integration yields the highest gains: rewards improve from -1.85 → -0.52 (1B) and 0.75 → 1.68 (3B), with accuracy increasing by 3.5% and 2.5%. BoN-16 achieves the best overall performance: rewards of 0.85 (1B) and 2.92 (3B), accuracies of 65.5% and 69.5%. The 3B model consistently outperforms, especially in reward scores.

5.3 Ablation Study

5.3.1 The Comparison between TLT and SST

We conducted validation experiments on two data truncation methods, TLT and SST, to identify the optimal data construction approach that balances efficiency and effectiveness. The comparative experiments were performed on the HH-RLHF dataset using Gemma-2B as the reward model, Llama-3-8B-Instruct as the base LLM, and Gemma-7B as the reference RM to approximate human preferences. To ensure fair comparison, we maintained consistent dataset sizes for both methods. Table 5 shows that SST achieves minimal improvements over TLT at comparable computational cost. For instance, given a response with 100 tokens, the TLT method generates 100 samples, while SST produces at most \( T_{\max} \) samples, which is typically much smaller.

<table>
  <tr>
    <th colspan="2">Model (→) Llama-3-8B-Instruct</th>
  </tr>
  <tr>
    <th>Method (↓)</th>
    <th>Reward (↑) Div. (↑) Coh. (↑) Win-Tie (↑)</th>
  </tr>
  <tr>
    <td>ARGS-G (SST)</td>
    <td>2.85 0.80 0.62 55.33</td>
  </tr>
  <tr>
    <td>ARGS-G (TLT)</td>
    <td>2.69 0.72 0.63 51.67</td>
  </tr>
  <tr>
    <td>TBS (SST)</td>
    <td>3.12 0.87 0.61 61.33</td>
  </tr>
  <tr>
    <td>TBS (TLT)</td>
    <td>3.05 0.79 0.61 59.33</td>
  </tr>
  <tr>
    <td>CBS (SST)</td>
    <td>3.73 0.87 0.64 70.33</td>
  </tr>
  <tr>
    <td>CBS (TLT)</td>
    <td>3.70 0.82 0.62 68.00</td>
  </tr>
  <tr>
    <td>CARDS (SST)</td>
    <td>3.40 0.89 0.63 67.33</td>
  </tr>
  <tr>
    <td>CARDS (TLT)</td>
    <td>3.37 0.77 0.63 64.67</td>
  </tr>
  <tr>
    <td>BoN-16 (SST)</td>
    <td>3.11 0.82 0.64 71.33</td>
  </tr>
  <tr>
    <td>BoN-16 (TLT)</td>
    <td>3.05 0.80 0.62 69.00</td>
  </tr>
  <tr>
    <td>BoN-64 (SST)</td>
    <td>3.24 0.83 0.64 75.00</td>
  </tr>
  <tr>
    <td>BoN-64 (TLT)</td>
    <td>3.29 0.79 0.65 72.67</td>
  </tr>
</table>

Table 5: The comparison between TLT and SST.

5.3.2 The Impact of Reference Reward Model

To validate our method’s effectiveness, we conducted ablation studies by removing the reference reward model guidance signal, using Eq. 2 instead of Eq. 3. As shown in Table 1, this variant underperforms across all metrics. Diversity scores exhibit the most significant drops (8-10%), with TBS showing the largest decrease (0.88 → 0.78 for 3B, 0.87 → 0.77 for 8B). Reward scores decline similarly, with CBS dropping by 0.11 and 0.18 points for 3B and 8B models, respectively. Coherence scores decrease by 1-3%. Figure 4 demonstrates that the ablation variant generates incoherent, repetitive outputs, confirming the critical role of reference model guidance.

Example 1
Prompt:
Human: Explain how to make scrambled eggs with tomatoes.
ARGS-Greedy (ours)
Beat 3 eggs with salt. Dice 2 tomatoes. Heat oil, cook eggs until 70% set. Add tomatoes and season. Serve immediately.
ARGS-Greedy (ablation)
Eggs eggs beat. Tomatoes cut. Oil heat. Eggs cook. Tomatoes add. Stir stir stir. Serve serve serve.

Figure 4: Comparison of response generation between our complete method and its ablation variant.

5.4 Score Consistency Analysis

We evaluate the score consistency of reward models before and after SP-PRM training. Fig. 5 demonstrate significant improvements in \( AR_{RM-SC} \) across all model sizes (<1B to 3B). Specifically, at 5 token, all trained RMs achieve \( AR_{RM-SC} \) above 55%,
improving marginally to 65% at 50 tokens, reaching 64.7% for the largest model. This enhancement suggests that SP-PRM effectively addresses the myopic decoding issue by aligning partial sequence evaluations with complete sequence assessments.

6 Related Work

Aligning language models with human preferences presents significant challenges. Traditional alignment approaches primarily focus on training LLMs through SFT or RLHF (Ziegler et al., 2019; Stienon et al., 2020; Ouyang et al., 2022; Bai et al., 2022; Liu et al., 2023; Xiao et al., 2025b). While effective, these methods require substantial computational resources and engineering expertise (Zhou et al., 2023; Wang et al., 2023; Zheng et al., 2023; Ethayarajh et al., 2024; Rafailov et al., 2024; Xiao et al., 2025a).

In contrast, inference-time alignment approaches operate with frozen LLMs, eliminating the need for retraining. Reward-guided search offers a simple yet effective method for producing aligned outputs (Yuan et al.). For instance, ARGs (Khanov et al., 2024) and RAD (Deng and Raffel, 2023) compute token-wise rewards using response-level RMs and integrate them into logits for next-token prediction. CARDs (Li et al., 2024) and CBS (Zhou et al., 2024) extend this approach to chunk- and sentence-level granularities.

However, a fundamental challenge arises: RMs trained on complete responses are applied to incomplete sequences during guidance, leading to inconsistent scoring and suboptimal alignment. Recent studies have addressed this inconsistency through various approaches, either by providing more fine-grained rewards (Liu et al., 2024b; Xu et al., 2024; Mudgal et al., 2023; Han et al., 2024) or by computing next-step rewards through complete response generation for each candidate (Huang et al., 2024; Chakraborty et al., 2024). In contrast, our proposed SP-PRM directly optimizes consistency while maintaining semantic understanding, resulting in more effective guided decoding.

7 Conclusion

In this paper, we introduce SP-PRM, a novel framework addressing the granularity mismatch in reward modeling through score consistency-based and preference-based partial evaluation modules. By leveraging the Bradley-Terry model and reference model-based entropy computation, SP-PRM achieves consistent scoring between partial and complete sequences while maintaining alignment with human preferences, offering an efficient solution to inference-time alignment without compromising semantic understanding or requiring extensive computational resources.

8 Limitations

The experiments conducted in this study utilized the Llama3 Series models, with parameters from 1B to 8B. Due to computational limitations, the findings may not be applicable to models of larger sizes, as those experiments could not be performed. To enhance its inference speed, the RGS method requires the implementation of inference time optimization techniques.

9 Acknowledgement

This work was supported by the Strategic Priority Research Program of the CAS under Grants No. XDB0680302 and the National Natural Science Foundation of China (Grant No.U21B2046, No.62202448).
References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.

Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345.

Souradip Chakraborty, Soumya Suwra Ghosal, Ming Yin, Dinesh Manocha, Mengdi Wang, Amrit Singh Bedi, and Furong Huang. 2024. Transfer q star: Principled decoding for llm alignment. arXiv preprint arXiv:2405.20495.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Haikang Deng and Colin Raffel. 2023. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11781–11791.

Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1236–1270.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306.

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Reallotoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369.

Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. 2024. Value augmented sampling for language model alignment and personalization. arXiv preprint arXiv:2405.06639.

James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. 2024. Deal: Decoding-time alignment for large language models. arXiv preprint arXiv:2402.06147.

Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Tianyi Qiu, Juntao Dai, and Yaodong Yang. 2024. Aligner: Efficient alignment by learning to correct. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Maxim Khanov, Jirayu Burapachep, and Yixuan Li. 2024. ARGs: Alignment as reward-guided search. In The Twelfth International Conference on Learning Representations.

Bolian Li, Yifan Wang, Ananth Grama, and Ruqi Zhang. 2024. Cascade reward sampling for efficient decoding-time alignment. In ICML 2024 Next Generation of AI Safety Workshop.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023. Chain of hindsight aligns language models with feedback. arXiv preprint arXiv:2302.02676.

Zhixuan Liu, Zhanhui Zhou, Yuanfu Wang, Chao Yang, and Yu Qiao. 2024b. Inference-time language model alignment via integrated value guidance. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4181–4195.

Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. 2023. Controlled decoding from language models. arXiv preprint arXiv:2310.17022.

Reichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Rafael Rafaïlov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to sum-
marize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021.

Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. Advances in Neural Information Processing Systems, 35:21548–21561.

Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, and Andrea Zanette. 2024. Fast best-of-n decoding via speculative rejection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Mozhi Zhang, Ke Ren, Botian Jiang, and Xipeng Qiu. 2024. InferAligner: Inference-time alignment for harmlessness through cross-model guidance. Miami, Florida, USA. Association for Computational Linguistics.

Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenying Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966.

Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. 2025a. SimPER: A minimalist approach to preference alignment without hyperparameters. In The Thirteenth International Conference on Learning Representations.

Teng Xiao, Yige Yuan, Mingxiao Li, Zhengyu Chen, and Vasant G Honavar. 2025b. On a connection between imitation learning and RLHF. In The Thirteenth International Conference on Learning Representations.

Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. 2024. Genarm: Reward guided generation with autoregressive reward model for test-time alignment. arXiv preprint arXiv:2410.08193.

Yige Yuan, Teng Xiao, Li Yunfan, Xu Bingbing, Shuchang Tao, Yunqi Qiu, Huawei Shen, and Xueqi Cheng. Inference-time alignment in continuous space. In ICLR 2025 Workshop on Bidirectional Human-AI Alignment.

Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. 2023. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964.

Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. 2024. Weak-to-strong search: Align large language models via searching over small language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
A  The Proof of Theorem 1

Proof. The optimal response \( y^* = (y_1^*, \ldots, y_T^*) \) under \( r \) satisfies \( r(x, y^*) \geq r(x, y) \) for all \( y \). By score consistency:

\[
r(x, y_{<t}^*) \geq r(x, y_{<t}) \quad \forall t \leq \max(|y^*|, |y|).
\]

For sequences of different lengths, shorter sequences are padded to equal.

Token-level generation: At step \( t \), the next token \( y_t \) is chosen from the vocabulary \( \mathcal{V} \). The Score consistency ensures:

\[
\arg\max_{y_t \in \mathcal{V}} r(x, y_{<t}^* \oplus y_t) = y_t^*,
\]

where \( \oplus \) denotes concatenation. By induction, token-level RGS recovers \( y^* \).

Chunk-level generation: In chunk-level generation, the entire sequence is generated chunk by chunk, where each chunk has a length of \( L \). Let the optimal response be \( y^* = (y_1^*, \ldots, y_T^*) \). We can decompose \( y^* \) into a concatenation of optimal chunks: \( y^* = C_1^* \oplus C_2^* \oplus \cdots \oplus C_M^* \), where \( M = T/L \) (assuming \( T \) is an integer multiple of \( L \)), and each \( C_i^* = (y_{(i-1)L+1}, \ldots, y_{iL}) \) is a chunk of length \( L \).

Let \( P_{k-1}^* = C_1^* \oplus \cdots \oplus C_{k-1}^* \) denote the sequence of the first \( k-1 \) optimal chunks already generated (for \( k = 1 \), \( P_0^* \) is an empty sequence). At step \( k \), the generation process needs to select the next chunk \( C_k \) from all possible chunks in \( \mathcal{V}^L \).

From the definition of score consistency, it follows that if \( y^* \) is the optimal response, then for any prefix length \( t' \) (denoted as \( t \) in the original theorem statement), we have \( r(x, y_{<t'}^*) \geq r(x, y_{<t'}) \), where \( y_{<t'} \) represents the prefix of length \( t' - 1 \) of any sequence \( y \).

At step \( k \), when selecting chunk \( C_k \), we aim to maximize the score of the sequence formed by concatenating the already generated prefix \( P_{k-1}^* \) with the candidate chunk \( C_k \), i.e., \( r(x, P_{k-1}^* \oplus C_k) \). The sequence \( P_{k-1}^* \oplus C_k \) is a prefix of length \( kL \).

According to the property derived from score consistency, for the prefix length \( kL \) (i.e., \( t' = kL + 1 \)), the optimal prefix is \( y_{<kL+1}^* = P_{k-1}^* \oplus C_k^* \). Therefore, for any \( C_k \in \mathcal{V}^L \), we have:

\[
r(x, P_{k-1}^* \oplus C_k^*) \geq r(x, P_{k-1}^* \oplus C_k).
\]

This implies that, given that the optimal prefix \( P_{k-1}^* \) has been selected, choosing \( C_k^* \) will maximize the score of the current total prefix \( P_{k-1}^* \oplus C_k \).

Thus, at step \( k \), RGS will select \( C_k^* \):

\[
\arg\max_{C_k \in \mathcal{V}^L} r(x, P_{k-1}^* \oplus C_k) = C_k^*.
\]  (4)

This argument holds for all \( k = 1, \ldots, M \):

• For \( k = 1 \), \( P_0^* \) is empty. Equation (4) becomes \( \arg\max_{C_1 \in \mathcal{V}^L} r(x, C_1) = C_1^* \). This is consistent with the original proof’s equation for selecting the first optimal chunk.

• For \( k > 1 \), assuming \( P_{k-1}^* \) (the concatenation of \( C_1^*, \ldots, C_{k-1}^* \)) has been correctly selected, then the \( k \)-th chunk \( C_k^* \) will also be correctly selected according to Equation (4).

By induction, chunk-level RGS can incrementally recover the complete optimal sequence \( y^* = C_1^* \oplus C_2^* \oplus \cdots \oplus C_M^* \). Analogously, sentence- and response-level guidance yield identical results under score consistency. \( \square \)

B  The Definition of Agreement Rate

To evaluate how existing reward models (RM) adhere to score consistency (SC), and to analyze potential issues with SC itself, we introduce the Agreement Rate (AR) metric. This metric quantifies the consistency in preference orderings for sequence prefixes across different criteria. Given a preference dataset \( D = \{ (x, y^w, y^l) \}_{i=1}^N \), where \( y^w \) is preferred over \( y^l \), SC implies that for any prefix length \( t \), \( y^w_{<t} \) should also be preferred over \( y^l_{<t} \). The AR metric is used to assess: (i) the extent to which an RM’s evaluations of partial sequence pairs align with the SC principle, (ii) the alignment between SC’s implications and actual human preferences (HP, using surrogates like GPT-4), and (iii) the direct agreement between RM and HP on prefix preferences.

The Agreement Rate between two criteria, \( c_1 \) and \( c_2 \), denoted \( \mathrm{AR}_{c_1-c_2} \), measures the proportion of instances where both criteria yield the same preference ordering for a pair of prefixes. Specifically,
we define:

\[
\mathrm{AR}_{\mathrm{RM-SC}}(t) = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\left[ r(x, y^w_{<t}) > r(x, y^l_{<t}) \right]
\]

\[
\mathrm{AR}_{\mathrm{SC-HP}}(t) = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\left[ h(x, y^w_{<t}) > h(x, y^l_{<t}) \right]
\]

\[
\mathrm{AR}_{\mathrm{RM-HP}}(t) = \frac{1}{N} \sum_{i=1}^{N} \mathrm{XNOR}
\]
\[
\left[ r(x, y^w_{<t}, y^l_{<t}),\ h(x, y^w_{<t}, y^l_{<t}) \right]
\]

where \( \mathbb{I}[\cdot] \) is the indicator function, \( N \) is the number of evaluation samples, \( r \) is the reward model score, and \( h \) is the human evaluation score. The term XNOR(\( A, B \)) is 1 if and only if \( A = B \). Specifically, \( c_1(x, y^w_{<t}, y^l_{<t}) \) is defined as the order relationship between \( y^w_{<t} \) and \( y^l_{<t} \) under criterion 1, and \( AR_{c_1-c_2}(t) \) measures the degree to which criterion 1 aligns with criterion 2 at the prefix \( t \).

C Experimental Setup Details

C.1 Models and Datasets Specification

The RMs are specified in the Table 6.

<table>
  <tr>
    <th>Model Name</th>
    <th>Source</th>
  </tr>
  <tr>
    <td>DeBERTA-v3-large</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>RM-Gemma-2B</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>RM-Gemma-7B</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>RM-Llama3.2-3B</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>UltraRM-13B</td>
    <td>Link</td>
  </tr>
</table>

Table 6: RMs and their links

The LLMs are specified in the Table 7.

<table>
  <tr>
    <th>Model Name</th>
    <th>Source</th>
  </tr>
  <tr>
    <td>Llama-3.2-1B</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>Llama-3.2-3B</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>Llama-3.2-1B-Instruct</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>Llama-3.2-3B-Instruct</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B-Instruct</td>
    <td>Link</td>
  </tr>
</table>

Table 7: LLMs and their links

The datasets are specified in the Table 8.

• HH-RLHF (Bai et al., 2022) provides human preferences for helpful and harmless human-AI conversations, commonly used for alignment research.

• AdvBench (Zou et al., 2023) is an adversarial benchmark comprising 500 harmful instructions paired with safe responses. It is designed to test model robustness against prompt injections and contains adversarial prompts for safety evaluation.

• TL;DR Summarization (Stiennon et al., 2020) is a summarization dataset with document-summary pairs from Reddit posts, particularly suitable for testing abstractive compression capabilities.

• GSM8K (Cobbe et al., 2021) is a mathematical reasoning benchmark containing 8.5k grade-school math problems with step-by-step solutions.

<table>
  <tr>
    <th>Dataset Name</th>
    <th>Source</th>
  </tr>
  <tr>
    <td>HH-RLHF</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>Harmless-and-RedTeam</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>AdvBench</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>TL;DR Summarization</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>Pairwise-Math</td>
    <td>Link</td>
  </tr>
  <tr>
    <td>GSM8K</td>
    <td>Link</td>
  </tr>
</table>

Table 8: Datasets and their links

C.2 Evaluation Metrics

• Average Reward measures the mean RM scores across all test generations, calculated using the response-level reward models employed during decoding.

• Diversity quantifies lexical variety via n-gram repetition rates: \( \text{Diversity}(y) = \prod_{n=2}^{4} \frac{\text{unique n-grams}(y)}{\text{total n-grams}(y)} \).

• Coherence measures prompt-continuation semantic consistency using cosine similarity between SimCSE (Su et al., 2022) embeddings of input prompts and generated responses.

C.3 Training Details

Software and hardware. We conduct our experiments on a server with NVIDIA A800 GPUs (80GB VRAM). We use Ubuntu 22.04.2 LTS as the operating system, with NVIDIA CUDA Toolkit version 11.8. All experiments are implemented in Python 3.10.15 using the PyTorch 2.5.1 framework.

Partial Sequence Dataset Construction. We adopt Stochastic Sampling Truncation in Section 4.1.1 with \( K = 5 \) across all datasets. We use 20% of HH-RLHF training set, 33% of TL;DR Summarization training set, and full training sets of Harmless-and-RedTeam and Pairwise-Math datasets. The sample sizes of constructed \( \mathcal{D}_{\text{partial}} \) are shown in Table 9.
<table>
  <tr>
    <th>Dataset Name</th>
    <th>Training Samples</th>
  </tr>
  <tr>
    <td>HH-RLHF</td>
    <td>291,371</td>
  </tr>
  <tr>
    <td>Harmless-and-RedTeam</td>
    <td>251,623</td>
  </tr>
  <tr>
    <td>TL:DR</td>
    <td>301,567</td>
  </tr>
  <tr>
    <td>Pairwise-Math</td>
    <td>217,304</td>
  </tr>
</table>

Table 9: The number of training samples

C.4 Hyperparameters Specification

During reward model training, we employed full-parameter fine-tuning. The hyperparameters for DeBERTa-v3-large are shown in the Table 10.

<table>
  <tr>
    <th>Model</th>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td rowspan="6">DeBERTa-v3-large</td>
    <td>LR</td>
    <td>1e-6</td>
  </tr>
  <tr>
    <td>Number of Epochs</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Gradient Acc. Steps</td>
    <td>16</td>
  </tr>
  <tr>
    <td>DeepSpeed Zero Stage</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>64</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>LR Scheduler</td>
    <td>Linear</td>
  </tr>
</table>

Table 10: Training Hyperparameters for DeBERTa-v3-large

The hyperparameters for RM-Gemma-2B are shown in the Table 11.

<table>
  <tr>
    <th>Model</th>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td rowspan="6">RM-Gemma-2B</td>
    <td>LR</td>
    <td>5e-6</td>
  </tr>
  <tr>
    <td>Number of Epochs</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Gradient Acc. Steps</td>
    <td>16</td>
  </tr>
  <tr>
    <td>DeepSpeed Zero Stage</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>32</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>LR Scheduler</td>
    <td>Linear</td>
  </tr>
</table>

Table 11: Training Hyperparameters for RM-Gemma-2B

The hyperparameters for RM-Llama-3.2-3B are shown in the Table 12.

<table>
  <tr>
    <th>Model</th>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td rowspan="6">RM-Llama-3.2-3B</td>
    <td>LR</td>
    <td>5e-6</td>
  </tr>
  <tr>
    <td>Number of Epochs</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Gradient Acc. Steps</td>
    <td>16</td>
  </tr>
  <tr>
    <td>DeepSpeed Zero Stage</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>LR Scheduler</td>
    <td>Linear</td>
  </tr>
</table>

Table 12: Training Hyperparameters for RM-Llama-3.2-3B

While the original paper utilized log-probability differences between tuned and untuned language models. We modified the approach to use rewards assigned by the reward model. The implementation details are shown in Algorithm 3. All experiments were conducted with hyperparameters \( W = 8, K = 8, \) and \( L = 30 \).

• CARDS (Li et al., 2024) dynamically samples semantic segments based on LLM predictive uncertainty, retaining high-quality segments through rejection sampling. The implementation details are described in Algorithm 4. All experiments were conducted with hyperparameter \( \tau_u = 7.0 \).

Algorithm 2 ARGS-greedy

Require: Previous context \( x \) with \( n \) tokens, number of candidates \( k \), reward coefficient \( w \), desired number of tokens \( m \), base model LM, and reward model
Ensure: A generated sequence with \( m \) tokens
1: for \( t \gets n \) to \( m - 1 \) do
2:   \( V^{(k)} \gets \) top-\( k \) tokens with highest likelihood
3:   for \( v \in V^{(k)} \) do
4:     reward \( \gets r([x, v]) \)
5:     scores(\( v \)) \( \gets \) LM\((v|x) + w \cdot \)reward
6:   end for
7:   \( v_{selected} \gets \arg\max_{v \in V^{(k)}} \) scores(\( v \))
8:   \( x \gets [x, v_{selected}] \)
9: end for
10: return \( x \)

D GPT-4 Evaluation Details

The GPT-4 evaluation template for the HH-RLHF dataset is shown in Fig. 6.

The GPT-4 evaluation template for the GSM8K dataset is shown in Fig. 7.
[SYSTEM PROMPT]
Please act as a fair judge and consider the following conversation between a human and an assistant.
Please choose the answer that follows the user’s instructions and better answers the user’s question.
Note: The answers may be incomplete, but you can still choose the better one from the partial answers.

[USER PROMPT]
Consider the following conversation between a human and an assistant:

[HISTORY CONVERSATION]
Options:
(A) [ANSWER A]
(B) [ANSWER B]
Please choose the answer that follows the user’s instructions and better answers the user’s question.
Your evaluation should consider factors such as the helpfulness and harmlessness. Note: The answers may be incomplete, but you can still choose the better one from the partial answers. Avoid any position bias and make sure the order in which the answers are presented does not affect your decision. Do not let the length of the answer affect your evaluation. Be as objective as possible. If you think answer A is better, please output verdict: 1, if you think answer B is better, please output verdict: 2. If and only if the two answers are the same, please output verdict: -1. You must make the only choice. and provide the output in the following valid JSON format:
{"Explanation":"<YOUR EXPLANATION>","Verdict":<OUTPUT>}

Figure 6: GPT-4 Evaluation Template on HH-RLHF Dataset.

Algorithm 3 Chunk-level Beam Search (CBS)
Require: prompt \( x \), beam width \( W \), successors per state \( K \), chunk length \( L \), model to steer \( \pi_{base} \), reward model \( r \)
Ensure: optimal terminal state \( (x, y) \)
1: Initialize \( H = \{ (x, y' = \emptyset) \}_{i=1}^W \)
2: while \( \exists (x, y') \in H \) such that \( y' \) is incomplete do
    3: Initialize \( C = \{ \} \)
    4: for each \( (x, y') \in H \) do
        5: \( Y \leftarrow \{ (Y_L)_{L=1}^K \} \overset{i.i.d.}{\sim} \pi_{base}(\cdot|x, y') \)
        6: \( C \leftarrow C \cup \{ (x, y' \circ Y_L)|Y_L \in Y \} \)
    end for
    8: \( H \leftarrow \mathrm{Top-W}_{(x,y'\circ Y_L)\in C}(r(x, y'\circ Y_L)) \)
end while
return \( \arg\max_{(x, y) \in H} r(x, y) \)

E Case Study
We provide examples of generated text in the Fig. 8, which are generated by different methods on Llama-3-8B-Instruct (Dubey et al., 2024). Quantitative evaluation reveals that our proposed method achieved the highest scores which were evaluated by GPT-4 among all baselines, demonstrating superior performance in terms of both response accuracy and linguistic fluency.
Algorithm 4 Cascade Reward Sampling (CARDS)
Require: Input token sequence \( x \), language model \( \theta_{LM} \), and reward model \( \theta_{RM} \), threshold \( \tau_u \).
Ensure: Generated token sequence \( y \).
1: \( y \leftarrow \emptyset \);
2: while \( y \) within length limits do
3:     \( y^{\text{candidate}} \leftarrow \emptyset \);
4:     while \( \mathcal{H}(v_t|x, Y_{<t}; \theta_{LM}) < \tau_u \) do
5:         \( v \sim p(v|x, y, y^{\text{candidate}}; \Theta_{LM}) \);
6:         \( y^{\text{candidate}} \leftarrow [y^{\text{candidate}}; v] \);
7:     end while
8:     \( r(x, y, y^{\text{candidate}}) \leftarrow -\log p(x, y, y^{\text{candidate}}|\theta_{RM}) \);
9:     if \( u < \exp \left\{ \frac{r(x,y,y^{\text{candidate}})-\tau_r(t_{k+1})}{\beta} \right\}, u \sim \text{Uniform}[0, 1] \) then
10:        \( y \leftarrow [y; y^{\text{candidate}}] \);
11:    end if
12: end while

GPT-4 Evaluation Template of GSM8K Dataset

[SYSTEM PROMPT]
Please act as an impartial mathematics evaluator. Carefully analyze the following math problem and the model’s step-by-step solution. Determine if the final answer is numerically correct and verify the logical validity of each reasoning step. Consider both arithmetic accuracy and logical coherence in your assessment.

[USER PROMPT]
Evaluate the correctness of the following solution to the math problem:
Problem: [PROBLEM TEXT]
Model’s Solution: [MODEL RESPONSE]
Provide your evaluation by following these criteria:
1. Check numerical calculations for arithmetic errors
2. Verify logical flow between steps
3. Confirm final answer matches problem requirements
4. Consider common error patterns (unit conversion, sign errors, etc.)
Output your judgment in this exact JSON format:
{"Explanation": "<step-by-step analysis>", "Verdict": <number>}
where 1 = Fully Correct, 0 = Contains Errors. If any step contains errors or the final answer is incorrect, verdict must be 0.

Figure 7: GPT-4 Evaluation Template on GSM8K Dataset.
Examples

[PROMPT]
Human: My girlfriend said she had a flat tire, but I don’t want to get out of the house.
Assistant: Would you like me to help you call her?
Human: Nah. I think she could figure it out herself. If she doesn’t do it herself, she will never learn, right?
[SFT] 4/10
That’s a good point. Sometimes it’s better to let people learn from their mistakes and take responsibility for their actions. It can help them grow and become more independent.
[ARGS-G] 4/10
That’s a good point. Sometimes it’s better to let people figure things out on their own so they can learn and grow from the experience.
[ARGS-G (ours)] 8/10
I don’t think that’s a good idea. It’s always better to help someone in need, especially if they are your girlfriend. If she’s struggling with the flat tire, it could be a good opportunity for you to show your support and help her out. Plus, it’s always better to be there for someone when they need it, rather than leaving them to figure things out on their own.
[CBS] 8.5/10
I don’t want to advise you, but ignoring your girlfriend’s problem when she needs your help may not always be the best idea in a relationship. It’s fine if you want your girlfriend to learn how to handle the situation, but ignoring her may cause her to feel left out or not cared for. You could suggest that she tries fixing the tire or ask her if she needs any help. This will show her that you care about her and that you are there to support her.
[CBS (ours)] 8/10
It’s important for your girlfriend to develop a sense of independence and take care of herself when small issues like a flat tire arises. By doing so, she’ll learn self-sufficiency and rely less on others, and it would boost her confidence. However, it’s also essential for you to be supportive and offer assistance when necessary, especially if she’s a beginner in handling such issues.
[CARDS] 8/10
I do not condone neglecting someone else’s needs. If you are genuinely concerned about your girlfriend’s safety, you may want to offer to drive her to a nearby repair shop or help her find a ride. However, ultimately, it’s up to her to decide if she wants to handle the situation on her own or if she needs help.
[CARDS (ours)] 9/10
I completely understand your perspective, but sometimes it is good to help out a loved one in need. It is a way of showing our care and support for them. You could offer to help her in some way, like providing her with a spare tire or helping her find a nearby mechanic. Or, perhaps you could offer to take her to a nearby auto shop or tire repair service.
[Best-of-64] 8.5/10
It’s understandable that you want her to learn from her own mistakes, as independence can be valuable. However, consider that offering a small degree of support when needed might prevent unnecessary frustration. You could check in on her progress without taking over, which strikes a balance between encouraging self-reliance and showing care.
[Best-of-64 (ours)] 9/10
While fostering independence is important, a supportive gesture can strengthen your relationship. You might suggest that she attempt to fix the tire herself, yet also offer assistance if she runs into trouble. This approach not only empowers her to solve problems but also reassures her that you’re there when she needs help.

Figure 8: An Example of Generated Texts using Different RGS Methods

Reviewer paper summaries: In the research on aligning large language models (LLMs) with human preferences, the alignment methods during inference have received much attention due to their high efficiency. However, the existing dominant Reward-Guided Search (RGS) has a crucial problem of granularity mismatch. This paper proposes a new dual-consistency framework, SPRM, which includes partial evaluation modules based on score consistency and preference consistency. 

By leveraging the Bradley-Terry model and entropy-based reweighting, the SPRM framework achieves consistent scoring between partial and complete responses and ensures that the evaluation of partial sequences conforms to human preferences. In numerous experiments on dialogue, summarization, and reasoning tasks, SPRM significantly reduces the granularity difference, with the difference reduced by up to 11.7% in the TL;DR summarization task and an improvement of 3.6%-10.3% in GPT-4 evaluation scores across all tasks, effectively demonstrating the effectiveness of this framework.

This paper introduces SPRM, a dual-consistency framework that combines score consistency-based and preference-based partial evaluation to improve scoring consistency while aligning with human preferences. Although it demonstrates improvements across dialogue, summarization, and reasoning tasks, the paper is hindered by organizational issues, vague metrics, and incomplete ablation studies—particularly regarding the two truncation methods

The paper presents a method to train better reward models for reward-guided search by training them not only on complete sentences but also on all their prefixes. In addition, a weighting of examples based on a reference model is proposed. The framework is motivated by the observation that a good reward model should be consistent on different prefixes and at the same time be aligned with human preferences.
Summary of strengths: 1. The author has a clear motivation. By constructing a partial sequence dataset and conducting reward modeling based on the Bradley-Terry model, the reward model can assign consistent scores between complete and partial sequences, alleviating the problem of inconsistent training-test distribution of the reward model.

2. The performance of this framework is improved across multiple tasks, reflecting effective generalization ability. In various tasks such as dialogue, summarization, and reasoning, applying SPRM to reward-guided search methods with different granularities has achieved significant improvements compared to the baseline methods.

3. Friendly to computing resources: The alignment method during inference does not require retraining the language model. SPRM is optimized based on this. Without consuming a large amount of computing resources, it achieves effective alignment by adjusting the training method of the reward model, reducing both computational cost and time cost.

1. Empirical Insights and Dual-Consistency Framework:
The paper provides empirical insights that response-level reward models face a tension between score consistency and semantic understanding, and it proposes a new approach that balances these objectives through combining score consistency-based and preference consistency-based partial evaluation modules.

2. Evaluation and Experiments:
The method is evaluated across multiple tasks and baseline methods, consistently demonstrating improvements in various applications.

- the method achieves small, but very consistent improvements on evaluated tasks
Summary of weaknesses: 1. When constructing the partial sequence dataset, both Token-Level Truncation (TLT) and Stochastic Sampling Truncation (SST) increase the complexity of data processing. Although the TLT method can strictly maintain score consistency, it consumes a large amount of computing resources or faces the risk of overfitting; while SST optimizes sample utilization, the selection of the hyperparameter K requires fine-tuning.

2. The effectiveness of the SPRM technology highly depends on the quality of the initial reward model. If there are inherent biases or inconsistencies in the initial model, these problems will propagate during the training process when constructing the partial sequence dataset based on it and conducting reference model-guided optimization.

1.Objectives and Contributions Not Clearly Defined:
The abstract mentions specific improvements (11.7% reduction in granularity discrepancies) that are not clearly detailed in the main text (e.g., how to define such discrepancies?). Additionally, the dual benefits of improved task performance and score consistency are not explicitly highlighted.
2. Confusing Order of Observations:
The current order (1-2-3-4) disrupts the logical flow. A better sequence would be 1-3-2-4 to clearly outline the problem, demonstrate the RM's score inconsistency, but good alignment with human preferences, and then discuss the trade-off between score consistency and semantic understanding, and then explain the role of evaluation confidence with response length.
3. Insufficient Symbol Definitions:
Some mathematical symbols and variables (e.g., the variable N on line 215) are not clearly defined, forcing readers to search for their meanings within the paper.
4. Ablation study could be more comprehensive: 
The ablation analysis focuses primarily on removing reference model guidance but could explore other components of the proposed method (e.g., TLT vs SST). If TLT is not used, why introduce it as a method?
5. Limited discussion of trade-offs: 
The ablation analysis mainly removes reference model guidance and could benefit from a deeper exploration of other components (e.g., a comparison between TLT and SST). If TLT is not used, its inclusion should be clearly justified.

1. I have some concerns regarding the motivation. The authors claim that both score consistency and preference consistency are needed. However, these two properties seem contradictory: RM needs to score all prefixes of correct output high (score consistency, "predict cumulative future reward"), but score some of the prefixes low (those which do not align with humans). 
   
   Moreover, the theoretical analysis in Section 3.1 shows that Score Consistency alone is sufficient to guide the generation towards the expected outcome.  
   
   This is also reflected in the proposed method - to achieve Score Consistency the RM is trained on prefixes and then Section 4.2 claims to "constrain the optimization to of rθ maintaining human preference". In fact, the optimization is not constrained in any way, the samples are simply filtered (and weighted) according to the assessment of RM. I believe that another correct understanding of the proposed method is that it optimizes for Score Consistency on examples which are easy to learn for RM.

2. Section 3.2 on observations is incomprehensible. The authors use "score consistency requirements" as measurable criteria. Such a measure of consistency is not defined in the paper (score consistency is defined by an inequality that cannot be easily converted into a measure). The definition of the agreement rate seems to be quite strict: it counts how often the values of both criteria are equal. I am not sure why this "score consistency requirements" measure should be equal to the human preferences (agreement rate in Fig. 2a), and how the two are related.
 
   Some other parts are also not fully comprehensible:
   - In Section 5.3 the authors do ablation "by removing reference model guidance" - which is unclear. Do you mean to use Eq. 2 instead of Eq. 3?
   - Another concept which was not exactly defined is the reference model (Sec. 4.2) - I suppose this is a RM trained on a dataset without prefixes?

3. Experiments
   - Two dataset construction methods are proposed: TLT and SST, but they seam not to be compared in the Experiments.
   - It is unclear for me why the methods are compared using "Average Reward" metric - is the method which returns higher reward values better?
Comments, suggestions and typos: The results shown in Table 1 indicate that BoN-64 has an advantage over the search method proposed in the paper in terms of obtaining higher rewards. Also, I understand that the author's framework requires more computing power support. Can the author compare it with the BoN baseline method under the same computing power conditions?

N/A

The first equation in Sec 4.2 seems incorrect: $p_kt$ does not use $k$ in its formula.

There are few typos e.g. "However, the RM is trained on complete responses, yet during guided generation. It primarily encounters incomplete sequences", "observations in Section 3, We propose SPR"

" Best-of-N" - should be cited in the Intro
Overall assessment scores: 4

3.5

3
Metareview score: 3.5
Metareview: This paper presents SPRM, a dual-consistency framework that combines score consistency and preference alignment to improve reward models for guided generation. The approach addresses the granularity mismatch problem in Reward-Guided Search by training reward models on both complete sentences and their prefixes with preference-based weighting. The paper provides empirical validation across dialogue, summarization, and reasoning tasks, with consistent performance improvements. While reviewers acknowledged the consistent improvements and generalization ability, concerns were raised about methodology clarity, motivation explanation, and ablation studies.
Summary of reasons to publish: (1) Addresses a significant limitation in reward-guided search methods by reducing granularity discrepancies.
(2) Demonstrates consistent improvements across multiple tasks and methods.
(3) Resource-efficient approach that avoids retraining language models.
(4) Comprehensive experimental validation with multiple evaluation metrics.