RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Models
Authors: Juan Diego Rodriguez
Wenxuan Ding
Katrin Erk
Greg Durrett
Venue: COLM 2025 (2025)
Keywords: consistency
robustness
ranking loss
generalizability
generator-validator gap
TL;DR: We reduce the generator-validator gap (a discrepancy between LLMs' generated answers and self-verification), with a ranking-based loss function, improving model consistency

Abstract: Although large language models (LLMs) have become more capable and accurate across many tasks, some fundamental sources of unreliability remain in their behavior. One key limitation is their inconsistency at reporting the same information when prompts are changed. In this paper, we consider the discrepancy between a model’s generated answer and their own verification of that answer, the generator-validator gap. We define this gap in a more stringent way than prior work: we expect correlation of scores from a generator and a validator over the entire set of candidate answers, i.e., candidate completions that could possibly arise during ordinary language use without breaking Gricean norms. We show that according to this measure, a large gap exists in various settings, including question answering, lexical semantics tasks, and next-word prediction. We then propose RankAlign, a ranking-based training method, and show that it significantly closes the gap, surpassing all baseline methods. Moreover, this approach generalizes well to out-of-domain tasks and lexical items.

Published as a conference paper at COLM 2025
RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Models
Juan Diego Rodriguez
Wenxuan Ding
Katrin Erk
Greg Durrett
Department of Computer Science
Department of Linguistics
The University of Texas at Austin [juand-r, wenxuand, katrin.erk,
edurrett)eutexxs...
Abstract
Although large language models (LLMs) have become more capable and ac- curate across many tasks, some fundamental sources of unreliability remain their behavior. One key limitation their inconsistency at reporting the same information when prompts are changed. In this paper, consider the discrepancy between a model's generated answer and their own veri- gap. We define this gap in
fication of that answer, the generator-oalidee more stringent way than prior work: we expect correlation of scores from a generator and validator over the entire set of candidate answers, i.e., can- didate completions that could possibly arise during ordinary language use without breaking Gricean norms. We show that according to this measure, a large gap exists in various settings, including question answering, lexical semantics tasks, and next-word prediction. We then propose RankAlign, a ranking-base based training method, and show that it significantly closes the gap, surpassing all baseline methods. Moreover, this approach generalizes well to out-of-domain tasks and lexical items.
1 Introduction
higher veliditoo lkkathodd
LLMs exhibit instability when prompted in different ways to answer the same question. One clear manifestation of this is the generator- validator gap (Li et al., 2024b; West et al., 2024; Hu & Frank, 2024), where a model may gen- erate answers that it does not verify as cor- rect, or vice versa. Resolving this inconsis- tency would lead to LLMs that report their underlying beliefs more consistently and do not reverse their answers when asked again, and may generally be more useful in evalua- tion settings (Wang et al., 2024c; Zheng et al., 2024).
G-V correlation
higher generator likelhood
Puuses kind
bag,
clocking
0.9
Do you think clothing?
Yes.
clives are kind of
fruit
0.0
you think are fruite?
No.
Suppose that a model places probability mass over several typical answers to a question. Which of these answers should the model val- idate as correct? For model to be consistent, the generator's probabilities should reflect the validator's 's judgments and vice versa: the gen- erator and validator's confidences should be correlated.
Figure 1: LLMs often have a discrepancy between generative and discriminative ver- sions of the same task. They may generate answers that contradict their discriminative judgments or endorse responses in which the generator has low confidence.
Equual contribution
Our code is available httph:: http:::://ggttu...oo///.. http://tthhhb.cmijiiaaara/aaaaaa

---

Published as a conference paper at COLM 2025
In this paper, we introduce new formulation of the generator-validator gap (G-V gap) that considers scores of the entire set of candidate answers at a time. Ideally, we want the model to refrain from generating responses that they disagree with discriminatively, We also expect the model to consistently assess answers even when they are less likely to generate the answers, setting which arises when LMs are used judges to evaluate arbitrary responses (Shi et al., 2024; Wang et al., 2024a; Li et al., 2024a) Both desiderata will be made precise through the correlation of generator and validator scores, which can be thought of as reflecting model credences (degrees of belief). low credence for proposition that one is unlikely to agree with when posed as Yes/ No question; some nuances this issue will be discussed in $2.
would be odd to have
We show empirically that these correlations are low for existing open-source LLMs across range of problems, including question answering, probing for lexical semantic knowledge (hypernymy and synonymy), and next-word prediction.
We then describe a new fine-tuning approach, RankAlign, which uses a pairwise ranking- based loss function to align validator rankings to rankings derived from generator log probabilities. We find this strategy significantly reduces the G-V gap across models by 31.8% average, giving substantially higher correlation between generator and validator across the population of sampled instances. Notably, it outperforms the reverse approach of aligning the generator to the validator. Moreover, our approach generalizes well to unseen tasks and to novel lexical items.
Our main contributions are: (1) a novel formulation of the generator-validatorr gap in terms of correlation between log-odds of generators and validators; (2) a lanking-based training objective for improving correlation between generator and validator close the G-V gap.
2 Problem Formulation
Giving the answer to a question and identifying whether a proposed answer is correct are two conceptually (Campbell, 1960) and computationally (Garey & Johnson, 1979) distinct problems. In the first, generative (generator) case, the LM must select an answer from among a combinatorially large set of options. In the second, discriminative (validator) case, the solution (or set of possible solutions) is presented along with the question, and the LM must select from among small set of options, such as indicating if an answer is correct or incorrect. LMS can give conflicting answers under generator and validator versions of the same question. This discrepancy is known as the Generator- Validdatr gap (Li et al., 2024b) or task demand gap between production and forced choice (Hu & Frank, 2024).
Figure 1 shows example of how this gap clothing, Gemma-2-2B prefers Yes over No, whereas clothing is a less likely continuation of 'Purses are kind of' than is the case for other examples with strong preference for Yes, ranking 32rd out of all examples. The opposite error can also occur. For example, fruit is an extremely likely continuation for 'Complete the sentence: Olives are kind of (in fact, it is the most likely token), even though the LM generates No when prompted with 'Do you think olives are fruit?'
arise. When asked whether purses are
Past work has focused on framing the G-V gap in terms of accuracy of validation decisions with respect to answers sampled from the generator (Li et al., 2024b); however, the example shows that this does not tell the whole story. This formulation fails to consider, for example, the alignment of lower-scoring (but still likely) options. We establish metrics to measure the pervasiveness of this discrepancy, and introduce a new method to help close the gap.
Desired behavior It is clear that cases with high generator scores and low validator scores are unacceptable- teebllleeeeeeeeeee LM is contradicting itself (Li et al., 2024b). The reverse case is more subtle: it may be appropriate to accept propositions (high validator score) that are unlikely to be uttered (low generator score); e.g., "Dolphins are entities." This is due to the
We take the intentional stance (Dennett, 1989) and do not address the issue of whether L.Ms actually have beliefs.
2

---

Published as a conference paper at COLM 2025
difference between believing and uttering proposition: it violation of Gricean principles (Grice, 1975) to say things that are too trivial, specific, or obscure, regardless of one's belief. For example, "Dolphins are entities" is uninformative; "Dolphins specific. Such extreme Gricean violations are rare in our data due to the nature of our tasks ($4.1): concept pairs derived from commonly-ocurriin generated synonyms in context, predictable cloze-style completions, or simple question answering. While the completions vary in terms of typicality (Murphy, 2004), they do not do drastically."
Odonoceti" is too
categories (hypernymy), human-
Setting We consider short-form natural language questionquestions which can be an- swered through single word, entity or multi-word expression. There may be a single correct answer (e.g., TriviaQA), set of correct answers (e.g., asking what superordinate category a concept belongs to), or a set of answers which vary in their plausibility, which arise in more subjective tasks such finding synonyms in context (Kremer et al., 2014), next word prediction (Paperno et al., 2016), or NLI judgmentss (Pavlick & Kwiatkowski, 2019).
Let the generator prompt XG be the sequence of tokens prompting the model to produce an answer, and denote possible answer to the generator prompt by УΛ- УĄ can be any sequence of tokens to which the LM can assign some probability. XG is often asking question about a specific entity or string (e.g,, 'A poodle is a kind of", or 'A synonym of chagrin is'), so we define template function G: z + Xg to construct generator prompts concerning some z of interest (e.g., poodle, chagrin).
To every generator prompt xg, there corresponds an associated validator prompt xv which consists of a polar (Yes/No) question asking whether УΛ is the correct answer to XG, where VΛ can take any candidate answer. We construct validator prompts via templates V: (z, > xv. Let yy be the token generated from the validator prompt, yv ~ PIM( VA))))))
For example, one probe language model's knowledge of hypernymy via the following, where z = poodle and / = mammal:
Generator prompt xG = G(z) = poodle is a kind of
Answer Чл = mammal
Validator prompt xv
= Vzz,yA) = Is it true that poodle is mammal?
Correlation of Log-Odds Validator prompts empirically have the property that most of the probability mass for the completion yy is con- centrated the Yes No tokens.
Class
Pasitive Negative
Valldator log odds
We define the G-V gap so that (1) we measure agreement
function of continuous gener- ator vvll/vaaidtoor scores rather than binary vari- ables, and (2) we evaluate on range of possible completions from the generator. We operational- ize these desiderata by evaluating the G-V gap through the correlation of generator and valida- tor log-odds over a range of (generator, valida- tor) prompts derived from a set of (question, answer) pairs (Figure 2).
-30
-20
-10
10
In the case of a validator prompt, we wish to measure the probability mass of Yes tokens, as ob-
Generator log odds
opposed to everything else. In practice, served the probability mass is concentrated on yes and no tokens, log-odds as:
Figure Generator and validator log- odds for Gemma-2-2B for hypernym pre- diction (Pearson p = 0.76).
define the validator
Moreover, experimental evidence (Hampton, 1998; 2007; Koriat & Sorka, 2015) shows that people's judgment of membership and typicality are in fact correlated.
3

---

Published as a conference paper at COLM 2025
LIPIM(Y: V(EyAA)) (LP V(z,yA))
(Z, = log
(1)
where := [ yes, _yes, Yes, _Yes J and N : [no, _no, No, No are the sets of Yes and No tokens. Similarly, the log-odds of the generator is defined as:
FLM G(z)) 1LMPL G(z))
lg(z,yA) log
(2)
where A the first token of the answer YA, and YA can be any candidate answer xg.
While the space of effective outputs is much larger for the generator than the validator, we can measure preferences for chosen answer continuations both cases using log-odds.
Log-odds Correlation (p) We measure the consistency between generator and validator across the range of natural (ecologically valid) answer choices (, hi via the Pearson correlation (R-all) of their log-odds (Z, and {lv(z, )A))),, (I,, Since this correlation will higher when generators and validators are both accurate (e.g., as in Figure 2), we also evaluate the Pearson correlation between lc and ly restricted to only the set of positive examples P (p-pos), negative examples N (p-neg).5
Training to Improve G-V Correlation
Is a poodle mammal?
RankAlign objectives Given a model that ex- hibits imperfect correlation between generator and validator, how do we go about closing this gap? When considering single datapoint, it is difficult to calibrate via training what precise log-odds val- ues the generator and the discriminator should re- turn. The correlation relationship is only exhibited when looking at larger collections of points time. Our aim is to train model to improve this correlation. We instantiate a simple objective do this, which enforces positive correlation between two sampled prompts. That is, the ranking of the points according to the generator and discriminator must be the same.
Validator log odds
Ut loloogpeess
Uww log p Yes
Generator log odds
Is an olive a fruit?
Figure 3: In RankAlign, pairwise logis- tic loss L log (c (1p u1)) is used to enforce the pair of validator log probabilities Wh,, и4 to be ordered as the generator log probabilities.
We introduce new ranking-based method to close the G-V gap. We wish to increase the correlation between the generator and validator log-odds IG and lv. Rather than optimize for this directly, we can instead encourage the LM to match the rank-
ings of the generator with the validator, or vice versa. Whether one wishes to train an LM to have generator scores aligned with its validator scores, validator scores aligned with its generator scores, may depend on the ultimate use case and on whether the generator or validator is more accurate.
We train the model to produce validator log probabilities that are ranked in the same way as the generator's 's log probabilities. Given a dataset of generator prompt-answer pairs { (xG,, A,),, we compute the generator log probabilities log( ( )) This gives rise
While some answers consist of multiple tokens, observe the first token be highly informative most cases; this is discussed $4.1.
'In some datasets (e.g,, most QA datasets) only positive labels are available since all answers in the dataset correct, in which N Ø.
4

---

Published as a conference paper at COLM 2025
to desired ranking between pairs of corresponding validator prompts { (v,) )
where
2V; X V whenever log(PLM(A) xG))) < log (PPPPMM44 xG.)).
In order to encourage the log-odds for Yes to be higher for Xv, than for XV,, we propose ranking-based loss function:
LG2v (pe) -x ED log or (lllog (Yes xv.,) log (Yes [x)ii]]
(3)
where (-) is the sigmoid function, f is hyperparameter controlling the sensitivity of the preference comparisons, and D is the set of prompts being sampled from, described in $3. Here Po is the LM being trained. Note that are not training the LM to prefer one response over another for a given prompt. Instead, we are training the model to assign higher likelihood to a given completion ("Yes") for one prompt compared to another.
In the case when 6 = 1, Eq. 3 simply minimizes the logistic cross-entropy loss over pairs (i,, Hw.), L = - log Log(o((us (o ui)) where the scores Ww := log Pe YYes xv,)) and un == log Pa(Yes xv,) are the log-probabilities under the validator (Figure 3). This en- courages uw to be greater than Ry.
RankAlign-V2G Alternative We also explore a variant of RankAlign, RankAlign-V2G, where we train the LM to produce log probabilities for the generator completions that match the ranking of the validator. The formulation is similar to the above, except for the slight asymmetry between the tokens over which log probabilities are computed. We rank pairs of generator prompts { (XG,, XG,)) where XG; < XG2 whenever (PLMP
xv)) < log(Pr M (Yes | xv,)),, and then use the same form of the loss as Eq. 3, except the prompts come from the generator, and the continuations are the associated answers Aw, Al.
CV26(Pa) (xmxx)eD log or (6log (a [xG.) - log Po(pee [x,)))
RankAlign-- V2G losses:
Finally, we explore the convex combination of RankAlign and RankAlign-
Cv2c+v22(Pe) = aLc2v + (1-2))22((PP))
(5)
in order to test whether aligning in both directions simultaneously can both reduce the G-V gap and achieve high generator and validator accuracy. We set 0.5 in our experiments.
Relation to DPO The surface form of the ranking loss in RankAlign resembles DPO (Rafailov et al., 2023) but has fundamental differences. It pushes the LMs to establish preferences between fixed completions given pairs of prompts, rather than pairs of completions given fixed prompls. DPO also involves comparison with the reference model to prevent the LM from straying too far from the initialization (Rafailov et al., 2024). We perform an ablation by adding Pref LG2V and LV2G in Appendix SC.2 to explore the effect of adding reference model in our setting.
Generator Sampling (RankAlign) To train with our RankAlign objective, we need to sample pairs of prompts XG and answers A such that there is clear ranking be- tween them. We sample pairs with a minimum margin separation of 8, i.e., such that (og(PMM(A.) (Aw X).,.) - log( (LM(A) XG))) > 8 for some d > 0.7 Pairs are sampled uni- formly at random over the dataset and filtered based on this criterion. We also report an ablation where pairs only sampled over the set of positive examples P in section $5.
Other hyperparameters for training are described in Appendix SA.
6Fture work could consider loss that selectively chooses the alignment direction per sample, based on whether generator or validator are more accurate ?Preliminary experiments with Gemma-2-2F across all tasks showed drop in both correlations and accuracy when setting 5 O.
5

---

Published as a conference paper at COLM 2025
4 Experimental Setup
4.1 Tasks and Datasets
We evaluate on four datasets, covering lexical relations (hypernymy and synonymy), next word prediction (cloze task), and question answering (QA). Generator and validator prompt templates for these tasks are given Appendix SF.
Hypernymy, the IS-A relation, is a lexical relation between words and their superordinate categories (e.g., (bee, insect) or food). We use the dataset from Rodriguez et al. (2024) which extends the set of hyponym-hypernymm pairs from THINGS (Hebart et al., 2019; 2023) with negative examples with varying degrees of conceptual similarity. The dataset is balanced between positive and negative examples.
Hypernymy (THINGS)
Lexical substitution (SWORDS) The lexical substitution task, or synonymy in context, is to determine whether one word can be substituted by another in a given context without altering its meaning. We use the SWORDS dataset (Lee et al., 2021), a high-quality, broad coverage dataset of (context, target word, substitutions) triples. Details on how we leverage this dataset are given in Appendix $B.1.
Next word prediction (LAMBADA) We adopt the LAMBADA dataset to evaluate models' capability of predicting the next word following given passage (Paperno et al., 2016). The dataset is designed so that final word prediction relies on the whole passage rather than just the last sentence, necessitating the understanding of the broad context.
Question answering (TriviaQA) In knowledge question answering, models are tasked with providing answers using their parametric knowledge. We use the open-domain QA dataset TriviaQA (Joshi et al., 2017) and present it to models without evidence documents.
4.2 Metrics
In addition to our main metrics p-all, p-pos and p-neg as defined in $2, we also employ other metrics to measure the accuracy of methods on their respective tasks, and to compare against previous work (Li et al., 2024b).
Validator performance metrics When the dataset contains both positive (Yes) and negative (No) labels, we can measure the validator accuracy via the area under the ROC curve of the validator log-odds Ig. This makes it possible to fairly compare methods which shift the entire distribution of bog-probabilities. For the datasets with only positive labels, we use recall of the validator when using to classify with a log-odds threshold of 0 (R@0).
Generator performance metrics We evaluate the accuracy of the generator via the Mean Re- ciprocal Rank (MRR-P) over the ranks the correct (positive) answers A, under PLM(- xc,). Similarly, we evaluate the Mean Reciprocal Rank (MRR-N) over the ranks of the incorrect answers A; under P.M(- xG. ), i.e. over the negative examples. Smaller values are better in this case, because incorrect answers should have higher rank. We also evaluate the accuracy of the generator by calculating the rank of each answer A; under the LM with the generator prompt XG, predicting positive if it falls below rank threshold 100 and negative otherwise, and computing the accuracy over the set of {xX,, A;) N pairs. We denote this Acc@100.
G-V Consistency In order to compare against previous work, we also evaluate G-V Consistency, which was introduced in Li et al. (2024b). G-V Consistency measures the accuracy of the validator when presented with answers УΛ which are the (top) generations from the generator, Y ~ PLM( xç). It measures aligmment between the generator and discriminator, but only when the generator is highly confident.

---

Published as a conference paper at COLM 2025
4.3 Baselines
To evaluate the effectiveness of RankAlign, we compare it against the following baselines in addition prompting the Base model.
SFT We fine-tune the base models with both generator and validator (prompt, comple- tion] pairs over positive examples, i.e., { (XG,, ) and corresponding (xv, Yes) J, where yA) E P. This measures the effect to which any in-domain training will increase G-V consistency by making both the generator and validator stronger.
We evaluate the performance of the consistency fine-tuning method presented in Li et al. (2024b), which fine-tunes the model over examples where the gen- erator and validator agree according to binary agreement criterion. In our setting, the yAS are not necessarily outputs of the generator, so we filter consistent pairs in a slightly different way. Specifically, we set thresholds tg = E[|c] and tv = [[ly] as the aver- age generator inveerttorvalldattee /vliddaoo log-odds over all examples, and keep examples {(z, where (l,
Consistency FT
tGl 1[ly > fv], ie., where generator and validator agree.
DPO Finally, we compare against a variant of DPO that attempts to close the G-V gap by aligning the validator with the generator based the generator's 's assessment of answers, or vice versa. DPO-V2G, given a validator prompt xy = V ((z, yA), we set (y/wnn Vlsse = (Yes,No) iff yA)( > tG and vice versa. Similarly, in DPO-G2V, we push the generator towards the validator by sampling pairs of (ywin, \/oos) such that ly (w,n) > lv Vose)).
4.4 Models
We evaluate all methods on four models: Gemma-2-22, Llama-3.2-33 Baama-3.2-3B-nnstrree and Llama-3.1-8B. We test are harder for non-instruction-turee models than Hypernymy or LAMBADA. We detail the hyperparameters and other training setting in Appendix SA.
on TriviaQA and lexical substitution, which
Results
Results on the Gemma-2-2B model experiments are shown in Tables 2 and 1. Full results for the other models follow broadly similar trends and are given Appendix SC. Our main objective is to improve the gemerator-validator correlation, but it is also important that generator and validator performance does not degrade substantially. In particular, when aligning the validator to the generator, the generator should be minimally impacted, and vice versa.
Task
Method
p-pos
R@0
A@100
MRR-P
5.1 In-domain Experiments
Base
6.1
90.8
99.3
79.0
SFT Consistency FT
9.8
100
99.7
83.5
RankAlign is effective at clos- ing the G-V gap On all models and tasks, RankAlign substan- tially enhances the correlations between generator and valida- tor, with an average gain of 31.8 on Pall- It outperforms all base- lines both overall and within individual classes. For Hyper- nymy, RankAlign nearly closes the G-V gap, with correlation p-all of 94.2 and per-class correla- tions of 87.5 and 89.0 for the posi- tive and negative classes, respec- tively. On SWORDS (Table 2),
LAMBADA
17.1
100
99.7
83.1
DPO V2G
-11.8
100
82.8
54.6
Ranking G2V
60.0
67.7
94.8
52.9
Ranking V2G
11.3
95.5
98.5
68.8
RankA
57.2
2.7
97.3
74.0
Base SFT
19.4
63.7
88.6
52.8
18.4
99.9
90.5
59.3
Consistency FT
20.1
99.9
90.9
59.3
TriviaQA
DPO V2G
18.2
85.7
50.3
Ranking G2V
56.8
39.8
44.5
9.6
Ranking V2G
29.9
99.9
71.6
17.0
RankAlign-w 73.1
75.2
80.0
36.9
Table 1: Performance metrics across methods for LAM- BADA and TríviaQA with the Gemma-2-2B model.
7

---

Published as a conference paper at COLM 2025
Task
Method
p-all
P-pos
p-neg
ROC
A@100
MRR-P
MRR-N (4)
Base
76.4
54.8
45.1
97.0
83.7
19.4
1.6
SFT
86.7
40.0
60.0
98.3
72.7
11.0
Consistency FT
73.0
49.1
34.5
97.9
49.7
66.1
17.7
Hypernym
DPO-G2V
77.6
53.6
49.8
97.0
85.3
20.1
1.5
DPO-V2G
80.8
64.2
57.6
94.0
84.3
16.4
1.3
RankAlign
94.2
87.5
89.0
93.5
83.8
22.1
1.6
RankAlign-V2G
87.1
73.2
73.7
95.6
90.2
43.1
1.1
RankAlign-a
89.3
82.1
83.1
93.9
84.6
41.8
3.7
Base
58.4
32.8
36.5
86.1
77.7
17.9
2.0
SFT
58.6
35.2
36.3
83.8
79.9
32.5
3.1
Consistency FT
55.0
31.6
31.4
83.7
77.7
31.0
3.5
SWORDS
DPO-G2V
57.3
31.3
36.4
85.9
78.8
20.0
2.1
DPO-V2G
-1.8
-6.8
1.1
50.0
77.5
20.4
2.5
RankAlign
76.6
67.6
60.4
87.0
51.1
0.02
0.0
RankAlign-V2G
47.1
29.6
33.9
72.4
79.3
19.2
2.9
RankAlign-a
81.0
69.2
67.1
87.9
78.2
17.6
2.3
Table 2:
Detailed performance metrics across tasks and methods for Gemma-2-2B on the Hypernymy and SWORDS datasets.
RankAlign increases p-all from a base 58.4 to 76.6, and roughly doubles p-pos and p-neg (from 32.8 and 36.5 to 67.6 and 60.4, respectively). Similar improvements from RankAlign also hold for Llama-3.2-3B and Llama-3.1-8B (Tables 7, 11).
...and has only mild degradation on task accuracy In the case of Hypernymy, the im- provement in correlations for RankAlign has little effect on model performance. Validator ROC from the base model slightly decreases (97 to 93.5 for Gemma-2, and 95.9 to 93.5 for Llama-3.2), while generator accuracy stays roughly constant, with similar trends observed for Llama. On the other hand, for SWORDS, RankAlign improves validator ROC at the expense of generator accuracy. ROC increases from 86.1 to 87 while both Acc@100 and MRR-P decrease (77.7 to 51.1 and 17.9 to 0.2, respectively). The effects model accuracy for LAMBADA and TriviaQA are mixed. While AankAlign with Llama and Llama-Instrue has little effect the generator accuracies of these datasets (Tables 8, 10), it causes large drop for Gemma (a decrease in MRR-P from 79 to 52.9, and from 52.8 to 9.6 for LAMBADA and TriviaQA, respectively; Table 1). While the SFT baseline does not outperform RankAlign- V2G on it has consistently high Acc@100 values and the highest MRR-P scores across models and datasets.
RankAlign variants Across all tasks and models (Tables 2-10), see that RankAlign significantly outperforms RankAlign-V2G; on improving correlations, while Rank Align-a performs similarly. appears that there is not enough information in the ranking of validator probabilities to successfully shift the distribution for the generator. We hypothesize that the generator's 's distríbution may be more precisely calibrated. Base LLMs are often calibrated (Kadavath et al., 2022a), and the generator is essentially using the model as base LLM. By contrast, the capacity to as a discriminator is more heavily induced by alignment, and aligned LLMs may exhibit lower calibration (Zhu et al., 2023).
5.2 Cross-domain Experiments
Next we investigate whether our methods generalize out of domain. Ideally, one would like the G-V gap remain small even in settings farther from the training set. We consider three cases: generalization across tasks, generalization to new lexical items, and generalization to different prompt formats.
Generalization across tasks We next evaluate the correlations of RankAlign in cross- domain fashion, i.e. training them on one dataset and evaluating them on another. While correlations are naturally lower than training and evaluating on the same dataset,
8

---

Published as a conference paper at COLM 2025
Hypemnyyy Ap-pos
SWORDS
LAMBADA
TriviaQA
Ap-all
Ap-neg
Ap-all
Ap-pos
4p-neg
Ap-pos
Ap-pos
Gemma-2
17.8
32.7
43.9
13.4
19.1
14.4
-10.9
19.7
Hypernmyy
Llama-3.2
16.5
33.2
46.1
11.8
14.4
10.6
0.5
11.9
Gemma-2
4.5
18.3
12.8
18.2
34.8
23.9
7.8
17.6
SWORDS
Llama-3.2
6.5
9.2
21.6
26.2
40.8
31.7
0.6
14.6
Gemma-2
-6.8
-8.7
-12.3
-26.5
-15.6
-20.9
57.8
-21.4
LAMBADA
Llama-3.2
-0.6
-2.8
-0.2
-5.7
-5.2
-3.6
45.2
-15.5
Gemma-2
-20.3
-12.7
-19.4
-18.6
-2.2
-18.3
1.9
37.4
TriviaQA
Llama-3.2
-0.3
7.5
22.4
13.3
-0.6
50.7
Table 3: Cross-dataset evaluation showing the difference in p scores (between the RankAlign trained model and the base LM) when training on a dataset (row) and evaluating on target datasets (column). Scores greater than 0 indicate that RankAlign generalizes across tasks. Values in gray are the in-domain results derived from Tables 2, 1, 8.
consider our method to have generalized the correlations exceed those of the original base model. The deltas in correlations between the fine-tuned models and the base models across domains are given in Table 3.
RankAlign generalizes well to OOD tasks when trained on Hypernymy and SWORDS in general, with large improvements over the base LM. On the other hand, LAMBADA and TriviaQA show limited generalizability to other tasks on Gemma-2, whereas they do
transfer on Llama. This may be because LAMBADA and TriviaQA are more difficult and less targeted tasks, making it hard to modify the model in a systematic way.
Lexical Generalization We evaluate whether RankAlign generalizes under varying degrees of lexical overlap be- tween the train and test sets for the Hy- pernymy task, shown in Table 4. We compare the previous results from Ta- ble 2 (Random split) against a setting where the train and test sets have no overlap between their hypernyms ya (No hypernym overlap), and where there no overlap between either hyponyms z or hypernyms Y (No overlap). 10 We find that RankAlign generalizes well in these new settings, with only a slight decrease in correlations, while SFT continues underperform here.
Train/ Teet split type
Method
p-all
p-pus
p-neg
Random split
Base
76.4
54,8
45.1
SFI
86.7
40,0
60.0
RankAlign
94.2
87.5
89.0
No hypernym overlap
Base
81.8
60.4
67.3
SFT
85.0
68.9
60.1
RankAlign
92.3
84.6
84.7
Base
75,9
53,0
32.5
No overlap
SFT
82.2
4.0
56.4
Rank
93.3
85.5
86.9
Table 4: Results on the Hypernymy task with vary- ing lexical overlap between train and test splits.
Generalization across prompt formats Given that RankAlign was trained with a fixed set of generator and validator prompt templates, we evaluate whether generalizes to closing the gap between variants of the generator and validator prompts. For the Hypernymy task, ( we consider prompt variants such as "Is the case that ... and "I love and other". (Detailed prompts in Table 19.) We find that correlations drop when compared to the original in- prompt (Table 5), but still greatly surpass the Base model, showing that RankAlign generalizes to unseen prompts.
Fulll results comparing how well methods generalize from SWORDS to Hypernymy are shown in Table 15. We note that RankAlign also has higher correlation than the baselines in this cross-domain setting, with generator and validator accuracy similar or higher than the base model. Additional results for RankAlign-a can be found in Table 16
"To test if the lack of transfer stems from LAMBADA and TriviaQA containing only positive examples, we re-trained them with balanced positive and negative samples (details Appendix $B.2), but observed minimal performance gains.
Details on the construction of these alternative train/test test splits given in Appendix $B.3.

---

Published as a conference paper at COLM 2025
6 Related Work
Model
G-V Prompt
p-all
p-pos
p-neg
Language Model consistency
Gemma-2 (Base)
Training prompt
76.4
54.8
45.1
Language models are expected to be consistent in reporting their beliefs even when queried differ- ently. Prior work has explored the instability of LMs with prompt paraphrasing (Sclar et al., 2024; Elazar et al., 2021; Moore et al., 2024), different option orders (Li et al., 2024a; Zheng et al., 2024; Ding al., 2024), inconsistency between token probability and output (Wang et al., 2024c;b; Wen et al., 2024; Song et al., 2025), and between logically related propositions (Li et al., 2024c; Cohen et al., 2024; Yin et al., 2024). Specifically, Li et al. (2024b) study the inconsistency where the models disagree with their own generative responses when prompted discriminatively. While they treat the generator- valldatoo consistency as binary agreement problem and only evaluate over candidate answers generated by the model, we provide a broader perspective, arguing that the generator and validator should align across the entire set of candidates and examples.
RRankAlign
94.2
87.5
89.0
Gemma-2 (Base) RRankAlign
78.0
51.9
7 82.9
Variant
92.0
84.0
Gemma-2 (Base) RRankAlign
68.4
22.8
35,9
Variant
80.3
43.9
66.8
Gemma-2 (Base) RRankAlign
53.6
21.8
21.0
Variant
76.6
57.0
64.5
Table 5: Correlations when evaluating RankAlign different generator-validator prompt pairs for the Hy- pernymy task. Prompt variants are shown in Appendix SD). Values in gray are from Table 2, for comparison.
Language Model as evaluators LLMs are widely used to asses their own responses for refinement (Press et al., 2023; Wadhwa et al., 2024; Feng et al., 2024), self-alignment and scalable oversight (Sun et al., 2024; Wu et al., 2024; Jiang et al., 2024; Bowman et al., 2022), as well as acting as judges and providing nuanced insights to open-ended generation (Dubois et al., 2024; Cui et al., 2024). While LLMssas-judges is promising alternative to human evaluation, it is critical to understand and enhance their reliability (Shi et al., 2024; Zhou et al., 2024; Li et al., 2024d). The items to be evaluated may come from any distribution, not just those that the validator model has high confidence in. Therefore, we argue that LMs should consistently express their assessment over any candidate.
Knowledge and belief in Language Models is common attribute propositional attitudes (Nelson, 2024) such as belief to LMs (Jiang et al., 2020; Kadavath et al., 2022b, i.a.). Several studies have discussed whether LMs can have beliefs and what normative constraints (Lin, 2024) would need to be enforced for the LM's beliefs to be consistent (Hofweber et al., 2024; Hase et al., 2024; Fierro et al., 2024). The ranking perspective in this paper is also conceptually similar to Spohn's ranking theory of belief (Spohn, 2009) which assigns ordinal ranks to propositions. Finally, Gekhman et al. (2025) find that LLMs encode significantly more factual knowledge in their parameters than they express in their outputs.
Conclusion
In this paper, we present new view of the generator-validator gap based on correlation between log-odds assigned under a generator and a validator. We describe a new method for training models to exhibit stronger correlation via ranking loss between pairs of examples. Results show that our RankAlign significantly outperforms baselines with mild task accuracy degradation, is robust to prompt variations, and generalizes well to unseen data and tasks.
We believe future work can improve the generalization of these gains further and also seek to mechanistically understand the origins of these gaps, leading to new methods for more reliable language modeling. Other directions include extending the investigation to longer- form completions, which are found in LLM evaluations of summarization or long-form QA (Xu et al., 2023). On the theoretical side, would be interesting to formulate the desired (ideal, rational) behavior relating generator and validator probabilities to wider range of cases which violate Grice's maxims. This forces one to grapple fully with the distinction between beliefs and speech acts One approach to this could be to take inspiration from Rational Speech Act theory (Frank& & Goodman, 2012) in order to model what a rational speaker should utter (rather than believe) under generator and validator settings.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
This work was supported by NSF CAREER Award IIS-2145280, the NSF Al Institute for Foundations of Machine Learning (IFML), a grant from Open Philanthropy, and Good Systems, 11 a UT Austin Grand Challenge to develop responsible Al technologies. We thank the anonymous reviewers whose helpful feedback helped improve the work.
References
Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilè Lukošiatë, Amanda Áskell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. ar preprint ar (rXi:2221..0354/ 2022.
Donald T Campbell. Blind variation and selective retentions in creative thought as in other knowledge processes. Psychological review, 67(6))380, 1960.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the
ripple effects of knowledge editing in language models. Transactions of the Associa- fion for Computational Linguistics, 12:283-298, 2024. doi: 10.1162 ttcl a 00644. URL https: http///aaannooloo.ooorooooo
tacl-1.16/.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guo- tong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. ULTRAFEEDBACK. Boosting language models with scaled Al feedback. Forty-first International Conference on Machine Learning, 2024. URL
Daniel C Dennett. The intentional stance. MIT press, 1989.
Wenxuan Ding, Shangbin Feng, Yuhan Liu, Zhaoxuan Tan, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge crosswords: Geometric knowledge reasoning with large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 2609-2636, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/vi 2024 0024.findings--cl..1 URL https: /tc//aallannnn aclaathllogyyyy.... org/2024. 1.44.finniin findings-add aatpgss-ac.11111
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori Hashimoto. Length-controoled alpacaeval: A simple way to debias automatic evaluators. Xrviv preprint ar Xiio:2404.04475, 2024.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hin- rich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012-1031, 2021.
Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don't hallucinate, abstain: Identifying LLM knowledge gaps via multi-LLM collaboration. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meefing of the Association for Computational Linguistics (Volume Long Papers), pp. 14664-14690, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653 00.18653/v1/2000 http/////222...llooo..... URL https://aclaatthoolge org/ 2tgg/2224 2024. ocl-long.786. 786/.
Constanza Fierro, Ruchira Dhar, Filippos Stamatiou, Nicolas Garneau, and Anders Søgaard. Defining knowledge: Bridging epistemology and large language models. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 16096-16111, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/202 emnlp-main.900. URL httpsh:://aaaaaahllllloooo org/
org/200 2024. wtt.2222222mmmmmmmeemmmmm mmnlp-main... 900//
"https:// httpss//gooddyteesstteaaccel
11

---

Published as a conference paper at COLM 2025
Michael C. Frank and Noah D. Goodman. Predicting pragmatic reasoning in language
games. Science, 336(6084):998-99) 2012. doi: 10.1126/ (0.1126/scciit science.1211663.. URL https:
//www.science... www..sccencee.rr/doo////eee wrt//o//a///////////////ooo///// /116/science....ee 121.122222 1218633.
M. R. Garey and David S. Johnson. Computers and Intractabilityy A Guide the Theory of NP-Completeness. W. H. Freeman, 1979. ISBN 0-7167-1044-7.
Zorik Gekhman, Eyal Ben-David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, and Roi Reichart. Inside-out: Hidden factual knowledge llms. In Second Conference on Language Modeling, 2025.
H. P. Grice. Logic and conversation. In Peter Cole and Jerry L. Morgan (eds.), Syntax and Semantics: Vol. 3: Speech Acts, pp. 41-58. Academic Press, New York, 1975. URL pdf.
http://wwn www. ac. wttp://.ww....///ttuutu/////eee.
James A Hampton. Similarity-based categorization and fuzziness of natural categories.
Cognition, 65(2-3):137-165, 1998.
James A Hampton. Typicality, graded membership, and vagueness. Cognitive science, 31): 355-384, 2007.
Peter Hase, Thomas Hofweber, Xiang Zhou, Elias Stengel-Eskin, and Mohit Bansal. Funda- mental problems with model editing: How should rational belief revision work in Ilms? ar Xrviv preprint arXi::2406. 19354, 2024.
Martin N Hebart, Adam H Dickter, Alexis Kidder, Wan Y Kwok, Anna Corriveau, Caitlin Van Wicklin, and Chris Baker. Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PloS one, 14(10):e0223792, 2019.
Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris I Baker. Things-data multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. eLife, 12:e82580, feb 2023. ISSN 2050-084X. doi: 10.7554/ 82580. URL / //toi.///////// org/10 7554/e5555 wewee...ife... 825800.
Thomas Hofweber, Peter Hase, Elias Stengel-Eskin, and Mohit Bansal. Are language models rational? the case of coherence norms and belief revision. CoRR, abs/ 0/5//2406.04444 2024. doi: 10.48550/ ARXIV.2406003442. URL https: / //oii org/10. 48550/arXiv.. 2406. 03442.
Jennifer Hu and Michael Frank. Auxiliary task demands mask the capabilities of smaller language models. In First Conference Language Modeling, 2024.
Dongwei Jiang, Jingyu Zhang, Orion Weller, Nathaniel Weir, Benjamin Van Durme, and Daniel Khashabi. Self-[in] correct: Llms struggle with discriminating self-generated responses. arv preprint arXio:2404.04298, 2024.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8: 423-438, 07 2020. ISSN 2307-387X. 1162/tacl.a.00224
doi: 1a11162/tacla.0000 URL https://doo org/10.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer: TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzi- lay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi:
URL
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfiel-Doddss, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer,
12

---

Published as a conference paper at COLM 2025
Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. CoRR, abs/2207.0522) 2022a. URL https://dii.orr org/10 (8550/arXiv.2222 .0221..
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. CoRR, URL https://doi. org/10.
/ 220222221, 2022b. doi: 10.48550/ ARXIV.2207.05222. 48550/arXiv.. 20..222222222 05221.
Asher Koriat and Hila Sorka. The construction of categorization judgments: Using subjective confidence and response latency to test a distributed model. Cognition, 134:21-38, 2015.
Gerhard Kremer, Katrin Erk, Sebastian Padó, and Stefan Thater. What substitutes tell us analysis of an "all-words" lexical substitution corpus. In Shuly Wintner, Sharon Goldwater, and Stefan Riezler (eds.), Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. pp. 540-549, Gothenburg, Sweden, April 2014. Association for Computational Linguistics. doi: 10.3115 h1///v1///111/1 URL https: / /ttp://aaaaaaaaoo....oooo org.:/4-1/5////
Mina Lee, Chris Donahue, Robin Jia, Alexander Iyabor, and Percy Liang. Swords: A benchmark for lexical substitution with improved data coverage and quality. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Pp. 1362-4379, Online, June 2021. Associ- ation for Computational Linguistics. doi: 10..8853// / :ttps://aclnnthology.ooom org/ naacl-main.
URL
Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, Qian Dong, and Yiqun Liu. Calibraeval: Calibrating prediction distribution to mitigate selection bias in llus-ss-judges. ariio preprint ar xrXvv22400113333 2024a.
Xiang Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto, and Percy Liang. Benchmarking and improving generator-validator consistency of language models. In The Twelffh International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL / net net/ForrumlddphBBsyyyza
Zhoubo Li, Ningyu Zhang, Yunzhi Yao, Mengru Wang. Xi Chen, and Huajun Chen. Unveil- ing the pitfalls of knowledge editing for large language models. In The Twelfth International Conference on Learning Representations, 2024c. URL https: //tp:://///eeeeiewww net net/fooummidd= FNktD3ib16.
Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. Split and merge: Aligning position biases in LLM-based evaluators. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11084-11108, Miami, Florida, USA, November 2024d. Association for Computational Linguistics. doi: 211853/v1/2022 emnlp-main.6221 URL httphttts///aee htthttts///aclaatholooy// org/2024. ennlp-main.622
Hanti Lin. Bayesian Epistemology. In Edward N. Zalta and Uri Nodelman (eds.), The Stan- ford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2024 edition, 2024.
Jared Moore, Tanvi Deshpande, and Diyi Yang. Are large language models consistent over value-laden questions? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen
13

---

Published as a conference paper at COLM 2025
(eds.), Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 15185- 15221, Miami, Florida, USA, November 2024. Association for Computational Linguis- /v/5553/1v/// tics. doi: 10.18653/ v1 www1//22..finmt (004..indiins--eml..8999 URL https:///allanthoooogyy org/2024.
Sindings-emnlp. 891/-
Gregory Murphy. The big book of concepts. MIT press, 2004.
Michael Nelson. Propositional Attitude Reports. In Edward N. Zalta and Uri Nodel- man (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2024 edition, 2024.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525-1534 Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/vi 10.18653/v1/P16-111 v1/116-1114..
URL https: httt::::/aalanttiie http://aacaaaaoooo...oooooo
Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677-694, 11 2019. ISSN 2307- 387X. doi: 10.162/taccla.000
URL htttht:://doi.oorg/
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687-5711, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ 221/223..indings-eeee... findings-emnlpp
URL https:
//aclanthology.org org/2023.323/32//0 wwtpaa2....iiinnnggggee findings-emiin em.... 378/.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL
https://openrev/// nett:orrmmid=HPPSSTI
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sushil Sikchi, Joey Hejna, Brad Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. Advances in Neural Information Processing Systems, 37: 126207-126242, 2024.
Juan Diego Rodriguez, Aaron Mueller, and Kanishka Misra. Characterizing the role of similarity in the property inferences of language models. CoRR, abs /2410.2259, 2024. doi: 10.48550/ ARXIV.2410.22590 URL https:///oat org/10. /8550//axiv.... 2410. 22590.
Melanie Sclar, Yejin Choi, Yulia Isvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How Iearedd to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OvenRevie 2024. URL https://openrevviw. net/forum?id-RRUSII
Lin Shi, Chiyu Ma, Wenhua Liang, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. arv preprint ar xrXio:2406.07791, 2024.
Siyuan Song, Jennifer Hu, and Kyle Mahowald. Language models fail to introspect about their knowledge of language. arviv preprint arXiv:2503.07513, 2025.
Wolfgang Spohn. A survey ranking theory. In Degrees of belief, pp. 185-228. Springer, 2009.
Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Daniel Cox, Yiming Yang, and Chuang Gan. SALMON: Self-alignment with instructable reward models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevieer wwwpeeeewwwwwweeeeeeeeeeeeeee
14

---

Published as a conference paper at COLM 2025
Manya Wadhwa, Xinyu Zhao, Junyi Jessy Li, and Greg Durrett. Learning to refine with fine-grained natural language feedback. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 12281-12308, Miami, Florida, USA, November 2024. Association for Computational finniings-eenll.7166 URL https://aclanthollogy
Linguistics. doi: 10.18653/V org/2024. handings-emnlp... 716/.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9440- 9450, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653 v1 2022..cl-looo..5550
URL
http:aaatttooooooooo/2222/e. org/2024. acl-long.55 511/.
Xinpeng Wang, Chengzhi Hu, Bolei Ma, Paul Rottger, and Barbara Plank. Look at the text: Instruction-tuned language models are more robust multiple choice selectors than you think. In First Conference on Language Modeling, 2024b. URL https: http::://opeeeevieeee net/ forum?iid-qddAA856yy.
Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. "My answer is C": First-token probabilities do not match text answers in instruction-tuned language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 7407-7416, Bangkok, Thailand, August 2024c. Association for Computational Linguistics. doi: 10.18653 v1/2024.1200 1122024.findiggs-de aadingg--acl4411 URL https: hatsss///llaaooogg org/ 2024. findings-acl. act.ggsacc..444 441/.
Bingbing Wen, Chenjun Xu, Bin HAN, Robert Wolfe, Lucy Lu Wang, and Bill Howe. From human model overconfidence: Evaluating confidence dynamics in large language models. In NeurlPS 2024 Workshop on Behavioral Machine Learning, 2024. URL https:
Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Raghavi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi. The generative Al paradox: "what can create, it may not understand". In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:///pperreview. net/
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with -lm-as-a-meta-judge. arviv preprint ar drXiv:2407.1955 19594, 2024.
Fangyuan Xu, Yixiao Song, Mohit lyyer, and Eunsol Choi. A critical evaluation of eval- uations for long-form question answering In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 3225-3245 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 /1/0 /wwp///........ooooo..... https://aclanthollogy wwwpooloooyoogg/222. org/2023 long.
URL
181/ 181/
Fangcong Yin, Xi Ye, and Greg Durrett. Lofit: Localized fine-tuning LLM representations.
In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
https://openreview. wtwprrewwwwwweeefoouunuuuuaa (ttefforummiddddixFFFFFFF
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2024. URL ht //tp://oopeeieeeeeeeeeee
Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In The Twelfth International Conference on Learning Representations, 2024. URL wttp://wwww.eeeooriiiiiiiiiiiii
15

---

Published as a conference paper at COLM 2025
Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao. On the calibration of large language models and aligmment. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: http//pprrriiieeeeeeeorrrrr///// Of2xc2GVid.
16

---

Published as a conference paper at COLM 2025
Model
Method
Hypernym
SWORDS
TriviaQA
LAMBADA
Base
0
SFT
4497
1052
6000
8306
DPO G2V
3000
696
3000
4153
Gemma-2-2B
DPO V2G
2149
329
Consistency FT
2152
390
1756
3156
RankAlign
3491
3919
3753
779
RankAlign-V2C,
3938
2690
3039
RankAlign-t
3491
3919
3203
3644
Base
0
SFT
4497
1052
6000
8306
DPO G2V
3000
696
3000
4153
Llama-3.2-3B
DPO V2G
2149
329
Consistency FT
2134
1884
2968
RankAlign
3386
3146
2537
865
RankAlign-V2G
4408
3762
986
1831
RankAlign-a
3386
3146
3019
3523
Base
0
0
SFT
1052
6000
DPO G2V
3000
Llama-3.2- -lama-3.2-Instruct
DPO V2G
Consistency FT
530
2788
RankAlign
3305
2902
RankAlign-V2G
4732
4019
RankAlign-a
3312
3573
Base
0
SFT
4497
1052
6000
8306
DPO G2V
3000
696
3000
4153
Llama-3.1-8B
DPO V2G
2149
329
Consistency FT
2364
510
2242
3214
RankAlign
3274
3116
3242
3484
RankAlign-VZZ V2G
4168
4505
3530
4598
RankAlign-a
3274
3166
3242
3484
Table 6: Training data size for each task and model.
A Hyperparameters and other experimental details
Hyperparameter details For SFT, we set the learning rate to 2e-5 and train for epoch; for DPO-V2G, we set the learning rate to le-5 and train for epoch, while in DPO-G2V, the learning rate is set to 2e-6 and train for 2 epochs according to preliminary results. In Consistency FT, we train with a learning rate of 2e-5 for 2 epochs.
learning rate of 1e-5 and train for 2 epochs. We use a minimum distance margin of 5 = 2.5 for RankAlign and = 0.15 for RankAlign- V2G in all cases (except for Llama-3.2-3B on TriviaQA, where found substantially better results with ó = 0.08). For Hypernymy, we also experimented with =0, 2.5, and 5, and found 2.5 worked best.
For RankAlign and RankAlign-V22 we
Dataset size for training and testing Our different training methods require different sampling strategies; note in particular that Rank-lign-V2G and RankAlign train on sampled pairs of instances, allowing them to use more data in some cases than methods like SFT that train on single instances.
For SFT, we keep only positive examples and train on both generator prompts and validator prompts. For Consistency FT, we keep examples where the assessment the generator and validator agree and train both generatively and discriminatively, For DPO and RankAlign, we sample pairs of data based on thresholds; this is described in $3 and $4.3. The specific number of examples used for training are presented in Table 6.
17

---

are shown in Table Published as a conference paper at COLM 2025
B Dataset construction details
B.1 Dataset construction for SWORDS
The SWORDS dataset contains ratings from human annotators on how appropriate a set of replacement words as substitute for target word in given context. We use this to select positive (good lexical substitutes) and negative (poor lexical substitutes) as answers yA for each context and target, by selecting the highest-ranking substitute as the positive answer and the lowest-ranked substitute as the negative answer.
We filter out examples where the target and replacement are the same word, where the replacement starts with a stopword (the, be, a, in, yet, at, by, do, dont, we, and, even, to, with), or where the replacement is a multiword expression with over words. This removes roughly 4% of the data.
B.2 Negative sampling for TriviaQA and LAMBADA
To investigate whether the lack of transfer can be attributed to the fact that LAMBADA and TriviaQA only consist of positive examples, we generate negative answers by prompting Qwen2.5-7B-Instruct, Specifically, for TriviaQA, we prompt with "Generate an incorrect answer to the following question directly in less than 3 words.", and filter out responses that have an F1 score larger than 0.3 to reduce false negative answers. For LAMBADA, we prompt the model with "Generate an incorrect answer to the question in one word." and filter out responses that have Jaccard-similarity greater than 0.5 to the gold completion. For each task, we then downsample to 3,000 examples consisting of balances positives and negatives and re-train with this set of data. We evaluate the resulting models on the same test set as in Table 1, which only consists of positive cases for fair comparison.
B.3 Alternative train/test splits for Hypernymy lexical generalization experiments
In order to investigate the lexical generalization of RankAlign, we constructed alternative train/test splits, with results shown in Table 4. The Random Split is the default split used throughout our paper, with a random sample of 3000 training and 1000 test samples.
For the No hypernym overlap setting, we made sure that there were no overlaps in hypernyms between train and test sets. Since there are only 44 unique hypernyms in the dataset, we manually isolated a set of 10 (jewelry, home decor, vehicle, musical instrument, tool, container, auto part, kitchen equipment, kitchen tool, garden tool) only use for the test set. This resulted in 3013 training instances and 1005 test instances. We verified there was minimal semantic overlap between this set and the hypernyms in the training set.
For the No overlap setting, we found train / test split where there is no overlap between either hyponyms or hypernyms. This was done by viewing the set of (hyponym, hypernym) pairs as a bipartite graph, where "no-overlap" split corresponds to partitioning of the nodes into two disjoint sets. We used the Kernighan-Lin algorithm as implemented in the networkx package (v. 4.2.3) for this purpose, resulting in train set of 2676 instances and test set of 419 instances.
C Additional results
C.1 In-domain results
Llama-3.2-3B The in-domain consistency and accuracy results for experiments on Hyper- nymy and SWORDS with Llama-3.2-3B are given in Table 7. Results for LAMBADA and TriviaQA for Llama-3.2-3B are in Table 8.
1lama-3.2-3B-Insttee Results for SWORDS with shown in Table 10.
9 and results for TriviaQA with Llama-3.2-3B-Innttuuu
18

---

Published as a conference paper at COLM 2025
Task
Method
p-all
p-pos
p-neg
ROC
A@100
MRR-P
MRR-N (1)
Base
78.5
55.7
44.1
95.9
84.7
9.7
0.9
SFT
88.6
54.5
61.4
98.0
51.4
74.6
12.2
Consistency FT
79.0
54.3
43.5
97.4
49.6
70.0
16.2
Hypernym
DPO-G2V
78.9
46.3
54.7
95.9
85.0
37.3
2.1
DPO-V2G
79.4
48.4
74.9
91.0
85.0
8.2
0.8
RankAlign
95.0
88.9
90.2
93.5
83.9
8,7
0.9
RankAlign-V2G
93.5
77.7
82.6
95.9
90.8
49.6
2.2
RankAlign-a
95.3
89.6
90.2
93.0
85.8
45.6
2.3
Base
53.8
27.0
32.5
81.6
76.2
23.6
3.8
SFT
53.2
21.7
34.3
80.4
75.7
33.8
4.2
Consistency FT
50.1
22.8
32.1
78.7
74.1
32.7
SWORDS
DPO-G2V
55.2
27.4
32.9
81.6
27.1
DPO-V2G
71.7
46.9
49.1
89.5
75.4
17.2
2.8
RankAlign
80.0
67.8
64.2
89.7
RankAlign-V2G
55.9
38.3
33.6
79.6
77.6
23.3
3.0
RankAlign-a
84.1
70.4
70.6
90.5
78.5
27.0
3.5
Table 7: Performance metrics across methods for Hypernymy and SWORDS tasks with the Llama-3.2-3B model.
Task
Method
p-pos
R@0
A@100
MRR-P
Base
9.3
84.6
99.8
80.5
SFT
22.6
100.0
99.7
85.1
Consistency FT
11.4
100.0
99.9
84.4
LAMBADA
DPO-V2G
39.3
100.0
99.2
79.2
RankAlign
54.4
78.8
99.8
78.5
RankAliig-V2G
8.9
93.9
57.3
RankAlign--t
37.2
59.4
99.2
85.1
Base
13.0
100.0
86.8
47.3
SFT
33.6
100.0
92.5
63.8
Consistency FT
37.8
100.0
92.2
64.0
TriviaQA
DPO-V2G
52.6
72.7
88.0
51.0
RankAlign
70.0
5.0
86.0
47.6
RankAlign-V2G.
53.9
100.0
64.3
20.2
RankAlign-at
70.8
13.3
85.5
40.0
Table Performance metrics across methods for LAMBADA and TriviaQA with the Llama-3.2-3B model
The in-domain consistency and accuracy results for experiments on Hyper- nymy and SWORDS with Llama-3.1-8B are given in Table 11. Results for LAMBADA and TriviaQA for Llama-3.1-88 are in Table 12.
Llama-3.1-8B
C.2 Comparison with DPO
The surface forms of our ranking-based loss functions bear some resemblance to DPO due to their ranking nature. Crucially, they do not incorporate the notion of reference model, as our goal is not to adjust the likelihoods of outputs relative to a reference, but instead bring the generator and validator into better absolute alignment.
Nevertheless, to compare directly with DPO, we explore whether adding comparison with reference model Pref would bring performance gains in out setting. Specifically, change the loss functions follows:
19

---

Published as a conference paper at COLM 2025
Task
Method
p-all
p-pos
p-neg
ROC
A@100
MRR-P
MRR-N (
Base
49.7
26.6
31.6
81.6
75.4
15.4
1.9
SFT
47.8
27.2
26.5
78.8
77.9
24.5
2.7
Consistency FT
38.1
23.0
19.2
72.8
76.3
22.2
2.7
SWORDS
DPO-G2V
52.3
29.5
31.4
82.9
78.6
21.9
2.1
DPO-V2G
50.1
28.3
22.1
87.6
63.9
3.1
0.2
RankAlign RankAlign-
65.3
49.3
43.5
88.7
75.1
11.7
1.5
57.0
37.9
38.2
82.8
49.1
0.0
0.0
RankAlign-at
72.1
57.5
49.6
90.1
76.8
24.1
3.0
Table
Performance metrics across methods for the SWORDS task with the -lama-3.2-3B-Insttte model.
Task
Method
P-pos
Recall
A@100
MRR-P
Base
19.3
64.3
92.2
54.6
SFT
34.6
100.0
94.1
58.2
Consistency FT
40.4
100.0
94.2
58.7
TriviaQA
DPO-V2G
37.4
61.8
79.4
21.7
RankAlign
65.4
0.0
91.3
52.8
RankAlign-V22
50.4
52.7
83.8
37.7
RankAlign--
37.1
0.0
78.7
20.3
Table 10: Performance metrics across methods for TriviaQA with Llama-3.2-3B-Im
Po xV,))
Po xy;)
Cc2v(Po) = - (xxxx)) logor (Blog
Blog
(6)
Pres xv.))
Prof Yes xvv)
Pa(Aw Avo
XG.)
Po(Ai xGG)
(P2)(Po) = - (x log or (Blog
B log
Pref (Aw XG..) Pref Gi
XG.
We report the results of RankAlign and RankAlign-V22 with reference term in the loss function in Tables 13 and 14. We note that the overall effect of using reference term in the loss function neutral to negative.
C.3 Cross-domain results
Cross-task results Additional results across models and methods when training SWORDS and evaluating on Hypernymy are shown in Table 15. Correlations for RankAlign are lower than in the in-domain setting (Tables 2, 7), but higher than all other baselines trained on SWORDS, including Base (Table 15). addition, RankAlign trained on SWORDS achieves a similar level of accuracy on Hypernymy as the in-domain model. Together, these results indicate that RankAlign
w generalizes well from the SWORDS to the Hypernymy tasks.
Results the generalization of RankAlign-a across the four tasks are in Table 16 and show a similar trend to the RankAlign results in Table
Generalization across classes We ablated the training sets for SWORDS and Hypernymy tasks to investigate whether training only the positive examples P improves correlation on the negatives N (Table 17). RankAlign generalizes well, substantially outperforming the Base model, although falling short of RankAlign when trained on PUN. in comparison, the non-ablated RankAlign obtained p-neg that were 15-18 points higher for Hypernym and points higher for SWORDS.
20

---

Published as a conference paper at COLM 2025
Task
Method
p-all
p-pos
p-neg
ROC
A@100
MRR-P
MRR-N (1)
Base
77.9
58.9
43.6
97.7
83.7
13.2
1.2
SFT
89.3
58.9
61.8
98.1
51.5
73.9
11.7
Consistency FT
79.8
51.7
47.1
97.9
58.9
69.9
12.9
Hypernym
DPO-G2V
81.1
53.2
56.9
97.7
81.7
56.9
3.3
DPO-V2G
76.1
45.2
76.6
82.8
82.9
12.7
1.3
RankAlign RankAlign-V2G
94.0
89.1
88.4
93.8
83.0
12.9
1.2
94.8
77.5
84.1
97.8
91.7
35.1
0.5
RankAlign-a
95.1
88.5
90.8
94.0
82.9
28.3
2.4
Base
71.2
44.5
48.1
89.7
77.6
29.9
3.7
SFT
71.5
40.9
46.1
90.7
74.6
40.1
4.7
Consistency FT
69.8
39,9
45.1
89.5
76.3
38.1
4.5
SWORDS
DPO-G2V
71.0
44.1
46.7
90.0
78.7
30.9
DPO-V2G
70.2
42.9
41.8
91.6
78.0
30.6
3.8
RankAlign
81.4
69.1
61.3
90.5
29.0
RankAlign-V2G
76.5
50.9
58.8
89.7
79.5
30.7
3.3
RankAlign-a
70.2
42.2
44.8
90.1
81.0
26.4
3.4
Table 11: Performance metrics across methods for Hypernymy and SWORDS tasks with the Llama-3.1-8B model.
Task
Method
p-pos
R@0
A@100
MRR-P
Base
0.6
79.6
99.9
82.5
SFT
100.0
86.7
Consistency FT
4.8
100.0
99.8
86.1
LAMBADA
DPO-V2G
62.7
99.3
99.9
82.3
RankAlign
65.2
74.9
92.1
61.6
BankAllgn-V2G
33.5
38.2
4.1
0.5
RankAlign-a
26.2
30.1
99.9
82.5
Base
32.2
100.0
92.8
62.7
SFT
36.9
100.0
95.4
69.9
Consistency FT
43.7
100.0
94.9
72.2
TriviaQA
DPO-V2G
77.6
92.5
62.7
RankAlign
65.2
0.0
92.1
61.6
RannkAlign-VGG
43.4
100.0
90.3
57.3
RankAlign--
63.6
2.5
92.6
62.7
Table 12: Performance metrics across methods for LAMBADA and TriviaQA with the Llama-3.1-BB model
Task
Method RankAlign +ref
p-all
p-pos
p-neg
ROC
A@100
MRR-P
MRR-N (1)
94.2
87.5
89.0
93.5
83.8
22.1
1.6
Hypernym
93.1
83.0
86.2
92.5
84.3
17.8
1.4
RankAlign
76.6
67.6
60.4
87.0
51.1
0.02
0.0
SWORDS
+ref
75.4
61.2
60.4
87.5
49.9
0.01
0.0
RankAlign +ref
60.0
60.0
94.8
52.9
LAMBADA
63.0
63.0
99.8
67.3
RankAlign +ref
56.8
56.8
44.5
9.6
TriviaQA
67.5
67.5
86.0
45.4
Table 13: Comparing RankAlign trained with and without reference terms in the loss function, for Gemma-2-2B.
21

---

Published as a conference paper at COLM 2025
Task
Method
p-all
p-pos
p-neg
ROC
A@100
MRR-P
MRR-N (1)
RankAlign
95.0
88.9
90.2
93.5
83.9
8.7
0.9
Hypernym
+ref
95.0
88.4
90.3
94.0
83.8
10.4
0.9
RankAlign
80.0
67.8
64.2
89.7
51.6
0.2
0.0
SWORDS
+ref
80.5
90.2
51.3
0.0
RankAlign
54.4
54.4
99.8
78.5
LAMBADA
+ref
60.8
60.8
99.6
77.9
RankAlign +ref
86.0
47.6
TriviaQA
67.4
85.5
33.2
Table 14: Comparing RankAlign trained with and without reference terms in the loss function, for Llama-332-3B.
Task
Model
p-all
p-pos
p-neg
ROC
A@100
MRR-P
MRR-N (
Base
76.4
54.8
45.1
97.0
83.7
19.4
1.6
75.5
53.9
97.2
83.7
20.7
1.7
Consistency FT
76.6
54.3
47.1
97.1
83.6
24.0
1.8
Gemma-2-2B
DPO G2V
76.7
55.5
45.5
97.0
84.3
21.0
1.7
DPO V2G
59.6
18.1
40.3
86.2
83.8
20.8
1.7
RankAllign
80.9
73.1
57.9
91.5
85.1
21.7
1.6
RankAlign-V2G
76.5
54.9
43.2
96.8
83.6
28.0
1.7
RankAlign-a
84.4
94.1
82.1
36.5
2.0
Base SFT
78.5
55.7
44.1
95.9
84.7
9.7
0.9
78.5
57.1
45.2
96.1
83.3
15.5
1.1
Consistency FT
78.5
96.1
83.6
15.5
Llama-3.2-3B
DPO G2V
79.1
56.3
45.3
96.0
84.8
9.6
0.9
DPO V2G
82.1
61.2
56.0
96.1
85.5
11.9
1.0
RankAlign
85.0
64.9
65.7
96.1
85.3
9.4
0,9
RankAlign-V2G
80.7
57.1
49.5
96.0
85.4
14.1
1.0
RankAlign-a
86.9
69.5
67.9
96.5
85.5
17.5
1.2
Table 15: Cross-domain results, SWORDS Llama-3.2-3B.
Hypernymy, for Gemma-2-2B and
Hypernymy Ap-pos
-> SWORDS
LAMBADA
TriviaQA Ap-pos
Ap-all
Ap-neg
Ap-all
Ap-pos
Ap-neg
Ap-pos
Hypemnymy
Gemma-2
12.9
27.3
38.0
24.7
16.6
-12.9
13.7
Llama-3.2
16.8
33.9
46.1
13.6
15.2
13.7
-44
25.8
Gemma-2
8.1
19.0
21.3
22.6
36.4
30.6
2.9
26.0
SWORDS
Tlama-3.2
8.4
13.8
23.8
30.3
43.4
38.1
-6.0
33.3
Gemma-2
-7.5
-4.2
-9.5
-44.9
-24.4
-42.9
51.1
-21.2
LAMBADA
Llama-3.2
7.3
-455
27.8
Gemma-2
6.5
5.2
5.1
-3.5
-5.7
53.7
TriviaQA
Llama-3.2
2.2
4.3
9.8
16.8
24.7
17.0
-13.8
57.8
Table 16: Cross-dataset evaluation showing the difference in p scores (between the RankAlign-a trained model and the base LM) when training on dataset (row) and evaluat- ing target datasets (column).. Scores greater than 0 indicate that RankAlign-t generalizes across tasks. Values in gray are the in-domain results.
C.4 G-V Consistency results
We evaluate G-V Consistency according to Li et al. (2024b). Specifically, we first prompt the model with generator prompts XG and set the corresponding y'A as the generator output. Yes]J, where 1/v PIM(- T(x;,
Then calculate G-V Consistency as [[[1[yv We compare model performance terms of Log-odds Correlation and G-V Consistency between the base model and Consistency FT-ed models.
22

---

Published as a conference paper at COLM 2025
Task
Model
p-pos
p-neg
Gemma-2 (Base) RankAlign
54.8
45.1
82.0
71.4
Hypernym
Llama-3.2 (Base)
55.7
44.1
+ RankAlign
88.0
72.4
Gemma-2 (Base)
32.8
36.5
RankAlign
71.5
53.0
SWORDS
Llama-3.2 (Base)
27.0
32.5
RankAlign
67.1
56.9
Table 17: Training RankAlign only on positive examples generalizes to closing the gap on negative examples N.
Model
Task
Method
p-all
GV-consistency
Base
76.4
Hypernymy
Consistency FT
73.0
81.2
Base
58.4
41.5
SWORDS
Gemma-2-2B
Consistency FT
55.0
93.4
Base
6.1
90.5
LAMBADA
Consistency FT
17.1
100
Base
19.4
80.9
TriviaQA
Consistency FT
20.1
99.9
Base
78.5
5.0
Hypernymy
Consistency FT
79.0
50.0
Base
53.8
16.3
SWORDS
Llama-3.2-3B
Consistency FT
50.1
56.7
Base
9.3
78.6
LAMBADA
Consistency FT
11.4
100
Base
13.0
3.3
TriviaQA
Consistency FT
37.8
Table 18: Comparing results on GV-consistency of base models and Consistency FI-ed models.
Results in Table 18 demonstrate that while Consistency FT does improve G-V Consistency, the gains in Log-odds Correlation are minimal or even negative. This suggests that G-V Consistency offers limited view of consistency across the entire dataset and space of possible answers, while Log-odds Correlation provides a broader understanding of the G-V Gap.
D Prompt Variations
Prompts variants used in the experiments shown in Table are given Table 19.
E Prompt templates for experiments
We present prompt templates for each task in Table 20, 21, 22, and 24 respectively. Optional exemplars in validator prompts are shown to non-instruct-tuned models.
23

---

Complete the sentence: {noun1) are kind of Published as a conference paper at COLM 2025
Prompt (Generator)
Prompt (Validator)
p-all
p-pos
p-neg
Complete the sentence:
Do you think z are V^? Answer:
94.2
87.5
89.0
2 are kind of
2 are kind of
Is it the case that z are УΛ? Answer:
92.0
84.0
82.9
I love 2 and other
In your view, are 2 yA? Answer:
80.3
43.9
66.8
Do you remember what our teacher used to tell us? She'd that contrary to appearances, actually
Deep down your bones, you believe that 2 are Y^?
76.6
57.0
64.5
are
Table 19: Generator and validator prompt variants and their associated cross-prompl corre- lations for the Hypernymy task. Here RankAlign was trained on the generator-validator prompt templates G == "Cmplltte the sentence: z are kind of"" and (z,yAA) := "Do you think are y Answer:"
Task
Prompt
Hypernymy (generator)
Hypernymy (validator)
Do you think bees furniture? Answer: No
Do you think corgis are dogs? Answer: Yes
Do you think trucks are fruit? Answer: No
Do you think robins are birds? Answer: Yes (optional exemplars)
Do you think {noun1} } are {noun2}? Answer:
Table 20: Generator and validator prompt templates for Hypernym.
Task
Prompt
LAMBADA
What word is most likely to come next in the following text? Text: [text)
(generator)
LAMBADA
Is the word "anyway" the most likely word to next in the following text? Text: "She gently takes him by his shoulders, forcing him to face her, and she adjusts the angle of his tie the way she might straighten picture on the wall. "I'm sure I don't need to tell you how important this gala is."
(validator)
"You don't, t but you will"
Answer: Yes (optional exemplar)
Is the word " "[completion]" the most likely word come next the following text? Text: (text)
Answer:
Table 21: Generator and validator prompt templates for LAMBADA.
24

---

Published as a conference paper at COLM 2025
Task
Prompt
SWORDS
Notice the word "[ [[targe]"" used in the context: " "[context)" In this context, the word " (target)" is synonymous with
(generator)
SWORDS (validator)
Determine whether the word in context can be replaced by another word or expression without changing the meaning of the sentence.
Notice the word "artists" used in the context: "Many painters, sculptors, and other "artists" were inspired by Duchamp.", ". In this context, is "artists" synonymous with "character"? Answer: No
Notice the word "happen" used in the context: did, one of three things would "happen'. Most likely: she would be meat..." In this context, is "happen" synonymous with "transpire"? Answer: Yes (optional exemplars)
"I could free Tasha, Ifl
Notice the word "[target]" used in the context: "(context)" In this context, is "[target]" synonymous with "(replacement)"" Answer:
Table 22: Generator and validator prompt templates for SWORDS.
Task
Prompt
TriviaQA (generator)
Question:
(question)
Answer:
TriviaQA (validator)
the correct answer the question "What kind song is Brindisi?" given by "drinking song"? Answer Yes or No: (optional exemplar)
Is the correct answer to the question "[question)" given by "{ ((answer)"? Answer Yes No:
Table 23: Generator and validator prompt templates for TriviaQA.
Task
Prompt
TriviaQA (generator)
Question:
(question)
Answer:
TriviaQA (validator)
Is the correct answer to the question "What kind of song is Brindisi?" given by "drinking song"? Answer Yes No: Yes (optional exemplar)
the correct answer to the question " "(question)" given by ((answer)" "
Answer Yes or No:
Table 24: Generator and validator prompt templates for TriviaQA.
25


Reviews:
[{"review_id": "ArHpzkPlrq", "reviewer": "Reviewer_R1Jh", "summary": "This paper addresses the generator-validator gap (G-V gap) in large language models (LLMs), where models give inconsistent responses when prompted in different ways to answer the same question. The authors propose a new formulation of this gap based on the correlation between log-odds scores from a model's generator (when creating answers) and validator (when verifying answers) modes across the entire set of candidate answers.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a more stringent and comprehensive definition of the generator-validator gap, moving beyond the binary agreement considered in previous work to analyze correlation across the entire distribution of possible answers.\n2. RankAlign shows clear improvements over all baseline methods, with an average gain of 31.8% in generator-validator correlation across all tasks and models tested. The authors conduct extensive evaluations across different models (Gemma-2-2B, Llama-3.2-3B, Llama-3.2-3B-Instruct) and four diverse tasks, providing a comprehensive analysis of their method's effectiveness.\n3. RankAlign demonstrates impressive generalization to unseen lexical items, different prompt formats, and even entirely different tasks, indicating that the method captures a fundamental aspect of model behavior.\n4. The ranking-based training approach is well-motivated and explained, with ablation studies and comparisons to related methods like DPO that help clarify its unique contributions.", "reasons_to_reject": "1. While RankAlign significantly improves correlation, this sometimes comes at the cost of task accuracy. For example, on TriviaQA with Gemma-2, MRR-P drops from 52.8 to 9.6, and on SWORDS, Acc@100 decreases from 77.7 to 51.1. These trade-offs are acknowledged but not fully addressed.\n2. The method's effectiveness varies considerably across models. For instance, RankAlign causes significant performance drops on Gemma-2 for LAMBADA and TriviaQA, but much smaller effects on Llama models. This variability is not fully explained.", "questions_to_authors": "None", "extra_scores": {}}, {"review_id": "6bB7Q7nkO6", "reviewer": "Reviewer_Zso5", "summary": "This work explores the generator-validator gap – where a model’s verification of an answer and generation of an answer to a question -- do not agree and proposes a new way to both formulate and mitigate the gap. Instead of the binary case where the gap is defined as the validator not picking the generator’s answer, this work instead operates in the continuous space where the generator and validator scores should correlate over a set of candidate answers. To do this, this paper sets up a simple case where examples are sentences with single word completions for the generator (e.g. *“A poodle is a kind of _”*) and corresponding yes/no questions for the validator (e.g. *“Is a poodle a kind of _?”*). The gap is measured using the correlation between the generator and validator’s next token log-odds.\n\nIn addition, this work proposes a new way to reduce the gap through finetuning called RankAlign. This approach aligns the validator’s candidate rankings with candidates’ log-likelihoods from the generator. They also test the reverse case, RankAlign-V2G, where the generator is trained to align with the validator’s rankings. Overall, RankAlign shows a significant improvement in the correlation between generator and validator without impacting task performance. These improvements are demonstrated by training in-domain and transferring cross-domain.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The overall formulation of the generator-validator gap is well laid out. The idea of exploring correlations across large sets of examples/candidate answers instead of a binary score seems very reasonable for better model generalizability.\n- RankAlign is fairly straightforward and has clear impacts in reducing the gap without compromising performance. This seems to hold across different language models as well. (I'm curious if how these results would scale to a larger model >3B).\n- The evaluation tasks chosen are pretty well varied. They test different aspects of semantic understanding (THINGS, SWORDS, LAMBADA) and there's the knowledge dataset (TriviaQA). The difference in correlation across tasks also fits with what I would intuitively expect, such as TriviaQA+LAMBADA being the more challenging and therefore having the lower correlations.", "reasons_to_reject": "- The evaluation data is somewhat limited. The input prompts and target generations are short-form. It’s not clear how well RankAlign would generalize to models used as judges with longer, more open-ended input prompts. These freeform cases would be the likely use case for LLM-as-judge.", "questions_to_authors": "- Do you think RankAlign can support thinking/reasoning models? They're useful LLM-as-judges but their step-by-step outputs don't seem to fit into your current framework.", "extra_scores": {}}, {"review_id": "y69G3Gi2ut", "reviewer": "Reviewer_p48J", "summary": "The generator-validator gap is a new type of language model inconsistency that is particularly important now that LLMs are starting to be used for judges. This paper propses a new metric, the correlation between a generator and it's validator, in order to quantify this type of inconsistency. The are able to ameliorate poorly aligned models by introducing a ranking-based loss that enforces alignment. While this approach results in degredation of accuracy and other measures of quality, they show that their method is able to close the gap (as quanitifed by their method but not previous methods).\n\nThe work is self is clear and the datasets and main experimental conditions are well chosen. The loss is novel and interested attempt to solve for problems introduced and initially attempted by [1]. However, the metric is conceptually quite confusing and generally not well validated which makes the work reads like: \"We have introduced our own metric which we have improved on\" this is concerning especially given the results in C.4 which seems to indicate that this metric is negatively correlated with previous metrics in the literature.\n\nAside from the problems in the metrics, this work has limited significance in the field of LLM consistency because the setting isn't connected to paraphrase inconsistency, inconsistency in sampling, or calibration literature on uncertainty quantification. It really isn't made clear why optimizing for G-V correlation as it is formulated in this paper is a good thing (since it introduces degenerate cases such as encouraging completion of low perplexity but correct words i.e. A poodle is a kind of thing or A poodle is a kind of mammal, which should not come before A poodle is a kind of dog.\n\n[1] Li, X. L., Shrivastava, V., Li, S., Hashimoto, T., & Liang, P. (2023). Benchmarking and improving generator-validator consistency of language models.", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The RankAlign objective is an interesting approach that works quite well for its stated purpose.  The experimental setup is generally well done (aside from the metrics validation). Supposing that we agreed the GV correlation metric was well validated and something the community wanted, then I would recommend this paper for acceptance.", "reasons_to_reject": "## (1) Metric Motivation and Formulation\n\nThis metric is conceptually flawed, using figure 1 we see naturally that \"Olivers are a kind of fruit\" and \"Do you think olives are fruits? No\" are inconsistent and we should not expect a discrepency. The second example \"Poodles are a kind of dog\" and \"Do you think poodles are mammals? Yes\", Is not inconsistent at all. There is no problem with that second example - the generator and the validator are already aligned in that case. I can understand the desire to formulate a metric unrelated to accuracy (lines 76 to 80) but this would apply for cases like: \"Olivers are a kind of dog\" and \"Do you think olives are dogs? Yes\" not for the poodle example. Further it doesn't seem like justification for why the poodle example should be included in the metric. This conceptual flaw likely explains why RankAlign results in degraded accuracy and isn't well validated (Appendix C.4). My recommendation for this is to reformulate the metric so it only captures the olive examples I provided but not the poodle examples. This isn't something in my view that can be resolved during the rebuttals but requires major revisions and resubmission.\n\n## (2) Metric and Method Validation\n\nSupposing the metric was fine conceptually, the metric isn't well validated. The only validation is in Table 15, where we see the metric does not align with previous approaches. The authors need to provide experiments showing that correlation corresponds to previous metrics of GV consistency.\n\nSince the results are all presented in the context of this metric and task accuracy alone, the paper reads as if the authors constructed a metric that they were able to improve instead of reading as a genuine improvement in generator-validator gap improvements. This is clear in Table 15, where the metric doesn't seem to align with how previous literature thought of GV-consistency. RankAlign itself is thus not validated, since alternative consistency measures are not provided. As above, I don't believe either of these can be addressed during a rebuttal period but I would be happy to provide further guidance and suggestion during the rebuttal period.\n\n## (3) Significance\n\nThe approach to sampling is not mentioned in the paper at all. I assume default configurations of deterministic sampling is used. \nSince this paper is about consistency, I would have expected for multiple seeds to have run with a random sampling algorithm such as nucleus sampling. I think this paper would be improved if this was added - It is hard to tell how the GV gap of each model varies with sampling.  Similarly, paraphrase variation is quite common in consistency literature yet is not provided - adding experiments over paraphrase varation would futher ehance the significance of this work to consistency research.  Without these experiments, metric isn't as interesting and useful to the community as it could be.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f0ugVeZ1D0", "reviewer": "Reviewer_JEXo", "summary": "RankAlign is a method of training a language model (LM) to increase the consistency of the model between generator and validator variations of the same task.  For example for a \"hypernymy\" task, a generator would be \"A horse is a type of ____\" (looking for mammal) while a validator woudl be \"Is a horse a type of mammal?\" (looking for \"Yes\").  Note that a high score for the first should imply a high score for the second, and to some degree vice versa.  The authors experiment with 4 such tasks, and experiment on how well their method improves the consistency (called the G-V) gap of several small/mid-sized LMs: Gemma-2-2B, Llama-3.2B, and Llama-3.2B-Instruct.  Their primary method adjusts the validator's log probabilities to be ranked in the sway way as the geneator's, and they also experiment with adjusting the other direction, and compare these to baselines such as Supervised Fine-tuning (SFT), Consistency Fine-tuning (Li et al., 2024), and Direct Preference Optimization (DPO).  Their technique often leads to the highest Pearsen correlation between generator and validator probabilities, while usually not losing much in terms of accuracy of the generator on the task.  They also try some generalization (cross-domain) experiments, for which their technique hold up in some cases (e.g., unrelated words, different forms of prompts) and not as well for others (training on one task only sometimes leads to improvement in correlation on a different task).\n\nOverall, the paper is an interesting experiment and an attempt to tackle one challenge with LMs.  The authors are reasonably successful on a small scale, and it may point the way to larger scale or scope techniques.\n\nThe limited scope is the main weakness of this paper.  The models used for the experiments are quite on the small end of LMs being used today.  At least from this paper, it is unclear if this G-V gap is really a problem with all models or just the smallest of them.  And the scope of tasks (4) is also quite limited.  This weakness wouldn't be an issue if the cross-task generalization was high, but since this is a weak point of their technique, it is hard to know how useful this kind of training will be for general use.  Overall, I do think it is an interesting step, but unless the scale and scope issues are tackled by future work, it will remain merely interesting.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- interesting topic/relevant problem\n- reasonable experiments\n- method seems to work for at least a limited scope", "reasons_to_reject": "- experiments are all small scale, hard to know if problems (or solution) will generalize to larger models\n- method shows limited ability to generalize beyond the scope of the training task", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper aims to measure and reduce the generator-validator gap in LLMs via a ranking-based fine-tuning method called RankAlign. The paper is clearly written with thorough experiments  showing significant gains at closing the gap. However, the majority of reviewers had concerns about generalization to larger models (>4B) and different tasks, as the results were not very conclusive on how much cross-task generalization exists. Furthermore, there were also concerns about the method causing performance drops on certain model/task pairs (e.g., TriviaQA and LAMBADA for Gemma-2). Overall, the paper is well executed and feels like a good first step towards measuring and closing the G-V gap, but the proposed method needs more empirical validation to argue convincingly that it doesn't harm performance across larger open-weight models.
Decision: Accept