Measuring the Inconsistency of Large Language Models in Ordinal Preference Formation (2024)

Abstract: Despite large language models’ (LLMs) recent advancements, their bias and hallucination issues persist, and their ability to offer consistent and preferential rankings remains underexplored. This study investigates the capacity of LLMs to provide consistent ordinal preferences, a crucial aspect in scenarios with dense decision space or lacking absolute answers. We introduce a formalization of consistency based on order theory, outlining criteria such as transitivity, asymmetry, reversibility, and independence from irrelevant alternatives. Our diagnostic experiments on selected state-of-the-art LLMs reveal their inability to meet these criteria, indicating a strong positional bias and poor transitivity, with preferences easily swayed by irrelevant alternatives. These findings highlight a significant inconsistency in LLM-generated preferential rankings, underscoring the need for further research to address these limitations.

heading: 1 Introduction
text: Expressing one’s preferences in an ordinal manner is a widespread and informative practice in human reasoning and communication (Arrow et al., 2010). By evaluating and comparing available options, individuals can make more informed decisions and communicate their values to others more effectively. In the domain of natural language processing (NLP), human preferential feedback serves as a valuable data type for aligning language models with human inclinations (Schulman et al., 2017; Rafailov et al., 2023).
Recent advances in large language models (LLMs) have prompted researchers to investigate the potential of LLMs in complex ranking-based tasks - such as recommendation (Li et al., 2023; Ren et al., 2024), web search (Sun et al., 2023), and text relevance comparison (Qin et al., 2023) - traditionally handled by task-specific models. Moreover, given that human annotation and evaluation
are resource-intensive, there is an increasing interest in augmenting or even substituting human preferential data with LLM-generated judgments to annotate, evaluate, or supplement as corpus (Wang et al., 2021; Zhao et al., 2022; Lee et al., 2023).
On the other hand, it is well-recognized that LLMs often exhibit severe bias and hallucination (Rawte et al., 2023; Zhang et al., 2023b). Specifically, prior research has identified undesirable behavioral patterns in LLMs when presented with multiple options (i.e., choices). For example, in Multiple-Choice Question Answering (MCQA), a task commonly used to benchmark LLM performance (Hendrycks et al., 2021a,b; Robinson et al., 2023), LLMs have shown a particular bias towards the position (Pezeshkpour and Hruschka, 2023) and the labeling of choices (Zheng et al., 2023).
Unlike MCQA, which requires single-selections, preferential ranking necessitates the ordinal preferences of all options, which is invaluable in scenarios lacking a definitive answer. Despite extensive research, the current literature on LLM bias has not fully addressed their behavior in preferential ranking tasks. To address this gap, our study en-
171
deavors to investigate a critical yet under-explored question: To what extent can LLMs consistently and coherently provide ordinal ‘preferences’?
This study makes an effort to measure the consistency (or more likely inconsistency) of LLMs in preferential ranking. Firstly, by incorporating order theory (Grätzer, 2002), we formalize the question and define five self-evident criteria that must be satisfied to achieve ‘consistency’ (§ 2.1). Through comprehensive diagnostic experiments on various state-of-the-art LLMs, we examine their adherence to preferential ranking criteria, namely transitivity, asymmetry, reversibility, and independence from irrelevant alternatives (IIA, as exemplified in Figure 1). We demonstrate that even the most advanced LLMs are incapable of providing consistent or coherent preferential rankings.
Specifically, we observe that: (1) The tested models generally fail to meet the asymmetry condition in preferential ranking (e.g., different answers for ‘compare A and B’ and ‘compare B and A’), indicating a strong positional bias (§ 3.2). (2) The preferences provided by the tested models exhibit poor transitivity; that is, concatenating binary preferences of choice pairs does not reliably yield an ordinal chain, and in fact, these preferences are often contested or even cyclic (§ 3.2). (3) The preferences of LLMs are significantly influenced by the addition or removal of irrelevant alternatives (§ 3.3). (4) When requested to provide rankings in different ordinal sequences (e.g., preferential descending and ascending), LLMs fail to produce logically equivalent outcomes (§ 3.4).
In summary, our contributions are threefold:
• We first formalize the measurement of consistency in LLM preferential ranking through the lens of order theory.
• We devise specific measurement metrics that align with the defined consistency conditions. A preliminary experiment not only corroborates some shared biases with the MCQA task but also highlights the unique challenges of preferential ranking.
• Through comprehensive experiments on a collection of state-of-the-art (SOTA) LLMs, we uncover a severe and widespread inconsistency in LLM preferential ranking. Our findings sound a serious alarm in related research and call for immediate mitigation efforts.

heading: 2 Experiment Setup and Preliminaries

heading: 2.1 Definition
text: Concretely, let A = {a, b, ..., n} be a finite set of n distinct alternatives, we define a preferential ranking as a strict partial ordering ≻ of A (Grätzer, 2002; Rosen, 2007). Such ordering satisfies that, for all a, b, c ∈ A:
• Irreflexivity: not a > a.
• Asymmetry: if a > b then not b > a.
• Transitivity: if a > b and b > c then a > c.
Besides above intrinsic conditions, In multiround preferential ranking scenarios, we also examine following criteria
• Independent from Irrelevant Alternative (IIA): if a > [...] > b in an ordering ≻original, given additional alternatives c, d, ..., then a > [...] > b in the new ordering ≻new.
• Reversibility: if a > [...] > b then b < [...] < a. This criterion can be regarded as a full ranking generalization of binary Asymmetry.
Note that there is also a non-strict partial ordering variation that allows a = b (i.e., preferential ties). For simplicity, all experiments are conducted under strict partial ordering scenario.

heading: 2.2 Datasets
text: Following prior research that benchmarks the reasoning capabilities of LLMs (Park et al., 2022; Liu et al., 2023; Zhang et al., 2023a; Google, 2023; Jiang et al., 2023), we choose the MMLU (Hendrycks et al., 2021a) as our principal testbed. This benchmark encompasses a total of 14,079 MCQA test cases across 57 varied subject areas. Given that our study’s main focus is on preference ranking rather than choice generation, the MMLU is particularly well-suited to our research interests, as the benchmark is uniformly formatted with multiple-choice options, and the options (i.e., choices) are predefined.
It should be noted that preferential ranking is a more challenging task than MCQA because it necessitates additional ordinal information. To create a balanced test set, we curate a collection by selecting the first 20 cases from each subject, resulting in a total of 1,140 cases. In line with the original MMLU framework (Hendrycks et al., 2021a), we employ a 5-shot example prompting strategy that leverages the dataset’s fixed development set.

heading: 2.3 Evaluated Models
text: To investigate the potential inconsistencies in LLM preferential ranking, we have compiled a selection of open-source models, including Llama-370B (AI@Meta, 2024) and Qwen-1.5-72/110B (Qwen, 2024). Our selection criteria prioritized models with relatively large parameters (exceeding 70B), as smaller models are generally outperformed by their larger counterparts in text-based task performance. For proprietary models, we have included gpt-3.5 (Brown et al., 2020) and gpt-4o (OpenAI, 2023), which are among the most widely utilized closed-source models in recent times. Specifically, we adopt the snapshot models gpt-3.5-turbo-0125 and gpt-4o-2024-05-13. Sources of tested open-source models are summarized in Table 6. Detailed specifications and sources for the selected models are provided in Appendix A. To ensure reproducibility, we have set the temperature for all experiments to zero (the temperature setting ranges from 0 to 2 for OpenAI models, and from 0 to 1 for others).

heading: 2.4 Preliminary Examinations
text: Prior to initiating the principal experiments, it is beneficial to ascertain whether the label tokens of choices affect LLMs’s preferences and whether LLMs exhibit differential performance across single-select and preference ranking tasks. To this end, we conduct two preliminary examinations.
Alternative Label Bias Following (Zheng et al., 2023), in comparison with the original Alphabetic label tokens ([A, B, C, D]) of MMLU, we add Arabic: [(1), (2), (3), (4)], and Roman: [I, II, III, IV] token sets. The parentheses in Arabic token to reduce ambiguity for numerical questions. Fewshot examples are modified in accordance with the altered labels.
As shown in Table 1, the first-preference accuracies vary slightly for tested models. We also evaluate similarity of rankings based on minimal editing distance, and the normalized (between 0 and 1) similarities are near 0.9, suggesting a minor influence of label tokens in preferential ranking.
Question Format Sensitivity Given that the firstpreference in a ranking context is logically congruent with a single-select choice, we juxtapose the accuracies of MCQA across these varying question formats.
Question Format Single Select
Ordinal Ranking
Cardinal Ranking
HitRate@N - @1 @2 @3 @1 @2 @3
llama-3-70b 77.3 72.1( -5.2) 85.6 93.1 73.9( -3.4) 87.4 94.2 qwen1.5-72b 78.0 72.2( -5.8) 85.5 91.6 70.4( -7.6) 83.7 91.4 qwen1.5-110b 76.5 71.7( -4.8) 85.9 93.2 71.8( -4.7) 86.0 93.2 gpt-3.5-trubo 67.3 62.1( -5.2) 81.0 91.6 61.1( -6.2) 80.7 91.4 gpt-4o 78.1 83.7( +5.6) 92.2 96.9 82.7( +4.6) 92.4 96.8
Table 2: Full results of Question Format Sensitivity test. @N denotes that the accuracies are calculated based on the first N elements of the preferential ranking lists.
Ordinal Ranking necessitates that outputs be sequenced lists, whereas Cardinal Ranking obliges models to assign a numerical score to each potential answer. As evidenced in Table 2, all models, with the exception of gpt-4o, demonstrate reduced accuracies relative to the single-select format. This leads us to infer that LLMs are indeed sensitive to the format of questions, thereby underscoring the importance of probing into LLMs’ performance in preference-based ranking.

heading: 3 Main Experiments and Key Observations

heading: 3.1 Irreflexivity
text: We elect to forgo further scrutiny of this criterion, as a pilot test that we performed indicates that carefully crafted prompt instructions effectively preclude the recurrence of options within modelgenerated rankings, with infrequent transgressions observed (less than 1% across all evaluated models).

heading: 3.2 Asymmetry and Transitivity
text: Symmetry checking in LLM reasoning is fundamentally a test for positional bias. Given that prior research has identified positional bias in singleselection tasks (Pezeshkpour and Hruschka, 2023; Zheng et al., 2023), it is imperative to ascertain whether LLMs exhibit a similar propensity to modify their preference rankings when options are se-
quenced differently in questions. Consider A as a list of n options: [a1, a2, a3, a4]. By soliciting the LLM to perform binary comparisons n× (n− 1) times, we can construct an n×n binary comparison matrix M . As depicted in Figure 2, we assign mij = 1 if the model shows a preference for ai when presented with the ordered pair [ai, aj ] (noting that [aj , ai] constitutes a distinct ordered pair), and mij = −1 if the model opts for aj when faced with the same ordered pair.
Next, we calculate an asymmetry score by comparing the agreement between mij and mji:
2 ∑n
i,j=0,i>j sij n(n− 1) , sij = { 0 if mij ≡ mji 1 if mij ̸≡ mji (1)
The average asymmetry scores, as delineated in Table 3, reveal a low degree of overall asymmetry among all models, indicative of significant positional biases in the preferential ranking task. Notably, gpt-4o, recognized as the SOTA proprietary LLM to date (Li et al., 2024), registers the lowest asymmetry score. Fundamentally, the positions of options in binary comparisons markedly affects the preferences of LLMs, culminating in a decrease in duly asymmetry. This finding also concurs prior observations that LLMs show a position bias in MCQA task (Robinson et al., 2023).
Upon identifying inconsistencies in asymmetry, we recognize that the upper and lower triangles of a binary comparison matrix do not perfectly correspond. Consequently, it is imperative to calculate transitivity separately for each triangular matrix.
Considering options as nodes and relations (‘<’ and ‘>’) as directions within a graph, we reconceptualize the problem as one of directed reachability.
For each relation, a relation matrix R can be derived from the triangular matrices, as depicted in Figure 2. Subsequently, we can compute the transitive closure matrix (Purdom Jr, 1970; Karp, 1990):
Mt = [rij ]n×n = R0 ∨R1 ∨ ... ∨Rn−1 (2)
where ∨ is logical and operation. If rij = 1 in Mt, then the relation has successfully transitioned; otherwise, it is deemed non-transitive.
As evidenced in Table 3, all models exhibit moderate transitivity, with the random baseline established at 59.4. Furthermore, there are subtle differences between the upper and lower triangles in all models. This observation is consistent with the positional biases identified in the asymmetry experiment. In contrast, the impact of relation symbols on transitivity is considerably less pronounced.

heading: 3.3 Independent from Irrelevant Alternative
text: IIA criterion assesses whether the introduction of an additional option affects the relative order of the original preference ranking. This condition is tested by calculating a normalized similarity score, Sim = 1−MED/2n, where MED represents the minimum edit distance between (1) preference rankings with three options and (2) preference rankings with four options, excluding the omitted choice from the three option rankings.
As suggested by Table 4, the removal of gold answers significantly alters the LLMs’ preferences for
the remaining options. Conversely, the elimination of non-gold options results in less pronounced, yet still noticeable, impacts on the preference rankings.

heading: 3.4 Reversibility
text: In all preceding experiments, the models were instructed to provide preferences in a descending order, placing the most favored option first. Maintaining all other conditions constant, we now instruct the models to rank in an ascending order, positioning the least favored option at the forefront.
Table 5 encapsulates the first-N option match rates and the overall ranking similarities between the original rankings and the reversed sequences under the alternative output order. All models exhibit suboptimal performance on full match length (with <45% match rate), while gpt-4o outperforms other models by significant margins.

heading: 4 Conclusion and Future Work
text: To conclude, we have formalized the consistency measurements in preferential ranking tasks by designing corresponding criteria and metrics. Through diagnostic experiments, we have evaluated some of the most advanced LLMs, uncovering severe inconsistencies and positional biases that are prevalent across all models, among other observations. Our study raises general awareness of discrepancies in LLMs and signals a call for future research efforts. Specifically, we highlight two areas of interest: the development of a non-MCQA benchmark for consistency measurement and the creation of mitigation methods to enhance the consistency of LLMs in ranking-based tasks.
Limitations
While the experiments on MMLU yield notable and insightful observations, we acknowledge that MCQA is not fully aligned with preferential ranking. Most QA benchmarks have predetermined ‘correct’ answers; however, preferential ranking
can also be relevant in scenarios where there is no absolute right or wrong. Therefore, an additional avenue for future work could involve constructing a benchmark that measures preference representativeness rather than one based on true-or-false judgments.

heading: Acknowledgements
text: Our colleague Jack provided the deployment and inference services of the tested open-source models, thanks to him.

heading: A Reproducibility
text: The two proprietary models we evaluated, gpt-3.5 and gpt-4o, are commercially available closesource models. The source for open-source models are shown in Table 6.

heading: Future Work
text: Our study raises general awareness of discrepancies in LLMs and signals a call for future research efforts. Specifically, we highlight two areas of interest: the development of a non-MCQA benchmark for consistency measurement and the creation of mitigation methods to enhance the consistency of LLMs in ranking-based tasks. Limitations
While the experiments on MMLU yield notable and insightful observations, we acknowledge that MCQA is not fully aligned with preferential ranking. Most QA benchmarks have predetermined ‘correct’ answers; however, preferential ranking
can also be relevant in scenarios where there is no absolute right or wrong. Therefore, an additional avenue for future work could involve constructing a benchmark that measures preference representativeness rather than one based on true-or-false judgments