Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Authors: Yinhong Liu
Zhijiang Guo
Tianya Liang
Ehsan Shareghi
Ivan Vulić
Nigel Collier
Venue: ICML 2025 spotlightposter (2025)
Keywords: LLMs
logical consistency
order consistency
transitivity
Primary area: deep_learning->large_language_models
TL;DR: We quantify, evaluate and improve the logical preference consistency of LLMs' judgements

Abstract: Large Language Models (LLMs) are expected to be predictable and trustworthy to support reliable decision-making systems. Yet current LLMs often show inconsistencies in their judgments. In this work, we examine \textit{logical preference consistency} as a foundational requirement for building more dependable LLM systems, ensuring stable and coherent decision-making while minimizing erratic or contradictory outputs.
To quantify the logical preference consistency, we propose a universal evaluation framework based on three fundamental properties: *transitivity*, *commutativity* and *negation invariance*.
Through extensive experimentation across diverse LLMs, we demonstrate that these properties serve as strong indicators of judgment robustness.
Furthermore, we introduce a data refinement and augmentation technique, REPAIR, that enhances logical consistency while maintaining alignment with human preferences. Finally, we show that improving consistency leads to better performance in LLM-driven logic-based algorithms, reinforcing stability and coherence in decision-making systems.

Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Yinhong Liu
Zhijiang Guo
Tianya Liang'
Ehsan Shareghi 2
Ivan Vulić Nigel Collier
Abstract
These limitations hinder the full deployment of LLMs, par- ticularly in professional and high-stakes applications."
Large Language Models (LLMS) are expected to be predictable and trustworthy to support reliable decision-making systems. Yet current L.Mss of- ten show inconsistencies in their judgments. In this work, we examine logical preference consis- tency as a foundational requirement for building more dependable LLM systems, ensuring stable and coherent decision-making while minimizing erratic or contradictory outputs. To quantify the logical preference consistency, we propose uni- versal evaluation framework based on three funda- mental properties: transitivity, commulativity and negation invariance. Through extensive experi- mentation across diverse LLMs, we demonstrate that these properties serve as strong indicators of judgment robustness. Furthermore, we introduce data refinement and augmentation technique, REPAIR, that enhances logical consistency while maintaining alignment with human preferences. Finally, we show that improving consistency leads to better performance in LLM-driven logic-based algorithmss. reinforcing stability and coherence in decision-makinn making systems. Code is available https//gttub..com/willlam////
The foundation of a reliable and trustworthy system is the consistency it predictions. A consistent system produces explainable and tractable decisions, enhancing its depend- ability and reliability. In this work, we focus on key form of consistency in LLMs: logical preference consistency, which is critical for applications requiring structured rea- soning and coherent decision -maiing (Creswell et al., 2023; Hamon et al., 2020). Logical inconsistencies can lead to unreliable conclusions (Restall, 2002) and even paradoxes (Hyde, 201 1), posing significant risks in domains that de- mand rigorous logical judgment, such as temporal or spatial reasoning (Mostafazadeh et al., 2016b), optimization (Guo et al., 2024), and automated decision systems.
This paper examines three key aspects of logical preference consistency LLMs: transitivity, commutativity, and negation invariance. We propose a universal framework for quantifying these consistency properties, applicable to any number of items, and adaptable across various domains. Our evaluations reveal strong correlation between con- sistency and L.LMs' judgment robustness, suggesting that logical preference consistency serves as a useful proxy for preference reliability. To enhance the logical preference consistency of LLMs, we propose REPAIR, a framework that refines noisy pairwise comparisons using rank aggre- gation and extrapolates additional comparisons logically, Models trained with this approach achieve better internal consistency without sacrificing alignment with human pref- erences. Moreover, in logic-dependent tasks, these models outperform less consistent ones, demonstrating improved efficiency in sorting- ranking algorithms (Liu et al., 2024) reliant on logical coherence.
1. Introduction
Recent research in Large Language Models (LLMs; Brown et al. 2020; OpenAl 2023; Anil et al. 2023a;b) has achieved substantial progress on various tasks, enabling them to gen- erate responses that better support their mecision-making and problem-solving (Dai et al., 2024). However, key chal- lenges still exist regarding the reliability and trustworthiness of L.L.Ms. Issues such as hallucination (Zhang al., 2023), bias (Gallegos al., 2024), and inconsistencies in reasoning (Huang & Chang, 2023) continue to affect their credibility.
In sum, our contributions are as follows: 1) We highlight the importance of logical preference consisenncy-alongsidee human alignment-in aligning LLM preferences. We define mathematical formulations for quantifying/measuring three key consistency properties: transitivity, commutativity and
University of Cambridge Moonss University. Correspon- dence to: Yinhong Liu eyll35@cam.ac....
'For instance, in the fields of behavioral economies and psy- chology systems are traditionally evaluated on two key dimensions: validity and reliability (Schmidt et al., 2000: Guion, 2004; Miller et al., 2023).
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
An input and several response candidates are provided compare
pairwise preference question set designed to check the logical consistency of the output judgments
Collect human pairwise preference annotations
Human judgements are usually noisy self-conflicting
rained a this week. We canceled the foorball game...
Transitivity inconsistency
F [A B)
+
B
F(A, B): [A] better than [B]?
B
[B, Cs: (B) better than [C]?
(B, C)
This is
week,
Q) better (C]
F(A. C)
Commutativity Inconsistency
(BB A): is (B) better than [A]?
F(B A)
[A, B) Is [A] worse than [B]?
F(A, B)
Negation Inconsistency
The weather...
We decided..
F TA. B)
B
Rank Estimation
FIB, C)
C
F [B. A)
LLM as Logical Operator
Depec-Den Algorithm
F(A B)
the Learn-to-Rank methods to estimate rankings
Refined conflict-free pairwise data
Align LLM preference with refined data to improve logical consistency
Logically consistent LLM serves as part of downstream algorithms
Figure I: Three types of logical inconsistencies are observed in real-world pairwise annotations (top row): Transitivity, Commutativity, and Negation Invariance. By refining the data for self-consistency using rank estimation, we can train LLMs with enhanced logical consistency, improving their performance in logic-dependent algorithms (bottom row).
negation invariance. 2) We conduct extensive experiments to evaluate logical preference consistency across state-of- the-art LLMs and analyze its correlation with model relia- bility. 3) We propose data refinement and augmentation method for instruction-tume ttuning that enhances logical prefer- ence consistency while maintaining alignment with human preferences and 4) we demonstrate that improving logical consistency enhances LLM performance in logic-dependent algorithms where L.LMs serve as logical operators. We re- lease our source code and the refined dataset ANONYMOUS.
consistency evaluation process by treating an LLM as an operator function that compares pairs of items and outputs decision on the relation between the items. Let X {ii .,, 2N represent set of items, and define relation (e.g., comparison) function F X x X + R, which compares two items, such as (Ii, and assigns relational decision F(z,, the directional relation between I; and øy. For simplicity, we consider R to be binary relation set, R (rig, where Fij represents preferential relation æ; > Iy (i.e., item x; is preferred over item Iz), and Tja indicates the reverse preference zj Tz.
r, where E R denotes
2. Measuring Logical Consistency
In evaluating logical consistency, we focus on whether the function F adheres to the following key properties over the item set X: 2) transitivity, 2) commutativity, and 3) negation invariance, as demonstrated in Figure 1. Transitivity ensures that the LLM's s predictions and judgements are internally co- herent and do not suffer from logical contradictions within given context. Commutativity tests whether the model's decisions are invariant to the order in which items are com- pared. The negation invariance checks whether the model maintains consistent understanding when dealing with re- lational negations. By systematically applying these tests, we are able assess the extent which the model's judg- ments conform to logically consistent behavior, providing
We evaluate the logical preference consistency of LLMs by assessing their ability to predict logically consistent relations among a set of items. These items could represent diverse entities or events with a uniform relation between them: such relation might be (i) comparing the preference among response candidates
query or (ii) determining the causal order of shuffled events, among other possibilities. This evaluation is grounded in relational algebra and order theory (A (Abiteboul et al., 1995; Imieliáski & Lipski Jr, 1984), with the goal of assessing whether the model maintains coherent and self-consistent judgments across its predictions.
To formalize this concept, we define the logical preference
2

---

Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
X1
X2
X1
X2
K3
X4
X5
Commutativity Conflict
A
A
B
F F(x, (a) + f14 (X X.) F41
X2
B
B
X
X3
X4
X3
X4
X
(a)
(b)
K4
A
B
A
X2
X3
X
Xs
A
B
A
X1
B
B
A
F [X2 x))
X2
Negation Conflict
X1
k3
A
B
X1
X3
F (Mg X2) + 22
X4
8.
A
B
F 0ks. X2) 25
Xs
B
B
X5 K
X4
X5
F [x] xp
(c)
(d)
Figure 2: Example of relation graphs illustrating transitivity, where items represented nodes, and directed edges indicate pairwise preferential relations. Red dashed cycles in the graph highlight violations of transitivity. The cycle in (d), spanning 4 items, cannot be captured by Stren (3). The Stron metric can be applied to partial relation graphs, as shown in (c) and (d).
Figure 3: Examples illustrating violations of commutativity and negation invariance. Each entry of the two preference matrices represents predicted judgments of 2, > Ty and Vi < Tyj labelled with A and B respectively. The top-left matrix is based the original relation, while the bottom- right matrix reflects the negated relation. Linked red cycles highlight non-commutative pairs, and linked dashed purple eycles indicate negation inconsistencies.
quantitative proxy measure of its decision reliability.
bitrary number of items. LLMs often struggle to maintain perfect/full transitivity, especially as the number of items in- creases. The proposed metric Stran (K)) captures the degree of transitivity across subsets of K sampled items, where 3 I K < |X]. The metric is defined as:
2.1. Measuring Transitivity
Grounded in our problem setup and definitions above, tran- sitivity implies that if a model predicts A > B and B > C, it must also predict A > C. Ensuring transitive predictions essential for coherent global understanding of item rela- tionships, preventing contradictions that could undermine reliability in feeisionnmaking or ranking tasks (Liu et al., 2024: Li et al., 2019; Qin et al., 2024).
M
1
Stran (K)
acyelic (S.
(I)
M
Here, S.K represents randomly sampled sub-graph of size K, and the indicator function 1 accylisc returns 1 if the sub- graph SKK contains no cycles (i.e., is transitive), and 0 otherwise. M denotes the total number of the sampled sub- graphs. As the size of the item set increases, the number of possible sub-graph combinations grows exponentially. To manage this complexity, 1,000 sub-graphs estimate transitivity for larger sets. The choice of M is supported by theoretical evidence explained in Appendix ŞI. Therefore, the metric ranges from 0 to 1, where represents nearly perfect transitivity.
We define that the function F is fully transitive if it does not predict any intransitive relation within the set X. This means that if F xz), = Tiy and F(I,, = Tyk. then #xx;, = Tik must hold for all i, E X. This can be visualized in Figure 2, where we represent the pairwise relations by F as a relation graph. If F is transitive over X, the corresponding relation graph should be a Directed Acyclic Graph (DAG). Coossequently, to determine if F is fully transitive over the item set X, we only have to verify whether the predicted relation graph contains any cycle. We show how to construct the relation graph from the judge- ments of the LLM operator function F, and the algorithm to check whether a graph contains cycles in Appendix SA.
cap the number of samples to
Maintaining transitivity becomes increasingly difficult as the size of the subset grows. For any set of K items, there are 2 possible combinations of pairwise relationships, but only K! of these can form transitive ranking. As result, the degree of transitivity tends to decrease with larger sub- graph sizes. This means that for the same LLM,, Stran ( (K) typically decreases as K increases, reflecting that preserving consistent rankings over larger sets of items is increasingly
We introduce Stran ( K) to quantify transitivity over ar-
"A DAG implies that no cyeles exist in the relation graph. We assume the comparison F is irreflexive, explained Appendix &J
3

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
challenging. Since Stran (K) measures transitivity for a fixed subset size, it allows for fair transitivity comparisons between item sets of different sizes.
visualize this comparison in Figure 3. The normalization factor averages across all pairwise permutations in set X, ensuring that 8neg ranges from 0 to 1. The maximum score of 1 indicates perfect negation invariance, where the model consistently handles negated relations.
2.2. Measuring Commutativity
Commutativity refers to the logical property that ensures the model's judgments remain consistent when the order of comparison between two items is reversed. Prior studies have shown that LLMs are susceptible to permutation bias, also referred to as positional bias (Wang et al., 2024b; Liusie et al., 2024a). To measure the degree of commutativity, we propose a metric Scomm. which evaluates whether the model's judgment changes when the order of the items is swapped in the prompt. Specifically, it is defined as follows:
3. Evaluating Logical Consistency of LLMs
After defining the measures, we proceed to evaluate LLMs' judgments from the consistency angle on three representa- tive tasks, each reflecting different levels of subjectivity.
3.1. Evaluation Setup
Tasks and Datasets. We employ three representative tasks to evaluate LLMs' logical consistency. The first task, ab- stractive summarization evaluation, uses the SummEval dataset (Fabbri et al., 2021) and focuses on the model's abil- ity to make preference judgments between summaries, par- ticularly regarding the coherence aspect. The second task, document reranking, leverages the NovelEval dataset (Sun et al., 2023), where LLMs assess the relevance of retrieved documents in response to queries. The final task, tempo- ral event ordering, uses the CaTeRS dataset (Mostafazadeh et al., 2016b) and tests the model's capability to judge tem- poral and causal relationships between events, critical for maintaining consistency narrative understanding. Further task and dataset details are available in Appendix $B.
2 Scamm XI(IX 1) Σ
(F((s, =F 2:))-
Here, F(z,, 2y) represents the model's judgment when com- paring items 2; and zz. The indicator function 1 returns if the model's judgment remains consistent when the order of the items is reversed, i.e., F(z,, 23) )- = F(æy,.;)) and O otherwise. We visualize this comparison in Figure 3. The normalization term ensures that Scomm is averaged across all pairwise combinations of the items in set X. As a result, the metric ranges from 0 to 1, with 1 indicating perfect com- mutativity, meaning that the model is completely robust to the order of item comparisons.
Metries and Reliability Measurement. In Appendix SG, we show metric computation details and the prompt tem- plates used. We compute the logical consistency metrics at the instance level and report averages across the test sets for each task. In addition, we report the human agreement rate (abbreviated as H.) by calculating the pairwise judge- ment accuracy between judgements made by LLMs and the provided human annotations. It serves for how closely the model's judgements align with human judgements. For the measurement of LLMs' reliability, we perform Monte Carlo Sampling of Chain-of- Thought (Wei et al., 2022) reasoning outputs, using a temperature of 0.7. Self-agreement defined as the percentage of outputs that agree with the majority judgment across multiple samples. This measurement ranges from 0.5 to 1, with higher values indicating greater stability in the model's judgments.
2.3. Measuring Negation Invariance
Negation invariance tests whether the model maintains con- sistency when confronted with the negation or inversion of a relational statement. Inconsistencies indicate a failure to correctly understand and apply the complementary nature of relations. Previous work suggested that LLMMs struggle to automatically infer appropriate inverse relationships when acquiring knowledge (Allen-Zhu& & Li, 2023; Berglund et al., 2024). To quantify negation invariance, we propose the met- ric Sneg. which examines if the model can correctly reverse its judgement when prompted with a negated relationship between items. The metric is defined as below:
reference
Σ UELJEIII (F
I(F(,,)))) = -F(1,
8neg IX((XI-1)
3.2. Results and Analysis
(3) In this formulation, -F æι) represents the negation of the original relation (c.g., reversing a preference or rela- tional direction). F(r,, refers to the model's judgment when explicitly prompted with the negated relation. The indicator function returns if the model's response to the negated relation matches the expected negated judgment (i.e., F(.,, xy) = -F(,,, ())) and 0 otherwise. We also
Performance of Different (Famillee of) Models. As shown in Table 1, recent LLMs like Gemma2-9B and Phi-3- medium demonstrate stronger overall consistency compared to earlier models. In particular, models such as Deepseek- chat-v2, Phi-3-medium, and Gemma-2-9B perform well across all three evaluated consistency dimensions. However, is important to note that strong performance in one aspect
4

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Table I: Logical consistency evaluation results. We report human accuracy (H.,, transitivity over 5 items (Stvan (5)).
commutativity csommm) and negation invariance (Sneg), all measured in accuracy.
SummEval (Coh)
NvvelEval
CaTeRS
Models
H.
Stran (5)
8comm
Snng
H
Stran
Scomn
8neg
H.
Stran (5)
Scome Sneg
Direct Judgements
Llama-2-7B
57.5
88.3
57.5
66.2
57.5
68.1
57.5
78.1
61.9
88.4
56.0
49.7
Llama-2-131
58.3
86.6
59.3
84.0
58.3
88.2
62.9
76.9
65.6
95.3
70.8
54.5
Llama-3-8B
67.8
91.0
76.1
48.9
60.6
73.0
73.3
79.1
73.1
88.2
79.9
63.3
Mistral-7B
63.6
95.1
59.9
51.2
60.1
90.5
68.0
82.1
70.2
95.9
73.9
76.9
Zephyr-7B-beta
61.3
87.8
52.8
74.1
60.5
91.5
63.8
86.5
70.6
93.5
77.8
80.8
Phi-3-mini
65.6
92.8
66.9
75.1
59.7
92.5
55.5
39.4
60.2
97.2
85.5
73.3
Phi-3-medium
68.8
96.2
71.0
78.1
62.7
93.7
76.3
77.6
67.2
93.4
87.9
84.1
Gemma-2-9B
73.8
94.8
78.1
78.2
63.6
89.9
85.3
88.2
72.4
96.0
86.5
66.9
GPT-3.5-0125
66.3
82.5
67.5
65.8
61.2
89.8
71.6
83.4
69.3
86.2
66.7
81.7
Deepseck-chat-v2
72.7
93.1
72.8
84.9
65.7
95.1
86.7
89.2
73.5
93.8
91.9
91.1
CoT Prompting
Llama-2-7B
55.0
42.6
45.7
68.9
56.5
50.8
66.7
60.1
59.5
40.5
56.7
56.1
Llama-2-15
66.6
55.8
63.2
40.3
59.2
60.8
71.0
56.4
60.6
61.1
63.5
74.3
Llama-3-8B
67.2
65.0
64.8
61.2
60.2
69.3
76.8
72.4
67.9
62.2
46.1
78.6
Mistral-7B
61.6
64.7
65.3
58.0
58.5
64.9
73.2
77.7
60.3
66.1
69.0
79.1
Zephyr-7B-beta
58.4
46.4
51.7
74.0
60.0
64.5
70.6
71.2
65.7
69.3
78.4
77.3
Phi-3-mini
65.1
67.9
61.9
43.0
56.5
61.2
47.8
80.5
58.4
81.2
82.6
78.7
Phi-3-medium
69.1
82.3
72.5
58.9
63.1
89.0
87.3
79.3
54.5
81.4
87.4
85.9
Gemma-2-9B
73.5
74.0
71.9
81.3
62.1
89.9
85.9
82.3
63.5
83.8
87.1
69.9
GPT-3.5-0125
63.9
63.3
64.8
68.4
59.3
79.3
73.4
67.5
62.8
68.6
70.5
74.9
Deepseek-chat-v2
74.9
79.4
76.1
82.8
61.5
94.2
90.4
83.0
69.1
81.6
88.2
88.4
of consistency does not guarantee similar performance in others. For example, while Mistral-7B excels in transitivity, its performance other consistency aspects weaker. The Phi-3 family consistently shows high logical consistency, which we hypothesize is due to its reliance synthesized data which is cleaner and less self-contradictorry
3.3. Consistency and Reliability
Transitivity as a Proxy for Self-Agreement. As shown in Figure 4, there are strong correlations between transitivity and self-agreement across three datasets, regardless of the task's level of subjectiveness. Self-agreement indicates the model's internal consistency and robustness, model should not fluctuate significantly in its responses. Higher self-agreement suggests that the model exhibits stable understanding of the underlying relations in the input. Given that transitivity captures this self-consistency, we argue that transitivity serves as a useful proxy for evaluating the local robustness of L.MMs.
reliable
Another observation is that LLMs tend to perform more consistently on the CaTeRS dataset. We attribute this to the dataset's focus on temporal and causal relations. The more objective and logical preferential relations may make easier for models maintain consistency.
Impact of CoT Prompting. We also investigated the effeect of Chain-of- Thought (CoT) prompting (Wei et al., 2022) on logical preference consistency. Surprisingly, CoT rea- soning did not generally improve consistency, and in some cases, it led to a decrease in transitivity performance, We hypothesize that this decline may be due to the additional CoT reasoning tokens could introduce variations in the judg- ment standards used for different comparisons. When the understanding of the preferential relations is not uniform, it may produce inconsistent con-trrnsitivv) outcomes. This suggests that CoT prompting, while beneficial for complex reasoning, might introduce unintended inconsistencies certain logical judgment tasks.
Commutativity Correlates Strongly with Human Pref- erences. For each task, we paraphrased the comparison prompt template 10 times using GPT-4-turbo to explore the sensitivity of LLMs to input variations. Despite these paraphrases being semantically equivalent, they produced di different performance outcomes, as shown Figure 5. We observe strong correlations between commutativity and hu- man agreement rates across nearly all datasets and models. lack of commutativity
This finding is not surprising, often indicates strong positional bias. These results align with previous research (Zhou et al., 2024), which demon- strated that positional bias can significantly impact align-
5

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
SummEval
NovelEval
CaTeRS
0.775
pivelue
0,003
despoodkchh chu
0.975
p-odlce m 0. 001
preslue - 0,000
0.250
0.22
gemmai min
seff-ggeeemen
0.223
0.500
0223
0.22
acptor 72
gtrig 2 44
0.875
0.00
Aiir 3.30
Jama- m.ma-3-00
0.50
Liams 3.7b
ephyr- leiia
2.
0.225
Lune2:73
47
glama 2:13b
0.22
Par5 mln1
0.600
0.025
2
0.775
040
0.45
0.50
0.55
0.00
0.55
0.70
6.75
0.50
0.5
0.6
0.7
0.5
0.9
9.5
0.4
0.6
0.7
Stran( (5)
Figure 4: Transitivity shows strong correlations with self-agreement across all three datasets. Self-agreement is measured as the percentage of majority choices from 10 CoT inferences, generated with temperature 0.7.
Meta-L Lamaa---a
Uama-2-76
Mata-Llaama-3-88
Llama-2-70
Meta-Llama-3aee p-sce- 6:000
Llama-2-77 S-0 on 2,000
2-00ee- 5.000
0.54
2000
p-sokin- 601
4.59
p-0taa in 0.225
Human judgement Accuracy
5.10
6.5
0080-
2330-
0.53
1:40
0.00
05
germaa-2-9b
Phi-3-mini
gemmme-2-90
Phi-3-mini
germma-2-99 2-9b
Phi-3-mini
Spearman
Spuerman CST
0.0
Spearnet 0.30
0.338
D = 05-1
cap
p- 0.006
6,136
0.8
6.7
3.5
0.5
066 6.0 6.0 032
1.6
62
05
1.4
85
775 0,00 8555 CaSC 8015 63K
05
0.5
0
3.75
6.6
835
0.5
05
0.1
Scomm
Figure 5: Commutativity shows a generally strong correlation with human preference across various LLMs and datasets.
ment with human judgments.
4.1. Estimating Rankings from Noisy Pairwise Data
We also show that the strength of this correlation varies between models. For instance, Llama-3-8B exhibits higher correlation with human preferences compared to Gemma-2- 9B. We hypothesize that this difference is due to Gemma- 2-9B's training, which was designed to be more robust to input prompts, while other models, like Llama-3-8B, are more fine-tuned to specific prompting styles.
Estimating global rankings from noisy pairwise annotations is essentially a rank aggregation problem. We use the win- loss rate method due its simplicity and two key advan- tages: 1) it performs well with partial and sparse compar- isons, which is common in real-world preference datasets, and 2) it remains unaffected by the order in which com- parisons are presented. In Appendix $N, we compare the performance of alternative rank aggregation methods, such as the ELO rating and the Bradlcy-Terry model, within the REPAIR framework. Additionally, we provide detailed analysis of their respective strengths and limitations.
4. Improve Logical Preference Consistency in LLMs via REPAIR
Our analysis so far has demonstrated that LLMs ex- hibit inconsistent logical behavior in making preferen- tial judgments. To address this, we propose REPAIR (Ranking Estimation and Preference Augmentation through Information Refinement), framework that aims to miti- gate these inconsistencies. REPAIR first estimates a rank- ing from noisy preference data, then generates additional conffict-free pairwise comparisons. This approach enhances logical preference coherence while maintaining alignment with human preferences, improving LLMs' reliability as logical operators. See Appendix SK for detailed motivation.
The win-loss rate for each item is calculated as the number of comparisons it wins minus the number of comparisons it loses and then divided by the number of comparisons it participates in. Following that, as shown in Figure 6, we rank the item by the value of its win-loss rate. Through this way, a full or partial ranking is aggregated. We then extrap- olate the ranking into self-consistent pairwise comparison set, which can be further augmented by adding comparisons with the negated relation.
6

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Raw Noisy Painwise Annotations
Preference Ranking Estimation:
Augmented conflict-free Annotations
Win-loss Rate
X1
X2
X3
X4
X5
X1
X2
X3
X4
X5
X1
A
K5 (xpx%))
X1
A
A
A
A
X1 X2
X2 2g
0g2%%)
0
X1 Xg
X2
X2
A
Xy (XF
X2
B
A
X1 X4
X5 <Xg
X3
+1
X3
B
A
X2 K2
X5 (X62X2)
(x5
X22
X4
X4
B
B
B
B
X5
B
X1 X2=x2=X=-X>>XX
X5
Figure 6: Illustration of data refinement and augmentation based on preference rank estimation using win-loss rates. The original annotations (left) are sparse and exhibit inconsistencies moonttransttivity). while the augmented annotations are consistent and include more comparisons. For partially ordered rankings, items are not compared when their preference relationship is unknown.
Table 2: Statistics at each augmentation stage. # Data means data size and AvgComp/Inst for average comparisons per instance. REPAIR-ed data represents data augmented by our REPAIR framework and Neg. denotes further augmentation with negated relation comparisons.
Table 3: cetticttioottttinn of Llama-3 Instruct (8B) using different variants (see Section 4.2). Only training on negated relations can enhance negation invariance.
Summarize from Feedback
Models
H.
Stran (5)
Scomm
8neg
Data
8 Styan
8comun
Data
AvgComp/Inst
Zero-shot inference
64.3
81.1
70.2
63.8
Raw data
98.4
14.8K
6.29
Raw data
74.4
93.1
89.0
61)66
Perturbed data
87.6
14.8K
6.29
Perturbed data
REPAIR-ed data
1
30.9K
13.2
REPAIR ed data
70.1
95.4
91.2
60.8
REPAIR-ed data+Neg.
1
61.8K
26.4
REPAIR-ed data + Neg.
69.7
95.2
90.5
87.9
4.2. Experiments
dataset with additional negated relation comparisons. Training hyperparameters are detailed in Appendix $ F. For evaluation, we randomly sample 200 instances from the test set and evaluate all models on this subset. The impact of the data augmentation is assessed in terms of logical consistency and model performance.
Experimental Setup. The previously used datasets (Sum- mEval, NovelEval, and CateRS) were primarily designed as evaluation benchmarks and are therefore too limited in size support model training. To address this limitation, em- ploy the Summarize from Feedback dataset (Mostafazadeh et al., 2016b), where annotators made pairwise comparisons between two summaries based on qualitative factors. Fur- ther details about this dataset can be found in Appendix $B. Additionally, we provide experiments on another dataset in Appendix SL. Given that original annotations are sparse and relatively clean, domly flipping 10% of the training labels. The objective of REPAIR is to both refine and augment this noisy dataset, thereby enabling the extraction of more reliable training signals. To this end, we apply REPAIR to enhance the con- sistency and increase the volume of pairwise comparisons, as shown in Table
Results and Findings. Results in Table highlight four key findings: 1) Zero-shot inference shows considerable logical inconsistency. However, training on perturbed data can already substantially improve both human preference alignment and consistency. 2) Training the original, clean dataset provides an upper bound in terms of align- ment with human judgments, as contains the complete and uncorrupted signals. Interestingly, this does not neces- sarily lead to superior logical consistency when compared to models trained on the REPAIR-ed perturbed data. 3) Models trained with the REPAIR-ee dataset demonstrate significant gains in transitivity and commutativity, while pre- serving strong alignment with human preferences. 4) Only training further on negated relations improves the model per- formance in negation invariance. However, adding negated relations to the broader dataset may introduce distractions and cause a forgetting effect (Luo al., 2023), resulting in slight reduction in performance on other logical properties.
introduce synthetic noise by ran-
To assess the effectiveness of REPAIR, instruction- tuned three original raw and clean training set data, 2) the flipped/per- turbed training set data, 3) the refined and augmented dataset, denoted as REPAIR-ed data, and 4) The REPAIR-
models on: 1) the

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Table 4: LLMs with better transitivity perform better with PairS. Improved commutativity indicates less need for algorithm calibration. For both PairS methods, we report the average Spearman correlations over 100 runs.
SvmmEval (Coh)
Models
H.
Stran (5)
8eomm
PairS
PairS calibrated
Mistral-7B
63.6
95.1
59.9
27.7
31.2 +3.5
Phi-3-mini
65.6
92.8
46.9
33,9
38,0 +4.1
GPT--355-tube
66.3
82.5
67.5
33.5
36.3 +2.8
Llama-3-8B
67.8
91.0
76.1
37.7
38.9
+1.2
Phi-3-medium
68.8
96.2
71.0
38.9
41.3 +.2,4
Overall, the findings support the effectiveness of REPAIR in improving the logical preference consistency of L.LMs.
insights: 1) Although Phi-3-mini has slightly lower accu- racy with human judgments than GPT-3.5-turbo, its stronger transitivity leads to better ranking performance with with- out calibration. 2) There is clear correlation between an LLM's commutativity and the performance gains from calibration in the PairS algorithm. LLMs that are more commutative rely less on calibration to achieve optimal per- formance, and therefore, require less computations.
5. Impact of Logical Preference Consistency on Downstream Applications
LLMs are increasingly used logical operators in high- level algorithms due to their ability to process and under- stand text semantics. For instance, Qin et al. (2024) use LLMs to enhance document reranking in information re- trieval systems. Similarly, Guo et al. (2024) and Yang et al. (2024) utilize LLMS as optimizers in prompt-tuning algo- rithms, while Liu et al. (2024) and Liusie et al. (2024b) employ LLMs as pairwise comparators to aggregate prefer- ence rankings. When LLMs are used as logical operators, maintaining a high level of logical consistency is critical to ensure predictable and efficient decision- focision-making, In this section, we examine how logical preference consistency in- fluences the performance of LLMdbased algorithms in such 'logically grounded' tasks.
6. Related Work
Consistency of LLMs has been explored in several contexts before, where previous research has predominantly focused on two domains: consistency of factual knowledge and en- tailment consistency across limited statements.
Consistency of Factual Knowledge. Previous work has demonstrated that in nhowledge-intensing question- answering tasks, concepts like symmetry and transitivity of knowledge snippets are important (Asai & Hajishirzi, 2020). To this end, a benchmark for studying language model consistency with paraphrased relations (Elazar et al., 2021) has been created. Additionally, different studies have examined whether LLMs have coherent understanding of knowledge graphs related to real-world entities (Jung et al., 2022; Gu et al., 2023; Kassner et al., 2023; Gu et al., 2023), The 'reverse curse' phenomenon highlights that unlearned factual knowledge cannot be deduced by reversing learned knowledge (Allen-Zhu & Li, 2023; Berglund et al., 2024).
We illustrate the impact of logical preference consistency through the proposed by Liu et al. (2024). PairS is a sorting-based based rank aggregation method that uses LLMs as pairwise evaluators (i.e., a particular juddee algorithm), comparing items based on specific attributes using a merge-sort ap- proach. Its performance is measured by comparing the LLM-gennrated rankings with human judgments via Spear- man's correlation. Sorting algorithms depend heavily logical properties. PairS assumes that LLMs used as com- parators have near-perfect transitivity and commutativity for optimal ranking results.
Search (PairS) algorithm
Entailment and Inference Consistency. In Natural Lan- guage Inference (NLI), the consistency of transitivity and symmetry in statement pairs has been explored using prede- fined inference rules (Li et al., 2019; Jang & Lukasiewicz, 2023). Jang et al. (2022) proposed a test set to exam the consistency of LLMs for NLI tasks. Prediction consistency has been improved through the use of adversarial first-order logic examples (Minervini & Riedel, 2018).
To evaluate this, we conduct controlled experiments on the coherence aspect of SummEval, using various LLMs with similar accuracy human judgments. As shown in Table 4, we follow Liu et al. (2024) by reporting both raw and cali- brated results. The calibration, performed as described in Zheng et al. (2023), averages the evaluation probabilities across both possible pairwise orders. While this technique increases inference cost, it reduces bias by balancing posi- tional preferences. Our findings from Table reveal two key
Despite the wealth of knowledge regarding the logical con- sistency of LLMs within specific domains, most studies have concentrated on first-order relations tions or implications directly linking two or three individual statements. Consequently, there is a notable gap in research
connec-
8

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
addressing the consistency and reliability of LLMs when applied to more complex decision-making scenarios or the evaluation of multiple items simultaneously. This limitation suggests a pressing need for further investigation into how LLMS can maintain coherence and consistency in broader, more comprehensive contexts, which is essential for their deployment in practical applications.
48550/arXiv. 2404. 14219.
Abiteboul, S., Hull, R., and Vianu, V. Foundations of
databases, volume 8. Addison Addison-Wesley Reading, 1995.
Allen-Zhu, Z. and Li, Y. Physics of language models: Part
3.2, knowledge manipulation. CoRR, abs/2309. 14402, 2023. doi: 10.48550/ARXIV:2300. 2tt.:/23299..44224.22 14402. URL https:
//doi.org/10 48000008555///// 2309. 1 14402.
7. Conclusion
Anil, R., Borgeaud, S., Wu, Y., et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312. 1805, 2023a. doi: 10.48550/ARXIV.3112.
In this work, we explored the critical role of logical prefer- ence consistency in enhancing the reliability and trustwor- thiness of LLMs, introducing a framework to quantify three key properties: transitivity, commutativity, and negation invariance, which serve as strong proxies for assessing over- all reliability. To address logical inconsistencies, we pro- posed REPAIR, data refinement and augmentation frame- work, demonstrating that models trained on REPAIR-ed data achieve higher logical preference consistency without compromising human alignment. Additionally, integrating logically consistent LLMs into logic-driven algorithms en- hanced global performance and computational efficiency, underscoring the practical benefits of logical coherence in downstream tasks. Our findings highlight logical consis- tency as complementary factor to human alignment in developing more reliable LLMs, encouraging its recogni- tion as a fundamental aspect of improving trustworthiness and reliability.
11805. URL https://doi org 48550/ 2312.11895
Anil, R., Dai, A. M., Firat, O., et al. Palm 2 technical report. CoRR, abs/2305. 10403, 2023b. doi: 10.48550/ARXIV, 2305. 10403 httpttts:://// org///..r/////// http//.0..8550///4 48550/arXiv. arxiv. 2305. 10403.
Arrow, K. J. A difficulty in the concept of social welfare. Journal of political economy, 58(4))328-346, 1950.
Asai, A. and Hajishirzi, H. Logic-guided data augmenta- tion and regularization for consistent question answering. In Jurafsky, D., Chai, J., Schluter, N. and Tetreault, J. (eds.), Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pp- 5642-5650, Online, July 2020. Association for Computational Lin- guistics. doi: (0.18653/v1/20222 wtw.://aa0.....maam.....9 main URL https://aclanthoooogg hwwphhhooogo.oro/////// org/ acl-main. 499.
Impact Statement
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernan- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D.. Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Training a helpful and harmless assistant with rein- forcement learning from human feedback, 2022. URL https://arxixs org ab20/ 228.22862..........
There are several potential societal consequences of our work. By ensuring logical coherence in LLMs, we hope to contribute to the development of more reliable and trustwor- thy Al systems. This could have positive implications for various sectors, including healthcare, finance, and educa- tion, where decision- makinn and reasoning tasks are critical. However, it important to consider the ethical aspects of de- ploying such systems. Ensuring logical consistency does not automatically guarantee fairness, transparency, or account- ability. Therefore, we encourage the broader community to continue exploring these ethical dimensions in parallel with technical advancements. In summary, while our work aims to advance the field of artificial intelligence by improving the logical consistency of LLMs, we recognize the need for ongoing discussions and research to address the ethical and socictal implications of deploying AI systems.
Barnard, G. Statistical inference. Journal of the Royal Statistical Society Series B: Statistical Methodology, 11 (2):115-139, 1949.
Batagelj, V. and Mrvar, A. A subquadratic triad census algorithm for large sparse networks with small maximum degree. Social networks. 23(3):237-243, 2001.
Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stick- land, A. C., Korbak, T., and Evans, O. The reversal curse: Lims trained on "a is b" fail to learn "b is a". In The Twelfth International Conference on Learning Represen- tations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreviee net//forumiid-GKTIIKK..
References
Abdin, M. I., Jacobs, S. A., Awan, A. A., et al. Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abs/2404.1422) 2024. doi: 10. 48550/ARXXIV2404. 14219. URL https:
9

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
Review.net, 2023. URL httos:///penreview..ne// forum?id-3PF3Wg6o-
Dai, J., Pan, X.. Sun, R., Ji, J., Xu, X., Liu, M., Wang. Y., and Yang. Y. Safe RLHF: Safe reinforcement learn ing from human feedback. In The Twelfth International Conference on Learning Representations, 2024. URL https:// http://wnee//oommmiiiyfrrrrooeee
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A.. Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen. M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https:
DeepSeek-AL. Deepseek-v2: A strong, economical, and efficient mixture-of-expert. language model. CoRR, abs/2405.04434 2024. doi: 10.48550/ARXIV.220055
04434. URL http:://doi.org//// /ttp://6b...0////044455/5//// 48550/arXiv. 2405.000 04434.
Dubey, A., Jauhri, A., Pandey, A., et al. The llama herd of models. CORR, abs/2407.21788. 2024. doi:
1048550/ARXXI.200...77.... URL https://dii.orrr 10. 48550/arxiv http:/rriiv.......2. 01.00.21788
//proceedings. neuriisssnnnnnnnissss...u.iii p00p:p///0/ppppppe//e/e///// /ttpc//abbccbbb.74444aaaaaacccaa html.
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., and Goldberg, Y. Measuring and improv- ing consistency in pretrained language models. Transac- tions of the Association for Computational Linguistics, 9:1012-1031, 2021. doi: 10. 10.1 aa 000410. URL https://aclantholoog hwwpolloyy.ogg///000 org/2021. wal.llaacll..c. 60.
Cachola, I., Lo, K., Cohan, A., and Weld, D. S. TLDR:
extreme summarization of scientific documents. In Cohn, T., He. Y., and Liu, Y. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACI., pp. 4766-4777. Association for Com- putational Linguistics, 2020. doi: 10.18653/V1/0200.
Elo, A. E. and Sloan, S. The rating of chessplayers: Past and present. (No Tirle), 1978.
URL https: //ds: org/10. 36653/v1/2228 findings-emnnn 428.
Fabbri, A. R., Kryšcifiski, W., McCann, B., Xiong, C., Socher, R., and Radev, D. SummEval: Re-evaluating summarization evaluation. Transactions of the Associa- tion for Computational Linguistics, 9:391 2021. doi: URL https::/aaee https://aclanthologgy org/2021.tacl-11
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G. Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Mísra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polo- zov, O., Lee, K., Zhou, Z., Wang, X., Sacta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstenn K., Eck, D., Dean, J., Petrov, S., and Fiedel, N. Palm: Seal- ing language modeling with pathways. Journal of Ma- chine Learning Research, 241240):1-113, 2023. URL /tjmrrr httpp/paps//////222/ 1144. html.
10. taal-1lcl2221 1.24 .4.
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncour, F., Yu, T., Zhang, R., and Ahmed, N. K. Bias and Fairness in Large Language Models: A Survey. Computational Linguistics, 50(3):1097-1179, 09 2024. ISSN 0891-2017. doi: 10. 10.110 162/coll aa0io2llla_00024__1__0 URI. https://doi.org org/ /00. 1162/coli. a.05524.
Gu, Y., Mishra, B. D., and Clark, P. Do language mod- els have coherent mental models of everyday things? Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 1892-1913. Association for Compu- tational Linguistics, 2023. doi: 10. 10.18653/V1/2023 ACL-LONG. 106. URL https://doi.org// /865// v1/2023, 11/2223.acl-1om. acl- 106. Guion, R. M. Validity and reliability. Handbook of research methods in industrial and organizational psychology, pp. 57-76, 2004.
Creswell, A., Shanahan, M., and Higgins, I. Selection- inference: Exploiting large language models for in- terpretable logical reasoning. In The Eleventh In- ternational Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Open-
10

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Guo, Z.,, Liu, M., Ji, Z., Bai, J., Guo, Y., and Zuo, W. Two optimizers are better than one: Llm catalyst empow- ers gradient-based optimization for prompt tuning, 2024. URL
Jung, J., Qin, L.., Welleck, S., Brahman, F., Bhagavamula, C., Bras, R. L.., and Choi, Y. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Gold- berg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 1266- 1279. Association for Computatioonal Linguisties. 2022. URL https:
org/abs htsp/bs//2400 24005 19732.
Hamon, R., Junklewitz, H., Sanchez, I., et al. Robustness and explainability of artificial intelligence. Publications Office of the European Union, 207:2020, 2020.
doi: 10. 1111166//1112222222..-.MMM 1011318663/222222 www.2222..emmmppmmaii.. 82.
Hansson, S. O. The structure of values and norms. Cam- bridge university press, 2001.
Kassner, N., Tafjord, O., Sabharwal, A., Richardson, K., Schütze, H., and Clark, P. Language models with ra- tionality. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2023, Sin- gapore. December 6-10, 2023, pp. 44190-14201, As- sociation for Computational Linguistics, 2023. doi: 10.8863/W1/2023EEMLP-PAMA
Huang, J. and Chang, K. C. Towards reasoning in large language models: A survey. In Rogers, A., Boyd- Graber, J. L., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 1049-1065. As- sociation for Computational Linguistics, 2023. doi: 10. 10.11663/W11/22233F FIN..N URL. https: //doi. org/10..r111.0 0t////.0.116555 101138886518288 (eeee..finningg Findings-acl.66
URL https: //doi.org/// 18653/v1 emn emn.p-main..mmm. 877.
Hyde, D. The sorites paradox. In Vagueness: A guide, pp. 1-17. Springer. 2011.
Krackhardt, D. and Stern, R. N. Informal networks and organizational crises: An experimental simulation. Social psychology quarterly, pp. 123-140, 1988.
Imieliński, T. and Lipski Jr, W. The relational model of data and cylindric algebras. Journal of Computer and System Sciences, 28(1):80-102, 1984.
Kumar, A. and Joshi, A. Striking a balance: Alleviating in- consistency in pre-trained models for symmetric classifi- cation tasks. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Findings of the Association for Computational Linguisties: ACL 2022, pp. 1887-1895, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. http:///ww///2222.nnnngggggo//e 148. URL https: waaclanthology.orggooooo
Jang, M. and Lukasiewicz, T. Consistency analysis of chatgpt. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2023, Sin- gapore, December 6-10, 2023, pp. 15970-15985. As- sociation for Computational Linguisties, 2023. doi: 10.18653/V1/20233 WEW.:02I.EEEEL---III.I..M URL https: //doi.oog//// 10113886651///// http///////22222....mpmmaaain
findings- acl. 148.
Li, T., Gupta, V., Mehta, M., and Srikumar, V. A logic- driven framework for consistency of neural models. In Inui, K. Jiang, J., Ng. V., and Wan, X. (eds.), Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing. EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp- 3922-3933. Association for Computational Linguistics, 2019. doi: 18.18653/V 1/D19- 1405. URL https://doi.org / 18653/v1 DD55://111199-19555
Jang, M., Kwon, D. S., and I Lukasiewicz, T. BECEL: Bench- mark for consistency evaluation of language models. In Calzolari, N., Huang, C.-R., Kim, H., Pustejovsky, J., Wanner, I.., Choi, K.-S., Ryu, P.-M., Chen, H.-H., Do- natelli, L., Ji, H., Kurohashi, S., Paggio, P., Xue, N., Kim, S., Hahm, Y., He, Z., Lee, T. K., Santus, E., Bond, F., and Na, S. (eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 3680- 3696, Gyeongju, Republic of Korea, October 2022. Inter- national Committee on Computational Linguisties. URL https: http::///aclaathololl. wttgg/////2222 coling-1. 324.
Liu, Y., Zhou, H., Guo, Z., Shareghi, E., Vulić, I., Korho- nen, A., and Collier, N. Aligning with human judgement: The role of pairwise preference in large language model evaluators. In First Conference on Language Model- ing, 2024. URL 9gdZI7c6yr.
wwwpennwwiiiw..eeeeorreeee net/forummid==
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, I., Lavaud, I.. R., Lachaux, M., Stock, P., Scao, T. L.. Lavril, T., Wang, T., Lacroix. T., and Sayed, W. E. Mistral 7b. CORR, URL https: 2023. doi: (0.48550/ARXIV23100066 /doii.or//11 48550/ arvi.. htt.://..........03333 2310.06825.
Liusie, A., Fathullah, Y., and Gales, M. F. Teacher-student training for debiasing: General permutation debiasing for large language models. In Ku, L., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Com- putational Linguisties, ACL. 2024, Bangkok, Thailand

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
and virtual meeting, August 11-16, 2024, pp. 1376-1387. Association for Computational Linguisties, 2024a. doi: URL https:
to-sequence rmns and beyond. In Goldberg, Y. and Riezler, S. (eds.), Proceedings of the 20th SIGNLL Conference Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 1 11-12, 2016, pp. 280- 290. ACI,, 2016. doi: 10.18653//1/K1166100 1028. URL https://doi.org/10 18653/v1/k16-1100
10, 10.1166331112244AEDDDNSSGGGGS //doi.org/10. 18653/ (tt/5///112222.finnnnggg findings-acl..
Liusie, A., Manakul, P., and Gales, M. Lim comparative assessment: Zero-shot nig evaluation through pairwise comparisons using large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 139-151, 2024b.
Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., and Deng, L. MS MARCO: A hu- man generated machine reading comprehension dataset. In Besold, T. R., Bordes, A., d' Avila Garcez, A. S.. and Wayne, G. (eds.), Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop Proceedings. CEUR-WS.org 2016. URL https: https://ceurrwue http///eurrrrrrreeee Vo1-1773/COCONIPS 2016 paper9. pddf.
Luo, Y., Yang, Z., Meng, F,, Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08877
2023.
Miller, B. P., Reed, D. D., and Amlung, M. Reliability measures: review
and validity of and synthesis of discounting and demand. Journal of the experimental analysis of behavior, 120(2):263-28) 2023.
OpenAL. GPT-4 technical report. CORR, abs/2303.08777, 2023. doi: 104455WARXXV22303007777 URL https: //doi.org/10 48550 ar..v. 2303. 08774.
Minervini, P. and Riedel, S. Adversarially regularising neu- ral NLI models integrate logical background knowl- edge. In Korhonen, A. and Titov, I. (eds.), Proceed- ings of the 22nd Conference on Computational Natu- ral Language Learning, CoNLL 2018, Brussels, Bel- gium, October 31- November 1, 2018, pp. 65-74. As- sociation for Computational Linguistics, 2018. doi: 10.18653/V1/K1 1007. URL 18653/v1/k18-11
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M. Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022. URL https: //arxiv.orge wow..orggggooo..ooo /t//abbs//2/// 221.
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Ba- tra, D., Vanderwende, L., Kohli, P., and Allen, J. F. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Knight, K., Nenkova, A., and Rambow, O. (eds.), NAACI HI. 2016, The 2016 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pp. 839-849 0499.849..68 The Association for Computational Linguistics, 201 doi: 10.18653/V1/11-1100 1099.. URL https://doi.onm org/00 10.
Qin, Z., Jagerman, R., Hui, K., Zhuang, H., Wu, J., Yan, L., Shen, J., Liu, T., Liu, J., Metzler, D., Wang, X., and Bendersky, M. Large language models are effective text rankers with pairwise ranking prompt- ing. In Duh, K., dómez-Adorno H., and Bethard, S. (eds.), Findings of the Association for Computational Linguisties: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 1504-1518, Association for Compu- tational Linguistics, 2024. doi: 10.18653/V1/2024, FINDINGS-NAACL.97, URL https://doi..rg/// 18653/v1/2824 findings-nacll.999
Mostafazadeh, N.. Grealish, A., Chambers, N., Allen, J., and Vanderwende, L. CaTeRS: Causal and temporal relation scheme for semantic annotation of event struc- tures. In Palmer, M., Hovy, E., Mitamura, T., and O'Gorman, T. (eds.). Proceedings of the Fourth Work- shop on Events, pp. 51-61, San Diego, California, June 2016b, Association for Computational Linguistics, doi: 10. 10.18653/v1/W16 1007. URL https:://claantholos org/w16-
Regenwetter, M., Dana, J., and Davis-Stober, C. P. Tran- sitivity of preferences. Psychological review, 118(1):42, 2011.
Restall, G. An introduction to substructural logics. Rout- ledge, 2002.
Rivière, M., Pathak, S., Sessa, P. G., et al. Gemma 2:
Improving open language models at a practical size. CoRR, abs/2408.001 2024. doi: 10.48550/ARXIV. 1408.00118. htts:: h/tpttttss::oo...oeeteteoototttt htt.t:::://ooi.o/////... org/10. 485504855555 2408.00118.
Nallapati, R., Zhou, B., dos Santos, C. N., Gülçehre, C., and Xiang, B. Abstractive text summarization using sequence-
12

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Schmidt, F. L., Viswesvaran, C., and Ones, S. Reliability is not validity and validity is not reliability. Personnel Psychology, 53(4):901-912, 2000.
Wang, R. and Henao, R. Unsupervised paraphrasing con- sisteney training for low resource named entity recogni- tion. In Moens, M.-F, Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5303-5308, Online and Punta Cana, Dominican Repub- lic, November 2021. Association for Computational Lin- guistics. doi: 10.18653/v 1tt0:1/111111.../ wet.:22..mmmmmmmmame main.430. URL https://aclanttoll// org/ //wgg emmll-mminn 430.
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., and Wang, H. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 38, pp. 18990-18998, 2024.
Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., and Ren, Z. Is chatgpt good at search? in- vestigating large language models re-ranking agents. In Bouamor, H., Pino, J., and Bali, K. (eds.), Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2023, Sin- gapore, December 6-10, 2023, pp. 14918-14937. As- sociation for Computational Linguisties, 2023. doi: URL https: //doi.org/10. 18653/v1 emnlp-main. 923.
Wasserman, S. Social network analysis: Methods and ap- plications. The Press Syndicate of the University of Cam- bridge, 1994.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Yang, C., Wang, X., Lu. Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. In The Twelfth International Conference on Learning Represen- lations, 2024. URL https://openreview. wwtpvviwwww...eeeeeeeee net/forum id-Bb4VGOWELI.
Tarski, A. On the calculus of relations. The journal of symbolic logic, 6(3):33-89 19,7399,1110 1941.
Touvron, H., Martin, L.,, Stone, K., et al. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.06 009288, 2023. doi: 10448550/RXXV.2300. 09288. URL 09288.
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., Wang. I.., Luu, A. T., Bi, W., Shi, F., and Shi, S. Siren's song in the Al ocean: A survey hallucination in large lan- guage models. CoRR, abs/2309.00219, 2023. doi: 10 10:4855/ARXIV.230000112 URL httpt::/ddi..rorr /rgg 10. 4855008550iii 2309.
Tunstall, L., Beeching, E. E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., Werra, L. V., Fourrier, C., Habib, N., Sarrazin, N., Sanseviero, O., Rush, A. M., and Wolf, T. Zephyr: Direct distillation of LM alignment. In First Conference on Language Modeling, 2024, URL https://openreviae httpvviwwwwneeefrrreeeeeeeee ntt//orummid--aaawWEaE
Zheng, C., Zhou, H., Meng. F., Zhou, J., and Huang, M. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2023.
Tversky, A. Intransitivity of preferences. Psychological review, 76(1):31, 1969.
Zhou, H. Wan, X., Liu, Y., Collier, N., Vulić, I,, and Ko- rhonen, A. Fairer preferences elicit improved human- aligned large language model judgments. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1241-1252, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 100118665//1/2022..mllml. https://aaclanthoolgg..oo org/ 2024. www.l-mainnn.......eemmmmmmme 72/.
Wang, B., Zheng, R., Chen, L., Liu, Y., Dou, S., Huang, C., Shen, W., Jin, S., Zhou, E., Shi, C., Gao, S., Xu, N., Zhou, Y., Fan, X., Xi, Z., Zhao, J., Wang, X., Ji, T., Yan, H., Shen, L., Chen, Z., Gui, T., Zhang, Q. Qiu, X., Huang. X.. Wu, Z., and Jiang, Y.GG. Secrets of rlhf in large language models part ii: Reward modeling, 2024a. wtgg////////0/00 06080.
URL https:://arxt
URL
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong. L., Liu, Q. Liu, T., and Sui, Z. Large lan- guage models are not fair evaluators. In Ku, L., Mar- tins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 9440-9450 Association for Computational Linguistics, 2024b. URL https://aclanthologg) org/ acl mlll-lng..51 511.
13

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
A. Transitivity Calculation
In this section, we show two algorithms: (i) construct a preference relation graph by LLM's judgements, and (ii) detect cycle in relation graph.
Algorithm 1 Construct Preference Relation Graph
1: Objective: Construct relation graph reflecting the understanding of LLM upon set of items.
2: Inputs: X (2;,,2;...5 NN set of items; LLM(:-I4) -I): LLM, for given instruction prompt preference operator.
Output: G: relation graph represented as adjacency list for each vertex.
4: Initialize: Initialize the relation graph G such that for each æ in X, G[æ] + 2, where G[æ] represents the pointed vertexes from
vertex a.
5: for i 1-4 n. do to do P(#; > ur)) <- aL
6: j 3
8:
if P(x; > xr) ) 2 then
9:
G[a.] t- G[a.] (x))
10:
else
12: end Gl, {
13: end for
14:
end for
15:
Return G
Algorithm 2 Check for Cycles in a Directed Graph
I: Objective: Detect if directed graph G contains cycle.
2: Inputs: G: directed graph represented an adjacency list.
3: Output: True a cycle exists, False otherwise.
4: Initialize: eisited +- Ø. recStack +- Ø
5: for each in G do
6: if & visited and CycleUtil(e,, visited, recStack) then
Return True end
9: end for
10: Return False
1: Function CycleUtil(v, visited, recStack)
2: visited + visited U {v}
3: recStack recStack U (u]
for each u G[v] do
5: u é visited A CycleUtil(u, visited, reeStack) then
6: Return True
7: else if recStack then Return True
9:
10: end for
11: recStack reaStack \ {w}
12: Return False
B. Dataset
SummEval (Fabbri et al., 2021) summarization meta-evaluation dataset comprising 100 source texts, each paired with 16 summary candidates generated by various language models (LMs). The dataset annotated based four criteria: coherence (COH), fluency (FLU), consistency (CON), and relevancy (RE).
NovelEval (Sun et al., 2023) is a document reranking test set consisting 21 novel questions. This set was constructed by compiling questions and passages from four domains published after the release of GPT-4. For each question, 20 candidate passages retrieved through Google search, and their relevance manually labeled.
CaTerS (Mostafazadeh et al., 2016b) is a dataset focused on temporal event ordering. It contains 1,600 annotated sentences derived from 320 five-sentence short stories sampled from the ROCStories corpus (Mostafazadeh al., 2016a). These
14

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
stories are organized based on causal and temporal relations. We filtered the dataset include only instances with seven or more events, resulting in 70 instances.
Summarize from Feedback. (Mostafazadeh et al., 2016b) consists of 64,832 summary comparisons drawn from the TL:DR (Cachola et al., 2020) and CNN/DM datasets (Nallapati et al., 2016). This dataset is divided into two sections: pairwise comparisons and axis annotations. We concentrate on the pairwise comparison section, which is labeled based on the overall quality of two summaries, For training, we utilize the entire training set, while for testing, we randomly sample 200 instances from the validation set.
MS MARCO. (Nguyen et al., 2016) is widely used large-scale benchmark for training and evaluating retrieval-based question answering systems. Following the methodology of Sun et al. (2023), we leverage the dataset for a document reranking task. Specifically, they randomly sample a subset of queries from the MS MARCO training set, retrieving 10 candidate passages for each query using BM25, Ranking preferences for these passages were distilled using GPT--35-ttrbb In this work, we decompose the preference rankings into full pairwise comparison annotations. We randomly sample 1K examples from their query subset and corresponding ranking labels. We then split them into a training set of 800 examples and test set of 200 examples.
C. Large Language Models in Evaluation
Open-Sourced LLMs. The -lama-2-7b-chatte and Llama-2- 1--3---hhat-hhh models (Touvron et al., 2023) were developed by Meta and are part of the Llama-2 family. Llama-3-8B-Instruet (Dubey et al., 2024) is another model from Meta's Llama series. The
(Jiang et al., 2023) model is an instruction-tune conetiiioo-teeddddee LLM developed by Mistral AI The zephyr-7b-beta model (Tunstall et al., 2024) is developed by Zephyr AI. The 1ia----mnii-4--nsttee instruct models (Abdin et al., 2024) are part of the Phi model series developed by Microsoft. The gemma-2-9b-it (Rivière al., 2024) model is created by the Gemma team.
and Phi-3-medium-44
API-Based LLMs. The 11t.3.5-turbo0111 model is part of OpenAI's GPT-3.5 series, which is renowned for its strong performance in natural language understanding and generation. Deepseek-chat- chattv2 (DeepSeek-AI, 2024) is another API-based L.M developed by DeepSeek.
D. Additional Related Work
Improving Consistency of LLMs. Asai & Hajishirzi (2020) proposed augmentation method and a training regularization term to improve the logical consistency of language models for QA tasks. Minervini & Riedel (2018) proposed an adversarial training procedure to improve the robustness of NLI models to ad adversarial examples. Wang & Henao (2021) focus on improving the paraphrasing consistency during training for the task of low-resource Named Entity Recognition (NER). Li et al. (2019) proposed a framework to regularize the loss function when training a neural model, based on logic rules to improve the logical consistency.
Kumar& & Joshi (2022) performed a symmetric consistency analysis on NLI and semantic textual similarity (STS) tasks in a more conservative manner, arguing that a model should generate not only the same predictions but also the same confidence scores if it is truly input-order invariant. They also observed that pretrained language models violated symmetric consistency and the authors then introduced consistency regularisation term to compensate for the issue.
Theory of Consistency. The concept of transitivity in preferences has been extensively explored, revealing its significance in both psychological and logical frameworks. Tversky (1969) highlighted various psychological factors that can lead to intransitive preferences, illustrating how human maaiinn often deviates from rational models. Building on this, Arrow (1950) introduced Arrow's Impossibility Theorem, which addresses the inherent challenges of constructing social choice functions that maintain transitivity.
Further contributions to the understanding of transitivity include comprehensive review by Regenwetter et al. (2011), which examines both theoretical and empirical perspectives on the subject. Their work underscores the critical role transitivity plays in preference structures. In a more formal context, Tarski (1941) examined the role of transitivity within relational calculus, an area that foundational to many logical systems. This connection demonstrates how transitivity not only influences preferences but also underpins logical reasoning.
Moreover, Hansson (2001) provided insights into the implications of violating transitivity in preferences, showing that such
15

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
violations can lead to logical inconsistencies within decision theory. This theme is further explored by Hyde (2011), who proved that apparent violations of transitivity can give rise logical paradoxes. Collectively, these works emphasize the importance of transitivity in ensuring consistency, both in decision-making and logical deduction.
E. Distinguishing Logical Transitivity from Social Network Analysis (SNA) Transitivity
Transitivity is a well-studied concept in Social Network Analysis (SNA). However, the definition and purpose for transitivity in our framework diverge significantly from those in SNA. Below, we highlight the key distinctions:
Motivations. In SNA, transitivity measures the tendency of nodes in network to form tightly connected groups. It is based on real-world social patterns, where "friends of friends" are likely to be friends, leading clustered communities. In contrast, our logical consistency framework employs transitivity to assess self-conflict in relation graphs generated by models or human judgments. Here, transitivity serves as an indicator of logical coherence rather than network clustering.
Definition. In graph theory, transitivity quantifies the formation of closed triangles (fully connected triplets), indicating the probability that two neighbors of a node are also connected. In our framework, however, it measures the extent to which relation graphs are acyclic. Specifically, it evaluates the probability that a randomly selected set of nodes forms an acyclic structure, extending beyond simple triplet closures.
Application and Scenarios. SNA transitivity is primarily applied to undirected social network graphs, while our logical transitivity operates exclusively on directed relation graphs. SNA transitivity focuses on the closure of triplets, whereas our logical transitivity evaluates logical circulations spanning to K nodes, with K being the size of the sampled sub- uub-graph. Extending SNA measures to higher-order relationships is nontrivial and computationally inefficient.
To illustrate these differences, we provide detailed analysis of popular SNA transitivity measures:
1. Clustering Coefficient (Wasserman, 1994): Global clustering coefficient measures
Number of Closed Triplets
Number of All Triplets (closed and open)"
and local clustering coefficient measures
C Number of closed triplets including node
All triplets including node i
While applicable to directed graphs, clustering coefficient still measures the ratio between closed triplets and total triplets instead of circular relations and cannot be applied larger structures.
2. Triad Census (Batagelj & Mrvar, 2001): Counts the relative frequency of 16 possible triadic configurations in directed graphs. However, this method is inherently constrained to triads.
3. E-1 (External-Intermaa)) Index (Krackhardt & Stern, 1988): Measures the ratio of external to internal ties within a given partition, capturing group closure rather than logical acyclicity.
F. intruuttionTunning Training Details
The model was fine-tuned using the following hyperparameters. A learning rate of 5 x 10-5 was employed over the course of 2 training epochs. A weight decay of x 10-2 was applied to prevent overfitting. For the LoRA (Low-Rank Adaptation) settings, the rank r was set to 16 and the scaling factor a was set to 64. The batch size during training was 4, with a gradient accumulation step of 2 to effectively handle smaller batches. All training was performed on an A100 machine.
G. Metric Computation details and Prompt Templates
Metrics Computation details. For all three datasets, they have the format of each input context is associated with multiple items, and the task is to determine specific relations among them.
For SvmmEval, the input context is a source text, and the task involves determining coherence preferences among multiple summary candidates. In Eva, the input context a query, with the task focused on assessing relevance preferences
16

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Prompt for (x,, x2)
Prompt for F (X2 X1)
Prompt for (x,, X2)
Source Text: { (nnput
Source Text: {( (input))
Source Text: {( (nput }}
Summary candidate A: output_1 }} Summary candidate B: {{ output_2]]
Summary candidate A: (I output_2 2 )) Summary candidate B: {f output_1 }}
Summary candidate A: output }}
Summary candidate B: {( output
Question: Which summary candidate better coherence?
Question:
Which summary candidate has better coherence? Please only answer with or B.
Which summary candidate worse coherence?
Question:
Please only answer with A
Please only answer with A
Answer:
Answer:
Answer:
(a) SummEval
Prompt for (x,,
Prompt for (X2. x,)
Prompt for (x,,
Query: (( ((iput))) }>
Query: (( input }}
Query: (( ((iput]] }}
Document candidate A: {( output_1 y) Document candidate B: {[ output
Document candidate {{ output 2 Document candidate {{ output_
Document candidate A: {( outpput_1 }
Document candidate B: {( output 2
Question: Which document candidate is more relevant to the query? Please only answer with A or B.
Question: Which document candidate is more relevant to the query? Please only answer with A or B.
Question: Which document candidate is less relevant to the query?
Please only answer with or B.
Answer:
Answer:
Answer:
(b) NovelEval
Prompt for (X,,
Prompt for F (X2. Xy)
Prompt for F (x₁,
Ien
Event context: { input
Event context: {{ input
Event context: {( input
Question: Does {( output_1 }) happen temporally causally later than
Question: Does (( output_22 2 happen temporally causally later than {( output_1 })? Please only answer with 'Yes' or 'No'.
Does {( output_1 }} happen temporally causally earlier than (( output 2 yy? Please only answer with 'Yes' or 'No'.
Question:
{( output_2 2 y Please only answer with 'Yes' or 'No'.
Answer:""
Answer:
Answer:
(c) CaTeRS
Figure 7: Prompt templates for pairwise comparisons. From left to right: Normal comparison, comparison with reversed item order, and comparison with negated relation.
17

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Prompt (g)
Prompt for F(X,
#W Source text: (( input }}
#MH Source text: {{ Input })
WWW Summary candidate A: {{ output_1 1 ?)
#M Summary Summary candidate A: {( output
WW Summary candidate B: {{ output
# Summary candidate B: {( output 2
## Question: Which summary candidate is
## Question: Which summary candidate is
overall better? Please select A B.
overall worse? Please select or
### Answer:
### Answer:
(a) Summary with Feeback
Figure 8: Pairwise comparison prompt templates for the instruction tuning Summary with Feedback, discussed in Section 4.
among multiple retrieved documents. For CaTeRS, the input context consists of a list unordered events, with the goal being to infer the temporal or causal order among these events.
To compute the consistency metries, a full pairwise comparison is conducted between every pair of items, including all permutations of the item set X. Both standard expression (F(,, RT)) and its reverse (F(r,, xr2)) are used. This will result two preference matrices shown in Fig. and can be used compute the Stran. Scomm and Sneg following the Equ. 1, Equ. 2 and Equ.
Prompt Templates. we demonstrate the prompt templates we used to perform pairwise comparisons for all three datasets in Figure 7. Figure 8 illustrates the prompt templates utilized for conducting the instruction tuning experiments on the Summary with Feedback dataset, as detailed in Section 4.
H. The Choice of K
In this section, we discuss the considerations regarding the choice of K for the transitivity metric Stran (K).
SummEval
Staan (K)
0.6
Llama-2-7b
Llama-3-8B
0.4
Mistral-7B
Phi-3-medium
gemma-2-9B
12
K
Figure 9: Transitivity metric (K) as a function of sub-graph size, K, on the SummEval dataset. Transitivity values decrease K increases.
Robustness of transitivity to K. Seran (K) is robust to variations in K. It uncommon for model perform well on Stran (3) but poorly Stran (5), shown in the Figure 9. Additionally, shows that larger values of K expand the value range of Stran (K), making comparisons between models more distinct.
Dependence on Prior Knowledge. Choosing K can depend on prior knowledge about the general consistency performance of current LLMs. For example, we observe that the consistency performance improves progressively from earlier LLMs, such as LLama-2 and Zephyr-7B-bete to more advanced models like Gemma-2 and Llama-3. This trend suggests that considering the evaluated L.MSs expected consistency levels can inform an appropriate choice of K.
18

---

Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Task-Specific Considerations. As discussed in Section 3.2, consistency performance varies depending the task. For objective tasks, consistency tends be higher, which means task-specific nuances should also guide the selection of K.
In summary, there is no definitive "gold standard" for choosing K. Similar to selecting the N-gram size in BLEU-N, the choice depends on the models and tasks under evaluation.
I. Theoretical Evidence for the Choice of transitivity Sub-graph sampling size M
Theoretical Basis. The transitivity metric Stran (K) is computed as the average result of binary indicator function (as defined in Equation 1), which follows binomial distribution. Intuitively, Stran (K) can be interpreted the probability (p) that randomly sampled K-node sub-graph is transitive.
Estimation Accuracy. Using the Central Limit Theorem (Barnard, 1949), the binomial distribution of Stran (K) approxi- mates a normal distribution when the sample size (n) is sufficienlly large. To estimate within a margin of error (E) at a confidence level z (e.g. z = 1.96 for 95% confidence), the required sample size is:
22. pp (1 (p) 71
(4)
E²
Worst-Case Scenario Analysis. Since the exact value of p is unknown, consider the worst-case scenario where p(1 p) maximized 0.5. Under this assumption, for a margin of error E at 95% confidence (= 1.96). The sample size should be at least 394, which is far exceeded by our choice of M = 1000. Therefore, our estimation of the Stron (K) is statistically accurate and stable.
J. The Irreflexive Assumption
As discussed in Section 2, have stated that if comparison function F is transitive, the corresponding relation graph is DAG. This statement is based on the assumption that F is irreflexive. For example, in Figure 3 and Equation 2 and 3, we omit the diagonal elements of the pairwise comparison matrices, and we compute the total pairwise comparisons as | ([X) 1) instead of I
This assumption is justified by practical considerations: in real-world applications- such as evaluating causal or temporal orders or comparing preferences it is uncommon to compare an item against itself, particularly in the context of large language models (LLMs).
K. Motivations for our Data Refinement and Augmentation Framework
Previous work has proven the benefits of using pairwise comparisons derived from preference rankings to train LLMs for a better understanding of preference orderings (Song et al., 2024). Additionally, Asai & Hajishirzi (2020) showed that incorporating logically derived counterparts of pairwise annotations improves consistency in knowledge-intensiee QA tasks. However, real- worldworld preference data are often noisy and self-contradictory (Chowdhery et al., 2023; Ouyang al., 2022), especially in more subjective tasks (Bai al., 2022). The inter-rater agreement rate typically falls between 60% 80% for preference or evaluation datasets. Prior work (Wang et al., 2024a) has shown that these self-confficting annotations reduce the efficiency of learning preferences, and we hypothesize that such annotations also contribute to the logical inconsistencies in trained models. Inspired by RLHF, where a reward model is used to establish complete order of responses, we propose first estimating a coherent ranking from the noisy data. This refined ranking can then be augmented with additional conflict-free pairwise comparisons to align LLMs more effectively. This method is expected to more efficient than aligning models to noisy and inconsistent annotations directly.
L. Performance of REPAIR on the MS-MARCO Dataset
In this section, we present the performance of the REPAIR framework on the MS-MARCO dataset (details provided in Appendix $B). The results, shown in Table 5, correspond to those in Table 3 and Table 4.
Our findings reaffirm the effectiveness of the proposed data augmentation framework. Specifically, we observe that REPAIR enhances consistency while maintaining alignment with human preferences. Additionally, we analyze the Spearman
19

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
Table 5: Instruction-tuning of Llama-3-Instruct-BB on the REPAIR-ed MS MARCO dataset and their performance on the PairS algorithm. LLMs trained on the REPAIR-ed dataset show improved performance on PairS with reduced need for algorithm calibration. For both PairS methods, we report the average Spearman correlations over 100 runs.
MS MARCO
Models
Stran (5)
commm
Sneg
PairS
PairS calibrated
Zero-shot inference
57.2
74.2
68.5
64.7
18.7
22.1(+3.4)
Perturbed data
74.7
86.7
81.1
62.9
58.1
60.3(+2.2)
REPAIR-ed data
75.0
91.3
85.9
63.1
61.0
62.6(+1.6)
REPAIR-ed data Neg.
74.9
90.2
86.2
85.5
correlations of the rankings predicted by the PairS algorithm. Our results indicate that models with higher transitivity produce more accurate overall rankings, while models with stronger commutativity require less calibration to achieve optimal performance.
M. Ablation Study: REPAIR Down-Sampling Performance
In this section, we analyze the performance of the REPAIR framework when sampling equivalent size of perturbed data from the REPAIR-ed dataset. This ensures that the comparison remains unaffected by dataset size differences, allowing for a clearer assessment of the framework's 's efficacy.
REPAIR a data augmentation framework, a widely adopted strategy to mitigate the challenges of acquiring high-quality, consistent data, particularly in the era of L.Mss. Given the increasing difficulty and resource intensity associated with obtaining such data through synthetic methods human annotations, data augmentation offers a practical and cost-effective alternative. Our framework enhances data quality solely by leveraging the perturbed dataset, without requiring any external knowledge.
Although the datasets differ in size, the comparisons in Table 3 remain valid, as they effectively showcase the ability of our approach to extract and expand information from the perturbed data.
To conduct this ablation study, we randomly sample an equivalent amount of REPA -e- data as the perturbed dataset while maintaining all other experimental settings consistent with those in Table 3. The experimental results, summarized in Table 6, yield the following key observations:
1. Information Expansion and Loss:
The REPAIR-ed dataset enriches the information content of the perturbed dataset. However, random sampling from this augmented dataset inevitably leads to some degree of information loss.
Without employing targeted sampling strategy mitigate this loss, the comparison may not fully reflect the optimal benefits of our augmentation approach.
2. Impact on Performance:
Due to the inherent information loss from random sampling, alignment with human preferences exhibits a slight decline. However, this performance drop remains relatively minor.
Despite the reduced dataset size, all three consistency metrics demonstrate noticeable improvements over the perturbed dataset. This highlights the robustness of our augmentation framework. even though the performance naturally lower than that of models trained the full augmented dataset.
These findings reaffirm the efficacy of proposed framework and offer deeper insights into its behavior under constrained data conditions.
N. Ablation Study: Alternative Ranking Estimation Methods for REPAIR
In the main paper, we presented REPAIR's 's performance using the win-loss rate as the ranking estimation method. this section, we conduct additional experiments to compare win-loss rate with alternative ranking methods.
20

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
on the down- sampled REPAIR-ed Summarize from Feedback dataset, matched size to the Perturbed data. Despite information loss from the down- sampling, all three consistency metries show steady improvements over the perturbed dataset.
Table 6: Instruction-tuning of
Table 7: Instruction-tuning of Llama-3-Instruett 8B on the REPAIR-ed Summarize from Feed- back dataset using various rank estimation meth- ods. All methods integrate well with the REPAIR framework and yield notable improvements.
Summarize from Feedback
Summarize from Feedback
Models
Models
Stran (5)
H.
Stran (5)
8pmmm
8wng
Zero-shot inference
62.9
84.7
68.2
Zero-shot inference
62.9
84.7
68.2
64.0
Perturbed data
71.3
92.4
79.6
Perturbed data
71.3
92.4
79.6
60.9
REPAIR-Ca data W-L
70.5
97.5
89.4
REPAIR-ed data (sampled)
69.1
96.3
88.9
61.7
REPAIR-ce data ELO
71.4
97.7
89.7
REPAIR-ed data + Neg. (sampled)
68.8
96.2
86.2
84.5
REPAIR-ed data B-T
71.2
98.6
91.0
We evaluate two widely used ranking methods: the Elo rating system (Elo & Sloan, 1978) and the Bradley-Terry (B-T) model (Bradley & Terry, 1952). Table 7 reports the results from these methods, maintaining consistency with the experimental setup in Table (Section 4). Our findings indicate that both Elo and the B-T model achieve slightly better performance than the win-loss rate, improving human accuracy and consistency metrics. However, the performance differences remain relatively small.
To further analyze the strengths and weaknesses ranking estimation methods, we summarize their characteristics below:
Win-loss rate: This method does not incorporate transitive information, treating all pairwise comparisons independently. While this may appear to be limitation, it has certain advantages:
Pros: Consider the pairwise comparisons {A > B, A > Cj. The win-loss rate does not infer a relationship Later, we show how Elo and B-T fail such cases.
between B and C, correctly predicting >
-Cons: In transitive scenario {A > B. B > C, C > D}, an ideal ranking should yield A > B > C> > D. However, the win-loss rate produces A > B = C > D, as it does not consider indirect comparisons.
This method effectively incorporates transitive information but is highly sensitive to the order of com- parisons. When pairwise comparisons sparse, this sensitivity can significantly impact ranking estimation. For example:
Elo rating:
- Given (A - A, > C}, Elo predicts A > C B.
If the order is reversed to {A C, A B}. Elo instead predicts A B C.
Since our dataset is static and comparison order carries no inherent meaning, Elo's sensitivity may introduce noise into the augmented preference data.
This probabilistic model estimates rankings using maximum likelihood, effectively handling transitive relationships and generating nuanced rankings. However, it has limitations:
Bradley-Teree model:
It struggles to produce tied rankings without explicit design modifications. For instance, given (A > B, A > C}. the B-T model predicts A > ~ C, where B and C receive slightly different scores, even if they should theoretically tie.
This forced ranking structure may introduce noise in scenarios where ties should be preserved.
Overall, the effectiveness of each ranking estimation method depends on factors such pairwise comparison sparsity and computational constraints:
Sparse comparisons: Methods relying on transitive relationships, such as Elo and B-T, may struggle when comparison data is limited.
Computational efficiency: Simpler methods like the win-loss rate are advantageous when computational resources or time are constrained.
21

---

Aligning with Logic: Mcasuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
These observations align with the axiomatic assumptions of the Von Neumann-Morgenstem utility theorem (?), which requires that preferences be complete- that is, preference judgment must exist between every pair of items-to aggregate preferences into consistent linear ranking. When comparison annotations are sparse, this completeness assumption is violated, leading to degradation in the performance of ranking estimation methods.
In summary, while Elo and the B-T model are effective in capturing transitive relationships and generating likelihood-based rankings, their sensitivity to noise and computational demands can be limiting certain contexts. In contrast, the win-loss rate offers a simple and robust baseline, especially when tied rankings are acceptable or when efficiency is a key concern. Additionally, extensions such TrueSkill build upon Elo and B-T by explicitly modeling ties and uncertainty, potentially mitigating some of these limitations.
22


Reviews:
[{"review_id": "LXbfd2hXx6", "reviewer": "Reviewer_sp8E", "summary": "The main topics of this paper are (1) introducing consistency metrics for several logic/preference orders of LLMs, (2) evaluation of a large number of models and datasets, and (3) impact on downstream applications.\nSpecifically, the metrics measure transitivity, commutativity, and negation invariance in LLMs based on in-context learning, as shown in Figures in the Appendix, such as Figure 7. The proposed data augmentation scheme, REPAIR, also enhances the consistency behavior.\n\n**Post rebuttal**\nI checked the paper again and read reviews from other reviewers.\nOne reviewer had a different evaluation (1) than others (4/3/3). \nIn short, two reasons for rejection were (1) related to the citation of one paper and (2) essentially, asking for more experiments.\nI wanted to clarify how serious these issues are, but I couldn't find substantial evidence supporting such arguments.\nTherefore, I will maintain my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's feedback and acknowledgment. We are grateful for the time and effort dedicated to reviewing our work.\n\nBest,\nAuthors", "claims_and_evidence": "The main claim is that given LLM preference orders under some relations, the proposed method measures the transitivity, commutativity, and negation consistency over the pairs of the statement.  The formulation is convincing.\n\nThe evaluation of LLMs shows the scores from various models over different datasets.\nThe outcome in Table 1 is also convincing.\n\nSome interesting hypotheses in Section 3.3, such as transitivity as a proxy for self-agreement, commutativity correlates with human preferences, are also provided with the evidence in Figure 4 and 5.", "methods_and_evaluation_criteria": "Yes, the definition of metrics, selection of the models, and datasets make sense.", "theoretical_claims": "I believe there is no theoretical claim in this paper.", "experimental_designs_or_analyses": "Overall, the experiment is well designed, and the appendix supports its validity and soundness.", "relation_to_broader_scientific_literature": "This empirical result might impact the consistency behavior around LLMs. The observation made in this paper is consistent with the earlier findings in smaller LMs and matches a new trend over very large LMs.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The main strength is the empirical work that supports various claims and reproducible details in the paper.", "other_comments_or_suggestions": "NA", "questions_for_authors": "I don't have any further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BzVPPd5Uq5", "reviewer": "Reviewer_3jpj", "summary": "This paper introduces a universal framework for evaluating logical preference consistency in LLMs, focusing on three properties: transitivity, commutativity, and negation invariance. The authors propose quantitative metrics for these properties, conduct comprehensive empirical analyses across state-of-the-art LLMs, and introduce REPAIR, a method for refining and augmenting data to improve logical consistency. Experimental results indicate enhanced consistency correlates with better model reliability and improved performance in logic-dependent tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's thoughtful feedback and acknowledgment. Below, we provide our clarifications and explanations.\n\n- Extending the Experiment and Method to Less Structured Datasets\n\n  We focus on analyzing logical preference consistency, and by '**universal** framework' we mean that our proposed method applies across **all domains with multiple rankable items**. As stated in Line 218, our dataset selection is guided by **subjectiveness** considerations: SummEval (evaluating summaries) represents a more subjective task, whereas CaTeRS (comparing temporal and causal order) is a more objective one. We believe covering this spectrum of subjectivity ensures broad applicability to real-world tasks. However, if the term \"universal framework\" seems overstated, we are open to refining it to \"improve in-domain consistency.\"\n \n---\n\n- CoT Analysis\n\n  Thank you for raising this point. Our findings show that evaluation with **CoT reasoning enhances accuracy but does not necessarily improve consistency**. While we recognize that systematically analyzing CoT’s impact on LLM judgments is a valuable research direction, conducting an exhaustive set of experiments is beyond the scope of this paper. Our primary focus is on **quantifying consistency, assessing its impact, and proposing methods for improvement**.\n\n---\n\n- REPAIR Method: Ablation Study on Components\n\n  We appreciate the reviewer's suggestion. The REPAIR method consists of two key steps:\n  1) **Rank estimation** from noisy and sparse human annotations.\n  2) **Preference augmentation** based on the estimated rankings.\n\n  In Appendix M and N, we conduct two ablation studies: \n  - **Evaluating different rank estimation methods.**\n  - **Assessing the performance impact of using the same amount of augmented data.**   \n\n  To the best of our knowledge, these cover all potential ablation studies relevant to the REPAIR method.\n\n---\n\n- REPAIR Method: Use of Human Judgments in Data Augmentation and Computational Overhead.  \n  As illustrated in Figure 6, **human judgment annotations are solely used in the first step of REPAIR, where a ranking is estimated from the noisy human annotations**. We will improve the visualization and explanation to make this clearer. Regarding computational overhead, since REPAIR is a data augmentation method, its **additional computational cost is directly proportional to the size of the augmented dataset**. Table 2 (# data, Avg Comp/Inst) details the amount of data generated by REPAIR and its computational implications.\n\n---\n\n- Sensitivity of Consistency Metric to Annotation Quality\n\n  Our consistency metrics—transitivity, commutativity, and negation invariance—**do not rely on human annotations**. Instead, they are derived solely from the model’s own judgments, making them independent of annotation quality.\n\n---\n\n- Trade-offs Between Consistency and Creativity in Model Outputs.  \n\n  This is an excellent question. We believe that consistency and creativity are distinct dimensions of LLM behavior, each desirable in different applications. For instance, when simulating human behavior, perfect logical consistency may not be necessary for all queries. Therefore, **we do not advocate for universal consistency improvements across all domains. Instead, we propose that when logical consistency is a critical factor for an LLM’s intended functionality**, e.g. used as an logical operator in a logic-dependent high-level algorithm, our method provides a systematic way to quantify and enhance this characteristic.\n\n---", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper improves reliability of LLMs by addressing logical consistency.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. Clearly defined formulations and rigorous metrics for evaluating logical consistency.\n2. Extensive experimentation across diverse LLMs and multiple tasks validates the utility of proposed metrics and methods.\n3. REPAIR method effectively improves consistency without sacrificing alignment with human preferences. Solid empirical analysis showing strong correlations between proposed consistency metrics and model reliability.\n\nWeaknesses:\n1. Evaluation is limited to datasets with relatively well-structured logical tasks; extending evaluation to more complex or noisy real-world applications could strengthen claims of universality.\n2. The observed negative effects of Chain-of-Thought prompting on logical consistency are intriguing but insufficiently explored; deeper analysis or theoretical insight would be valuable.", "other_comments_or_suggestions": "1. The paper lacks a detailed ablation study on components of the REPAIR method.\n2. Clarify more explicitly how human judgments were utilized in the data augmentation step.\n3. It would be useful to clarify the computational overhead introduced by the REPAIR method.", "questions_for_authors": "1. Have you explored the effectiveness of the proposed method on larger-scale and less structured datasets (e.g., open-ended reasoning tasks)?\n2. How sensitive is your consistency metric to variations in annotation quality?\n3. Could you discuss potential trade-offs in consistency versus expressivity or creativity of the model outputs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QcBTJeJtEg", "reviewer": "Reviewer_hAwv", "summary": "This paper investigates whether large language models (LLMs) make pairwise preference judgments without logical contradictions, focusing on three consistency properties: transitivity (A > B > C implies A > C), commutativity (choices remain the same regardless of order/phrasing), and negation invariance (consistent answers when preferences are reversed or negated). The authors propose metrics for each property and show that state-of-the-art LLMs often violate them, despite aligning well with human judgments. To address this, they introduce REPAIR, which aggregates noisy preference data into a self-consistent global ranking and then adds logically implied or negated comparisons for fine-tuning. REPAIR raises consistency scores without jeopardising human alignment and boosts performance in downstream LLM-as-judge tasks.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's thoughtful feedback. Below, we provide our clarifications and explanations.\n\n---\n\n- Overstatment on \"universal framework\":\n\n  We agree that our work focuses on the logical preference consistency of pairwise preference-ranking scenarios. However, we would like to clarify that **by \"universal framework\" we mean that our proposed method applies across all domains with multiple rankable items**, unlike previous works that primarily focus on tasks such as NLI and factual knowledge consistency.  \n  If the reviewer still finds \"universal framework\" to be an overstatement, we are happy to revise or remove the term accordingly.\n\n---\n\n- Concerns about the Win-loss rate rank estimation method\n\n  We acknowledge the potential limitations of using the win-loss rate as a rank estimation method. However, we would like to refer to our **ablation study in Appendix N** (referenced in the main paper, Line 326), where we provide **detailed analysis of the advantages and disadvantages of two alternative rank estimation methods, ELO rating and Bradley-Terry model**. Additionally, we compare REPAIR's performance across different rank estimation methods. We believe this ablation study sufficiently addresses the concerns raised. However, if further clarification is needed, we would be happy to expand on this discussion.\n\n---\n\n- Concerns about experimental design\n\n  The use of different datasets for the REPAIR method (Summary from Feedback and MS MARCO, Appendix L) and for the quantification method (SummEval, NovelEval, and CaTeRS) is intentional. The latter datasets are designed as evaluation benchmarks and are too small to be split for training.\n\n  For details on dataset sizes, please refer to Appendix B.\n\n---\n\n- Concerns about Generalization:\n\n  We believe the reviewer is questioning whether the learned consistency can generalize across different domains. We would like to clarify that **we do not claim that the REPAIR method improves consistency across domains**. While we agree that cross-domain generalization is an important future direction, it is beyond the scope of this paper.\n\n  Systematically justifying such a claim is challenging due to the multiple factors influencing generalization performance, such as the specific preference learning method (e.g., DPO or PPO) and the domain gap between training and target tasks. For example, training on summaries with feedback may improve SummEval but not necessarily document reranking tasks.\n\n  If the reviewer is concerned about potential overstatement, we are open to revising our claim to specify \"improving in-domain logical consistency.\"\n\n---\n\n- Comparison to previous work PairS[1].\n\n  We would like to clarify that **our work and PairS address completely different research problems and are not directly comparable**. PairS focuses on ranking estimation via search-based methods, whereas our paper aims to **quantify and improve the logical preference consistency of LLMs**.\n\n  The only connection to PairS appears in Section 5, where we use it to illustrate **how logical preference consistency impacts real-world logic-dependent algorithms**.\n\n[1] Liu, et al. \"Aligning with human judgement: The role of pairwise preference in large language model evaluators.\" COLM (2024).\n\n  \n---\n\n- Error in Figure 6\n\n  We thank the reviewer for spotting this typo. We will revising it accordingly.\n\n---\n\nWe appreciate the reviewer's feedback and hope our clarifications address their concerns. We are open to revising our claims and further refining our discussion if needed.", "claims_and_evidence": "The paper makes several claims supported by empirical evidence to certain extent:  \n- Logical preference consistency (as defined by transitivity, commutativity, and negation invariance) is crucial for reliable LLM decision-making, yet current models often exhibit inconsistencies.  \n- The proposed consistency metrics serve as indicators of model robustness and alignment, showing strong correlations with human preference agreement rates.  \n- The REPAIR method effectively improves an LLM’s logical consistency without compromising alignment with human preferences.  \n- Models with higher logical consistency perform better in downstream tasks that require iterative judgments, such as ranking algorithms.  \n\nHowever, the paper overstates its contributions by describing the evaluation framework as \"universal\". In reality, the study is specific to pairwise preference-ranking scenarios, and its applicability to broader LLM tasks remains unverified. Also, the experiments do not sufficiently support the claims (see one critical aspect in Experimental Designs Or Analyses).", "methods_and_evaluation_criteria": "The methodology introduced in this paper is mostly appropriate for the stated problem. The authors identify transitivity, commutativity, and negation invariance as key measurable aspects of logical consistency in pairwise judgments, and they formalise metrics for each. These choices are sensible: transitivity directly addresses the avoidance of preference cycles, commutativity tests that the order or wording of input does not flip the outcome, and negation invariance ensures a model handles logically equivalent inquiries consistently. \n\nHowever, one particular thing remains arguable is the use of win-loss rate in its REPAIR algorithm:\n- Win-loss treats all preferences equally, whereas in real-world data, some preferences may be stronger or more certain than others. Methods that incorporate confidence scores or margins of preference may produce more reliable rankings.\n- If a model produces inconsistent rankings (e.g., due to randomness or bias), win-loss does not provide an effective way to smooth or account for contradictions.\n- Win-loss does not estimate a likelihood of preference correctness but rather a raw win ratio, which lacks formal statistical grounding.", "theoretical_claims": "This work is largely empirical. One nuanced theoretical claim is its choice of sampling sub-graphs for transitivity check.", "experimental_designs_or_analyses": "An critical issue lies in the experiment design is that the authors conduct their logical consistency evaluation on three datasets (SummEval, NovelEval, CaTeRS) but then demonstrate the effectiveness of their REPAIR method on a different dataset (Summarize From Feedback). This raises several concerns and questions:\n\n- Did the fine-tuned model show improved logical consistency when re-evaluated on SummEval, NovelEval, and CaTeRS?\n- If so, why were those results not reported?\n- If not, does this suggest that REPAIR’s improvements are dataset-specific or do not generalise across tasks?", "relation_to_broader_scientific_literature": "As illustarted, this is largely within the scope of preference ranking in LLMs.", "essential_references_not_discussed": "This submission largely builds upon [1], which is cited. However, it does not sufficiently discuss how it improves over that prior work, leaving unclear whether its contributions are meaningful advancements or merely incremental refinements.\n\n[1] Liu, et al. \"Aligning with human judgement: The role of pairwise preference in large language model evaluators.\" COLM (2024).", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "- There appears to be a contradiction in Figure 6’s right-most subfigure, where the preference matrix entry $(X_2, X_4)$ is labeled \"A\" (indicating $X_2 > X_4$), but the side prompt states $X_2 < X_4$ (indicating the opposite relationship).", "questions_for_authors": "My main questions and concerns have been discussed above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2YqiNFGfMm", "reviewer": "Reviewer_paNc", "summary": "The paper introduces a novel framework for enhancing the logical consistency of large language models (LLMs). The authors propose a universal evaluation framework based on three key properties: transitivity, commutativity, and negation invariance. They also introduce REPAIR, a data refinement and augmentation method that improves logical consistency while maintaining alignment with human preferences. The study demonstrates significant improvements in model performance across various tasks, including abstractive summarization evaluation, document reranking, and temporal event ordering. The main findings include substantial enhancements in logical consistency metrics and better performance in logic-dependent algorithms.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's thoughtful feedback and acknowledgment. Below, we provide our clarifications and explanations.\n\n---\n\n- Potential Limitations of the proposed methods:\n\n  In the appendix, we analyze the limitations of different rank estimation methods. Another potential limitation is that the proposed REPAIR method may not be effectively applied to preference data when there are only two responses per query (i.e., when the data is too sparse). In such cases, the rank estimation method cannot reliably distinguish between true preference and noisy preference.\n\n---\n\n- Future direction:\n\n    A promising future direction is to investigate whether the improved logical consistency achieved by REPAIR can be generalized. For example, if we augment a general-purpose query-response preference dataset using REPAIR, would training on this augmented dataset improve logical consistency in other domains? To what extent? While we believe these are valuable research directions, they require more exhaustive investigations, which are beyond the scope of this work.\n\n---\n\n- Additional visualizations:\n\n  We will make every effort to improve the clarity of the REPAIR-related figures to enhance their readability and comprehension.\n\n---\n\n- Code and dataset release:\n\n    We confirm that we will release all code and datasets to facilitate reproducibility and further research.\n\n---\n\n- Comparison to adversarial training (Minervini & Riedel, 2018)\n\n  We appreciate the reviewer for pointing out this relevant work. The primary differences between Minervini & Riedel (2018) and our REPAIR method are as follows:\n\n  - The previous work focuses solely on the Natural Language Inference (NLI) task, where a well-defined logical relationship of entailment exists. In contrast, our framework is more general and applicable to any rankable items, including entailment relations, causal relations, temporal orders, and preference orders.\n\n  - The prior work mitigates inconsistency by adding regularization to the loss function, whereas REPAIR focuses on a data augmentation approach to address inconsistency. As discussed in the Related Work section and Appendix D, other data-centric methods exist, but they are often restricted to NLI relations or factual knowledge consistency, which limits their applicability to broader domains.\n\n---\n\nWe will revise the manuscript according to the reviewer's suggestions and hope we have addressed all the concerns.", "claims_and_evidence": "The claims made in the submission are well-supported by clear and convincing evidence. The authors provide extensive experimental results across multiple datasets (SummEval, NovelEval, CaTeRS) to demonstrate the effectiveness of their proposed framework. The metrics used (transitivity, commutativity, negation invariance) are clearly defined and appropriately applied to evaluate logical consistency. The results show significant improvements in logical consistency and performance, which validate the claims made in the paper.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for the problem at hand. The authors use a combination of ranking estimation and data augmentation (REPAIR) to enhance logical consistency. The evaluation criteria, including transitivity, commutativity, and negation invariance, are relevant and effectively measure the logical coherence of LLMs. The use of benchmark datasets (SummEval, NovelEval, CaTeRS) makes sense for evaluating the proposed methods.", "theoretical_claims": "The paper does not present any formal theoretical proofs. However, the conceptual framework and the proposed methods are logically sound. The authors provide a clear rationale for the choice of logical consistency properties (transitivity, commutativity, negation invariance) and their application to LLMs. No specific theoretical claims require further validation.", "experimental_designs_or_analyses": "The experimental designs and analyses are valid. The authors conduct extensive experiments across multiple datasets to evaluate the effectiveness of REPAIR. The results are presented in a clear and structured manner, with detailed comparisons to baseline methods. The experimental setup is appropriate for the tasks considered, and the results support the claims made in the paper.", "relation_to_broader_scientific_literature": "The key contributions of the paper are well-aligned with the broader scientific literature on improving logical consistency in neural models. The paper builds on prior work in natural language processing (NLP) and machine learning (ML) that focuses on enhancing model reliability and coherence. Specifically, the paper relates to works such as:\n\nAdversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge (Minervini & Riedel, 2018): This work investigates the problem of generating adversarial examples to improve logical consistency in NLI models.\n\nLogic-Guided Data Augmentation and Regularization for Consistent Question Answering (Asai & Hajishirzi, 2020): This study integrates logical rules with neural models to improve consistency in QA tasks.", "essential_references_not_discussed": "To my knowledge, I do not find any other related work essential to understanding the key contributions of the paper, but not currently cited/discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper presents a novel and comprehensive framework for evaluating and improving logical consistency in LLMs.\n\nThe proposed REPAIR method is innovative and shows significant improvements in logical consistency and performance.\nThe experimental results are robust and validate the effectiveness of the proposed methods.\n\nWeaknesses:\n\nThe paper could be strengthened by exploring more potential limitations of the proposed methods, such as their applicability to other types of models or tasks.", "other_comments_or_suggestions": "The paper could benefit from additional visualizations or examples to illustrate the effectiveness of the REPAIR method.\n\nThe authors may ensure that the code and refined datasets are publicly available to facilitate reproducibility.", "questions_for_authors": "Limitations and Future Work: What are the potential limitations of the proposed methods, and what are the key areas for future research? How do the authors plan to address these limitations?\n\nComparison to Adversarial Training: How does REPAIR compare to adversarial training methods (e.g., Minervini & Riedel, 2018) in terms of improving logical consistency? Are there any specific advantages or disadvantages?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper provides an important, interesting, and timely evaluation of preference consistency for LLMs given the prevalence of LLM-based WinRate comparison-based evaluation in a very broad range of experimental settings.  The paper further goes on to provide a repair method to improve preference consistency.  Appendices provide many additional interesting evaluations and results to augment the primary contributions of the paper.  

Three out of four reviewers vote for acceptance, while one reviewer raises experimental concerns and recommends rejection.  I understand and agree with the concerns of that reviewer (as discussed further below), but having read the paper, I don't think this experimental issue alone is sufficient to reject the paper in light of it's novelty, insights, and otherwise expansive and diverse experimental evaluation.  

The authors are strongly encouraged to take reviewer feedback into account to improve clarity and explanation when revising the paper.  I highlight one particular point of concern I share with a reviewer along with three additional points I would like to raise that can hopefully improve and enhance the paper presentation:

1. One reviewer and I retain a concern that there appears to be an unjustified shift (in the present paper) between the datasets used in Section 3 and Section 4.  The authors provide a response regarding this in their rebuttal.  The reviewer who initially raised this issue is not convinced, nonetheless I suggest that the authors include their justification on revision so they can make their experimental rationale clear.  

   To further elaborate on concerns about this experimental shift, I don't understand why three datasets and many LLMs are presented initially but the proposed method is only applied to one dataset ("Summarize from Feedback") with only one LLM, and even then (line 365) they sample 200 test instances randomly, which they do not report doing for Table 1.  

2. In Table 3 the authors test a (single) Llama-3 Model fine tuned on the "Summarize from Feedback" dataset (it is not clear if they are using a dedicated train set, which should be clarified) which has been augmented in three ways: (1) perturbed by corrupting 10% of labels, (2) REPAIRed, and (3) REPAIRED with negative examples, but they do not report the results of a tuning the Llama model on the original training data, which is an important comparison and unexplained omission. The comparison between their methods (2), (3), a zero-shot model and (1) which is a corrupted dataset does not seem like a fair or informative comparison.  More explanation is needed to justify this experiment and its conclusions.

3. Seemingly out of the blue, results on MS-Marco reranking appear in the Appendix in what appears to be a reranking task, but it is not clear to me where the initial list being reranked comes from (BM25?). The authors somehow apply their REPAIR method to augment MS-Marco labels, but labels in MS MARCO are not pairwise comparisons or preference orderings on passages/documents, they are pointwise relevance judgements, so it is extremely unclear what the authors are doing here.  This requires clarification.

4. I remark that when converting preferences to a score-based total order ranking (e.g., ELO, Bradley-Terry) seems to implicitly require the [Von Neumann–Morgenstern utility theorem](https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem) to verify that such a score-based utility exists along with it's very strong axiomatic assumptions on preferences, i.e., [completeness, transitivity, continuity, and independence](https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem#The_axioms) (or variations thereof).  I think this theory strongly informs the results in Appendix N since the authors claim "Sparse comparisons: Methods relying on transitive relationships, such as Elo and B-T, may struggle when comparison data is limited".  If the preferences are sparse, this violates the completeness requirements to identify a scoring-function and could very clearly explain why ELO and Bradley-Terry do poorly in this setting.  I remark that extensions such as [TrueSkill](https://proceedings.neurips.cc/paper_files/paper/2006/file/f44ee263952e65b3610b8ba51229d1f9-Paper.pdf) added "indifference" (tied rankings) to ELO and this may mitigate concerns that ELO and Bradley-Terry do not perform well with tied rankings.
Decision: Accept (spotlight poster)