Logical Consistency of Large Language Models in Fact-Checking
Authors: Bishwamittra Ghosh
Sarah Hasan
Naheed Anjum Arafat
Arijit Khan
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
Logical consistency
Fact-checking in Knowledge graphs
Primary area: foundation or frontier models, including LLMs
TL;DR: Assessing and improving the logical consistency of large language models on fact-checking queries in the knowledge graph.

Abstract: In recent years, large language models (LLMs) have demonstrated significant success in performing varied natural language tasks such as language translation, question-answering, summarizing, fact-checking, etc. Despite LLMs’ impressive ability to generate human-like texts, LLMs are infamous for their inconsistent responses – a meaning-preserving change in the input query results in an inconsistent response and attributes to vulnerabilities of LLMs such as hallucination. Consequently, existing research focuses on simple paraphrasing-based consistency assessment of LLMs, and ignores complex queries that necessitate an even better understanding of logical reasoning by an LLM. *Our work therefore addresses the logical inconsistency of LLMs under complex logical queries with primitive logical operators, e.g., negation, conjunction, and disjunction.* As a test bed, we consider retrieval-augmented LLMs on a fact-checking task involving propositional logic queries from knowledge graphs (KGs). Our contributions are three-fold. **Benchmark:** We introduce three logical fact-checking datasets over KGs for community development towards logically consistent LLMs. **Assessment:** We propose consistency measures of LLMs on propositional logic queries and demonstrate that existing LLMs lack logical consistency, especially on complex queries. **Improvement:** We employ supervised fine-tuning to improve the logical consistency of LLMs on the complex fact-checking task with KG contexts. We have made our source code and benchmarks available.

Published as a conference paper at ICLR 2025
LOGICAL CONSISTENCY OF LARGE LANGUAGE MODELS IN
Bishwamittra Ghosh *, Sarah Hasan *, Naheed Anjum Arafat', Arijit Khan?
Maa Planck Institute for Software Systems, Germany
2 FAllborg University, Denmark
"Independent Researcher, USA
"bghoshhmpi-swe. org. "(sarahh, arijick)ecs. aau.ck, "naheed.anjumeu nus. edu
ABSTRACT
recent years, large language models (LLMs) have demonstrated significant suc- cess in performing varied natural language tasks such as language translation, question-anse answering, summarizing, fact-checking, etc. Despite LLMs' impressive ability generate human-like texts, LLMs are infamous for their inconsistent re- sponses - a prreernng change in the input query results in an inconsis- tent response and attributes vulnerabilities of LLMs such as hallucination. Con- sequently, existing research focuses on simple paraphrasing- bbsed consistency as- sessment of LLMs, and ignores complex queries that necessitate an even better understanding of logical reasoning by an LLM. Our work therefore addresses the logical inconsistency of LLMs under complex logical queries with primitive log- icul operators, e.g. negation, conjunction, and disjunction. As test bed, we consider retrieval-augmentedd L.LMs on fact-checking task involving proposi- three- tional logic queries from knowledge graphs (KGs). Our contributions fold. Benchmark: We introduce three logical fact-checking datasets over KGs for community development towards logically consistent LLMs. Assessment: We propose consistency measures LLMs on propositional logic queries and demonstrate that existing LLMs lack logical consistency, especially on complex queries. Improvement: We employ supervised fine-tuning to improve the logical consistency of LLMs the complex fact-checking task with KG contexts. We have made our source code and benchmarks available'.
INTRODUCTION
Large language models (LLMs), e.g., ChatGPT (OpenAI, 2022), PaLM (Chowdhery et al., 2023), LLaMA (Touvron al., 2023) permeate natural language processing with their remarkable capacity to comprehend and generate human-like texts (Min et al., 2023; Kamalloo et al., 2023). As such, there a host of applications of LLMs in high-stake domains such as healthcare (Singhal et al., 2023; Wang et al., 2023b; Dash et al., 2023), finance (Wu et al., 2023; Yang et al., 2023a), law (Xiao et al., 2021), and education (Kasneci et al., 2023). For instance, by implicitly storing pre-trained/fin- tuned knowledge, LLMs are increasingly being applied in fact-checking and question-answeriig tasks (Saeed al., 2023; Quelle & Bovet, 2024).
Consistency of LLM Response. Despite the success story, notable challenge in LLMS is the inconsistency of generated responses. Consistency of an I.M the invariance of output under mocaing--resservee alternations in its input is a highly desirable property (Elazar et al., 2021; Liu et al., 2023c). Existing works mostly focus on semantically similar sentences (paraphrases) for consistency assessment. For example, Kuhn al. (2023) measures the consistency of LLMs by computing semantic uncertainty of output responses under paraphrased inputs. Elazar et al. (2021) consider quasi- parapihaappssssss in cloze-style input format and assess masked language models (Devlin et al., 2018) for consistency (additional related works are discussed in the Appendix B).
*Jintly first authors. 'ttpps:/ / /ig thub. com/bi shwami ttrat 1m mm_logica logical,_.. ca__oon conos stency

---

Published as a conference paper at ICLR 2025
Knowledge graph (relevant sub-graph)
T.S.
Aidous
Fact
Eliot
vieaa
Nobel Price + voviiee
Husley
Thomas Mann award winner Nobel Prize Literature
Find relevant context
Literature
Tolstoy
snard Slaac
intomeell
Hemi Bergion
Kafka
Franz
iclewoed
Thomtas
Mana
T.S. Elitt award winnrr Nohel Prize in Litezaure Thomas Mann inffuenced
Consider the context: SConlext.
Frauz Kafka
Thomas Mann award winner Nobel prize in Literature
Ts the fact correct given the context? &Fact.
LLMQuery (Prompt)
Context
Our LLM- based fact-checking framework on simple fact with context from a knowledge graph. A representative LLMQuery is in Figure 3, and the extension to complex facts is in Figure 4.
Figure 1:
Existing works focus on the consistency of an LLM on simple queries and do not assess consistency on complex queries requiring improved logical reasoning by LLMss. Therefore, an unexplored re- search question is to systematically assess and improve the logical consistency of LLMSs complex logic queries with primitive logical operators: negation (-), conjunction (^), and disjunction (v). Informally, an LLM is logically consistent with a query under the negation operator if the LLM re- sponse to the base query and the negated query are semantically opposite. Subsequently, an LLM is consistent on a conjunction query with (at least) two sub-queries the individual responses to each sub- uub-queries are logically consistent with the conjunction query. Building on logical semantics, naturally extend the consistency assessment of LLMs to propositional logic queries and rules such as commutative, distributive, associative, syllogism, and to first-order logic queries (Boole, 1847). Our hypothesis is that logically consistent LLMs are essential in trustworthy systems (Sun et al., 2024b): When an LLM consistent, easier to verify whether the LLM right or wrong, without sophisticated benchmark being logically consistent provides the end users additional confidence in applying LLMs in their own workload.
Logical Consistency in Fact-Checking via Retrieval-Augmented Generation. In this paper, we assess and improve the logical consistency of LLMs in propositional logic queries and logic rules, particularly in the context of fact-checking. Automated fact-checking is the task of deploying an intelligent system such as an L.MM to verify a queried fact with respect to baseline knowledge (Guo et al., 2022). Since LLMs might be inconsistent unknown facts, we consider a controlled setup of LLMs in retrieval-augmenter generation (RAG) that references an authoritative knowledge base to an LLM beyond training data while generating response (Lewis al., 2020). As a RAG context, we resort to real-world knowledge graphs (KGs) (Agrawal et al., 2023; Yang et al., 2023b; Suchanek & Luu, 2023; Huang al., 2024b), which store large-scale facts in a structured format of (subject, relation, object) triples. Indeed, a KG is a rich source of highly-curated facts on entities connected by relationships, and provides test bed for logical consistency assessment. Therefore in our setup, we aim to provide KG context and a propositional fact-checking query to an LLM and assess the consistency of responses under logical manipulation. However, such a logical fact-checking dataset is absent in the literature.
Contributions: We have three- fold contributions in the paper.
1. Benchmark. We address the lack of a logical fact-checking dataset (LFC) over KGs by introduc- ing three datasets: FreebaseLFC,, NELLLFC, and WikiLFC ($5). They are derived from their KG counterparts, Freebase (Bordes et al., 2013), NELL (Carlson et al., 2010), 0), and WikiKG90Mv2 (Hu al., 2021b), respectively. Existing KG datasets, while being widely used, are not suitable for testing LLM's logical consistency in their native format. Therefore, we transform (subject, relation, object) triplet from the KG into a suitable input format for an LLM as a (Fact, Context) pair, referred to LLMQuery - Figure shows example of fact, its corresponding KG context, and the LLMQuery from the FFe dataset.
2. Assessment of Logical Consistency. We propose logical consistency measures of LLMs on propositional basic queries, which cover fact-checking queries over KGs using retrieval- augmented L.MSs ($3). The queried facts involve one more triplets connected by logical operators
2

---

Published as a conference paper at ICLR 2025
such negation (-), conjunction (A), and disjunction (v). For consistency assessment, sider a variety of scenarios, such as both simple and complex facts, diverse logic rules, first-order logic, and LLMQuery with and without KG contexts ($5). We demonstrate that existing LLMs such as Llama2-7B, Llama2-13B, and Gemma-2B, while being more accurate with KG contexts, are not consistent when considering logical equivalents or variants of these facts.
3. Improvement of Logical Consistency. We demonstrate how existing LLMs can be fine-tuned to improve their consistency via supervised fine-tuning (Radford et al., 2019; Devlin al., 2018) (84). We showcase that pre-train, prompl, and predict paradigm is often insufficient to improve the con- sistency of LLMs for complex fact-checking with KG contexts, thus we resort to pre-train, fine-tune, and predict approach. Although supervised fine-tuning is shown to be effective in other downstream tasks (Wei et al., 2021; Li Liang. 2021), it has not yet been conducted for logical fact-checking queries over KGs. Furthermore, fine-tuning is feasible in our case due to the availability of abundant training data with ground-truth labels from the KG ($4.2). Additionally, we leverage parameter- efficient fine-tuning based on QLoRA (Quantized Low-Rank Adaptation) (Dettmers et al., 2024), thus improving the fine-tuning efficiency.
Experimental results show that the fine-tuned LLMs improve the consistency by 14% on average, while our optimization techniques scale both fine-tuning and inferencing to large KGs and complex fact-checking tasks. Our experiments also show that models fine-tuned on simple facts generalize well to more complex, unseen facts and rules ($5). This is possible due to complex facts and rules being decomposable into collection of simple facts (Proposition 1)2.
BACKGROUND
We provide preliminaries on knowledge graphs and propositional logic, which constitute the back- ground for assessing logical consistency in LLMs.
Knowledge Graph (KG). A knowledge graph stores large-scale and real-world facts as graph, denoted by G = (V. R), where V is an entity and r E R is relation. r is binary function r V.V. V {0, 13 indicating whether the relation holds between pair of entities or not. In the KG, such binary relation indicates a directed edge between the pair of entities, formally u I. v indicates r(u, u) 1. A KG G can be represented by set of facts or triplets T s V x R V: For a triplet T (u, r, v) E T with u, U E V, we refer to u as a subject entity and v as an object entity.
Propositional Logic Facts. At the of a KG, propositional logic facts constitute one multiple triplets connected by logical operators, including conjunction (^), disjunction (v), and negation (-). As per the DNF theorem (Davey & Priestley, 2002; Halbeisen & Krapf, 2020), any proposi- tional logic fact q can be expressed in the disjunctive normal form (DNF), which a disjunction of conjunctive facts, wherein each conjunctive fact is conjunction of atomic relation facts. Formally,
q = Ci V C2 V V. Gn where C; Ci Gig A
[c/l1 < i < n) are conjunction facts. A conjunction fact comprises one or more atomic relation facts = A Am eig. An atomic relation fact Cij is a relation projection between a pair of entities. Formally,
Eg r(u, v) or eiy =r(u,e)
where u E V is subject entity, v E V is an object entity, and e R is a relation. We refer to an atomic relation fact as the simple fact, while a propositional logic fact involving multiple triplets connected by logical operators is called a complex fact.
"W analyze the impact of considering KG contexts (Appendix G.2), context length (Appendix G.3), dif- ferent KG retrieval methods to build the KG context in LLMQuery (Appendix G.4), different prompting strate- gies (Appendix G.5), and the impact of learning rate (Appendix G.6) and epachs (Appendix G.7) on logical consistency and accuracy in fact-checking. We discuss the generalization of logical consistency to complex queries (Appendix H). We extend consistency assessment beyond structured format natural text queries in Appendix Also, we demonstrate the superiority of our faet-checking framework over existing fact-checker in Appendix J. We experiment with Llama2- -0BB and GPT-40, where instruction prompting tends to be suf- ficient to improve logical consistency since fine-tuning is often infeasible in large and closed-son source models (Appendix K). Finally, we empirically verify whether L.LMs fine-tuned to be logically consistent retain their performance on general language benchmarks in Appendix I..
3

---

Published as a conference paper at ICLR 2025
Given a propositional logic fact 9, let denote its truth value T(q) {0, 1}. The focus of this work is to find the truth value of a propositional logic fact, also referred to as the fact-checking, using an LLM and KG context.
Example 1. @i value of q), denoted as T(q)) 0, Vatican City did not participate in the 2008 Olympies. Olympics, Nelly Korda) A
Olympics, Vatican City) is simple fact. The truth
Let us consider complex fact SeardedT(Ollyppp)) Gold Medal, Nelly Korda). In this case, the truth value T(g)) since both of the constituent simple facts are true.
3 MEASURING LOGICAL CONSISTENCY ON PROPOSITIONAL LOGIC FACTS
In this section, we motivate and define the quantitative measure of logical consistency of an LLM response on propositional logic facts, presented as LLMQuery (Figure 1). When prompted the LLM with a fact the LLM either accepts g as a valid fact or rejects it, that is, LLM(g) E {0, 1}, which binary function that either validates q with response 1, invalidates g with response 0. In this paper, we guide the LLM to be a binary classifier by auto-regressively generating binary responses such as yes/no, accept/reject, etc., via prompt instructions. Next, we discuss the correctness criteria of LLM(q) followed by consistency measures.
Definition (Correctness of fact-checking via LLM response). Given fact q, an LLM response is correct if LLM(y) Tyy). In other words, correct LLM response produces I for true fact (T(g) = 1) and 0 for false fact (T(4) = 0). The correctness can be extended to measure the average fact-checking accuracy of the LLM on batch Q {gk < k & N) of N facts as Accuracy(LLM, Q) Σk=1 L T(()))) where 1 is an indicator function taking value the LLM is correct for the fact gk, and 0 otherwise.
Note that an LLM response being correct (or incorrect) says nothing about whether the response is consistent not. We clarify this point with example, where the LLM is correct in one out of two fact-checking queries, yet inconsistent as a whole.
Example 2. Consider a true fact honoredFor(Golder Globe Award for Best Director, Born on the Fourth of July) from Freebase KG and a false fact - = honoredFor oGoleen Globe Award for Best Director, Chamada Cobrar). In both 9 and -q. the subject entity is an award category and the object entity is a film. Next, we formulate an LLMQuery as a (Fact Context)-pair as in Figure we defer the construction of KG context & 3.4, where the context intuitively contains the necessary knowledge for the LLM in validating q or -q Upon prompting Llama2-7B model with (Fact g. Context), we obtain the response LLM(g) 1. On the contrary, when we prompt Llama2-7B with negative LLMQuery (Fact -g. Context) with the same context but negated fact, we receive an identical response LLM(-q) 1. Therefore, the LLM responses are inconsistent as it returns to both true and false facts and fails to distinguish between logically opposing facts.
well-known that the set of operators {-, A, v} is functionally complete (Howson, 2005), im- plying that any propositional logic statement can be expressed using these three operators. Hence, we start with defining an LLM response's is consistency for these three operators.
Definition 2 (Logical Consistency Primitive Operators). Let consider p and proposi- tional logic facts. An LLM response is logically consistent if the following conditions are satisfied.
LLM( (q) -LLM(g)
LLM(P q) LLM(p) v LLM(4)
LLM(p q) = LLM(p) A LLM(4)
(3)
Informally, the consistency criteria states that LLM responses constituent sub-queries must com- ply with the semantics of the logical operators connecting those sub-queries.
Below, we discuss the LLM's consistency w.r.t. these three operators in detail with KG contexts.
3.1 CONSISTENCY OF SIMPLE FACTS
A simple fact is an atomic relation; therefore, consistency on negation is applicable (Equation 1). As stated in $2.1, 1, simple fact in KG is either relation r(u, v) or its negation -r(u, v). The negation
4

---

Published as a conference paper at ICLR 2025
-r(u, u) can be computed in two ways: (1) by replacing in the triplet (u, r) with alternative falsified relation s.t. (u, v) = 0; or (2) via negating entities, e.g. by finding an alternate object entity o' such that r(u, r²) = O. We adopt the second approach for practical purposes since the notion of consistency does not depend on how the negation of a triplet is computed.
Given simple fact q r(u, v). check consistency with respect to the negation operator. First, we compute the negation of this fact, =q = r(u, v²) by finding an alternate object entity v', and finally, we check the consistency of LLM responses based on whether the following holds (equa- tion? 1).
LLM(r(u, e')) -LLM(r(u, v))
Example showed an example of logical inconsistency. In the following, we show an example of logical consistency of a simple fact.
Example 3. Consider true fact q = capital Hamoi) from Freebase and a false fact
Sorkh Qalch North Khorasan). We formulate LLMQuery as a (Fact q. Context)-pair similar to that in Figure 1. Upon prompting Llama2-7B, we obtain the response LLM(g) 1. On similarly constructed negation LLMQuery (Fact -q. Context), we find the response LLM( (q) = O. Since LLM(-q) = 0 = -1 = -LLM(g), Llama2-7B is logically consistent.
-q = capital
Similar accuracy, measure the average consisteney of LLM responses on given batch containing N pairs of simple facts and negations, Q = {(g. -9k) & k E N):
Consistency(LLM, Q) N E -uM()))--(((())>>
3.2 CONSISTENCY OF COMPLEX FACTS
The canonical example of a complex fact is DNF fact since a DNF fact consists of functionally complete logical operators {-, A, v). Indeed, one can construct complex facts using other opera- tors, such as implication (=) and biconditional (+), however, such facts can be transformed into DNF (Davey & Priestley, 2002). Therefore, define the consistency of complex fact as the consistency of a DNF fact and apply similar constraints as in Definition consistent DNF fact if the LLM response the DNF fact is equal to applying a hierarchy of disjunctions followed by conjunctions on the LLM responses over the lowest level atomic facts.
Informally, an LLM is
Proposition 1. An LLM is consistent on a DNF fact = V where 6 = A Aー= Bij. if LLM(g) =
fm LLM(e,)) Here, eij is an atomic relation fact for any 1 < i < and < & im- Via ( -
The proof is given in the Appendix C. Since any propositional logic fact can be ex- pressed in the DNF form, Proposition encompasses the consistency criteria of any propo- sitional logic fact. addition, we measure the average consistency given a batch of N DNF queries Q {gx = fm (n = N} as Consistency(LIM, Q)
1 EN
N
The choice of DNF in Proposition is not the only design choice, since the proposed method for assessing logical consistency not reliant on any specific normal form. In fact, the definition of consistency is flexible enough to be adapted to other canonical forms such as CNF (Proposition 3). In the following, as basic example of complex facts, we show an LLM'ss consistency assessment using conjunction with two atomic relation facts from Freebase.
Example 4. Consider two true facts from Freebase: p = filmProducti onniuuttriinnivvrssalal Studios, Media, Valentine's Day). By prompting
The Family Man) and Llama2-7B with an LLMQuery constituting a complex fact with ^ operator, i.c., the (Fact p Fact 4. Context)-pair, we obtain the response LLM(p ^ 4) = 1. Similarly, by prompting the (Fact p. Context)-pair, we receive the response LLM(p) 1. We obtain the same for q. LLM(g) 1. this case, the LLM is consistent because LLM(p q) LLM(p) LLM(g).
Artistic gymnastics) implying that Artistic Gymnastic is played in Brazil and another true fact & sport(2012 Summer Olympics, Artistic gymnastics). Upon prompting Llama2-7B with an LLMQuery with (Fact A Fact 9 Context) obtain the response LLM(P q) 1. However, upon prompting Llama2-7B with the constituent simple facts individually, we receive the response LLM(P) 0 and LLM(g) = 1. In this case, the L.L is inconsistent because LLM(p A q) 1 0 LLM(p) LLM(g).
Consider the true fact = countryReverse
5

---

Published as a conference paper at ICLR 2025
Algorithm LLM fact-checking with KG contexts (for simple facts)
Input: A language model LLM, knowledge graph G, simple fact r(u, v), maximum hop À
Output: A binary value {0, 1
1: Gabb BoundedBPS u,
Find the À hop subgraph of the subject in t Get the relevant context for relation r in Geub
2: (p BuildContext( (GenttttettGGebrrr r)
3: return LLMQuery(r(u, v),
Get binary response on concatenated context and target fact
3.3 CONSISTENCY FOR LOGIC RULES
An important family of true facts comes from various logical rules of inference, such as the commu- tative, associative, and distributive properties. One critical question is: Do LLMs respond such that these laws of inference are respected? Let s be propositional logic facts. An LLM is consistent w.r.t. the commutative law if reordering logic facts results in an identical response.
LLM(y q) LLM(4 p) and LLM(P q) LLM(q p)
An LLM is consistent w.r.t. the associative law the following holds.
LLM((P v q) v s) LLM(P v (q v s)) and LLM((P ^ q) ^ s) LLM((P a An LLM is consistent w.r.t. the distributive law if the following holds.
LLM(P ^ (g v s)) (p A 9) v (p v s)) and LLM(p v (q A s)) = LLM((P v g) a (p 8))
Given batch of N logic rules I. (1x & l & N), we measure the average consistency of LLM
responses as Consistency( [LM, L) = N -- 1 Consistency(LIx (.), where Consistencc((LM..4 (.) is an
>
indicator function taking value if the LLM is consistent for the logic rule (g. and 0 otherwise.
3.4 CONSTRUCTING LLMQuery WITH FACTS AND KG CONTEXTS
Our goal is to assess the logical consistency of LLMs on fact-checking queries from KGs, as discussed above. is plausible that LLMs lack the necessary knowledge to answer the fact- checking query accurately, let alone being consistent, especially on recent facts never seen during grraiiing/finettuning As such, motivated by the RAG-based framework, we intend to provide necessary knowledge or context from the KG in LLMQuery before assessing for consistency. Algo- rithm provides template pipeline for LLMQuery, where key step is to find the relevant context for the target fact. Among different choices, we first discuss a graph traversal- -based method for context retrieval, followed by alternate methods such as vector-embedding--eee based retrieval.
Graph Traversal. Given target fact, apply breadth-first search (BFS) to get relevant sub- graph from the KG. For simplicity, let us consider a simple fact either in the form r(u, v). or in the negation form -r(u, v). The relevant information of an entity is generally expected to be in the local neighbourhood in the KG (Dettmers et al., 2018; Wang et al., 2020b). Hence, we conduct BFS from the subject entity up certain number of hops (e.g., À 2 3) in the KG, and get a connected subgraph (line 1). To analyse the complexity of BFS, let the X-hop neighbourhood of contain [v(x) nodes and [x)] edges in the KG. The time complexity of BFS-based context building is (()) + Ex (w)l). Thereafter, we provide the list of triplets in the subgraph as a context to LLMQuery (line 2). Our hypothesis is that the context contains the necessary information for the LLM to validate the target fact: If the fact is implied by the context, the LLM is expected to return the binary response (validating the fact), and otherwise (invalidating it).
Optimizing Context Generation. LLMs are restricted to processing finite-length context. Be- sides, LLMs' performance often deteriorates with long contexts (Liu et al., 2023a). Therefore, we further prune the context/subgraph from BFS, especially when the subject entity u is connected to many other entities within a few hops. We adopt three potential approaches: (1) We prioritize triplets having semantically similar relations as the target fact's relation r. We identify such semantically similar relations by utilizing the ontology and relation hierarchy, often available with KG (Chah, 2018; Chen et al., 2023). For complex fact, split the context size equally for each simple fact involved in it. (2) We apply emberdig based methods to project triplets in the sub- graph to high-dimensional embedding space (Reimers & Gurevych, 2019) and find relevant triplets

---

Published as a conference paper at ICLR 2025
Consider the context: SContext. the fact correct given the context? Answer Yes when the fact is the context and otherwise. SFact.
The adaptation of LLMQuery from Figure 1 for instruction prompting. The prompt contains a clear instruction (in blue) to guide the LLM towards outputting a correct response.
Figure 2:
close to the embedded target fact using approximate nearest neighbor (ANN) algorithms (Malkov & Yashunin, 2020). (3) As the last approach, we bypass applying BFS, embed the entire KG triplets, and retrieve relevant triplets to the target fact. Thus, we provide the most relevant context to the LLM so that we can preciscly assess its logical consistency. Furthermore, sessing consistency is modular and can benefit from future development in graph-based retrieval of related contexts such graphRAG (Liu et al., 2024; Sanmartin, 2024).
framework for as-
4 IMPROVING THE LLM CONSISTENCY BY SUPERVISED FINE-TUNING
Supervised fine-tuning is the process of fine-tuning LLMs on specific task by providing them with explicit instructions or prompts along with examples of inputs (training data) and outputs (ground- truth labels) relevant to that task. These prompts guide the model's ' learning process and help it specialize in that task. The task considered in this paper is logically consistent fact-checking. Fine- tuning is feasible for this task because there are abundant facts available in the KG for fine-tuning. We first discuss the reasons behind adopting supervised fine-tuning rather than zero-shot instruction prompting for this task. Afterwards, we discuss the steps involved in the fine-tuning.
4.1 MOTIVATION: INSTRUCTION PROMPTING vs. SUPERVISED FINE-TUNING
Supervised fine-tuning requires time and computational resources, and in some cases, it is not needed all (Radford et al., 2019; Ziegler al., 2019). Hence, is important to verify if our task necessi- tates fine-tuning. We first consider zero-shot instruction prompting shown in Figure 2, tationally cheaper alternative. Our goal is to instruct the LLM to output yes when the fact triplets are inside the context, and no otherwise. The context length is limited to 1000 tokens. We empirically find that even for simple facts checking, such zero-shot instruction prompting does not improve the logical consistency, while supervised fine-tuning outperforms zero-shot instruction prompting; the consistency improves by up to 46% (see Tables 3 and 11).
compu-
The reasons for inferior performance of zero-shot instruction prompting compared supervised fine-tuning are two-fold: (1) Nuances and intricacies of logical facts. It is well known that su- pervised fine-tuning of pre-trained LLMs is preferable for tasks that require precise and nuanced predictions, especially when the task involves multiple classes or fine-grained distinctions (Wei al., 2021; Li & Liang, 2021). In such cases, providing explicit supervision through labeled data allows the LLM to adjust its behaviour based on the nuances and intricacies of the task domain and learn to make accurate predictions based on the task-specific examples (Sainz et al., 2024). For a simple fact-checking task, the intricacies arise from understanding the differences between true fact and its corresponding false fact. (2) Limited knowledge. Fine-tuning a pre- trained LLM with task-specific labeled examples enables it directly optimize its parameters for the target task, lead- ing to potentially better performance compared to zero-shot instruction prompting (Ziegler et al., 2019; Xu et al., 2023), which relies on generalization from instructions given in the prompt. For instance, supervised fine-tuning has shown better performance than zero-shot prompting on tasks such named entity recognition, relation extraction, and relation classification (Sainz al., 2024; Xu et al., 2023), while Ziegler et al. (2019) showed that supervised fine-tuning with explicit task- specific instructions or preferences performs better than zero-shot prompting on text summarization task. In the case of our simple fact-checking task, without sufficient data, the LLM fails to learn significantly different representations of false fact from a true fact.
4.2 METHODOLOGY FOR SUPERVISED FINE-TUNING
We consider supervised fine-tuning to align LLM responses be logically consistent. We consider a two-step procedure: dataset creation and supervised fine-tuning. Here, we discuss dataset creation and refer to Appendix D for details on fine-tuning and the two-step Algorithm 2.

---

Published as a conference paper at ICLR 2025
Instruction Dataset Creation. We construct binary instruction dataset from primitive logic facts containing negation, conjunction, and disjunction operators - the goal is to help the LLM correctly classify logic facts as true/false. More specifically,, and p A preceded by context y. For simple facts, the ground truth response of p is true when p E yp and false otherwise- the LLM is expected to learn function where if the target fact is in the context, it should recognize this and outputs 1, and 0 otherwise. For complex facts, the ground truth response for p v q true when p E ip or q E ç, and false otherwise. Similarly, the response for p A q is true when both p y and q é y. Assume a simple fact p having subject node u, and the X-hop neighbourhood of contains [Vx(u) nodes and [x)] edges in the KG. Following $3.4, the time complexity of creating instruction data for the simple fact p is V (w)l + |Ex (u))). In addition, we prioritize keeping both positive and negative training examples
construct dataset with four facts: p. P.
the same batch so that the LLM learns to distinguish between them and subsequently becomes more consistent. Below, we formalize the sufficient condition to achieve logical consistency by achieving accuracy in fact-checking.
Proposition 2. (I) An LLM is consistent on a simple atomic fact if it is accurate both on the fact and its negation. (II) For complex DNF fact, the LLM is consistent if it is accurate on the DNF fact as well as on all constituent atomic facts.
We explain Proposition 2 using examples and defer the proof to the Appendix C. Consider simple fact p. where p E sp. Let the LLM accurately classify p as and -p as O. Then, the LLM is consistent on p. Further, consider a conjunction fact p q. where p E yp and $ y. Let the LLM accurately classify p A as O, and constituent atomic facts p as and as O. Therefore, the LLM is consistent on p A q- Notice that Proposition 2 provides sufficient (but not necessary) condition for the LLM's logical consistency on both simple and complex facts. Therefore, we expect that fine-tuning to enhance the accuracy following Proposition 2 would lead the LLM's logical consistency.
Generalization to Complex Facts and Logic Rules. Our instruction dataset focuses on accu- rately classifying single logic operators such as negation, conjunction, and disjunction. Recognizing primitive operators is the building block for classifying facts and rules with multiple heterogencous operators. Our hypothesis therefore is that fine-tuning to recognize single operators should gener- alize to complex logic facts and rules with multiple operators, which we informally attribute as the emergent abilities of LLMS (Lu et al., 2023; Schaeffer et al., 2024). this work, we empirically showcase the generalization to out-of-distributio complex facts and logic rules in $ 5.
5 EXPERIMENTAL RESULTS
We conduct experiments to evaluate the logical consistency of LLMs and the impact of supervised fine-tuning on improving consistency. Additional details on experiments, results, and code are the supplementary materials. Specifically, we investigate the following research questions empirically.
Logical Consistency via Improved Accuracy. Does the logical consistency of LLMs improve via fine-tuning task of correctly classifying primitive logical operators?
Generalizability. Does the fine-tuned models generalize to in-distribution facts from different logic facts containing more operators?
datasets, and more generally, Efficiency. How is the efficiency of PEFT vs. full fine-tuning and LLM inference on logic facts? Ablation Study. Does instruction prompting improve logical consistency compared to fine-tuning across different model-sizes? What is the impact of different KG retrieval methods on the consis- tency of the LLM? Does adding KG contexts help LLMS in improving accuracy and consistency, and how does the length of context play a role? What is the impact of hyperparameters, e.g., learning rate in fine-tuning?
5.1 DATASETS AND EXPERIMENTAL SETUP
KG Datasets and Facts We consider three KG benchmarks: Freebase (FB15K), NELL, and large-scale dataset from OGB: WikiKG90Mv2 (Wiki) (Hu et al., 2021b). We obtain FB15K and NELL from the codebase of Query2Box (Ren et al., 2020). We create logical fact-checking datasets FreebaseLFC, NELLLFC, and WikiLFC using the FB15K, NELL, and Wiki KGs, respectively. Statistics of these KGs and fact-checking datasets are in Tables 4-5 in the Appendix G. In all exper- iments, we resort to our proposed fact-checking datasets, since to our best knowledge none of the earlier works focus assessing and improving LLM's logical consistency in fact-checking.
8

---

Published as a conference paper at ICLR 2025
Table Accuracy and logical consistency LLMs before and after fine-tuning (FT). Bold numbers denote an improved result in accuracy and consistency for fine-tuning. Training is performed on FreebaseL.FC dataset only (marked with '*'), while performance improved all datasets.
Accuracy
Logical Consistency
Model
Dataset
Fact
Before FT
After FT
Before FT
After FT
0.90
0.93
0.81
0.86
FreebaseLFCC
p
0.61
0.93
0.67
0.83
p v
0.73
0.76
0.73
0.97
Llama2-13B
-p
0.88
0.97
0.76
0.93
NELLLFC
paq
0.38
0.89
0.69
0.88
pvg
0.73
0.76
0.73
WikiLFC
p. PP
0.96
0.96
0.92
0.93
PP
0.87
0.97
0.76
0.94
Freebasel. LFC*
p A
0.39
0.87
0.47
0.86
p V
0.78
0.81
0.83
0.77
Llama2-7B
P. mM
0.71
0.94
0.44
0.88
NELLLFC
p
0.26
0.81
0.71
0.91
pvq
0.75
0.73
0.95
0.78
WikiLFC
PP
0.90
0.90
0.80
0.81
p. pp
0.82
0.98
0.66
0.96
FreebaseLFGe
p A
0.45
0.83
0.70
0.73
0.94
0.91
0.90
Gemma-28
P: PP
0.94
0.62
NELLLFC
p A
0.33
0.83
0.78
0.83
0.75
0.88
0.98
0.87
WikiLFC
"P
0.76
0.90
0.52
0.80
Average
0.69
0.88
0.74
0.88
LLMs. We evaluate the logical consistency of three open-source LLMs: the chat version of Llama2- 7B (Touvron al., 2023) and Llama2-13B (Touvron et al., 2023) with 7B and 13B parameters, respectively, and instruction tuned Gemma-2D with 2B parameters (Team et al., 2024). We fine- tune epochs and save the intermediate models. In order to select the best model for each dataset and fact type, consider the following principle: compute the sum of accuracy and consistency on both evaluation and test set (to prioritize both accuracy and consistency), find the optimal epoch having the maximum sum the evaluation set, and report results respective test set.
5.2 EXPERIMENTAL RESULTS
Improved Logical Consistency via Improved Accuracy. Table shows the result for simple facts (p, -p) and facts with one logical operator. In all LLMs and FreebaseLFC dataset, fine-tuning results in higher accuracy and consistency in most fact types the exception is for disjunctive fact v where fine-tuning demonstrates a higher accuracy yet slight decrease in consistency (further explanation Appendix G.3). We observe a similar trend in NELL and Wiki datasets which are not included in fine-tuning- revealing the generalization effect of fine-tuning on similar facts in different datasets. Within Llama2 base models (before fine-tuning), the 13B model has, in general, higher accuracy and consistency logic facts than the 7B model thus, more model parameters achieve improved accuracy and consistency, which is further improved via fine-tuning. Therefore, fine-tuning logic facts for increasing accuracy in dataset increases accuracy (on average 19%) and logical consistency (on average 14%) across all datasets.
Generalizability of Fine-tuned Models. Table 2 shows the generalizability of fine-tuned models, ie., increase in accuracy and consistency across facts with more operators and commutative rules. In particular, while accuracy is always higher out-of-distribution facts, there is occasional

---

Published as a conference paper at ICLR 2025
Table 2: Generalization of fine-tuning to complex facts and rules in Llama2-13B. For results on
Llama2-7B and Gemma-2B, refer to Table 13 in the Appendix G
Accuracy
Logical Consistency
Model
Dataset
Fact
Before FT
After FT
Before
After
PV
0.74
0.85
0.78
0.81
PA (q v P < qVP
0.53
0.82
0.63
0.79
FreebaseLFFC
0.73
0.76
0.85
0.98
Llama2-13B
PA AAQ+QAA
0.61
0.93
0.84
0.91
p (q r) P (q V
0.67
0.77
0.72
0.80
0.42
0.38
0.75
0.85
NELLLFC
PVQ q p PAq+
0.73
0.76
0.80
0.98
0.38
0.70
0.90
0.85
Table Impact of instruction prompting (Prompt) vs. fine-tuning (FT) on simple facts. Fine-tuning results in higher accuracy and consistency than instruction prompting. Extended result is in Table 1..
Accuracy
Logical Consistency
Model
Dataset
Before FT
Prompt
After FT
Before FT
Prompt
After FT
FreebaseLFCC
0.90
0.89
0.93
0.81
0.79
0.86
Llama2-13B
NELLLFC
0.88
0.85
0.97
0.76
0.72
0.93
WikiLFC
0.96
0.91
0.96
0.92
0.90
0.93
decrease in consistency, specially in facts containing a disjunctive operator. Therefore, nudging the LLMs to recognize primitive operators can generalize to complex facts and rules.
Efficiency. Our empirical results demonstrate that PEFT is 3X more efficient than full fine-tuning for logical consistency. The and-to-end fact-checking time is also efficient (less than 0.63 seconds per sample) in our datasets. For details, refer to Tables 6-7 in the Appendix G.
Ablation Study: Supervised Fine-tuning vs. Zero-shot Instruction Prompting. Table shows the effect of zero-shot instruction prompting (an example in Figure 2) vs. fine-tuning on the accuracy and logical consistency of simple facts. First, instruction prompting negatively impacts the LLM reasoning on Llama2 base models (before fine-tuning), where both accuracy and consistency lower compared to not considering any instruction. In contrast, fine-tuning results in higher accuracy and logical consistency than instruction prompting in all datasets and fact types, thereby validating the need for fine-tuning to improve accuracy and logical consistency of logic facts.
6 CONCLUSIONS
We propose a methodology to assess and improve the logical consistency of LLMs in fact-checking from KGs. We formalize the definition of logical consistency of I. on propositional logic queries comprising multiple logical operators. LLMS are usually less consistent in answering logic facts. A supervised fine-tuning for recognizing primitive logic operators (negation, conjunction, and disjunc- tion) improves their logical consistency
show 14% increase in consistency via fine-tuning.
Our work has potential impacts on trustworthy language modeling. logically consistent LLM less likely to hallucinate since its response is unlikely to change due to the logical manipulation of the prompt. Future works can assess consistency when LLM responses extend consistency assessment to natural language queries containing logical relations. We will consider unstructured context unlike this paper, which further complicates LLM's ability to aceu- rately and consistently answer fact-checking queries. On the KG front, we investigate the efficacy of knowledge injection methods, or graph-to-text conversion, input more relevant contexts in the LLMQuery. Finally, incorporating link prediction and reasoning capabilities into LLMs to conduct logic fact-checking over incomplete KGs is an interesting problem.
several directions. Beyond binary responses, we aim to not trivially classified into finite categories. We will
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Garima Agrawal, Tharindu Kumarage, Zeyad Alghami, and Huan Liu. Can knowledge graphs reduce hallucinations in LLMs? survey. CoRR, abs/231 2023.
Ammar Ammar and Remzi Celebi. Fact validation with knowledge graph embeddings. In ISWC, pp. 125-128, 2019.
Akari Asai and Hannanch Hajishirzi. Logic-guided data augmentation and regularization for con- sistent question answering. In ACL, pp. 5642-5650, 2020.
Jinheon Back, Alham Fikri Aji, and Amir Saffari. Knowledge-augmentee language model prompt- ing for zero-shot knowledge graph question answering. In NLRSE@ACL, 2023.
George Boole. The mathematical analysis of logic. CreateSpace Independent Publishing Platform, 1847.
Antoine Bordes, Nicolas Usunier, Alberto Garefa-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NeurlPS, pp. 2787-2795, 2013.
Agustín Borrego, Daniel Ayala, Inma Hernández, Carlos R. Rivero, and David Ruiz. CAFE: Knowl- edge graph completion using neighborhood-awwan features. Engineering Applications of A'ttfficial Intelligence, 103: 104302, 2021.
Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam Hruschka, and Tom Mitchell. Toward an architecture for never-ending language learning. In AAAI, pp. 1306-1313, 2010.
Niel Chah. OK Google, what your ontology? or: Exploring freebase classification to understand Google's knowledge graph. CoRR, abs/1805.03888, 2018.
Lihu Chen, Gael Varoquaux, and Fabian M. Suchanek. GLADIS: A general and large acronym disambiguation benchmark. In EACL, pp. 2073-2088, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer. Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov. Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1-240: 13, 2023.
Douglas
Dev Dash, Eric Horvitz, and Nigam Shah. How well large language models support clinician information needs. Stanford University Human-Centered Artificial Intelligence, 2023.
Brian A Davey and Hilary Priestley. Introduction to lattices and order. Cambridge university press, 2002.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2D knowledge graph embeddings. In AAAI, pp. 1811-1818, 2018.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized L.MMs. In NeurIPS, 2024.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 2018.

---

Published as a conference paper at ICLR 2025
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph RAG approach to query-focused summariza- tion. CoRR, abs/2404.16130. 2024.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language mod- els. Transactions of the Association for Computational Linguistics, 1:1012-1031, 2021.
Allyson Etringer. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48, 2020.
Luis Galárraga, Christina Teflioudi, Katja Hose, and Fabian M. Suchanek. Fast rule mining in ontological knowledge bases with AMIE+. VLDBB J., 24(6) 2015.
Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. A survey automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178-206, 2022.
Lorenz Halbeisen and Regula Krapf. Gödel's Theorems and Zermelo's oss Axioms: A Firm Foundation of Mathematics. Springer Nature, 2020.
Ahmed Abdeen Hamed, Alessandro Crimi, Magdalena M. Misiak, and Byung Suk Lee. Establishing trust in ChatGPT biomedical generated text: An ontology-basee knowledge graph to validate discase-sympton links. CoRR, abs/2300.003929 2023.
Shibo Hao, Bowen Tan, Kaiwen Tang, Bin Ni, Xiyan Shao, Hengzhe Zhang, Eric P. Xing, and Zhit- ing Hu. BertNet: Harvesting knowledge graphs with arbitrary relations from pretrained language models. In ACL, pp. 5000-5015, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009. 03300, 2020.
Colin Howson. Logic with trees: an introduction to symbolic logic. Routledge, 2005.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv preprint 1eXiv:2/06.09660 2021a.
Nan Hu, Jiaoyan Chen, Yike Wu, Guilin Qi, Sheng Bi, Tongtong Wu, and Jeff Z. Pan. Benchmarking large language models in complex question answering attribution using knowledge graphs. CoRR, 0os/2401.146444 2024a.
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovee, OGB- LSC: A large-scale challenge for machine learning graphs. arXiv preprint arXiv:2 2021b.
Xiangkun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, and Zheng Zhang. Refchecker: Reference-based fine-grained hallucination checker and benchmark for large language models. 2024b.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Truns. Inf. Syst., 2024a.
Wenyu Huang, Guancheng Zhou, Mirella Lapata, Pavlos Vougiouklis, Sebastien Montella, and Jeff Z. Pan. Prompting large language models with knowledge graphs for question answering involving loig-tail facts. CoRR, abs/2405.06524, 2024b.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21226, 2024.
12

---

Published as a conference paper at ICLR 2025
Viet-Phi Huynh and Paolo Papotti. Towards benchmark for fact checking with knowledge bases. In Reasoning on Data Workshop WWW, pp. 1595-1598, 2018.
Myeongjun Jang. Frank Mtumbuka, and Thomas Lukasiewicz. Beyond distributional hypothesis: Let language models learn meaning-text correspondence. NAACL, pp- 2030-2042, 2022.
Myeongjun Erik Jang and Thomas Lukasiewicz. Improving language models' meaning understand- ing and consistency by learning conceptual roles from dictionary. In EMNLP, pp. 8496-8510, 2023.
Ehsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, and Davood Rafiei. Evaluating open-domain question answering in the era of large language models. In ACL, pp. 5591-5606, 2023.
Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and individual differences, 103:102274, 2023.
Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In ACL. pp. 7811-7818, 2020.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664 2023.
Ashutosh Kumar and Aditya Joshi. Striking a balance: Alleviating inconsistency in pre-trained models for symmetric classification tasks. In ACL, pp. 1887-1895, 2022.
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Prin- ciples, pp. 61 611-626, 2023.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. serrieval-augment generation for mnowledg-iintensing NLP tasks, In NeurIPS, 2020.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. A logic-driven framework for consistency of neural models. In EMNLP-IJCNLP, pp. 3922-3933, 2019.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL-IJCNLP, pp. 4582-4597, 2021.
Xingxuan Li. Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Li- dong Bing- -owwoeedge-ow Grounding large language models via dynamic knowledge adapting over heterogeneous sources. In ICLR, 2023.
Peng Lin, Qi Song, Yinghui Wu, and Jiaxing Pi. Discovering patterns for fact checking in knowledge graphs. ACM J. Data Inf. Qual., 1133):131113::7, 2019.
Ruixi Lin and Hwee Tou Ng. Does BERT know that the IS-A relation is transitive? In ACL, pp. 94-99, 2022.
Guangyi Liu, Yonggi Zhang, Yong Li, and Quanming Yao. Explore then determine: A GNN-LLM synergy framework for reasoning over knowledge graph. CoRR, abs/2406.01 145, 2024.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models long contexts. CoRR, abs/2307.03177 2023a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Comput. Surv. 19: 2023b.
13

---

Published as a conference paper at ICLR 2025
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-BERT: Enabling language representation with knowledge graph. In AAAI, pp. 2901-2908, 2020.
Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy LLMs: A survey and guideline for evaluating large language models' alignment. arXiv preprinl arXiv:2308.0533 2023c.
Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. Are emergent abilities large language models just in-context learning? arXiv preprint arXiv:2309.01809 2023.
Weichen Luo and Cheng Long. Fact Checking on Knowledge Graphs, pp. 149-168. Springer International Publishing, 2021.
Qitan Lv, Jie Wang. Hanzhu Chen, Bin Li, Yongdong Zhang, and Feng Wu. Coarse-to-fine high- lighting: reducing knowledge hallucination in large language models. In ICMI., 2024.
Yu A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4))824-836, 2020.
Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Nakatumba-Nabenta Prompt engi- neering in large language models. In International Conference Data Intelligence and Cognitive Informatics, pp. 387-402. Springer, 2023.
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Comput. Surv., 56(2), 2023.
OpenAl. Introducing ChatGPT. https: /top:::///peaai.... http://pppeaaa..m//oopaoaac
com/ www.o////o/.a///////ocooo//oee/e 2022.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In ACL, 2016.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In EMNLP-IJCNLP, pp- 2463-2473, 2019.
Dorian Quelle and Alexandre Bovet. The perils and promises of fact-checking with large language models. Frontiers in Artificial Intelligence, 7, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Harsh Raj, Vipul Gupta, Domenic Rosati, and Subhabrata Majumdar. Semantic consistency for assuring reliability of large language models. arXiv preprint arXiv:2308.00/33 2023.
Abhilasha Ravichander, Eduard H. Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Che- ung. On the systematicity of probing contextualized word representations: The case of hypernymy in BERT. In SEM@ COLING, pp. 88-102, 2020.
Arijit Ray, Karan Sikka, Ajay Divakaran, Stefan Lee, and Giedrius Burachas. Sunny and dark out- side?! improving answer consistency in VQA through entailed question generation. In EMNLP- IJCNLP, pp. 5859-5864, 2019.
Nils Reimers and Iryna Gurevych. Sentence-Bel BERT: Sentence embeddings using siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2019. URL https://arxt /raxiv. org/abs/ / 908. 10084.
14

---

Published as a conference paper at ICLR 2025
Hongyu Ren, Weihua Hu, and Jure Leskovec. Query2box: Reasoning knowledge graphs in vector space using box embeddings. In ICLR, 2020.
Mohammed Saeed, Nicola De Cao, and Paolo Papotti. A DB-first approach to query factual infor- mation llms. Table Representation Learning Workshop at NeurIPS, 2023.
Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. GoLLIE: Annotation guidelines improve zero-shot informatiinn-extraciion In ICLR, 2024.
Diego Sanmartin. KG-RAG: Bridging the gap between knowledge and creativity. abs/2405. 2035,, 2024.
CoRR,
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage? In NeurlPS, 2024.
Baoxu Shi and Tim Weninger. Discriminative predicate path mining for fact checking in knowledge graphs. Knowl. Based Syst., 104:123-133, 2016.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei. Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972), 2023.
Fabian M. Suchanek and Anh Tuan Luu. Knowledge hases and language models: Complementing forces. In RuleMI,+RR RR, 2023.
Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowl- edgeable are large language models (LLM)? a.k.a. will LLMs replace knowledge graphs? In NAACL-HLT, pp. 311-325, 2024a.
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wen- han Lyu, Yixuan Zhang, Xiner Li, et al. TrustLLM: Trustworthiness in large language models. arXiv preprint ar 240555400,, 2024h.
Liyan Tang, Philippe Laban, and Greg Durrett. Minicheck: Efficient fact-checking of llms on grounding documents. In EMNLP, 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint wrXiv:2403.08295, 2024.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: large- scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 809-819, 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. CoRR, 139/2332.11971,, 2023.
Chenguang Wang, Xiao Liu, and Dawn Song. Language models are open knowledge graphs. CoRR, abs/2010.11967, 2020a.
Haohan Wang, Da Sun, and Eric P. Xing. What if we simply swap the two text fragments? A straightforward yet effective way to test the robustness of methods to confounding signals in nature language inference tasks. In AAAI, pp. 7136-7143, 2019.
Liang Wang, Nan Yang, and Furu Wei. Learning retrieve in-context examples for large language models. In EACL, pp. 1752-1767, 20244a
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V. Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In ICLR, 2023a.
15

---

Published as a conference paper at ICLR 2025
Yuqing Wang, Yun Zhao, and Linda Petzold. Are large language models ready for healtheare? A comparative study on clinical language understanding. In Machine Learning for Healthcare Conference, 2023b.
Yuxiang Wang, Arijit Khan, Tianxing Wu, Jiahui Jin, and Haijiang Yan. Semantic guided and response times bounded top-k similarity search over knowledge graphs. In ICDE, pp. 445-456, 2020b.
Zeyuan Wang, Qiang Zhang. Keyan Ding, Ming Qin, Xiang Zhuang, Xiaotong Li, and Huajun Chen. InstructProtein: Aligning human and protein language via knowledge instruction. In ACL, pp. 1114-1136, 2024b.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
Yanbin Wei, Qiushi Huang, Yu Zhang, and James T. Kwok. KICGPT: Large language model with knowledge in context for knowledge graph completion. In EMNLP, pp. 8667-8683, 2023.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In NUT@ EMNLP. 2017.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prab- hanjan Kambadur, David Rosenberg, and Gideon Mann. BloombergGPP: A large language model for finance. arXiv preprint arXiv:2303. /7564, 2023.
Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, and Maosong Sun. Lawformer: A pre-trained language model for Chinese legal long documents. Al Open, 2:79-84 84, 2021.
Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, and Enhong Chen. Large language models for generative information extraction: A sur- vey. arXiv preprint arXiv:2312.17611 2023.
Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. Retrievallaugmmened generation with knowledge graphs for customer service ques- tion answering. In SIGIR, pp. 2905-2909, 2024.
Hongyang Yang. Xiao-Yang Liu, and Christina Dan Wang. FinGPT: Open-source financial large language models. arXiv preprint arXiv:2306.06031, 2023a.
Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. ChatGPT is not enough: En- hancing large language models with knowledge graphs for fact-aware language modeling. CoRR, abs/2306.1 1489, 2023b.
Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. Jaket: Joint pre-training of knowl- edge graph and language understanding. In AAAI, pp. 11630-11638, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In ACL, 2019.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML, pp. 12697-12706, 2021.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? In EMNLP, pp. 4862 4876, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu. Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: /909 08593, 2019.
16

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND BROADER IMPACTS
Our accuracy and consistency measures are given over logic facts for which ground-truths are avail- able from the input KG and LLM responses are binary. In this work, we do not consider link pre- diction and multi-hop reasoning capabilities of LLM models to validate facts over incomplete KGs. is vital for both researchers and investigators to exercise prudent judgment and validate findings through additional comparative methods before arriving at any definitive conclusions or undertaking consequential actions.
LLM's general reasoning ability has been under much scrutiny in recent years, and experiments suggest that there is still a gap between LLMs and human-like logical reasoning ability, Given such a state of affairs, one limitation of our current study is that we do not evaluate how much logical consistency improves the general reasoning ability of LLMs. Another limitation is that our paper mainly focuses on binary responses, and does not consider extensions multi-class logic and non- binary answer consistency. We will consider such extensions in the future.
B RELATED WORKS
B.I LARGE LANGUAGE MODELS WITH KG CONTEXTS
Large language models (LLMs), pre-trained on large-scale web and enterprises corpus, encode sig- nificant knowledge implicitly in its parameters without human supervision, which can be probed for various question-answering and querying tasks, thus LLMs act as knowledge bases (Petroni et al., 2019; Hao al., 2023; Wang et al., 2020a). LLMS have also shown tremendous promise in new tasks for which they have not been pre-trained. Users can interact with state-of-the-art I. L.LMs through prompting. They simply specify instructions for new tasks as text prompts to guide an LLM for the desired outputs (Liu et al., 2023b). However, LLMs are proficient at learning probabilistic language patterns and may not explicitly store consistent representations of knowledge, hence they can output unreliable and incoherent responses and often experience hallucinations by generating factually incorrect statements (Elazar et al., 2021; Liu et al., 2023e; Sun et al., 2024a).
Knowledge graphs (KGs) can enhance pre-training (Yu et al., 2022), fine-tuning (Wang et al., 2024b), inference (Wei et al., 2023), prompting (Baek et al., 2023), retrieval/ knowledge augmented generation (Xu et al., 2024), knowledge editing (Zheng et al., 2023), and knowledge validation (Hu et al., 2024a) of LLMs. In particular, KGs can assist in the inference phase of LLMs through retrreval-augmeented generation (RAG), e.g., to provide KG context within prompts for reducing hallucinations, thus improving accuracy and consistency. For instance, Liu et al. (2020) inject KG triplets into LLMs to enhance the domain knowledge, meanwhile Li et al. (2023) uses external data sources to improve the factual correctness of large language models. GraphRAG is the latest tech- nology that models external knowledge base as knowledge graph improve retrieval methods with a more comprehensive contextual understanding and thereby assists in obtaining more precise search results via LLMs (Xu et al., 2024; Edge et al., 2024; Sanmartin, 2024; Liu al., 2024). Since we consider a RAG paradigm for LLM's consistency assessment, good retrieval technique (e.g., future developments on graphRAG) can be leveraged in our framework further improve the LLM's accuracy and consistency in fact-checking.
B.2 MEASURING AND IMPROVING THE CONSISTENCY OF LLMS
With the prevalence of large language models, measuring their consistency has become criti- cal. LLMs can generate logically contradicting outputs, e.g., different predictions for meaning- preserving text alternations (Ravichander et al., 2020; Elazar et al., 2021). They may violate im- portant relational properties such negation (Kassner & Schütze, 2020; Ettinger, 2020; Asai & Hajishirzi, 2020; Jang et al., 2022), symmetry (Wang et al., 2019; Li et al., 2019; Kumar & Joshi, 2022), and transitivity (Asai & Hajishirzi, 2020; Lin & Ng. 2022). Raj et al. (2023) measures the semantic similarity of LLM outputs in response to paraphrased versions of a question.
To improve the consistency of LLM, several techniques are developed, e.g., adding special consistency loss function to the original loss that penalizes inconsistent answers and continuing the pre-training step (Elazar al., 2021). Others achieve this through data augmentation (Ray et al.,
17

---

Published as a conference paper at ICLR 2025
2019; Asai & Hajishirzi, 2020). Some other works learn inter-relationships between concepts using external dictionaries (Jang & Lukasiewicz, 2023). There are also general approaches to improve an LLM's accuracy, but the same methods could be applied to enhance consistency, such reinforce- ment learning from human feedback (Ouyang et al., 2022), prompt design optimization (Marvin al., 2023), as well model fine-tuning.
First, unlike the bulk of the literature that measures an LLM's consistency on natural language pro- cessing tasks, we develop systematic consistency measurement techniques of retrieval-augmenter LLMs for propositional logic facts and logic rules, with knowledge graph contexts. Second, existing approaches generally update all model parameters and are inefficient. We avoid full fine-tuning by using parameter-efficient supervised fine-tuning through low-rank adaption. We have experimen- tally demonstrated that our approach effective, efficient, and generalizes to complex facts and logic rules, can handle large-scale KGs, and LLMs having billions of parameters.
B.3 LARGE LANGUAGE MODELS HALLUCINATIONS
LLMs may not explicitly store consistent representation of knowledge. Hence, they halluci- nate by generating unreliable and incoherent responses, and often plausible yet factually incorrect statements. There are several types of LLM hallucinations. In this work, our focus is faithfulness hallucination (Huang et al., 2024a), resulting in divergence of LLM--geeerted contents from user inputs (c.g., KG contexts, instructions, examples, etc.), as well as the lack of consistency within the generated contents when the input queries are logically perturbed. Subsequently, various strategies have been developed detect and reduce LLMs' hallucinations. For instance, serrival--augmette generation (RAG) enhances models with up-to-date knowledge from external sources. More ad- vanced approaches (Wang et al., 2024a) retrieve high-quality candidates to avoid the context poi- soning issue, e.g., L.LM-Eebedder (Lv et al., 2024) is learns vector embedding by rewarding high-quality retrieval candidates, contrastive learning, and knowledge distillation. In fact, we have already shown that vector embedding methods improve logical consistency over traversal-based context retrieval (Appendix E, Table 11). In the future, would be interesting replace the Sentence Transformer in our vector embedding method via LLM- Embedder to construct more relevant contexts. Another approach to reduce LLMS' hallucinations is the chain-of-thought (CoT) prompting (Wei et al., 2022), which encourages LLMs generate rea- soning steps before responding. We have also conducted experiments with CoT (Appendix & G.5.3), but it does not demonstrate any improvement over fine-tuning. As such, we hypothesize that achiev- ing logical consistency requires an update in the internal weights of the LLMs, e.g., via supervised fine-tuning. Moreover, self-consistency (Wang et al., 2023a) employs post-processing approach that does not improve the internal weights of the model and instead addresses the decoding module of the model - it samples multiple responses of the L.M and chooses a response with the major- ity vote that is denoted as consistent. In contrast, our fine-tuning procedure improves the internal weights the LLM to be a consistent responder under logical perturbation of input queries, thereby reducing faithfulness hallucination.
retrieval technique that
B.4 FACT-CHECKING WITH KGs AND LLMS
Knowledge graphs that store high-quality facts are reliable and valuable sources for fact-checking. We refer to Luo & Long (2021); Huynh & Papotti (2018) for surveys and experimental benchmarks. In particular, KG approaches for fact-checking can be broadly classified into paths (Shi & Weninger, 2016), rules and patterns (Lin et al., 2019; Galárraga et al., 2015), and embedding- based (Ammar & Celebi, 2019; Borrego al., 2021). Recent embedding methods determine whether given fact, currently missing in an incomplete knowledge graph, should appear in it. In contrast, our focus is to validate logic facts from the input KG.
LLMs like GPT-3.5 and GPT-4 have also been exploited for automated fact-checking with and with- out access to external contexts, e.g., Google search results (Quelle & Bovet, 2024). Hamed et al. (2023) studied fact-checking in biological graphs constructed from ChatGPT contents. Recently, MiniCheck (Tang et al., 2024) and RefChecker (Hu et al., 2024b) perform fact-checking of LLM outputs, primarily simple facts claim- trimeeti grounding documents. To the best of our knowl- edge, ours is the first work on measuring and improving the consistency of LLMs for propositional logic-based fact-checking with KG contexts.
18

---

Published as a conference paper at ICLR 2025
C
FORMAL PROOFS
Proposition 1. An LLM is consistent with respect to DNF fact V=1 where c₁ s=nneih
(LM( V LLM(eN))
Here, Eij is an atomic relation fact for any < E and 1 & & im.
Proof. Applying the definition of consistency (Definition 2) on the LLM response to g, we obtain LLM(g) LLM(c1) LLM((2) .... LLM(cn)
LLM \ eiy) LLMI A c2s) ... LLM
Ens)
2m (A LLMesS) v LLM(ess)
LLM(enS)
LLX(e))
Proposition 2. (I) An LLM consistent on a simple atomic fact if it is accurate both on the fact and its negation. (II) For complex DNF fact, the LLM is consistent if it is accurate the DNF fact as well as on all constituent atomic facts.
Proof. (I) Given a fact and its truth value T(4) {0, 1J, an LLM response LLM(4) is correct (LM(4) T(4) (Definition 1). Similarly, for the negation fact -g, an LLM response LLM(-q) is correct if LLM( (q) = T(-4) Since qq is the negation of g.
T(ạ) -T(-4)
LLM(g) - LLMM -4)
-LLM(g) = LLM(-U)
LLM is consistent on (4, -q)
The last line follows from the definition of consistency (Definition 2)
V (A/m Eig) we similarly assume that the LLM accurately classifies
(II) For DNF fact
q and its constituent atomic fact eij. That is, LLM(g) = T(g) and LLM(c;)) = T(e;y). Relating with constituents Eij as y = ViL (Ajm ess). we derive the following.
T(4)
V
T(G))
==1
LLM(g) V
LIM(CE)
Therefore, the LLM is consistent to the DNF fact according to Proposition 1.
Proposition 3. An LLM consistent with respect CNF fact AR where e vln v/
LLM(g) = ^ V LIM(es)
1=1
Here, eij is atomic relation fact for any E i and jisk. im.
19

---

Published as a conference paper at ICLR 2025
Proof. Applying the definition of consistency to the LLM response to q, obtain
LLM(g) LLM(CI) A LLM((2)
A... LLM(cn)
1m
2m
LLM V e1j) A LLM V e2s) ... LLM
V eng)
1=1
f=1
(LLMsss)
( LLM(ZS) a A...A VLLM(en)) 4-1
LLM())))
D ADDITIONAL DISCUSSIONS ON METHODOLLOGY
Algorithm 2 Supervised fine-tuning for logical consistency
Input: A language model LLM, facts R
Output: fine-tuned model LLM*
1:
Instruct BuildDataset(R)
Create instruction dataset from knowledge queries
2: LLM < SupprvvisedFinetuuinggnnn ( Dinatruct)
D.1 SUPERVISED FINE-TUNING
We adopted a parameter-efficient fine-tuning (PEFT) method based on QLoRA (Quantized Low- Rank Adaptation) (Dettmers et al., 2024), which is a more wemory-effiiiie adaptation of LoRA (Hu al., 2021a). The main idea behind LoRA freeze the pre-trained model weight-matrix Wa of size d x d and represent the trainable update-matrix (AW) two low-rank matrices A and B of dimensions d and r x d, respectively. Here, r << d, e.g., r 64 and > 16K for Llama2-7B and d ~ 8K for Gemma-2B in experiments. Given input vector during forward-pass, instead of computing (Wo- + a, LoRA computes (Woz + BAx). The benefit of storing two low-rank matrices instead of AW is that the model needs update only 2rd parameters, which more efficient than updating de parameters. QLoRA quantizes the precision of the weights to bits. Due quantization, OLoRA is more memory efficient; for instance, it has been reported that one can fine-tune a 65B parameter Llama model on a single 48GB GPU while preserving full 16-bit finetuning task performance (Dettmers et al., 2024).
E DATASETS
E.
DATA SOURCES
We consider existing KGs, such as Freebase, NELL, and WikiKG90Mv2, and convert to logical fact-checking datasets. Freebase contains real-world entities such as people, places, institutions, and their relations from the Freebase KG. NELL is a dataset built from the Web via an intelligent agent called Never-Ending Language Learner. This agent attempts learn over time read the web. NELL has accumulated over 100 million candidate triplets by reading the web (April 2018) with different levels of confidence. WikiKG90Mv2 KG extracted from the entire Wikidata knowledge base.
E.2 OUR CURATED DATASETS
Our curated datasets contain the following logical facts: simple facts p and -p, facts with one operators and q. complex facts with two operators p (g v r) and p (q r), and commutative rules + and A + A p. fine-tuning, we split each fact type into training, evaluation, and test sets having 1K, 5K, and 5K samples, respectively (Table 5). For
20

---

Published as a conference paper at ICLR 2025
Table 4: Statistics of KGs (left) and simple facts obtained from them (right)
KG
#Entities
#Relations
#Triplets (simple facts)
FB 15K
14, 951
1,345
592, 213
NELL
63, 361
200
142, 804
Wiki
9, 10, 610
1,387
60, 10, 62, 811
LLMQuery
((SS)) You are helpful assistant. Always follow the instructions precisely and output the response exactly in the requested format. ((/YYS)
Consider the context Answer question according to the context.
set of triplets where entries separated by 'I' symbol.
T.S. Eliot award winner Nobel Prize in Literature Thomas Mann influenced
Franz Kafka
Thomas Mann award winner Nobel prize in Literature
Do not add additional text. Is the following triplet factually correct? Answer with Yes or No.
Thomas Mann award winner Nobel prize in Literature
Yees or No}
Figure 3:
(Continuing Figure 1) A representative LLMQuery on a simple fact used in our experi-
ments. The expected LLM response is in blue color.
instance, Freebasel.FC contains 1K facts from each of the 4 fact-types: p, P, PAg q. pv resulting 1K 4K facts. fine-tuning, we only consider training facts from FreebaseL hence the other datasets do not have any additional training facts. In Wiki dataset, randomly sample 10 million triplets to prepare subgraph creation via BFS and training/evaluatioonesst split.
E.3 MORE COMPLEX AND HIGHER-ORDEL LOGIC DATASETS
complex logical expressions, especially the law of Syllogism (LoS) and the first-order logic (FoL). In the LoS dataset, there are 100 queries of the form (1 q) r2(g, s) encapsulating the logic (p q) s), which, according to the law of Syllogism should be equivalent to p s.
We have curated two
datasets from Freebase containing
Consider the context: $Context.
the fact correct given the context? $Fact-1 and SFact-2,
LLMQuery on ($Fact-1 A SFact-2, $Context)
Consider the context: SContext.
the fact correct given the context? $Fact-1 or SFact-2.
LLMQuery on ($Fact-1 v SFact-2, $Context)
Figure 4: (Continuing Figure 1) LLMQuery when considering conjunctive and disjunctive facts. We replace logical operators with their natural language description, such as A with 'and' and with 'or', when constructing LLMQuery.
21

---

Published as a conference paper at ICLR 2025
Table 5: Fact-checking datasets
Datasets
Fact types and Rules
Training
Evaluation
Test
FC
p. P.p.AapV
IK (x4)
(x4)
5K (x4)
PA (g (q A (q -
5K (x4)
5K (x4)
NELLLFC
All above fact types and rules
(x8)
(x8)
WikiLFC
p
5K (x2)
5K (x2)
Example: Contains(Oceanias New Zealand) Country(New Zealand, Auckland) logically equivalent to Contains( Montains(Oceani,, Oceania, Auckland).
In the FoL dataset, there are 500 queries of the form (pri(p, x) the form of v (p.ri(pp.0 z) v -r2(2,9) q). The negation of the former should be logically equivalent to the latter. LLMQuery, we consider the natural language description "There exists an entity a such that' and 'For all entity x such that' to express existential and for-all V quantifiers, respectively.
r2(2,0 q) and their negations in
F PARAMETER SETTINGS AND SYSTEM CONFIGURATIONS
F.1 PARAMETER SETTINGS
For an efficient fine-tuning, we adopted QLoRA implementation from Huggingface (Dettmers et al., 2024). Llama2-77 and 13B, the hyperparameter choices of QLoRA are the following: learning rate 2 x 10-5, weight decay 0.001, warm-up ratio 0.03, batch size 8, r = 16, 16, and dropout 0.1. Gemma-2B, we consider learning rate 10-6 and keep other hyper-paramett to Llama. For high-throughput and memory-efficiet inference, we adopt the vL.MM library (K (Kwon et al., 2023), and set temperature to 0 for a deterministic output. We limit the context length in the LLMQuery by selecting relevant triplets of around 1000 tokens.
similar
F.2 SYSTEM SETUP
All experiments are conducted in Python version 3.8.0. Fine-tuning is conducted on cluster with two NVIDIA A40 45 GB GPUs having Intel(R) Xcon(R) Gold 5317 CPU 3.00 GHz, 48 core and 1007 GB RAM. Inference is conducted on a cluster with two Tesla V100-PCIE 32 GB GPUs having Intel(R) Xeon(R) Gold 6134M CPU 3.20 GHz, 32 core, and 755 GB RAM.
G ADDITIONAL EXPERIMENTS
G.1 EFFICIENCY
compare the per-epoch running time of PEFT vs. full fine-tuning of Llama2-7B (Table 6). PEFT requires average 4.43 hours complete an epoch including the computation of evaluation loss, whereas full fine-tuning requires 12.52 hours. Therefore, PEFT is more efficient than full fine-tuming in improving the efficiency of supervised fine-tuning for logical consistency.
Table Running time of PEFT vs. Full Fine-tuning of Llama2-7B
PEFT
Full fine-tuning
Time per epoch
4.43 hours
12.52 hours
Table 7 shows the end-to-end inference time for facts with different number of operators. Increasing the number operators results in a higher time for BFS and subsequent context creation time. In particular, the BFS time for FreebaseLFC is higher than that in NELLLFC, and WikiLFC has the lowest BES time. BFS time is correlated with the sparsity of the KG and can be approximated by how many edges extracted in the BFS subgraph for simple facts, the median number of edges
22

---

Published as a conference paper at ICLR 2025
Table 7: Per-sample end-to-end fact-checking time in seconds using Llama2-7B
Dataset
Fact
BFS
Context Creation
LLM Inference
Total
P. p
0.04 + 0.10
0.06 + 0.10
0.21 : 0.00
0.30 i 0.14
Freebasel.FC
p A g
0.07 + 0.15
0.15 + 0.18
0.22 + 0.00
0.44 + 0.25
V (q A r)
0.09 + 0.20
0.32 i 0.34
0.22 = 0.01
0.63 + 0.39
0.02 + 0.08
0.04 i 0.05
0.19 + 0.00
0.24 + 0.10
NELLLFC
p A q
0.04 + 0.07
0.08 0.13
0.21 0.00
0.32 + 0.16
PV (q r)
0.05 + 0.15
0.09 + 0.09
0.22 + 0.00
0.36 + 0.18
WikiLFC
P.
0.01 + 0.05
0.01 + 0.01
0.08 = 0.00
0.09 + 0.06
in the BFS subgraph is 9773, 763.5, and 7 for FFeebaseLFC., NELLLFC, and WikiLFC, respectively. Further, since WikiLFCC is the sparsest among KGs in our experiments, the context creation time and subsequent LLM inference time is lower in WikiLFC compared to other two datasets for ample, the LLM inference time the lowest WikiLFC (0.08 seconds) compared to FreebaseLFC and NELLLFC (~ 0.20 seconds). In summary, the end-to-end fact-checking time is efficient (less than 0.63 seconds per sample) in our implementation.
G.2 IMPACT OF KG CONTEXTS
Table 8: Impact of using context in the LLMQuery the accuracy and logical consistency of simple facts in FreebaseLFC dataset. Results are for the base model without fine-tuning.
Accuracy
Logical Consistency
Model
Dataset
Without Context
With Context
Without Context
With Context
Llama2-7B
FreebaseI.FC
0.51
0.88
0.03
0.77
Table 8 shows the effectiveness of using a context on the accuracy and logical consistency of Llama2-7B for simple facts. Adding context provides the LLM premise answer the subse- quent fact. As such, Llama2-7B base model demonstrates a higher accuracy and logical consistency in the presence of context, and hence, in all our experiments we have provided KG context the LLMQuery.
G.3 IMPACT OF CONTEXT LENGTH
The inference performance of LLMs is affected by the length of the context of the underlying prompt. In our study, we vary the context length of the KG context using 1000 (default) and 200 tokens and report the performance of logical consistency and accuracy before and after fine-tuning in the FreebaseLFC dataset (Table 9). Limited context results in an improved accuracy the base model (before fine-tuning). However, the consistency is in-general unaffected due to reduced con- text length. Upon fine-tuning the model as proposed in the paper, both accuracy and consistency improve with reduced context length. Therefore, context length provides a useful knob to improve the accuracy of an LLM in answering fact-checking queries, whereas improve logical consistency, supervised fine-tuming is still needed.
G.4 IMPACT OF DIFFERENT CONTEXT RETRIEVAL METHODS
In addition to the BFS-based context retrieval method in $3.4, we consider two alternative retrieval methods to compare their effectiveness in providing the best context to the LLM and the resultant consistency the LLM in answering fact-checking queries.
BFS+ Vector embedding method. We compute 2-hop neighbours by BFS to extract rel- evant triplets the query. The relevant triplets first embedded using the Sentence Transformer (Reimers & Gurevych, 2019). Finally, we prune the irrelevant triplets by ap- plying nearest neighbour search on the embeddings and selecting the top-k relevant triplets
23

---

Published as a conference paper at ICLR 2025
Table Impact of reduced context length in FreebaseLFC dataset.
Accuracy
Logical Consistency
Model
Context length
Fact
Before FT
After FT
Before FT
After FT
P: p
0.82
0.98
0.66
0.96
1000 tokens
p q
0.45
0.83
0.70
0.84
Gemma-2B
p V q
0.73
0.94
0.91
0.90
p
0.92
1.00
0.84
0.99
200 tokens
p A q
0.50
0.98
0.63
0.97
V
0.75
0.93
0.90
0.93
with the highest relevance scores (w.r.t the query) construct the KG Context. Here, rel- evance scores are computed based on the T22 norm of the embedding vector for the query subtracted by the embedding vector for the triplet.
Vector Embedding Method. We bypass the BFS step, rather compute and index the em- beddings of all the triplets the KG, such as Freebase. Finally, we extract relevant triplets by applying nearest neighbour search the embeddings and selecting top-k triplets with the highest relevance scores (w.r.t. the query) to construct the KG Context.
Table 10: Impact different retrieval methods. Experiments performed simple facts from the FreebaseLFC dataset and on Llama2-7B model.
Accuracy
Logical Consistency
Retrieval Method
Before FT
After FT
Before FT
After FT
BF$ Relation hierarchy ($3.4)
0.88
0.97
0.76
0.94
BFS Vector Embedding
0.92
0.96
0.85
0.92
Vector Embedding (without BFS)
0.89
0.98
0.78
0.96
We observe that before fine-tuning, accuracy and consistency of BFS + embedding based retrieval better than that of retrieving context from relation hierarchy and ontology applied to BFS subgraph of 2-hop neighbours. addition, bypassing BFS and only applying vector embedding results similar performance as BFS relation hierarchy. Thus, vector embedding appears effective way to provide context to the L.M.
We further verify that the supervised fine-tuning improve both accuracy and consistency in all retrieval methods. Note that we reuse the fine-tuned model on context from BFS relation hierarchy and apply embedding based retrieval has a higher preprocessing time (almost 100x than the retrieval time based on relation hierarchy) embed the subgraph than relation hierarchy based retrieval. However, such vector embeddings can be performed offline.
data from the vector-embedding based retrieval methods. In preprocessing.
Handling Dynamically Varying Contexts. Real-world applications may need context retrieval to quickly adapt to new information, e.g., the knowledge graph evolving with time. In the case of evolving KGs, the "BFS Relation hierarchy" retrieval is computationally better alternative than embedding (without BFS)" method. Although the embedding- babeedding-based methods slightly improve the accuracy and consistency (as shown in Table 10), due to their higher pre-process processing overhead, they must be conducted offline repeatedly as the KG changes to obtain better entity embeddings.
retrieval methods such as "BFS + Vector Embedding" or "Vector Embedding
24

---

Published as a conference paper at ICLR 2025
G.5 RESULTS WITH -ERO,SHOT, 2-SHOT AND CHAIN-OF-THOUGHH PROMPTING
G.5.1 SUPERVISED FINE-TUNING VS ZERO-SHOT PROMPTING
We demonstrate the extended results on comparing instruction prompting with fine-tuning in Ta- ble 11. We find that fine-tuning is superior to instruction prompting on simple facts, across datasets and models.
Table 11: Impact of instruction prompting (Prompt) vs. fine-tuning (FT) on simple facts. Fine-tuning results in higher accuracy and consistency than instruction prompting.
Accuracy
Logical Consistency
Model
Dataset
Before FT
Prompt
After FT
Before FT
Prompt
After FT
FreebaseLFC NELLLFC
0.90
0.89
0.93
0.81
0.79
0.86
Llama2-
0.88
0.85
0.97
0.76
0.72
0.93
WikiLFC
0.96
0.94
0.96
0.92
0.90
0.93
FFecbaseL.CC
0.87
0.82
0.97
0.76
0.64
0.94
Llama2-7B
NELLLEC
0.71
0.61
0.94
0.44
0.23
0.88
WikiLFC
0.90
0.82
0.90
0.80
0.64
0.81
FFecbaseLFC
0.82
0.88
0.98
0.66
0.77
0.96
Gemma-28
NELLLFC
0.81
0.81
0.94
0.62
0.63
0.89
WikiLFC
0.76
0.69
0.90
0.52
0.43
0.80
G.5.2 2-SHOT PROMPTING
Table 12: Zero-shot, 2-shot and Chain-of-TThoueee (CoT) prompting results on Llama2-7B model.
Accuracy
Logical Consistency
Dataset
Before FT
0-shot
2-shot
CoT
After FT
Before FT
0-shot
2-shot
CoT
After FT
FreebaseLFC
0.87
0.89
0.78
0.93
0.97
0.76
0.79
0.58
0.86
0.94
In Table 3, showed that for our task, supervised fine-tuning produces better consistency and curacy compared to zero-shot prompts. We have also tested 2-shot prompts to understand whether providing examples (one positive and one negative example) to the LLM can improve its perfor- mance on fact-checking consistency (see Figure 5). The prompt is shown in the following text box along with the accompanying result in Table 12. the 2-shot prompt, we denote the example contexts in red color and the example facts in blue color.
We observe that 2-shot prompting performs worse than zero-shot prompting. This is due the rea- son mentioned in $4.1: There are nuances and intricacies in logical facts, and providing examples only adds to the complexity rather than helping the LLM understand the differences between true and false facts. Such results are not unprecedented in the literature. For instance, Zhao et al. (2021) showed that there are biases in GPT-3 and GPT-2, e.g., their tendency to output recent or common tokens which caused these models to produce drastically different accuracy various tasks, de- pending on the prompt format and ordering of examples the prompt. was also shown that in some of the eextclaasifiiattio task datasets, 2-shot prompting had worse performance than zero-shot prompting (see Table in Zhao et al. (2021)).
G.5.3 CHAIN-OF-THOUGII (CoT) PROMPTING.
We have created a chain-of-the oo-thuhtt (CoT) prompt to test whether providing CoT to the LLM can improve its performance fact-checking consistency (see Figure 6). Similar the 2-shot prompt, the CoT prompt contains a positive example and a negative example. A positive example one where the context contains the triplets in the logic query, and negative example is where the logic query is absent from the context. In addition to the examples, we also provide a reasoning behind the correct answer. Table 12 presents the result for CoT prompting.
25

---

Published as a conference paper at ICLR 2025
2-shot Prompt
((SS)) You are a helpful assistant. Always follow the instructions precisely and output the response exactly in the requested format. ((/SYS)
# Example 1:
Consider the context as set triplets where entries are separated by 1. Answer the question according to the context.
Context:
County Durham contains reverse England
Hartlepool contains reverse County Durham Hartlepool administrative parent County Durham
Question:
Hartlepool contains reverse
County Durham
Answer: Yes
# Example 2:
Consider the context as set triplets where entries are separated by 9. Answer the question according to the context.
Context:
County Durham contains reverse England
Hartlepool contains reverse County Durham Hartlepool administrative parent County Durham
Question: Hartlepool contains reverse
Cottam, East Riding Yorkshire
Answer: No
Test Context & Fact
Consider the context set of triplets where entries separated by ' Answer the question according the context.
Context: SContext
Question: SFact
Answer: YYes No}
Figure 5: Example of 2-shot prompt, where representative positive and negative examples are pro- vided to guide the LLM to answer the target fact.
As we compare the results of CoT with "After FT", we conclude that CoT does not demonstrate any improvement over fine-tuning our specific case. We hypothesize that achieving logical consis- tency requires an update in the internal weights of the LLM, such as via supervised fine-tuning, and simply providing examples is insufficient.
G.6 IMPACT OF LEARNING RATE
While fine-tuning Llama2-7B, we consider different learning rates (Figure 7). A higher learning rate such as x 10 quickly decreases training loss and results in an over-fitting as such, there is an increase in evaluation loss over epochs. On the other hand, lower learning rate 2 10-7 neither decreases training loss nor evaluation loss. Finally, both learning rates x 10 and 2 x balance between train/evaluation loss; and select 2x 10-5 as the final learning rate for subsequent
26

---

Published as a conference paper at ICLR 2025
Chain of Thought Prompt
(<SY))) You are a helpful assistant. Always follow the instructions precisely and output the response exactly in the roquested format. <(/SY))
Consider the context Answer the question according to the context.
of triplets where entries separated by 'I' symbol.
Hartlepool ladministrative children reverse County Durham County Durham contains reverse England
Hartlepool contains reverse County Durham
County Durham containedby reverse Durham University
Do not add additional text. Is the following triplet factually correct? Answer with Yes or No.
Hartlepool contains
County Durham
The provided triplet 'Hartlepool contains reverse County Durham' appears in the context on the ninth line. The answer is Yes.
Consider the context Answer the question according the context.
of triplets where entries separated by 'I' symbol.
Hartlepool administrative children reverse County Durham County Durham contains reverse England
Hartlepool contains reverse County Durham County Durham containedby reverse Durham University
Do not add additional text. Is the following triplet factually correct? Answer with Yes or No.
Hartlepool
contains reverse Cottam, East Riding of Yorkshire
The provided triplet 'Hartlepool contains reverse Cottam, East Riding of York- shire' does not appear in any line of the context. The answer is No.
Test Context & Fact
Consider the context as a set of triplets where entries are separated by 'I' symbol. Answer the question according to the context.
SContext
Do not add additional text. Is the following triplet factually correct? Answer with Yes or No.
SFact
[Reasoning followed by No answer]
Figure 6: Example of chain of thought (CoT) prompt, where representative positive and negative examples are followed by CoT reasoning and answers.
experiments in Llama2-7B due to achieving lower evaluation loss. Therefore, learning rate provides precise control over generalization performance in fine-tuning.
G.7 ACCURACY AND LOGICAL CONSISTENCY ACROSS EPOCHS IN FINE-TUNING.
We demonstrate how accuracy and logical consistency vary across fine-tuning epochs in Llama2- 13B (Figure 8), Llama2-7B (Figure 9), and Gemma-2B (Figure 10). We present results on Freeba- seLFC dataset, which are used in training, and on NELLLFC dataset to understand the generalization in fine-tuning. In general, both accuracy and consistency increase increase the epochs across
27

---

Published as a conference paper at ICLR 2025
Learning Rate
1.4
Learning Rate
1.3
2e-07
Training Loss
1.2
2e-07
1.2
Evaluation
2o-05
2c-05
0.0002
0.0002
0.9
0.2
Epoch
Epoch
Figure 7: Impact of different learning rates in fine-tuning Llama2-7B model. Higher learning rate results in over-fitting and degraded performance on the evaluation data.
different fact types: simple fact p, p, and complex facts PA and pv q. In most cases, accuracy and consistency surpass their initial values the base models (shown in dashed horizontal lines). Fur- ther, accuracy and consistency are often correlated across epochs. Exceptionally in LLama2 models, accuracy and consistency occasionally decreases for v q fact, whereas Gemma-2B does hibit this phenomenon. In summary, both accuracy and logical consistency improve over epochs in fine-tuning across models and datasets.
Llama2 13B PPeeboseLPC wish
Llamaā 13B NELLLPC with
0.9
Accuracy
Accuracy
Logical
0.9
Logical Consistency
Consistency
0.7
0.6
0.7
0.5
0.6
0.3
0.4
Epoch
Epoch
Llama2-13B FreebaseL.PC p.gg
Elama3-13B (B NELLLEG fact p.A
Accuracy
0.8
Consistency
Logical
0.7
0.8
Logical
Consistency
0.5
0.4
0.4
0.3
Epoch
Epoch
Lismx2-198 on FrwbeeLLC with p
Lhumd-1555 om NELLLEG with fudd V
0.95
Accuracy Logical
Accuracy
0.9
Logical
Consistency
Consistency
0.85
0.75
Epoch
Epoch
Figure 8: Accuracy and logical consistency of evaluation data across fine-tuning epochs in Llama2- 13B. The left row denotes results on FreebaseLFC (included in training). while right row denotes results on NELLLFC. Dashed line denotes accuracy and consistency before fine-tuning
28

---

Published as a conference paper at ICLR 2025
Lisma2-7B on Frocbael.FCC wilh fact p5,
Llama2-88 on NELLLPC with Back ps
Accuracy
Accuracy
0.9
Logical
0.9
Lagical
Consistency
0.8
Consistency
0.8
0.7
0.6
0.6
0.5
0.5
Epoch
Epoch
Llama2-7B on Prochasel.FCC with p./g
Llama2-7D NELILPC wish fact A.g
0.9
Accuracy Logical
0.9
Accuracy
0.8
Logical
Consistency
Consistency
0.7
0.6
0.6
0.5
0.3
0.4
12
Epoch
Epoch
Llama2-BB PFeebasel.FC with fact
Llama3-7B NELLLPC with P
0.95
Accuracy Logical
Accuracy
Logical
Consistency
0.85
Consistency
0.75
0.7
0.65
12
2
Epoch
Epoch
Figure 9: Accuracy and logical consistency of evaluation data across fine-tuning epochs Llama2- 7B. The left row denotes results on Freebasel.FC (included in training), while the right row denotes results on NELLLFC. Dashed line denotes accuracy and consistency before fine-tuning.
29

---

Published as a conference paper at ICLR 2025
Gemma-28 un Freeswed.55C with fack
Gemma-28 on NELLLFF with fact p4:
Accuracy
0.95
Accuracy
0.95
Logical
Logical
0.9
Consistency
0.85
Consistency
0.8
0.75
0.7
0.7
0.65
Epoch
Epoch
Cemma-20 on PreebaseL C with fact 9.A.
Gemmo-2B NELLLFC with p
Accuracy Logical
Accuracy
Logical
Consistency
Consistency
0.7
0.6
0.5
0.5
10
Epoch
Epoch
Gemma-2B PF with fact
Gemma-2B NELLEPC with fact
Accuracy
0.95
Accuracy
0.9
Consistency
Logical
0.9
Logical
Consistency
0.85
0.75
0.7
0.7
2
10
12
10
Epoch
Epoch
Figure 10: Aceuracy and logical consistency of evaluation data across fine-tuning epochs in Gemma- 2B. The left row denotes results on Freebasel.FC (included in training), while the right row denotes results on NELLLFC. Dashed line denotes accuracy and consistency before fine-tuning.
30

---

Published as a conference paper at ICLR 2025
H GENERALIZABILITY TO COMPLEX FACTS AND RULES
Table 13: Generalization of fine-tuning complex facts and rules (Extended Table 2).
Accuracy
Logical Consistency
Model
Dataset
Fact
Before FT
After FT
Before FT
After FT
(g
0.74
0.85
0.78
0.81
PA PV q
0.53
0.82
0.63
0.79
0.73
0.76
0.85
0.98
Freebasel.Fe
LJama2-13B
AAP
0.61
0.93
0.84
0.91
(g PA (q pVg+qvD
0.67
0.77
0.72
0.80
0.42
0.38
0.75
0.85
NELLLFC
0.73
0.76
0.80
0.98
pAgOgA
0.38
0.70
0.90
0.85
PV (g
0.68
0.74
0.78
0.71
PA (gvr
0.42
0.74
0.54
0.75
Freebasel.I
PVg qVgqv
0.78
0.84
0.87
0.85
Llama2-7B
PAgegAL
0.39
0.89
0.91
0.92
PV (gA PA (V
0.63
0.70
0.94
0.76
0.37
0.66
0.80
0.79
NELLLFC
PVg QVgOqVP
0.75
0.78
0.98
0.82
PAge
0.26
0.81
0.99
0.95
PV (g
0.62
0.83
0.91
0.77
PA (g T)
0.38
0.74
0.77
FreebaseLFC
Dvg-qva
0.73
0.94
0.90
0.93
Gemma-2B
RAgOI 1999000
0.45
0.83
0.80
0.90
PV (g PA (q V
0.62
0.81
0.97
0.80
0.37
0.66
0.82
0.80
NELLLEC
PVQ +
0.75
0.88
0.99
0.93
PA pAg ~ AAP
0.33
0.83
0.92
0.90
H.1 DE-MORGAN'S LAWS
We have tested the generalizability of the fine-tuned models on more challenging logic rules such as De-Morgan's laws. order to finetune Llama2-13B, we have incorporated in the training datasets simple facts, conjunction, disjunction, first-order logic rules, law of syllogism, and a small number of De Morgan's 's law rules. The results are shown in Table 14.
We observe that before any fine-tuning, the accuracy of De-Morgan's 's law on conjunctive (-(p ^ 4) ->> v -ạ) and disjunctive (-(p v ?) -p A -q) facts the same as random guess model while being highly consistent. This is because the base Llama2-133 model classifies each fact (both sides of ++) as true, resulting in a consistent yet inaccurate fact-checker. Upon fine-tuning, the accuracy both conjunctive and disjunctive fact-types improves significantly, while retaining similar performance in logical consistency. Therefore, our experiments demonstrate the generality of applying supervised fine-tuning in improving both logical consistency and accuracy in logical fact-checking across multiple fact types, including De Morgan's 's law.
H.2 FIRST-ORDER LOGIC AND LAW OF SYLLOGISM
We demonstrate results on first-order logic (FoL) and law syllogism (LoS) queries when con- sidering only proposition logic queries for fine-tuning in Table 15 and all propositional logic, FoL, and LoS queries for fine-tuning in Table 16. When all queries are considered, we observe higher accuracy and logical consistency unseen Fol. and LoS queries.
31

---

Published as a conference paper at ICLR 2025
Table 14: Evaluating Generalizability of fine-tuned Llama2-13B using De-Morgan's law before and after fine-tuning. Experiments are conducted on FreebaseLFCC dataset.
Accuracy
Logical Consistency
Fact
Before FT
After FT
Before
After FT
4
0.25
0.99
1.00
0.98
A 9)
4
0.75
0.97
0.99
0.95
Table 15: Performance on more complex logic settings: first-order logic (FoL) with an existential quantifier (3) and the law of syllogism (LoS), curated from the FreebaseLFC benchmark. FT results are on propositional logic benchmark.
Accuracy
Logical Consistency
Model
Dataset
Fact
Before FT
After FT
Before FT
After FT
FoL
0.57
0.66
0.13
0.37
Llama2-13B
Freehasel
LoS
0.68
0.95
0.73
0.91
FoL
0.50
0.61
0.00
0.45
Llama2-7B
FreebaseL.FC
LoS
0.55
0.77
0.45
0.55
FoL. LoS
0.50
0.57
0.00
0.80
Gemma-2B
FreebaseLFC
0.55
0.91
0.76
0.82
I PERFORMANCE ON NATURAL TEXT FACT-CHECKING DATASET
It is important assess how well the proposed method translates to real-world situations where queries are of textual nature. In particular, does the supervised fine-tuning still generalize to textual data as opposed triple context and query? We experiment with widely known real-world fact- checking benchmark containing textual facts: FEVER (Fact Extraction and VERification) (Thorne al., 2018).
FEVER consists of 185,445 4444 claims generated by altering sentences extracted from Wikipedia, which subsequently verified without knowledge of the sentence they were derived from. The claims are classified supported, refuted, not-enough-info. We consider supported and refuted claims as simple query p and manually create natural language negation of each claim (-p) assess for logical consistency. Our augmented FEVER dataset contains 580 supported claims and 220 refuted claims, totaling 580 220 = 800 simple claims. We split these 800 simple claims into 400-200-200 for training, validation, and test, respectively, to perform supervised fine-tuning.
We adopt the Gemma-2B-it model (Team et al., 2024) as proof of concept. order to create the context, retrieve 5 most relevant claims to the given query claim using vector embedding retrieval method. Table 17 shows that our method improving logical consistency also works well in real-world situations where both queries and context to answer the queries are in natural text. In
Table 16: Performance on more complex logic settings: first-order logic (FoL) with an existential quantifier (3) and the law of syllogism (LoS), curated from the Freebasel. FFC benchmark. FT results are on propositional logic, FoL, and LoS queries.
Accuracy
Logical Consistency
Model
Dataset
Fact
Before FT
After FT
Before FT
After FT
FoL.
0.57
0.94
0.13
0.88
Llama2-13B
FreebaseL.FC
LoS
0.68
0.97
0.73
0.94
Fol.
0.50
0.91
0.00
0.86
Llama2-7B
FreebaseLFC
LoS
0.55
0.98
0.45
0.97
Fol.
0.50
0.69
0.00
0.69
Gemma-2B
FreebaseLFC
LoS
0.55
0.83
0.76
0.67
32

---

Published as a conference paper at ICLR 2025
summary, the LLM (prior fine-tuning) still lacks logical consisteney and accuracy natural text facts. Our supervised tinettnning improves both the logical consistency and accuracy the LLM and hence also generalizes to textual data.
Table 17: Improvement in accuracy and consistency on the FEVER dataset
Accuracy
Logical Consistency
Model
Fact
Before FT
After FT
Before FT
After FT
Gemma-2B
Pp
0.73
0.95
0.95
0.96
J EXPERIMENTAL RESULTS ON ADDITIONAL BASELINE: MINICHECK
We present the results by considering state-of-the-art fact-checking methods using KGs such as MiniCheck (Tang et al., 2024) in Table 18. The key technique employed in MiniCheck is to use GPT-4 and then construct synthetic training data having challenging instances and factual errors. Training on this data teaches the model, MiniCheck-FTSs to recognize complex information across input claim and check each fact it. When compared against our fine-tuned version of the Llama2-7B model over the FreebaseL.FC dataset, we find that Llama2-7B shows better accuracy and consistency than MiniCheck-FTS. The reason behind MiniCheck's 's underwhelming performance is that it is designed for fact-checking grounding documents, not for verifying propositional logic queries unlike ours, neither to ensure whether the trained LLM is logically consistent in these facts not.
Table 18: Experimental results MiniCheck
Accuracy
Consistency
Facts
MiniCheck-FTS Fine-tuned I Llama2-7B
MiniCheck-FT5
Fine-tuned I. Llama2-7B
P, p
0.57
0.97
0.13
0.94
P g
0.83
0.87
0.84
0.86
p g 0.39
0.81
0.41
0.77
K
EXPERIMENTS WITH LARGER AND CLOSED-SOURCE MODELS
K.1 LLAMA2-70B
In this section, we present experimental results on models with a larger number of parameters, such as Llama2-70B The experiments on Llama2-70B are presented in Table 19, Table 20 and Table 21.
From Table 19 and Table 20, observe that despite having more parameters than the correspond- ing 7B and 13B versions of LLaMA2, the Llama2-70B base model does not always demonstrate an improved logical consistency in all query types: simple facts, complex facts, and logic rules, demanding further supervised fine-tuning or instruction prompting. Finc-tuning of Llama2-70B is incomplete due to its higher demand on computational resources.
Llama2-708 demonstrates higher effectiveness in understanding instruction prompts than the lower- size models from the same family. For example, in Table 21, we observe that Llama2-700 improves accuracy from 0.77 to 0.99 and consistency from 0.53 to 0.99 due instruction prompting, which superior to Lllam2-133B. As such, we hypothesize that fine-tuning may not necessary on larger models, which already demand high computational resources; instruction prompting may be com- petitive as fine-tuning.
K.2 GPT-40
In this section, present experimental results GPT-4o. Table 22 presents the results simple facts and complex facts and logic rules on the Freebase dataset. In Table 23, we present the results on more complex logic rules such as the law of syllogism and first-order logic.
33

---

Published as a conference paper at ICLR 2025
Table 19: Comparison among models of different sizes regarding aceuracy and logical consistency of LLMs before and after fine-tuning (FT). Bold numbers denote an improved result in accuracy and consistency for fine-tuning. Training is performed on FreebaseLFCC dataset only (marked with '*'), while performance improved in all datasets. "_" denotes incomplete fine-tuning due to computa- tional resources.
Accuracy
Logical Consistency
Model
Dataset
Fact
Before FT
After FT
Before FT
After FT
0.77
0.53
Ilama2-708
FreebaseLFC*
P q
0.91
0.86
p q
0.32
0.62
p. mp
0.90
0.93
0.81
0.86
Llama2-13B
FreebaseLFC*
p ^
0.61
0.93
0.67
0.83
p v
0.73
0.76
0.73
0.97
p: -p
0.87
0.97
0.76
Llama2-78
FreebaseLFC*
p A
0.39
0.87
0.47
0.86
p
0.78
0.81
0.83
0.77
p 0.82
0.98
0.66
0.96
Gemma 2B
FreebaseLFC*
0.45
0.83
0.70
0.84
p 0.73
0.94
0.91
0.90
Table 20: Comparison among models of various sizes regarding their generalizability to complex facts and rules.
Accuracy
Logical Consistency
Model
Dataset
Fact
Before FT
After FT
Before
After FT
PV (q PA (q V
0.38
0.70
0.70
0.90
Llama2-70B
Freebasel FC
PV + qVP
0.32
0.94
PAQ
0.91
0.93
PV (T V
0.74
0.85
0.78
0.81
0.53
0.82
0.63
0.79
Llama2-18B
FreebaseLFCC
P ovqey qVp
0.73
0.76
0.85
0.98
P pAg+aga
0.61
0.93
0.84
0.91
PV (
0.68
0.74
0.78
0.71
PA (g
0.74
0.54
0.75
Llama2-7B
FreebaseLFC
pV AAg+-vp
0.78
0.84
0.87
0.85
PAq+aAP
0.39
0.89
0.91
0.92
PV (g
0.62
0.83
0.91
0.77
P V
0.38
0.74
0.77
0.81
Gemma-2B
Freebasel.FF LFC
P p
0.73
0.94
0.90
0.93
P pageaAr
0.45
0.83
0.80
0.90
From table 22, we observe that GPT-4o demonstrates improved logical consistency on negation, conjunction, and disjunction facts, but performs comparatively poorly when multiple logical opera- tors are involved, such as p (q r) and v (q r) the accuracy and consistency reduce to (0.84, 0.81) and (0.76, 0.87), respectively. Considering the API cost of rumning GPT-4o, we consider a smaller number of queries (around 1000 queries per query type). resulting in a total cost of 45 US dollars.
The results in Table 23 suggest that GPT-4o lacks logical consistency complex logical expressions or higher-order Horn rule settings, justifying the motivation of our work, that is, to measure and improve the logic consistency of LLMs.
Finally, we proceeded to the instruction prompt for GPT-4o to improve its consistency on simple facts. We find that the logical consistency improves to 0.98, shown in table 24.
34

---

Published as a conference paper at ICLR 2025
Table 21:
Impact of instruction prompting (Prompt) vs, fine-tuning (FT) on simple facts across
models of various sizes.
Accuracy
Logical Consistency
Model
Dataset
Before FT
Prompt
After FT
Before FT
Prompt
After FT
Llama2-700 70B
FreebaseLFCC
0.77
0.99
0.53
0.99
Llama2- 13B
FreebaseLFC
0.90
0.89
0.93
0.81
0.79
0.86
Llama2-1
FreebaseLFC
0.87
0.82
0.97
0.76
0.64
0.94
Gemma-28
FreebaseLFC
0.82
0.88
0.98
0.66
0.77
0.96
Table 22: Logical consistency and accuracy of GPT-4o model (before fine-tuning) across different facts and logic rules in FreebaseLLCC benchmark.
Fact
Accuracy
Logical Consistency
P. p
0.91
0.91
y
0.97
0.95
p q
0.97
0.95
P A Q + q Ap
0.96
0.95
p v (q A r)
0.76
0.87
PA (q v
0.84
0.81
Comparing GPT-40 with smaller LLMs. We compare GPT-4o with smaller models such as Gemma-2B 2B. Llama2-7B, and Llama2-13B on FoL and LoS queries in Table 23. We observe that the base 2B, 7B, and 13B models do not offer similar levels accuracy and consistency on first-order logic. This is not surprising, since GPT-4o has been exposed structured logie (Hurst al., 2024). We also observe that Llama2-13B and Gemma-2B have better consistency than GPT-4o on the LoS dataset, but their accuracy is lacking in comparison. is plausible for (base) model demon- strate higher consistency yet lower accuracy since consistency indicates the logical robustness of the model on logically manipulated queries independent from the matter of accuracy. In our attempt to further improve GPT-4o, we have conducted instruction prompting with simple facts. The results are shown Table 24. This result indicates that since fine-tuning is often infeasible in large closed- source models like GPT-4o, instruction prompting becomes effective in improving both accuracy and consistency to 98%.
Table 23: Performance of the LLMs on more complex logic settings: first-order logic with an exis- tential quantifier (3) and the law of syllogism curated from the FreebaseLFC benchmark.
Fact
Models
Accuracy
Logical Consistency
GPT-4o
0.91
0.78
Llama2-
13BB
0.57
0.13
First-order Logic (FoL.
Llama2-7B
0.50
0.00
Gemma- 2B
0.50
0.00
GPT-4o
0.94
0.64
Llama2-133
0.68
0.73
Law of Syllogism (LoS)
Llama2-7B
0.55
0.45
Gemma- 2B
0.55
0.76
35

---

Published as a conference paper at ICLR 2025
Table 24:
Impact of instruction prompting (Prompt) on GPT-4o in improving the accuracy and logical consistency of simple facts in FreebaseLFC benchmark.
Accuracy
Logical Consistency
Before FT
Prompt
Before FT
Prompt
0.91
0.98
0.91
0.98
0.945
SCIQ Accuracy
Accuracy
0.94
0.945
0.935
0.94
0.925
0.935
0.92
10
Epoch
Epoch
(a) Gemma-2B
(b) Llama2-7B
Lambada Accuracy
0.53
Lambada Accuracy
0.69
0.51
0.5
0.48
0.68
0.47
Epoch
Epoch
(c) Gemma-2B
(d) Ilama2-7B
0.47
MMLU Accuracy
MMLU Accuracy
0.37
0.46
0.36
0.45
0.34
0.44
0.33
Epoch
Epoch
(e) Gemma-2 2B
Llama2-7B
Hellaswag Accuracy
0.495
Hellaswag Accuracy
0.58
0.49
0.575
0.485
0.57
0.48
0.475
0.56
0.47
Epoch
Epoch
(g) Gemma-2B
(h) Llama2-7B
Figure 11: Performance on Language benchmarks
L PERFORMANCE ON LANGUAGE BENCHMARKS
We conduct an evaluation of the fine-tuned SFT model four general language benchmarks:
SCIQ (Welbl et al., 2017). Lambada (Paperno et al., 2016), MMLU (Hendrycks et al., 2020), and Hellaswag (Zellers et al., 2019). Figure 11 shows the performance across epochs.
36

---

Published as a conference paper at ICLR 2025
We observe that the logical consistency- based fine-tuning improves the benchmark accuracy SCIQ (from 0.926 to 0.94 in Gemma-2B and 0.94 to 0.945 in Llama2-7B), which is a science exam question-answer benchmark. In addition, the Lambada benchmark, the performance increases in the initial few epochs. In the rest of the benchmarks, MMLJ and Hellaswag, increasing fine-tuning epochs results in monotonic decrease in benchmark accuracy with epochs.
Therefore, consistently across multiple models like Gemma and Llama, consistency- -aasseeenc--b-bed fine- tuning is helpful in the scientific domain benchmark. At the same time, performance degradation may be observed in other multi-task abilities with excessive fine-tuning. Therefore, as evident from our experiments, we suggest applying consistency-foeut SFT for few epochs before the model overfits and loses its generalization performance on other benchmarks.
M ACKNOWLEDGMENT
AK and SH acknowledge support from the Novo Nordisk Foundation grant NNF 220C0072415.
37


Reviews:
[{"review_id": "m67BDnF0rg", "reviewer": "Reviewer_pZwi", "summary": "This paper presents a method for bench marking LLMs for logical consistency and ways to improve the logical consistency of LLMs through a supervised fine tuning using the fine tuning data. Paper shows how LLMs perform on different logical consistency checks with triples given a context to answer some logical queries and show the effectiveness of improving them with SFT. They propose three benchmark datasets based on three different knowledge bases and show how SFT improves across benchmarks.", "questions": "After SFT, the models still perform as good on the original language benchmarks as the original model?\nWas the goal of the peper to produce a model that does logical consistency in KG RAG kind of setting or in general to improve the logical consistency of LLMs ? if its later I don't see any experiments around other evaluation to show that.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall benchmarks for checking logical consistency for a given triple context and logical query are good and can help in evaluating logical consistency of LLMs.\nEmpirical results showing zero shot/prompt and fine tuning results show the improvements across benchmarks.", "weaknesses": "Experiments are more on the benchmark created and not sure how well they translate to real world situations or places where queries are in textual nature and does he supervised model still be able to generalize to textual data as opposed to triple context?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ht7PEujhQX", "reviewer": "Reviewer_Eq6F", "summary": "This paper discusses the topic of the logical consistency of LLM when answering complex queries that can be expressed by propositional logic and knowledge extracted from the knowledge graph. The logical consistency is defined by whether the model can answer the logical equivalent query with the same answer. To this end, the author first showcases some examples of non-consistency in query answering of LLM and then uses finetuning instead of prompting to improve the logical consistency of LLM.", "questions": "Similar with weakness, I wonder whether the logical consistency helps with the reasoning ability of LLM?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe definition of logical consistency is strict and the paper constructs three new datasets for finetuning and evaluating the consistency of LLM.\n2.\tThe construction of the LLMQUERY seems interesting and the author proposes many optimizations towards that.\n3.\tThe experiments show that the logical consistency can be improved by finetuning but not prompting.", "weaknesses": "1.\tThough being logical consistent is a desirable attribute of LLM, we are also interested in whether it retrieve the correct answer. The logical consistency is only computed by whether the model gives the consistent answer with itself, but whether the answers are true or false are neglected in the experiment.\n2.\tThe writing of this paper can be further improved. For example, the Proposition 3 is trivial and kind of redundant. There are also some mistakes like in Line 346 ``more computationally cheaper’’.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zU4IyY97wR", "reviewer": "Reviewer_7P9X", "summary": "This paper tackles the significant issue of logical consistency in responses generated by large language models (LLMs). It emphasizes a gap in current research, where consistency assessments focus primarily on simple paraphrasing rather than complex logical reasoning. The authors address this by examining how LLMs handle propositional logic (involving operators such as negation, conjunction, and disjunction) within a fact-checking framework supported by knowledge graphs (KGs).", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "There are three main contributions of this paper.\n1) Novel Dataset and Benchmarking: Three logical fact-checking datasets FreebaseLFC, NELLLFC, and WikiLFC, derived from knowledge graphs.\n2) Evaluating LLMs on these new fact-checking datasets. The paper includes a variety of experimental setups, including comparisons of zero-shot instruction prompting and supervised fine-tuning, adding depth to the evaluation of LLMs' logical consistency. Results show that existing LLMs are not consistent when considering\nlogical equivalents or variants of these facts.\n3) Improving the consistency of LLMs via supervised fine-tuning", "weaknesses": "1)  While the paper presents methods for extracting relevant KG context through BFS and embedding-based retrieval, it is unclear how effective these methods are in dynamically varying real-world contexts. Expanding this section could strengthen the applicability of the approach.\n2) The paper suggests fine-tuning as a solution to improve logical consistency but does not deeply discuss the associated computational overheads and limitations, which may impact the scalability of this approach for large datasets or diverse domains.\n3) The paper mainly focuses on binary responses, which may limit the generalizability of the method in complex domains. Exploring extensions to multi-class logic and non-binary answer consistency could open avenues for broader applications. Note that a fact might not be binary, e.g., Crimean is a part of Russian. The answer is nether yes or no, but depends on who claims it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JmSXyns9DF", "reviewer": "Reviewer_JETn", "summary": "This paper focuses on the logical consistency of LLMs in fact-checking using KGs. The authors first formalize the definition of logical consistency for LLMs and concentrate on retrieval-augmented LLMs within a fact-checking task. They propose three benchmarks, FreebaseLFC, NELLLFC, and WikiLFC to demonstrate that several existing LLMs lack logical consistency when handling complex queries, and employ supervised fine-tuning to enhance the logical consistency of these models.", "questions": "1. Have the authors considered more complex logical expressions or some higher-order Horn rules?\n\n2. The tables in the paper are not formatted as three-line tables, which appears somewhat unusual.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n\n2. The logical consistency evaluation of LLMs is rational and interesting.\n\n3. The employed supervised fine-tuning method on LLMs has enhanced the logical consistency performance.", "weaknesses": "1. The authors may want to provide more results across various scales of LLMs (including both smaller models such as Vicuna-68M and larger models like LLaMA2-70B) to comprehensively evaluate the logical consistency issue.\n\n2. The authors may want to include more results from existing fact-checking methods using KGs, such as MiniCheck [1] or RefChecker [2], on the proposed benchmarks.\n\n[1] Minicheck: Efficient fact-checking of llms on grounding documents. EMNLP, 2024.\n\n\n[2] Refchecker: Reference-based fine-grained hallucination checker and benchmark for large language models. 2024.\n\n3. The third contribution, \"Improvement of Logical Consistency,\" appears somewhat underwhelming. In Table 1, Vanilla LLaMA-2-13B already achieved strong results before fine-tuning, which leads me to wonder whether models like GPT-4o, GPT-o1 exhibit lower levels of logical inconsistency which may be hardly noticeable.\n\n4. The authors may want to discuss and summarize more LLMs with KG contexts methods in related work including [3] to provide a more comprehensive insight.\n\n[3] Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Models. ICML2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)