Logical Consistency of Large Language Models in Fact-Checking

Authors: Bishwamittra Ghosh, Sarah Hasan, Naheed Anjum Arafat, Arijit Khan
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, Logical consistency, Fact-checking in Knowledge graphs

Abstract:
In recent years, large language models (LLMs) have demonstrated significant success in performing varied natural language tasks such as language translation, question-answering, summarizing, fact-checking, etc. Despite LLMs’ impressive ability to generate human-like texts, LLMs are infamous for their inconsistent responses – a meaning-preserving change in the input query results in an inconsistent response and attributes to vulnerabilities of LLMs such as hallucination. Consequently, existing research focuses on simple paraphrasing-based consistency assessment of LLMs, and ignores complex queries that necessitate an even better understanding of logical reasoning by an LLM. *Our work therefore addresses the logical inconsistency of LLMs under complex logical queries with primitive logical operators, e.g., negation, conjunction, and disjunction.* As a test bed, we consider retrieval-augmented LLMs on a fact-checking task involving propositional logic queries from knowledge graphs (KGs). Our contributions are three-fold. **Benchmark:** We introduce three logical fact-checking datasets over KGs for community development towards logically consistent LLMs. **Assessment:** We propose consistency measures of LLMs on propositional logic queries and demonstrate that existing LLMs lack logical consistency, especially on complex queries. **Improvement:** We employ supervised fine-tuning to improve the logical consistency of LLMs on the complex fact-checking task with KG contexts. We have made our source code and benchmarks available.

Decision: accept (poster)
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729184196977

thread_type:
official_review

<thread_object>
creator: reviewer_jetn

<content>
summary:
This paper focuses on the logical consistency of LLMs in fact-checking using KGs. The authors first formalize the definition of logical consistency for LLMs and concentrate on retrieval-augmented LLMs within a fact-checking task. They propose three benchmarks, FreebaseLFC, NELLLFC, and WikiLFC to demonstrate that several existing LLMs lack logical consistency when handling complex queries, and employ supervised fine-tuning to enhance the logical consistency of these models.

strengths:
1. This paper is well-written and easy to follow.

2. The logical consistency evaluation of LLMs is rational and interesting.

3. The employed supervised fine-tuning method on LLMs has enhanced the logical consistency performance.

weaknesses:
1. The authors may want to provide more results across various scales of LLMs (including both smaller models such as Vicuna-68M and larger models like LLaMA2-70B) to comprehensively evaluate the logical consistency issue.

2. The authors may want to include more results from existing fact-checking methods using KGs, such as MiniCheck [1] or RefChecker [2], on the proposed benchmarks.

[1] Minicheck: Efficient fact-checking of llms on grounding documents. EMNLP, 2024.


[2] Refchecker: Reference-based fine-grained hallucination checker and benchmark for large language models. 2024.

3. The third contribution, "Improvement of Logical Consistency," appears somewhat underwhelming. In Table 1, Vanilla LLaMA-2-13B already achieved strong results before fine-tuning, which leads me to wonder whether models like GPT-4o, GPT-o1 exhibit lower levels of logical inconsistency which may be hardly noticeable.

4. The authors may want to discuss and summarize more LLMs with KG contexts methods in related work including [3] to provide a more comprehensive insight.

[3] Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Models. ICML2024.

questions:
1. Have the authors considered more complex logical expressions or some higher-order Horn rules?

2. The tables in the paper are not formatted as three-line tables, which appears somewhat unusual.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730490217456

thread_type:
official_review

<thread_object>
creator: reviewer_7p9x

<content>
summary:
This paper tackles the significant issue of logical consistency in responses generated by large language models (LLMs). It emphasizes a gap in current research, where consistency assessments focus primarily on simple paraphrasing rather than complex logical reasoning. The authors address this by examining how LLMs handle propositional logic (involving operators such as negation, conjunction, and disjunction) within a fact-checking framework supported by knowledge graphs (KGs).

strengths:
There are three main contributions of this paper.
1) Novel Dataset and Benchmarking: Three logical fact-checking datasets FreebaseLFC, NELLLFC, and WikiLFC, derived from knowledge graphs.
2) Evaluating LLMs on these new fact-checking datasets. The paper includes a variety of experimental setups, including comparisons of zero-shot instruction prompting and supervised fine-tuning, adding depth to the evaluation of LLMs' logical consistency. Results show that existing LLMs are not consistent when considering
logical equivalents or variants of these facts.
3) Improving the consistency of LLMs via supervised fine-tuning

weaknesses:
1)  While the paper presents methods for extracting relevant KG context through BFS and embedding-based retrieval, it is unclear how effective these methods are in dynamically varying real-world contexts. Expanding this section could strengthen the applicability of the approach.
2) The paper suggests fine-tuning as a solution to improve logical consistency but does not deeply discuss the associated computational overheads and limitations, which may impact the scalability of this approach for large datasets or diverse domains.
3) The paper mainly focuses on binary responses, which may limit the generalizability of the method in complex domains. Exploring extensions to multi-class logic and non-binary answer consistency could open avenues for broader applications. Note that a fact might not be binary, e.g., Crimean is a part of Russian. The answer is nether yes or no, but depends on who claims it.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711102095

thread_type:
official_review

<thread_object>
creator: reviewer_eq6f

<content>
summary:
This paper discusses the topic of the logical consistency of LLM when answering complex queries that can be expressed by propositional logic and knowledge extracted from the knowledge graph. The logical consistency is defined by whether the model can answer the logical equivalent query with the same answer. To this end, the author first showcases some examples of non-consistency in query answering of LLM and then uses finetuning instead of prompting to improve the logical consistency of LLM.

strengths:
1.	The definition of logical consistency is strict and the paper constructs three new datasets for finetuning and evaluating the consistency of LLM.
2.	The construction of the LLMQUERY seems interesting and the author proposes many optimizations towards that.
3.	The experiments show that the logical consistency can be improved by finetuning but not prompting.

weaknesses:
1.	Though being logical consistent is a desirable attribute of LLM, we are also interested in whether it retrieve the correct answer. The logical consistency is only computed by whether the model gives the consistent answer with itself, but whether the answers are true or false are neglected in the experiment.
2.	The writing of this paper can be further improved. For example, the Proposition 3 is trivial and kind of redundant. There are also some mistakes like in Line 346 ``more computationally cheaper’’.

questions:
Similar with weakness, I wonder whether the logical consistency helps with the reasoning ability of LLM?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712519365

thread_type:
official_review

<thread_object>
creator: reviewer_pzwi

<content>
summary:
This paper presents a method for bench marking LLMs for logical consistency and ways to improve the logical consistency of LLMs through a supervised fine tuning using the fine tuning data. Paper shows how LLMs perform on different logical consistency checks with triples given a context to answer some logical queries and show the effectiveness of improving them with SFT. They propose three benchmark datasets based on three different knowledge bases and show how SFT improves across benchmarks.

strengths:
Overall benchmarks for checking logical consistency for a given triple context and logical query are good and can help in evaluating logical consistency of LLMs.
Empirical results showing zero shot/prompt and fine tuning results show the improvements across benchmarks.

weaknesses:
Experiments are more on the benchmark created and not sure how well they translate to real world situations or places where queries are in textual nature and does he supervised model still be able to generalize to textual data as opposed to triple context?

questions:
After SFT, the models still perform as good on the original language benchmarks as the original model?
Was the goal of the peper to produce a model that does logical consistency in KG RAG kind of setting or in general to improve the logical consistency of LLMs ? if its later I don't see any experiments around other evaluation to show that.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732111480354

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comment and summary of changes

comment:
We thank the reviewers for their valuable feedback to improve our work. We have uploaded a revised draft in which we made changes (in blue color) to address the concerns of the reviewers. In the following, we summarize the key changes. 

- **Limitations and Broader Impacts** (Appendix A):  We have extended the discussion on Limitations of the paper, in particular, regarding (1) the general reasoning ability of LLMs and (2) our paper’s focus on binary responses (Reviewers Eq6F, 7P9X)

- **Dynamically Varying Contexts** (Appendix E, lines 1165-1171): We have discussed that the “BFS + Relation hierarchy” retrieval method is computationally efficient than embedding-based retrieval methods in the case of dynamic KGs. (Reviewer 7P9X)


- **Empirical Evaluation on Natural Text Dataset** (Appendix F): We have empirically evaluated our method on FEVER dataset, a widely known real-world fact-checking benchmark containing textual facts. Experiments show that our supervised fine-tuning improves both the logical consistency and accuracy of the LLM on natural text dataset, thereby suggesting that the proposed method also generalizes to textual data.  (Reviewer pZwi)


- **Performance of the Fine-tuned Model on Language Benchmarks** (Appendix G): We have conducted an evaluation of the fine-tuned SFT model on four general language benchmarks: SCIQ, Lambada, MMLU, and Hellaswag. We found that the accuracy improved on the scientific question benchmark SCIQ and open-ended cloze task dataset Lambada. We have discussed remedies that may help improve the performance on other benchmarks. (Reviewer pZwi)


- **Experimental Results with the MiniCheck Baseline** (Appendix H): We have empirically compared the baseline MiniCheck-FT5 and our fine-tuned Llama2-7B model on the Freebase dataset. We found that our fine-tuned Llama2-7B shows better accuracy and consistency than MiniCheck-FT5. This highlights the importance of our supervised fine-tuning on propositional logic queries. (Reviewer JETn)

- **Experiments with Larger Models** (Appendix I): In an effort to evaluate the need for supervised fine-tuning on larger-sized models, we have evaluated Llama2-70B on simple facts, complex facts, and logic rules. We found that despite having more parameters than the corresponding 7B and 13B versions of LLaMA2, Llama2-70B does not always improve logical consistency in all three query types, demanding further supervised fine-tuning or instruction prompting. We further found that 70B model demonstrates better instruction prompting capability than the lower size models, while improving accuracy and logical consistency. (Reviewer JETn)

- **Grammatical Error** (line 346): We have corrected the grammatical error at line 346 in section 4.1 (Reviewer Eq6F)

Below, we address the questions and concerns specific to each reviewer in further detail.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary of the paper: This paper addresses the critical issue of logical consistency in LLMs, particularly in the context of answering complex queries derived from propositional logic and knowledge graphs. This paper highlights a gap in current assessments, which often focus on simple paraphrasing rather than deeper logical reasoning. They present a method to benchmark LLMs for logical consistency through three proposed datasets (FreebaseLFC, NELLLFC, and WikiLFC) that evaluate how well models maintain consistency when answering logically equivalent queries. To enhance logical consistency, this paper advocates for SFT over prompting, demonstrating its effectiveness in improving model performance across the benchmarks. Overall, the paper emphasizes the need for rigorous evaluation and improvement of LLMs in logical reasoning tasks, particularly in applications such as fact-checking.

Strengths of the paper:
- Clear Presentation: In general, this paper is well-written, presenting complex concepts in an accessible manner.
- Useful Resources: This paper establishes a definition of logical consistency and introduces three new datasets (FreebaseLFC, NELLLFC, WikiLFC) for fine-tuning and evaluation.
- Comprehensive Experiments: The study employs multiple experimental setups, comparing zero-shot instruction prompting with supervised fine-tuning, revealing limitations in existing LLMs' logical consistency. Results show significant improvements in the logical consistency of LLMs when applying supervised fine-tuning techniques.

Weaknesses of the paper:
- Benchmark Limitations (Reviewer pZwi): Experiments are heavily reliant on created benchmarks, raising concerns about their applicability to real-world scenarios, particularly with textual queries and the generalizability of supervised models.
- Dynamic Context Effectiveness (Reviewer 7P9X): The methods for extracting relevant KG context via BFS and embedding-based retrieval lack clarity on their effectiveness in dynamically varying real-world situations, indicating a need for further exploration.
- Limitation of Reasoning Type (Reviewers Eq6F, 7P9X): The focus on binary responses could restrict the method's applicability in complex domains, suggesting a need to explore multi-class logic and non-binary answer consistency for broader applications.
- More Analysis are Needed (Reviewer JETn, pZwi): Including using larger LLMs, more baselines and more benchmarks to provide more empricial evidence to justify the effectiveness of the proposed method. 

Reason for the decision: After considering the rebuttal, I believe that the authors have adequately addressed most of the concerns raised, as detailed in the additional comments in the reviewer discussion box. Reviewers JETn and 7P9X support accepting this paper, while reviewers pZwi and Eq6F recommend rejection. Reviewers pZwi and Eq6F did not respond to the authors' last comments. Upon reviewing the authors' responses, I find that they have addressed, at least in part, the concerns raised by these reviewers. After careful consideration, I am leaning toward accepting this paper due to its proposed logical consistency, which I believe is an important perspective that is currently somewhat overlooked by the research community. Additionally, the paper contributes valuable resources, including three datasets and an effective SFT method. It has the potential to engage the broader community and make an impact.