KnowHalu: Multi-Form Knowledge Enhanced Hallucination Detection

Authors: Jiawei Zhang, Chejian Xu, Yu Gai, Freddy Lecue, Shuang Yang, Dawn Song, Bo Li
Primary area: foundation or frontier models, including LLMs
Keywords: Hallucination Detection; Large Language Model; Factual Checking; Multi-Form Knowledge

Abstract:
As large language models (LLMs) become increasingly integral to a wide array of applications, ensuring the factual accuracy of their outputs and mitigating hallucinations is paramount. Current approaches, which primarily rely on self-consistency checks or post-hoc fact-checking, often fall short by disregarding the nuanced structure of queries and the diverse forms of contextual knowledge required for accurate response generation.
To address these shortcomings, we introduce KnowHalu (pronounced “No Halu”), the first multi-form knowledge-based hallucination detection framework. We also introduce a new category of hallucinations, off-target hallucinations, which occur when responses are factually accurate but irrelevant or nonspecific to the query (e.g., answering "What’s the primary language in Barcelona?" with "European language").
In particular, KnowHalu employs a rigorous two-phase process to detect hallucinations. In the first phase, it isolates off-target hallucinations by analyzing the semantic alignment between the response and the query. In the second phase, it conducts a novel multi-form knowledge-based fact-checking through a comprehensive pipeline of reasoning and query decomposition, knowledge retrieval, knowledge form optimization, judgment generation, and judgment aggregation.
Extensive evaluations demonstrate that KnowHalu significantly surpasses state-of-the-art (SOTA) baselines across diverse tasks, achieving over 15% improvement in question answering (QA) and 6% in summarization tasks when applied to the same underlying LLM. These results underscore the effectiveness and versatility of KnowHalu, setting a new benchmark for hallucination detection and paving the way for safer and more reliable LLM applications.

Decision: reject
Reviewer scores: 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729075326024

thread_type:
official_review

<thread_object>
creator: reviewer_ns6f

<content>
summary:
This paper proposes a KnowHalu method to detect hallucination, which is roughly divided into two stages, including off-target hallucination detection and multi-step facutal checking. The authors conducted experiments on multi-hop QA and text summariztion tasks on the HaluEval dataset. Three LLMs including Mistral-7B, Starling-7B and gpt-3.5-turbo-1106 were used. The experimental results show that the accuracy is improved compared with baseline models such as self-consistency that do not rely on retrieval evidence.

strengths:
1. The authors conducted sufficient experiments and the method description is generally clear. The motivation of the proposed method is intuitive
2. The paper includes in-depth analyses of various components of KnowHalu. These studies may provide valuable insights into the contribution of each component.

weaknesses:
1. Although the proposed multi-step reasoning process decomposes the query into multiple sub-queries to improve the detection accuracy, it involves multiple steps of knowledge retrieval, reasoning and aggregation, but it introduces engineering complexity and difficulty in reproduction, such as shared key-value caching and multi-form queries. I suggest that the authors open source the code and provide detailed reproducible instructions. While some latency comparisons are provided, a more thorough analysis of the computational overhead compared to simpler approaches would be valuable, especially for real-time applications.
2. The evaluation relies heavily on the HaluEval dataset, where hallucinations are generated by chatgpt and may not fully represent hallucination behavior in real-world scenarios. I would like the authors to test on a wider dataset with more sources of hallucinations or real sources of hallucinations (1000/500 given the limited size of the test set).
3. In addition, I hope that the author conduct manual evaluation tests on the detected hallucinations and reasoning processes, and conduct corresponding error analysis to gain insight into their limitations and potential improvement directions, such as errors in early stages (query or knowledge retrieval, rewriting queries) may be amplified through the pipeline. Robustness analysis of cascading errors is needed.
4. The authors are encouraged to take a systematic exploration of how different scales of model size affect performance.
5. The writing and presentation still need improvement. Such as Line 222 - one-hope, the size of Figure 7, Table 10, Table 15

questions:
1. Is there a clear and strict definition of Off-Target Hallucination?
2. What is the reason for the performance difference of two query formulations? How are the two queries generated?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730355297495

thread_type:
official_review

<thread_object>
creator: reviewer_h9rq

<content>
summary:
This paper presents a novel approach, referred to as KnowHallu, for detecting hallucinations in texts generated by large language models (LLMs). Firstly, it isolates a new type of hallucination, referred to as the off-target hallucination, and then it conducts a novel multi-form knowledge-based fact-checking through a comprehensive four-stage pipeline. Experimental results demonstrate the effectiveness of the proposed framework for detecting hallucinations.

strengths:
1.The figure is presented in a clear and organized manner, effectively incorporating a wealth of details that contribute to KnowHallu’s comprehensibility.

2.The proposed framework demonstrates effectiveness based on a substantial number of experiments conducted.

weaknesses:
1.Lack of clarification of the concept of ‘off-target’ hallucinations. The author claims in the abstract that they introduce a new category of hallucinations. However, it is noteworthy that prior research, such as HaluEval, has already categorized model responses into "unverifiable", "non-factual" and "irrelevant" classifications. Furthermore, earlier studies focusing on fact-checking have highlighted that not all statements are suitable for this process, which has led to the emergence of the claim detection task. Therefore, why is the concept of off-target hallucination emphasized in this context? Additionally, I am interested to know whether there is statistical data in Table 1 regarding the percentage of different types of questions present in actual LLM responses.

2.Usage of testing tasks. The original HaluEval framework includes both Dialogue and General scenarios; however, it is unclear why these two scenarios were not included in the current evaluation. The proposed design of step-wise reasoning is a common approach in addressing multi-hop questions. However, it raises the question of its applicability across all hallucination detection tasks.

questions:
See weakness.

1.Regarding the experimental results presented in Table 2, it is noteworthy that GPT-3.5 does not exhibit superior performance compared to other models. Isn't that somewhat counterintuitive?

2.The overall framework appears to combine various manually designed prompting strategies with established fact-checking techniques. For example, the approach involves decomposing the question into smaller components and verifying each part through retrieval on a one-by-one basis. From your view, what's the difference in the current hallucination detection task and the original fact-checking task? 

I would like to improve my score if my aforementioned questions are answered properly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700570873

thread_type:
official_review

<thread_object>
creator: reviewer_wylp

<content>
summary:
This paper presents KnowHalu, a novel hallucination detection framework leveraging multi-form knowledge to enhance the factual accuracy of Large Language Models (LLMs) and mitigate the generation of hallucinated content. KnowHalu employs a two-phase approach: initially identifying off-target hallucinations through semantic alignment, followed by a multi-step, knowledge-based fact-checking process that includes reasoning, query decomposition, knowledge retrieval, optimization, judgment generation, and aggregation. Extensive evaluations demonstrate KnowHalu's superiority over state-of-the-art baselines, with over 15% improvement in question answering tasks. The study introduces the concept of off-target hallucinations and explores the impact of query formulations and knowledge forms on detection accuracy, proposing an aggregation method to refine judgments based on different knowledge forms. While KnowHalu shows promising results on standard datasets, the experimental section could be enhanced, particularly in handling more complex dialogues and longer responses, and the analysis could benefit from deeper insights into the nuances of hallucination detection.

strengths:
The "Off-Target Hallucination Checking" concept is indeed compelling as it tackles a critical shortcoming in the output of language models. The detection pipeline for addressing these hallucinations is not only effective but also yields encouraging results across question answering and text summarization tasks. Furthermore, the methodology section is articulated in a clear and comprehensible manner, which enhances the understandability of the approach taken.

weaknesses:
The shortcomings of this paper are quite apparent. First, it lacks a thorough analysis of the RAG data sources. For instance, it does not address the specific impact of the knowledge base (KB) data source on the results, nor does it consider how data missingness or redundancy might affect the outcomes. Additionally, it would be beneficial to explore whether data conflicts could impact the model's robustness.

Moreover, Section 3.2 presents a rather lengthy description of the methods. It would be advisable to streamline this section to enhance overall clarity.

questions:
In Tables 4 and 5, does KnowHalu perform better in unstructured scenarios than in structured ones? However, doesn't Table 3 present the opposite conclusion?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732313963999

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Summarization

comment:
We thank all the reviewers for their valuable feedback on our paper. We are pleased to note that all reviewers found our paper well-written and recognized it as providing a novel and systematic LLM hallucination detection framework, along with a significant performance improvement over other methods. We have made the following revisions following the reviewers' suggestions to further improve our work.

- We have added the detailed inconclusive rate, which is proposed for considering data missingness, following the suggestion of **Reviewer wyLP** in Table 2.
- We added more clarification of the motivation for introducing a new category of hallucination, i.e., off-target hallucination, which we believe helps build a comprehensive hallucination categorization for the community, following **Reviewer H9RQ** and **Reviewer NS6F**.
- We included an analysis of the percentage of different types of questions shown in Table 1, following **Reviewer H9RQ**.
- We added additional experimental results on dialogue data from HaluEval and WikiQA, following the suggestions from **Reviewer H9RQ** and **Reviewer NS6F**. We show that our method still outperforms all other baselines with these two additional datasets.
- We conducted a systematic exploration of how different scales of model size affect performance, following **Reviewer NS6F**. The results show that the performance of the knowledge increases with the scaling of the model size linearly.

We sincerely appreciate the reviewers for their precious time in reviewing our paper and look forward to any further discussion to improve the quality of our work!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces KnowHalu, a novel framework for hallucination detection in outputs of large language models (LLMs). The method employs a two-phase approach: isolating off-target hallucinations through semantic alignment and performing multi-step, knowledge-based factual checking. KnowHalu introduces the concept of off-target hallucinations and demonstrates significant improvements over state-of-the-art baselines in both question answering and summarization tasks.

(a) Scientific Claims and Findings
KnowHalu is a multi-form knowledge-enhanced hallucination detection framework comprising two phases: detection of off-target hallucinations and multi-step factual checking. The authors propose a new category of hallucination, off-target hallucinations, characterized by responses that are factually correct but irrelevant or nonspecific to the query. Extensive experiments on question answering and summarization tasks demonstrate that KnowHalu outperforms state-of-the-art methods with improvements of over 15% in question answering and 6% in summarization accuracy.

(b) Strengths

- Novel categorization of hallucinations with the introduction of off-target hallucinations.
- Rigorous two-phase detection framework addressing both factual inaccuracies and relevance issues.
- Comprehensive experimental validation across diverse tasks, datasets, and models.

(c) Weaknesses

- Heavy reliance on HaluEval dataset, limiting generalizability to real-world scenarios.
- Computational complexity and cascading errors in the multi-step pipeline were identified but not fully resolved.
- Presentation issues and lack of clarity in some sections, although improved after revisions.
- Limited robustness analysis of cascading errors and real-time applicability.

(d) Decision: weak reject

While the paper proposes an interesting approach and demonstrates promising results and reviewers slightly improved their scores after rebuttal, critical issues remain that limit its readiness for acceptance. The reliance on a single dataset raises concerns about generalizability, and the computational complexity of the method challenges its practical applicability. Addressing these concerns with broader real-world evaluations and streamlined implementations could significantly strengthen the work. I encourage the authors to refine these aspects and resubmit in the future, as this line of research has substantial potential.