Detecting Hallucination Before Answering: Semantic Compression Through Instruction

Authors: Yeongbin Seo, Dongha Lee, Jinyoung Yeo
Primary area: foundation or frontier models, including LLMs
Keywords: hallucination, hallucination detection, feeling of knowing, LLM, large language model

Abstract:
Large language models (LLMs) excel in various tasks but often suffer from hallucinations, providing incorrect information with high confidence. To address this, we focus on detecting when an LLM knows or does not know an answer, a concept referred to as the ``feeling of knowing'' (FoK). We propose a novel approach called Semantic Compression by trying to Answer in One-word (SCAO), which enables efficient FoK detection before generating full sentences, with only minimal computational cost. Additionally, we introduce a method to measure confounding variable effects in benchmarks, an Approximate Misannotation Effect (AME) test. Our experiments demonstrate that the feature fusion model of our SCAO and probing achieves enhanced performance in FoK detection in both short and long-form entity questions. The code and the dataset is available online (https://anonymous.4open.science/r/SCAO-2FF8).

Decision: reject
Reviewer scores: 5, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730343172281

thread_type:
official_review

<thread_object>
creator: reviewer_qsgq

<content>
summary:
The paper presents a novel approach to detect when a large language model (LLM) knows or doesn't know an answer, a concept referred to as the "feeling of knowing" (FoK). The authors propose Semantic Compression by trying to Answer in One-word (SCAO), an efficient method for FoK detection that requires minimal computational resources. The paper also introduces a method to measure the impact of confounding variables in benchmarks, called the Approximate Misannotation Effect (AME) test. The experiments show that combining SCAO and probing improves FoK detection in both short and long-form entity questions. The paper argues that LLMs are structurally similar to dense retrievers and that token-level confidence can be a key source of information representing the vividness of memory. It also highlights the issue of misannotation in previous research and proposes an AME test to measure it. The results show enhanced performance of the feature fusion model of SCAO and probing, demonstrating their synergistic nature.

strengths:
1) The authors propose a novel approach, Semantic Compression by trying to Answer in One-word (SCAO), to detect when a Large Language Model (LLM) knows or doesn't know an answer. The paper also introduces the Approximate Misannotation Effect (AME) test, a new method to measure confounding variable effects in benchmarks. 
2) By detecting when an LLM knows or doesn't know an answer before generating a full sentence, the SCAO method could significantly reduce computational cost and improve user experience. The AME test could also help researchers better understand and control the effect of misannotation in benchmarks.

weaknesses:
The paper has some foundamental flaws as outlined. Nonetheless, after addressing these problems, it could potentially be a captivating piece of work.
1) The problem that the paper is attempting to solve is not well justified. The concept of "Detecting hallucination before answering" appears contradictory for two primary reasons. Firstly, the term "detecting" implies that an answer already exists to be detected, which is inconsistent with the concept of doing so prior to providing an answer. From my perspective, it appears more akin to "predicting hallucination before answering". Secondly, the necessity for "predicting" before answering is not adequately explained. Although there may be valid reasons for this, the paper fails to provide them.
2) The paper conflates the concept of "feeling of knowing" with actual "hallucination". The authors assert that their focus is on determining when a Large Language Model (LLM) knows or does not know an answer, but they do not reference or compare their work to existing studies on the subject, such as the well-regarded "Do Large Language Models Know What They Don't Know? (Z Yin, 2023)". Furthermore, while the "feeling of knowing" is related to hallucination, the two are not identical.
3) The experimental design is biased. The paper specifies that "Our baseline should predict FoK label y from question q without allowing the target LLM \theta to infer more than two steps". However, this limitation is based on the paper's specific methodology, not the task itself, and as such is unfair to the baselines.
4) The research on the impact of misannotation does not have a significant link with the SCAO, suggesting it could be divided into two separate papers.

questions:
LN013: "we focus on detecting when an LLM knows or does not know an answer", feeling of knowing is a different task from hallucination detection.

LN068: "the effect of misannotation", what specific effect does it refer to?

LN080: I cannot see the connections between the contribution 1) and 2).

LN105: "FoK of LLM" misses the studies about "know and don't know in LLMs". For example, "Do Large Language Models Know What They Don't Know? (Z Yin, 2023)"

LN312: "misannotation effect" need literature support or justification. Why is it a problem and why is it important for your topic?

LN388: "Our baseline should predict FoK label y from question q without letting the target LLM \theta infer more than two steps", why should the baseline follow the restriction of your method?

LN429: "impossible to address without further clue", what does it mean? Need further explanation.

LN430: "we only use English questions and exclude instances with multiple labels", why?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680448112

thread_type:
official_review

<thread_object>
creator: reviewer_a5np

<content>
summary:
This paper proposed a method namely SCAO to detect the hallucination before full generation of LLMs. The key idea is to prompt LLMs to answer in one word. It can compress the semantic information, and assess the model's confidence. This work also introduced a method namely AME to measure mis-annotation effects in benchmarks.

strengths:
1. The experiments are thorough. The method AME assessing benchmark quality is a valuable contribution.

2. The paper is well-written and easy to follow. The key concepts like FoK and SCAO are explained clearly. It is important for LLMs on hallucination detection, especially before full generation. It minimizes the computational overhead.

weaknesses:
1. Although the retriever analogy is interesting, it is not very clear to me that why compressing to one word should work better than other forms of compression.

2. The gains are relatively modest in some cases. It will be great if more in-depth analysis of when and why the proposed SCAO can outperform.

questions:
Did you try different forms of semantic compression rather than with one-word answers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730747489086

thread_type:
official_review

<thread_object>
creator: reviewer_rzuc

<content>
summary:
This work focuses on addressing hallucinations in large language models, by estimating to what extent the model has relevant knowledge— relying on the fact that model hallucinations primarily arise when the model is making extrapolations. The work proposes a method “semantic compression via trying to answer in one word” to estimate to what extent the model has relevant knowledge or not (FOK).

strengths:
(1) The paper is well-written and clear.

(2) The experimental methodology is well laid out and the rationale behind various decisions is clearly explained in the manuscript.

(3) This work joins a growing line of research in estimating model confidence and knowledge, which is an interesting research direction.

weaknesses:
(1) The evaluation is mostly done focusing on QA tasks where there is a concept of a right answer or a set of right answers. However, many times hallucinations can happen in more open-ended settings, like with the prompt “Tell me a Biography of Barack Obama” (see Min et al., 2023). Models may interleave true facts and false facts when generating a response, how can this method be extended to these typical long-form settings?

(2) It would be great to include more analysis of when this method works and when it doesnt, for example where in the model response the correct answer string occurs when making the FOK dataset.

Min, Sewon, et al. "Factscore: Fine-grained atomic evaluation of factual precision in long form text generation." arXiv preprint arXiv:2305.14251 (2023)

questions:
Could the authors explain how SCAO works when a question is not answerable by a single word?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730878096875

thread_type:
official_review

<thread_object>
creator: reviewer_p2rf

<content>
summary:
This paper introduces a method for detecting when an LLM knows the answer to a question *before* generating it. This method relies on probing the last token logits and hidden state obtained via a special "answer-in-one-word" prompt attached to the original query. The paper also introduces a procedure for identifying datasets with misannotations and excludes these from the analysis. Experimental results on entity-centric datasets suggest that the proposed method significantly improves the accuracy of detecting questions where the LLM is likely to provide a correct answer.

strengths:
- The problem formulation of detecting a hallucination *without* generating an answer is quite interesting and potentially useful.
- The main idea of representing the query via a "answer-in-one-word" prompt is intuitive and interesting.
- Several baselines are evaluated and the proposed methods shows notable gains in accuracy of detecting hallucinated answers.

weaknesses:
- The AME analysis for detecting misannotations is unconvincing. The main idea is that if a classifier can detect hallucinated answers based on the question alone, this points to some issue with the question. But such a classifier might be simply detecting difficult questions or questions about tail knowledge. In this case, it is not clear why datasets with such questions should be excluded? Moreover there isn't any further evidence provided that the datasets excluded based on AME were indeed problematic.
- Max token length is set to 50 when generating answers for the long-form QA datasets -- this seems very short for a dataset like ELI5. Further, it is not clear why a new dataset like Explain was used rather than other existing long-form QA datasets (e.g., ASQA).
- The main motivation for detecting a hallucination before generating is to save the cost of decoding. However, most of the empirical analysis in this paper is on datasets where the answers are short entities. In such a case, appending the answer-in-one-word prompt is perhaps equally expensive as decoding the answer. Hence, the paper would benefit from evaluating on more realistic setups where the answers are longer. (There are some results in Table 3, but these are not very convincing).

questions:
- How does SCAO compare to R-tuning on the same datasets as used in the latter paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731951753517

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Rebuttal

comment:
We first thank all reviewers for their thoughtful feedback on our work. We believe that constructive suggestions from all reviewers, such as the clear use of terminology between hallucination detection and FoK (Feeling of Knowing), can significantly enhance the clarity and advancement of our paper.

We have incorporated this feedback to update our paper, and the key updates are summarized below.

---
## 1. Clarification of the term  "hallucination detection" and "FoK"
As reviewer qsGQ pointed out, hallucination detection and FoK are distinct concepts. While we have already explained this in Sections 1 and 2, there still seems to be room for more emphasis. Therefore, we have revised our paper to clarify this further.

**Definition of "hallucination detection" and "FoK"** 

Here, we review how the definitions of "hallucination detection" and "FoK" differ. While hallucination indicates general incorrect answering, its causes can be categorized into various factors. In open-book tasks, hallucination is primarily caused by issues with reasoning and reading comprehension, whereas in closed-book tasks, it is mainly due to a lack of certain knowledge. Additionally, the question itself being unanswerable or vague can also be a cause.

Among these, FoK refers to detecting the lack of certain knowledge in closed-book scenarios. Therefore, FoK can be considered a subset of hallucination detection. Further, our work focuses on the scenario of detecting before generation.

Despite these differences, we used "hallucination detection" in the title for the following reasons: 1) In common service environment like chatGPT, it is well-known that a major cause is when the model is asked questions in subjects where it lacks pre-trained knowledge. LLMs lack the ability to say "I don't know" when they encounter unknowns, leading to a hallucinated answer instead. 2) Previous works have often regarded these two concepts as identical [1], without providing a separate definition of FoK. In connection with prior research, we chose this expression.

However, this can cause confusion. Thus, to avoid confusion, we thoroughly emphasized the distinction between the two concepts in the paper. Nevertheless, as there is still potential for misunderstanding, we update the paper as follows.

**Key updates in our paper**

1) We change the title of the paper as follows for clarity.

*Detecting Unknowns to Predict Hallucination, Before Answering: Semantic Compression Through Instruction*

2) We revised Sections 1 and 2 to further clarify the distinctions between the two concepts, "hallucination detection" and "FoK."

3) Our proposed metric for assessing datasets, AME, is designed to clarify the evaluation of FoK itself rather than hallucination detection. Accordingly, we revised the explanation of AME in Section 5 to better reflect this purpose.

---
## 2. Redefining AME (approximate misannotation effect) to AQE (approximate question-dependency effect)

Regarding the review of p2RF, we redefined AME as AQE and revised its explanation in Section 5 to emphasize the following aspects:
1) We emphasize that AQE not only measures the effect of misannotated questions but also captures broader question-dependency in the FoK datasets.
2) We clarify that AQE is a metric designed to evaluate the model-awareness of FoK methods, rather than being solely focused on hallucination detection.

The revised explanation of AQE is as follows. (identical to the paper):

**Explanation on AQE**

For a more precise evaluation of FoK, we provide a metric that can assess whether a dataset is fit for evaluating FoK. While FoK is the ability to be aware of whether LLM possesses specific knowledge, what we can explicitly measure is only the amount of incorrect answers (hallucinations, denote $H = \sum_i  1 (y_i = False)$), which is not enough to measure self-awareness. 

Causes of incorrect answers can be naively divided into two: question-dependent and model-dependent. **(1) Question-dependent:** This occurs when the question is difficult or unanswerable (denote $Q$). **(2) Model-dependent:** This happens when the model lacks the necessary knowledge to answer the question (denote $M$). When $H=Q+M$, FoK refers to predicting the portion of $M$. Since $H$ is measurable, we should measure $Q$ to accurately determine $M$. But measuring $Q$ directly is non-trivial, as it is for $M$.

AQE is devised to approximate the $Q$.  $Q$ is defined as the portion of incorrect answers ($H$) that can be predicted solely with the properties of the question, independent of $\theta$. To fit this definition, we train and test a model $\phi$ to predict $y=False$ case using only the question as input. This accuracy of $\phi$ is the AME score. The closer AME is to 1, the lower the model-dependency of the dataset (paired with a certain model), making it unsuitable for measuring FoK. 

**Reference**
[1]Hanning Zhang, et al. R-Tuning: Instructing Large Language Models to Say ‘I Don’t Know’.  2024 NACCL
</content>
</thread_object>

</thread>
<thread>
creation date:
1732466997746

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers.

For a more convenient review of the revised paper, we provide a version where the major edits compared to the first submitted draft are highlighted in **blue** ([click to see the pdf [link]](https://openreview.net/pdf?id=YFOg1LUGG1)). After the discussion period ends, we will remove the highlight for the final version submission.

Through the rebuttal and updates of our paper, we believe we have addressed all of the reviewers' concerns. This enabled us to write a more refined and comprehensive draft. We deeply appreciate the reviewers' insightful feedback.

As the end of the discussion period approaches, please feel free to raise any remaining questions or suggestions.

Thank you so much for your time and valuable feedback!

Best regards,

The Authors of Paper 8793
</content>
</thread_object>

</thread>
<thread>
creation date:
1732792096536

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers.

As the PDF revision period came to an end, we convert the version with blue-highlighted revisions back to a version without highlights ([click to see the pdf[link]](https://openreview.net/pdf?id=YFOg1LUGG1))). 
The blue-highlighted version is still accessible in the revision history (revised version on 25 Nov 2024 ([click to see the pdf [link]](https://openreview.net/notes/edits/attachment?id=jis0eX23UI&name=pdf))).

If you have any remaining concern, we would be happy to address any further questions/suggestions that might come up until the end of the discussion period.

Thank you so much for your time and valuable feedback!

Best regards,

The Authors of Paper 8793
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents an approach to assess whether an LLM knows the answer to a question prior to generating a response. The approach involves probing the LLM's final token logits and hidden states using a specialized "answer-in-one-word" prompt. Additionally, the paper proposes a process for identifying and excluding datasets with misannotations from the analysis.

While we appreciate the authors’ responses during the rebuttal phase, several key concerns raised by the reviewers remain unresolved:

- The experimental findings lack sufficient significance to bolster the paper’s contributions. The results across datasets such as Mintaka, HotpotQA, and ELI5-small are not particularly compelling.
- The experiments focus solely on factoid QA, which is insufficient to justify the proposed approach. As highlighted by one reviewer, a core motivation of the paper is to reduce the cost of generating responses from an LLM. Consequently, it is crucial to evaluate the approach on datasets that require longer generations.

These unresolved issues limit the overall value of the work. As a result, we agree that the current submission does not meet the ICLR standard. We hope these reviews provide useful feedback for the authors’ future revisions.