Title: FEWL: Measuring and Mitigating LLM Hallucination Without Gold-Standard Answers
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM Hallucination, LLM Evaluation

Abstract: LLM hallucination, i.e. generating factually incorrect yet seemingly convincing answers, is a major threat to the trustworthiness and reliability of LLMs. The first step towards solving this problem is to measure it. However, existing hallucination metrics require having a benchmark dataset with gold-standard answers, i.e. ``best'' or ``correct'' answers written by humans. Such requirement makes hallucination measurement costly and prone to human errors. In this work, we propose Factualness Evaluations via Weighting LLMs (FEWL), a novel  hallucination metric that is specifically designed for the scenario when gold-standard answers are absent. FEWL leverages the answers from off-the-shelf LLMs that serve as a proxy of gold-standard answers. The key challenge is how to quantify the expertise of reference LLMs resourcefully. We show FEWL has certain theoretical guarantees and demonstrate empirically it gives more accurate hallucination measures than naively using reference LLMs. We also show how to leverage FEWL to reduce hallucination through both in-context learning and supervised fine-tuning. Experiment results on Truthful-QA, CHALE, HaluEval datasets demonstrate the effectiveness of FEWL.

TL;DR: We propose Factualness Evaluations via Weighting LLMs (FEWL), an innovative hallucination metric that is specifically designed for the scenario when gold-standard answers are absent.