K-HALU: Multiple Answer Korean Hallucination Benchmark for Large Language Models
Authors: Jaehyung Seo
Heuiseok Lim
Venue: ICLR 2025 Poster (2025)
Keywords: Hallucination
Benchmark dataset
Multiple answer
Korean
Large language model
Primary area: datasets and benchmarks
TL;DR: Multiple-answer Korean hallucination benchmark for large language models

Abstract: Recent researchers and companies have been developing large language models (LLMs) specifically designed for particular purposes and have achieved significant advancements in various natural language processing tasks. However, LLMs are still prone to generating hallucinations—results that are unfaithful or inconsistent with the given input. As a result, the need for datasets to evaluate and demonstrate the hallucination detection capabilities of LLMs is increasingly recognized. Nonetheless, the Korean NLP community lacks publicly available benchmark datasets demonstrating the faithfulness of knowledge-based information. Furthermore, the few existing datasets that evaluate hallucination are limited in their access to the entire dataset, restricting detailed analysis beyond simple scoring, and are based on translated English knowledge. To address these challenges, we introduce K-HALU, a Korean benchmark designed to evaluate LLMs' hallucination detection in Korean. This benchmark contains seven domains, considering the faithfulness of statements based on knowledge documents compiled from Korean news, magazines, and books. For more strict evaluation, 40% of the dataset is structured as multiple-answer questions, requiring models to select all possible correct answers from the given options. Our empirical results show that open-source LLMs still struggle with hallucination detection in Korean knowledge, emphasizing the need for a more detailed analysis of their limitations.

Published as a conference paper at ICLR 2025
K-HALU: MULTIPLE ANSWER KOREAN HALLUCINA- TION BENCHMARK FOR LARGE LANGUAGE MODELS
Jaehyung Seo Korea University Scoul, Republic of Korea seo seo-aae770korees ac.ek
Heuiseok Lim
Korea University
Scoul, Republic of Korea
1 iimseok@korre ac kr
ABSTRACT
Recent researchers and companies have been developing large language models (LLMs) specifically designed for particular purposes and have achieved signifi- cant advancements various natural language processing tasks. However, LLMs are still prone to generating hallucinations--reulll that are unfaithful or incon- sistent with the given input. As result, the need for datasets to evaluate and demonstrate the hallucination detection capabilities of LLMs is increasingly rec- ognized. Nonetheless, the Korean NLP community lacks publicly available bench- mark datasets demonstrating the faithfulness of knowledge- based information. Furthermore, the few existing datasets that evaluate hallucination are limited their access to the entire dataset, restricting detailed analysis beyond simple scor- ing, and are based on translated English knowledge. To address these challenges, we introduce K-HALU" a Korean benchmark designed to evaluate LLMs' hallu- cination detection in Korean. This benchmark contains seven domains, consider- ing the faithfulness of statements based on knowledge documents compiled from Korean news, magazines, and books. For more strict evaluation, 40% of the dataset is structured as multiple-answer questions, requiring models to select all possible correct answers from the given options. Our empirical results show that open- source LLMs still struggle with hallucination detection in Korean knowledge, em- phasizing the need for a more detailed analysis of their limitations.
INTRODUCTION
Large language models (LLMs) have achieved remarkable advances, surpassing human capabilities various natural language processing (NLP) tasks (Ouyang al. 2022 OpenAl 2023). Recently, compact and high-performing open-source LLMs have emerged Touvron et al.) 2023 Jiang et al., 2023). and researchers and companies are leveraging these models to develop their own purpose- specific systems (Kim et al. [2244] Research al., 2024). Within the Korean NLP community, over 2,000 models were evaluated on the Open Ko-LLM leaderboard (Park et al., 2024) from October 2023 to August 2024, illustrating an increase in proprietary LIM development.
However, as result of limited parameter sizes and constrained training data, open-source LLMs continue to struggle with the problem of generating hallucinated outputs (Ji et al. 2023a Zhang et al. 2023 et al. 2023). Hallucinated outputs cannot guarantee faithfulness with the provided data and often include unsupported unverifiable content (Huang al. 2023a). Hallucinations present a significant threat to the reliability and practical applications of LLMs (Chen et al.) 2024), and LLMSs with relatively fewer parameters incomplete data even more vulnerable to this
phenomenon (Rawte et al. 2023 Guerreiro et al. 2023
A more pressing issue in the Korean NLP community is the lack of benchmarks to verify the po- tential risks associated with hallucinations in these numerous proprietary LLMss The few available Korean datasets related to hallucinations are typically closed and used solely for leaderboard-style scoring, limiting access to the data for detailed analysis (Park et al. 2024). Furthermore, most bench- marks focus on parametric knowledge and linguistic nuances specific the English-speaking world,
"Corresponding Author
https://gi thub. com/J- Sec/ HAIU

---

Published as a conference paper at ICLR 2025
Table 1: Examples of K-HALU benchmark according to the instruction type for selecting halluci- nated statements and faithful statements. This table has been translated from Korean into English for the convenience of non-Korean speakers. (Refer to the Korean version in Appendix
##f Hallucinated statements selection type Society Domain
#date: 20210711
#document: The Korean Association for Public Administration estahlished in 1956... Red tape symbol of bureaueratic formalism.
#instruction: Select the hallucinated statements that differ from or are unsupported in relation to the content of the given document. Note that there can be multiple hallucinated statements.
#1: Professor Park Soon-ac working to expand women's 's participation the public sector.
#2: Professor Park Soon-ac went to the United States to pursue Ph.D. while raising two children.
#3: Professor Park Soon-ae declares an intention to reform the bureaucratic field in the 3G era.
#4: Professor Park Soun-ae points out that udministrative convemience is an issue in Korea.
#5: Professor Park Soon-ae states that civil servants prefer maintaining regulations over deregulation.
#label: [1.2] (Index number choice) #choice number: [2.3] (Numbering of choice)
##* Faithful statements selection type International Domain ###
#date: 20150819
#document: Google is launching new 'Android One' smartphone in six African countries It is one of the major projects being pursued.
#instruction: Select the faithful statements that correspond to the information identifiable from the given doc- ument. Note that there may be multiple correct statements.
#1: Google selling the 'Hot 2' model in six North American countries, including the Canada and Mexico.
#2: The price of the newly launched model is under 100 dollars.
#3: Google is launching a new 'iPhone One' smartphone in six African countries.
#4: The 'Hot model. produced by Infinix, features a 10-inch touchscreen and a 5GHz quad-core processor.
#5: satellite internet service project is underway in Kampala, the capital of Uganda.
#label: [2] (Index number of choice) #choice number: [3] (Numbering of choice)
making them less ideal as resources for evaluating underrepresented languages |Exxaniz et al., 2024)) In particular, English hallucination benchmark datasets are challenging to apply to the Korean lan- guage caused by linguistic and socio-cultural differences (Hendrycks et al., 2021 Seo et al. 2024). The absence of publicly available Korean hallucination benchmarks restricts the ability to evaluate the reliability of LLMs thoroughly, hinders the continuous accumulation of findings needed for im- provement, and makes it challenging to capture the robustness of LLMs in detecting hallucinations.
To overcome these limitations, we introduce the multiple-answer Korean hallucination benchmark for large language models (K-HALU). K-HALU consists of 2,170 test samples, each including a textual document, a publish date, an instruction, and statements. The textual documents sourced from seven knowledge Technology-from Korean news, magazines, and books. As described in Table [ LLMS should select the appropriate statements in multiple-choice format, considering the given textual document and publish date. Unique to K-HALU is the multiple-answe question setup, which requires LLMs to identify all possible correct answers. This approach ensures a more rigorous evaluation of model reliability and assesses the models' ability to recognize multiple simultaneous hallucinations. Also, K-HALU includes statements considering the publish date, allowing us to determine whether LLMs maintain temporal consistency in faithfulness.
Economy, History. International, Medical, Society, and
We evaluate open-source multilingual I.MMs frequently used in the Korean NIP community, such as Llama2 (Touvron et al.] 2023). I.lama (AT@ Metal 2024). Mistral (Jiang et al., 2023). and Korean- centric models such as KULLM3 (Kim et al. 2024) and ExaOne (Research et al. 2024). Addi- tionally, we test closed API models with high performance and usability, including GPT-3.5 Turbo Ouyang et al. 2022), , GPT-4 Turbo, and GPT-4 omni (OpenAl 2023). The results reveal that open- source LLMs exhibit low accuracy, with less than 35% in our evaluations, and perform particularly poorly-under 15%-on instruction types designed to differentiate hallucinated statements. Com- pared to API models such as GPT-3.5 and beyond, open-source models present a performance gap exceeding 27%, showing greater weakness hallucination the number of answers increases.
2 RELATED WORK
Hallucination in NLP With the prominent advancements of LLMs and their applicability to vari- ous tasks, the importance of research on hallucinations in NL. has increased significantly. Maynez
2

---

Published as a conference paper at ICLR 2025
AI Hulb
Fishion 2524012 Wocanes danang Nersach radjer X: vep
521..
Culture
Medical
Economy
Sackety
9 Rasionalationn
@ K-HALU Benchnurk
History
Technology
International
Craembre Verfication
Ihalkctrude 5
Korean textual knowledge
Astomatic Dotaunt Creation
Figure I: Overview of K-HALU benchmark construction pipeline.
et al.] (2020) highlighted issues of faithfulness and factuality in abstractive summarization, while
Raunak al. (2021) pointed out similar concerns in neural machine translation, emphasizing that
language model-based natural language generation can lead to hallucinations. Since then, research focusing on hallucinations in natural language generation tasks has become
prominent (Zhao iet al. 2020 Shuster et al. 2021: Liu et al. [021 Fabbri et al.) [2022] Zhang et al. 2022). Following
the emergence of outstanding generative LLMs (Ouyang et al. [0222) the definition and scope of hallucinations in NLP have been discussed in greater detail (Ji et al. 2023a Huang et al. 2023a
Zhang et al., 2023). Pi et al. (2023a) categorized hallucinated outputs into two types: intrinsic hal- lucinations, which result from confficts with the source content, and extrinsic hallucinations, which include information that cannot be verified by the source content. They focus on the contributors to hallucinations and explore task-centric mitigation strategies. Huang et al. (2023a) explored the causes and solutions to hallucinations based on faithfulness and factuality, while Zhang et al. (2023) classified hallucinations arising from conflicts between input, context, and factual knowledge with the model's i generated output. Subsequent studies have actively explored hallucination detection
(Huang al.) [023b] Manakul al.) 2023 Tiang al.
2024 Chen al., 2024) and mitigation 2023b Choubey et al. (002)..
strategies (Maheshwari et al. 2023 Chuang el al. 2023
Ji et al.
Several English hallucination benchmarks have been developed to evaluate language models in var- ious tasks. These include Fever ( (Thorne et al.) 2018)), which evaluates factual consistency against textual sources. QAGS (Wang et 2020), which measures factual inconsistencies in summaries, SummEval (Fabbri et al.) (222)) which assesses the quality of summaries between human evalua- tors and models, FailhDial (Dziri et al. 2022), a dialogue-focused dataset for evaluating response faithfulness, HaluEval (Li et al.) 2023). which determines faithful hallucination presence in model- generated samples, and FELM (Zao et al. 2024), which measures hallucination across multiple domains. However, research that provides specific insights into hallucinations in the Korean lan- guage is still lacking, and there is a significant shortage of publicly available benchmark datasets that could serve as the foundation for hallucination studies in Korean.
Korean Benchmark Benchmarks serve as essential tools for the quantitative assessment of the strengths and weaknesses of L.Mss, offering critical insights into the future direction of NL. re- search and development (Zellers et al. 2019; Son et al., (202aa). Existing Korean benchmarks have evaluated models based linguistic skills or language understanding (Park al. 2022). and tasks involving universal reasoning were often translated for applicability to other languages (Conneau et al.] 2018 Ham et al.] 2020 Ponti al. 2020 Seo et al. 2022 Park et al. 2024). However, the need for native knowledge derived from textual sources written in Korean, beyond simple machine translation, has increasingly been recognized within the Korean NLP community for more precise performance measurement and knowledge verification (Son et al. 2024a; Seo er al. 2024). Tasks in benchmarks such as KorNLI & KorSTS (Ham et al., (022)). Korean- CommonGEN (Seo et al. (022), and Ko-H5 (Park et al. 2024) are based on reasoning and have been constructed by translating ex- isting datasets. KoBBQ (Jin et al. 2024) addresses biases using partial translations specific to the Korean cultural context. Meanwhile, benchmarks such as HAE-RAE (Son et al., 2024b). KMMLU (Son et al. 2024a), and KoCommonGEN v2 (Seo et al. 2024) have been developed using Korean textual sources and human annotators. However, these benchmarks do not evaluate hallucinations in Korean. Ko- Ko-TruthfulQA (Lin et al. 2022 Park et al., 2024). which partially addresses elements of hallucination, is closed dataset, and even the latest Open Ko-LLM leaderboard, which could verify reliability, does not provide access to the datasets. Thus, we propose a new Korean hallucination benchmark K-HALU and plan to release the dataset and evaluation code to the public entirely.
3

---

Published as a conference paper at ICLR 2025
Table 2: Number of K-HALU Benchmark examples according knowledge domain, instruction type, and multipleaanswer questions.
K-HALU Benchmark
Culture
Economy
History
International
Medical
Society
Technology
Total
# exomples
299
300
329
325
317
300
2,170
Instruction
-# hallucination solect
150
150
150
165
162
158
150
1,085
fact select
150
149
150
164
163
159
150
1,085
Multiple-nnseers
single-answer
180
179
180
197
196
190
180
1,302
# Ewo-answers
90
90
90
100
98
653
threc-answers
30
30
32
31
K-HALU BENCHMARK
K-HALU is a hallucination detection benchmark composed of 2,170 multiple-choice tasks, where there can be more than one correct answer. As shown in Table the seven domains are sourced from Korean textual knowledge and include Culture, Economy, History, International, Medical, Society, and Technology. The Culture domain covers topics related to entertainment, literature, and films; the Economy addresses issues such as administration, corporations, real estate, and the market; History contains Korean, world, and East Asian history; International deals with diplomacy, global corporations, and foreign affairs; Medical focuses on medical knowledge, pharmaceutical research, and health tips: Society covers education, North Korea, sel--oevelopment, and interest conflicts; and Technology includes patents, research papers, and IT services as the textual sources. Each domain contains average of 310 examples, and each example consists of the following components: "id," "'ocumentt"" "date," "instruction," "five statements," "choice number", and "label". All documents include the publish date. The statements are categorized as either hallucinated or faithful, and the proportion of these categories varies depending on the instruction type and the number of correct answers. Details regarding the policies and responsibilities related to the use of the K-HALU dataset provided in Ethics Statements.
3.1 TASK DEFINITION
The objective of the K-HALU task is for LLMs to the provided document and publish date as source knowledge to discriminate the faithfulness of the given statements and detect hallucinations. K-HALU employs a multiple-answe
question evaluation format, allowing for more than one correct answer. LLMs are required to select all possible correct statements based on the given instruction's 's question type. Given that the instruction demands "all possible correct answers," " even single in- correct selection by the LLM will result in an incorrect response, reflecting the strict nature of this evaluation. For example, be classified incorrect, as all correct answers must be identified for the response to be considered correct (the order of the selected answers is not taken into account during scoring).
question has three correct answers, selecting only two correctly will
3.2 DATASET CREATION
Source Dataset We utilized the publicly available Knowledge Graph-to-Tex -ext dataset from AI- Hub, an integrated AI platform operated by the Korean National Information Society Agency (NIA), as our textual source" This dataset consists of textual documents, including news articles, magazine articles, and books, describing faithful relationships in the form of statements. Each statement tagged as subject, predicate, and object, forming triples-context pair. Each document contains one or more triples-context pairs, amounting to a total 300,178 samples, each of which has processed human labeling and data refinement. Among these samples, 89.8% are news articles, 4.9% are mag- azine articles, and 5.3% are sourced from books. The textual documents are divided into seven knowledge domains: Culture, Economy, History, International, Medical, Society, and Technology, with all domains-except for History-containinn over 40,000 samples. Each sample includes one human-annotated statements that describe the corresponding textual document, and each
4

---

Published as a conference paper at ICLR 2025
statement is labeled with tags for subject, object, and predicate. Out of 300.178 samples, used 67,740 training samples to create options, considering domain proportions.
To construct the five answer options, extracted documents from each domain (cxcept for History) that contain at least five human-annen -nnotatd statements. Since the History domain had relatively smaller number of source data samples, extracted documents that included at least two human- annotated statements. From each domain, we selected an average of 310 documents as the final textual sources, excluding those with statements that referenced content mentioned in the text.
Hallucinated Statement To create hallucinated statements that violate faithfulness, we leveraged the human-annotated statements, tagging labels, and published dates from the extracted documents as basis for modification. Using GPT-4 omni (gpt 1ont-4c-200 200-2024 2222200-1100 05-13) (OpenA]] 2023). we transformed the human-ane aanaant-d statements into factually incorrect or unfaithful statements, mis- representing event dates based on the publish date, including unverifiable content. We generated five hallucinated statements per document. For the History domain, we added hallucinated state- ments that were directly generated without referencing human-annotated statements.
Faithful Statement To prevent overfitting or unintended cheating from model training the orig- inal dataset, we avoid using the numan-annotated statements literally as faithful statements. Instead, employed GPT-4 omni to either modify the existing faithful statements with high inferential quality that did not overlap with the originals. This process involved expressing tags at higher conceptual level paraphrasing while maintaining the same context. The faithful statements generated included higher-level reasoning, such as inferring event dates based the publication date or combining information from multiple sentences. As with the hallucinated statements, we generated five faithful statements per document.
statements or generate
Instruction and Multiple Answers We created instruction types that evenly required selecting hallucinated faithful statements for each domain. We adjusted the number of hallucinated and faithful statements for each sample, considering the proportion of multiple-answer questions. These proportions were based on MultiSpanQA (Li et al. 2022), where the distribution of answer spans is 58%/35%/7% and CLEAN (Luo et al. 2024). which uses 46% of question-answer pairs as multi- answer instances. We set the distribution of the multiple-answer question with one correct answer at 60%, two correct answers at 30%, and three correct answers at 10%, ensuring that more than two answers comprised 40% of the dataset. For example, if the ith test example's 's instruction requires selecting faithful statements and the label has two correct answers, two of the five statements are faithful, and three are hallucinations. To minimize bias from differences token counts between options, we selected the final candidate options by choosing statements closest to the average length of the ten generated statements.
The final dataset comprises 2.170 examples constructed through the aforementioned process. The instruction types are divided into two categories based on the type statements required, and the distribution of correct answers (1, 2, or 3) follows 6:3:1 ratio with varying proportions of halluci- nated and faithful statements across the five options.
3.3 QUALITY CONTROL
We implemented two mechanisms within the dataset generation pipeline to enhance the quality of automatic creation: (I) rationalization and (2) ensemble verification.
Rationalization As illustrated in Figure I GPT-4 omni was instructed to provide the evidence for each statement generated by the generation of hallucinated and faithful statements. This approach aims to enhance the model's reasoning capabilities during the automatic execution of statements, ensuring that it consistently produces high-quality outputs by utilizing the rationale it established as contextual knowledge (Lei et al.) 2016 Wang et al.) 2023 Schimanski et al.) 2024). Moreover, the generated rationale serves valuable resource for assisting both the meta-evaluator and human annotator in verifying the correctness of the outputs during the ensemble verification process.
Ensemble Verification To further improve the quality of the final dataset samples, we con- ducted cross-validation by ensemble three top-performing models meta-evaluaaooss (Dutschmann
5

---

Published as a conference paper at ICLR 2025
Table 3: Ensemble verification results for quality control across domains in the final dataset. The quality acceptance rate represents the proportion of examples in the dataset that received a verifica- tion score of 1. Underline indicates the lowest quality scores
Quality ucceptance rate
Culture
Economy
History
International
Medicul
Society
Technology
Average
GPT-4 (upL-4-065 1------0133
99.54%%
99.59%
98 94%
99.54%
99.35%
99.08%
99.72%
99.39%
CharGPT-4 (chat.gpt 40- att
99%
99.63%
99.31%
99.49%
99.03%
99.08%
99.45%
99.36%
GPT-4 omni (gpt- 10- 40-2022---22222200 08---0--000 06)
99.45%
99.45%
99.22%
99.54%
99.22%
98.99%
99.59%
99.35%
et al., 2023 Manakul et al. 2023 Gilardi et al. 2023). The selected models included GPT-4
--6- 13), which demonstrates state-of-the-art performance in both benchmark tasks and hand-engineered tasks (OpenAT] 2023), ChatGPT-4 (chat.cpt-4o-1atte
intended for evalu- ation, and GPT-4 omni (gpt 40 2024-0222200 - 00), OpenAl's most advanced flagship mode
Table 3 presents the results of the quality evaluation conducted by the meta-evaluators on the five hallucinated faithful statements generated for each of the 2,170 test examples, alongside the ra- tionalized evidence provided for each statement. The verification score was binary: score of 1 was given if both the statement and the rationalized evidence were valid, and if either was problematic. The evaluation results show that even the lowest-scoring domain, History, achieved a high-qualing level, with 08.94% of its hallucinated statements deemed valid. Among the low-quality statements, 68% were flagged by two more models, and 21% were flagged by all three models.
We employed three human annotators, all native Korean speakers, and graduates of four-year uni- versities located in Seoul, Republic of Korea. The annotators reviewed 158 test examples flagged by at least one model as containing low-quality statements. They performed binary classification to determine whether revisions were necessary. A statement two or more annotators agreed on the need for revision. As a result of the human evaluation, 137 test examples flagged as low-quality by the LLM meta-evaluators were found to reflect instances where the models either misinterpreted instructions or hallucinated during the evaluation process. For 21 test examples, at least two human annotators agreed that revisions were necessary. These problematic statements were subsequently revised by one of the authors, a native Korean speaker and Ph.D. candidate. Krippendorff's alpha for inter-annotator reliability was 0.923 among the three meta-evaluators and 0.828 among the three human annotators, indicating high and moderate inter- annotator agreement, respectively (Krippendorfi) 2011).
considered for direct revision if
4 EXPERIMENTS
4.1 SETUP
Models To ensure a broad representation of the LLMs employed in our experiments, we select models that have demonstrated strong performance within the open-source Korean NLP community and have been widely used as baselines in subsequent research or indus- tries. We conduct experiments across three different model types: (I) open-source multilin- gual LLMs, including Llama2 (mtaaaaaaaaaaaaaaaaaaaaaaa
Llama3 (fattar-LLmm//eeeraa-aaaa------- --- 7bb-cchtttt (Touvron et al., 2023),
(Al @ MMtal 2024). and Mistral- Jiang et al. 2023 MistralAl (2) Korcan-centric LLMs, such as KULLM3 (nlpai- lab/ (ab/KKULM3) (Kim et al.
Nemo
2024
2024 which has been solely fine-tuned with limited instruction tuning dataset, and ExaOne (Research et al., 2024), which have under-
- -- gone pre-training and instruction-tuning on broader dataset. These five open-source LLMs are ca- pable of generating high-quality outputs based on the provided instructions with long-form docu- ments and calculating log probabilities for our K-HALU evaluation framework. However, closed APIs, including GPT-3.5 Turbo 3oee-33.-turbb-0122 (Ouyang et al.) 2022). GPT-4 Turbo (spt-4-turbo- - not provide access to log probabilities. As result, evaluation for closed API models is limited to directly generating the indices of the statement choices.
and GPT-4 omni (gpt-4o- 2444002240000 08 6) (OpenAl 2023) do
https://plat openaaaroooooooao.oo.mmmmaaaaaaa http:///w...pppaa..ooocc///////e ccom//ccs//mmmml

---

Published as a conference paper at ICLR 2025
Log Probabilities We adopt multiple choice and log probabilities-basee performance measure- ment as the default approach to ensure stability in performance reproduction and minimize umin- tended interference in evaluating hallucination detection capabilities. To evaluate the multiple-choiee in K-HALU, we compute the conditional probabilities of sequence generation, leveraging the auto- regressive of LLMs (Gao et al.] 2024). For each statement (choice) and its input source S, the
sequence generation probability P(æ) is calculated as follows:
P(x)
log P(a mec))
Here, P(x) represents the token probability computed by the model, with ":" indicating sequence concatenation. The input source content S consists of the instruction 7, the textual document 1, and the publish date d, which can be expressed as S = [/=t:d]. For each of the 5 choices, the cumulative log probabilities of tokens are calculated independently by concatenating the input source S with each choice x. Finally, the Top-N answers, corresponding to the number of correct answers for the task, are selected based on their probabilities. This approach minimizes unintended interference from other choices and mitigates score distortion caused by differences in choice length and structure, maintaining stability in performance reproduction.
Exact Match To enable LLMs to provide judgments directly, we set up the task to have the models generate binary outputs ("0" or "I") indicating the validity of each choice and evaluate them using exact match. We incorporate a post-processing step to minimize errors caused by unnecessary special characters or slight variations in format during the generation process. e.g., "[1, 1, 0, 0, 0J"
LLM-as-a-Judge We use an LLM-as-a-Judge style prompt (See Table 13 to assign scores for hallucination detection results, enabling the LLM evaluate the quality and accuracy of generated text (Liu et al. 2023] Chiang & Lee 2023 Zheng et al. 2024). LLMs are tasked with directly generating faithful or hallucinated statements based on the given instructions. These outputs tend to be descriptive, which increases the potential for errors when evaluated using exact match. To mitigate this issue, we utilize GPT-4 omni (gpt-46 11-400-2022 2202--2-22 1-244008 (-8-06) as an evaluator to classify the validity of the generated statements binary values (0 for invalid, for valid).
4.2 RESULTS
Baseline Accuracy Figure presents the performance of baseline models on the K-HALU test set. Open-source LLMs exhibit an overall low performance, with an average accuracy of 29.05%. ExaOne achieves the highest accuracy 32.95%, while Llama2 shows the lowest performance at 24.75%. Given that the model sizes of the open-source LLMs used in the experiment range between 7B and 13B, the relatively newer models, ExaOne and Mistral-Nemo, demonstrate better perfor- mance. Although KULLM3 is Korean-specific L.M,, it demonstrates limited improvement in de- tecting hallucinations in Korean, likely attributable to the tuning processes constrained by a limited dataset. In comparison, open-source LLMs demonstrate substantial performance gap of 27.9 27.91* 91% when compared to the average performance of 56.96% achieved by closed API LLMS excluding GPT-4 omni, which is directly involved in dataset creation. These results highlight the vulnerabil- ity of open-source LLMs to hallucinations compared to their commercial counterparts. Proprietary LLMs exhibit inherent weaknesses in hallucination detection, stemming from constrained resources and incomplete tuning strategies. Even state-of-the-ard closed API models, whether directly or in- directly involved in the dataset creation process, demonstrate suboptimal performance in detecting hallucinations in Korean, indicating a clear need for further enhancement.
Domain Analysis Table compares the baseline performance across the seven domains. Among the open-source LLMs, the highest average accuracy of 31.8% is observed in the Culture domain, while the lowest performance is seen in the Society domain, with an average accuracy of 27.06%. In contrast, closed API models show the highest average performance in the Technology domain
'Closed API M. are analyzed and compared with open-source M. using the exact match measurement the default setup, given the restrictions accessing log prohabilities.

---

Published as a conference paper at ICLR 2025
Liama2
Liama3
ExaOne
Nistral
GPT-3.5
GPT-4o
0.79
0.8
0.73
0.6
0.4
0.25
0.29
0.29
0.33
0.3
0.2
Accuacy
Figure 2: Baseline accuracy of models on the K-HALU test set. Open-source LLMs show lower accuracy, with ExaOne performing the best among them, while GPT-4 omni achieves the highest overall performance in comparison to other models.
Table 4: Model performance across seven knowledge domains, with accuracy evaluated using log probabilities. Bold indicates the domain where each model achieved the highest performance, while underline represents the domain where each model recorded the lowest performance.
Models
Culture
Economy
History
International
Medical
Soclety
Technology
0.2533
Llama3
ATe 2024
0,3200
0.2843
0.2733
0.3100
0,2923
0.2618
0.2735
KULLM
Kim 2024
0.3333
0.2876
0.2833
0.2736
0.2862
0.2744
0.2667
ExaOne
Reseurch 2ie
0.3367
0.3077
0.3033
0.3526
0.3385
0.3186
0.3467
Mistral-Ner Nemo MilranAl RUM
0.3200
0.3110
0,2933
0,3100
0.3077
0.2808
0.2867
(Onyan
GPT-4 Turbo (UDenA) 2025
0.7667
0.7157
0.6967
0.7325
0.7384
0.7476
0.7433
GPT-4 OpenAA
0.7633
0,8000
at 65.77%, and the lowest in the History domain at 60.44%. ExaOne exhibits similar performance variations GPT-3.5, while the remaining open-source LLMs show comparable trends in their strengths and weaknesses across the domains. However, the overall performance variation across domains not pronounced for any models. This suggests that the general-purpose models used in the experiment are relatively unaffected by domain-specific knowledge gaps or biases toward particular knowledge domains.
Multiple Answers Table pressens the baseline performance according to the number of correct answers across each domain. Open-source LLMs achieve the highest average accuracy of 31.67% for single-answer types while showing the lowest performance of 24.63% for questions requiring two correct answers. Models that perform well on single-answer types also tend to exhibit higher performance on more than two-answer types. Notably, there is sharp decline performance when inferring single-answer in the Technology domain, two in the Society domain, and three in the History domain. These results suggest that the textual sources in the Society and History domains, which involve complex events, may have confused hallucination detection. Additionally, ExaOne's approximately 10% outperforming over other models in the Technology domain for single-answer questions suggests that the lack of training technology-relate knowledge written in Korean likely influenced the results. On the other hand, closed API models demonstrate a marked increase in per- formance more than two-answer types. This outcome appears indicate that these models recog- nized the correlation between answer candidates during the generative evaluation process or that the marginal differences among candidates in the hallucination faithful sets resulted in higher scores when selecting multiple answers, regardless of the ranking of generation probabilities. This result supports the effectiveness of post-hoc sampling methods, such as the one proposed by Manakul al. 2003), in mitigating hallucinations for closed API models.
Instruction Types Table 6 compares the performance based on whether the instruction type in- volves selecting hallucinated or faithful statements. All models, except GPT-3.5 Turbo, perform bet- ter when selecting faithful statements. Open-source LLMs exhibit a significant performance gap be- tween the two question types, averaging 33. 33.18%..... In contrast, the closed API models show marginal performance difference, averaging only 2.06%. These results indicate that open-source multilingual and Korean-centir LLMs still struggle detect hallucinated outputs, raising concerns that their instruction tuning seems biased towards selecting faithful statements. Moreover, as the number of correct answers increases, performance question types that require selecting hallucinated state-
8

---

Published as a conference paper at ICLR 2025
Table 5: Model performance across seven domains based the number of multiple answers with accuracy evaluated using log probabilities. (1/2/3) indicates the model's performance according to the number of labels. Bold: highest, underline: lowest performance.
Modele
Culturn (1/2/))
Reonumy (1/23)
History (1/25))
International (1/215)
Medical (1/2/)
Society (1/2/)
Vechnology (1/21)
Total (172/5)
Liamz
0.32/0.000.30
0.30/023 70.33
0.29/021 10.3
0.28 0.20/015/3.p
0.26/0.15.0.9
0.26 0.11 /0.22
0.28/0.30/0.23
0.25 /0.10/000/ 0.20
Lamed
0:36/426.11
03220022003 20.20
0.120036 /8/0.
0.3/00/5500
0.31 /0.18 70:35
0.79 1279/027 70:29
0.32/0 (0:
KULMM
0:38/426/0..1
63120221 70.20
0.33003240000
0:910034 70.25
0.30/025/00
0.35/0.33 70:35
0.26 0
4.32/0.25/
FoaDre
0:35/030/0.3
6.900032/00
0.377627740
0.3/0.37 0044
0.31/0.36/04
0.37/0.22 70:31
0.38/0.30 70:27
0.34/0.31
Misral
0.35/0..7711 10:30
0:347027/00 /0.30
0.54/0.66001
0.52/0.77 20.39
0.31/0.30/0.35
0.33/0.19 00225
0:27/03000.33
0.32/0.22
0,30//0.0911 10:55
0:301163/
70073
(.22/
GPT-4
0.59 /0.77/
0.57 /0.37 /0.90
0.62 /0.97
0.65/0.83 /0.91
0.62/0.4/008
10.95
GPT-4o
0.66/0.98/1.
0..6/0
0,64/0.93
0,69/096/1.0
0.67/0.550.99
0.70/
0.69/0..7
0.07 0.95
Table 6: Model performance in hallucination and fact selection types across the number of multiple answers with accuracy evaluated using log probabilities. H. Selection refers to the hallucination
selection type and F Selection refers the fact selection type. A number in () represents the number of labels, and Avg. denotes the average score for each type. Bold: highest performance.
Models
H H.Selectoon (1)
F_Selection (1)
H. Selelection (2)
F.Selection (2)
H. H.Selection (3)
F.Selection (3)
H.Avg.
F_Avg.
Llama2
0.1656
0.4015
0.0736
0,0373
0.3611
0.1253
0.3696
Llama3
0.1825
0-4554
0.0859
0.3884
0.0280
0:4815
0.1382
0.4378
KULLMB
0.1656
0.4723
0,0644
0,4006
0,0186
0.4815
0.1207
0.4516
ExaOne
0.1488
0.5338
0.0644
0,5505
0.0467
0.6019
0.1134
0.5456
Mistral
0.1626
0.4815
0.0767
0.4465
0.0467
0.5463
0.1253
0.4774
GPT:3.5
0.3098
0.2215
0,5920
0. 0.6697
0,4860
0.6296
0.4120
0.3972
GPT-4
0.6043
0.623]
0.8834
0.9266
0,9626
0.9352
0.7235
0.7456
GPT-40
0.6288
0,7185
0,9479
0.9480
0,9813
0.9815
0.7594
0.8138
ments drops sharply. However, performance on faithful statement selection tends to improve. As discussed in Multiple Answers, this is considered to result from the smaller differences among can- didates within the faithful set, whereas the larger dissimilarities within the hallucinated set arise from the model's inability to accurately identify hallucinations.
Few-shot and CoT To address open-source LLMs' low hallucination detection capabilities, we apply few-shot samples and Chain-of-Thoughh (CoT) reasoning (Wei et al. 2022) to evaluate per- formance improvements. We create three samples containing multiple-answer types to measure few- shot accuracy. These samples are constructed based on the results from Table [ with two selecting hallucinated statements and one selecting faithful statements. Additionally, we incorporate CoT rea- soning steps, guiding the models to choose appropriate statements and rationalize their choices based on the provided documents and publication dates according to the instruction type. Appendix (F iFliluus trates the design of the CoT prompts. Table 7 compares the performance of the baseline models with the results after applying few-shot and COT. Llama2 and ExaOne exhibit a slight decrease in perfor- mance, while Llama3, KULLM3, and Mistral show modest gains within 2%. These findings align with the results of HaluEval (Li et al.] (023). suggesting that few-shot sampling and CoT reasoning steps are not sufficient as fundamental solutions for improving hallucination detection.
Exact Match and LLM-as-a-Judge Table g presents the results of applying evaluation meth- ods where models generate answers directly based on the prompt's 's instruction and context. When using exact match to evaluate performance, it becomes more challenging for models to achieve high accuracy compared to multiple choice accuracy. This is because exact match heavily relies on instruction-fole follooong abilities, which amplifies performance differences among models. Addition- ally, the influence of instruction-followiii or CoT settings compared to other evaluation methods. To mitigate potential distortions caused by descriptive outputs in exact match evaluations, we employ the LLM-as-a-Judge evaluation method. Baselines show low performance as they struggle to generate answer choices that align with the given instructions. Interestingly, KULLM3 performs better the L.-Maas-a-Judee evaluation compared to other methods. ExaOne and Mistral-NEMO demonstrate generally higher performance across the multiple choice, exact match, and -s-a-gee evaluations. Similar to multiple choice and log- nog-probability-baed based evaluations, applying 3-shot or CoT reasoning results in partial performance improvements. However, consistent performance enhancement is not observed across all settings. These results highlight the challenges ooen-source LLMs face in directly solving K-HALU's hal- lucination detection tasks through generative outputs or CoT prompting. This implics the need for further research to improve model capabilities in hallucination detection tasks effectively.
appears have a greater impact on performance in 3-shot

---

Published as a conference paper at ICLR 2025
Table 7: Model performance by # shots and CoT with accuracy evaluated using log probabilities.
Models
Zero-shot
3-shot
3-shot CoT
Llama2
Touvron et al. 2025
0.2475
0,2410
0.2429
Llama3 ATeMaR 2024
0.2880
0.2880
0.2926
KULLMS [Kun 2024
0.2862
0,3018
0.2959
ExaOne
[Rseaach ei c.. 2024
0.3295
0.2922
0.2991
Mistral Nemoo
2024
0.3014
0.3060
0,3115
Table 8: Performance comparison of models using Exact Match and auddee-a-Judge
Models
Exact Match
L.M.assa-Judge
Zero-shot
3-shot
3-shot + CoT
Zero-shot
3-shot
3-shot + CoT
I.aam22 Nouvrrn et
0,0106
0.0051
0.0060
D.0101
0:0115
0.0189
Llamas Meta
X2A
0.1203
0.1760
0.0213
0,0212
KULLMS Kim
202
0,0191
0.0083
0.0070
0,1134
0,1005
0.0797
EsaOne Research et
2024
0.0484
0.1885
0.1839
0.0922
0.1276
0.1023
Mistral-N NEMO MistralAT
2024
0.1240
0.2668
0.2770
0.0995
0.1138
0.1106
Qualitative Analysis We qualitatively analyze cases where six or more models from the eight baselines select incorrect answers for given example in each domain. Incorrect examples are cat- egorized based on the rational evidence for faithfulness or hallucination in the answer options. We focus on the incorrect answer choices commonly identified across multiple models to summarize representative examples of hallucinations within the K-HALU benchmark. Figure 3 from our review of these incorrect examples across domains. Errors such as misrepresenting the publish date, misunderstanding numerical relationships, and conflicting knowledge are prevalent and recurrent issues across the models. In another example, models produce incorrect outputs for faithful statement about 'NAVER's ' intelligent search robot announced in September 2000,' they incorrectly believe the phrase "intelligent search robot" inappropriate. Likewise, when the document mentions 'the largest project in history.' models generate incorrect answers, arguing that this does not align with their prior knowledge. Also, models frequently produce domain -ppeciff errors when trained on inaccurate knowledge of specific entities. For example, documents mentioning "President Syngman Rhee" consistently lead to incorrect answers across multiple models.
contains notes
CONCLUSION
We propose the K-HALU benchmark, designed evaluate hallucinations in Korean, K-HALU eval- uates ballucination detection by verifying faithfulness using Korean knowledge-ber based textual docu- ments drawn from seven different domains. The benchmark introduces a strict evaluation framework that includes multiple-answer questions, requiring models to select all possible correct answers. This approach enables us to quantify the hallucination detection capabilities of LLMs, providing deeper insight into their ability discern faithfulness. Our analysis demonstrates that open-source LLMs are still struggling with hallucinations and that there remains a significant performance gap com- pared to closed API models. In future work, aim to focus enhancing datasets and models to mitigate hallucination in Korean, building on the insights gained from this research. We hope K-HALU helps improve the Korean NLP community's LLM hallucination performance.
ETHICS STATEMENTS
The authors are solely responsible for any legal issues related to the K-HALU benchmark and ac- knowledge all associated risks. We state that the source dataset used to construct the K-HALU benchmark, Knowledge Graph-to-Text was developed as part of the Intelligent Information Indus- try Infrastructure Construction project, supported by the Ministry of Science and ICT and the Korea National Information Society Agency (NIA). In utilizing this source dataset to create K-HALU, we comply with the Open AI Dataset (AI-Hub, S. Korea) copyright policy and confirm that its use and distribution have been pre-approved through prior consultation with NIA. The K- K-HALU benchmark available for download and use by individuals who agree to the terms regarding its intended usage and the AI-Hub data usage policy. The link to access the dataset is provided in the K-HALU GitHub
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
K-HALU used datasets from The Open AI Dataset Project (AI-Hub, S. Korea). All dataset-related information can be accessed through AI-Hub (www.aihub.orkt)) This work was supported by In- stitute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) RKS-2024-00398113 Research on the reliability and coherence of outcomes produced by Generative AI). This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-20218) 1RIAA A6A A03045425). This work was supported by Institute of Information & commu- nications Technology Planning & Evaluation (IITP) under the Leading Generative Human Re- sources Development (UTP-2024-22408111) grant funded by the Korea govemment (MSIT)
REFERENCES
AI@ Meta. Llama 3 model card. 2024.
URL https://giithb..cc//eeeet
1lama3/blok wwt.//ainnniii /omminn/MMELL..... CARD md
Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 967-976, 2023.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned 2023.
lens. arXiv preprint arXiv:2303.088.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs' internal states retain the power of hallucination detection. In The Twelfth International Conference on Leurning Reprexentations, 2024. TRLLhttpss///penrreet net/ forum?
id=z312nz Qbz
Cheng-Han Chiang and Hung-Yl Yi Lee. Can large language models be an alternative to human evalua- tions? Proceedings of the st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15607-15631, 2023.
Prafulla Kumar Choubey, Alex Fabbri, Jesse Vig, Chien-Sheng Wu, Wenhao Liu, and Nazneen Ra- jani. Cape: Contrastive parameter ensembling for reducing hallucination in abstractive summa- rization. In Findings of the Association for Computational Linguisties: ACL 2023, pp. 10755- 10773, 2023.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2023.
Alexis Conneau, Ruty Rinott, Guillaume Lample. Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp- 2018
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers embedding space. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguis- tics (Volume I: Long Papers), pp. 16124-16170, 2023.
Thomas-Marte Dutschmann, Lennart Kinzel, Antonius Ter Laak, and Knut Baumann. Large-scale evaluation of k-fold cross-validation ensembles for uncertainty estimation. Journal of Chemin- formatics, 15(1):49, 2023.
Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar R Zaiane, Mo Yu, Edoardo M Ponti, and Siva Reddy. Faithdial: A faithful benchmark for information-seekke seeking dialogue. Transactions of the Association for Computational Linguistics, 10: (0:1473-1490. 2022.
Julen Etxaniz, Oscar Sainz, Naiara Miguel, Itziar Aldabe, German Rigau, Eneko Agirre, Aitor Ormazabal, Mikel Artetxe, and Aitor Soroa. Latxa: An open language model and evaluation suite for Basque. In Lun Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of

---

Published as a conference paper at ICLR 2025
the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pp. -4952-14972, Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 10. 1/8833//112024.aclloooooo acc- 1og..oog.... 799
Alexander Richard Fabbri, Wojciech Kryšcińskki, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409, 2021.
Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. Qafacteval: Im- proved qa-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 2587-2601, 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffreey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL httthhttt///eeeeoo 12608602]
org/ records/
F Gilardi, M Alizadeh, and M Kubli. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences of the United States of America, 120 2023.
(80)=22305016110023000006
Nuno M. Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. Hallucinations in large multilingual translation mod- els. Transactions of the Association for Computationall Linguistics, 11:1500-1517, 2023. doi: 10.1162/tacl a 00615. httttttss: a/cl //t.a/////o/......l.o.o........ org 2aarg22223. 2023. tacl-1lll1. 1.. 85
Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. Kornli and korsts: New benchmark datasets for korean natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 422 222-430, 2020.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt, Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL bettps://openrewyee net forum?id-
d7KB 3GmQ.
Lei Huang, Weijiang Yu. Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint 1tXiv:23111052222 2023a.
Yuheng Huang, Jiayang Song, Zhijie Wang. Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:2307.10236 2023b.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of ballucination in natural language generation. ACM Computing Surveys, 55(12):1-388 2023a.
Ziwei Ji, Zihan Liu, Nayeon Lee, Tiezheng Yu, Bryan Wilie, Min Zeng, and Pascale Fung. Rho: Reducing hallucination in open-domain dialogues with knowledge grounding. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 4504-4522, 2023b.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.068/ 2023. URL actps: : mi starl.rr....... ai/news/
announcings ng- rtral-
12

---

Published as a conference paper at ICLR 2025
Che Jiang, Biqing Qi, Xiangyu Hong, Dayuan Fu, Yang Cheng, Fandong Meng, Mo Yu, Bowen Zhou, and Jie Zhou. On large language models' hallucination with regard to known facts. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 1041-1053, 2024.
Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Alice Oh, and Hwaran Lee. Kobbq: Korean bias benchmark for question answering, Transactions of the Association for Computational Linguis- fics, 12:0075524, 2024.
Jeongwook Kim, Taemin Lee, Yoonna Jang, Hyeonseok Moon, Suhyune Son, Seungyoon Lee, and Dongjun Kim. Kullm3: Korea university large language model 3. https:///ithub. com/ n Alppi-lab/kull 2024.
Klaus Krippendorff. Computing krippendorff's alpha-reliability, 2011.
URL
https:
//citeseerx pwt.:/isssssp..uuuueeeeeuuuuue http:ppppp.....ccouu..oooeoooooo
&doi-
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 107-117, 2016.
Haonan Li, Martin Tomko, Maria Vasardani, and Timothy Baldwin. Multispanqa: A dataset for multi-span question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1250-1260, 2022.
Junyi Li, Xiaoxue Cheng. Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Halueval: A large- scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1449-6464, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers). pp. 3214-3252, 2022.
Tianyu Liu, Xin Zheng, Baobao Chang, and Zhifang Sui. Towards faithfulness in open domain table-to-text generation from an entity-centric view. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13415-13423, 2021.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511-2522, 2023, URL https:
//aclanthology http:/hhoooogyo.gg2222222222 www.//ggg/22.... emnlp moo-main.... 153/
Zhiyi Luo, Yingying Zhang, Shuyun Luo, Ying Zhao, and Wentao Lyu. A dataset of open-domain question answering with multiple-span answers. arXiv preprint arXiv:2402.0992. 2024.
Himanshu Maheshwari, Sumit Shekhar, Apoorv Saxena, and Niyati Chhaya Open-world factually consistent question generation. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 2390-2404, 2023.
Potsawee Manakul, Adian Liusie, and Mark Gales, Selfcheckgpt: Zero-resource black-box halluci- nation detection for generative large language models. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp.
2023.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1919, 2020.
MistralAI. Mistral nemo model card. 2024. URL Ittpss:/haaginggaaeecc////tit/es
Mistral Nemo Base 2407 httpc///lbbbb///
RERRREA.EE.... md
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-super sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 16-30, 2023.
13

---

Published as a conference paper at ICLR 2025
nostalgebraist. Interpreting gpt: the logit lens, 2020. URL wttp:ttttwwww.eswsst..s..
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow in- structions with human feedback. Advances in Neural Information Processing Systems, 35:27730-
iew. 27744, 2022. URL 2ttp:///222222033333310030001000 2022 2022/Elle/blefdeee
Chanjun Park, Hyeonwoo Kim, Dahyun Kim, SeongHwan Cho, Sanghoon Kim, Sukyung Lee, Yungi Kim, and Hwalsuk Lee. Open Ko-LLM leaderboard: Evaluating large language models in Korean with Ko-h5 benchmark. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Vol- ume 1: Long Pupers), pp. 3220-3234, Bangkok, Thailand, August 2024, Association for Compu- tational Linguistics. doi:
177. URL https://aclanthon
177
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Ji Yoon Han, Jangwon Park, Chisung Song, Junseong Kim, Youngsook Song, Tachwan Oh, et al. Klue: Korean language understanding evaluation. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2022.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. Xcopa: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2362-2376, 2020.
Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowe Dowmunt. The curious case of hallucinations in neural machine translation. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1172-1183, Online, June 2021. Association for Computational Linguistics. Linguitt doi: 10..8853/v maacl-man.2- URL https:
//aclanthology. wrt.yy.or//22/00 hrt.aa/hoooogy.oor/2222...aae. naacl www.::0ma..aa.mmm............... 92
Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S.M Towhidul Islam Ton- moy, Aman Chadha, Amit Sheth, and Amitava Das. The troubling emergence of hallucination in large language models
extensive definition, quantification, and prescriptive remediations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empiricul Methods in Natural Language Processing, pp. 2541-2573, Singapore, December 2023. Association for Computational Linguisties. doi:
wtt.3//1/222.emmlpp.. httpss:/aclaathooooe wttp:yyyorg//2000 emnlp ma maiin........ 155
URL
LG Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Stanley Jungkyu Choi, Yemuk Choi, Seokhee Hong, Yeonjung Hong, Junwon Hwang, Hyojin Jeon, et al. Exaone 3.0 7.8 b instruction tuned language model. arXiv preprint wrXiv:240.00341, 2024.
Tobias Schimanski, Jingwei Ni, Mathias Kraus, Elliott Ash, and Markus Leippold. Towards faithful and robust IIm specialists for evidence-based 2024.
Available at SSRN 4728973,
Jaehyung Seo, Seounghoon Lee, Chanjun Park, Yoonna Jang, Hyeonseok Moon, Sugyeong Eo, Seonmin Koo, and Heui-Seok Lim. A dog is passing over the jet? text-generation dataset for korean commonsense reasoning and evaluation. In Findings of the Association for Computational Linguistics: NAACL 2022. pp. 2233-2249, 2022.
Jaehyung Seo, Jaewook Lee, Chanjun Park, SeongTae Hong, Seungjun Lee, and Heui-Seok Lim. Kocommongen v2: A benchmark for navigating korean commonsense reasoning challenges in large language models. In Findings of the Association for Computational Linguistics ACL 2024, pp. 2390-2415, 2024.
14

---

Published as a conference paper at ICLR 2025
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguis- tics: EMNLP 2021, pp. 3784-3803, 2021.
Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. Kmmlu: Measuring massive multitask language understanding in korean. arXiv preprint arXiv:2402. 11548, 2024a.
Guijin Son, Hanwool Lee,, Suwan Kim, Huiseo Kim, Jac cheol Lee, Je Won Yeom, Jihyu Jung, Jung woo Kim, and Songseong Kim. Hae-rae bench: Evaluation of korean knowledge in language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLIN) 2024), pp. 7993-8007, 2024b.
James Thorne, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. Fever: a large- scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Papers), pp. 109-819, 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foun- dation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. URL https:
//doi. /00 48550/ 2307.09288
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions evaluate the fac- tual consistency of summaries. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Aunual Meeting of the Association for Computational Linguistics, pp. 5008-5020, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.acl- main.450. URL
1 ogy. wrgg:.rgg/200 /elgg//2200 2020 acl manlllmmin.... 450
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. The Eleventh International Conference on Learning Representations, 2023. URL
hittps://opt http:s////ppeeeeeeee..omm. net/ wtwp://wwwweeeeooruieeeeeee forum?id=1PLINIII
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 -48377, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791 4800, 2019. URL
o
P19-1477 144277
Haopeng Zhang, Semih Yavuz, Wojeiech Kryšciñíski, Kazuma Hashimoto, and Yingbo Zhou. Im- proving the faithfulness of abstractive summarization via entity coverage control. Findings of the Association for Computational Linguistics: NAACL 2022, pp.
2022.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's 's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint 2rXiv:2309.01211 2023.
Yiran Zhao, Jinghan Zhang, Chern, Siyang Gao, Pengfei Liu, Junxian He, et al. Felm: Benchmark- ing factuality evaluation of large language models. Advances in Neural Information Processing Systems, 36, 2024.
Zheng Zhao, Shay B Cohen, and Bonnie Webber. Reducing quantity hallucinations in abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp- 2237-2243 2020.
Liammin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging jlue with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. URL https: 7/d1. am.. wwt...moo/////////oo//
466814
15

---

Published as a conference paper at ICLR 2025
A QUALITATIVE DETAILS
Figure 3 shows the analysis of hallucination error types frequently occurring in eight baselines across each domain.
Catagorias
Examples
Culture
1. Knowledge Conflict 2. Publish Date
1. Conflict with the latest knowledge regarding the adverb "most"
Inaccurate Knowledge Entities
Mscziculatio dale in expressores "thes year."
Lexical Ambiguity S. Author Blussoo
Confusion regarding the inppeahment order the president.
Emors due to ambiguty in terns retated association membership.
mepping of book's
Economy
1. Knowledge Conflict Publishh Date
1. Jucged the phrese "intelligntt sear ch robot" as inappropriate late in the text.
Hallucmation
Macaiculation
retated "this year."
Error calculiting the bid emount.
History
Inaccurate Knowledge Entilles Numerfeal Hallucination Negation
3. Incorrect knowledge about the Koree historical incident. Error year.
7. Msiieepretatiin of negation.
International
Knowledge Publissh
Consistert errors regarding content Eirors concemiin the publication date of Journa..
cemsorship.
Lexical Ambiguity Numerical Hallucination
wver-generatiiaiion the term "efficient"
Error calcuiating factory's
Medical
1. Knowledge Conflict Publish Date
1. Knowledge contict regerding terms like "latest" and "new drug."
Knowledge Entities
event
Numerical Hallucination Negation
Falure disanguish between academic conference and forum.
Emor piobability ty calculation.
Msinterpeetation negative expressions.
Society
2. Publish Date
2. Frequent enors regarcing the publicatior year.
Lexical Ambiguity Author Illusion
Ambigutty-relteed errors Interpreting educational content. Spacing errors suthor's name.
Technology
Knowledge Conflict Publish Date
Knowlddge conflict about the new product. Miscaiculation years related
Numerfeal Hallucination Negation
year 6. Labeling error reçarding the version of the game. Maintarpretation of nagative expressions.
Figure 3: A review note for incorrect answers from baseline models.
B F1 SCORE
Table the Fl score, precision, and recall metrics for evaluating model performance. These metrics are calculated using macro-average approach, which considers the multiple correct answer distribution (6:3:1) and the uniform label distribution across the dataset. Accuracy requires selecting all correct answers to receive a score, meaning partially correct responses are treated as entirely incorrect. In contrast, the F1 performance for the models.
accommodates partial correctness, resulting in improved overall
Open-source LLMs, such LLAMA, KULLM3, and ExaOne, tend to predict only subset of the correct answers, prioritizing those with higher log probabilities. This behavior leads to higher precision compared recall, as these models adopt more conservative strategy in their predictions. On the other hand, closed API LLMs, including GPT-3.5, GPT-4, and GPT-4 omni, generate answers more flexibly, often producing greater number of options than the predefined correct answers. This results in higher recall than precision, as these models aim capture all possible correct answers occasionally overpredict.
C EXTENDING K-HALU TO MULTILINGUAL
Setup We conduct multilingual experiments using English (a high-resource language) and Malay (a fow-resource language) to provide insights into the adaptability of the K-HALU benchmark across languages. K- C-HALU's hallucination detection criteria and settings for multiple-answe questions are designed to be languages.
making the evaluation framework adaptable to different
16

---

Published as a conference paper at ICLR 2025
Table 9: Performance of L.MSs K-HALU evaluated by Accuracy, F1, Precision, and Recall.
Model
Accuracy
F1
Precision
Recall
Llama2 Touvron et al.
2023
0.2475
0.3939
0.4152
0.3979
Llama3 MeT@Meta, 2024
0.2880
0.4332
0,4503
0.4356
KULLM3 (Kim (Kim el al..
2024
0.2862
0.4156
0,4323
0,4174
ExaOne (Research
2024
0.3295
0.4770
0.5117
0,4760
Mistral-NEMOO (MistralA) 2024
0.3014
0.4393
0.4530
0,4417
GPT-3.5 Turbo (Ouyang et al
2022
0.4046
0.7237
0.6476
0.8260
GPT-4 Turbo (OpenAl 2023
0.7346
0.8538
0,8211
0.8899
GPT-4 omni (OpenAl 2023
0.7866
0.8732
0.8552
0.8924
Table 10: Performance of Models K-HALU (English and Malay Versions).
Models
Accuracy
F1
Precision
Recall
K-HALU (English ver.)
Llama2 Touvron al. 2023
0.2757
0.4124
0.4261
0.4131
Llama3 Meta 2024)
0.2727
0,4012
0.4139
0,4038
KULLM3 (Kim al. 2024
0,2846
0.4082
0.4235
0.4091
ExaOne
Research et al.
2024
0.2638
0.4079
0.4288
0,4106
Mistral-Nemo (MistralA) (MistralAA
20124
0.2906
0.4354
0.4496
0.4364
K-HALU (Malay ver.)
Llama2 (Tourron et al.
2023
0.2620
0.4317
0.4393
0,4314
Llama3 Meta 20244
0.2715
0.4443
0.4528
0.4452
KULLM3 (Kim et al. 2024
0.2600
0.4268
0,4373
0,4274
ExaOne (Research et al.
2024
0,2505
0.4171
0,4297
0,4189
Mistral-Nemo [MstralAA]
2024
0.2772
0.4491
0.4591
0.4476
To set up K-HALU for multilingual evaluation, we translate 2,170 samples into English and Malay using the ChatGPT-4 carefully crafted to minimize changes in hallucination-rele related content linguistic nuances that could alter faithfulness (See Table To ensure the quality of translated samples, we implement an API verify whether the translations preserved the original labels. Low-quality samples that failed this binary classification check are filtered out (See Table [2p. As result, 671 English samples and 523 Malay samples remain, indicating that only approximately 27.51 1%5 of the samples retain their validity after translation.
1o-et-4--latee Mo-latest) API. During the translation process, prompts are
I.LM-as-a-Judee evaluation script using the GPT-4 omni (gpt-4o- hoptt-4--202408-10
Results Table 10 presents the results of hallucination detection evaluation performed on the K- HALU benchmark using translations of the Korean textual documents and statements into English and Malay. The results indicate that LLMs continue struggle with hallucination detection, even when K K-HALU is evaluated in different languages. The Korean version of K-HALU achieves ap- proximately 2.7% higher performance compared the translated versions, highlighting the influ- ence of language-specific embedded knowledge on hallucination detection performance.
To achieve optimal evaluation outcomes, is essential to reconstruct reliable knowledge documents and faithful statements based on the original source in each language. Each language's cultural context and linguistic nuances require adjustments to hallucination detection methods and multiple- answer question approaches to ensure accuracy and relevance.
D LENS OBSERVATION
We employ two-lens observation methods to examine how each layer of LLM calculates the next token probability for both hallucinated and faithful statements.
17

---

Published as a conference paper at ICLR 2025
Table 11: Prompt for translating K-HALU English and Malay. This table demonstrates the in- struction and system setup for translation tasks,
Nitt Translation Task Prompt- Korean to English whri
##System:
You are a human annotator and a Korean Korean-Engllihh translator.
Your lask to translate the provided Korean lext into English without altering meuning structure.
#UInstruction:
The text consists of "id". "document", "date", "instruction", statementss "1", "2", "3",, "4",, "5",, and "label".
Your task is to:
Translate the "documenn". "instruction", and statements ("1", "2", "3", "4", "5") from Koreun English.
##Requirements:
Ensure changes made beyond accurate translation.
MMintaai the structure the original text your translation.
**Genneate the result in the form of ISON dictionary identical to the Output Format **
##Output Format:
("id":" (idd" ".""""""n "document" "" (translated translated_oocummnn) "aaamnetttt""""t"""" ""date"""
"instruction" "
"translated "arastoo""""aasslataaeee
"I"""(translaa" (translated (translated_sttteemennt
statssttttttttt1... 1)" "":" translated_ssatemmnn..2)
"3":"" setinlated_statemmeee 3) """""""" (manclatte statement,
"5" (translated stttttttttttt... """""lab""" "[label)")
#9Input Format:
Translation Task Prompl-
Malay
#0System:
a annotator l Korean Malay translator. Your task is to translate the provided Korean text into Malay without altering its meaning or stracture.
#eInstruction:
The text consists "id", "document", "date", "instruction"" statements "1", "2", "3", "4",, "5",, and "label".
Your
Translate the "document", "instruction", and statements ("1", "2", "3", "4", "5") from Korean to Malay.
##Requirements:
Ensure no changes are made beyond accurate translation. original
Maintain the structure
your translation.
**Generate the result in the form of ISON dictionary identical to the Output Format.* **
##Output Format:
("id"." (id)" "d"""""men"""" ((transaated (manslate__oocumeen] " "ddate" "date":""danc""""" (daree)
"instruction""" "[translated_insttr
"I""" (trassaated_stttemmen. 1)"" """""" (ranslated.stttmmentt]]
"3"5" gounslated__sattemmeet 3/"--"""" (tansliaddd [unnclated_sattement...))
"5"h" (trrnnslated statementt 5) "label" "(label)"))
#9Input Format:
Logit Lens (nosalleebraist 2020) maps the model's 's representation space the vocabulary space for each token, leveraging the residual stream for this process. This method enables us to observe how the model's final output evolves according to token and layer positions. It is widely used for interpreting the internal representations of Transformer-baae based language models (Dar et al. 2023 Hanna et al. 2024).
Tuned Lens (Belrose al., 2023) addresses some limitations of the Logit Lens, such as its diffi culty in dealing with inconsistent readiness for final decoding across different positions. The Tuned Lens improves upon the Logit Lens by aligning intermediate layer outputs more closely with final predictions, facilitating the capture of more abstract and semantic information (Nanda et al. 2023 Jiang et al. 2024).
The selection of positions for measuring token-wise changes in hidden states across layers through lens observation depends on the experimental objectives (Azaria & Mitchell 2023 Chen et al., 2024]. In our analysis, we focus on the final token of the output to assess the model's l'ss judgment and the potential occurrence of hallucinations in response to the input.
Results To further analyze the results from Instruction Types, we track the model's internal state during next-token prediction using lens observation. Figure Soureeiillen iillutrates the probability shift across each layer for Llama2, according to hallucinated and faithful statements. Under the Logit Lens (000)) (nostalgebraist) (020). slight probability fluctuations are detected in the early layers, with a gradual increase starting from the middle layer (15*), and a sharp rise of approximately 40% near the final layer. For instructions that differentiate hallucinated statements, there is slightly higher probability than for instructions that differentiate faithful statements up until the final layer, where the model becomes more confident about faithful statements. Under the Tuned Lens (Belrose et al. 2023). sig- nificant probability fluctuations are detected in the early layers, followed by another notable increase around the 15th layer. After this point, little no probability shifts are observed. Similar to the Logit
18

---

Published as a conference paper at ICLR 2025
Table 12: Prompt for evaluating translation quality and hallucination label correctness. This table outlines the system and task instructions for assessment.
#iй Translation Evaluation Prompi English Label Verification ###
##Syystem:
You are professional annotator tasked with evaluating quality translations dataset and ensuring the correctness of hallucination-rclatee labels. Your role strictly whether the provided "label" aligns
with the corresponding "document" and "instruction."
## Tassk:
Evaluate the following ISON object for its correctness based on the criteria below:
Label Validity: Verify if the provided "label" correctly reflects the instruction each statement compured the "document" and "instruction."
difRequirements:
Do not provide explonations justifications for scores.
Only output the score, c.g., 1 or
##Scoring:
Assign point the "label" valid.
Assign points if the "label" is invalid.
dffExample Input:
"001".
"document": "The National Assembly is the legislative body of South Korca.",
"date": "2024-11-01"
"instruction": "Identify which statements are factually consistent with the document.", "1": "The National Assembly is located in Sevul.", "2": has 300 members elected by the public.".
the judiciary.". "4"h "It was established 1948.".
"3": "The Assembly also
"5": "The President of South Korea is a member of the National Assembly.",
"label": [1,2,4]
#xExmmlle Output:
#llnput to Evaluate:
#*# Translation Evaluation Prompt Malay Label Verification # ##System:
You are professional amnotator tasked with evaluating the quality of Malay translations in dataset and ensuring the correctness of ballucination--elated labels. Your role is to strictly assess whether the provided "label" aligns the corresponding "document" "instruction."
#aTakk:
Evaluate the following JSON object for correctness based the criteria below:
1. Label Validity: Verify if the provided "label" correctly reflects the instruction of each statement compured to the "document" and "instruction." ##Requirements:
Do provide explanations justifications for your scores.
Only output the score, e.g. O.
#ilSeoring:
Assign point if the "label" valid.
Assign points the "label" invalid.
##Example Input:
'id": "001",
"document": "Majlis Perundangan Kebungsaun udalah badan perundangan Korea Selatan.", "date": "2024-1100"
"instruction": "Kenal pasti pemyaataan yang konsisten dengan fakta dalam dokumen.". "I" "Majlis Perundangan Kehangsaan terletak di Scoul.", "2": "la mempunyai 300 ahli yang dipilih olch rakyat.",
"3": "Majlis ini juga menyelia badan kehakiman.",
"4"; ditubuhkkan pada tahun 1948."
"5": "Presiden Korea Selatan adalah abli Majlis Perundangan Kebungsaan.",
"label": [1.2,4]
Example Output:
#lnput Evaluate:
Lens results, the model shows greater confidence in differentiating faithful statements compared to hallucinated ones as it approaches the final layer.
These findings suggest that open-source LLMs exhibit stronger certainty for faithful statements in their final layer outputs, contributing the performance gap when handling instructions that differ-
19

---

Published as a conference paper at ICLR 2025
Tuned
Figure 4: Token probability shift of Llama2 on the K-HALU under Logit and Tuned Lens. true/halu indicates that the instruction used selects faihhful/hlllccintted statements. answer/wrong refers to cases where the correct/incorrect option is chosen according to the instruction.
entiate hallucinated statements. The probability shift near the final layer reaffirms the importance of this stage in hallucination detection, as highlighted by Jiang et al. (2024) and Chen et al. (2024), and serves a key factor explaining the performance difference observed in Table 6 Moreover, the Logit and Tuned Lens observations show that the internal token probabilities do not exhibit sig- nificant differences when instructions involve correct and incorrect statements. This indicates that the Llama2 model struggles to accurately assess hallucinated statements, which explains its notably poor performance our previous experiments.
E LLM-AS-A-JUDGE STYLE PROMPT
Table 13 presents the prompt used for the LLM--as-a-ludge evaluation.
F CHAIN OF THOUGHT (COT) PROMPT
Table 14 presents the CoT prompt used for evaluating log probabilities, Table 15 for evaluating Exact matches, and To for the LLM-as-a-Judge evaluation method.
G K-HALU EXAMPLE
Figure 6
8
and illustrate examples of the original Korean texts from the K-HALU
dataset for each domain.
20

---

Published as a conference paper at ICLR 2025
Table 13: Prompt for evaluating LLMs' prediction quality and hallucination label correctness. This table outlines the system and task instructions for assessment.
### Prediction Evaluation Prompt - Label Verification ####
##System:
You are a professional annotator tasked with evaluating the quality of predictions in a dataset and ensuring the correctness of ballucination-relaee labels.
## Task:
Evaluate the following JSON object for its correctness based the criteria below:
1. Prediction Validity: Verify the provided "preds" correctly reflects the instruction of each statement pared to "document" and "instruction." #RRequirements:
Do not provide explanations or justifications for your scores.
Only output the score. e.g., or O.
##Scoring:
Assign 1 point if the "preds" is valid.
Assign 0 points if the "preds" is invalid.
##Example Input:
"001",
"document": "The National Assembly the legislative body of South Korea",
"date": 2024-11-01
"instruction": "Identify which statements are factually consistent with the ducument",
"choices":
The National Assembly is located in Seoul.",
"2. It has 300 members elected by the public.". "3. The Assembly also oversees the judiciary.".
*4. It established 1948.".
"5. The President of South Korea memher of the National Assembly."
].
"gold":
"1. The National Assembly is located Seoul.",
"2. It has 300 members elected by the public.",
"4. It was established in 1948."
].
"preds":
*1. The National Assembly is located Scoul""
22. has 300 members elected by the public."
##Example Output:
#mnput to Evaluate:
21

---

Published as a conference paper at ICLR 2025
Table 14: CoT reasoning prompt used for hallucination detection tasks evaluated using log probabil- ities, Each example illustrates the evaluation process, where statements are analyzed for consistency with the given document and classified as hallucinated or faithful.
### CoT Reasoning Prompt Hallucination Detection (Log probabilities) #ill
##Example I:
date: (INPUT DATE)
document: (INPUT DOCUMENT)
instruction: (INPUT INSTRUCTION]
choice:
1. {#1 INPUTT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL)
2. {#2 INPUTT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED FAITHFUL)
3. (#3 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL)
4. {#4 INPUT STATEMENT)
analysis: (REASON WHY conclusion: ( (HALLUCINATED OR FAITHFUL)
5. (#5 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: ( (HALJLCINAAHH OR FAITHFUL.)
output: (LIST OF CORRECT STATEMENTS)
##Example 2:
date: (INPUT DATE)
document: (INPUT DOCUMENT]
instruction: (INPUT INSTRUCTION]
choice:
1. {#1 INPUT'S STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL)
2. (42 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL)
3. (#3 INPUT STATEMENT)
analysis: [REASON WHY) conclusion: (HALLUCINATED FAITHFUL)
4. {#4 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLLUCINATTE OR FAITHFUL)
5. (#5 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL)
output: [LIST OF CORRECT STATEMENTS]
##Example 3:
date: (INPUT DATE)
document: (INPUT DOCUMENT)
instruction: (INPUT INSTRUCTION]
choice:
1. (#1 INPUT STATEMENT)
analysis: [REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL)
2. (#2 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: [HALLUCINATED OR FAITHFUL)
3. [#3 INPUT STATEMENT)
analysis: (REASON WHY conclusion: (HAIJLUCIIAAHH OR FAITHFUL.)
4. (#4 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL)
5. {#5 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL)
output: (LIST OF CORRECT STATEMENTS)
22

---

document: (INPUT DOCUMENT) Published as a conference paper at ICLR 2025
Table 15: CoT reasoning prompt used for hallucination detection tasks evaluated using exact match. Each example illustrates the evaluation process, where statements are analyzed for consistency with the given document and classified as hallucinated or faithful.
### CoT Reasoning Prompt Hallucination Detection (Exact Match) ##i
##Example I:
date: (INPUT DATE)
document: (INPUT DOCUMENT)
instruction: (INPUT INSTRUCTION]
choice:
1. {#1 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) (0V1)
2. {#2 INPUTT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) - 10V
3. (#3 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) + (OV
4. {#4 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: ( (HALLUCINATED OR FAITHFUL) + [0V
5. (#5 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HAIJJUCINATHD OR FAITHFUL.] + (0
output: ([BIARY VECTOR OF S ELEMENTS]]
##Example 2:
date: (INPUT DATE)
document: (INPUT DOCUMENT]
instruction: (INPUT INSTRUCTION]
choice:
1. {#1 INPUTT STATHMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) - (OV
2. (42 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) (0V1)
3. (#3 INPUT STATEMENT)
analysis: [REASON WHY) conclusion: (HALLUCINATED FAITHFUL) (0V 13
4. (#4 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALJLUCINATHE OR FAITHFUL.] + (0V1)
5. (#5 INPUT STATEMENT)
analysis: [REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) - 10V1)
output: [[BINARY VECTOR OF 5 ELEMENTS1]
##Example 3:
date: (INPUT DATE)
instruction: (INPUT INSTRUCTIÓN]
choice:
1. (#1 INPUT STATEMENT)
analysis: [REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) + 13
2. (#2 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: [HALLUCINATED OR FAITHFUL) + [0V1)
3. [#3 INPUT STATEMENT)
analysis: (REASON WHY conclusion: ( (HAIJLUCINAAHH OR FAITHFUL.) (0V1) 1]
4. (#4 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) (0V1)
5. {#5 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) + (0V1)
output: (IBINARY VECTOR OF 5 ELEMENTS]]
23

---

Published as a conference paper at ICLR 2025
Table 16: CoT reasoning prompt used for hallucination detection tasks evaluated using LLM-as-a- Judge. Each example illustrates the evaluation process, where statements are analyzed for consis- tency with the given document and classified as hallucinated faithful.
### CoT Reasoning Prompt Hallucination Detection (LLM-as-a-Judge) #i##
##Example I:
date: (INPUT DATE)
document: (INPUT DOCUMENT)
instruction: (INPUT INSTRUCTION]
choice:
1. {#1 INPUTT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) (OV 1]
2. {#2 INPUTT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) - 10V
3. (#3 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) + (OV
4. {#4 INPUT STATEMENT)
analysis: (REASON WHY conclusion: ( (HALLUCINATED OR FAITHFUL) + [0V1)
5. (#5 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALJLCINAAHH OR FAITHFUL.] + (0
output: (LIST OF CORRECT STATEMENTS)
##Example 2:
date: (INPUT DATE)
document: (INPUT DOCUMENT]
instruction: (INPUT INSTRUCTION]
choice:
1. {#1 INPUT' STATHMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) - (OV
2. (42 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) (0V1)
3. (#3 INPUT STATEMENT)
analysis: [REASON WHY) conclusion: (HALLUCINATED FAITHFUL) (0V 13
4. {#4 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLLUCINATHE OR FAITHFUL.] + (0V1)
5. (#5 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) - 10V1)
output: [LIST OF CORRECT STATEMENTS]
##Example 3:
date: (INPUT DATE)
document: (INPUT DOCUMENT)
instruction: (INPUT INSTRUCTIÓN]
choice:
1. (#1 INPUT STATEMENT)
analysis: [REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) + 13
2. (#2 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: [HALLUCINATED OR FAITHFUL) + [0V1)
3. [#3 INPUT STATEMENT)
analysis: (REASON WHY conclusion: (HAIJLUCINATHH OR FAITHFUL.) (0V1) 1]
4. (#4 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) (0V1)
5. {#5 INPUT STATEMENT)
analysis: (REASON WHY) conclusion: (HALLUCINATED OR FAITHFUL) + (0V1)
output: (LIST OF CORRECT STATEMENTS)
24

---

Published as a conference paper at ICLR 2025
K-HALU: Culture Domain
("id": "p30ts_ 235",
"document": "0 म 7H22
f 2 014
x| E क [ 3 E
OHE - ttihho Fn 말 #ol = X ]
E . 2244 1956
S -
7 T 2 \n'n
0
2 W
||
| A 0'
_ 21= 014 T HO E | मे OH
~ # ~ ž ø 19550001-334
C 7
0 AA 8 # = 2 H 말 @ 4
7|
4 O [| E + M 2 4 \nino| /
@ I/ll/ A Tnllol 42q,
712 14.6am, E|E 18.5cm
_ E717|
22 2 4 A AF7 E ΣMffffI 0
j 말 100 72 0 172 / 21711124-20055)/ 1982 전 T
- @ C. \nAn0/EHof| FNoo E
E OIQ / 7 IH @ CE ž = Ε d To
_ フ|ş _ コセ 0 22 = 2 WH OJ 0121 10 ö|
o C 2/ 2 ( Iooo 22 31 2 \n # 7 O =
- 2 E|Н (15011117) 5 Σ # U ( 92 '
O 16-194|7| 42 @ 1 غ 180 श 7/ 2/ 13 / su
193 lo 4 + ME.. E| 0 7 (15271117)) 2122 W
h 말ेE 01 7 2
~ 'MD 216, ! ' 3
"
1
+ 0|
2
lloo8| 전 1 1 galt 20/4.
전 ह 0
_ 2 Z $ 87 0 5 HEE -
///oo|| 212 ~
E2Z 0 - - 2 Σ / / 전 MOMiI 2 ii
ø 90000)-244810J1, | T O X|E 10 말マS ||HI| \| 7|4/ // Ø //7/ 100 E (21 (539)5)))), (90P))) E 4 \n/n0/@@t 1985 / '5 S,8508B' サ 169|~2391 Oll, 2008 전 "Dialogue(4( 594-621201 E 1970 - ||2| 1980 E 0 H ||2|, 1 X ||2| /| E/gEl \|2| х 0 ] n 7142 श E X To 7 HIMEE D + 720 0/2 ニ gF =| MEEL 5OJAA| 7 @ IIHI ~| T @
/ + A 0 こ
\n|n 2005 λęΙ 7///I| 'HE No. 050615' 2500 AII 2 ž
=0. 1 지| द EN L =1= - A|HHO|| natn # 'R''SS
1o 2 A ~ 5 7 JIIIE H RE A|2/ 말 _ E 전
ž & 1987 전 'IXe 7 サ| 3480000!
l 4 인사 - || ž ! @ 21= æ A/Z {
전 x 말 2E. A _ 19 म| 21 (1913199))1 X 7
E. X|
S/2 // Ea44 'VI 68', 1968 + $ ö 2 @0
$ = l2t10aa
24 III 68 #4', 1960 $ 7 _ Δ/2/ 2 말
0 Of ||
= _ '1 IIII 69#49' 50| Egd,. \n ~ 거
$ + A1 20y =2 1 사|ズ 2 전 _ T "date": "20200110",
oljo| 사μ @/0 774 Paa
"instruction": " E द 1 + 2 4|gal बह E t 7 전 + @C..".
eyS 1 Et, 전
"1". " 1 l/l X o|
2 3 ~ O
"0/ _ 55 S.8508B 2021 Aylokkoss 50%
-
"0 [ 전
@ ~he 39 OIL हIा E -5 ~
"E"> 0
7 7|
L 204|7| ž T 42/O/C."
"5": ~0 प 전
/
인 The 39 Ho! н g - 2
"label": [0, 4],
"choice_number": [1, 5]}
Figure 5: An example of K-HALU in Culture domain
25

---

Published as a conference paper at ICLR 2025
K-HALU: Economy Domain
{"id": "p60hs 82",
"document": "GS 2005 18 LG] Iii I LG3 22 1947
2 | 70 | a MP} 35 _ 7 1
2005 5 [00 2 _ 20/EE) nninGS _ w GS cH
= || GS 인 2 A GS ह * 69/HO| a
사/ 67H o 71 2 8.7%0f Z क 7 A = H 사/Oii| o 지Пン
_ 16.6% पCJ. GSJ 4 5.5% 30T I _ 2 8. 1 2 / 2013 3.3% / $ d I A1 Yo1 ~ o |n 7 E z| GSa 2 8 IA T GS2]E/210), Ha
ト T G 7| [ 2015 7 S 6 & GS
E ROE({77/H=0)) 5.9%F.. GS 871 7|0 E? 7 2 A ト GS2II 65.7%) || 21.4%% EIC 124 말
i Ld|l 57HA|2
= 7 2 7 O|E * GS7| XIE 50% = FA GS
Inin 2 1 ø l GS 7 0 OJ /|// - ~ 1
- 1M LI ^ / ब 2 8 7 E7| 4|
ininGS 7 D| * (Chevronn)) X| 50% = X
पn GS7 100% E GSOI | 50% E O Iyts || 5 @
t ?ル 6t J 2HE 2 XHU GS T - - O14. & GS A 0 + # 2012 2015 T A |0 3939/8!
2 E. O e E x//\\/2| @
2033 1381 644
/
E 7 ( | 말 OJ 7121 GSE EZ 12799/8 E 인
" \ E 2 8 & 5 こ 0| 42.8% 2 oj
2 4 |)
7 A GSJE
2 69.55% Xl
K (60.2%%) 22 C 22944
X 2 FOOE. Inin 22
$ 1 4 E O क | 4 X THS 2 A
J N| CHai E E d EΩIC). ( E GS, GSZNN, A
J _ ッ OF 04* & 7 R1.. ह GSZIS N á
/e - > GS2 XI DIE _ ~ II/ & 7| X # AI _ (20CC) In 8|4 - 22 $4 2712 571 5X 34.8%
I! 7 21.1% H|E 22 O 20|E| 0 27 / E| J C||i| _ ž! O ø 20 @IC. GS P #4 27//2| 47 49 E \n @|
I _ 200 IntMMliMI ø 2 S 4 IETE= ø || 말 ||
GS ø=2 [H 87 Z 21
N ह (H G
~ 0 //
_ ME.. nn 전 T лM //////
yn A ž
Φ7 2 EE a A
(|7] 24.. 2 8 c
R
5. A 7 ||UIPAIIプ GS 5 47H H Z//8/D l 인 G 5 47H I S "date": : "20170419", "instruction": " 1 4 o221 H/ ट 4 Q
2E. 전 0
a @["..",
02714 I 2g Taλ E
2
"1" 9 $ 인712 |7 2 सM =
"2": A 2 |I / @
"3": "G" 2
J Ti &C..",
"4" "4 2 W ~ G 0 7|0 こ E "
"5": "G":" 전 2 2 c | GS 전 말 @
Ct.,
"label": [4].
"rnoice_number"s [5]}
Figure 6: An example of K-HALU in Economy domain
26

---

Published as a conference paper at ICLR 2025
K-HALU: History Domain
("id": "p30ts 233",
"document": " 2 flolre a S 2 e मे = 2 0
- =
2 / = 말
1895 1 8 t 지l a lye o "date": "20200320",
ar E
_ |O 2004 X क
\ XO/C.""
"instruction": "øo|"! 2 ह प| + W 4|gol / E 2 of 7/ ズ + AC..", "1":" 1895 10 8 q| 전 @o|"""" "2": 2 1895 10 / WW Φ @o
e 1 E, ~
"3". " _ lye d|인2| ニミ X| d Io
OlC.."
"4": "CAN
8 인 Il 242th _ SHEE 말
ক| A
Ch.",
"5": "It 전 - IMe
Ct."
"label": [3, 2],
"choice_number" [4, 3}}
Figure 7: An example of K-HALU in History domain
27

---

Published as a conference paper at ICLR 2025
K-HALU: International Domain
( "p60hs 4", "document": 0 o 4i 1 - # SE 2 o||
o 말 a _ 인 - fl 7|
20 _ 5F
- R 7H
20 2 0|
LI 1 2 ब a
= / 27-30% _ / 2 A O _2|2 - T 2011 * x K AI SF
H R मा _ / EI @ H. \n'n
0 4IEE 1 _ E 말 0 OHHI| / 전 0 6 J
E F - O[C. 2
2
@ T x L X !! 2
100 7/11/ E 50 IT 1871 /
O - / A
/ OHL|S SITE T 2 X|
d22r -
o0
- E J BE 5 4 7
प 3 a Z lo إ
2 e - 7 1
a 2 J / E L RE Nnn SE X /| 15
b X C J- E J ||| UE X O ह 5 20
( 2 E -| 18 # \nin W RRILRE
- \"O E D 7 CHōl /| E /
2/ 2 2
- O EI E />|
T AI
C 2 1/ 2I, / 전
J 5 인인 D 2/72 2 в EE, E 전 5 2 인 A
OI| / 2O _ \nNOHO)) E EE 6 (I フ| 지 O
म - e // 407 F Lalll
인 O "date": "20130521", "instruction": = @ 4|g4| 42714 0421 7/ श + oI.."" "1": "OJull E 2| बा 말
2ge T2ハeΩ E, शे egg
2
"2": 2 i| 2 E 말 ~ 말인 -
2 ø R|म
- U
"3": " 0/ | L х フ 6|हl 20 2215
"4". 22 L 1
2 ब 말 न I|o @! 50 E E 거|
OI
"5": 2 क
ब 말े دy 사 ",
"label": [3]
somoice_number!::
[4]]
Figure 8: An example of K-HALU in International domain
28

---

Published as a conference paper at ICLR 2025
K-HALU: Medical Domain
("id": "p60hs_70 70",
"document": "E 7 a Llalo T L8 L 5 OF
o G = g 4 6 u 86 075 T
m 2 - # ini 2006-20154 22 = 7 4 - 4 येय = " 2 - HA
)), - 2 65 0 - 135 _ dIo2 - = \n'n zn HIll a 1000 16.157|a #I II 1. = N H
@ ö/2i 4t 말ास 2 1000 6.4 1.478|| E | 4E 282 1000
13 f 1.25HH = X E| F 3 O _ 8/पi 4 말ेे 전 @ 1.05 2 1.138H 7 4E 57/ O Q - Z 4 말ेय 전 O 1.644H, 282 1. 1.69 EI Ø E O 말 \n E 0 G|O|| 2 T E L 2 X ह 4 말े 2 20% T
/ NE S 21 17%2 7%, 25% Z G LIEWEE 2 O | EU 말 말 ЯА {| ž АEU TIlL= G Æ ~ 전 & ж|l | 48 말 0 2.74HII, 2 1.32 EE 1.5HII, & 2.9H G dg L - _ 2 2 UFEE LU Oo : zo
#| ह 2 - قق н O 2/2/ 2.4 CI g LIELH do2 = !n 075 ह e ZHORS 7/ M+E OL @g
_ A 전 TH A 7HAN 5 LU E ह 인인 + & """."" "date": "20181206",
"instruction":""400 말 4 0 말 मय g nateº. E शे g o 7/ क + I "1": S0, _ LU I
"2": 2 48 말े D 0I e O|
"3": " - a JN "
"4": ~ 7 +U E ~y + -
"5": "Of - R +
E a O E
"label": [1].
[2]]
Figure 9; An example of K-HALU in Medical domain
29

---

Published as a conference paper at ICLR 2025
K-HALU: Society Domain
("id": "p30hs 230", "document": " (AI 2 7| alee dルズハニ H @UD @ Ct. in 2 _ Eg 'Al gे N a 15 전 34 ~ I 9 C| Al g _ ag 4 _ 2023 X ma 1
15%o e =
=[ #4, G|O/E पे 2g (7) & Al 말े
= d द O = dE 2 1 24R O = E 2
c m 전 ALm 0 7 x| λ E E Inino/ Al g m श aaTE E Th M
O - 50/C, b/ of # 50
- 417 ب / 사ZSH dIS C..- A / I x| & - M 0d 2
AF 1 म| 06On SE C Vnln = हे 2 a 4 onL 2915000 д >| S λ @ श 1 A λ 2021 2023 OHE 5000 $7| X ofl Wo|c. N V εF, T J E 88) M EI 2 7
I 4 N O aa T [. OJ ΣI EO ! &8
| C KaA X S 2 HIJe - 2 O A @ 1 *
x ( ह ) ह AI 말े N TEONE T >/ \ninēł
@ E ズ Té (SW)) = E NOEL SHEE - E 인 _ 7 प " "date": "20200309",
"instruction": ": "Z _ 4 4274 02 7H श + O...",
말य E
श2 2 E, n2ハ egg
"1": " AI y 2 - 8/887>
2 K
"2": "H Al 인 2 0 15% E O
n5 हका = 말ेे 인
"3": "S E @ शा 10 d π 2021
2023
OHL 1 $ ズ "F
"4": T क 2 EgGl= 'AI 말े R 인 15 전
340 ~
9 "
"5": " - Al 인 2a1Ra OØ 1 4 0 2
OI
"label": [3. 2]. "choice_nnmber [4, 3]}
Figure 10: An example of K-HALU in Society domain
30

---

Published as a conference paper at ICLR 2025
K-HALU: Technology Domain
("id": "p60hs 46", "document": (AI) d 인 3| λ 2|| 4 Y @ ~ ž AI 인 ' 2| 1 - = 0 Λ Al 전 - a EΣ A !n E A 71213, X \| (IoT), ~ _ _ ह F| A t MZ क _ 7 MZ =| 1980 HH x E @ x|E|인 1990 Z _ a / Z 5 _E, E 20
0 _ A X T 2
\n 11 11 = @ ! T E|| E 1 A 135, X|ø 2 53 72 H PIDIQIA! 7/243, EeI Al Of =| AI O 24A/2 Π @ AI O X 0 Ool D C/ A
- O 1 E =
2 श 3R A @ O| E AI 7/722 O| | प| 4
HΩ C/ Ω म !oo [ En0 전 ~ 말 A मप 2 E
7 @ HIGHL! AI ~ 7. || T 2 7240 || 전
E H
I EE = 22 7 2 말2S ~ 2
NI 2| E 전 5 I
2 T E ΔE| 2 (Keyess) H I # ALl zμ Slll E/ EE/o _ E BR 8/1 2174, 말 &o 인 ||| l || H|CH AI 말 E EUMORM, @ @ 4
C| 5 S + Q1 7o2 C Inin 2|EE||| OHE 29 0 oF
전 ~ H| T 2 2771 / B| 722 F _ AI の
전 말æ य |/ в 2 전 T "
"date": "20201013",
"instruction": = _ 4|ghl o2 7 ह + & "1": SEISEE Al a _
말 말 E n2t22 E, 말े _
"2": " =| CH A
MZ # _ جi O
"3": EEI 80 0O ~
WIHES 0 E| $c..". h H| AI ~ -
"4": EEO E| उ /|
"5": "I S| ~ 1 13 79% / O
"label": [1].
"choice_numble [2]}
Figure 11: An example of K-HALU in Technology domain
31


Reviews:
[{"review_id": "wzVwXLEpPz", "reviewer": "Reviewer_cEAG", "summary": "This work introduced a Korean benchmark namely K-HALU designed to evaluate LLM's hallucination detection in Korean. Specifically, K-HALU consists of more than 2,000 challenging test samples from Korean news, magazines and books, covering seven domains. The evaluated LLMs involve open-source models: Llama 2, Llama3, Mistral and Korean models such as KULLM3 and ExaOne, and API models: GPT-3.5 and GPT-4 series. Results show that the open-source LLMs exhibit low accuracy less than 35% and 15% on harder tasks, compared with the API LLMs show better accuracy by 27%.", "questions": "I hope the authors can clarify my concerns about the above weakness. But if I missed some important thing, please correct me in the rebuttal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces a Korean benchmark to evaluate LLM's ability for hallucination detection. The benchmark consists of more than 2,000 examples and is labeled by both human annotators and top-performing models.\n\n2. The writing is clear and easy to follow.\n\n3. The analyses are extensive for different LLMs and different scenarios such as domain, instruction type and few-shot.", "weaknesses": "1. The comparisons and discussion with the English are missing. Given the fact that there exist benchmarks for hallucination detection in English, the difficulty of Korean hallucination detection by LLMs and the corresponding novelty is unclear to me. This will hurt the contribution of this work. \n\n2. The evaluation measure is a bit simple to some extent. The measurement used in the benchmark is selecting the hallucinated statements from the given statements. However, it is more useful for LLMs to directly give judgments or reasons on whether an input has a hallucination. The problem setting in such two cases is different in the existence of reference selections. \n\n3. The metrics on how different human annotators consensus with each other is unclear. Besides, the linguistic background of human experts can be essential to the correctness of data labeling. \n\n4. (Minor) I notice from Table 7 that The CoT approach can hardly improve the accuracy, it is peculiar from the literature of CoT. More discussion on this can be helpful. \n\n5. (Minor) The question in the benchmark can have multiple selections, thus other measurements such as precision, recall or F1 can be more comprehensive to evaluate hallucination detection than accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2c8zu6ZeEQ", "reviewer": "Reviewer_Fjh4", "summary": "This paper presents a benchmark for evaluating the performance of LLMs on determining whether a statement (short sentence) written in Korean is hallucination or not with respect to a given document also written in Korean and its publication date. Concretely, there is a set of statements for each document and the LLM is tasked to either pinpoint all hallucination ones or all correct ones (i.e., two types of multiple-choice questions). The motivation to create this benchmark is due to the lack of evaluation resources in Korean. The benchmark creation pipeline is described together with its statistics. The performance of several open- and closed-source, multi-lingual and Korean-tuned, LLMs is evaluated.\n\nFirst, the authors should revise the terminology (i.e., hallucination, factual, faithful) which seems confusing as used in the paper. As the benchmark and task is stated, it seems that the actual task is to distinguish between faithful (i.e., supported by the input document) or unfaithful (not supported by the input document) statements. A statement can be not supported by the given document but still be factual (see definition in Maynez et al. 2020).\n\nSecond, it is not clear why the format of the benchmark is multiple-choice and why the strict evaluation on sets of statements (select n hallucination/non-hallucination statements from given N). It is not clear why this task setup, it seems adding extra complexity that is not needed to see whether an LLM can distinguish whether a statement is faithful (or not). Cannot it just evaluate the LLM on each individual statements?\n \nAs for the LLM execution of the task, why the selection between hallucination or not is based on probabilities? Why not prompting the model to generate/select the options (i.e., to generate \"1, 2\")? The authors should explain why they chose this method and may be also add results with the prompt approach. Previous evaluation methods based on prompt ask the model to choose among plausible responses. Also, how was this probability approach implemented for closed-source LLMs (i.e., GPT variants)?\n\nMore details about the Chain-of-Thought implementation should be provided. The prompts and partition into reasoning sub-steps?\n\nThe authors should avoid using the term \"uncertain\" throughout the paper. It is confusing and under-specified, do the authors mean that the statement is not supported by the document? Then should be said 'unsupported'.\n\nRelated work should discuss other work on creating evaluation resources for less represented languages. eg., [1].\n[1] https://arxiv.org/abs/2403.20266\n\nSection qualitative analysis, Appendix A and Figure 4 should go into the main paper.\n\nMinor comments:\n\n- the source dataset should be better described (Knowledge Graph-to-Text dataset from AI-Hub) what is this graph?\n- Lens Observation section. At this point in the paper, it is not clear what the used frameworks are nor why suddenly we are looking at these frameworks.\n- Line 360, \"handling hallucinations\" should be called \"detecting hallucinations\" as the task is to detect whether the sentence is hallucination or not rather than doing something to mitigate hallucinations.", "questions": "- why the format of the benchmark?\n- why only the implementation using probabilities?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The collected evaluation suit from Korean sources is valuable for the evaluation of Korean language models.\n- Experiments show that LLMs struggle to reason about content in Korean (though see comments above about the experimental setup).", "weaknesses": "Execution of the work: task formulation (the formulation of the task on multiple-choice seems unjustified and unnecessary more complex) and implementation with LLMs only using probabilities.\n\nOverall, the contribution seems rather thin.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h9TirWkBUZ", "reviewer": "Reviewer_zS85", "summary": "This paper introduces K-HALU, a multiple-answer Korean hallucination benchmark for evaluating LLMs on their ability to detect factual inaccuracies or hallucinations within Korean-specific contexts. Unlike previous Korean benchmarks, which often rely on translations of English datasets or offer limited dataset access, K-HALU provides a culturally and linguistically relevant resource. It consists of 2,170 test samples. By requiring models to select all possible correct answers in a multiple-choice format, K-HALU presents a stricter evaluation standard than traditional benchmarks, especially for open-source models, which often struggle with hallucination detection.", "questions": "1. For common benchmarks like MMLU, the majority of samples can be sourced online to verify the correctness of extractions. Can K-HALU samples similarly be sourced online for users to verify correctness?\n\n2. In evaluating multiple-choice answers (as mentioned in the weaknesses section), is exact match used as the criterion for correctness, or are alternative evaluation methods applied?\n\n3. What would the model accuracy be if strict exact match criteria were applied, where an incorrect answer is not picked and the entire question being marked as incorrect? A breakdown of accuracy under these conditions for each model would provide valuable insights into model performance.\n\n4. What would the feasibility of extending K-HALU to other non-English or low-resource languages be? Is the framework adaptable beyond Korean, or are there limitations specific to the Korean language context?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The main strength of this paper lies in the originality and cultural relevance of K-HALU, as it fills a significant gap in the Korean NLP community by using carefully selected and annotated content rather than translations from existing benchmarks. K-HALU’s focus on Korean linguistic and socio-cultural contexts ensures that factuality assessments and hallucination detection are appropriately designed, making it particularly valuable for future NLP research in non-English settings. The multiple-answer format provides a more rigorous evaluation by requiring the selection of all correct answers, offering a strict measure of model robustness and enhancing the benchmark's reliability in assessing hallucination vulnerabilities in real-world applications. Additionally, figures in the paper reveal that open-source models still struggle with hallucinations in Korean, highlighting opportunities for further development and application in multilingual hallucination research.", "weaknesses": "A weakness of the paper is the lack of clarity regarding the human-annotated statements. The paper does not provide details about the annotators' backgrounds, such as their expertise in relevant domains (e.g., whether they hold advanced degrees or come from diverse educational backgrounds). This information is critical for understanding the quality and reliability of the annotations, especially in fields that require specialised knowledge. Secondly, the evaluation section appears unclear and lacks specificity. It is unclear whether exact match is used as the primary metric for assessing correctness in multiple-answer questions or if external models are used in evaluation.  A more detailed breakdown of the evaluation methods would improve clarity and provide stronger support for the paper’s conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)