Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScore
Authors: Sheikh Shafayat
Eunsu Kim
Juhyun Oh
Alice Oh
Venue: COLM (2024)
Keywords: multilinguality, factuality, knowledge representation
TL;DR: We fact check multilingual generations of LLMs

Abstract: Evaluating the factuality of long-form large language model (LLM)-generated text is an important challenge. Recently there has been a surge of interest in factuality evaluation for English, but little is known about the factuality evaluation of multilingual LLMs, specially when it comes to long-form generation.
This paper systematically evaluates multilingual LLMs' factual accuracy across languages and geographic regions.
We introduce a simple pipeline for multilingual factuality evaluation, by applying FActScore \citep{min2023factscore} for diverse languages. In addition to evaluating multilingual factual generation, we evaluate the factual accuracy of long-form text generation in topics that reflect regional diversity. We also examine the feasibility of running the FActScore pipeline using non-English Wikipedia and provide comprehensive guidelines on multilingual factual evaluation for regionally diverse topics.

Published as a conference paper at COLM 2024
Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScore
Sheikh Shafayat, Eunsu Kim*, Juhyun Oh", Alice Oh
School of Computing
KAIST
Daejeon, Korea
(sheikh. shafayat, ken.kkee,,,,,11uuuuuuuuuuuuuu juanjuunnjkkksstt..m.. ac. ar,a edu
Abstract
Evaluating the factuality of long-form large language model (LLM)- generated text an important challenge. Recently there has been a surge of interest in factuality evaluation for English, but little is known about the factuality evaluation of multilingual LLMs, specially when comes to long- form generation. We introduce simple pipeline for multilingual factuality evaluation, by applying FActScore (Min et al. 2023) for diverse languages. In addition to evaluating multilingual factual generation, factual accuracy of long-form text generation in topics that reflect regional diversity. We also examine the feasibility of running the FActScore pipeline using non-English Wikipedia and provide comprehensive guidelines on multilingual factual evaluation for regionally diverse topics.
evaluate the
Introduction
Large Language Models (LLMs) are susceptible to factuality hallucination, a phenomenon in which the generated text contradicts established world knowledge (Huang et al. 2023
Zhang et al. 2023). Despite extensive research focusing on hallucination and factuality of
T.LMs in form generation, the previous works have predominantly studied English
(Huang et al. 2023 Min et al. 2023 Mishra et al. 2024 Wei et al. 2024 Wang et al. 2023).
Consequently, there exists wide gap in our comprehension
the factual accuracy of LLMs when producing content in non-English languages. As highlighted by Kang et al.
(2024), current metrics for detecting hallucination are inadequate in multilingual settings. The extent to which LLMs exhibit factual hallucination remains unclear across different languages. Similar to other capabilities, there may be discernible decline in performance when L.Mss are tasked with non- Mon-English contexts (Ahuja et al. 2023; Bang et al. 2023).
In this paper, we address this gap by systematically evaluating the factual accuracy of multi- lingual LLMs across different languages and geographic regions. We explore the following research questions: R1: How can we effectively evaluate the factuality of multilingual LLMs in free-form text generation? R2: How do multilingual models' free-form answers compare in factual accuracy across languages and geographically contextualized questions?
To address these questions, introduce Multi-FAct, simple pipeline tailored for evalu- ating factuality in a multilingual context. Leveraging open-source models, we adapt the FActScore (Min et al. 2023) for multiple languages and validate it using human annotations from Min et al. (2023). Furthermore, we explore the use of non-English Wikipedia content to augment factuality evaluation, finding that while English Wikipedia remains a reliable source, concatenating articles from multiple non-English languages can sometimes provide estimates close to English Wikipedia.
We conduct an evaluation of the factual accuracy of recent proprietary multilingual LLMs in nine languages using biography generation task. We ensure geographical diversity by curating subjects from various regions. Our analysis reveals two significant findings. First,
"qqull contribution.

---

Published as a conference paper at COLM 2024
1. Obtaining multilingual generation
3. Measuring Factuality
Write a bio of Shinzo Abe
Breaking down into Atomic Facts
Shinzo Abe was lived during 09/21, 1954
32 7
2154
_
07/08, 2022.
E ,
0 1
He was a Japanese politician.
H B ^
He served as Prime Minister of Japan twice.
fa Shinzo Abe,
He holds the record for the longest-serving
Japan
Prime Minister Japanese history.
2. Translating into English Translate w/ GPT3.5
Fact- Fact-Check
Shinzo Abe (09/21, 1954 07/08, 2022) was a Japanese politician who served as Prime Minister of Japan twice and holds the record...
Union Wiki Long Wiki Multilinggul reference
Multi-FActScore 0.63
Figure 1: Our Multi-FAct Pipieline. The pipeline structured into three main stages: 1) Obtaining multilingual generations, Translating these facts into English using GPT-3.5, and 3) Measuring factuality, by breaking them down into smaller atomic facts (Min et al. and then verifying them by asking LLMs with context provided.
2023
English consistently maintains an advantage in both factual accuracy and the quantity of generated facts compared to other languages. Second, content produced by multilingual language models exhibits better factual accuracy about North America and Europe across the languages.
Our contributions are follows:
We propose novel pipeline Multi-FAct, that applies FActScore in multilingual setting, and demonstrates the feasibility of conducting factuality evaluation of long-form generation in the multilingual biography task.
We explore the potential and limitations of using non-English resources, particularly non-English Wikipedia articles, for factuality evaluation of long-form generation, showing that reasonably large non-English Wikipedia articles can also serve viable source.
We introduce Multi-FAct as a tool to investigate cultural and geographic biases in LLMs, offering insights into how these biases manifest in LLM generated content.
Related Work
LLM Factuality Evaluation The assessment of factual accuracy natural language texts predates the advent of LLMs (Guo et al. 2022). Our Multi-FAct pipeline follows the previous fact evaluation research, which involves validating claims based on external resource such
as Wikipedia article (Thorne et al., 2018 Krishna et al., 2022 Zhong et al. 2019) or Google search (Chern et al. 2023 |023| Wei et al. 2024). Our work is inspired by and based on Min et al.) (2023) which proposes FActScore to measure the factuality of long-form generated output of LLMS by breaking it down into atomic facts. Another line work focuses on evaluating factual accuracy solely based on model output internal states, eliminating the need for external knowled ledge bases like Wikipedia (Manakul et al. 2023 Azaria & Mitchell 2023 Dhuliawala et al. 2023). While such methods ofter resource efficiency, they often sacrifice accuracy compared to approaches leveraging external resources for factuality ev evaluation.
Multilingual Factuality Evaluation To evaluate factual accuracy multilingual genera- tion, Gupta & Srikumar (2021) presents the X-Fact dataset. Similarly, Thorne al. (2018)
2

---

Published as a conference paper at COLM 2024
presents frameworks for multilingual fact-extraction and verification evaluation. Addition- ally, |Aharoni et al. (2022) introduces mFACE, dataset aimed at evaluating multilingual factual consistency. While these papers advance the understanding of multilingual factu- ality, they focus on the assessment of factual accuracy in existing articles or benchmarks rather than examining the factual accuracy in the long-form generation of LLMss. Recently, Kang et al. (0024) explores the efficacy of existing metrics designed to detect hallucinations of LLL-ggenerations in English in multilingual setting, showing that the English-based metrics fall short in multilingual contexts. Our study extends the endeavor of developing factuality evaluation metric that can be applied in multiple languages by introducing Multi-Fact pipeline.
Geo-culture biases of LLMs
Research shows LLMs exhibit cultural and linguistic bias
(Hovy & Yang 2021 Cao et al., 2023 Hershcovich et al. 2022 Huang & Yang 2023: |
al.
2023 Naous et al. 2023). A recent paper conducts true-talse evaluation task that
shows GPT models are more accurate for facts about the Global North compared to the Global South (Mizza et al. 2024), consistent with our findings. We conduct much more
comprehensive study with long-form generation of geographically diverse information in multiple languages.
3 Multi-FAct Pipeline
We introduce Multi-FAct, a novel pipeline for automatically measuring factuality of multilingual LLM in multilingual setting. Multi-FAct evaluates the biographies of multi- regional topics in multiple languages. We choose the biography generation task because biographies suitable for being decomposed into verifiable and independent "atomic" facts (Min et al. 2023
The Multi-FAct pipeline is structured into three main steps: 1) Obtaining multilingual generations ($ 3.1 2) Translating these facts into English using GPT-3355 ($ 3.2), and 3) Measuring factuality of the translated facts 3.3).
3.1 Generating Facts
Model Most of the results in this paper GPT4 (gpt-4-1106-p and GPT3.5 sapt-3.5-turbo000 models because at the time of writing this paper, those were the only two strong multilingual modele] for LLM multilingual biography generation. All experiments are conducted between January and March 2024 and the model temperatures are set to their defaults for respective models.
Language We choose a total of nine languages: English, German, French, Spanish, Arabic, Swahili, Chinese, Korean, and Bengali. We carefully choose languages to represent the level of existing resources high, medium, and low, and to represent diverse regions-Africt Europe, Asia, and America
Prompt We use the following prompt: Write a biography of (name). We translate the prompt with GPT-4 into the eight non-English langauges, then ask the native speakers to verify the translation. We also transliterate each person's 's name into corresponding languages either using Wikipedia crosslinking or GPT-4, of which we take subset and the native speakers verify that almost all the transliterations are correct.
We also provide FActScore estimation for some newer models in Appendix Fxxept for GPT4o and GPT4o-mini in the Appendix.
3 Afrria: Arabic, Swahili, and French, Europe: Spanish, French, German; Asia: Korean, Bengali, Chinese; America: Spanish and French. English is widely spoken all over the world, and kept as the baseline.

---

Published as a conference paper at COLM 2024
3.2 Translating Generated Facts
To com compare the biographies written in the eight non-English languages, translate the generated content from the original languages to English using gpt-3. 5it--bbbo-tto0000 0125. We do not use commercially available machine translation systems, such Google Translate, based on the preliminary result that they do not maintain the consistent gender of the person throughout the tex 4
We translate the generation and verify facts in English, rather than doing fact verification in corresponding languages with respective Wikipedia articles for multiple reasons. First, Wikipedia differs in size and scope for each language (Wikipedia contributors 2024), which makes it difficult to compare cross-lingual factuality if the knowledge base is different. Additionally, key components of the original FActScore pipeline, such as RAG and NPM, optimized for English and not available in other languages. Our quantitative analysis of LLM-based translation's impact on FActScore evaluation reveals that GPT-3.5's translation minimally influences the FActScore of texts (See $ 3.4
3.3 Measuring Factuality
To evaluate the accuracy of the generated model's response M, we use the FActScore metric. This involves breaking down the translated generation into atomic facts short sentences, each conveying a single piece of information, as defined in Min et al. (2023). We denote the set of atomic facts as A, and we measure the accuracy of these atomic facts with the corresponding English Wikipedia article serving the reference knowledge source C.
# of Correct Facts(M) = [a is supported by C)
(1)
FActScore (M) =
Σ of Correct Facts(M)
A
We replace all proprietary models of the original FActScore pipeline with open-source mod- els, ensuring cost-effectiveness while maintaining the quality of system. We decompose LLM responses M into atomic facts using Mistral-7B Jiang et al. (2023). The verification step also uses Mistral-7B along with RAG and NPM for the best pertormance. We set the NPM threshold at 0.3.
3.4
Reliability of Multi-FAct
3.4.1 Replication of original FActScore
We conduct comparative analysis using our implementation, which includes Mistral 7B model, instead of InstructGPT for atomic break-down, against the outcomes presented in Min et al. (2023), using subset of human-annotated factual data generated by ChatGPT (See lable 1). A comparison with other baselines and different parts of the Multi-FAct pipeline for different languages is provided in appendix
Metric
Human FS
FS (ours)
Error
Value 0.626 + 0.238
0.640 + 0.127 -0.014 + 0.128
Table 1: Summary of Replication. Error refers to the difference between human FActScore and FActScore determined by our method. Standard deviation is reported alongside each metric. Note that the Human FActScore value is slightly different from what is provided in Min al. (2023 as this value averaged over our subset (44 samples).
However, do provide an analysis of what happens when Google Translation is used instead of GPT3.5 in Appendix C
4

---

Published as a conference paper at COLM 2024
3.4.2 Effect of GPT-3.5 Translation in Multi-FAct
As our pipeline crucially relies on the GPT-3.5 translation (Step in figure 1 is imperative to check whether the translation step affects the automated estimation of FActScore in the multilingual setting. We evaluate the effect of GPT-3.5 translation based on the premise that the factual content of a text should remain consistent across translations into different languages. In other words, if the factuality of an English text is established, its translated version in another language should exhibit the same level of factuality.
Language
Correlation
Mean error
Standard Deviation
English
0.84
-0.014
0.13
German
0.83
-0.007
0.13
French
0.80
-0.004
0.14
Spanish
0.81
-0.014
0.14
Swahili
0.81
-0.015
0.13
Arabic
0.82
0.021
0.14
Chinese
0.81
0.040
0.14
Korean
0.80
0.009
0.14
Bengali
0.84
0.027
0.13
Table 2: FActScore evaluation across languages. English refers to FActScore evaluation on original human annotation data provided in English; other languages refer to Multi- FAct evaluation on GPT-4 translation of the human annotation. Correlation refers the correlation of individual FActScore with human-annotated FActScore provided in Min et al. (2023
To simulate and test the reliability of Multi-FAct with respect to translation, we first take the
numan-annotatee examples provided in the original work (Min et al. 2023) and translate them into the eight non-English languages using GPT-45 Then, we we apply our multilingual FActScore pipeline (Figure 1) on the translated texts Table indicates that the execution of Multi-FAct non-English data does not significantly degrade the estimation of FActScore, and the correlation between FActScore in different languages and human evaluation remains comparable. This outcome implies that adding translation step to our pipeline does not significantly influence the estimation of FActScore.
2
4 Reference Sources in Multilingual Factuality Evaluation
We evaluate the factual accuracy of LLM-generated biographies against English Wikipedia for its reliability and comprehensive coverage of information about individuals, following prior works (Petroni et al. 2020 Chen et al. 2017 Min et al. 2023). However, we need to analyze the reliability of non-English Wikipedia articles as references. While the Multi-FAct pipeline is effective for topics with established English Wikipedia articles, its applicability may be limited because many regional topics (e.g., local politicians) lack English Wikipedia articles. If we cannot extend our Multi-FAct pipeline to these topics, it would be major obstacle to ensuring LLM factuality with geographical diversity. However, as figure P illustrates, languages included in our study exhibit a wide range of variability in their representation within Wikipedia.
In this section, we explore the following question: How should measure multilingual factuality, when we cannot find a sufficient golden source of reference in English? First, we examine whether simple translation of Wikipedia articles from languages other than English would sustain the quality ($ 4.1. Then, we reveal that the "length of articles" is one factor contributing to performance degradation when non-English Wikipedia is used as fact verification ground truth corpus, and we propose a new method of concatenating articles from different languages to enhance the comprehensiveness of the reference 4.2
5e use GPT-4 as a previous study (Jiao et al. 2023 demonstrated GPT-4 to be a good translator. We did not GPT-4 for back-translation in Muln-FAcc pipeline reduce APT costs.
5

---

Published as a conference paper at COLM 2024
Scniid 502
Marber ar cancar scalo)
Mumber Aetho lia
13
(c) Number of original FActScore annotated topics that have Wikipedia articles in corresponding languages.
(a) Number of total Wikipedia articles available per language (in millions).
(b) Number of active Wikipedia users per lan- guage (note the log scale).
Figure 2: The Wikipedia size distribution for languages in this study, (c) shows the number of thuman-annotteer examples that also have representation the corresponding Wikipedia. Note that Wikipedia size differs widely across languages, which usually means non non-English Wikipedia articles are as comprehensive English Wikipedia articles for fact verifica- tion.
Averagg FActScore estimaation enror
Average Fictscore ezimation cmor
0.40
0200
Language
Swehil
0.35
0.30
6-66
6.23
Chinese
Korean
0.20
n-57
99- riws
Spanish
0.15
n-37
0.10
French
0.10
English
0.05
#c157
0.00
Snahll BengellChnese Korenn Arak GommihGemmnn French English
2007
1,50
Wikieeiaa Corous
Average Length English
(a) FActScore estimation error when using differ- ent Wikipedia corpus as ground truth.
(b) FActScore estimation error scales linearly with the length of the Wikipedia article.
Figure Effect of using non-English Wikipedia as knowledge corpus for FActScore estima- tion. The x-axis in subfigure (b represents the word count (in English) after translation. As the length of the non-Eng -onnEnglishh Wikipedia article increases, its utility for evaluating factuality increases.
4.1 Using Translated Wikipedia Articles as Reference
We first test whether we can simply translate the Wikipedia articles written in eight non- English languages into English then replace the English Wikipedia articles within Multi-FAct pipeline with the translations. Using GPT-3.5, we translate the respective Wikipedia articles into English and conduct factual evaluation in English. We chose 157 individuals topics for whom human-annotated data exists in Min et al. (2023), which served as the ground truth. We then compare this ground truth with the experimental results.
Figure illuutratees Siguuraaill the errors for each language's Wikipedia against ground truth, and we include English Wikipedia corpus the baseline. The results show a near-zero error rate for English, and lower error rates in French, Spanish, and German. We observe relatively higher error rates for Bengali, Korean, Chinese, and Arabic. Note that the size and coverage of Wikipedia varies across languages, thus the number of samples used in the experiments also varies by language, as indicated by in the figures.
4.2 Effect of Reference Length
We hypothesize that non- -on-lsh Wikipedias result higher errors compared to English Wikipedia due to the lower coverage of topics, evidenced by the shorter length of non-

---

Published as a conference paper at COLM 2024
0.40
Indivicual Wiki
Lagogg
0.35
0.10
FesScot Dffoortte fran
Reench
0.30
Sponiich
Chinose
0.14
Arabic
0.20
0.12
Bengall
0.10
0.05
0.08
0.00
Al-En
Adian
Euro
1000
1500
2300
2500
3000
2500
Average Wiki Length (in English words, after translating)
Ensembling multiple non-English
(a) Multi-FAct estimation for persons with 'large' Wikipedia articles in corresponding language.
Wikipedias work better than single language
Wikipedia.
Figure Effect of corpus size on FActScore Estimation. Figure shows that comprehensive non-English sources might be used for factuality evaluation after translating into English.
English articles. Figure 3b shows that there linear relationship between FActScore estimation error and the word count of Wikipedia articles, indicating that longer Wikipedia articles tend to serve better as references for factuality evaluation.
To further test the linear relationship observed in Figure 3b] we select 100 individuals from each of our studied languages who have both English Wikipedia articles and substantially long Wikipedia articles in their regional language We generate biographies for these 100 individuals in English and evaluate these biographies against both the English Wikipedia and the Wikipedia in the corresponding regional language. Since FActScore estimation for English is nearly perfect, we use it benchmark to estimate the error when using non-English articles as references. Figure 4a illlustrates that the linear relationship between the length of Wikipedia articles and FActScore holds for longer articles as well.
Ensembling Wikipedia Building upon the insights about the reference length discussed in $4..1 this section investigates whether combining multiple mon-English Wikipedias, referred to as "ensembling Wikipedias," can enhance the accuracy of Multi-FAct measurements. In our experiments, we explore three cases: the union of wikis from Asian languages (Asian), the union of wikis from European languages (Euro), and the union of all wikis except English (All-Eng). As depicted in the Fig 3a among the two wikis with errors close to zero-namely, "Euro" and "All-Eng" cases it is speculated that this could be attributed to their comprehensiveness
or the high content overlap with the English Wikipedia. In the case of "Asian," an error rate of 0.2 is observed, representing decrease compared to using only one wiki of Asian language, and even approaching the performance observed for European languages (Spanish, French, and German).
Our findings suggest that even for the same topic, the coverage of Wikipedia articles may vary across languages, thereby demonstrating that an "ensemble Wikipedia" comprising multiple articles can mitigate the performance gap between non-English Wikipedias and the English Wikipedia. However, the challenge of Wikipedia containing only single, brief article remains unaddressed.
5 Factual Accuracy of Multilingual models
We use Multi-FAct to investigate the factual accuracy of multilingual LLMs in biography generation. We select "National Leaders of Each Country" as the topic for factuality mea- surement, as it is fairly common across various cultures and languages to have well-known and well-described national leaders such as presidents and prime ministers. Specifically, we choose to focus on the president or head of state in the year 2015. This year is randomly
7he detailed sampling method provided in appendix C.2
7

---

Published as a conference paper at COLM 2024
selected from those prior to the emergence of LLMs and also to make sure that at least some amount of data about the leader is online, which might not be true for very recent leaders. The lists of countries and their corresponding leaders used in our experiments are detailed in Table of Appendix.
5.1 Does factuality differ across languages?
GPT-3.5 TectScors
GPT-4 NactSorre
CFTS.5 hellddnieed
connect wits
000
Dome
Figure 5: FActScore of GPT-3.5 and GPT-4 for each language. Note that GPT-4 makes strong jump from GPT-3.5, especially in low-resource languages.
Figure 6: Number of correct and halluci- nated facts by GPT-3.5 and GPT-4 for each language. The number on top of each bar is the FAAtScorre.
Figure illustrates the factuality across various languages using FActScore as a metric, uncovering significant variations among languages. English, Spanish, French, German, and Swahili exhibit notably higher FActScore for both GPT-3.55 and GPT-4.
@ossure@ccopann factuality across languages through the average number of correct and hallucinated facts, highlighting a significant gap the quantity of facts generated between English and other languages. Notably, even with similar FActScore as shown in Figure English surpasses other languages in generating a higher number of correct facts, thus delivering more useful responses.
These observations emphasize the influence of output length differences on the number of correct and hallucinated facts across languages, despite similar FActScore values. English and other high-resource languages (e.g,, Chinese, Spanish) typically produce more facts due to longer outputs. In contrast, leworece languages like Bengali yield fewer facts because of shorter responses, even when their FActScore are comparable to English (0.58 for Bengali, 0.61 for English). This discrepancy further widens the factuality gap between low-resource and high-resource languages. Hence, the denormalization of output length in multilingual contexts suggests that evaluating factuality requires considering both FActScore and the number of correct facts for thorough assessment.
5.2 Does the Factuality Geographically Differ Across Languages?
In this section, we explore whether the factuality of models varies with the geographic distribution of the language. Since GPT-4 performs better in low resource language (Figure 5), from here on, we conduct our analysis using only the outputs from GPT-4.
Table presents the overall FActScore by language and continent. Interestingly, languages with higher mean values tend to have lower standard deviations, indicating that these languages maintain a relatively uniform level of factual accuracy regardless of the geo- graphical area. Western languages such as English, Spanish, and German notably score high across continents, while Chinese, although high-resource language, exhibits low factual precision. Additionally, among the nine languages evaluated, six demonstrate their highest performance in content related to the American continent, highlighting a prevalent American-centric bias in GPT's knowledge across most languages.
Next, we analyze the geographic distribution of the most factually accurate information across different languages. For each language, we identify the continents associated with
8

---

Published as a conference paper at COLM 2024
Language
Africa
America
Asia
Europe
Mean
STD
Spanish 0.624
0.641
0.627
0.640
0.633
0.087
German
0.622
0.642
0.616
0.628
0.627
0.094
Swahili
0.643
0.637
0.585
0.610
0.619
0.096
English
0.607
0.632
0.613
0.607
0.615
0.086
French
0.595
0.655
0.609
0.594
0.613
0.110
Bengali 0.579
0.574
0.585
0.589
0.582
0.149
Arabic 0.493
0.559
0,485
0.522
0.515
0.167
Korean 0.481
0.501
0.490
0.476
0.487
0.208
Chinese 0.453
0.502
0.479
0.514
0.487
0.230
Table 3: Overall FActScore breakdown of GPT4 by language with mean and standard devia- tion (STD). The continents with the highest and second-highesi values for each language are emphasized in bold and underlined, respectively. Note that for all language except for Swahili, GPT4 is leaning towards American and European leaders in terms of factual accuracy.
top-20 FActScore. Our findings reveal strong bias towards American and European topics across all languages studied. This pattern persists even in languages like geographicallly distant from these continents, such as Korean and Chinese. For Arabic, while the highest FActScore is linked to Africa, likely reflecting its geographical roots, American topics still rank second place. This trend highlights Western-centric bias in the model's factual content distribution across languages. The result figure and detailed analysis of geographical biases across more fine-grained subregions are presented in Appendix D
Conclusion
In this paper, we introduce Multi-FAct, extension of FActScore in multilingual setting to address the critical gap factuality evaluation for oon-English free-form generation from LLMs. We empirically demonstrate the viability of conducting factuality evaluation in the multilingual biography generation task. Our approach of translating generated content and comparing it against English Wikipedia proved effective, offering scalable method for assessing factual precision across multiple languages. Additionally, we find that sufficiently comprehensive non- -onglsh sources serve alternatives to English sources using the same pipeline.
Our analysis of LLMs' outputs across different languages and geographically contextualized questions reveals consistent advantage for English in terms of both factual accuracy and the quantity of generated facts, as well as better performance for content related North America and Europe across languages. Multi-FAct serves as a valuable tool for investigating these biases, emphasizing the need for efforts to improve the cultural and geographic fairness of factual generations from LLMs.
Limitations
Our work shares some of the same limitations that original FActScore faces. Throughout the whole paper, we assume English Wikipedia as the most authoritative knowledge base for fact verification. However, it is known that Wikipedia itself has a Western-centric biasNaowe et al. (203). Moreover, when combining multiple non-English Wikipedia, we do not take into account when Wikipedia articles in different languages contradict each other, which can happen for controversial topics. It is also worth noting that FActScore and Multi-FAct both do not differentiate between the informational value of facts. For example, statements treated
like "Person X attended MIT" and "Person X attended renowned university" equally, despite the former being more specific and informative. Future research should aim to distinguish between specific, valuable facts and generic, less informative ones.

---

Published as a conference paper at COLM 2024
Acknowledgments
This research project has benefitted from the Microsoft Accelerate Foundation Models Research (AFMR) grant program through which leading foundation models hosted by Microsoft Azure along with access Azure credits were provided to conduct the research. We thank Pang Wei Koh and Sewon Min for providing us with valuable feedback on an earlier draft of this work.
References
Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. mface: Multilingual summarization with factual consistency evaluation. arviv preprint ar arXiv:2212.10622, 2022.
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. MEGA: Multilingual evaluation of generative AI. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4232-4261 Singapore, December 2023, Association for Computational Linguistics. doi: (0.1863//1122033mmmppp
URL https
booglanthology.orr/22800 htt.2/222..mnllmmmm
258
Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when it's lying, 2023.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.), Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 675-718, Nusa Dua, Bali, November 2023. Association for Computational Linguistics. URL https \/aclanthology.orr//2Z22 1in pajinnp--mann...
Yang Trista Cao, Anna Sotnikova, Jieyu Zhao, Linda X. Zou, Rachel Rudinger, and Hal Daume III au2. Multilingual large language models leak human stereotypes across language boundaries, 2023.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions. arxiv preprint arXiv:1704.0005: 2017.
I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. Factool: Factuality detection in generative ai-a tool augmented framework for multi-task and multi-domain scenarios. arviv preprint drXie:2307.33528, 2023.
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celiky- ilmaz, and Jason Weston. Chain-of-verification reduces hallucination in large language models, 2023.
Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. A survey on automated fact- checking. Transactions of the Association for Computational Linguistics, 10:178-206, 2022.
Ashim Gupta and Vivek Srikumar. X-fact: A new benchmark dataset for multilingual fact checking. arXiv preprint arXiv:2106.09248, 2021.
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. Challenges and strategies in cross-cultural NLP. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Anmual Meeting of the Association for Compu- fational Linguistics (Volume 1: Long Papers), pp- 6997-7013, Dublin, Ireland, May 2022.
10

---

Published as a conference paper at COLM 2024
Association for Computational Linguistics. doi: 10.1865//112002.dd-logg// URL https://accamttioooyy..r////////
.82
Dirk Hovy and Diyi Yang. The importance of modeling social factors of language: Theory and practice. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Online, June 2021.
Computational Linguistics: Human Language Technologies, pp. Association for Computational Linguistics. doi: 10. (0.18653/1 10/553/11// v 2021 www.....a..mmmi.naaam URL https://ac.antholoog..oo org/ naacc-main.
Jing Huang and Diyi Yang. Culturally aware natural language inference. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7591-7609, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/vi /v11 weed.fiinnnnneminnn emnlp.500
URL https://aclantthol
0rg/2023.findiin
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arviv preprint ar XrXiv:2311.05232, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. Xiv preprint ar Xio::2310.06825, 2023.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. Is chatgpt good translator? preliminary study. arviv preprint ar drXiv:2301.08745 1(10), 2023.
Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury, and Srijan Kumar. Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries, 2023.
Haoqiang Kang, Terra Blevins, and Luke Zettlemoyer. Comparing hallucination detection metrics for multilingual generation. arXiv preprint arXio:2402.10496 2024.
Amrith Krishna, Sebastian Riedel, and Andreas Vlachos. Proofver: Natural logic theorem proving for fact verification. Transactions of the Association for Computational Linguistics, 10: 1013-1030, 2022.
Potsawee Manakul, Adian Liusie, and Mark J. Gales. Selfcheckgpt: Zero-resource black- box hallucination detection for generative large language models, 2023.
Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. Nonparametric masked language modeling. arXiv preprint ar axXi::2212.01349, 2022.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation, 2023.
Shujaat Mirza, Bruno Coelho, Yuyuan Cui, Christina Pöpper, and Damon McCoy. Global- liar: Factuality of llms over time and geographic regions. ar Xiv preprint arXiv:2401.17839 2024.
Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models. ar Xiu preprint ar XrX:::2401.066 2024.
Tarek Naous, Michael J Ryan, and Wei Xu. Having beer after prayer? measuring cultural bias in large language models. ar Xix preprint arXit:2305.14455 14456,, 2023.
11

---

Published as a conference paper at COLM 2024
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. Kilt: a benchmark for knowledge intensive language tasks. arXiv preprint (7X::2000.002252, 2020.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER:
large-scale dataset for fact extraction and VERification. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langwage Technologies, Volume I (Long Papers), pp. 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1. /v1111111111 URL ptttp:://aclanthol.gg. org/N18- 1074
Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, et al. Factcheck-gpt. End-to-end fine-grained document-level fact-checking and correction of I1m output. ar Xiv preprint ar Stt::231.090000 2023.
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. Long-form factuality in large language models, 2024.
Wikipedia contributors. Wikipediaamultilnnguall statistics - Wikipedia, the free encyclope- dia, 2024. URL https: http://www..iiiiiiaaa.a.a htt.://cc........e........aaaaaa [Online; accessed 24-Febrruar-2024).
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's 's song in the ai ocean: a survey on hallucination in large language models. aio preprint Xiv:2309.01219, 2023.
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. Reasoning over semantic-level graph for fact checking. arv preprin! ar arX1::1909.00745, 2019.
12

---

Published as a conference paper at COLM 2024
A Comparison with Baseline
We provide a comparison of different components of Multi-FAcc below.
Trivial baselines: Random prediction: 8.86% Always "Supported": 41.15% Always "Not
Supported"; 58.86%
Method
Arabie
Swahili
Bengali
Chinese
Korean
French
German
Spanish
Mean
Retrieval ChatGPT
-0.268
-0.282
-0.260
-0.265
-0.277
-0.278
-0.288
-0.290
-0.276
Retrieval + ChatGPT
-0.163
-0.190
-0.165
-0.168
-0.177
-0.177
-0.175
-0.187
-0.175
Retrieval Mistral
-0.150
-0.180
-0.169
-0.162
-0.172
-0.171
-0.175
-0.174
-0.169
NPM
-0.097
-0.096
-0.098
-0.121
Retrieval Mistral NPM
0.028
-0.016
0.022
0.029
0.006
-0.013
-0.013
-0.012
0.004
Table 4: Multi-FAct pipeline component comparison across different languages.
In Table
ChatGPT refers GPT3.5-turbo.
No Retrieval ChatGPT means simply asking ChatGPT whether a fact is correct or not (closed book setting).
Retrieval ChatGPT implies retrieving relevant passages from Wikipedia and then answering whether fact is correct or not using GPT3.5.
Retrieval + Mistral refers to retrieving passages and then using Mistral 7B to answer whether a fact is correct or not. This is similar to the previous setting except for the fact verification LLM is being switched to open-source model.
NPM refers to Non-Parametric Model (Min et al. (2022)), which requires having access to the Wikipedia article (ie, retrieval enabled).
The final method is the ensemble version of NPM and Retrieval Mistral and performs the best (also reported in original FActScore work (Min et al. (2023 )) This
is the one we used throughout our paper for other analyses.
While GPT3..5 and Mistral 7B are very similar at atomic fact breakdown, we use Mistral 7B in all experiments to save cost.
B Effect of non-LLM Based Translation
showed that GPT3.5-based translation does not hurt the FActScore estimate. We have also explained the reason we use GPT3.5 over commercial translation systems like Google Translate is because Google Translate is not able to keep the gender of the subject consistent across the generation. Table provides a comparison translation of FActScore estimation when the translation done by Google Translate vs GPT3.5.
In section 3.4.2
The experiment set up for Table was the same as Table 2 The small difference in Error with GI71..5 column arises from the run-to-run stochasticity of the FActScore estimation.
Can we skip the translation step altogether?
We also experimented with generating atomic facts in the target language (thus, skipping the translation step) with French and Korean using GPT3.5 Turbo. Qualitative analysis with native Korean speakers revealed that the generated atomic facts are quite reasonably good. However, because there is no NPM model available for Korean and French, the FActScore estimates using atomic facts generated with those methods do not match the best- best-performing method Misrrvvall-istral-NPPP tokenizer fertility for non-English languages is often much worse than for English, which makes it much more computationally expensive to do atomic fact breakdown in Korean. At the time writing this paper, no small open-source model, like Mistral 7B, could perform atomic fact breakdown in diverse multilingual setting.
provided in the above table. Moreover,
13

---

Published as a conference paper at COLM 2024
Source Language
GPT3.5 Error
GTranslate Error
Arabic
0.028
0.051
Swahili
-0.0157
0.0168
Bengali
0.0219
0.057
Chinese
0.0289
0.0883
Korean
0.0063
0.0453
French
-0.0126
-0.0106
German
-0.0127
-0,0019
Spanish
-0.0115
0.0581
Mean
0.0041
0.0380
Table 5: FActScore estimation error rate comparison when translation to English done using GPT-3.5 and GTranslate across different languages
C Additional Results
C.1 Results from other models
Table shows the benchmark results from Claude-3 and Gemini-1.0-pro model. We chose to use GPT4 for most of our analysis because GPT4 has the best performance in low resource languages.
Language
(Claude) haiku
opus
sonnet
gemini-1.0-pro
8pt3.5
gpt4-turbo
gpHoommni
gptio
English
0.68
0.70
0.62
0.44
0.57
0.62
0.54
0.58
French
0.65
0.64
0.60
0.42
0.59
0.61
0.56
0.58
Spanish
0.66
0.70
0.60
0.47
0.57
0.63
0.59
0.59
German
0.63
0.66
0,58
0.43
0.57
0.63
0.55
0.60
Chinese
0,27
0.43
0,40
0.40
0.49
0.49
0.53
Korean
0.39
0.40
0.18
0.48
0.49
0.52
Arabic
0.33
0.45
0.26
0.32
0.24
0.53
0.46
0.51
Bengali
0.26
0.36
0.18
0.40
0.12
0.58
0.48
0.58
Swahili
0.57
0.62
0.54
0.46
0.59
0.62
0.65
0.65
Mean
0.49
0.56
0.43
0.42
0.43
0.58
0.53
0.57
Table 6: FActScore estimation of the other models on the same national leaders' biography generation task. Note that only GPT4-turbo has the overall highest performance and on average GPT4 series models have good performance across all languages.
Note that model response size varies significantly across languages and models, which is not captured in this table. Similar to what shown in figure 6 most of the models in this table tend to generate much longer responses for English and high-resource languages. The average response length also differs from model family to model family.
C.2 Effect of Reference Length
Sampling Method We randomly collected 10,000 people from 9 different Wikipedia using Wikipedia APT and sorted them by their byte length. After that, we kept the top 100 entities that also had an English Wikipedia. Note that since we did this for each language separately, the list of 100 people differs for each language. However, manual inspection revealed that these entities are usually famous politicians, religious figures, famous athletes, and singers.
Table 7able/ shows the word count distribution for the selected Wikipedia articles.
14

---

Published as a conference paper at COLM 2024
Language
Mean Text Length (Language Wiki) Mean Text Length (English Wiki)
Spanish
3353.94
6895.68
French
3617.13
6569.71
German
3519.03
5723.78
Korean
1520.79
5144.43
Chinese
2879.44 44
4937.70
Arabic
2829,49 A8
5648.35
Swahili
865.85
5003.25
Bengali
2772.49
7829.74
Table 7: Comparison of mean text lengths (in words) in Various Language Wikipedia after translating English vs length of the same articles English Wikipedia. For the same subjects, English Wikipedia has much larger articles than the other languages.
D Geographical Biases in Factual Precision Across Subregions
Continent
America
Europe
Acia
Africa
Fequency
6
English
Bengali
Arabic
Swahill
Karean
Franch
Sparteon
Chinesse
German
Figure 7: Continental distribution of top 20 countries that had the highest FActScore in each language for GPT4. The way to interpret this plot is, say for German, we sort the twenty most accurate presidents' biographies and then look up which continents they belong to. Note that for almost all languages, the top twenty biographies belong to subjects from America and Europe.
We examine how languages with notable variations in factuality across continents exhibit distinct geographical biases. Further analysis involves breaking down the continents into sub-regions to compare the number of correct facts and FActScore among three languages: Chinese and Korean, which exhibit he highest standard deviation (SD), and English, which demonstrates the lowest SD among the four continents, detailed in Table 3 of & 5.2
Chinese stands out in Eastern Asia, achieving the highest number of correct facts among all regions analyzed, indicating pronounced bias towards its main linguistic area. Korean, also categorized in Eastern Asia, shows comparable levels of factuality in Eastern Asia to that of regions associated with America and Europe, underscoring geographical biases in the model's factual precision. English, while displaying a bias towards North America, its main region, exhibits relatively even factuality across other regions.
Figure shows the fine-grained geographic distribution of languages with the highest standard deviation (Chinese, Korean, Arabic) and the least standard deviation (English).
15

---

Published as a conference paper at COLM 2024
Africa
Ansedca
(a) English
(b) Chinese
200
200
390
Leper
Loten
Oodin
Abiia
Amrrica
Eurpe
Atkic
Adla
Eоroaa
(c) Korean
(d) Arabic
Figure 8: Fine graiegg geographic distribution of languages with the highest standard deviation (Chinese, Korean, Arabic) and the least standard deviation (English) given. The top bar represents the average number of hallucinated facts, and the bottom one denotes correct facts. The number on top of each bar represents FActScore.
E
List of Countries and Presidents
Table lists all the names of the countries, presidents and their Geographic locations. Please note that, we changed Australia and New Zealand to Oceania in the UN geoscheme for plotting purposes.
16

---

Published as a conference paper at COLM 2024
Country
Continent
Region
President
Ethiopia
Africa
Eastern Africa
Hailemariam Desalegn
Tanzania
Africa
Eastem Africa
lakaya Kikwete
Kenya
Africa
Eastem Africa
Uhuru Kenyatta
Uganda
Africa
Eastem Africa
Yoweri Museveni
Mozambique
Africa
Eastem Africa
Filipe Nyusi
Madagascar
Africa
Eastem Africa
Hery Rajaonarimampianina
DR Congo
Africa
Middle Africa
Joseph Kabila
Angola
Africa
Middle Africa
José Eduardo dos Santos
Cameroon
Africa
Middle Africa
Paul Biya
Egypt
Africa
Northern Africa
Abdel Fattah el-Sisi
Algeria
Africa
Northern Africa
Abdelaziz Bouteflika
Sudan
Africa
Northern Africa
Omar al-Bashir
Morocco
Africa
Northern Africa
Abdelilah Benkirane
South Africa
Africa
Southern Africa
Jacob Zuma
Nigeria
Africa
Western Africa
Muhammadu Buhari
Ghana
Africa
Western Africa
John Mahama
Côte d'Ivoire
Africa
Western Africa
Alassane Ouattara
Niger
Africa
Western Africa
Mahamadou Issoufou
Burkina Faso
Africa
Western Africa
Kafando
Mali
Africa
Western Africa
Ibrahim Boubacar Keita
Australia
America
Australia and New Zealand
lony Abbott
New Zealand
America
Australia and New Zealand
John Key
Haiti
America
Caribbean
Michel Martelly
Cuba
America
Caribbean
Raúl Castro
Dominican Republic
America
Caribbean
Danilo Medina
Mexico
America
Central America
Enrique Peña Nieto
Guatemala
America
Central America
Otto Pérez Molina
Honduras
America
Central America
Juan Orlando Hernández
Nicaragua
America
Central America
Daniel Ortega
United States
America
Northern America
Barack Obama
Canada
America
Northern America
Justin Trudeau
Brazil
America
South America
Dilma Rousseff
Colombia
America
South America
Manuel Santus
Argentina
America
South America
Cristina Femández de Kirchner
Peru
America
South America
Ollanta Humala
Venezuela
America
South America
Nicolás Maduro
Chile
America
South America
Michelle Bachelet
Ecuador
America
South America
Rafael Correa
Bolivia
America
South America
Morales
Paraguay
America
South America
Horacio Cartes
Uzbekistan
Asia
Central Asia
Islam Karimov
China
Asia
Eastem Asia
Xi Jinping
Japan
Asia
Eastem Asia
Shinzo Abe
South Korea
Asia
Eastem Asia
Park Geun-hye
Indonesia
South-Fastern Asia
Joko Widodo
Philippines
Asia
South-Eastern Asia
Benigno Aquino III
Vietnam
Asia
South-Fastern Asia
Trng Tn Sang
Thailand
Asia
South-Eastern Asia
Prayut Chan-a-cha
Myanmar
South-Fastern Asia
Thein Sein
Malaysia
Asia
South-Easterm Asia
Najib Razak
India
Asia
Southern Asia
Narendra Modi
Pakistan
Asia
Southern Asia
Mamnoon Hussain
Bangladesh
Southern Asia
Sheikh Hasina
Iran
Asia
Southern Asia
Hassam Rouhani
Afghanistan
Southern Asia
Ashraf Ghani
Nepal
Asia
Southern Asia
Sushil Koirala
Turkey
Western Asia
Reeep Tayyip Erdogan
Iraq
Asia
Western Asia
Fuad Masum
Saudi Arabia
Western
Salman Saudi Arabia
Yemen
Asia
Western Asia
Abdrabbuh Mansur Hadi
Russia
Europe
Fastemn Furope
Vladimir Putin
Ukraine
Europe
Easter Europe
Petro Poroshenko
Poland
Europe
Eastem Europe
Andrzej Duda
Romania
Europe
Europe
Klaus lohannis
Czech Republic
Europe
Eastem Europe
Miloš Zeman
Hungary
Europe
Fastem Europe
János Ader
Belarus
Europe
Eastem Europe
Alexander Lukashenko
Bulgaria
Europe
Fastem Furope
Rosen Plevneliev
United Kingdom
Europe
Northern Europe
David Cameron
Sweden
Europe
Northern Europe
Stefan Läfven
Italy
Europe
Southern Europe
Sergio Mattarella
Spain
Europe
Southern Europe
Mariano Rajoy
Greece
Europe
Southern Europe
Prokupis Pavlopoulas
Portugal
Europe
Southern Europe
Marcelo Rebelo de Sousa
Germany
Europe
Western Europe
Joachim Gauck
France
Europe
Western Europe
François Hollande
Netherlands
Europe
Western Europe
Mark Rutte
Belgium
Europe
Western Europe
Charles Michel
Austria
Europe
Western Europe
Heinz Fischer
Switzerland
Europe
Western Europe
Simonetta Sommaruga
Table 8: List of Countries and Presidents Used in Our Experiment
17


Reviews:
[{"review_id": "3YbDASRzjN", "reviewer": "Reviewer_a4f9", "summary": "This paper explores multilingual LLM evaluation by adapting existing factual evaluation method FActScore. The paper also studies the potential and limitations of using non-English resources such as non-English wikipedia and geographic biases in LLM.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer a4f9,\n\n\nFirst of all, thanks for your comments and reviews. \n\n\nOn the points mentioned under “Reasons to Reject”, here are our thoughts:\n\n* It is true that our method uses an existing factuality evaluation method after translating multilingual generations into English. We made this choice for several reasons:\n    * Most LLMs, and all open-source LLMs, are not able to follow instructions properly in a multilingual setting, which means much of the original FActScore pipeline (which strongly depends on models’ capability to break down sentences into atomic facts and then do RAG) is unusable in other languages. This means it is simply impossible to evaluate and compare long-tail languages in a scalable fashion without translating them to English with current models. \n    * One of the main contributions of our paper (Section 3.4) was to show that we can use existing English factuality evaluation methods, like FActScore for non-English languages by translating the generations to English, which is not trivial at all. In fact, many tasks and metrics for LLMs that are originally built for English are not transferable to other languages without significant additional research, and it is unfair to ask for technical novelty on top of the 9 different languages (including low-resource languages) that we included in this work. Section 3.4.2 shows that the translation step does not significantly hurt factuality evaluation.\n\n* About providing more details on the FActScore details: We will modify our writing and the main figure accordingly so that it is easy for new readers to understand our method. Thanks for pointing this out!\n\n* On the matter of only using GPT4 and GPT3.5 for evaluation: Our method uses GPT3.5 based translation and Mistral 7B for factuality measurement. We did not evaluate llama, mistral (or any other open-source models) since they are meant to be English-only models and are not able to follow instructions/generate properly well for the languages we worked on. Most of our experiments are conducted on GPT3.5 and GPT4 because those two were the only strong multilingual models available at the time of writing the paper. More multilingual models like Claude 3 and Gemini 1.0 came out much later and they are also included in the appendix (Table 5). However, we plan to include models that came out since we submitted our paper (Command R+ and Gemini 1.5) in our final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper adapts FActScore to a multi-lingual factuality evaluation framework, and experiment the method using 8 languages\n\nProviding insight about reasonably large non-English wikipedia articles can also serve as a viable source.", "reasons_to_reject": "Overall, the originality of the work is limited, it uses GPT3.5 for language translation then adapts an existing factuality method.\n\nThe paper can be improved by providing more details about FActScore, it is hard for readers to understand the approach with no knowledge of the method.\n\nExperiments only use GPT3.5 and GPT4 for evaluation, it is better to have more baseline methods such as llama, mistral, etc.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vZRF4yW1Vj", "reviewer": "Reviewer_r8Mi", "summary": "It is nice that this paper tackles the multilingual challenge – this is very, very underexplored in our field. Overall, the paper makes a few design decisions that I would have probably done differently, but the authors do justify their decisions relatively well and I think I would be able to replicate their methods. In particular, I think the paper could be improved with two experiments:\n•\tIncluding at least one small experiment using a non-LLM translation compared to a translation LLM.\n•\tTrying to see how FActScore would work on a language without translation. Aka an experiment similar to section 4.1, but just done monolingually in the languages (such as Bengali generation and Bengali evaluation).\n\nOne of my first questions when I read the paper was “Why translate with GPT-3.5?”. The authors do address this. It is ok for some languages, but if you are going to do automatic evaluation, another commercial solution like Google Translate may make more sense – especially for low-resource languages (Kocmi et al, 2023; Robinson et al., 2023). The authors make a nice point about gender in Google Translate and why they did not use it, but I would have still liked to have at lease seen one experiment showing what happens if you use that as well. A comparison even for one language with Google Translate and GPT-3.5.\n\n\nI would have also liked to have seen how the method works monolingually (aka not translating to English). Again, the authors make claims as to why they made these decisions such as “which makes it difficult to compare cross-lingual factuality if the knowledge base is different.” However, it would be nice to still see how it works in different resource settings. You are generating in that language, it could be another line or two in some tables. This feels like it could also answer some of my questions about translationese impacts.\n\nOverall, this paper is a nice attempt at multilingual fact checking, which is woefully underexplored. This alone is a worthwhile contribution to the field.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer r8Mi,\n\n\nThank you very much for your long, detailed and insightful review! We agree on your opinion on translationese and impacts of doing factuality verification in English.\n\n\nOn the matter of concerns listed under Reasons to Reject:\n\n\n* About non-LLM based translations: Thanks for pointing this out. It is our oversight not to include an experiment on the effects of different types of translation methodology in the appendix. We are conducting the experiment right now and will share the results with you in this thread as soon as we are done.\n\n* About the experiment doing FActScore monolingually:\n    * This is a very valid point, and the reason we didn’t do it before was that our method uses Mistral for atomic fact generation and then doing RAG on wikipedia to verify the validity of the facts. None of these two steps are possible by Mistral, since it is also English only. Besides, there is was concern regarding which languages’ corpus to use for a fair comparison across languages.\n\n\n    * However, provided that you asked, we are conducting the experiment on a subset of the languages using InstructGPT/GPT3.5-Turbo [which are multilingual]. We will let you know the experiment results in this thread as soon as it is done.\n\nIf you have any more questions or concerns, please let us know!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There are very few papers looking at multilingual fact checking in LLMs. The field needs more of these.", "reasons_to_reject": "I'd like to see the following two experiments:\n•\tIncluding at least one small experiment using a non-LLM translation compared to a translation LLM.\n•\tTrying to see how FActScore would work on a language without translation. Aka an experiment similar to section 4.1, but just done monolingually in the languages (such as Bengali generation and Bengali evaluation).", "questions_to_authors": "Figure 5, the x-axis should be the same between a and b. Either (0.0-1.0) for both or (0.0-0.8) for both - not for 0.8 for one and 1.0 for the other.", "extra_scores": {}}, {"review_id": "Cr1KFdOmOT", "reviewer": "Reviewer_5m3n", "summary": "This paper the problem of factuality evaluation of multilingual LLMs. It introduces a pipeline, Multi-Fact, for multilingual factuality evaluation, adapting FactScore for diverse languages. This pipeline first translates texts in other language into English, and then using FactScore for evaluation. In experiments, this paper explores the potential and limitations of using non-English resources for factuality evaluation of long-form generation. The Multi-Fact tool can also be used to investigate cultural and geographic biases in LLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 5m3n,\n\n\nThank you for your review and comments. \n\n\nAs to the concerns you have raised in the Rejections to Reject section, it may be helpful for us to clarify the main novelty and contribution of our paper. Please read below and let us know if you have additional questions or comments:\n\n* Yes, it is true that the technical complexity of the proposed method is relatively simple, translating texts from other languages to English and then applying  FactScore to evaluate them. \n\n    * The core contribution of this work is proposing a methodology that develops upon FactScore, which was only applicable in English, that can measure the factuality of text generated by multiple languages.\n    * We show that this methodology works for 8 different languages, which opens up possibilities to study the factuality of LLMs in many non-English languages. Our method is simple and easy to use, which makes it more powerful. \n\n    * We also rigorously tested that the translation step does not hurt the estimation (Section 3.4), and show how current LLMs are not performing differently for different languages, which is problematic.\n\n* About your concern regarding baseline: would you please explain a little more about what baseline we should have included?\n\n    * While we agree with you that comparison with baselines is important to show the validity of a metric, we did not include experiments comparing with baseline since our work is the first work that tries to evaluate multilingual models’ factuality in long form generation. Our factuality evaluation pipeline is based on FActScore (for English only) and we used the best-performing long-form factuality evaluation method. \n\n    * We will conduct an experiment by simply doing everything in the original language (without translation), as we think this is a valid baseline. If you are thinking of another, more valid baseline, it would be very helpful if you can tell us your idea.\n\n* About the use of Wikipedia data:\n\n\n    The proposed method is general and can be applied using any ground truth corpus with reasonably good coverage. None of the steps in our pipeline is dependent on the corpus being Wikipedia. In fact, you can use any corpus, in any language, into our pipeline.\n\n\n    We use Wikipedia because a previous study (Min et al 23) has shown it to be of reasonably good coverage in agreement with works (Chen et al., 2017;Petroni et al., 2021) that treat Wikipedia as a general-purpose knowledge corpus.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Factuality evaluation in multi-lingual LLMs is an interesting and important research problem.\n- The proposed method is simple yet effective.\n- There are some interesting findings in the experiments, which are inspiring for future research on building multi-lingual LLMs.\n- The paper is well-written, and the idea is easy to follow.", "reasons_to_reject": "- The novelty and technical complexity of the proposed method are quite basic. It mainly translates texts from other languages into English and then uses the FactScore tool for evaluation. There is no innovative approach or model created specifically for multilingual scenarios.\n- The experiments do not compare the proposed method with any baseline methods. It is unclear whether the proposed method is effective, even though it seems reasonable.\n- The experiments only use Wikipedia as a data source, which limits how widely the proposed method can be applied.", "questions_to_authors": "N/A", "extra_scores": {}}, {"review_id": "pcjFgk0IER", "reviewer": "Reviewer_bkdk", "summary": "This paper proposes to extend FACT-Score to be multilingual and shows various analyses across languages and geographies. They find that shorter Wikipedia pages in other languages provide less information to use for FACT-Score resulting in lower scores. They show this can be partially resolved through translation and ensembling of Wikipedia pages. They also show a western bias in FACT-Score. \n\nMy score is marginally below as of right now, but I think with fixing the grammar and other issues it would go to marginally above.\n\nEDIT: increased my score, see response to their rebuttal.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for comments and reviews. \n\nOn the points mentioned under “Reasons to Reject”, here are our thoughts:\n- Regarding our experimental setting, \n  - We used 80 instances per language, with references from the English Wikipedia. All datasets and code will be open-sourced and made publicly available on GitHub.\n  - Table 3 shows the results of our experiment comparing the original InstructGPT implementation with our modified Mistral implementation. The use of Mistral resulted in a 2% point increase in performance compared to InstructGPT (originally reported in FActScore paper). We will ensure that the experimental details are clearly explained in the manuscript.\n- We appreciate the feedback about the title and understand that it may have been misleading. We had no intention to overstate our contributions. We will revise the title to “MultiFAct: Multilingual Factuality Evaluation using FActScore” and related sentence in the abstract to “we apply FActScore to multilingual settings.”\n- Thank you for the detailed feedback on the areas that need improvement. We apologize for the hasty writing and will tidy up overall manuscript. We will accept all the suggestions provided in the minor comments. To demonstrate how we would revise the paper to be more clear and concise, we have revised the first two paragraphs of Section 4.2, \"Effect of Reference Length,\" as follows:\n\n   We hypothesize that non-English Wikipedias result in higher errors compared to English Wikipedia due to the lower coverage of topics, as evidenced by the shorter length of non-English articles. Figure 3b shows that there is a linear relationship between FActScore estimation error and the word count of Wikipedia articles, indicating that longer Wikipedia articles tend to serve better as references for factuality evaluation. \n  To further test the linear relationship observed, we select 100 individuals from each of our studied languages who have both English Wikipedia articles and substantially long Wikipedia articles in their regional language. We generate biographies for these individuals in English and evaluate them against both the English and regional language versions of Wikipedia. Since FActScore estimation for English is nearly perfect, we use it as a benchmark to estimate the error when using non-English articles as references. Figure 4a illustrates that the linear relationship between the length of articles and FActScore holds for longer articles as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides many interesting analyses into how different Wikipedia languages play a role in impacting FACT-Score values. \n- They propose several ways of countering this length bias that are intuitive and seem to help. Figures are well done and easy to understand\n- The topic is important when discussing the evaluation of LLMs across languages", "reasons_to_reject": "1.  Some of the experimental settings are unclear. For Table 3 and Figures 6-7, which references are being used? How many instances of data are used per language?  Will these datasets be open-sourced?\n- Other questions include, how does each of the changes you made in terms of using open models impact these results? E.g. Does using Mistral cause a change?\n\n2. There are a decent amount of mistakes in the paper in terms of spelling and latex references missing, which are minor and can be resolved but indicate a hurried submission\n\n3. The paper title seems to indicate they extended Fact-Score itself, but in reality they applied it to new data - which is a fine contribution but somewhat misleading since there are no methodological contributions to the base FACT-Score.", "questions_to_authors": "Minor comments:\n1. The related work should be a cite paper with parentheses for many of the citations there (some are correctly using the name, others should be in parens).  This should also be followed throughout the paper - only use the non-parens version when citing the name directly.\n2. 5.2 should be low resource language**s**\n3. Footnote 5 on page 6 has an undefined reference\n4. Table 3 could be made clearer by multiplying the values by 100 and underlining less - it is a little busy", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper uses FactScore to perform an evaluation of factuality in multiple languages. To do, so it generates responses in non-English languages, translates them to English, then runs FactScore. It also replaces several of the components of FactScore with open source models. 

The reviews on this paper were split, two positive and two negatives. Reviewers appreciated the topic of multilingual fact checking, and the interesting analyses of the effect of different Wikipedias. There were questions about the effect of translating into English, and although the authors already investigated this in their original paper, they added more analyses, including translating with another translation software and performing some factuality evaluation in non-English languages. These additions greatly improve confidence in the validity of the results and should be included in any revision. 

The main concerns raised by the reviewers were the limited novelty and missing baselines. While the paper doesn't propose a new method, I agree with the authors and one of the reviewers that this isn't a problem -- the investigation and results are interesting and useful to the community as they are. Concerning missing baselines, the author made an attempt to add a simple random baseline and provided ablations for their method (https://openreview.net/forum?id=lkrH6ovzsj&noteId=S3yAsA5YMt), which are helpful. These, too, should be included in any revision.

Given the improvements to the paper, and the valuable topic and insights, I believe this paper provide a useful contribution to the community and should be accepted, on the assumption that the above comments are addressed for the next revision.

[At least one review was discounted during the decision process due to quality]
Decision: Accept