CaLMQA: Exploring culturally specific long-form question answering across 23 languages
Authors: Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit Iyyer, Eunsol Choi
Accepted at: acl25

CaLMQA: Exploring culturally specific long-form question answering across 23 languages

Shane Arora*†  Marzena Karpinska*  Hung-Ting Chen◊
Ipsita Bhattacharjee◊  Mohit Iyyer◊†  Eunsol Choi◊†
The University of Texas at Austin*
New York University◊
University of Massachusetts Amherst◊
shane.arora@utexas.edu {hung-ting.chen, eunsol}@nyu.edu
{mkarpinska, ibhattacharj, miyyer}@umass.edu

Abstract

Despite rising global usage of large language models (LLMs), their ability to generate long-form answers to culturally specific questions remains unexplored in many languages. To fill this gap, we perform the first study of textual multilingual long-form QA by creating CaLMQA, a dataset of 51.7K culturally specific questions across 23 different languages. We define culturally specific questions as those that refer to concepts unique to one or a few cultures, or have different answers depending on the cultural or regional context. We obtain these questions by crawling naturally-occurring questions from community web forums in high-resource languages, and by hiring native speakers to write questions in under-resourced, rarely-studied languages such as Fijian and Kirundi. Our data collection methodologies are translation-free, enabling the collection of culturally unique questions like “Kuber iki umwami wa mbere w’uburundi yitwa Ntare?” (Kirundi; English translation: “Why was the first king of Burundi called Ntare (Lion)?”). We evaluate factuality, relevance and surface-level quality of LLM-generated long-form answers, finding that (1) for many languages, even the best models make critical surface-level errors (e.g., answering in the wrong language, repetition), especially for low-resource languages; and (2) answers to culturally specific questions contain more factual errors than answers to culturally agnostic questions – questions that have consistent meaning and answer across many cultures. We release CaLMQA to facilitate future research in cultural and multilingual long-form QA.

github.com/2015aroras/CaLMQA
hf.co/datasets/shanearora/CaLMQA
© CC BY 4.0

*These authors contributed equally to this work.

1 Introduction

While large language models (LLMs) are increasingly used by people across the world, most NLP efforts are focused on English and western cultures. Growing evidence reveals significant gaps in their performance across languages (Qiu et al., 2023; Guerreiro et al., 2023) and their understanding of diverse cultures (Tao et al., 2024; Li et al., 2024), as well as a persistent bias toward Western-centric perspectives (Palta and Rudinger, 2023; Durmus et al., 2024; AlKhamissi et al., 2024; Naous et al., 2024). Existing research of multilingual QA largely focuses on assets derived from English resources (Singh et al., 2024; Zhang et al., 2023; Lai et al., 2023), limiting their coverage of culturally unique concepts especially in low-resource languages. While some prior work collects short-answer and multiple-choice questions in non-English languages (Myung et al., 2025; Clark et al., 2020; Liu et al., 2019), multilingual long-form QA, a task more aligned with real-world applications, remains unexplored.

In this work, we develop a translation-free multilingual QA dataset of long-form culturally specific questions: Cultural Long-form Multilingual Question Answering (CaLMQA). Questions are posed in the language of the target culture and demand nuanced, long-form responses. We only collect culturally specific questions that (1) refer to concepts unique to one or a few cultures, such as “Kuber iki umwami wa mbere w’uburundi yitwa Ntare?” (Kirundi),1 or (2) have different answers depending on the cultural or regional context, as in “बंदूक का लाइसेंस कैसे बनता है?” (Hindi).2 We contrast the quality of an LLM’s answers to these questions with its answers to culturally agnostic questions that have consistent meaning and

1English translation: “Why was the first king of Burundi called Ntare (Lion)?”
2English translation: “How do you get a gun license?”
Figure 1: Distribution of topics in CALMQA, with box size indicating the frequency of each topic. Each topic is accompanied by an example and its English translation. Table 12 contains descriptions of the topics, and §B describes our topic classification method.

answer across many cultures (e.g., “Why is smoking bad for the heart?”), which are prevalent in many translation-centric multilingual QA works.

Evaluation of multilingual long-form QA is challenging: lexical metrics for short-form QA do not correlate with human preferences in long-form QA (Krishna et al., 2021; Xu et al., 2023) or transfer from English to other languages (Kang et al., 2024; Koto et al., 2021; Min et al., 2023; Song et al., 2024). We thus adopt a multi-aspect evaluation protocol including (1) surface-level measures of language identification and repetition; (2) automatic factuality and relevance metrics run on translated answers; and (3) human evaluations from native speakers. To distinguish the effects of culture and language on model performance, we use a baseline set of parallel culturally agnostic questions created by translating a seed set of 51 English questions into the 22 other languages, following common practice in prior work (Vayani et al., 2024; Artetxe et al., 2020; Lewis et al., 2020; Alonso et al., 2024).

We show that seven popular LLMs, including closed models such as CLAUDE-3-OPUS, GEMINI-1.5-PRO and GPT-4o, suffer from basic surface-level issues, especially on low-resource languages (e.g., none of them reliably generate text in Afar). Also, open-weight models such as Mixtral-8x22B and Llama-3-70B often apologize instead of providing an answer or generate text in English when prompted with non-English questions. We observe that the factuality and relevance of LLM-generated culturally specific answers is significantly lower than that of culturally agnostic answers, underscoring the importance of studying culturally specific questions. Factuality and relevance drop considerably on low-resource languages, with GPT-4-TURBO and GPT-4o performing best.

We conduct a human evaluation on a subset of the data (spanning five languages) for the best-performing models. Native speakers rate and rank answers from different LLMs, and an analysis of their annotations reveals that omissions and factuality issues are strong predictors of answer quality ratings. This human evaluation also supports our automatic factuality and relevance evaluations in that culturally agnostic questions are twice as likely to receive higher ratings than culturally specific questions, regardless of the generation model.

Overall, our work establishes a foundation for studying multilingual long-form question answering by releasing CALMQA – the first textual multilingual long-form question answering (LFQA) dataset, with 51.7K questions across 23 languages derived from culturally specific sources.
2 CaLMQA: Cultural Long-form Multilingual Question Answering

Each of the 51.7K examples in CaLMQA consists of (1) a culturally specific question written in one of 23 languages, (2) an optional human-written English translation (for low-resource languages), and (3) an optional human-written reference answer (for high- and mid-resource languages). We detail CaLMQA’s collection process and statistics below.

2.1 What questions are culturally specific?

Culture is a multifaceted and abstract concept that eludes a simple definition (Adilazuarda et al., 2024; Liu et al., 2024). We define culturally specific questions as questions that (1) refer to topics, concepts, objects, entities or events that are unique to one or a few cultures, or (2) have different answers depending on the cultural or regional context. Our notion of culturally specific questions is based on Liu et al. (2024): “1) basic concepts that are ‘configured’ differently, reflecting the cultural-specific way of thinking, and 2) concepts that are unique to a culture”; our definition embeds the former by including questions with answers dependent on culture, and the latter by including questions that refer to concepts related to culture. Liu et al. (2024) taxonomizes cultural NLP works into 10 categories including values, norms and morals, and knowledge; we collect that cultural knowledge in CaLMQA.

2.2 Data Collection

We collect our dataset through two processes. For high- and mid-resource languages, we follow prior work (Fan et al., 2019) and collect questions from community Q&A forums. For low resource languages where such web content is scarce, we hire freelancers to write culturally specific questions.

Culturally specific questions for high- and mid-resource languages: Many countries have their own community forums where people can exchange information, similar to Quora, Reddit or StackExchange in English. We collect culturally specific questions from these websites via a crowdsourcing process that we scale with LLM assistance: first, we ask English-proficient Prolific3 crowdworkers from different countries to provide a link to a community web forum in their language that contains many complex questions that cover a diverse range of topics. Next, we ask workers to collect culturally specific questions and real users’ answers from the identified websites, for $0.65-1.33 USD per question. We manually review all provided examples and websites. Our workers yielded 923 questions across 11 languages with answers at a cost of $1427 USD (Table 4, left). Refer to §A.2 for more details.

We scale our question collection process by automating the collection and verification of questions. We obtain around 10k questions for each language. For English, Chinese and Russian, we use existing Hugging Face datasets containing questions scraped from our chosen websites (Gao et al., 2021; Wang, 2023; its5Q, 2022). For the remaining high-resource languages (except Hebrew, for which we were unsuccessful), we implement and utilize website-specific question extraction scripts. We do not collect answers due to the challenges of extracting them. We filter our questions using GPT-40-MINI, with two model passes that assess each question’s cultural specificity and general quality, retaining 52% of questions (prompts in Table 5 and Table 6). We apply these filters on the worker-collected questions too, retaining >90% of questions. This procedure yielded 50,227 additional questions at a cost of $34 USD.

Culturally specific questions for low-resource languages: Unlike existing LFQA datasets, CaLMQA also includes twelve low-resource languages (Table 4, right). We choose languages with scarce online resources that are not well-studied in prior work, but for which we can also find at least one annotator bilingual in English. We hire 29 native speakers (1-3 annotators per language, depending on their availability) on Upwork,4 each of whom receives guidelines, takes a paid ($7 USD) comprehension test, and then writes culturally specific questions with English translations for $0.65-1.00 USD per question. As having them write answers for all of these languages is prohibitively expensive, we collect answers and their English translations only for Kirundi ($2 USD per question, $106 USD total). This process yielded a total of 548 questions with English translations at a cost of $833 USD. The protocol was reviewed and deemed exempt by an Institutional Review Board. Please refer to §A.3 for more details.

Quality control: We screened crowdworkers through a qualification task to ensure understand-

3https://www.prolific.com/
4https://www.upwork.com/
ing of culturally specific, long-form questions. Each question curated by workers was manually reviewed by at least one author; workers provided clarifications or replaced unsuitable questions as needed. In the case of low-resource languages, the questions were also checked by another annotator of that language. See §A.2 for detailed guidelines.

2.3 Dataset Analysis

Table 1 and Table 4 summarize the statistics CALMQA’s 51.7K culturally specific questions. We measure the length of questions with bytes (Clark et al., 2020) as token count is not comparable across languages due to different compression rates (Ahia et al., 2023). High- and mid-resource language questions are generally longer than low-resource language questions, except for Arabic and Balochi. This can be largely attributed to different collection method (gathered from community forums vs. manually written by crowdworkers); see Table 9 for examples.

Finally, we categorize CALMQA’s questions based on their topic by first manually curating a set of categories and developing GPT-4-TURBO-based pipeline. Figure 1 shows a treemap of the question categories with examples. We find that the distribution of categories of culturally specific questions is similar between different languages. See §B for details.

3 Evaluating LLMs on CALMQA

We evaluate answers from seven state-of-the-art LLMs using automatic metrics for surface quality, relevance and factuality, combining these into a unified metric. We supplement this with human evaluation of LLM answers across five languages.

Models: We evaluate four closed-source LLMs (Claude-3-Opus, Gemini-1.5-Pro, GPT-4-TURBO, GPT-4o (Anthropic, 2024; Gemini Team, 2024; OpenAI, 2024a,b) and three open-weights LLMs (AYA-EXPANSE-32B, LLAMA-3-70B, Mixtral-8X22B (Dang et al., 2024; AI@Meta, 2024; Mistral AI, 2024). Model details are in Appendix Table 15.

Inference Setting: Each model is prompted with a question from our dataset in a zero-shot setup without instructions. We use greedy decoding and limit outputs to 2048 tokens. The total cost of API calls is $530 USD.5

5We note the total cost of calls for each model as follows: GEMINI-1.5-PRO $17 USD, GPT-4o $40 USD, GPT-4-TURBO $80, LLAMA-3-70B and Mixtral-8X22B $4 USD, and CLAUDE-3-OPUS $390 USD.

Data: For controlled comparison of LLM performance on questions with and without cultural knowledge requirements, we assemble an evaluation set of 3,644 questions from three sources: (1) all 1471 human-collected culturally specific questions, (2) 100 randomly sampled automatically collected questions per language, and (3) 51 culturally agnostic questions from r/explainlikeimfive translated into 22 languages using GPT-4-TURBO, which has demonstrated superior translation performance (Yan et al., 2024; Jiao et al., 2023). For Balochi, Fijian, and Kirundi, where translation quality was poor, we hire native speakers. This subset allows comprehensive evaluation while managing computational costs compared to using our full dataset of 51.7K questions.

3.1 Automatic Evaluation Metrics

Since common QA metrics like BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004a) do not correlate well with human judgement for long-form QA (Xu et al., 2023; Krishna et al., 2021), we (1) identify answers with surface level issues (e.g. incorrect language), (2) measure factuality and relevance of the remaining answers using the VeriScore pipeline of Song et al. (2024) and LLM-as-a-Judge (Zheng et al., 2023) with GPT-4o respectively, and (3) combine our individual measures to produce a single metric of answer quality.

Identifying surface-level issues (\( S_{surf} \in \{0, 1\} \)): Useful answers must be in the correct language (i.e., the language of the question) and free from word or phrase repetition. We start by detecting answers in the wrong language using a pipeline that combines polyglot6 and langid7, which yields optimal results for most languages (see Table 13 for accuracy). Balochi, Kirundi, Papiamento, and Hiligaynon are excluded due to low language identification accuracy. Then, we identify responses with repetitions by employing tiktoken8 with the o200_base encoding and flagging any answers in which a sequence of 20 tokens is repeated four or more times.9 See §C.1 for details. We assign a score of 1 if there is no surface issue, 0 otherwise.

6https://pypi.org/project/polyglot/
7https://pypi.org/project/pylangid/
8https://github.com/openai/tiktoken
9GEMINI-1.5-PRO often returned an API error for questions in low-resource languages; we mark such answer as invalid.
<table>
  <tr>
    <th>LANGUAGE</th>
    <th># Q</th>
    <th># A</th>
    <th>Q. BYTES (AVG/STD)</th>
    <th>A. BYTES (AVG/STD)</th>
    <th>LANGUAGE</th>
    <th># Q</th>
    <th>Q. BYTES (AVG/STD)</th>
    <th>A. BYTES (AVG/STD)</th>
  </tr>
  <tr>
    <td>English</td>
    <td>2617</td>
    <td>78</td>
    <td>205.1 / 209.4</td>
    <td>674.1 / 475.9</td>
    <td>Afar</td>
    <td>25</td>
    <td>43.7 / 16.5</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>5300</td>
    <td>85</td>
    <td>127.0 / 77.2</td>
    <td>2105.0 / 2378.6</td>
    <td>Balochi</td>
    <td>65</td>
    <td>122.7 / 52.4</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>5901</td>
    <td>75</td>
    <td>69.0 / 49.2</td>
    <td>588.8 / 939.7</td>
    <td>Faroese</td>
    <td>30</td>
    <td>47.8 / 16.6</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>German</td>
    <td>4091</td>
    <td>96</td>
    <td>427.8 / 451.9</td>
    <td>1169.0 / 744.7</td>
    <td>Fijian</td>
    <td>75</td>
    <td>75.0 / 36.9</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Hebrew</td>
    <td>96</td>
    <td>96</td>
    <td>142.5 / 84.2</td>
    <td>2043.6 / 1934.9</td>
    <td>Hiligaynon</td>
    <td>65</td>
    <td>93.4 / 39.1</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>6404</td>
    <td>91</td>
    <td>133.7 / 46.5</td>
    <td>3618.8 / 1867.1</td>
    <td>Kirundi</td>
    <td>53</td>
    <td>64.6 / 21.2</td>
    <td>557.2 / 160.9</td>
  </tr>
  <tr>
    <td>Hungarian</td>
    <td>3843</td>
    <td>75</td>
    <td>366.0 / 441.8</td>
    <td>379.3 / 333.2</td>
    <td>Papiamento</td>
    <td>10</td>
    <td>66.8 / 28.5</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>6466</td>
    <td>75</td>
    <td>814.2 / 696.7</td>
    <td>920.6 / 637.1</td>
    <td>Pashto</td>
    <td>75</td>
    <td>64.8 / 26.9</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>5875</td>
    <td>75</td>
    <td>248.8 / 198.5</td>
    <td>1008.6 / 936.3</td>
    <td>Samoan</td>
    <td>25</td>
    <td>51.2 / 19.3</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>5403</td>
    <td>75</td>
    <td>291.0 / 487.7</td>
    <td>4546.7 / 5067.9</td>
    <td>Tongan</td>
    <td>10</td>
    <td>81.2 / 19.2</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>5058</td>
    <td>102</td>
    <td>547.3 / 544.0</td>
    <td>852.0 / 817.9</td>
    <td>Tswana</td>
    <td>65</td>
    <td>87.2 / 43.4</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>51150</td>
    <td>923</td>
    <td>152.8 / 140.8</td>
    <td>1640.8 / 2291.3</td>
    <td>Wolof</td>
    <td>50</td>
    <td>45.3 / 18.9</td>
    <td>N/A</td>
  </tr>
</table>

<table>
  <tr>
    <th>LANGUAGE</th>
    <th># Q</th>
    <th>Q. BYTES (AVG/STD)</th>
    <th>A. BYTES (AVG/STD)</th>
  </tr>
  <tr>
    <td>Total</td>
    <td>548</td>
    <td>75.1 / 41.3</td>
    <td>557.2 / 160.9</td>
  </tr>
</table>

Table 1: Data statistics of high- & mid-resource language (left) and low-resource language (right) culturally specific questions. We report the number of bytes in the UTF-8 encoding as token counts will significantly vary between the languages. For high- & mid-resource languages, answers were only obtained for a subset of questions collected by crowdworkers, due to challenges with extracting and ranking answers automatically. For low-resource languages, we collect answers for Kirundi only. See Table 4 for culturally agnostic questions.

We only evaluate factuality and relevance for answers without surface-level issues.

Evaluating factuality (\( S_{fact} \in [0, 1] \)): To evaluate factuality of long-form texts, FACTSCORE (Min et al., 2023) verifies automatically extracted claims against retrieved evidence, and recent work expands this to multilingual texts by translating the non-English responses into English (Shafayat et al., 2024). Following this, we translate our questions and answers into English using GPT-4o. Then, we apply the claim extraction and verification pipeline introduced in Song et al. (2024), which improves the robustness of FACTSCORE by focusing exclusively on verifiable, non-trivial claims and using Google Search to obtain evidence.\footnote{We use Google Search via the Serper API at a total cost of $510 USD.} Finally, for every valid answer (i.e., answer without surface-level issues), we obtain a list of claims, the corresponding top 10 search results, and faithfulness labels (supported or unsupported); see Figure 7 for more details. The \( S_{fact} \) score will be the fraction of claims that are deemed supported, or 0% if there are no verifiable claims.

Evaluating relevance (\( S_{rel} \in \{0, 1\} \)): LLM prompting has been shown to have reasonable agreement with human annotations in English and multilingual settings (Hada et al., 2023; Hu et al., 2024). Hence, to evaluate the relevance of long-form answers to their questions, we employ LLM-as-a-Judge (Zheng et al., 2023) using GPT-4o. That is, we prompt GPT-4o to decide whether each answer is relevant to its question, using the prompt in Figure 15 with the English translation of the question and answer from our factuality evaluation, at a total cost of $120 USD.

Overall performance: We combine three metrics to measure the overall quality of the general answer. We obtain the overall quality score at the instance level \( S \) by multiplying the surface-level quality, factuality and relevance scores (\( S = S_{surf} * S_{fact} * S_{rel} \)).

3.2 Results of automatic evaluation

Table 2 reports micro-averaged automatic metrics of each model on culturally agnostic and culturally specific sets, respectively.

Answers to culturally agnostic questions are more factual: Generated answers to culturally agnostic questions tend to be more factual (64%–71%) than answers to culturally specific questions (45%–52%).\footnote{Models generate a similar number of factual claims on average for both culturally specific and culturally agnostic questions, with the former yielding slightly lower mean claim counts (see Figure 8).} By contrast, surface issues and relevance are relatively consistent between culturally specific and culturally agnostic questions.

Open-weight models perform much worse than closed-weight models in low-resource languages:
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">Surface Level</th>
    <th colspan="2">Fine-Grained</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>Wrong Lang. (%) ↓</th>
    <th>Repetitions (%) ↓</th>
    <th>W/o Issues (%) ↑</th>
    <th>Factual Pr. (%) ↑</th>
    <th>Relevance (%) ↑</th>
    <th>Overall Score \( S \) ↑</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>2.7 / 1.2</td>
    <td>7.5 / 1.4</td>
    <td>90.4 / 97.4</td>
    <td>69.6 / <b>52.2</b></td>
    <td>88.2 / 95.7</td>
    <td><b>56.9</b> / <b>49.2</b></td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>3.6 / 1.6</td>
    <td>3.0 / 0.5</td>
    <td>93.4 / 97.9</td>
    <td>69.9 / 51.9</td>
    <td>85.0 / 94.4</td>
    <td><b>56.9</b> / <b>48.7</b></td>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td>4.3 / 1.2</td>
    <td>0.6 / <b>0.1</b></td>
    <td><b>95.0</b> / <b>98.7</b></td>
    <td>63.6 / 45.5</td>
    <td>84.4 / 93.6</td>
    <td>52.9 / 42.6</td>
  </tr>
  <tr>
    <td>AYA-EXPANSE-32B</td>
    <td>19.8 / 6.7</td>
    <td>7.9 / 1.8</td>
    <td>73.4 / 91.7</td>
    <td>63.8 / 45.6</td>
    <td>84.7 / 91.9</td>
    <td>43.4 / 39.5</td>
  </tr>
  <tr>
    <td>GEMINI-1.5-PRO</td>
    <td><b>0.3*</b> / <b>0.3*</b></td>
    <td><b>0.1*</b> / <b>0.1*</b></td>
    <td>58.2 / 82.9</td>
    <td><b>71.1</b> / 48.7</td>
    <td><b>98.2</b> / 96.5</td>
    <td>40.9 / 46.6</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>33.2 / 11.2</td>
    <td>10.7 / 7.7</td>
    <td>57.2 / 81.3</td>
    <td>64.0 / 46.2</td>
    <td>95.5 / 93.0</td>
    <td>35.6 / 35.7</td>
  </tr>
  <tr>
    <td>LLAMA-3-70B</td>
    <td>76.0 / 70.0</td>
    <td>0.5 / 0.6</td>
    <td>23.5 / 29.5</td>
    <td>66.6 / 46.7</td>
    <td>97.4 / <b>97.5</b></td>
    <td>15.3 / 13.5</td>
  </tr>
</table>

Table 2: Model performance aggregated across languages. Each cell reports values for culturally agnostic / culturally specific portions. Due to language identification errors, we exclude Balochi, Kirundi, Papiamento, and Hiligaynon from the aggregation. Fine-grained metrics are only computed over answers that lack surface-level issues. *GEMINI-1.5-PRO returned API errors for 41.4% (agnostic) / 16.7% (specific) of answers, which likely obscures surface-level errors that it makes.

Figure 2 shows the overall scores for each model by language. Open-weight models are comparable to their closed counterparts on high- and mid-resource languages, with AYA-EXPANSE-32B outperforming CLAUDE-3-OPUS in 8 of these languages on culturally agnostic questions. The closed models significantly outperform the open models on the low-resource languages, scoring mostly 22 – 66 while the open models mostly score below 10. This gap is attributed to surface-level issues, which are present in as high as 70% for LLAMA-3-70B (see Table 2). The exception is GEMINI-1.5-PRO, which throwing API errors when prompted in low-resource languages.

We show specific categories of answer deficiencies detected by our surface-level issue metrics in Table 16, and provide further analysis in Appendix C.2.

4 Human Evaluation

Given the limitations of automatic metrics, we supplement our evaluation with native speaker judgments across five languages: Kirundi, Fijian, Hindi, German, and English.

Evaluation setup: We evaluate CLAUDE-3-OPUS, GPT-4-TURBO, and MIXTRAL-8X22B. For each language we sampled 10 culturally specific and 10 culturally agnostic questions.13

We recruit native speakers via Prolific and Upwork, all of whom participated in the question collection process, paying $7.50 USD per question and an additional $8.00 USD for reviewing the guidelines, totaling $720 USD. Annotators are presented with a question, reference answer (if available), and answers generated by the three models in random order. For each candidate answer, they are asked to: (1) identify whether it is in the correct language, (2) mark minor and major errors,14 (3) evaluate factuality, (4) note significant omissions, (5) comment on the answer’s overall quality (Figure 3), and (6) rate it on a 5-point scale (excellent, good, average, poor, unusable). Finally, annotators rank the three answers from best to worst and provide a free-form explanation for their ranking. We provide details of the workflow in Figure 17 and §D. The study was reviewed by the Institutional Review Board and received a non-human subject determination.

13For culturally specific questions, annotators selected 10 questions they were confident in answering accurately. For culturally agnostic questions, we supplied annotators with bullet-point answers in English.

14This step was included to help the annotators visualize any issues with the answer and encourage them to read the entire answer. Hence, we did not require annotators to classify errors beyond a simple minor vs major distinction.

15We use clmm from the ordinal package (Christensen, 2023) because of the ordinal nature of our response variable (ratings) and repeated measures, with annotators rating each model multiple times for different questions.

4.1 Results of human evaluation

Looking at the overall answer ratings, human annotators prefer GPT-4-TURBO’s answers, followed by CLAUDE-3-OPUS’s and then lastly MIXTRAL-8X22B’s (Figure 4). To confirm, we fit a cumulative link mixed model (clmm()) for predicting ratings from models (Table 18), with annotators nested within language included as a random effect.15 We find that a MIXTRAL-8X22B answer has an 88% chance of having a lower rating than a CLAUDE-3-OPUS answer (\( p<.001 \)) and a 94% chance of having a lower rating than a GPT-4-TURBO answer (\( p<.001 \)). Also, a CLAUDE-3-OPUS answer has a 30% chance of having a lower rating than a GPT-4-TURBO answer (\( p<.001 \)).
<table>
  <tr>
    <th colspan="12">Overall quality score for culturally agnostic questions</th>
    <th colspan="12">Overall quality score for culturally specific questions</th>
  </tr>
  <tr>
    <td>English</td><td>74.5</td><td>74.1</td><td>74.2</td><td>73.0</td><td>71.0</td><td>76.3</td><td>68.5</td><td>62.5</td><td>60.7</td><td>59.6</td><td>57.4</td><td>54.9</td><td>62.8</td><td>57.3</td>
  </tr>
  <tr>
    <td>Spanish</td><td>71.9</td><td>74.2</td><td>68.5</td><td>74.4</td><td>70.2</td><td>66.0</td><td>67.8</td><td>51.5</td><td>52.0</td><td>42.8</td><td>48.4</td><td>44.0</td><td>43.6</td><td>39.8</td>
  </tr>
  <tr>
    <td>German</td><td>73.9</td><td>70.7</td><td>67.9</td><td>71.8</td><td>72.4</td><td>68.8</td><td>54.8</td><td>53.0</td><td>54.8</td><td>45.9</td><td>53.4</td><td>45.4</td><td>50.3</td><td>23.5</td>
  </tr>
  <tr>
    <td>Hindi</td><td>71.7</td><td>70.9</td><td>66.9</td><td>67.5</td><td>74.0</td><td>49.5</td><td>5.4</td><td>66.0</td><td>66.3</td><td>58.9</td><td>54.3</td><td>60.3</td><td>39.3</td><td>6.7</td>
  </tr>
  <tr>
    <td>Russian</td><td>70.3</td><td>69.9</td><td>62.3</td><td>65.9</td><td>71.1</td><td>62.7</td><td>41.8</td><td>53.3</td><td>50.8</td><td>41.6</td><td>47.4</td><td>46.6</td><td>46.0</td><td>22.1</td>
  </tr>
  <tr>
    <td>Arabic</td><td>72.3</td><td>70.2</td><td>65.2</td><td>69.8</td><td>70.3</td><td>59.1</td><td>5.5</td><td>58.3</td><td>56.0</td><td>49.4</td><td>50.5</td><td>49.5</td><td>32.5</td><td>2.0</td>
  </tr>
  <tr>
    <td>Chinese</td><td>70.0</td><td>71.2</td><td>62.5</td><td>68.2</td><td>70.1</td><td>62.1</td><td>0.0</td><td>52.0</td><td>50.5</td><td>43.7</td><td>51.8</td><td>48.1</td><td>48.3</td><td>0.0</td>
  </tr>
  <tr>
    <td>Hungarian</td><td>71.1</td><td>70.1</td><td>65.0</td><td>51.8</td><td>73.3</td><td>62.7</td><td>42.9</td><td>45.0</td><td>47.6</td><td>40.1</td><td>34.2</td><td>41.5</td><td>37.4</td><td>10.2</td>
  </tr>
  <tr>
    <td>Japanese</td><td>69.0</td><td>70.4</td><td>65.5</td><td>69.2</td><td>70.7</td><td>66.0</td><td>0.0</td><td>47.0</td><td>50.2</td><td>40.4</td><td>44.6</td><td>36.5</td><td>34.8</td><td>0.4</td>
  </tr>
  <tr>
    <td>Korean</td><td>71.7</td><td>73.3</td><td>67.7</td><td>71.4</td><td>66.4</td><td>62.2</td><td>1.5</td><td>46.7</td><td>44.9</td><td>42.6</td><td>39.5</td><td>38.3</td><td>32.3</td><td>0.4</td>
  </tr>
  <tr>
    <td>Hebrew</td><td>67.8</td><td>67.1</td><td>64.0</td><td>67.7</td><td>64.7</td><td>37.1</td><td>1.6</td><td>48.9</td><td>48.3</td><td>38.9</td><td>46.8</td><td>39.7</td><td>27.9</td><td>1.7</td>
  </tr>
  <tr>
    <td>Faroese</td><td>66.2</td><td>66.0</td><td>64.3</td><td>0.0</td><td>0.0</td><td>2.4</td><td>0.0</td><td>54.7</td><td>51.5</td><td>48.6</td><td>8.9</td><td>0.0</td><td>9.4</td><td>6.3</td>
  </tr>
  <tr>
    <td>Samoan</td><td>57.4</td><td>55.2</td><td>49.9</td><td>15.2</td><td>0.0</td><td>0.0</td><td>1.8</td><td>40.7</td><td>38.0</td><td>41.4</td><td>9.1</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Pashto</td><td>66.1</td><td>55.7</td><td>54.6</td><td>27.1</td><td>0.0</td><td>2.0</td><td>0.0</td><td>29.0</td><td>28.4</td><td>25.8</td><td>9.9</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Tswana</td><td>47.1</td><td>53.4</td><td>42.5</td><td>14.2</td><td>0.0</td><td>0.0</td><td>0.0</td><td>23.6</td><td>21.8</td><td>15.9</td><td>4.7</td><td>0.0</td><td>0.0</td><td>0.7</td>
  </tr>
  <tr>
    <td>Tongan</td><td>21.9</td><td>33.9</td><td>29.8</td><td>6.9</td><td>0.0</td><td>0.0</td><td>0.0</td><td>26.1</td><td>33.7</td><td>20.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Fijian</td><td>29.7</td><td>30.0</td><td>26.3</td><td>7.5</td><td>3.3</td><td>0.0</td><td>0.0</td><td>26.2</td><td>18.5</td><td>18.5</td><td>4.9</td><td>1.4</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Wolof</td><td>7.3</td><td>4.3</td><td>9.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>6.4</td><td>7.7</td><td>9.7</td><td>2.2</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Afar</td><td>2.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
</table>

Figure 2: Answer scores S based on our quality criteria: surface issues, factuality and relevance. The left heatmap shows the results for culturally agnostic questions while the right heatmap shows the results for culturally specific questions. Closed- and open-weight models perform comparably on high- to mid- resource languages, while open-weight models are much worse on low-resource languages. Scores degrade on culturally specific questions due to factual imprecision (see Figure 10).

Answer ratings are lower for culturally specific questions: Figure 4 suggests that LLMs generate worse answers for culturally specific questions than for culturally agnostic questions. To check this, we fit a cumulative link mixed model for predicting ratings from question type (Table 20), with annotators nested within language included as a random effect. We see that an answer to a culturally agnostic question has a 67% chance of having a higher rating than an answer to a culturally specific question (\( p<0.001 \)). CLAude-3.0Pus’s performance drop on culturally specific questions is notable: its answer to a culturally specific question has an 80% chance of receiving a lower rating compared to a culturally agnostic question (\( p<0.001 \)).

Factuality and omission issues are strong predictors of answer rating: To determine which variables of this experiment (e.g., model, question type, factuality issues, omissions) correlate with answer rating, we fit cumulative link mixed models for predicting the rating, with each variable being used as the sole predictor of a separate model. Table 23 shows R^2 values of these models. We observe high marginal R^2 for the factuality issues model (R^2=0.560) and the omissions model (R^2=0.740), indicating that these factors are strong predictors of answer rating.\footnote{In case of mixed effects models, *marginal* R^2 refers to the proportion of variance explained by the fixed effects (predictors) alone.}

4.2 Analyzing annotator comments

We analyze annotators’ comments to gain insights into answer quality. For each comment field, we iteratively develop and apply an annotation schema, linking the results to the corresponding ratings and scores. (See Appendix §D for details.)

Factuality errors are more frequent for culturally specific answers: All 12 issues regarding incorrect dates, entity and events (e.g., “It is mentioned that Nifty was launched in 1995 but it was actually launched in 1996.”) occur in culturally specific answers, likely due to a greater prevalence of dates, entities and events in culturally specific questions about topics like HISTORY than culturally agnostic topics like HEALTH AND WELLNESS.

GPT-4-Turbo answers rank first due to content.
We analyze the reasons mentioned for ranking each model’s answers as best. Having good CONTENT (e.g. due to being complete; see Table 28 for a description) is a reason for GPT-4-Turbo being chosen as best answer 51% of the time (e.g. “*Answer 1 (GPT-4-Turbo) is the perfect answer and
<table>
  <tr>
    <th>Question</th>
    <th>Generated Answer</th>
    <th>Model</th>
    <th>Comment</th>
  </tr>
  <tr>
    <td>Kirundi<br><i>Why was it forbidden for kids to eat at their neighbors’ house at the new year?</i></td>
    <td>Mu mucu nyarwanda, Kubera iki cazira ko umwana arya mugahinga ku mwaka mushasha? zikurikira! Mugahinga in ibyito by’abakuru: Mu nyarwanda, mugahinga inifwa nkibyoro by’abakuru kubera ko arri byino bikomeye.(…)</td>
    <td>Claude 3 Opus</td>
    <td>The answer is not answering the question. It talks about something unrelated to the question asked. It’s really off-topic. It is also written in Kinyarwanda and talks about Rwanda culture, which is beyond the scope of the question. The first three words, “Mu mucu nyarwanda…” are translated as “In Rwandan culture…”, which shows that it is purely about Rwanda, not Burundi.</td>
  </tr>
  <tr>
    <td>Hindi<br><i>जनसंख्या और ओबीसी में क्या अंतर हैं?</i></td>
    <td>General (G) और Obstetrics and Gynecology (OB/GYN) दोनों जन-जन अंक के निर्धारण में और उनके बीच एक अंतर है।<br>विवरण: General (जनसंख्या) व्यापक रूप से और बीमारियों के लिए जाना जाता है, जबकि OB/GYN (विषयवस्तु) को लेकर कोई व्यापक बीमारियों और गर्भावस्था में संबंधित बहावलियों के लिए प्रयुक्त होता है।(…)</td>
    <td>Mixtral 8x22B</td>
    <td>The entire answer is wrong. The question is to state the difference between General and OBC category of people (based on caste and society status), but the answer is about population and gynecology (doctors). It is irrelevant to the question. The answer is written partially in Hindi and English. It is not clear. It is fluent. It sounds artificial.</td>
  </tr>
  <tr>
    <td>German<br><i>Was ist Überhangmandat? Geglückt, aber nichts passendes gefunden. Ist zu schwer erklärt.</i></td>
    <td>Ein Überhangmandat ist ein Begriff aus dem Wahlrecht, der vor allem in Verhältnissystemen mit Direktkandidaten eine Rolle spielt, wie zum Beispiel in Deutschland bei Bundestagswahlen. Ich versuche es einfach zu erklären: (…)</td>
    <td>GPT-4 Turbo</td>
    <td>Very helpful, has a good mixture of theory and a well explained example. Complete, clear and fluent. Definitely it would go for “human-like”.</td>
  </tr>
</table>

Figure 3: Examples of comments on LLM-generated answers written by human annotators.

![Bar chart showing distribution of human ratings of answers by model and question type](page_109_670_1267_312.png)

Figure 4: Distribution of human ratings of answers by model and question type. Each model generates 50 answers per question type. Humans give higher ratings for culturally agnostic answers, especially for CLAUDE-3-OPUS.

and explains all the points needed to understand how to play the game ‘Teen Patti’.”). In the culturally agnostic setting, where CLAUDE-3-OPUS and GPT-4-TURBO perform comparably, more GPT-4-TURBO wins (48%) are attributed to CONTENT than CLAUDE-3-OPUS wins (32%). The full result can be found in §D (Table 24).

5 Related Work

Cultural & Multilingual NLP: Cultural knowledge has been explored through the creation of knowledge bases (Fung et al., 2024; Nguyen et al., 2022) as well as datasets for tasks like probing (Keleg and Magdy, 2023; Yin et al., 2022; Zhou et al., 2024), short-form QA and visual QA. Short-form QA work for multilingual cultural knowledge includes MMLU (Hendrycks et al., 2020) translations or MMLU-style datasets (Singh et al., 2024; Lai et al., 2023; Kim et al., 2024; Koto et al., 2024a), common sense datasets (Myung et al., 2025; Wibowo et al., 2023; Koto et al., 2024b), and evaluations (Shen et al., 2024). Visual long-form QA (LVQA) is less explored and mostly monolingual (Yu et al., 2024; Alwajih et al., 2024), but the contemporaneous work Vayani et al. (2024) looks at LVQA in 100 languages. We are not aware of any textual LFQA datasets of cultural knowledge.

Some multilingual cultural works rely on translation for their multilinguality (Singh et al., 2024), potentially limiting their coverage of cultural concepts. Surveys (Adilazuarda et al., 2024; Liu et al., 2024) call out a lack of multilingual datasets that cover a diverse set of cultural concepts. Our work seeks to make progress in this gap of the literature.

Evaluation of Long-Form QA: Evaluating long-form QA (LFQA) remains challenging. Lexical metrics of text generation like ROUGE (Lin, 2004b) and some neural-based metrics like BERTScore (Zhang et al., 2019) and BLEURT (Sellam et al., 2020) show poor correlation with human ratings (Krishna et al., 2021; Xu et al., 2023; Cambazoglu et al., 2021; Chen et al., 2019). For most other model-based evaluations (Zheng et al., 2023; Fu et al., 2023; Zhong et al., 2022), correlation with human annotations is measured for tasks like instruction-following, summarization and machine
translation but mostly not LFQA. Jiang et al. (2023) assess effectiveness of metrics for LFQA, however this is done only on GPT-4-created data.

6 Conclusion & Future Work

We introduce CALMQA, the first textual multilingual long-form QA dataset, which contains 51.7K culturally specific questions across 23 high- and low-resource languages. Our evaluation of seven state-of-the-art LLMs reveals that culturally specific questions are more difficult for models than culturally agnostic ones, evidenced by lower factuality and human ratings. Furthermore, we observe critical surface-level issues (wrong language, repetition) in all models, especially for low-resource languages. Our results stress the importance of diversifying pre- and post-training datasets to emphasize cultural knowledge acquisition, which can help improve culturally specific QA. Also, improving cross-lingual transfer to address data scarcity may help for underrepresented languages like Afar.

Limitations

While we strive to cover as many aspects of the cultures represented in CALMQA as possible, we acknowledge that it is not feasible to encompass every cultural nuance. Additionally, for low-resource languages, we employed workers to manually write questions, which impacts scalability. Finally, our culturally agnostic questions are translations from English performed by GPT-4-TURBO, and thus may not match the quality of human translations.

It would be ideal to have identical distributions of topics across language and type (culturally specific vs culturally agnostic). However, topics like religion, food & drinks, history and literature, among many others, are naturally bound to the culture, making it impossible to have similar distributions for culturally specific and culturally agnostic questions. Moreover, such topics may have different relative significance for different cultures. Consequently, collecting questions representative of the topics important to people conflicts with having identical distributions between languages. Nevertheless, we found that the topic distribution is similar between languages.

Our automatic evaluation relies on surface-level measures such as language detection and token repetitions. While this approach allows us to determine that current LLMs still struggle with producing outputs in the correct language and without repetitions, it does not assess the fluency or completeness of outputs that lack these surface-level issues. This underscores the need for comprehensive metrics to evaluate overall answer quality in multilingual LFQA, which we leave to future work.

We assess factuality of model generated answers by translating them into English and extracting verifiable claims and validating them against evidence retrieved through web searches. However, this evaluation is influenced by three factors: (1) the quality of translation, (2) the quantity of extracted claims and (3) the availability of relevant online evidence. Our relevance evaluation also depends on the quality of translation. While we do not observe any evident issues with our pipelines during data inspection, it is possible then these factors influenced the results.

Our human evaluation uses 100 questions across 5 languages to demonstrate that models struggle to generate well-written, factual, and complete answers in non-English languages. Large-scale human evaluation is time-consuming and prohibitively expensive, and finding workers proficient in low-resource languages presented a significant challenge, constraining our evaluation efforts. However, we have shown that we can statistically justify various insights about LLM multilingual capabilities with our scale of data.

Ethical consideration

The protocols for data collection and human evaluation described in this paper were reviewed and deemed exempt by the Institutional Review Board. All annotators provided informed consent for the use and publication of their annotations and collected questions. They were compensated fairly for their work, with their preferred rates respected for both the question collection and evaluation tasks.

Acknowledgments

We extend our gratitude to the annotators from Prolific and Upwork for their hard work and for sharing their expertise about their culture. This project was supported by NSF grant RI-2312948.

References

Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Alham Fikri Aji, Jacki O’Neill, Ashutosh Modi, and Monojit Choudhury. 2024. Towards measuring and
modeling "culture" in llms: A survey. Preprint, arXiv:2403.15412.

Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Julia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. ArXiv, abs/2305.13707.

AI@Meta. 2024. Llama 3 model card.

Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404–12422, Bangkok, Thailand. Association for Computational Linguistics.

Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. 2024. Medexpqqa: Multilingual benchmarking of large language models for medical question answering. Preprint, arXiv:2404.05590.

Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. 2024. Peacock: A family of Arabic multimodal large language models and benchmarks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12753–12776, Bangkok, Thailand. Association for Computational Linguistics.

Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. Technical report, Anthropic. Accessed: 2024-05-23.

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.

Berkant Barla Cambazoglu, Valeriia Bolotova-Baranova, Falk Scholer, Mark Sanderson, Leila Tavakoli, and W. Bruce Croft. 2021. Quantifying human-perceived answer utility in non-factoid question answering. Proceedings of the 2021 Conference on Human Information Interaction and Retrieval.

Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Evaluating question answering evaluation. In Conference on Empirical Methods in Natural Language Processing.

Rune H. B. Christensen. 2023. ordinal—Regression Models for Ordinal Data. R package version 2023.12-4.

Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Preprint, arXiv:2003.05002.

John Dang, Shivalka Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosst, Marzieh Fadaee, Beyza Ermiş, Ahmet Üstün, and Sara Hooker. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261.

Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024. Towards measuring the representation of subjective global opinions in language models. Preprint, arXiv:2306.16388.

Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELi5: Long form question answering. Preprint, arXiv:1907.09190.

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. In North American Chapter of the Association for Computational Linguistics.

Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. 2024. Massively multi-cultural knowledge acquisition & lm benchmarking. Preprint, arXiv:2402.09369.

Jingsong Gao, Qingren Zhou, and Rui Qiu. 2021. ELI5-Category: a categorized open-domain qa dataset.

Gemini Team. 2024. Gemini 1.5: Unlocking multi-modal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.

Nuno M. Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André Martins. 2023. Hallucinations in large multilingual translation models. Transactions of the Association for Computational Linguistics, 11:1500–1517.

Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. 2023. Are large language model-based evaluators the solution to scaling up multilingual evaluation? In Findings.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. ArXiv, abs/2009.03300.
Xinyu Hu, Li Lin, Mingqi Gao, Xunjian Yin, and Xiaojun Wan. 2024. Themis: A reference-free nlg evaluation language model with flexibility and interpretability. In Conference on Empirical Methods in Natural Language Processing.

its5Q. 2022. its5q/yandex-q - datasets at hugging face.

Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhui Chen. 2023. Tigerscore: Towards building explainable metric for all text generation tasks. Trans. Mach. Learn. Res., 2024.

Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. Is chatgpt a good translator? yes with gpt-4 as the engine. Preprint, arXiv:2301.08745.

Haoqiang Kang, Terra Blevins, and Luke Zettlemoyer. 2024. Comparing hallucination detection metrics for multilingual generation. Preprint, arXiv:2402.10496.

Amr Keleg and Walid Magdy. 2023. Dlama: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models. Preprint, arXiv:2306.05076.

Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. 2024. Click: A benchmark dataset of cultural and linguistic intelligence in korean. ArXiv, abs/2403.06412.

Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Evaluating the efficacy of summarization evaluation across languages. ArXiv, abs/2106.01478.

Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024a. Arabicmmlu: Assessing massive multitask language understanding in arabic. ArXiv, abs/2402.12840.

Fajri Koto, Rahmad Mahendra, Nurul Aisyah, and Timothy Baldwin. 2024b. Indoculture: Exploring geographically-influenced cultural commonsense reasoning across eleven indonesian provinces. ArXiv, abs/2404.01854.

Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. Preprint, arXiv:2103.06332.

Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Demontercourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. Preprint, arXiv:2307.16039.

Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315–7330, Online. Association for Computational Linguistics.

Huihan Li, Liwei Jiang, Nouha Dziri, Xiang Ren, and Yejin Choi. 2024. Culture-gen: Revealing global cultural perception in language models through natural language prompting. Preprint, arXiv:2404.10199.

Chin-Yew Lin. 2004a. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Chin-Yew Lin. 2004b. Rouge: A package for automatic evaluation of summaries. In Annual Meeting of the Association for Computational Linguistics.

Chen Cecilia Liu, Iryna Gurevych, and Anna Korhonen. 2024. Culturally aware and adapted nlp: A taxonomy and a survey of the state of the art.

Jiahua Liu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2019. XQA: A cross-lingual open-domain question answering dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2358–2368, Florence, Italy. Association for Computational Linguistics.

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. Preprint, arXiv:2305.14251.

Mistral AI. 2024. Cheaper, Better, Faster, Stronger. Technical report, Mistral AI. Accessed: 2024-06-19.

Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Víctor Gutiérrez-Basulto, Yazmín Ibáñez-García, Hwaran Lee, Shamsuddien Hassan Muhammad, Kiwoong Park, Anar Sabuhri Razeyev, Nina White, Seid Muhie Yimam, Mohammad Taher Pilehvar, Nedjima Ousidhoum, Jose Camacho-Collados, and Alice Oh. 2025. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. Preprint, arXiv:2406.09948.

Tarek Naous, Michael Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics.

Tuan-Phong Nguyen, Simon Razniewski, Aparna S. Varde, and Gerhard Weikum. 2022. Extracting cultural commonsense knowledge at scale. Proceedings of the ACM Web Conference 2023.
OpenAI. 2024a. Gpt-4 technical report. Preprint, arXiv:2303.08774.

OpenAI. 2024b. Model release blog: GPT-4o. Technical report, OpenAI. Accessed: 2024-05-23.

Shramay Palta and Rachel Rudinger. 2023. FORK: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9952–9962, Toronto, Canada. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Yifu Qiu, Yftah Ziser, Anna Korhonen, E. Ponti, and Shay B. Cohen. 2023. Detecting and mitigating hallucinations in multilingual summarisation. ArXiv, abs/2305.13632.

Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Annual Meeting of the Association for Computational Linguistics.

Sheikh Shafayat, Eunsu Kim, Juhyun Oh, and Alice Oh. 2024. Multi-fact: Assessing factuality of multilingual llms using factscore. Preprint, arXiv:2402.18045.

Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Soujanya Poria, and Rada Mihalcea. 2024. Understanding the capabilities and limitations of large language models for cultural commonsense. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5668–5680, Mexico City, Mexico. Association for Computational Linguistics.

Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limknochotip, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, and Sara Hooker. 2024. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation. Preprint, arXiv:2412.03304.

Yixiao Song, Yekyung Kim, and Mohit Iyyer. 2024. Veriscore: Evaluating the factuality of verifiable claims in long-form text generation. Preprint, arXiv:2406.19276.

Yan Tao, Olga Viberg, Ryan S. Baker, and Rene F. Kizilcec. 2024. Cultural bias and cultural alignment of large language models. Preprint, arXiv:2311.14096.

Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2022. Label Studio: Data labeling software. Open source software available from https://github.com/heartexlabs/label-studio.

Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, Henok Biadgling Adentew, Yahya Hmaiti, Amandeep Kumar, Kartik Kuckreja, Mykola Maslych, Wafa Al Ghallabi, Mihail Mihaylov, Chao Qin, Abdelrahman M Shaker, Mike Zhang, Mahardika Krisna Ihsani, Amiel Esplana, Monil Gokani, Shachar Mirkin, Harsh Singh, Ashay Srivastava, Endre Hamerlik, Fatihahn Asma Izzati, Fadillah Adamsyah Maani, Sebastian Cavad, Jenny Chim, Rohit Gupta, Sanjay Manjunath, Kamila Zhumakhanova, Feno Heriniaina Rabevohtira, Azril Amirudin, Muhammad Ridzuan, Daniya Kareem, Ketan More, Kunyang Li, Pramesh Shakya, Muhammad Saad, Amirpouya Ghasemaghaei, Amirbek Djanibekov, Dilshod Azizov, Branislava Jankovic, Naman Bhatia, Alvaro Cabrera, Johan Obando-Ceron, Olympiah Otieno, Fabian Fastrem, Muztoba Rabbani, Sanoojan Baliah, Santosh Sanjeev, Abduragim Shtanchaev, Maheen Fatima, Thao Nguyen, Amrin Kareem, Toluwani Aremu, Nathan Xavier, Amit Bhatkal, Hawau Toyin, Aman Chadha, Hisham Cholakkal, Rao Muhammad Anwer, Michael Felsberg, Jorma Laaksonen, Thamar Solorio, Monojit Choudhury, Ivan Laptev, Mubarak Shah, Salman Khan, and Fahad Khan. 2024. All languages matter: Evaluating llms on culturally diverse 100 languages. Preprint, arXiv:2411.16508.

Ray Wang. 2023. wangrui6/zhihu-kol - datasets at hugging face.

Haryo Akbarianto Wibowo, Erland Hilman Fuadi, Made Nindytama Nityasya, Radityo Eko Prasojo, and Alham Fikri Aji. 2023. Copal-id: Indonesian language reasoning with local culture and nuances. ArXiv, abs/2311.01012.

Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. Preprint, arXiv:2305.18201.

Jianhao Yan, Pingchuan Yan, Yulong Chen, Judy Li, Xianchao Zhu, and Yue Zhang. 2024. Gpt-4 vs. human translators: A comprehensive evaluation of translation quality across languages, domains, and expertise levels. Preprint, arXiv:2407.03658.

Da Yin, Hritik Bansal, Masoud Monajatipoor, Lui-nian Harold Li, and Kai-Wei Chang. 2022. GeoMLAMA: Geo-diverse commonsense probing on multilingual pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039–2055, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Weihao Yu, Zhengyuany Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. Mm-vet: Evaluating large multimodal models for integrated capabilities. Preprint, arXiv:2308.02490.

Chen Zhang, L. F. D’Haro, Chengguang Tang, Ke Shi, Guohua Tang, and Haizhou Li. 2023. xdial-eval: A multilingual open-domain dialogue evaluation benchmark. In Conference on Empirical Methods in Natural Language Processing.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.

Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Peng Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi-dimensional evaluator for text generation. In Conference on Empirical Methods in Natural Language Processing.

Li Zhou, Taelin Karidi, Nicolas Garneau, Yong Cao, Wanlong Liu, Wenyu Chen, and Daniel Hershcovitch. 2024. Does mapo tofu contain coffee? probing llms for food-related cultural knowledge. ArXiv, abs/2404.06833.

Ethical Considerations

The protocols for data collection and human evaluation described in this paper were reviewed and deemed exempt by the Institutional Review Board. All annotators provided informed consent for the use and publication of their annotations and collected questions. They were compensated fairly for their work, with their preferred rates respected for both the question collection and evaluation tasks.

A Data Collection

This appendix provides extra details about the data collection process for CALMQA. §A.1 describes the identification of websites used for data collection. §A.2 outlines the data collection methods for high- and mid-resource languages, and §A.3 details the data collection process for low-resource languages. Table 9 contains example entries from the dataset. Table 4 and Table 10 provide more details on the number of questions and languages included in the dataset.

A.1 Website Survey

We conducted a survey to find websites with non-English cultural questions. The instructions outlined the survey’s goal, defined a good website, and specified what constitutes a culturally specific question. Our criteria for a good website included:

• At least 500 answered “good” questions (as defined below). Websites could contain other questions as we could filter them out.

• Most questions and answers should be in a non-English language.

• Questions should cover a diverse range of topics, not just one or two broad areas (e.g., fashion, technology).

• The website should contain culturally specific questions not found on English websites or in English QA datasets.

• The website should have a large community of contributors with many questions answered.

The survey evolved through an iterative process of piloting and refining based on the results.

Survey participants were English-proficient crowdworkers on the Prolific platform (https://www.prolific.com), whose native language was not English. The survey took about 10 minutes to complete, and we paid $10 for valid responses, totaling $510. We considered a response valid if it showed a good-faith effort, even if the website was of insufficient quality or duplicated in another response. From 51 responses, we obtained 4 websites used for question collection. Some websites were rejected despite having good questions because the proportion of good to bad questions was too low for feasible collection. Remaining websites were identified by the authors. See Table 3 for the full list of websites employed.

A.2 High- and Mid-Resource Culturally Specific Questions

Culturally specific questions in high-resource languages were collected by workers on the Prolific17 platform from the websites in Table 3. All crowdworkers were English-proficient with their native language matching the language of their allocated websites. Each collector was required to read guidelines, pass a guidelines understanding test and complete a test pilot of 5 questions in order to qualify

17https://www.prolific.com/
<table>
  <tr>
    <th>WEBSITE</th>
    <th>LANGUAGE</th>
    <th>ISO</th>
    <th># Q</th>
    <th>URL</th>
  </tr>
  <tr>
    <td>Ejaba</td>
    <td>Arabic</td>
    <td>ar</td>
    <td>29</td>
    <td><a href="https://www.ejaba.com/">https://www.ejaba.com/</a></td>
  </tr>
  <tr>
    <td>Ujeeb</td>
    <td>Arabic</td>
    <td>ar</td>
    <td>56</td>
    <td><a href="https://ujeeb.com/">https://ujeeb.com/</a></td>
  </tr>
  <tr>
    <td>Zhihu</td>
    <td>Chinese</td>
    <td>zh</td>
    <td>75</td>
    <td><a href="https://www.zhihu.com/">https://www.zhihu.com/</a></td>
  </tr>
  <tr>
    <td>Reddit ELI5</td>
    <td>English</td>
    <td>en</td>
    <td>78</td>
    <td><a href="https://www.reddit.com/r/explainlikeimfive/">https://www.reddit.com/r/explainlikeimfive/</a></td>
  </tr>
  <tr>
    <td>Gutefrage</td>
    <td>German</td>
    <td>de</td>
    <td>96</td>
    <td><a href="https://www.gutefrage.net/">https://www.gutefrage.net/</a></td>
  </tr>
  <tr>
    <td>Quora</td>
    <td>Hebrew</td>
    <td>he</td>
    <td>96</td>
    <td><a href="https://he.quora.com">https://he.quora.com</a></td>
  </tr>
  <tr>
    <td>Let’s Diskuss</td>
    <td>Hindi</td>
    <td>hi</td>
    <td>91</td>
    <td><a href="https://hi.letsdiskuss.com/">https://hi.letsdiskuss.com/</a></td>
  </tr>
  <tr>
    <td>Gyakori kérdések</td>
    <td>Hungarian</td>
    <td>hu</td>
    <td>75</td>
    <td><a href="https://www.gyakorikerdesek.hu/">https://www.gyakorikerdesek.hu/</a></td>
  </tr>
  <tr>
    <td>Yahoo Japan</td>
    <td>Japanese</td>
    <td>ja</td>
    <td>17</td>
    <td><a href="https://chiebukuro.yahoo.co.jp/">https://chiebukuro.yahoo.co.jp/</a></td>
  </tr>
  <tr>
    <td>OKWave</td>
    <td>Japanese</td>
    <td>ja</td>
    <td>58</td>
    <td><a href="https://okwave.jp/">https://okwave.jp/</a></td>
  </tr>
  <tr>
    <td>Naver</td>
    <td>Korean</td>
    <td>ko</td>
    <td>75</td>
    <td><a href="https://kin.naver.com/qna/">https://kin.naver.com/qna/</a></td>
  </tr>
  <tr>
    <td>Yandex</td>
    <td>Russian</td>
    <td>ru</td>
    <td>75</td>
    <td><a href="https://yandex.ru/q/">https://yandex.ru/q/</a></td>
  </tr>
  <tr>
    <td>Todoexpertos</td>
    <td>Spanish</td>
    <td>es</td>
    <td>102</td>
    <td><a href="https://www.todoexpertos.com/">https://www.todoexpertos.com/</a></td>
  </tr>
</table>

Table 3: Websites from which cultural questions were obtained, with the number of questions retrieved by website. Multiple websites were used for a given language if workers were struggling with a given website.

for the main task. This protocol was reviewed by the Institutional Review Board. Overall, our process yielded 923 questions across 11 languages with answers at a cost of $1427 USD.

Guidelines We provided a guidelines slideshow detailing the rules for selecting questions. The main rules for questions where:

1. The question should require long answer.
2. The question should be culture specific.
3. A native speaker would ask this [question].
4. The question should be objective.
5. Questions should not need pictures/links.

Guidelines Understanding Test Our guidelines understanding test consisted of a form consisting of 11 multiple-selection multiple-choice graded questions. The first question assessed question was “Which of these are listed as important rules for questions in the guidelines? (you should select all correct answers)”, which required showing understanding of long-form culturally specific information-seeking questions. The remaining 10 questions were curated examples of questions that each may or may not have had issues. Test takers were required to select all the reasons why a question was not suitable according to the guidelines, or select that the question was suitable. We reviewed test results manually, and accordingly chose which workers to pass. We provided passing workers with the test answers, so that they could learn from their mistakes. We paid workers $3.33 USD for completing the test.

Main Collection Task We asked workers to provide examples of culturally specific questions and real users’ answers from the identified websites. We manually reviewed all provided examples, using Google Translate to get English translations of website content. In cases where we deemed that an example did not meet our guidelines, we provided feedback and the worker either clarified how their example met the guidelines or replaced the example. For the final dataset, we used GPT-4-TURBO with the prompt in Table 7 instead of Google Translate to obtain the English translations of questions. We translated answers using GPT-4o, which was released after we had conducted our human eval-
<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="3">Culturally Specific</th>
    <th colspan="3">Culturally Agnostic</th>
  </tr>
  <tr>
    <th># Q</th>
    <th>Q. Bytes (AVG/STD)</th>
    <th>A. Bytes (AVG/STD)</th>
    <th># Q</th>
    <th>Q. Bytes (AVG/STD)</th>
    <th>A. Bytes (AVG/STD)</th>
  </tr>
  <tr>
    <th colspan="7">High- & Mid-Resource Languages</th>
  </tr>
  <tr>
    <td>English</td>
    <td>78</td>
    <td>275.7 / 189.0</td>
    <td>674.1 / 475.9</td>
    <td>51</td>
    <td>67.1 / 31.7</td>
    <td>632.3 / 636.9</td>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>85</td>
    <td>74.3 / 61.3</td>
    <td>2105.0 / 2378.6</td>
    <td>51</td>
    <td>108.7 / 56.4</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>75</td>
    <td>193.4 / 1329.5</td>
    <td>588.8 / 939.7</td>
    <td>51</td>
    <td>68.1 / 31.4</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>German</td>
    <td>96</td>
    <td>304.6 / 227.4</td>
    <td>1169.0 / 744.7</td>
    <td>51</td>
    <td>82.2 / 39.8</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Hebrew</td>
    <td>96</td>
    <td>142.5 / 84.2</td>
    <td>2043.6 / 1934.9</td>
    <td>51</td>
    <td>93.0 / 42.9</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>91</td>
    <td>122.4 / 52.8</td>
    <td>3618.8 / 1867.1</td>
    <td>51</td>
    <td>184.2 / 90.3</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Hungarian</td>
    <td>75</td>
    <td>301.1 / 279.8</td>
    <td>379.3 / 333.2</td>
    <td>51</td>
    <td>82.3 / 38.2</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>75</td>
    <td>512.0 / 359.3</td>
    <td>920.6 / 637.1</td>
    <td>51</td>
    <td>104.3 / 50.6</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>75</td>
    <td>126.3 / 138.7</td>
    <td>1008.6 / 936.3</td>
    <td>51</td>
    <td>93.0 / 43.3</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>75</td>
    <td>310.3 / 438.3</td>
    <td>4546.7 / 5067.9</td>
    <td>51</td>
    <td>134.6 / 70.8</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>102</td>
    <td>429.9 / 271.1</td>
    <td>852.0 / 817.9</td>
    <td>51</td>
    <td>83.6 / 36.1</td>
    <td>N/A</td>
  </tr>
  <tr>
    <th colspan="7">Low-resource languages</th>
  </tr>
  <tr>
    <td>Afar</td>
    <td>25</td>
    <td>43.7 / 16.5</td>
    <td>N/A</td>
    <td>51</td>
    <td>81.1 / 39.8</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Balochi</td>
    <td>65</td>
    <td>122.7 / 52.4</td>
    <td>N/A</td>
    <td>51</td>
    <td>96.1 / 48.5</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Faroese</td>
    <td>30</td>
    <td>47.8 / 16.6</td>
    <td>N/A</td>
    <td>51</td>
    <td>75.1 / 34.5</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Fijian</td>
    <td>75</td>
    <td>75.0 / 36.9</td>
    <td>N/A</td>
    <td>51</td>
    <td>92.5 / 40.6</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Hiligaynon</td>
    <td>65</td>
    <td>93.4 / 39.1</td>
    <td>N/A</td>
    <td>51</td>
    <td>83.6 / 39.7</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Kirundi</td>
    <td>53</td>
    <td>64.6 / 21.2</td>
    <td>557.2 / 160.9</td>
    <td>51</td>
    <td>88.2 / 43.1</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Papiamento</td>
    <td>10</td>
    <td>66.8 / 28.5</td>
    <td>N/A</td>
    <td>51</td>
    <td>74.1 / 35.3</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Pashto</td>
    <td>75</td>
    <td>64.8 / 26.9</td>
    <td>N/A</td>
    <td>51</td>
    <td>118.1 / 55.6</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Samoan</td>
    <td>25</td>
    <td>51.2 / 19.3</td>
    <td>N/A</td>
    <td>51</td>
    <td>80.5 / 37.6</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Tongan</td>
    <td>10</td>
    <td>81.2 / 19.2</td>
    <td>N/A</td>
    <td>51</td>
    <td>102.4 / 47.9</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Tswana</td>
    <td>65</td>
    <td>87.2 / 43.4</td>
    <td>N/A</td>
    <td>51</td>
    <td>88.8 / 43.4</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Wolof</td>
    <td>50</td>
    <td>45.3 / 18.9</td>
    <td>N/A</td>
    <td>51</td>
    <td>78.2 / 44.1</td>
    <td>N/A</td>
  </tr>
</table>

Table 4: Combined data statistics for culturally specific and culturally agnostic questions. For each language, we report the number of questions (# Q), average and standard deviation of question bytes (Q. Bytes) and answer bytes (A. Bytes) in UTF-8 encoding. Answer bytes for culturally agnostic questions are not available, and are marked as N/A.

uation, with the prompt in Table 8. We paid the workers $0.65-1.33 USD per question.

A.3 Low-Resource Culturally Specific Questions

Questions for low-resource languages were collected by hiring native speakers proficient in English through Upwork. They were paid $0.65 to $1.00 USD per submitted question with its English translation. Annotators were required to read the guidelines and complete a short comprehension task, for which they were paid $7 USD. Additionally, answers to all Kirundi questions were paid $2 USD per answer. This protocol was reviewed by the Institutional Review Board.

Annotators were instructed to write up to 25 questions in their native language along with English translations, ensuring the questions met the following criteria:

• The question requires a long-form answer (at least 3-4 sentences).
• The question is culturally specific, meaning it is more likely to be asked in the region where the language is spoken.
• The question is something a native speaker of the language might ask.
• The question has an objective answer (i.e., not based on opinions).

B Question Categorization

In this section we describe the process of categorizing all questions into predefined set of categories.

Method We selected 25 random culturally specific questions from the dataset. We manually created a list of broad categories with descriptions and examples, and then 2 authors independently applied the categorization on the 25 questions. We reviewed disagreements and accordingly refined the categories. Then we used GPT-4o to categorize using the prompts in Table 11, with temperature set to
You are to evaluate whether a given question is *culturally specific* to [language]. A question is considered *culturally specific* to a language if it meets both of the following conditions:
1. The question is written in [language].
2. **At least one** of the following applies:
- The question refers to topics, concepts, objects, entities, or events that are unique to one or a few cultures associated with speakers of [language].
- The answer to the question may vary depending on the cultural or regional context—meaning that the answer would be specific to the [language] culture and may differ when considered in the context of other cultures.

Culturally specific topics may include, but are not limited to, areas such as heritage, current events, history, religion, literature, society, travel, cuisine, fauna, economy, education, law, and the arts. **Note:** A question being written in [language] does not automatically make it culturally specific.

**Example 1 (Culturally Specific to English):**
- *Question:* "Why does Eminem give so much praise to Dr Dre? He talks about him like a father in his music."
- *Explanation:* This question refers to Eminem and Dr Dre, who are prominent figures in the American music industry—a distinct aspect of American (and related Western) culture. The question’s cultural context makes it culturally specific to English.

**Example 2 (Culturally Specific to English):**
- *Question:* "How do you obtain a gun license?"
- *Explanation:* The process for obtaining a gun license differs significantly between countries. In the USA, the process is well-established and culturally significant, while it may differ or be non-existent in other countries. Thus, the question is culturally specific to English when referring to American cultural context.

**Example 3 (Not Culturally Specific to English):**
- *Question:* "Do ships cause the ocean to be higher than it normally would be?"
- *Explanation:* This question is general and universal; its answer does not depend on any cultural or regional context. Therefore, it is not culturally specific.

—

**Task:**
Assess whether the following question is culturally specific to [language] or not. Provide your explanation in English, wrapping it in `<explanation></explanation>` tags. Then, output `<result>PASS</result>` if the question is culturally specific, or `<result>FAIL</result>` if it is not. Your response should contain **only** these two tags and nothing else.

**Here is the question to assess:**
<question>
[question]
</question>

Table 5: Prompt used with GPT-4o-Mini to filter questions collected from community QA websites for culturally specific questions. Strings in the form [form] are placeholders that are replaced at runtime.
You are evaluating questions in a dataset for quality. Your task is to determine whether a given question meets the following quality criteria:

1. **Language:** The question is written in [language].
2. **Long-form Answer:** The question cannot be answered with just a short phrase or entity; it requires a few sentences to answer.
3. **Not Asking For Answerer’s Opinions:** The question does not ask for opinions, personal experiences, perspectives or recommendations of the **answerer**. A question that exhibits bias or implies a particular view point of the **asker** does **not** violate this criterion.
4. **Public Information:** Answering the question does not require access to non-public information.
5. **Privacy:** The question does not contain any personally identifiable information (e.g., name, username, phone number, or home address).

**Example 1 (Satisfies Criteria):**
- *Question:* "Why is Norton hated so much? What makes an antivirus/antimalware program good or bad anyway?"
- *Explanation:* 1. The question is in English. 2. The question requires a explanation comprising of multiple sentences. 3. The question does not ask for an opinion, even though it indicates a negative viewpoint towards Norton. 4. Answering the question does not require access to non-public information. 5. The question does not contain any personally identifiable information. The question meets all the criteria and so is satisfactory.

**Example 2 (Does Not Satisfy Criteria):**
- *Question:* "How would you suggest I revise mathematics before my first economics class?"
- *Explanation:* The question is explicitly asking for a recommendation and so does not meet the quality criteria.

__

**Task:** Assess whether the following question satisfies all of the quality criteria listed above. Provide a detailed explanation of your assessment in English, wrapped in `<explanation></explanation>` tags. Then, output `<result>PASS</result>` if the question satisfies the quality criteria, or `<result>FAIL</result>` if it does not. Do not output anything outside of the `<explanation></explanation>` and `<result></result>` tags.

**Here is the question to assess:**
<question>
[question]
</question>

Table 6: Prompt used with GPT-4o-MINI to filter questions collected from community QA websites based on general quality criteria. Strings in the form [form] are placeholders that are replaced at runtime.

Your task is to translate a question from [language] into English. You will be given the [language] answer as the context.

Here is the [language] answer. Use it as the context to make the translation sound natural in the English: [answer]

Translate the following question from [language] into English. Make it sound as natural as possible: [question]

Table 7: Prompt used with GPT-4-TURBO to translate non-English questions into English. Strings in the form [form] are placeholders that are replaced at runtime.
Your task is to translate the answer of a [language] question from [language] into English. You will be given the [language] question as the context.

Here is the [language] question. Use it as the context to make the translation sound natural in the English: [question]

Translate the following answer from [language] into English. Make it sound as natural as possible: [answer]

Table 8: Prompt used with GPT-4o to translate non-English answers into English. Strings in the form [form] are placeholders that are replaced at runtime.

0.0. After minor clarifications to category descriptions, we found that GPT-4o produced adequate categories for all 25 questions. We consequently used the model to categorize all of CaLMQA. Our final categories, with descriptions and examples, can be found in Table 12.

Results Figure 5 shows the number of questions by category and language. We observe that one of RELIGION, BELIEFS, CUSTOMS, AND TRADITIONS, GOVERNANCE AND SOCIETY and HISTORY is the top category of almost every language (the exceptions being English and Korean). Furthermore, RELIGION, BELIEFS, CUSTOMS, AND TRADITIONS is the predominantly the top category for low-resource languages (10/12). This difference is likely due to the question collection process for low-resource languages.

To compare the distribution of categories between languages, we compute pairwise Bhattacharyya coefficients between the data from the languages (Figure 6). The Bhattacharyya coefficient ranges from 0 to 1 with a higher number meaning similar distributions. We see generally high coefficients, indicating that the category distributions are similar between languages.

C Automatic Evaluation

In this section of the appendix we present the details of automatic evaluation. All evaluated models are listed in Table 15. Examples of the model tendencies detected by automatic evaluation are in Table 16.

C.1 Method Details

Language accuracy Figure 13 displays the percentage of responses each model generated in the correct language, independent of correctness or fluency of the answer. We used polyglot (https://pypi.org/project/polyglot/) and langid (https://pypi.org/project/py3langid/) for language identification, choosing them based on their performance for specific languages. This identification was also applied to the questions to estimate its performance across languages. Our pipeline accurately recognized 100% of instances in 14 languages. For other languages, accuracy typically remained above 90%, with Fijian at 98.67%, Russian at 97.33%, Tongan at 96.92%, Samoan at 92.00%, and Wolof at 90% (see Table 13). However, identification accuracy for Kirundi was notably lower at 35.85%, as the libraries frequently misclassified it as the closely related Kinyarwanda. The automatic identification process failed entirely for Balochi, Hiligaynon, and Papiamento, which is reflected in seemingly low performance for these languages across all the models.

Repetitions Figure 14 illustrates the percentage of responses affected by repetitions, analyzed by language across different models. To identify these repetitions, we employed tiktoken (https://github.com/openai/tiktoken) with the o200_base encoding. We specifically identified instances where at least 20 consecutive tokens were repeated at least four times within an answer.

Claim extraction and verification pipeline We first translated the answers into English with GPT-4o. Then we extract claims using a finetuned Mistral 7B model and use them to query Serper API for evidence. Then we prompt a finetuned Mistral 7B model for verification. Both models were introduced in Song et al. (2024). The pipeline is visualized in Figure 7.

We report the mean claim count by model, language of the question, and question type in Figure 8. We exclude all answers with surface-level issues as
<table>
  <tr>
    <th>FIELD</th>
    <th>CONTENT</th>
  </tr>
  <tr>
    <td>Language</td>
    <td>English</td>
  </tr>
  <tr>
    <td>Question (Original)</td>
    <td>Why does the President of the United States need to be born in the United States to be eligible to run? It seems like the country that a person was born in has little to do with their abilities to lead.</td>
  </tr>
  <tr>
    <td>Question (English)</td>
    <td>Why does the President of the United States need to be born in the United States to be eligible to run? It seems like the country that a person was born in has little to do with their abilities to lead.</td>
  </tr>
  <tr>
    <td>Culturally Specific</td>
    <td>True</td>
  </tr>
  <tr>
    <td>Answer</td>
    <td>It’s not a matter of their leadership ability, it’s intended to guarantee loyalty. The idea is that most people are loyal to their birth country, so you’d want someone born in the US to be the one acting as commander in chief of the US armed forces, among other things. May not be perfect logic, but it’s tradition at this point. Nobody has made a big enough deal of it accompanied by a strong enough argument to get it changed.</td>
  </tr>
  <tr>
    <td>Language</td>
    <td>Hungarian</td>
  </tr>
  <tr>
    <td>Question (Original)</td>
    <td>Hogyan lehet kikeverni a Horthy-kori sisakok színét?</td>
  </tr>
  <tr>
    <td>Question (English)</td>
    <td>How can you mix the color of helmets from the Horthy era? (translated by GPT-4-TURBO)</td>
  </tr>
  <tr>
    <td>Culturally Specific</td>
    <td>True</td>
  </tr>
  <tr>
    <td>Answer</td>
    <td>M35 tábori sisak. Neten rákeresve találsz róla képeket. Nem kell megijedni a sok árnyalattól, annak idején sem volt tökéletesen egységes.<br>Ez egy zöldesbarna szín. Talán a RAL6025 áll a legközelebb hozzá. Festékboltban kikeverik géppel. Nem drágább, mint külön megvenni hozzá egy egy dobozzal a festéket és kevergetni. Ecsettél festették az eredetit.</td>
  </tr>
  <tr>
    <td>Language</td>
    <td>Fijian</td>
  </tr>
  <tr>
    <td>Question (Original)</td>
    <td>Na cava na vuna era vinakata kina na Nasi ni veiyanuyanu mera sa lesi i Viti Levu?</td>
  </tr>
  <tr>
    <td>Question (English)</td>
    <td>What is the reason the nurses from the outer islands want to be assigned to Viti Levu?</td>
  </tr>
  <tr>
    <td>Culturally Specific</td>
    <td>True</td>
  </tr>
  <tr>
    <td>Answer</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Language</td>
    <td>Kirundi</td>
  </tr>
  <tr>
    <td>Question (Original)</td>
    <td>Ni kubera iki twama dukeneye gushira ama aprikasiyo ku gihe? Hoba iki iyo tutabikoze?</td>
  </tr>
  <tr>
    <td>Question (English)</td>
    <td>Why do we need to constantly do software updates? What happens if I don’t?</td>
  </tr>
  <tr>
    <td>Culturally Specific</td>
    <td>False</td>
  </tr>
  <tr>
    <td>Answer</td>
    <td>N/A</td>
  </tr>
</table>

Table 9: Examples of entries in CALMQA. Metadata like questions source (specific website or annotator) are omitted here for simplicity.
<table>
  <tr>
    <th>LANGUAGE</th>
    <th>ISO</th>
    <th>FAMILY</th>
    <th>Branch</th>
    <th>MORPHOLOGY</th>
    <th>ORDER</th>
    <th>SCRIPT</th>
    <th>Region</th>
    <th>SPEAKERS</th>
  </tr>
  <tr>
    <th colspan="9">High- & Mid-Resource</th>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>ar</td>
    <td>Afro-Asiatic</td>
    <td>Semitic</td>
    <td>fusional</td>
    <td>SVO</td>
    <td>Arabic alphabet</td>
    <td>Arab world</td>
    <td>720M</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>zh</td>
    <td>Sino-Tibetan</td>
    <td>Sinitic</td>
    <td>analytic</td>
    <td>SVO</td>
    <td>Hanzi</td>
    <td>Mainland China, Taiwan, Singapore</td>
    <td>1.38B</td>
  </tr>
  <tr>
    <td>English</td>
    <td>en</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>analytic</td>
    <td>SVO</td>
    <td>Latin</td>
    <td>World-wide</td>
    <td>1.5B</td>
  </tr>
  <tr>
    <td>German</td>
    <td>de</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>fusional</td>
    <td>SVO</td>
    <td>Latin</td>
    <td>Germany, Austria, Switzerland, etc.</td>
    <td>133M</td>
  </tr>
  <tr>
    <td>Hebrew</td>
    <td>he</td>
    <td>Afro-Asiatic</td>
    <td>Semitic</td>
    <td>fusional</td>
    <td>SVO</td>
    <td>Hebrew script</td>
    <td>Israel</td>
    <td>9.3M</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>hi</td>
    <td>Indo-European</td>
    <td>Indo-Iranian</td>
    <td>fusional</td>
    <td>SOV</td>
    <td>Devanagari</td>
    <td>India</td>
    <td>610M</td>
  </tr>
  <tr>
    <td>Hungarian</td>
    <td>hu</td>
    <td>Uralic</td>
    <td>Finnno-Ugric</td>
    <td>agglutinative</td>
    <td>SVO</td>
    <td>Latin</td>
    <td>Hungary</td>
    <td>13M</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>ja</td>
    <td>Japonic</td>
    <td>Japanese</td>
    <td>agglutinative</td>
    <td>SOV</td>
    <td>Kanji, Kana</td>
    <td>Japan</td>
    <td>123M</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>ko</td>
    <td>Koreanic</td>
    <td>Korean</td>
    <td>agglutinative</td>
    <td>SOV</td>
    <td>Hangul</td>
    <td>Korea</td>
    <td>82M</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>ru</td>
    <td>Indo-European</td>
    <td>Balto-Slavic</td>
    <td>fusional</td>
    <td>SVO</td>
    <td>Cyrillic</td>
    <td>Russia, Russian-speaking world</td>
    <td>255M</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>es</td>
    <td>Indo-European</td>
    <td>Italic</td>
    <td>fusional</td>
    <td>SVO</td>
    <td>Latin</td>
    <td>Spain, Central and South Americas, the US</td>
    <td>559M</td>
  </tr>
  <tr>
    <th colspan="9">Low-Resource</th>
  </tr>
  <tr>
    <td>Afar</td>
    <td>aa</td>
    <td>Afro-Asiatic</td>
    <td>Cushitic</td>
    <td>agglutinative</td>
    <td>SOV</td>
    <td>Latin</td>
    <td>Ethiopia, Djibouti, Eritrea</td>
    <td>2.6M</td>
  </tr>
  <tr>
    <td>Balochi</td>
    <td>bal</td>
    <td>Indo-European</td>
    <td>Indo-Iranian</td>
    <td>agglutinative</td>
    <td>SOV</td>
    <td>Balochi Standard Alphabet</td>
    <td>Pakistan, Iran, Afghanistan</td>
    <td>8.8M</td>
  </tr>
  <tr>
    <td>Faroese</td>
    <td>fo</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>fusional</td>
    <td>SVO</td>
    <td>Latin</td>
    <td>Faroe Islands, Denmark</td>
    <td>69K</td>
  </tr>
  <tr>
    <td>Fijian</td>
    <td>fj</td>
    <td>Austronesian</td>
    <td>Malayo-Polynesian</td>
    <td>agglutinative</td>
    <td>VOS</td>
    <td>Latin</td>
    <td>Fiji</td>
    <td>640K</td>
  </tr>
  <tr>
    <td>Hiligaynon</td>
    <td>hil</td>
    <td>Austronesian</td>
    <td>Malayo-Polynesian</td>
    <td>analytic</td>
    <td>VSO</td>
    <td>Latin</td>
    <td>Philippines</td>
    <td>9.1M</td>
  </tr>
  <tr>
    <td>Kirundi</td>
    <td>rn</td>
    <td>Niger-Kongo</td>
    <td>Atlantic–Congo</td>
    <td>agglutinative</td>
    <td>SVO</td>
    <td>Latin</td>
    <td>Burundi</td>
    <td>12-13M</td>
  </tr>
  <tr>
    <td>Papiamento</td>
    <td>pap</td>
    <td>Portuguese-based creole</td>
    <td>Afro-Portuguese</td>
    <td>analytic</td>
    <td>SVO</td>
    <td>Latin</td>
    <td>Aruba, Curaçao, Bonaire</td>
    <td>300K</td>
  </tr>
  <tr>
    <td>Pashto</td>
    <td>ps</td>
    <td>Indo-European</td>
    <td>Indo-Iranian</td>
    <td>fusional</td>
    <td>SOV</td>
    <td>Pashto alphabet</td>
    <td>Afghanistan, Pakistan and Iran</td>
    <td>58.8M</td>
  </tr>
  <tr>
    <td>Samoan</td>
    <td>sm</td>
    <td>Austronesian</td>
    <td>Malayo-Polynesian</td>
    <td>analytic</td>
    <td>VSO</td>
    <td>Latin</td>
    <td>Samoa</td>
    <td>510K</td>
  </tr>
  <tr>
    <td>Tongan</td>
    <td>to</td>
    <td>Austronesian</td>
    <td>Polynesian</td>
    <td>agglutinative</td>
    <td>VSO</td>
    <td>Latin</td>
    <td>Tonga</td>
    <td>187K</td>
  </tr>
  <tr>
    <td>Tswana</td>
    <td>tn</td>
    <td>Niger-Kongo</td>
    <td>Atlantic–Congo</td>
    <td>agglutinative</td>
    <td>SVO</td>
    <td>Latin</td>
    <td>Botswana, South Africa, Zimbabwe</td>
    <td>13.9M</td>
  </tr>
  <tr>
    <td>Wolof</td>
    <td>wo</td>
    <td>Niger-Kongo</td>
    <td>Atlantic–Congo</td>
    <td>agglutinative</td>
    <td>SVO</td>
    <td>Latin primarily</td>
    <td>Senegal</td>
    <td>12.3M</td>
  </tr>
</table>

Table 10: Linguistic and usage information of the languages in the CALMQA dataset
<table>
  <tr>
    <th>LANGUAGE</th>
    <th>PROMPT</th>
  </tr>
  <tr>
    <td>English</td>
    <td>You are categorizing questions about different cultures into specific categories. Your task is to assign one category to each question. Here are the available categories:

education and career – <i>[Education and Career Description]</i>. Example: <i>[Education and Career Example]</i><br>
[categories]…<br>
health and wellness – <i>[Health and Wellness Description]</i>. Example: <i>[Health and Wellness Example]</i><br>
other – <i>[Other Description]</i>

Here is the question to categorize:
<original_question>{question}</original_question>

Categorize this question into one of the categories. Output your choice in the following format:
<category>category name</category>

Your choice:</td>
  </tr>
  <tr>
    <td>All except English</td>
    <td>You are categorizing questions about different cultures into specific categories. Your task is to assign one category to each question. Here are the available categories:

education and career – <i>[Education and Career Description]</i>. Example: <i>[Education and Career Example]</i><br>
[categories]…<br>
health and wellness – <i>[Health and Wellness Description]</i>. Example: <i>[Health and Wellness Example]</i><br>
other – <i>[Other Description]</i>

Here is the question to categorize:
<original_question>{question}</original_question>
<translation>{translation}</translation>

Categorize this question into one of the categories. Output your choice in the following format:
<category>category name</category>

Your choice:</td>
  </tr>
</table>

Table 11: Prompts used with GPT-4o to categorize questions. Strings in the form [form] are placeholders that are replaced at runtime. The categories used are in Table 12.
<table>
  <tr>
    <th>CATEGORY</th>
    <th>DESCRIPTION</th>
    <th>EXAMPLE</th>
  </tr>
  <tr>
    <td>EDUCATION AND CAREER</td>
    <td>Questions related to school, education system, jobs and career paths. Includes developing new skills for new jobs.</td>
    <td><i>Why do young children drop out of school?</i></td>
  </tr>
  <tr>
    <td>GOVERNANCE AND SOCIETY</td>
    <td>Questions about laws, governance and policies, as well as politics and social issues.</td>
    <td><i>What are the reasons why Japan cannot have casinos?</i></td>
  </tr>
  <tr>
    <td>GEOGRAPHY, TOURISM, AND CLIMATE</td>
    <td>Questions concerning the geography, climatic conditions, environmental factors of a region, tourism and travelling.</td>
    <td><i>What is the significance of Gorée Island?</i></td>
  </tr>
  <tr>
    <td>TECHNOLOGY</td>
    <td>Questions about the technology, technological advancements, uses of technology and digital innovation.</td>
    <td><i>Are stores that accept VISA debit cards marked VISA? Or is it a store with a VISA PLUS mark?</i></td>
  </tr>
  <tr>
    <td>ECONOMY AND INDUSTRY</td>
    <td>Questions regarding modern-day economic practices, key industries, trade, and economic development.</td>
    <td><i>Why is our country not developing like others?</i></td>
  </tr>
  <tr>
    <td>MEDIA AND ENTERTAINMENT</td>
    <td>Questions about the media and entertainment specific to the region.</td>
    <td><i>Why are trademarks obscured on broadcast?</i></td>
  </tr>
  <tr>
    <td>FOOD AND DRINKS</td>
    <td>Questions related to culinary traditions, typical foods and beverages, preparation methods, culinary practices, and cultural significance of meals.</td>
    <td><i>Why is bread with sausage called a sandwich in Russia?</i></td>
  </tr>
  <tr>
    <td>HISTORY</td>
    <td>Questions about historical events, significant figures, and important periods that have shaped a culture.</td>
    <td><i>How did the Bujumbura market burn?</i></td>
  </tr>
  <tr>
    <td>LANGUAGE, ART AND LITERATURE</td>
    <td>Questions about the language, dialects, as well as art forms and literary works/traditions.</td>
    <td><i>Why is the Balochi Language categorized into its three main dialects?</i></td>
  </tr>
  <tr>
    <td>RELIGION, BELIEFS, CUSTOMS, AND TRADITIONS</td>
    <td>Questions regarding religious practices, beliefs, rituals, customs, traditions, and holiday.</td>
    <td><i>In the Islamic religious teaching, what's the meaning of the seven tens?</i></td>
  </tr>
  <tr>
    <td>HEALTH AND WELLNESS</td>
    <td>Questions related to traditional and modern health practices, public health issues, and well-being.</td>
    <td><i>Why methadone? What makes it "better" than other opioids for maintenance therapy or tapering off another drug?</i></td>
  </tr>
  <tr>
    <td>OTHER</td>
    <td>Questions that do not fit neatly into the above categories.</td>
    <td></td>
  </tr>
</table>

Table 12: Categories of questions in CALMQA.
<table>
  <tr>
    <th></th>
    <th>Economy And Industry</th>
    <th>Education And Career</th>
    <th>Food And Drinks</th>
    <th>Geography, Tourism, And Climate</th>
    <th>Governance And Society</th>
    <th>Health And Wellness</th>
    <th>History</th>
    <th>Language, Art And Literature</th>
    <th>Media And Entertainment</th>
    <th>Religion, Beliefs, Customs, And Traditions</th>
    <th>Technology</th>
    <th>Other</th>
  </tr>
  <tr>
    <td>Afar</td>
    <td>1.00</td>
    <td>0.92</td>
    <td>0.92</td>
    <td>0.58</td>
    <td>0.46</td>
    <td>0.84</td>
    <td>0.87</td>
    <td>0.63</td>
    <td>0.69</td>
    <td>0.92</td>
    <td>0.77</td>
    <td>0.52</td>
    <td>0.51</td>
    <td>0.86</td>
    <td>0.67</td>
    <td>0.87</td>
    <td>0.88</td>
    <td>0.80</td>
    <td>0.90</td>
    <td>0.51</td>
    <td>0.79</td>
    <td>0.88</td>
    <td>0.99</td>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>1.00</td>
    <td>0.94</td>
    <td>0.76</td>
    <td>0.59</td>
    <td>0.85</td>
    <td>0.88</td>
    <td>0.80</td>
    <td>0.82</td>
    <td>0.86</td>
    <td>0.83</td>
    <td>0.71</td>
    <td>0.72</td>
    <td>0.80</td>
    <td>0.79</td>
    <td>0.88</td>
    <td>0.92</td>
    <td>0.84</td>
    <td>0.72</td>
    <td>0.80</td>
    <td>0.86</td>
    <td>0.94</td>
    <td>0.84</td>
    <td>0.84</td>
    <td>0.84</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>Balochi</td>
    <td>1.00</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>1.00</td>
    <td>0.88</td>
    <td>0.83</td>
    <td>0.73</td>
    <td>0.93</td>
    <td>0.90</td>
    <td>0.90</td>
    <td>0.92</td>
    <td>0.92</td>
    <td>0.72</td>
    <td>0.82</td>
    <td>0.70</td>
    <td>0.69</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.92</td>
    <td>0.90</td>
    <td>0.85</td>
    <td>0.89</td>
    <td>0.92</td>
    <td>0.89</td>
    <td>0.92</td>
    <td>0.89</td>
    <td>0.92</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>English</td>
    <td>1.00</td>
    <td>0.67</td>
    <td>0.63</td>
    <td>0.89</td>
    <td>0.82</td>
    <td>0.53</td>
    <td>0.86</td>
    <td>0.83</td>
    <td>0.84</td>
    <td>0.61</td>
    <td>0.81</td>
    <td>0.61</td>
    <td>0.53</td>
    <td>0.73</td>
    <td>0.54</td>
    <td>0.75</td>
    <td>0.34</td>
    <td>0.56</td>
    <td>0.46</td>
    <td>0.56</td>
    <td>0.46</td>
    <td>0.56</td>
    <td>0.46</td>
    <td>0.56</td>
    <td>0.46</td>
  </tr>
  <tr>
    <td>Faroese</td>
    <td>1.00</td>
    <td>0.83</td>
    <td>0.82</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.83</td>
    <td>0.66</td>
    <td>0.77</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>Fijian</td>
    <td>1.00</td>
    <td>0.88</td>
    <td>0.84</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.78</td>
    <td>0.74</td>
    <td>0.70</td>
    <td>0.61</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>German</td>
    <td>1.00</td>
    <td>0.88</td>
    <td>0.84</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.78</td>
    <td>0.74</td>
    <td>0.70</td>
    <td>0.61</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Hebrew</td>
    <td>1.00</td>
    <td>0.92</td>
    <td>0.70</td>
    <td>0.69</td>
    <td>0.53</td>
    <td>0.78</td>
    <td>0.63</td>
    <td>0.68</td>
    <td>0.87</td>
    <td>0.61</td>
    <td>0.76</td>
    <td>0.55</td>
    <td>0.67</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Hiligaynon</td>
    <td>1.00</td>
    <td>0.81</td>
    <td>0.60</td>
    <td>0.61</td>
    <td>0.90</td>
    <td>0.71</td>
    <td>0.93</td>
    <td>0.84</td>
    <td>0.81</td>
    <td>0.61</td>
    <td>0.81</td>
    <td>0.61</td>
    <td>0.81</td>
    <td>0.61</td>
    <td>0.81</td>
    <td>0.61</td>
    <td>0.81</td>
    <td>0.61</td>
    <td>0.81</td>
    <td>0.61</td>
    <td>0.81</td>
    <td>0.61</td>
    <td>0.81</td>
    <td>0.61</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>1.00</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.78</td>
    <td>0.74</td>
    <td>0.70</td>
    <td>0.61</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Hungarian</td>
    <td>1.00</td>
    <td>0.92</td>
    <td>0.70</td>
    <td>0.69</td>
    <td>0.53</td>
    <td>0.78</td>
    <td>0.63</td>
    <td>0.68</td>
    <td>0.87</td>
    <td>0.61</td>
    <td>0.76</td>
    <td>0.55</td>
    <td>0.67</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>1.00</td>
    <td>0.92</td>
    <td>0.70</td>
    <td>0.69</td>
    <td>0.53</td>
    <td>0.78</td>
    <td>0.63</td>
    <td>0.68</td>
    <td>0.87</td>
    <td>0.61</td>
    <td>0.76</td>
    <td>0.55</td>
    <td>0.67</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>Kirundi</td>
    <td>1.00</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.78</td>
    <td>0.74</td>
    <td>0.70</td>
    <td>0.61</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>1.00</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.78</td>
    <td>0.74</td>
    <td>0.70</td>
    <td>0.61</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Papiamento</td>
    <td>1.00</td>
    <td>0.71</td>
    <td>0.80</td>
    <td>0.97</td>
    <td>0.67</td>
    <td>0.42</td>
    <td>0.36</td>
    <td>0.84</td>
    <td>1.00</td>
    <td>0.81</td>
    <td>0.78</td>
    <td>0.54</td>
    <td>0.72</td>
    <td>0.76</td>
    <td>0.92</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Pashto</td>
    <td>1.00</td>
    <td>0.81</td>
    <td>0.78</td>
    <td>0.54</td>
    <td>0.72</td>
    <td>0.76</td>
    <td>0.92</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>1.00</td>
    <td>0.79</td>
    <td>0.80</td>
    <td>0.61</td>
    <td>0.85</td>
    <td>0.82</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Samoan</td>
    <td>1.00</td>
    <td>0.72</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>1.00</td>
    <td>0.85</td>
    <td>0.59</td>
    <td>0.52</td>
    <td>1.00</td>
    <td>0.72</td>
    <td>0.81</td>
    <td>1.00</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td>Tongan</td>
    <td>1.00</td>
    <td>0.72</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>Tswana</td>
    <td>1.00</td>
    <td>0.72</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>Wolof</td>
    <td>1.00</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
  </tr>
</table>

Figure 5: Number of human-collected questions by category and language.

![Heatmap of human-collected questions by category and language](page_104_104_1247_484.png)

Figure 6: Bhattacharyya coefficients of the category distributions, pairwise between languages. The Bhattacharyya coefficient ranges from 0 to 1, with a higher number meaning more similar distributions.

![Bhattacharyya coefficient matrix](page_104_624_1247_484.png)
<table>
  <tr>
    <th>LANGUAGE</th>
    <th>LANG ID</th>
    <th>ACCURACY (%)</th>
  </tr>
  <tr><td>Afar</td><td>aa</td><td>100.00</td></tr>
  <tr><td>Arabic</td><td>ar</td><td>100.00</td></tr>
  <tr><td>Balochi</td><td>bal</td><td>0.00</td></tr>
  <tr><td>Chinese</td><td>zh</td><td>100.00</td></tr>
  <tr><td>English</td><td>en</td><td>100.00</td></tr>
  <tr><td>Faroese</td><td>fo</td><td>100.00</td></tr>
  <tr><td>Fijian</td><td>fj</td><td>98.67</td></tr>
  <tr><td>German</td><td>de</td><td>100.00</td></tr>
  <tr><td>Hebrew</td><td>he</td><td>100.00</td></tr>
  <tr><td>Hiligaynon</td><td>hil</td><td>0.00</td></tr>
  <tr><td>Hindi</td><td>hi</td><td>100.00</td></tr>
  <tr><td>Hungarian</td><td>hu</td><td>100.00</td></tr>
  <tr><td>Japanese</td><td>ja</td><td>100.00</td></tr>
  <tr><td>Kirundi</td><td>rn</td><td>35.85</td></tr>
  <tr><td>Korean</td><td>ko</td><td>100.00</td></tr>
  <tr><td>Papiamento</td><td>pap</td><td>0.00</td></tr>
  <tr><td>Pashto</td><td>ps</td><td>100.00</td></tr>
  <tr><td>Russian</td><td>ru</td><td>97.33</td></tr>
  <tr><td>Samoan</td><td>sm</td><td>92.00</td></tr>
  <tr><td>Spanish</td><td>es</td><td>100.00</td></tr>
  <tr><td>Tongan</td><td>to</td><td>100.00</td></tr>
  <tr><td>Tswana</td><td>tn</td><td>96.92</td></tr>
  <tr><td>Wolof</td><td>wo</td><td>90.00</td></tr>
</table>

Table 13: Accuracy of the language detection pipeline on the test set made from questions in the given language. Note that the language detection libraries are often more accurate on longer texts (i.e., texts longer than the length of a single question).

well as languages for which the model produced less than 50% of valid answers (i.e., answers without identified surface level issues).

C.2 Further Analysis

Model surface-level errors  We further analyzed the responses for specific textual indicators. Detected patterns in model responses are presented with examples in Table 16.

Our textual analysis demonstrates issues in Mixtral-8x22B responses for low-resource languages. 31.47% of Mixtral-8x22B responses to questions in low-resource languages contain phrases like “sorry”, “apologize” or “understand” (e.g., “I’m sorry for any confusion, but it seems you’re using a language that I’m not currently able to understand or translate.”). Mixtral-8x22B responses to questions in high-resource languages do not contain these apology-related keywords, revealing an inability to answer the question specifically in low-resource languages. The apologetic textual markers were seen in less than 1% of other models’ responses except for Llama-3-70B’s, where they were present in 14.74% of low-resource and 10.48% of high-resource language answers.

Textual indicators also uncover deficiencies in Llama-3-70B responses. Notably, 37.87% of responses from Llama-3-70B explicitly mention the English name of the language (e.g., “I see you’re speaking in Balochi!”), indicating that although the system recognizes the language of the question, it nonetheless responds in English. This is in contrast to Mixtral-8x22B, which does so in 7.21% of responses, GPT-4-Turbo at 1.84%, and less than 1% for other models. Additionally, approximately 19.71% of Llama-3-70B responses include terms like “translate” or “translation” (e.g., “I apologize, but I’m having trouble understanding your question. Could you please rephrase or translate your question into a language I can understand, such as English?”), where the system either declines to answer (with or without apology), requests an English translation, or provides a translation itself. In comparison, 8.43% of Mixtral-8x22B responses
Figure 7: Claim extraction and verification pipeline. Example showing extraction and verification of claims for a question and answer in Japanese. English translations were obtained with GPT-4o. Only part of the answer is provided for readability.

<table>
  <tr>
    <th>English</th>
    <th>Spanish</th>
    <th>Russian</th>
    <th>Chinese</th>
    <th>German</th>
    <th>Hungarian</th>
    <th>Hebrew</th>
    <th>Arabic</th>
    <th>Hindi</th>
    <th>Korean</th>
    <th>Japanese</th>
    <th>Pashto</th>
    <th>Faroese</th>
    <th>Samoan</th>
    <th>Tswana</th>
    <th>Fijian</th>
    <th>Tongan</th>
    <th>Wolof</th>
    <th>Afar</th>
  </tr>
  <tr>
    <td>39.6</td>
    <td>34.9</td>
    <td>27.9</td>
    <td>30.0</td>
    <td>39.9</td>
    <td>28.1</td>
    <td>24.4</td>
    <td>25.0</td>
    <td>21.1</td>
    <td>26.8</td>
    <td>24.5</td>
    <td>20.4</td>
    <td>24.5</td>
    <td>23.1</td>
    <td>22.8</td>
    <td>23.2</td>
    <td>17.7</td>
    <td>18.6</td>
    <td>1.8</td>
  </tr>
  <tr>
    <td>34.3</td>
    <td>30.7</td>
    <td>25.4</td>
    <td>29.9</td>
    <td>28.1</td>
    <td>25.0</td>
    <td>23.9</td>
    <td>23.1</td>
    <td>20.4</td>
    <td>23.7</td>
    <td>22.5</td>
    <td>20.7</td>
    <td>22.3</td>
    <td>20.0</td>
    <td>18.4</td>
    <td>17.3</td>
    <td>15.7</td>
    <td>11.4</td>
    <td>1.8</td>
  </tr>
  <tr>
    <td>36.5</td>
    <td>35.0</td>
    <td>34.4</td>
    <td>35.6</td>
    <td>35.8</td>
    <td>25.6</td>
    <td>33.0</td>
    <td>31.3</td>
    <td>31.0</td>
    <td>30.4</td>
    <td>27.5</td>
    <td>13.0</td>
    <td>16.7</td>
    <td>7.0</td>
    <td>7.6</td>
    <td>6.3</td>
    <td>7.3</td>
    <td>7.7</td>
    <td>1.8</td>
  </tr>
  <tr>
    <td>24.3</td>
    <td>21.4</td>
    <td>21.3</td>
    <td>24.6</td>
    <td>19.4</td>
    <td>18.9</td>
    <td>16.9</td>
    <td>16.9</td>
    <td>16.4</td>
    <td>18.1</td>
    <td>17.9</td>
    <td>13.0</td>
    <td>16.7</td>
    <td>10.8</td>
    <td>10.6</td>
    <td>10.3</td>
    <td>9.9</td>
    <td>7.7</td>
    <td>1.8</td>
  </tr>
  <tr>
    <td>3.1</td>
    <td>2.8</td>
    <td>2.5</td>
    <td>2.6</td>
    <td>1.9</td>
    <td>2.4</td>
    <td>2.1</td>
    <td>2.3</td>
    <td>2.5</td>
    <td>2.1</td>
    <td>2.1</td>
    <td>1.4</td>
    <td>1.6</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>0.9</td>
    <td>0.7</td>
    <td>0.8</td>
  </tr>
  <tr>
    <td>22.7</td>
    <td>17.0</td>
    <td>14.1</td>
    <td>17.9</td>
    <td>15.9</td>
    <td>13.3</td>
    <td>10.0</td>
    <td>13.2</td>
    <td>10.5</td>
    <td>14.5</td>
    <td>13.0</td>
    <td>6.4</td>
    <td>18.3</td>
    <td>9.6</td>
    <td>10.8</td>
    <td>8.1</td>
    <td>9.0</td>
    <td>10.0</td>
    <td>1.8</td>
  </tr>
</table>

<table>
  <tr>
    <th>English</th>
    <th>Spanish</th>
    <th>Russian</th>
    <th>Chinese</th>
    <th>German</th>
    <th>Hungarian</th>
    <th>Hebrew</th>
    <th>Arabic</th>
    <th>Hindi</th>
    <th>Korean</th>
    <th>Japanese</th>
    <th>Pashto</th>
    <th>Faroese</th>
    <th>Samoan</th>
    <th>Tswana</th>
    <th>Fijian</th>
    <th>Tongan</th>
    <th>Wolof</th>
    <th>Afar</th>
  </tr>
  <tr>
    <td>37.0</td>
    <td>21.4</td>
    <td>21.9</td>
    <td>26.8</td>
    <td>20.1</td>
    <td>16.5</td>
    <td>22.4</td>
    <td>20.5</td>
    <td>17.9</td>
    <td>19.0</td>
    <td>16.9</td>
    <td>20.5</td>
    <td>22.7</td>
    <td>21.5</td>
    <td>22.4</td>
    <td>21.5</td>
    <td>20.8</td>
    <td>20.0</td>
    <td>20.0</td>
  </tr>
  <tr>
    <td>34.1</td>
    <td>20.3</td>
    <td>20.8</td>
    <td>26.1</td>
    <td>18.7</td>
    <td>14.5</td>
    <td>22.4</td>
    <td>19.5</td>
    <td>17.4</td>
    <td>16.3</td>
    <td>16.7</td>
    <td>17.9</td>
    <td>17.1</td>
    <td>16.9</td>
    <td>16.9</td>
    <td>14.0</td>
    <td>18.3</td>
    <td>13.7</td>
    <td>1.8</td>
  </tr>
  <tr>
    <td>33.0</td>
    <td>20.7</td>
    <td>26.5</td>
    <td>29.5</td>
    <td>22.3</td>
    <td>15.7</td>
    <td>29.2</td>
    <td>24.2</td>
    <td>24.0</td>
    <td>19.0</td>
    <td>17.7</td>
    <td>10.6</td>
    <td>18.3</td>
    <td>9.9</td>
    <td>6.9</td>
    <td>6.4</td>
    <td>9.0</td>
    <td>10.0</td>
    <td>1.8</td>
  </tr>
  <tr>
    <td>21.1</td>
    <td>12.8</td>
    <td>16.6</td>
    <td>19.9</td>
    <td>12.7</td>
    <td>10.9</td>
    <td>14.5</td>
    <td>14.5</td>
    <td>14.2</td>
    <td>15.3</td>
    <td>11.7</td>
    <td>14.6</td>
    <td>18.3</td>
    <td>9.6</td>
    <td>10.8</td>
    <td>8.1</td>
    <td>9.0</td>
    <td>10.0</td>
    <td>1.8</td>
  </tr>
  <tr>
    <td>22.7</td>
    <td>11.9</td>
    <td>22.5</td>
    <td>28.9</td>
    <td>17.3</td>
    <td>8.8</td>
    <td>15.2</td>
    <td>20.1</td>
    <td>19.4</td>
    <td>20.3</td>
    <td>17.9</td>
    <td>6.4</td>
    <td>18.3</td>
    <td>9.6</td>
    <td>10.8</td>
    <td>8.1</td>
    <td>9.0</td>
    <td>10.0</td>
    <td>1.8</td>
  </tr>
</table>

Figure 8: Mean claim count for answers without surface-level issues. The left heatmap shows the results for culturally agnostic questions while the right heatmap shows the results for culturally specific questions. Only languages where at least 10 answers were free from surface-level issues are included.
Figure 9: Percentage of model answers without surface issues per language. The left heatmap shows the results for culturally agnostic questions while the right heatmap shows the results for culturally specific questions.

Figure 10: Factual precision for answers without surface-level issues. The left heatmap shows the results for culturally agnostic questions while the right heatmap shows the results for culturally specific questions. We remove model-language combinations for which there are not at least 10 answers without surface-level issues. Factual precision degrades on culturally specific questions, especially for low-resource languages.
Figure 11: Relevance for answers without surface-level issues. The left heatmap shows the results for culturally agnostic questions while the right heatmap shows the results for culturally specific questions. We remove model-language combinations for which there are not at least 10 answers without surface-level issues. Answer relevance degrades for low-resource languages but is similar on culturally specific and culturally agnostic questions.

exhibit similar behavior, with less than 1% for other models. Lastly, we observed an unusually high proportion of emojis in responses generated by LLAMA-3-70B, with 17.54% containing at least one emoji.

Human- vs automatically-collected questions
Table 14 shows model performance scores on human collected and automatically collected questions. We see comparable results between the two question sets, though some model rankings change. Specifically, GPT-4-TURBO and GEMINI-1.5-PRO move ahead of GPT-40 and CLAUDE-3-OPUS respectively in overall performance. Nevertheless, we see important trends like poor factuality scores and high Mixtral-8x22B repetitions on both sets of questions.

Figure 12 breaks down model overall performance on the human collected and automatically collected questions by language. We observe that model performance is higher on the automatically collected question for most languages. To determine whether the performance difference between the two question sets is significant, for each language we conduct a 2-sample Kolmogorov–Smirnov test on the overall answer scores for those languages, with the null hypothesis that the answer scores are drawn from the same distribution. The tests refute the null hypothesis with p-values below 0.01 for all languages except English and Hindi, for which p-values are 0.22 and 0.08 respectively. Although model performance is not identical on both question sets, analysis like Table 14 shows that model performance on automatically collected questions is an effective proxy for performance on human collected questions.

Answer statistics: We compute the lengths of generated answers using tiktoken with the o200k_base encoding. Table 17 presents statistics for the length of answers generated by each model. To account for variations in token count due to the language of generation and the presence of repetitions, we provide separate statistics for all answers and for those produced in the correct language without repetitions. Finally, we provide the percentage of answers produced in English for a non-English question in Figure 16.

D Human Evaluation

In this section, we present the details of human evaluation.

Evaluation Task The evaluation was conducted using LabelStudio (Tkachenko et al., 2020-2022). On the UI, annotators were presented with a question, a gold answer (if applicable), and three competitive answers in random order. The annotation
Figure 12: Answer scores \( S \) based on our quality criteria: surface issues, factuality and relevance. The left heatmap shows the results for human collected questions while the right heatmap shows the results for a subset of automatically collected questions (100 per language). Model performance differs notably between human collected and automatically collected questions in non-English languages.

Figure 13: Percentage of responses generated in the correct language by model for culture specific and culturally agnostic questions. Blank cells for GEMINI-1.5-PRO indicate languages where the API returned an error message. Balochi, Hiligaynon, and Papiamento are omitted since language detection libraries performed poorly for these languages. Additionally, detection accuracy for Kirundi was compromised, with instances of Kirundi being incorrectly identified as Kinyarwanda. Please see Table 13 for details.
<table>
  <tr>
    <th></th>
    <th colspan="7">% of answers with repetitions for cult. agnostic questions</th>
    <th colspan="7">% of answers with repetitions for cult. specific questions</th>
  </tr>
  <tr>
    <th></th>
    <th>English</th>
    <th>Arabic</th>
    <th>Chinese</th>
    <th>German</th>
    <th>Hebrew</th>
    <th>Hindi</th>
    <th>Hungarian</th>
    <th>Japanese</th>
    <th>Korean</th>
    <th>Russian</th>
    <th>Spanish</th>
    <th>Afar</th>
    <th>Faroese</th>
    <th>Fijian</th>
    <th>Pashto</th>
    <th>Samoan</th>
    <th>Tongan</th>
    <th>Tswana</th>
    <th>Wolof</th>
  </tr>
  <tr>
    <td>English</td>
    <td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>2.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>7.8</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>11.5</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.5</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>German</td>
    <td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.5</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Hebrew</td>
    <td>11.8</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>14.8</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>3.9</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>11.9</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Hungarian</td>
    <td>3.9</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>2.5</td><td>0.0</td><td>1.0</td><td>0.0</td><td>1.0</td><td>2.1</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>10.7</td><td>0.0</td><td>1.7</td><td>0.5</td><td>0.5</td><td>1.2</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>2.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>2.0</td><td>0.0</td><td>0.0</td><td>5.2</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>1.5</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.5</td><td>0.5</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>2.0</td><td>0.0</td><td>0.0</td><td>1.1</td><td>0.0</td><td>0.5</td><td>0.0</td><td>3.8</td><td>1.1</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Afar</td>
    <td>0.0</td><td>9.8</td><td>35.3</td><td>21.6</td><td>0.0</td><td>11.8</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>16.0</td><td>8.0</td><td>0.0</td><td>4.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Faroese</td>
    <td>17.6</td><td>2.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>13.3</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Fijian</td>
    <td>3.9</td><td>15.7</td><td>15.7</td><td>0.0</td><td>2.0</td><td>0.0</td><td>0.0</td><td>10.7</td><td>18.7</td><td>5.3</td><td>1.3</td><td>1.3</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Pashto</td>
    <td>84.3</td><td>15.7</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>72.0</td><td>10.7</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Samoan</td>
    <td>11.8</td><td>25.5</td><td>3.9</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>8.0</td><td>8.0</td><td>4.0</td><td>4.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Tongan</td>
    <td>31.4</td><td>52.9</td><td>37.3</td><td>2.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>20.0</td><td>60.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Tswana</td>
    <td>19.6</td><td>23.5</td><td>0.0</td><td>0.0</td><td>2.0</td><td>0.0</td><td>0.0</td><td>18.5</td><td>18.5</td><td>1.5</td><td>0.0</td><td>4.6</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
  <tr>
    <td>Wolof</td>
    <td>5.9</td><td>5.9</td><td>31.0</td><td>33.3</td><td>2.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>33.0</td><td>14.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
  </tr>
</table>

Figure 14: Percentage of answers with identified repetitions for culturally specific and culturally agnostic questions. Blank cells for GEMINI-1.5-PRO indicate languages where the API returned an error message.

You will be provided with a question and an answer, both translated from [language] into English. The answer was produced by an LLM.

Your task is to evaluate whether the answer is relevant to the question. **Relevance means that the answer makes a genuine attempt to address the question as stated.** The factual accuracy of the answer is not important. An answer should be judged as "irrelevant" if it:
- Does not attempt to answer the question,
- Focuses on a completely different topic,
- Is merely a refusal to answer, or
- Simply translates the content of the question without addressing it.

**Instructions:**

1. Read the question and the answer carefully.
2. Provide an explanation of your reasoning enclosed in `<explanation></explanation>` tags. 3. Then, give your final judgment enclosed in `<judgment></judgment>` tags with either the word `relevant` or `irrelevant`.

**Content:**

**Question:**
<question>
[question]
</question>

**Answer:**
<answer>
[answer]
</answer>

Figure 15: Prompt used with GPT-4o to determine the relevance of an LLM-generated answer to its question. Strings in the form [form] are placeholders that are replaced at runtime. The question and answer are provided in their English translations.
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">Surface Level</th>
    <th colspan="2">Fine-Grained</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>Wrong Lang. (%) ↓</th>
    <th>Repetitions (%) ↓</th>
    <th>W/o Issues (%) ↑</th>
    <th>Factual Pr. (%) ↑</th>
    <th>Relevance (%) ↑</th>
    <th>Overall Score S ↑</th>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td><b>0.0 / 0.0</b></td>
    <td><b>0.0 / 0.1</b></td>
    <td><b>100.0 / 99.9</b></td>
    <td><b>52.3 / 55.8</b></td>
    <td><b>98.4 / 98.3</b></td>
    <td><b>52.0 / 55.0</b></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td><b>0.0 / 0.0</b></td>
    <td><b>0.1 / 0.5</b></td>
    <td><b>99.9 / 99.5</b></td>
    <td><b>52.3 / 56.5</b></td>
    <td><b>98.7 / 98.5</b></td>
    <td><b>51.8 / 55.5</b></td>
  </tr>
  <tr>
    <td>AYA-EXPANSE-32B</td>
    <td>0.1 / 0.1</td>
    <td><b>0.0 / 0.0</b></td>
    <td><b>99.9 / 99.9</b></td>
    <td>48.3 / 51.2</td>
    <td>96.6 / 95.8</td>
    <td>46.8 / 49.8</td>
  </tr>
  <tr>
    <td>GEMINI-1.5-PRO</td>
    <td>0.8 / 3.1</td>
    <td>0.8 / 2.9</td>
    <td>99.2 / 96.9</td>
    <td>47.7 / 48.6</td>
    <td>94.8 / 95.2</td>
    <td>46.1 / 47.8</td>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td><b>0.0 / 0.0</b></td>
    <td><b>0.0 / 0.0</b></td>
    <td><b>100.0 / 99.9</b></td>
    <td>44.6 / 50.4</td>
    <td>98.5 / 97.1</td>
    <td>44.0 / 49.0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>0.3 / 0.1</td>
    <td>4.4 / 4.7</td>
    <td>95.4 / 95.2</td>
    <td>45.5 / 47.7</td>
    <td>93.8 / 94.3</td>
    <td>41.6 / 43.8</td>
  </tr>
  <tr>
    <td>LLAMA-3-70B</td>
    <td>61.2 / 66.0</td>
    <td>0.7 / 0.5</td>
    <td>38.2 / 33.7</td>
    <td>44.9 / 49.8</td>
    <td>96.4 / 96.3</td>
    <td>16.2 / 16.5</td>
  </tr>
</table>

Table 14: Model performance aggregated across 10 high resource languages on human collected and a subset of automatically collected questions (100 per language). Each cell reports values on human collected / automatically collected culturally specific question.

<table>
  <tr>
    <th>MODEL</th>
    <th>CHECKPOINT</th>
    <th>AVAIL.</th>
    <th># PARAM</th>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>gpt-4-0125-preview</td>
    <td>closed</td>
    <td>?</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>gpt-4o-2024-05-13</td>
    <td>closed</td>
    <td>?</td>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td>claude-3-opus-20240229</td>
    <td>closed</td>
    <td>?</td>
  </tr>
  <tr>
    <td>GEMINI-1.5-PRO</td>
    <td>gemini-1.5-pro-preview-0514</td>
    <td>closed</td>
    <td>?</td>
  </tr>
  <tr>
    <td>LLAMA-3-70B</td>
    <td>Meta-Llama-3-70B-Instruct x</td>
    <td>open-weight</td>
    <td>70B</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>Mixtral-8x22B-Instruct-v0.1 x</td>
    <td>open-weight</td>
    <td>8x22B</td>
  </tr>
  <tr>
    <td>AYA-EXPANSE-32B</td>
    <td>aya-expanse-32b x</td>
    <td>open-weight</td>
    <td>32B</td>
  </tr>
</table>

Table 15: General information about models we evaluate using CaLMQA.

process for each answer involved: (1) marking any mistakes,\footnote{This step was included to help the annotators visualize any issues with the answer.} (2) stating whether the answer is in the correct language, (3) evaluating factual accuracy, (4) noting any content omissions, (5) commenting on the overall quality of each answer, (6) rating each answer on a 5-point scale (excellent, good, average, poor, unusable). Upon completing the ratings, annotators ranked the three answers from best to worst and provided a free-form explanation for their ranking. Figure 17 illustrates the overall flow of the evaluation task. The study was submitted for the review to Institutional Review Board and received a non-human subject determination.

Guidelines and Consent We provided human evaluation guidelines, describing how to use the interface (including a tutorial video) and explaining each of the steps in the annotation process. The guidelines link to the consent form.

Data Human evaluation was done for answers generated by CLAUDE-3-OPUS, GPT-4-TURBO, and MIXTRAL-8X22B for questions in English, German, Hindi, Fijian and Kirundi. For culturally specific questions, annotators chose 10 questions in their language that they felt confident they knew the answer to. For culturally agnostic questions, we sampled 10 English culturally agnostic questions, and used the original English and the translations into the 4 other languages. We provided annotators with bullet-point answers in English for the culturally agnostic questions.

Workers and Cost German and Hindi annotators were recruited via Prolific, while Fijian and Kirundi annotators were recruited via Upwork. English annotations were performed by one of the authors. All annotators were native speakers of their respective languages and had participated in the question collection. Each question took approximately 20–40 minutes to evaluate, with annotators receiving compensation of $7.50 USD per question and an additional $8.00 USD for reviewing the guidelines, totaling $158 USD per language. The overall cost of the evaluation amounted to approximately $720 USD.\footnote{We also covered Upwork charges which the platform impose on the freelancers.}

Results Figure 18 and Figure 19 show the results of annotation for whether the answer was generated in the same language as the question (see Table 25
<table>
  <tr>
    <th>ISSUE</th>
    <th>QUESTION</th>
    <th>GENERATION</th>
  </tr>
  <tr>
    <td>The model answers a non-English question in English</td>
    <td>A BKV járatain egy (külföldi) nyugdíjas minkét igazolja, hogy nyugdíjas a sofőrnel? ...<br><i>How can a (foreign) senior citizen prove their eligibility for free travel on Budapest’s public transport (BKV) to the driver?</i> ...</td>
    <td>I understand your concern! As a foreign pensioner, your mother is entitled to free travel on BKV (Budapest public transportation) services, but she needs to prove her pensioner status to the driver or inspector. According to the BKV website, foreign pensioners can use their national pensioner ID or a document that proves their pensioner status, along with a valid ID (e.g., passport). The document should be in English or have an English translation. Here are some options your mother can consider: ...[Hungarian/LLAMA-3-70B]</td>
  </tr>
  <tr>
    <td>A sequence of at least 20 tokens is repeated at least 4 times</td>
    <td>Ke eng se se neng se le pharologano ka lenyalo la ga Sir Seretse Khama le Ruth Khama mme Ruth Khama o ne a itshwara jang lefa go nale pharologanyo eo? <i>What was different about Sir Seretse Khama and Ruth Khama’s marriage and how did Ruth Khama behave despite the difference?</i></td>
    <td>Ke a leboga gore o ne a reng dithlong tse di ntle. Ka kakaretso, ga Sir Seretse Khama le Ruth Khama o ne a itshwara jang lefa go nale pharologano ya gagwe. Ruth Khama o ne a neng a reng mabaka a a latelang a a nang le bona, a a nang le lefatsheh la Botswana, a a nang le baagi ba bangwe ba bangata ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ba bangwe ...[Tswana/Aya-EXPANSE-32B]</td>
  </tr>
  <tr>
    <td>The model apologizes and is unable to answer the question</td>
    <td>Duni daddarba leh yan iyyaan duma duni core kee akah yan iyya? <i>How do they know how hot the earth’s core is?</i></td>
    <td>I’m sorry for any confusion, but it seems like the text you’ve provided is in a language that I don’t currently support. I can understand and generate text in several languages, but I need to know which one you’re using to provide an accurate response. If you’re able to translate your question into English, I’d be more than happy to help! [Afar/MIXTRAL-82x2B]</td>
  </tr>
  <tr>
    <td>The model recognizes the language of the text</td>
    <td>Umugirwa wo gufuba umwami wagenda gute mu Burundi? <i>How was the Kings’ funeral done in Burundi?</i></td>
    <td>A question in Kirundi! Umugirwa wo gufuba umwami wagenda gute mu Burundi? Translation: “What is the significance of the drum in Burundian culture?” In Burundian culture, drums (umugirwa) play a significant role in various aspects of life, including: ...[Kirundi/LLAMA-3-70B]</td>
  </tr>
  <tr>
    <td>The model translates the question into English</td>
    <td>Mahimo bala kontrolon ukon himuon sang tawo ang tiempo? Kon mahimo, hasta din kag paano? <i>Can weather be controlled or man made? If so, to what extent and how?</i></td>
    <td>What a fascinating question! In English, your question translates to: “Can we control time like we control a person? If so, up to what extent and how?” Unfortunately, the answer is no, we cannot control time like we control a person. Time is a fundamental concept that governs the universe, and it is not a physical entity that can be manipulated or controlled. ...[Hiligaynon/LLAMA-3-70B]</td>
  </tr>
</table>

Table 16: Examples of issues in model responses that are detected as surface-level errors.

![Percentage of answers produced in English by model which produced the answer for culturally specific and culturally agnostic questions. Blank cells for GEMINI-1.5-PRO indicate languages where the API returned an error message.](page_1092_1342_682_312.png)

Figure 16: Percentage of answers produced in English by model which produced the answer for culturally specific and culturally agnostic questions. Blank cells for GEMINI-1.5-PRO indicate languages where the API returned an error message.
<table>
  <tr>
    <th rowspan="2">MODEL</th>
    <th colspan="3">All Data</th>
    <th colspan="3">Correct Lang / No Repetitions</th>
  </tr>
  <tr>
    <th>MEAN</th>
    <th>MEDIAN</th>
    <th>STD</th>
    <th>MEAN</th>
    <th>MEDIAN</th>
    <th>STD</th>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td>296.4</td>
    <td>293</td>
    <td>88.9</td>
    <td>302.2</td>
    <td>297</td>
    <td>79.2</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>472.6</td>
    <td>482</td>
    <td>155.2</td>
    <td>468.9</td>
    <td>477</td>
    <td>147.2</td>
  </tr>
  <tr>
    <td>GPT-4O</td>
    <td>446.6</td>
    <td>425</td>
    <td>268</td>
    <td>434.9</td>
    <td>430</td>
    <td>184.8</td>
  </tr>
  <tr>
    <td>GEMINI-1.5-PRO</td>
    <td>265.6</td>
    <td>270</td>
    <td>247.1</td>
    <td>421.6</td>
    <td>421</td>
    <td>177.7</td>
  </tr>
  <tr>
    <td>AYA-EXPANSE-32B</td>
    <td>449.4</td>
    <td>437</td>
    <td>187.3</td>
    <td>476.3</td>
    <td>460</td>
    <td>289.7</td>
  </tr>
  <tr>
    <td>LLAMA-3-70B</td>
    <td>395.9</td>
    <td>410</td>
    <td>171.4</td>
    <td>478.7</td>
    <td>484</td>
    <td>138.8</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>305.3</td>
    <td>237</td>
    <td>281.9</td>
    <td>255.4</td>
    <td>252</td>
    <td>114</td>
  </tr>
</table>

Table 17: Mean, median, and standard deviation of token counts in answers generated by different models. To account for variations in token count due to the language of generation and the presence of repetitions, we provide separate statistics for all answers and for answers produced in the correct language without repetitions. Token counts were computed using tiktoken with the o200k_base encoding.

for detailed counts). Figure 20 and Figure 21 display the annotations of the severity of factual issues in each answer (see Table 26 for detailed counts). Figure 22 and Figure 23 present the annotations of the severity of omissions in each answer (see Table 27 for detailed counts). Figure 24 and Figure 25 show the rankings of the models for both culturally specific and culturally agnostic questions. Figure 4 shows ratings by model by question type. Finally, Figure 26 shows the distributions of scores assigned for each model by the question type and language of generation.

Statistical analysis We conducted a statistical analysis using the clmm() function from the ordinal package in R. Each model was fitted with the ordinal ratings (1–5) as the response variable and different predictors, allowing for random intercepts for annotators. Table 20 shows the results of a model with question type (either culturally specific or culturally agnostic) as the predictor. Table 21 presents the results of an analysis with model type, question type, and their interaction as predictors, complemented by Table 22, which shows the results of a post-hoc analysis. Finally, Table 23 displays the R^2 values for models with different predictors, namely model type, question type, omission ratings, factuality ratings, and language accuracy ratings.

Analysis of the annotations We conducted manual analyses of the comments provided by the annotators. For each analysis, we iteratively designed an annotation schema to analyze the submitted comments. Table 28 describes the annotation schema used for analyzing the comments on model ranking (i.e., the annotator’s reason for ranking a model 1st, 2nd, or 3rd). The results of this analysis are presented in Table 29. Table 30 describes the categories used to analyze the comments on factuality. The results of this analysis are presented in Table 31. Finally, Table 32 describes the categories used to analyze the general comments left by the annotators for each answer. The results of this analysis are presented in Table 33 and Table 34.

D.1 Additional Insights

We capture here insights gained from analyzing human evaluation that we could not fit in the main body of text.

Enumerating facts makes responses seem less human-like. German and Hindi annotators remarked about the presence fact enumerations (often in the form of dot points) for some model answers. For German, the enumeration structure made responses seem artificial (e.g. ‘Again very AI made structure. “Here are common methods” and a following enumeration plus the asterisk titles...’). For Hindi, listing facts makes the responses not seem human-like, though not necessarily like an AI either (e.g. ‘The answer is just stating points on why is smoking harmful, so it neither sounds human-like nor artificial.’). More broadly, the fact enumeration structure was described negatively in 5 responses, neutrally in 18 responses and positively in 2.

GPT-4-TURBO made the most grammar/spelling errors. 9 out of 12 spelling and grammar issues were noted for GPT-4-TURBO responses. 8 of these issues occurred for Fijian (e.g. ‘There is a minor error, and the system might have spelled “nodra”’
Was ist Überhangmandat?

Answer 2
Ein Überhangmandat ist ein Sitz im Bundestag, der eine Partei zusätzlich zu den ihr nach dem Zweitstimmenergebnis zustehenden Sitzen erhält.
Das passiert, wenn eine Partei in einem Bundesland mehr Direktmandate (durch Erststimmen) gewinnt, als ihr nach ihrem Zweitstimmenanteil in diesem Bundesland eigentlich zustehen würden.
Ein Beispiel:
- Partei X erhält 30% der Zweitstimmen in einem Bundesland, das 50 Sitze hat. Ihr stehen also 15 Sitze zu.
- Also: Partei X gewinnt 20 Wahlkreise und damit 20 Direktmandate in diesem Bundesland.
- Die 5 zusätzlichen Sitze über die 15 hinaus sind Überhangmandate.
(...)

Correct language?
[ ] yes [ ] partially [ ] no

Comment on the factual mistakes
"Überhangmandat" explicitly limited to the German Bundestag which is wrong. It can also...

Rate the severity of factual mistakes
[ ] none [ ] minor [ ] major

Comment on the content omissions
None

Rate the severity of missions
[ ] none [ ] minor [ ] major

Comment on the overall quality
Helpful and with a vivid example, which is necessary. But fails in the basic explanation, that...

Rate the overall quality
★★★★☆

Rank the answers from best (top) to worst (bottom)

Answer 3
Answer 1
Answer 2

Why did you choose this order? Explain your reasons.

Answer #3 give best explanation and example. Best content, good language. Answer #4 lacks example, fair otherwise. Answer #2 has a major content mistake in my opinion, as this topic is not limited to the Bundestag elections only.

Figure 17: Our human evaluation pipeline. The annotator has to first read the answer, mark and classify all the mistake, and then comment and rate different properties of the answer. Once they have completed evaluating all three answers they are asked to rank them with respect to each other and provide a justification for the ranking. The example shows a culturally specific questions and one answer in German. The answer was produced by CLAUDE-3-OPUS.

Figure 18: Annotations on Language Correctness for Culturally Specific Questions

Figure 19: Annotations on Language Correctness for Agnostic-Specific Questions

Figure 20: Factuality issues as assessed by the annotators by model for culturally specific questions

Figure 21: Factuality issues as assessed by the annotators by model for culturally agnostic questions

Figure 22: Omissions as assessed by the annotators by model for culturally specific questions

Figure 23: Omissions as assessed by the annotators by model for culturally agnostic questions
Figure 24: Number of times each model was ranked as first, second, and last for culturally specific questions.

Figure 25: Number of times each model was ranked as first, second, and last for culturally agnostic questions.

Figure 26: Scores distribution by language and model for Culturally Specific and Culturally Agnostic questions
<table>
  <tr>
    <th>Formula</th>
    <th colspan="4">rating ~ model + (1 | language/annotator)</th>
  </tr>
  <tr>
    <th colspan="5">Random Effects</th>
  </tr>
  <tr>
    <th>Group</th>
    <th>Name</th>
    <th>Variance</th>
    <th>Std. Dev.</th>
  </tr>
  <tr>
    <td>language</td>
    <td>(Intercept)</td>
    <td>0.7175</td>
    <td>0.847</td>
  </tr>
  <tr>
    <td colspan="2">Number of groups:</td>
    <td colspan="2">5</td>
  </tr>
  <tr>
    <th colspan="5">Fixed Effects</th>
  </tr>
  <tr>
    <th>Coefficient</th>
    <th>Estimate</th>
    <th>Std. Error</th>
    <th>z value</th>
    <th>Pr(&gt;|z|)</th>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>0.8635</td>
    <td>0.2885</td>
    <td>2.993</td>
    <td>0.00276 **</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>-1.9493</td>
    <td>0.2844</td>
    <td>-6.854</td>
    <td>7.18e-12 ***</td>
  </tr>
  <tr>
    <td colspan="5">Significance codes: 0 '****' 0.001 '***' 0.01 '**' 0.05 '*' 0.1 '.' 1</td>
  </tr>
</table>

Table 18: Results of cumulative link mixed model with ordinal ratings as the response variable and model as the predictor.

<table>
  <tr>
    <th>Contrast</th>
    <th>Estimate</th>
    <th>SE</th>
    <th>df</th>
    <th>z-ratio</th>
    <th>p-value</th>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS — GPT-4-TURBO</td>
    <td>-0.863</td>
    <td>0.288</td>
    <td>Inf</td>
    <td>-2.993</td>
    <td><b>0.0078</b></td>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS — MIXTRAL-8X22B</td>
    <td>1.949</td>
    <td>0.284</td>
    <td>Inf</td>
    <td>6.854</td>
    <td><b>&lt;.0001</b></td>
  </tr>
  <tr>
    <td>GPT-4-TURBO — MIXTRAL-8X22B</td>
    <td>2.813</td>
    <td>0.315</td>
    <td>Inf</td>
    <td>8.936</td>
    <td><b>&lt;.0001</b></td>
  </tr>
  <tr>
    <th colspan="6"><b>P value adjustment: Tukey method for comparing a family of 3 estimates</b></th>
  </tr>
</table>

Table 19: Post-hoc analysis for the model in Table 18. Tests performed using the emmeans library in R.

incorrectly. However, the language content is relevant so the rating is 4 out of 5, and it sounds like a human.’) and the last was in German (‘Defninety helpful, complete and clear. Also fluent. One spelling mistake found: Zusammengefasend is no German word should be "zusammengefasst" or similar. But that could be a human-alike typo.’). This mistakes were present in otherwise mostly positive responses, suggesting that the issues were not due to lack of language understanding. We suspect that this phenomenon may be the result of a tokenizer issue.
<table>
  <tr>
    <th>Formula</th>
    <th colspan="4">rating ~ type + (1 | language/annotator)</th>
  </tr>
  <tr>
    <th colspan="5">Random Effects</th>
  </tr>
  <tr>
    <th>Group</th>
    <th>Name</th>
    <th>Variance</th>
    <th>Std. Dev.</th>
  </tr>
  <tr>
    <td>annotator</td>
    <td>(Intercept)</td>
    <td>0.9418</td>
    <td>0.9705</td>
  </tr>
  <tr>
    <td>Number of Groups: annotator</td>
    <td></td>
    <td>5</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="5">Fixed Effects</th>
  </tr>
  <tr>
    <th>Coefficient</th>
    <th>Estimate</th>
    <th>Std. Error</th>
    <th>z value</th>
    <th>Pr(&gt;|z|)</th>
  </tr>
  <tr>
    <td>Culturally Agnostic</td>
    <td>0.7259</td>
    <td>0.2192</td>
    <td>3.312</td>
    <td>0.000926***</td>
  </tr>
  <tr>
    <td>Significance codes:</td>
    <td>0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 20: Results of cumulative link mixed model with ordinal ratings as the response variable and question type (culturally specific vs culturally agnostic) as the predictor.

<table>
  <tr>
    <th>Formula</th>
    <th colspan="4">rating ~ model * type + (1 | language/annotator)</th>
  </tr>
  <tr>
    <th colspan="5">Random effects:</th>
  </tr>
  <tr>
    <th>Groups</th>
    <th>Name</th>
    <th>Variance</th>
    <th>Std.Dev.</th>
  </tr>
  <tr>
    <td>Language</td>
    <td>(Intercept)</td>
    <td>0.7566</td>
    <td>0.8698</td>
  </tr>
  <tr>
    <td>Annotator:Language</td>
    <td>(Intercept)</td>
    <td>0.7566</td>
    <td>0.8698</td>
  </tr>
  <tr>
    <th colspan="5">Fixed effects:</th>
  </tr>
  <tr>
    <th>Coefficient</th>
    <th>Estimate</th>
    <th>Std. Error</th>
    <th>z-value</th>
    <th>Pr(&gt;|z|)</th>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>1.1679</td>
    <td>0.3868</td>
    <td>3.020</td>
    <td>0.002531 **</td>
  </tr>
  <tr>
    <td>MIXTRAL-8x22B</td>
    <td>-1.7013</td>
    <td>0.3853</td>
    <td>-4.415</td>
    <td>1.01e-05 ***</td>
  </tr>
  <tr>
    <td>Culturally Agnostic</td>
    <td>1.3561</td>
    <td>0.3956</td>
    <td>3.428</td>
    <td>0.000607 ***</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO:Culturally Agnostic</td>
    <td>-0.6182</td>
    <td>0.5849</td>
    <td>-1.057</td>
    <td>0.290489</td>
  </tr>
  <tr>
    <td>MIXTRAL-8x22B:Culturally Agnostic</td>
    <td>-0.7163</td>
    <td>0.5467</td>
    <td>-1.310</td>
    <td>0.190109</td>
  </tr>
</table>

Table 21: Cumulative link mixed model fitted with the Laplace approximation fitted with clmm() in R. The response variable is the ratings (an ordinal variable on a 5-point scale), with predictors being model (CLAUDE-3-OPUS, GPT-4-TURBO, or MIXTRAL-8x22B) and question type (culturally specific and culturally agnostic). Annotator nested within language is included as a random effect. The baseline model is CLAUDE-3-OPUS and the baseline question type is culturally specific. Model’s conditional \( R^2 \) is 0.497 (including random effects) and marginal \( R^2 \) is 0.266 (only fixed effects). Please refer to Table 22 for post-hoc analysis.
<table>
  <tr>
    <th>Contrast</th>
    <th>Estimate</th>
    <th>SE</th>
    <th>df</th>
    <th>z-ratio</th>
    <th>p-value</th>
  </tr>
  <tr>
    <td>Spec. CLAUDE-3-Opus- Agn. CLAUDE-3-Opus</td>
    <td>-1.356</td>
    <td>0.396</td>
    <td>Inf</td>
    <td>-3.428</td>
    <td><b>0.0091</b></td>
  </tr>
  <tr>
    <td>Spec. CLAUDE-3-Opus- Spec. GPT-4-Turbo</td>
    <td>-1.168</td>
    <td>0.387</td>
    <td>Inf</td>
    <td>-3.020</td>
    <td><b>0.0380</b></td>
  </tr>
  <tr>
    <td>Spec. CLAUDE-3-Opus- Agn. GPT-4-Turbo</td>
    <td>-1.906</td>
    <td>0.424</td>
    <td>Inf</td>
    <td>-4.492</td>
    <td><b>0.0001</b></td>
  </tr>
  <tr>
    <td>Spec. CLAUDE-3-Opus- Spec. Mixtral-8x22B</td>
    <td>1.701</td>
    <td>0.385</td>
    <td>Inf</td>
    <td>4.415</td>
    <td><b>0.0002</b></td>
  </tr>
  <tr>
    <td>Spec. CLAUDE-3-Opus- Agn. Mixtral-8x22B</td>
    <td>1.061</td>
    <td>0.374</td>
    <td>Inf</td>
    <td>2.835</td>
    <td>0.0687</td>
  </tr>
  <tr>
    <td>Agn. CLAUDE-3-Opus- Spec. GPT-4-Turbo</td>
    <td>0.188</td>
    <td>0.412</td>
    <td>Inf</td>
    <td>0.457</td>
    <td>1.0000</td>
  </tr>
  <tr>
    <td>Agn. CLAUDE-3-Opus- Agn. GPT-4-Turbo</td>
    <td>-0.550</td>
    <td>0.442</td>
    <td>Inf</td>
    <td>-1.242</td>
    <td>1.0000</td>
  </tr>
  <tr>
    <td>Agn. CLAUDE-3-Opus- Spec. Mixtral-8x22B</td>
    <td>3.057</td>
    <td>0.429</td>
    <td>Inf</td>
    <td>7.123</td>
    <td><b>&lt;.0001</b></td>
  </tr>
  <tr>
    <td>Agn. CLAUDE-3-Opus- Agn. Mixtral-8x22B</td>
    <td>2.418</td>
    <td>0.414</td>
    <td>Inf</td>
    <td>5.842</td>
    <td><b>&lt;.0001</b></td>
  </tr>
  <tr>
    <td>Spec. GPT-4-Turbo- Agn. GPT-4-Turbo</td>
    <td>-0.738</td>
    <td>0.436</td>
    <td>Inf</td>
    <td>-1.694</td>
    <td>1.0000</td>
  </tr>
  <tr>
    <td>Spec. GPT-4-Turbo- Spec. Mixtral-8x22B</td>
    <td>2.869</td>
    <td>0.420</td>
    <td>Inf</td>
    <td>6.836</td>
    <td><b>&lt;.0001</b></td>
  </tr>
  <tr>
    <td>Spec. GPT-4-Turbo- Agn. Mixtral-8x22B</td>
    <td>2.229</td>
    <td>0.404</td>
    <td>Inf</td>
    <td>5.514</td>
    <td><b>&lt;.0001</b></td>
  </tr>
  <tr>
    <td>Agn. GPT-4-Turbo- Spec. Mixtral-8x22B</td>
    <td>3.607</td>
    <td>0.462</td>
    <td>Inf</td>
    <td>7.800</td>
    <td><b>&lt;.0001</b></td>
  </tr>
  <tr>
    <td>Agn. GPT-4-Turbo- Agn. Mixtral-8x22B</td>
    <td>2.967</td>
    <td>0.445</td>
    <td>Inf</td>
    <td>6.669</td>
    <td><b>&lt;.0001</b></td>
  </tr>
  <tr>
    <td>Spec. Mixtral-8x22B- Agn. Mixtral-8x22B</td>
    <td>-0.640</td>
    <td>0.383</td>
    <td>Inf</td>
    <td>-1.669</td>
    <td>1.0000</td>
  </tr>
</table>

Table 22: Post-hoc analysis for the model in Table 21 with Bonferroni adjustment. Spec. refers to culturally specific questions while Agn. refers to culturally agnostic questions. Tests performed using the emmeans library in R.

<table>
  <tr>
    <th>Predictor</th>
    <th>Conditional R<sup>2</sup></th>
    <th>Marginal R<sup>2</sup></th>
  </tr>
  <tr>
    <td>Model</td>
    <td>0.214</td>
    <td>0.189</td>
  </tr>
  <tr>
    <td>Omission</td>
    <td>0.752</td>
    <td>0.740</td>
  </tr>
  <tr>
    <td>Factuality</td>
    <td>0.614</td>
    <td>0.560</td>
  </tr>
  <tr>
    <td>Language Acc.</td>
    <td>0.339</td>
    <td>0.327</td>
  </tr>
  <tr>
    <td>Q-Type</td>
    <td>0.093</td>
    <td>0.061</td>
  </tr>
  <tr>
    <td>Model * Q-Type</td>
    <td>0.497</td>
    <td>0.266</td>
  </tr>
</table>

Table 23: Conditional and Marginal R<sup>2</sup> values for different predictors. We fit cumulative link mixed models (c1mm() in R) with ratings as the response variable and different predictors. All models included random intercepts for annotators. Omission, Factuality, and Language Accuracy were treated as ordinal variables (no issues > minor issues > major issues), whereas Q-Type and Model are categorical variables with two and three levels respectively. The last model was fitted with the interaction between the Model and the Q-Type. The Conditional R<sup>2</sup> refers to the variance explained by both fixed effects (predictors) and random effects (annotators), while Marginal R<sup>2</sup> refers to the variance explained by fixed effects only.
<table>
  <tr>
    <th>MODEL</th>
    <th>WIN RATE</th>
    <th>REASON</th>
    <th># (spec./agn.)</th>
    <th>COMMENT</th>
  </tr>
  <tr>
    <td rowspan="3">GPT-4-TURBO</td>
    <td rowspan="3">55%</td>
    <td>CONTENT</td>
    <td>27 / 24</td>
    <td><i>Answer 1 (GPT-4-TURBO) is the perfect answer and and explains all the points needed to understand how to play the game 'Teen Patti'.</i></td>
  </tr>
  <tr>
    <td>LANGUAGE/FORMAT</td>
    <td>11 / 7</td>
    <td><i>Answer 3 (GPT-4-TURBO) is very well structured and easy to follow. It covers all the information as well.</i></td>
  </tr>
  <tr>
    <td>FACTUALITY/RELEVANCE</td>
    <td>10 / 3</td>
    <td><i>A3 (GPT-4-TURBO) is more factual than A1 (CLAUDE-3-OPUS) and A2 (MIXTRAL-8X22B).</i></td>
  </tr>
  <tr>
    <td rowspan="3">CLAUDE-3-OPUS</td>
    <td rowspan="3">36%</td>
    <td>CONTENT</td>
    <td>6 / 16</td>
    <td><i>Answer 3 (CLAUDE-3-OPUS) covers the topic in its entirety and hence is ranked 1st.</i></td>
  </tr>
  <tr>
    <td>LANGUAGE/FORMAT</td>
    <td>5 / 13</td>
    <td><i>Answer 2 (CLAUDE-3-OPUS) is more readable because the information is listed as points.</i></td>
  </tr>
  <tr>
    <td>FACTUALITY/RELEVANCE</td>
    <td>6 / 6</td>
    <td><i>A3 (CLAUDE-3-OPUS) is more detailed and factual than A1 (GPT-4-TURBO) and A2 (MIXTRAL-8X22B).</i></td>
  </tr>
  <tr>
    <td rowspan="3">MIXTRAL-8X22B</td>
    <td rowspan="3">9%</td>
    <td>CONTENT</td>
    <td>2 / 1</td>
    <td><i>A1 (MIXTRAL-8X22B) is better explained than A2 (CLAUDE-3-OPUS) and A3 (GPT-4-TURBO).</i></td>
  </tr>
  <tr>
    <td>LANGUAGE/FORMAT</td>
    <td>3 / 2</td>
    <td><i>All answers have equal quality content, so they are distinguished by their structure/verbosity. Answer 3 (MIXTRAL-8X22B) has a very natural structure. Answer 2 (GPT-4-TURBO) and answer 1 (CLAUDE-3-OPUS) have redundancies and answer 1's are slightly worse.</i></td>
  </tr>
  <tr>
    <td>FACTUALITY/RELEVANCE</td>
    <td>1 / 0</td>
    <td><i>Answer 2 (MIXTRAL-8X22B) was slightly more specific to Western Europe than Answer 1 (GPT-4-TURBO), but both were roughly equal in quality.</i></td>
  </tr>
</table>

Table 24: Win rates of the three models in human-evaluated 3-way comparisons of answers for 100 questions. Reasons behind the annotators’ decisions are provided, with separate reason counts for culturally specific and culturally agnostic questions. A breakdown of reasons into finer-grained categories is provided in Table 28.
<table>
  <tr>
    <th rowspan="2">LANGUAGE</th>
    <th rowspan="2">MODEL</th>
    <th colspan="3">CULTURALLY SPECIFIC</th>
    <th colspan="3">CULTURALLY AGNOSTIC</th>
  </tr>
  <tr>
    <th>YES</th>
    <th>PARTIALLY</th>
    <th>No</th>
    <th>YES</th>
    <th>PARTIALLY</th>
    <th>No</th>
  </tr>
  <tr>
    <td rowspan="3">German</td>
    <td>CLAUDE-3-OPUS</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>9</td>
    <td>1</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td rowspan="3">Hindi</td>
    <td>CLAUDE-3-OPUS</td>
    <td>9</td>
    <td>1</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>8</td>
    <td>2</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td rowspan="3">Kirundi</td>
    <td>CLAUDE-3-OPUS</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>1</td>
    <td>9</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>1</td>
    <td>9</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>0</td>
    <td>7</td>
    <td>3</td>
    <td>1</td>
    <td>5</td>
    <td>4</td>
  </tr>
  <tr>
    <td rowspan="3">Fijian</td>
    <td>CLAUDE-3-OPUS</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>3</td>
    <td>0</td>
    <td>7</td>
    <td>2</td>
    <td>0</td>
    <td>8</td>
  </tr>
  <tr>
    <td rowspan="3">English</td>
    <td>CLAUDE-3-OPUS</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
</table>

Table 25: Count of instances generated in the language of the question by model and question-type, and the language being evaluated

<table>
  <tr>
    <th rowspan="2">LANGUAGE</th>
    <th rowspan="2">MODEL</th>
    <th colspan="3">CULTURALLY SPECIFIC</th>
    <th colspan="3">CULTURALLY AGNOSTIC</th>
  </tr>
  <tr>
    <th>NONE</th>
    <th>MINOR</th>
    <th>MAJOR</th>
    <th>NONE</th>
    <th>MINOR</th>
    <th>MAJOR</th>
  </tr>
  <tr>
    <td rowspan="3">German</td>
    <td>CLAUDE-3-OPUS</td>
    <td>8</td>
    <td>1</td>
    <td>1</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>8</td>
    <td>2</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>8</td>
    <td>2</td>
    <td>0</td>
    <td>9</td>
    <td>1</td>
    <td>0</td>
  </tr>
  <tr>
    <td rowspan="3">Hindi</td>
    <td>CLAUDE-3-OPUS</td>
    <td>7</td>
    <td>2</td>
    <td>1</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>7</td>
    <td>3</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>1</td>
    <td>4</td>
    <td>5</td>
    <td>6</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td rowspan="3">Kirundi</td>
    <td>CLAUDE-3-OPUS</td>
    <td>0</td>
    <td>1</td>
    <td>9</td>
    <td>3</td>
    <td>4</td>
    <td>3</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>0</td>
    <td>4</td>
    <td>6</td>
    <td>3</td>
    <td>6</td>
    <td>1</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
  </tr>
  <tr>
    <td rowspan="3">Fijian</td>
    <td>CLAUDE-3-OPUS</td>
    <td>7</td>
    <td>3</td>
    <td>0</td>
    <td>8</td>
    <td>2</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>8</td>
    <td>2</td>
    <td>0</td>
    <td>3</td>
    <td>7</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>5</td>
    <td>5</td>
    <td>0</td>
    <td>5</td>
    <td>5</td>
    <td>0</td>
  </tr>
  <tr>
    <td rowspan="3">English</td>
    <td>CLAUDE-3-OPUS</td>
    <td>8</td>
    <td>1</td>
    <td>1</td>
    <td>9</td>
    <td>1</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
    <td>9</td>
    <td>1</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>7</td>
    <td>2</td>
    <td>1</td>
    <td>9</td>
    <td>1</td>
    <td>0</td>
  </tr>
</table>

Table 26: Factuality issues in model generation by model, question type and language of the question
<table>
  <tr>
    <th rowspan="2">LANGUAGE</th>
    <th rowspan="2">MODEL</th>
    <th colspan="3">CULTURALLY SPECIFIC</th>
    <th colspan="3">CULTURALLY AGNOSTIC</th>
  </tr>
  <tr>
    <th>None</th>
    <th>Minor</th>
    <th>Major</th>
    <th>None</th>
    <th>Minor</th>
    <th>Major</th>
  </tr>
  <tr>
    <td rowspan="3">German</td>
    <td>CLAUDE-3-OPUS</td>
    <td>6</td>
    <td>1</td>
    <td>3</td>
    <td>7</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>6</td>
    <td>4</td>
    <td>0</td>
    <td>8</td>
    <td>2</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>1</td>
    <td>7</td>
    <td>2</td>
    <td>3</td>
    <td>6</td>
    <td>1</td>
  </tr>
  <tr>
    <td rowspan="3">Hindi</td>
    <td>CLAUDE-3-OPUS</td>
    <td>8</td>
    <td>1</td>
    <td>1</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>9</td>
    <td>0</td>
    <td>1</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>3</td>
    <td>1</td>
    <td>6</td>
    <td>4</td>
    <td>4</td>
    <td>2</td>
  </tr>
  <tr>
    <td rowspan="3">Kirundi</td>
    <td>CLAUDE-3-OPUS</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
    <td>4</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>2</td>
    <td>2</td>
    <td>6</td>
    <td>6</td>
    <td>3</td>
    <td>1</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
  </tr>
  <tr>
    <td rowspan="3">Fijian</td>
    <td>CLAUDE-3-OPUS</td>
    <td>6</td>
    <td>3</td>
    <td>1</td>
    <td>6</td>
    <td>4</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>8</td>
    <td>1</td>
    <td>1</td>
    <td>7</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>3</td>
    <td>0</td>
    <td>7</td>
    <td>0</td>
    <td>0</td>
    <td>10</td>
  </tr>
  <tr>
    <td rowspan="3">English</td>
    <td>CLAUDE-3-OPUS</td>
    <td>6</td>
    <td>3</td>
    <td>1</td>
    <td>8</td>
    <td>2</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>9</td>
    <td>1</td>
    <td>0</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>3</td>
    <td>2</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
    <td>0</td>
  </tr>
</table>

Table 27: Count of omission issues by severity type, model, and language for culturally specific and culturally agnostic questions
<table>
  <tr>
    <th>TYPE</th>
    <th>DESCRIPTION</th>
    <th>EXAMPLE</th>
  </tr>
  <tr>
    <th colspan="3">Content: Completeness/Explanation</th>
  </tr>
  <tr>
    <td>COMPLETENESS</td>
    <td>The answer was perceived as complete.</td>
    <td><i>Answer 3 (Claude-3-Opus) covers the topic in its entirety and hence is ranked 1st.</i> [Hindi]</td>
  </tr>
  <tr>
    <td>EXPLANATION/EXAMPLES</td>
    <td>The answer included useful explanation and/or examples.</td>
    <td><i>A3 (Gpt-4-Turbo) is better explained than A1 and A2.</i> [Fijian]</td>
  </tr>
  <tr>
    <td>DETAILS/BACKGROUND</td>
    <td>The answer included details and/or necessary background.</td>
    <td><i>Answer 1 (GPT-4-Turbo) and 2 (Claude-3-Opus) are similar but answer 1 has detailed information about the methods to measure body mass compared to 2.</i> [Hindi]</td>
  </tr>
  <tr>
    <td>GENERAL</td>
    <td>The answer was general, which was appropriate for the given question.</td>
    <td><i>Answer 3 (GPT-4-Turbo) is perfect. The writing style of the answer is the best compared to the other answers. For instance, it mentions the timeline in general rather than pointing out exact years of the event (In my opinion, different sources and online transcripts have a little variation in years in terms of history so it is the best to keep it general).</i> [Hindi]</td>
  </tr>
  <tr>
    <th colspan="3">Language/Presentation</th>
  </tr>
  <tr>
    <td>LANGUAGE</td>
    <td>The answer was fluent/used better language or was less AI-like.</td>
    <td><i>Both answer 1 (GPT-4-Turbo) and 3 (Claude-3-Opus) are good. Answer 1 sounds more human-like which is why it is ranked 1st.</i> [Hindi]</td>
  </tr>
  <tr>
    <td>STRUCTURE</td>
    <td>The structure of the answer was better.</td>
    <td><i>All answers have equal quality content, so they are distinguished by their structure/verbosity. Answer 3 (Mixtral-22B) has a very natural structure.</i> [English]</td>
  </tr>
  <tr>
    <td>SIMPLE/CLEAR/SPECIFIC</td>
    <td>The answer was clear and/or simple, to the point.</td>
    <td><i>Answer 3 (GPT-4-Turbo) is slightly clearer than answer 1 (Claude-3-Opus).</i> [English]</td>
  </tr>
  <tr>
    <td>SUCCINCT</td>
    <td>The answer was succinct.</td>
    <td><i>All three answers are complete by content in my view (good answer requires more, but that is more than question covers). Answer two (Mixtral-22B) I regard the best, as the density of content in a few lines is awesome - in most of the other questions, "death by long text and details" is valid, here I opt for short and good.</i> [German]</td>
  </tr>
  <tr>
    <th colspan="3">Factuality/Correctness</th>
  </tr>
  <tr>
    <td>FACTUAL</td>
    <td>The answer is better in terms of factuality.</td>
    <td><i>A3 (GPT-4-Turbo) is more factual than A1 (Claude-3-Opus) and A2 (Mixtral-22B).</i> [Fijian]</td>
  </tr>
  <tr>
    <td>RELEVANT</td>
    <td>The answer is the most relevant to the question. Often mentioned when other answers were irrelevant.</td>
    <td><i>Answer 3 (GPT-4-Turbo) is placed in the first position because it is relevant though not specific to Burundi. It could be used if it were specific. It has some important information (...).</i> [Kirundi]</td>
  </tr>
  <tr>
    <td>NO ISSUES</td>
    <td>There were no apparent issues in the answer.</td>
    <td><i>Answer 3 (Claude-3-Opus) had no notable issues.</i> [English]</td>
  </tr>
  <tr>
    <td>PARTIAL ANSWER</td>
    <td>The answer at least partially addresses the question (while other answers may be refusals, repetitions, or simply irrelevant/wrong).</td>
    <td><i>Answer 3 (GPT-4-Turbo) is the only one that tries to answer the question. The other 2 just point out differences between the medications.</i> [English]</td>
  </tr>
</table>

Table 28: Categories used for analysis of reasons for specific ranking of the answers
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Type</th>
    <th colspan="4">Completeness - Explanation</th>
    <th colspan="3">Language - Presentation</th>
    <th colspan="4">Factuality - Correctness</th>
  </tr>
  <tr>
    <th>Complete</th>
    <th>Explanation</th>
    <th>Details</th>
    <th>General</th>
    <th>Language</th>
    <th>Structure</th>
    <th>Simple</th>
    <th>Succinct</th>
    <th>Factual</th>
    <th>Relevant</th>
    <th>No issues</th>
    <th>Partial ans</th>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td><i>Spec.</i></td>
    <td>4</td>
    <td>0</td>
    <td>2</td>
    <td>0</td>
    <td>1</td>
    <td>2</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td><i>Spec.</i></td>
    <td>11</td>
    <td>5</td>
    <td>9</td>
    <td>2</td>
    <td>3</td>
    <td>2</td>
    <td>4</td>
    <td>2</td>
    <td>4</td>
    <td>1</td>
    <td>2</td>
    <td>3</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td><i>Spec.</i></td>
    <td>1</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>4</td>
    <td>0</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td><i>Agn.</i></td>
    <td>11</td>
    <td>0</td>
    <td>4</td>
    <td>1</td>
    <td>2</td>
    <td>1</td>
    <td>7</td>
    <td>3</td>
    <td>1</td>
    <td>1</td>
    <td>4</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td><i>Agn.</i></td>
    <td>14</td>
    <td>4</td>
    <td>6</td>
    <td>0</td>
    <td>1</td>
    <td>4</td>
    <td>1</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>2</td>
    <td>1</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td><i>Agn.</i></td>
    <td>0</td>
    <td>0</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>1</td>
    <td>0</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
</table>

Table 29: Count of different reasons mentioned by the annotator for ranking each model’s answer as the best out of three. Note that in some cases more than one reason might have been give by the annotator. <i>Spec.</i> refers to Culturally Specific questions, while <i>Agn.</i> refers to Culturally Agnostic questions.
<table>
  <tr>
    <th>TYPE</th>
    <th>DESCRIPTION</th>
    <th>COMMENT EXAMPLE (LANGUAGE/MODEL)</th>
  </tr>
  <tr>
    <th colspan="3">Direct Factual Errors</th>
  </tr>
  <tr>
    <td>DATE</td>
    <td>Issues involving incorrect temporal references.</td>
    <td><i>It is mentioned that Nifty was launched in 1995 but it was actually launched in 1996.</i> [Hindi/GPT-4-TURBO]</td>
  </tr>
  <tr>
    <td>ENTITY</td>
    <td>Incorrect entity such as a person, place, or organization.</td>
    <td><i>Almost everything is incorrect because the answer states that Ntare Rugamba is the person who accepted to die in the place of the king, while Ntare Rugamba is the king who ruled before the King Mwezi Gisabo.</i> [Kirundi/GPT-4-TURBO]</td>
  </tr>
  <tr>
    <td>EVENT</td>
    <td>Errors in the details or occurrence of events.</td>
    <td><i>It says that Aurangzeb got the mosque built at the place, however this claim is very strong. He got the temple destroyed but it is not sure if he got the mosque built, as it was a decade after the demolition of the temple.</i> [Hindi/MIXTRAL-8x22B]</td>
  </tr>
  <tr>
    <td>REASON</td>
    <td>Incorrect reasons or causative explanations for events or situations.</td>
    <td><i>"Ni o lobika na ligamu, o sa vakalevutaka na kena yawa mat yolomu" means folding your elbow increase the distance from your spirit"</i> [Fijian/GPT-4-TURBO]</td>
  </tr>
  <tr>
    <th colspan="3">Contextual and Logical Errors</th>
  </tr>
  <tr>
    <td>SCOPE</td>
    <td>Errors involving the incorrect extent or range of a fact.</td>
    <td><i>"Überhangmandat" explicitely limited to the German Bundestag which is wrong. It can also apply for regional votes for a single state for instance.</i> [German/CLAUDE-3-OPUS]</td>
  </tr>
  <tr>
    <td>ILLOGICAL</td>
    <td>Statements that are logically inconsistent or defy common sense.</td>
    <td><i>Yes, there is an error in the phrases "e rawa ni vakavana na turu ni wai ni mata ena vuku ni kena vakacacani na mata ni mata" means the cutting of onion can damage the eye of the eye sounds grammatically incorrect.</i> [Fijian/GPT-4-TURBO]</td>
  </tr>
  <tr>
    <td>NEGATION</td>
    <td>Incorrect use of negation or statements that present the opposite of what is true.</td>
    <td><i>Answer implies that internists are not PCPs</i> [English/CLAUDE-3-OPUS]</td>
  </tr>
  <tr>
    <td>PLACEMENT</td>
    <td>Inaccuracies resulting from the improper contextual placement of information.</td>
    <td><i>Putting "7. secondhand smoke" as a dot point incorrectly suggests that it is a reason why smoking is bad for the heart</i> [English/CLAUDE-3-OPUS]</td>
  </tr>
  <tr>
    <th colspan="3">Interpretative and Relevance Errors</th>
  </tr>
  <tr>
    <td>IRRELEVANT OR HALLUCINATED</td>
    <td>All information in the answer is irrelevant or is fabricated.</td>
    <td><i>The entire answer is wrong. The question is to state the difference between General and OBC category of people (based on caste and society status), but the answer is about general and gynecology practitioners (doctors). It is irrelevant to the question.</i> [Hindi/MIXTRAL-8x22B]</td>
  </tr>
  <tr>
    <td>Different Culture</td>
    <td>The answer includes references to the wrong culture.</td>
    <td><i>The answer given is incorrect because it is not specific to Burundi. It is in Kinyarwanda and talks about Rwanda.</i> [Kirundi/CLAUDE-3-OPUS]</td>
  </tr>
  <tr>
    <td>UNCLEAR OR UNDOCUMENTED</td>
    <td>Facts that are not clearly expressed or lack sufficient documentation or sourcing.</td>
    <td><i>The phrase "leweka" is not clear in the content. This issue might have a typo error.</i> [Fijian/CLAUDE-3-OPUS]</td>
  </tr>
  <tr>
    <td>NOT AN ANSWER</td>
    <td>The model refuses to answer usually producing the response in English sometimes accompanied with an attempt to translate the question.</td>
    <td><i>Incorrect language. (refusal to answer produced in English)</i> [Fijian/MIXTRAL-8x22B]</td>
  </tr>
</table>

Table 30: Categories used for analysis of comments on the factuality of the answers
<table>
  <tr>
    <th>MODEL</th>
    <th>TYPE</th>
    <th>UNCLEAR/ UNDOC.</th>
    <th>DATE</th>
    <th>ENTITY</th>
    <th>EVENT</th>
    <th>REASON</th>
    <th>SCOPE</th>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td><i>Spec.</i></td>
    <td>1</td>
    <td>1</td>
    <td>2</td>
    <td>1</td>
    <td>0</td>
    <td>2</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td><i>Spec.</i></td>
    <td>0</td>
    <td>1</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>1</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td><i>Spec.</i></td>
    <td>1</td>
    <td>1</td>
    <td>2</td>
    <td>2</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td>Agn.</td>
    <td>2</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>Agn.</td>
    <td>5</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>1</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>Agn.</td>
    <td>2</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>1</td>
    <td>0</td>
  </tr>
</table>

<table>
  <tr>
    <th>MODEL</th>
    <th>TYPE</th>
    <th>ILLOGICAL</th>
    <th>PLACEMENT</th>
    <th>NEGATION</th>
    <th>IRRELEVANT OR HALLUCINATED</th>
    <th>DIFFERENT CULTURE</th>
    <th>NOT AN ANSWER</th>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td><i>Spec.</i></td>
    <td>4</td>
    <td>0</td>
    <td>1</td>
    <td>6</td>
    <td>4</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td><i>Spec.</i></td>
    <td>2</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>7</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td><i>Spec.</i></td>
    <td>3</td>
    <td>0</td>
    <td>2</td>
    <td>7</td>
    <td>0</td>
    <td>8</td>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td>Agn.</td>
    <td>1</td>
    <td>1</td>
    <td>0</td>
    <td>6</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td>Agn.</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
    <td>4</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td>Agn.</td>
    <td>0</td>
    <td>0</td>
    <td>1</td>
    <td>5</td>
    <td>0</td>
    <td>10</td>
  </tr>
</table>

Table 31: Count of different types of factuality issues mentioned by annotators in their comments. The issues are presented by question type (<i>culturally specific</i> or <i>culturally agnostic</i>) and by model which generated the answer. The taxonomy used for this annotation can be found in Table 30.
<table>
  <tr>
    <th>CATEGORY</th>
    <th>DESCRIPTION</th>
    <th>COMMENT EXAMPLE (LANGUAGE/MODEL)</th>
  </tr>
  <tr>
    <th colspan="3">Content Issues</th>
  </tr>
  <tr>
    <td>UNHELPFULNESS</td>
    <td>Annotator cannot discern the question’s answer from the provided answer text</td>
    <td>This answer is not very helpful because it not specific to Burundi. Of course it contains some relevant information, but it lacks specificity. [Kirundi/GPT-4-Turbo]</td>
  </tr>
  <tr>
    <td>IRRELEVANCE</td>
    <td>Contains information not needed to answer the question</td>
    <td>The answer is irrelevant to the question and it isn’t helpful. It explains a completely different thing (explains the importance of blood sample rather than explaining why the blood sample is taken from the elbow area). [Hindi/Mixtral-8x22B]</td>
  </tr>
  <tr>
    <td>UNDERSPECIFICITY</td>
    <td>Ideas are not explained in sufficient detail</td>
    <td>Some instances of generic reasons without much detail about how it applies in Western Europe in particular. [English/GPT-4-Turbo]</td>
  </tr>
  <tr>
    <td>INCOMPLETENESS</td>
    <td>Extra information is needed to understand the answer</td>
    <td>The points mentioned in the answer are true but they do not cover the entirety of the topic. ... So it could be said that the answer is subjective rather than complete and clear. [Hindi/GPT-4-Turbo]</td>
  </tr>
  <tr>
    <td>REPETITIVENESS</td>
    <td>Some phrase or idea is repeated</td>
    <td>The phrases “Sa vakuamuria na noda vakarau ni dautama o ira na iTaueki vulagi…..” is repeated in almost every line... [Fijian/Mixtral-8x22B]</td>
  </tr>
  <tr>
    <td>GRAMMAR/SPELLING</td>
    <td>Grammar or spelling mistakes are present</td>
    <td>There is a minor error, and the system might have spelled "nodra" incorrectly. [Fijian/CLAUDE-3-OPUS]</td>
  </tr>
  <tr>
    <th colspan="3">Content Merits</th>
  </tr>
  <tr>
    <td>HELPFULNESS</td>
    <td>Information needed to answer the question is present and understandable</td>
    <td>It is helpful because it tells us the reasons for updating apps: Security, Good performance and New features. [Kirundi/GPT-4-Turbo]</td>
  </tr>
  <tr>
    <td>COMPLETENESS</td>
    <td>Answer does not need extra information to be understood</td>
    <td>It is complete, even makes an arc from outer core to inner core and a comparison to the sun surface temperature. So it covers more for what has been asked for but I like this additional information. [German/GPT-4-Turbo]</td>
  </tr>
  <tr>
    <th colspan="3">Presentation Issues</th>
  </tr>
  <tr>
    <td>ARTIFICIALITY</td>
    <td>Perceived by the annotator to be written by AI</td>
    <td>Structure with asterix and enumeration very artificial / non human like. [German/GPT-4-Turbo]</td>
  </tr>
  <tr>
    <td>NON-FLUENCY</td>
    <td>Reading the text is awkward/difficult for the annotator</td>
    <td>...Also Argument 5 is "estimation" and this is again mentioned in the summary -> a bit stange and hence not fluent. [German/CLAUDE-3-OPUS]</td>
  </tr>
  <tr>
    <td>INCLARITY</td>
    <td>Answer ideas or terms are difficult to understand</td>
    <td>Fluent and complete, but not clear, as complex word i.e. Zytokinen (Cytokines) are not explained. [German/CLAUDE-3-OPUS]</td>
  </tr>
  <tr>
    <th colspan="3">Presentation Merits</th>
  </tr>
  <tr>
    <td>NATURALITY</td>
    <td>Perceived by the annotator to be human-written/natural</td>
    <td>Complete, but so different to the other answers, most human-like answer as being relatively general to a more or less vague question. [German/CLAUDE-3-OPUS]</td>
  </tr>
  <tr>
    <td>FLUENCY</td>
    <td>Written in a smooth manner and easy to read</td>
    <td>There are no inaccuracies in the answer which makes it helpful. It makes use of easier vocabulary which sounds fluent. [Hindi/CLAUDE-3-OPUS]</td>
  </tr>
  <tr>
    <td>CLARITY</td>
    <td>Ideas are expressed in an easy-to-understand manner</td>
    <td>The answer is complete and helpful. It is very clear because the information is subdivided into general and OBC sections and it is easy to follow. [Hindi/GPT-4-Turbo]</td>
  </tr>
</table>

Table 32: Categories used for the analysis of annotators’ general comments on the quality of answers
<table>
  <tr>
    <th>MODEL</th>
    <th>TYPE</th>
    <th>UNHELPFULNESS*</th>
    <th>INCOMPLETENESS*</th>
    <th>ARTIFICIALITY*</th>
    <th>NON-FLUENCY*</th>
    <th>INCLARITY*</th>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td><i>Spec.</i></td>
    <td>9</td>
    <td>13</td>
    <td>4</td>
    <td>4</td>
    <td>10</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td><i>Spec.</i></td>
    <td>6</td>
    <td>7</td>
    <td>4</td>
    <td>2</td>
    <td>5</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td><i>Spec.</i></td>
    <td>20</td>
    <td>17</td>
    <td>19</td>
    <td>10</td>
    <td>15</td>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td><i>Agn.</i></td>
    <td>3</td>
    <td>5</td>
    <td>5</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td><i>Agn.</i></td>
    <td>1</td>
    <td>4</td>
    <td>11</td>
    <td>1</td>
    <td>5</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td><i>Agn.</i></td>
    <td>16</td>
    <td>14</td>
    <td>17</td>
    <td>8</td>
    <td>9</td>
  </tr>
</table>

<table>
  <tr>
    <th>MODEL</th>
    <th>TYPE</th>
    <th>IRRELEVANCE</th>
    <th>UNDERSPECIFICITY</th>
    <th>REPETITIVENESS</th>
    <th>GRAMMAR/SPELLING</th>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td><i>Spec.</i></td>
    <td>1</td>
    <td>1</td>
    <td>0</td>
    <td>1</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td><i>Spec.</i></td>
    <td>4</td>
    <td>1</td>
    <td>0</td>
    <td>2</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td><i>Spec.</i></td>
    <td>4</td>
    <td>7</td>
    <td>4</td>
    <td>1</td>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td><i>Agn.</i></td>
    <td>1</td>
    <td>1</td>
    <td>2</td>
    <td>1</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td><i>Agn.</i></td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>7</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td><i>Agn.</i></td>
    <td>3</td>
    <td>1</td>
    <td>4</td>
    <td>0</td>
  </tr>
</table>

Table 33: Counts of different types of issues noted in annotators’ comments about general answer quality. The issues are presented by question type (<i>culturally specific</i> or <i>culturally agnostic</i>) and by model which generated the answer. The taxonomy used for this annotation can be found in Table 32. Our UI suggested to annotators to make comments (positive or negative) about categories marked with *.

<table>
  <tr>
    <th>MODEL</th>
    <th>TYPE</th>
    <th>HELPFULNESS</th>
    <th>COMPLETENESS</th>
    <th>NATURALITY</th>
    <th>FLUENCY</th>
    <th>CLARITY</th>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td><i>Spec.</i></td>
    <td>24</td>
    <td>19</td>
    <td>24</td>
    <td>22</td>
    <td>22</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td><i>Spec.</i></td>
    <td>24</td>
    <td>20</td>
    <td>27</td>
    <td>22</td>
    <td>19</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td><i>Spec.</i></td>
    <td>13</td>
    <td>7</td>
    <td>18</td>
    <td>15</td>
    <td>12</td>
  </tr>
  <tr>
    <td>CLAUDE-3-OPUS</td>
    <td><i>Agn.</i></td>
    <td>26</td>
    <td>20</td>
    <td>30</td>
    <td>22</td>
    <td>30</td>
  </tr>
  <tr>
    <td>GPT-4-TURBO</td>
    <td><i>Agn.</i></td>
    <td>29</td>
    <td>24</td>
    <td>26</td>
    <td>17</td>
    <td>25</td>
  </tr>
  <tr>
    <td>MIXTRAL-8X22B</td>
    <td><i>Agn.</i></td>
    <td>16</td>
    <td>12</td>
    <td>22</td>
    <td>13</td>
    <td>16</td>
  </tr>
</table>

Table 34: Counts of different types of merits noted in annotators’ comments about general answer quality. The merits are presented by question type (<i>culturally specific</i> or <i>culturally agnostic</i>) and by model which generated the answer. The taxonomy used for this annotation can be found in Table 32. Our UI suggested to annotators to make comments (positive or negative) about all these categories.

Reviewer paper summaries: The authors propose CaLMQA, a multilingual long-form cultural QA dataset. The questions in the dataset are sourced from community forums for high- and mid-resource languages, and written by human annotators for low-resource languages. To ensure that the questions genuinely reflect local cultures and perspectives, the data collection process avoids direct translation from English.

Each question is accompanied by an English translation (for low-resource languages) and, for a subset of high- and mid-resource questions, a reference answer from either the forum or the annotator. In addition, the authors compile a set of culturally agnostic questions by translating general knowledge questions, which serve as a control set for their experiments.

The dataset is evaluated using 7 LLMs, with more than half being closed-source models. The authors automatically evaluate the generated answers across three dimensions: surface quality (fluency and repetitiveness), relevance, and factuality. They used a combination of existing metrics to quantify each of these aspects. In addition to automatic metrics, the authors also conduct manual evaluation on a smaller set of their data. Their findings include: even strong models frequently produce significant format errors, such as using the wrong language or generating incoherent responses. Furthermore, culturally specific questions tend to be more challenging for LLMs.
Summary of strengths: - This is one of the first datasets to address culturally specific questions in a long-form QA format, with broad language coverage.
- The use of community Q&A forums as a source for high- and mid-resource languages is a strong point, as it captures authentic, naturally occurring questions that reflect genuine user curiosity.
- The comparison between culturally agnostic and culturally relevant questions provides interesting insights.
Summary of weaknesses: - There may be differences in the types of questions between low-resource and high-/mid-resource languages due to differing data collection methods. While questions for high- and mid-resource languages come from natural user-generated content, those for low-resource languages are manually written by annotators, who may not necessarily reflect genuine curiosity. The annotator guidelines, particularly for question creation in low-resource languages, would benefit from more detailed clarification.
- There is an imbalance in the number of questions across languages, which may impact the comparability and generalizability of the results.
- Despite the large number of questions, the availability of gold-standard reference answers is limited.
- The selection of evaluated LLMs is not very comprehensive. Including one or two monolingual LLMs trained specifically on one of the target languages could provide valuable comparative insights and strengthen the analysis.
- The automatic evaluation metrics have several limitations:
  - Surface-level issues like incorrect language usage rely on existing language detection systems that perform poorly for some low-resource languages (reportedly with detection performance as low as zero for certain languages in Table 13).
  - Factuality and relevance are evaluated using translations generated by GPT-4o, but similar to the surface-level metric, it is unclear whether the approach using translation can generalize to the low-resource languages. This raises concerns about the validity of automatic evaluation metrics.
Comments, suggestions and typos: - Some methodological details would benefit from further elaboration. For example, how many annotators were assigned per question to produce the reference answers? How was subjectivity in annotation handled or mitigated?
- Typo in Table 1: significantly very -> significantly vary
Overall assessment scores: 3