NativQA: Multilingual Culturally-Aligned Natural Queries for LLMs

## Abstract

Abstract Natural Question Answering (QA) datasets play a crucial role in developing and evaluating the capabilities of large language models (LLMs), ensuring their effective usage in real-world applications. Despite the numerous QA datasets that have been developed, there is a notable lack of region-specific datasets generated by native users in their own languages. This gap hinders the effective benchmarking of LLMs for regional and cultural specificities. In this study, we propose a scalable framework, NativQA , to seamlessly construct culturally and regionally aligned QA datasets in native languages, for LLM evaluation and tuning. Moreover, to demonstrate the efficacy of the proposed framework, we designed a multilingual natural QA dataset, Multi NativQA , consisting of ∼ similar-to \sim 72K QA pairs in seven languages, ranging from high to extremely low resource, based on queries from native speakers covering 18 topics. We benchmark the Multi NativQA dataset with open- and closed-source LLMs. We made both the framework NativQA and Multi NativQA dataset publicly available for the community. 1 1 1 https://nativqa.gitlab.io/

## 1 Introduction

Recent advancements in Large Language Models (LLMs) have revolutionized the landscape of artificial intelligence, significantly pushing the state-of-the-art for a broad array of Natural Language Processing (NLP) and Speech Processing tasks, such as machine translation, question answering, automatic speech recognition, text-to-speech generation among others.
Their potential in language understanding and generation, across multiple (high- and low-resourced) languages has attracted researchers to benchmark the LLM capabilities across diverse tasks, domains, and disciplines OpenAI (); Touvron et al. (, ).
Moreover, the rapid integration of LLMs with various applications necessitates measuring cultural discrepancies in the responses generated by LLMs to ensure alignment with users’ cultural values and contexts.

Figure: Figure 1: Distribution of the Multi*NativQA* dataset across different languages.
Refer to caption: /html/2407.09823/assets/x1.png

Evaluating the generalization capabilities of LLMs across different tasks and languages has recently garnered significant attention.
The HELM project Liang et al. () assessed English LLMs across various metrics and scenarios. BIG-Bench Srivastava et al. () introduced a large-scale evaluation with 214 tasks, including low-resource languages. Recently, GPT-2.5 Radford et al. (), ChatGPT OpenAI (), and BLOOM Scao et al. () were evaluated by Bang et al. (); Ahuja et al. (); Hendy et al. ().

Figure: Figure 2: Examples of questions and answers in different languages.
Refer to caption: /html/2407.09823/assets/figures/examples_qa.png

Such evaluations have been conducted on standard QA, NLP, and/or speech datasets. LLM developers measure different capabilities of their released models, such as common sense reasoning (e.g., HellaSwag Zellers et al. ()), world knowledge (e.g., MMLU Hendrycks et al. () and Natural Questions Kwiatkowski et al. (), TriviaQA Joshi et al. ()), and reading comprehension (e.g.,
SQuAD Rajpurkar et al. ()).

Interestingly, most of these datasets are in English. Hence, multilingual and non-English LLMs have been evaluated by using machine translation, with or without human involvement, to translate the existing English datasets into corresponding languages. For example, to evaluate Jais Sengupta et al. () and AceGPT Huang et al. (), evaluation datasets have been translated into Arabic. Other examples on translated datasets include Korean MMLU Son et al. () and Okapi Lai et al. (), a translation of three benchmark datasets in 26 languages.
Machine translation has its drawbacks in terms of accuracy, but most notably, inability to reflect cultural and regional specificities of a targeted language.

Consequently, our observation suggest that using automatically-translated datasets is not ideal for an evaluation that truly reflects real users needs and tasks, and captures their cultural and regional interests and preferences as conveyed through their native languages. At the same time, the typical alternative of developing datasets in new languages by human annotators is a costly and time-consuming process.

Therefore, we propose a framework, *NativQA*, specifically designed to seamlessly develop regionally and culturally specific datasets following a human-machine collaborative approach. Datasets developed through *NativQA* serve two primary functions: (i) evaluating the LLM performance over real users information needs and interests expressed in their native languages, and (ii) facilitating fine-tuning and instruction tuning of LLMs to adapt to cultural contexts. Moreover, to show the efficacy of the *NativQA* framework, we developed a natural question-answering (QA) dataset, Multi*NativQA*, including $\sim$72K QA pairs in seven extremely low to high resource languages (as shown in Figure [1](#S1.F1)), covering 18 different topics from various regions (see examples in Figure [2](#S1.F2)).

Figure: Figure 3: *NativQA* framework, demonstrating the data collection and annotation process.
Refer to caption: /html/2407.09823/assets/figures/data_collection_pipeline_nativqa.png

Our contribution in this study is as follows:

- •
We propose the *NativQA* framework for developing culture- and region-specific natural question-answering datasets. This framework helps enhance LLM inclusivity and provides comprehensive culturally-aligned benchmark datasets.
- •
We develop and release a dataset, Multi*NativQA*, in seven languages with a size of over 72K QA pairs, covering 18 different topics based on real queries from native speakers.
- •
We establish baselines and provide evaluation results over the Multi*NativQA*  dataset using different open and closed models, promoting research in this area.

## 2 NativQA Framework

Figure [3](#S1.F3) presents the *NativQA* framework consisting of three inter-connected modules – Query Collection, QA Collection and QA Validation.

### 2.1 Query Collection (QC)

The objective of this module is to gather open-ended queries focusing on various predetermined topics derived from common concepts in everyday communication. We believe that the set of topics should be manually constructed, as this step requires identifying topics that are culture- or region-dependent (e.g., Events, Literature, etc.) (see Table [2](#S3.T2)).
Next, query collection can start by recruiting native speakers in the target countries. Each speaker is encouraged to write $M$ queries per topic, in their native (or second) language, focusing on issues they might encounter as residents of a corresponding major city.

Once the initial collection of queries, $Q_{m}$, is completed, the next step is to expand the set with synthesized queries, $Q_{s}$. The aim of expanding the set is to increase variability in sub-topics and writing styles in the final set of queries. For $Q_{s}$, we opt to prompt LLM to generate $x=10$ similar queries for each input query, $q_{m}^{i}\in Q_{m}$.
Finally, we de-duplicate $Q_{s}$ against $Q_{m}$ using exact string matching, resulting in the final set of seed queries, $Q_{0}=Q_{m}\bigcup Q_{s}$.

### 2.2 QA Collection (QAC)

The next step is to collect QA pairs that potentially cover topics represented by $Q_{0}$, using a search engine, e.g., Google.
We specifically select Google, since when a query is issued against it, it can return a data structure called “People also ask" that lists few questions asked by real users and potentially relevant to the initial user query, as shown in Figure [4](#S2.F4). Moreover, the questions are associated with answers extracted by the search engine and links to the answers sources.

Our QA curation module implements Algorithm [1](#alg1), using the seed queries $Q_{0}$ along with the number of iteration, $N_{iter}$, as input. For each iteration $i\in N_{iter}$, we collect QA pairs $P_{QA}^{i}$, and related queries $S^{i}_{rel}$ for each query, $q\in Q$, and then pass it to the filtering module and update the current query set $Q$. We repeat the process for all the iterations to obtain the final QA set, $S_{QA}$ for enriched queries $Q$.

Figure: Figure 4: Google’s QA list in response to a query.
Refer to caption: /html/2407.09823/assets/x2.png

Figure: Algorithm 1 Collecting QA pairs using seed queries $Q_{0}$. $P_{QA}^{i}$: QA pair, $S_{rel}^{i}$: related queries. QA(*) and RQ (*) are functions that return questions and answers, and related queries, respectively, which are obtained from the search engine for a given query.

### 2.3 QA Validation (QAV)

The last step of the *NativQA* framework is to validate the QA pairs, considering at least two aspects: (i) the quality and answerability of questions, and (ii) reliability and completeness of answers. We validate the QA pairs through the following steps.

##### Domain Reliability Checking (DRC).

The answers collected by our approach include a link to the web page from which an answer was extracted. Thus, our answer validation step starts by a semi-supervised approach that aims to keep QA pairs based on the reliability of the Web domain where the answer appears. We hypothesize that answers from web pages of a reliable domain are likely to be trustworthy.
In our approach, unique domains for the QA pairs in $S_{QA}$ are listed. Next, annotators manually annotate each domain by reliability based on an annotation guideline we designed for this task, inspired by several relevant studies Selejan et al. (); Flanagin and Metzger (); Metzger and Flanagin (). We then only keep the QA pairs with answers from reliable sources.

For developing large-scale fine- and instruction-tuning data, this approach is practical and scalable because it reduces manual effort required to obtain reliable answers. However, even if a domain is considered reliable (e.g., BBC, Guardian), this does not guarantee that the search engine effectively extracted an answer that accurately satisfies a given question. Thus, we added two more steps to validate the select QA pairs as discussed below.

##### QA Annotation (QAA).

Given the curated questions with answers from reliable sources, in this step, we opt for the manual checking and editing of answers for the remaining QA pairs. For each QA pair, we apply three types of annotations. (i) Question validation: human annotators should verify questions quality, and filter out lower-quality questions. Specifically, we define a “good question” as one that is fact-seeking, and can be answered with an entity or explanation. While a “bad question” is either ambiguous or incomprehensible, depends on clear false presupposition, opinion-seeking, or does not seek factual information.
(ii) Answer categorization: for a good question, annotators are asked to categorize answers based on correctness (see Section [3.2.2](#S3.SS2.SSS2)) by examining each QA pair and assessing whether the answer provides sufficient information to satisfy the question.
(iii) Answer editing: If an answer is not fully answering a question, annotator must edit the answer by adding more content from the answer source Web page such that the answer is complete. We limit the annotators to using the provided source Web pages to maintain the scope and reliability of answers we collect during this phase.

In the following section, we illustrate the effectiveness and scalability of the *NativQA* framework by showcasing steps to create the large-scale and multilingual Multi*NativQA* dataset.

## 3 Multi NativQA Dataset

**Table 1: List of languages with initial seed queries along with the number of QA pairs collected per language from different locations. CC: Country code, Lang.: Language, SQ: Seed Query. F. QA: Final QA set. Cat.: Categorization in terms of high (H), medium (M), low (L), and extremely low (X) as per Lai et al. ().**
| Lang. | Cat. | City | CC | #SQ | #QA | F.QA |
| --- | --- | --- | --- | --- | --- | --- |
| Arabic | M | Doha | QA | 3,664 | 12,311 | 7,548 |
| Assamese | X | Assam | IN | 900 | 21,009 | 572 |
| Bangla | L | Dhaka | BD | 889 | 13,688 | 10,724 |
| Bangla | L | Kolkata | IN | 900 | 13,378 | 10,969 |
| English | H | Dhaka | BD | 1,339 | 17,744 | 7,075 |
| English | H | Doha | QA | 3,414 | 25,621 | 12,806 |
| Hindi | M | Delhi | IN | 1,184 | 16,328 | 13,720 |
| Nepali | L | Kathmandu | NP | 1,222 | 11,503 | 581 |
| Turkish | M | Istanbul | TR | 900 | 23,143 | 8,027 |
| Total |  | – | – | 14,412 | 154,725 | 72,022 |

Our Multi*NativQA* dataset encompasses 7 languages,
ranging from high- to extremely low-resource on 7 different location/cities,
covering 18 predetermined topics (see Table [1](#S3.T1) for details).
Multi*NativQA* captures linguistic diversity, by including several dialects for dialect-rich languages like Arabic.(^2^22In addition to Modern Standard Arabic (MSA), used formally and officially, we incorporated six Arabic dialects: Egyptian, Jordanian, Khaliji, Sudanese, Tunisian, and Yemeni, representing the diverse linguistic landscape of Doha, Qatar.)
We also added two linguistic variations of Bangla to reflect differences between speakers in Bangladesh and West Bengal, India. Furthermore, we opted to cover English queries from Dhaka and Doha, where English is commonly used as a second language.

### 3.1 Implementing NativQA Framework

Query Collection: For multilingual query collection, we started with various predetermined topics (see Table [2](#S3.T2)) derived from common concepts in everyday lives of users (more details are discussed in Section [A.1](#A1.SS1)). Next, we asked the native speakers to write 10 to 50 queries per topic focusing on issues they encounter in their major cities and then extended to urban areas.
As there was no strict limit on the number of queries, some topics exceeded 50 queries. Then, we prompted GPT-4 to generate 10 similar queries based on each input query. The resultant number of seed queries, after de-duplication, for each language are reported in Table [1](#S3.T1).

QA Collection: Consequently, we use the QAC Module to enrich queries and QA pairs for each language and its respective city. For each language, we ran our collection algorithm for 3-7 iterations ($N_{iter}$) based on the convergence rate. After the QAC, we collected $\sim$154K QA pairs across all languages as presented in Table [1](#S3.T1) (column #QA).

QA Validation: The QAV is the final step of the *NativQA* framework.
It includes two sub-steps: DRC and QAA.
We implemented DRC across all target languages and cities.
As for QAA, we only implement this sub-step for the evaluation (test) split of the Multi*NativQA* dataset. This is because the QAA is often time-consuming and costly but highly effective for designing evaluation datasets. However, due to resource constraints, we completed the annotation for Arabic, Bangla, English, and Assamese.
Annotations for other languages are currently ongoing.

**Table 2: Selected topics used as seed to collect manual queries.**
| Query Topics |
| --- |
| Animal, Business, Cloth, Education, Events,<br>Food & Drinks, General, Geography, Immigration Related,<br>Language, Literature, Names & Persons, Plants, Religion,<br>Sports & Games, Tradition, Travel, Weather |

### 3.2 Annotation Guidelines

#### 3.2.1 Domain Reliability

The objective for the domain reliability annotation task is to verify the credibility of the source domain, which can
be used to judge the factuality and reliability of answers sourced from that domain.
We adopt the following definition of the credibility of the domain/website: “A credible webpage is one whose information one can accept as the truth without needing to look elsewhere. If one can accept information on a page as true at face value, then the page is credible; if one needs to go elsewhere to check the validity of the information on the page, then it is less credible” Schwarz and Morris ().

Annotators were tasked to review each web domain to determine its credibility and assign one of the following four reliability labels:

- •
Very reliable: The information is accepted without additional verification.
- •
Partially reliable: The information may need further verification.
- •
Not sure: Unable to verify or judge the website for any reason.
- •
Completely unreliable: The website and the information appear unreliable.

In Section [A.2](#A1.SS2) (in Appendix), we provide additional details of the instructions.

#### 3.2.2 QA Annotation

This phase of the *NativQA* framework involves three types of annotations. Below, we discuss the guidelines for each type.

##### 1. Question selection:

The purpose of this task is to evaluate the quality of the questions. The annotators assessed whether the questions are factual or meet the criteria discussed below. We defined two types of questions inspired by the NQ dataset Kwiatkowski et al. ().

- •
Good question: A good question is a fact-seeking question that can be answered with an entity or explanation.
- •
Bad question: A question is considered a bad question if it meets any of the following criteria:
–
Ambiguous or based on a false presupposition, making it incomprehensible.
–
Opinion-seeking, such as “Can you give me your thoughts on$\dots$?”
–
Does not ask for factual information.

Based on whether a question is marked as good or bad, the annotator’s subsequent tasks will vary. If a question is marked as good, the annotator will review the answer, its source page, and perform answer categorization tasks. Otherwise, further annotation is skipped, and the annotator proceeds to the next QA pair.

##### 2. Answer categorization:

An answer can be categorized into one of these categories: (i) correct answer, (ii) partially correct answer, and (iii) incorrect answer, and (iv) the answer can’t be found in the source page. Complete definition for each category is provided in Section [A.3](#A1.SS3).

##### 3. Answer editing:

The purpose of this step is to ensure that the answer accurately responds to the question and is correct, fluent, and informative. If the answer was incomplete, annotators are required to check the answer source page and extract content from it that can complete the answer (if such content is available in the page).

### 3.3 Annotation Task Setup

The annotation team consists of native speakers of the respective languages, who worked on the entire process starting from query collection to QA pair annotation. The annotators have diverse educational backgrounds, ranging from undergraduate students to those holding graduate and PhD degrees. The team was trained and monitored by an in-house expert annotator. To ensure quality, periodic checks of random annotation samples were conducted, and feedback was provided.
Depending on the availability of the annotators for a language, we opted to go for one to three annotators for the domain reliability task. For the DRC task, three annotators were assigned for the annotation.
When multiple annotators label a domain, the majority label is used as its final label. For other languages, domains were automatically matched with those already identified as reliable by annotators.
For the QAA task,
each QA pair was annotated by two annotators.

### 3.4 Annotation Platform

We utilized our in-house annotation platform for the annotation task. Separate annotation interfaces (as presented in Section [B](#A2)) were designed for each phase and each language. To facilitate the annotation process, the annotation interface included the annotation guidelines throughout the phases.

### 3.5 Annotation Agreement

To evaluate the reliability of manual annotations, we computed the Inter-Annotator Agreement (IAA) using a Fleiss’ Kappa coefficient ($\kappa$) for the domain reliability task for Arabic, Bangla, and English. The Kappa ($\kappa$) values for these languages are 0.53, 0.66, and 0.37, respectively, which correspond to fair to substantial agreement according to Landis and Koch’s scale Landis and Koch (). Note that we selected the final label where the majority agreed, meaning that we have above 66% agreement on the final label.

For the QA annotation task,
we first directly select only the questions where both annotators agree. For the disagreed cases, another annotator revises them; ultimately, we select based on the agreement of at least two annotators. For the answer editing, 75.79% (Bangla) to 88.5% (Arabic) of the cases were agreed upon by both annotators, computed based on exact string matching.

### 3.6 Statistics and Analysis

In Figure [1](#S1.F1), we report the initial collection of data distribution across languages, irrespective of the country they were collected from. English, Arabic, and Bangla are higher in proportion due to the fact that (i) English consists of data collected from Qatar and Bangladesh, (ii) Arabic consists of queries from different dialects, and (iii) Bangla consists of data from Bangladesh and India.
As table [1](#S3.T1) shows, our annotation process resulted in a decrease in QA set size by half (comparing initial QA set (column #QA) to final QA set (column F.QA)). We also faced a significant drop for Assamese and Nepali. This drop is due to the fact that the search engine returned QA pairs in non-native languages (in these cases, either Hindi or English) rather than the native language.
As part of our process, we filtered out QA pairs that are not in the target language. We identify the native language using a language detection tool(^3^33[http://fasttext.cc/docs/en/language-identification.html](http://fasttext.cc/docs/en/language-identification.html)) and then manually revise them.

In Figure [10](#A3.F10) and [11](#A3.F11) (in Appendix), we report topic wise distribution for all languages and regions. It appears that we have a very good coverage of topics for all languages.

**Table 3: Data split distribution for different languages.**
| Lang | Train | Dev | Test | Total |
| --- | --- | --- | --- | --- |
| Arabic | 5,284 | 747 | 1,517 | 7,548 |
| Assamese | – | – | 572 | 572 |
| Bangla-BD | 7,507 | 1,062 | 2,155 | 10,724 |
| Bangla-IN | 7,678 | 1,086 | 2,205 | 10,969 |
| English-QA | 8,964 | 1,268 | 2,574 | 12,806 |
| English-BD | 4,952 | 701 | 1,422 | 7,075 |
| Hindi | 9,604 | 1,358 | 2,758 | 13,720 |
| Nepali | – | – | 581 | 581 |
| Turkish | 5,619 | 795 | 1,613 | 8,027 |
| Total | 49,608 | 7,017 | 15,397 | 72,022 |

**Table 4: Reported results for different languages with different LLMs. MET.: METEOR, Rou.: Rouge1. Bold results are best per column per language.**
| Models | BLEU | Rou. | MET. | BLEU | Rou. | MET. | BLEU | Rou. | MET. | BLEU | Rou. | MET. | BLEU | Rou. | MET. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Arabic | Bangla-IN | English-BD | Hindi | Turkish |  |  |  |  |  |  |  |  |  |  |
| GPT-4o | 0.315 | 0.052 | 0.107 | 0.248 | 0.019 | 0.108 | 0.325 | 0.276 | 0.231 | 0.337 | 0.056 | 0.168 | 0.305 | 0.247 | 0.131 |
| GPT-4 | 0.275 | 0.039 | 0.090 | 0.225 | 0.021 | 0.093 | 0.325 | 0.278 | 0.234 | 0.296 | 0.048 | 0.148 | 0.288 | 0.228 | 0.119 |
| Gemini | 0.239 | 0.037 | 0.102 | 0.291 | 0.015 | 0.014 | 0.202 | 0.230 | 0.217 | 0.288 | 0.044 | 0.167 | 0.259 | 0.226 | 0.124 |
| LLama-3 | 0.143 | 0.026 | 0.054 | 0.051 | 0.004 | 0.047 | 0.285 | 0.215 | 0.183 | 0.112 | 0.027 | 0.080 | 0.098 | 0.145 | 0.067 |
| Mistral | 0.190 | 0.016 | 0.050 | 0.148 | 0.005 | 0.049 | 0.012 | 0.016 | 0.017 | 0.023 | 0.005 | 0.015 | 0.054 | 0.046 | 0.031 |
|  | Assamese | Bangla-BD | English-QA | Nepali |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o | 0.110 | 0.028 | 0.110 | 0.231 | 0.012 | 0.103 | 0.333 | 0.268 | 0.216 | 0.300 | 0.005 | 0.114 |  |  |  |
| GPT-4 | 0.086 | 0.019 | 0.085 | 0.210 | 0.011 | 0.085 | 0.331 | 0.268 | 0.217 | 0.280 | 0.004 | 0.090 |  |  |  |
| Gemini | 0.107 | 0.500 | 0.221 | 0.271 | 0.009 | 0.095 | 0.236 | 0.241 | 0.212 | 0.114 | 0.071 | 0.111 |  |  |  |
| LLama-3 | 0.016 | 0.004 | 0.040 | 0.053 | 0.002 | 0.045 | 0.287 | 0.219 | 0.183 | 0.058 | 0.000 | 0.048 |  |  |  |
| Mistral | 0.099 | 0.004 | 0.032 | 0.143 | 0.004 | 0.049 | 0.039 | 0.039 | 0.032 | 0.006 | 0.002 | 0.009 |  |  |  |

## 4 Experimental Setup

To establish baselines over Multi*NativQA*, we benchmark LLMs performance in the QA task over it. Our experiments setup is described next.

##### Data Splits.

The dataset for each language is split into training, development, and test sets using stratified sampling, considering topics as labels, with proportions of 70%, 10%, and 20%, respectively.
Due to the small size of the Nepali and Assamese sets, we use their full datasets as a testing sets. Details of the final splits are in Table [3](#S3.T3).

##### Models.

We experiment with both open and close LLMs. For the close models we use GPT-4o, GPT-4 (version 0314), Achiam et al. () and Gemini.(^4^44gemini-1.5-flash-preview-0514) For open models, we opt for llama-3-8b-instruct,(^5^55[https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/)) and mistral-7b-instruct.(^6^66[https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)) We use zero-shot learning as our setup with all models.
For reproducibility, we set the temperature to zero, and designed the prompts using concise instructions.
We utilized the LLMeBench framework to run our experiments Dalvi et al. ().

##### Evaluation Metrics.

We measure the performance the models using standard metrics commonly used for QA evaluation, such as BLEU, ROUGE and Meteor.

Figure: Figure 5: Average BLEU scores by language. X-Low: Extremely low.
Refer to caption: /html/2407.09823/assets/figures/BLEU_scores_by_language_grouped.png

## 5 Results

##### Open vs Close LLMs

Table [4](#S3.T4) shows the complete results for each LLM and language. We observe that closed models, especially GPT-4o, clearly outperform all other models across majority of languages(^7^77Except Bn-BD and BN-IN) with an average BLEU score of 0.278. This performance is then followed by GPT-4 and Gemini, with BLEU scores of 0.258 and 0.223, respectively. Results also show that in terms of open models, considering BLEU scores, Mistral is outperforming LLama3 in three
languages such as Arabic, Assamese, and Bangla (BD, IN) and underperforming for the rest.

##### High- vs Low-resource Languages

We also look at the the average performance of models per language (Reporting average BLEU scores in Figure [5](#S4.F5)). The average performance over English surpasses that over other languages, which is expected given that English is the highest resource language among those we consider. Medium-resource languages such as Arabic, Turkish, and Hindi rank just below English in terms of performance. Assamese, categorized as an extremely low-resource language, exhibits very poor performance. Overall, the representation and/or richness of digital content for a language is reflected in the performance of the models.

## 6 Related Work

LLMs have consistently showcased impressive capabilities spanning diverse disciplines and tasks and there have been efforts to evaluate the performance of LLMs on standard NLP tasks Achiam et al. (); Touvron et al. (); Bubeck et al. (); Bang et al. (); Ahuja et al. (); Hendy et al. ().
In a variety of domains, including finance Wu et al. (), law Liu et al. (), medicine Wang et al. (); Singhal et al. (), programming Li et al. () and intellectual property Ni et al. ().
NLP tasks has also been benchmarked in many studies. For example, Lai et al. () evaluated ChatGPT by considering seven different NLP tasks and covering 37 diverse languages with high, medium, low, and extremely low resource settings.
While there have been several efforts to develop resources and benchmark LLMs with those resources, most of the prior works are limited to English. Furthermore, regarding the evaluation for other languages, translated forms are commonly used. With our effort we aimed to address this gap by proposing a framework (*NativQA*) and a dataset.

##### Existing QA Datasets

There are many QA datasets in different languages. Below, we discuss the most widely used datasets.
Kwiatkowski et al. () and Yang et al. () proposed two extractive QA datasets including Natural Questions (NQ), both containing long-form answers for questions that include large-scale question-answer pairs. The generated long answer’s faithfulness is estimated by measuring the ratio of the golden short answer span contained in it.
Joshi et al. () developed TriviaQA dataset, which consists of 650K question-answer-evidence triples. These triples are created by merging 95K question-answer pairs.
Rajpurkar et al. () developed SquAD, which is a collection of 100K crowdsourced questions and answers paired with shortened Wikipedia articles. To develop the dataset, paragraphs/passages were given to annotators, and they were asked to write QA pairs based on the passage.
HelpSteer Wang et al. () is another QA dataset, which comprises a 37K sample dataset with multiple attributes of helpfulness preference that covers verbosity, accuracy, coherence, and complexity in addition to overall helpfulness.

**Table 5: Existing most notable QA datasets in compare to ours (Multi*NativQA*).**
| Dataset | # of Lang | Lang | Domain | Size |
| --- | --- | --- | --- | --- |
| NQ Kwiatkowski et al. () | 1 | En | Wiki | 323K |
| HotpotQA Yang et al. () | 1 | En | Wiki | 113K |
| TriviaQA Joshi et al. () | 1 | En | Wiki, Web | 650K |
| SquAD Rajpurkar et al. () | 1 | En | Wiki | 100K |
| HelpSteer Wang et al. () | 1 | En | Helpfulness | 37K |
| BanglaRQA Ekram et al. () | 1 | Bn | Wiki | 3k |
| Multi*NativQA* dataset | 7 | Ar, As, Bn,<br>En, Hi, Np, Tr | Open | 72K |

##### Evaluations of LLMs for QA

For benchmarking there are many notable datasets covering world knowledge Hendrycks et al. (), commonsense reasoning Zellers et al. (), reading comprehension Bandarkar et al. (), factuality Lin et al. (), and others. These datasets are usually transformed into multiple-choice questions. Additionally, standard QA datasets have also been used for LLM evaluation Hu et al. (). Kamalloo et al. () performed the analysis of different open-domain QA models, including LLMs by manually judging answers on a benchmark dataset of NQ-open Lee et al. (), and reported a systematic study of lexical matching. Their investigation shows that LLMs attain state-of-the-art performance but fail in lexical matching when candidate answers become longer.

In Table [5](#S6.T5), we report the most notable existing QA datasets along with ours. Compared to existing datasets, Multi*NativQA* dataset is novel in terms of the number of languages, wide coverage of topics, and being native to the region and culturally aligned.

## 7 Conclusions

In this study, we propose the *NativQA* framework, which enables constructing culturally and regionally-aligned natural QA datasets. Resulting datasets can aid in training/fine-tuning and evaluating LLMs over native and culturally-aligned real users information needs and tasks. The proposed framework is scalable and reduces human involvement in a dataset construction by automating several processes. We show the efficacy of the *NativQA* framework, by designing and developing a multilingual native QA dataset, Multi*NativQA*. We further enrich our study by benchmarking QA performance of five open and closed LLMs over Multi*NativQA*. Our results demonstrate that the latest closed model, GPT-4o, shows superior performance in general across languages.
Our study is an ongoing effort; therefore, we aim to extend the framework to include more languages and implement more measures to improve the quality of both the framework and the dataset.
In future, the dataset will be used to tune LLMs to improve cultural and regional alignment.

## 8 Limitations

While the proposed framework enables the development of datasets with cultural and native information, it currently has several limitations. Firstly, the *NativQA* framework still relies on human-in-the-loop processes, from seed query creation to manual revision of QA pairs. This dependency limits large-scale data collection. Although we consider the human-in-the-loop setting a limitation, we also note that ensuring a high-quality dataset without it would be challenging. Secondly, the semi-supervised approach, based on domain reliability (DRC), to dataset development is a reasonable starting point; however, full supervision (QAA) would ensure higher quality. Thirdly, in our current study, we relied on one search engine. This can be extended to include other search engines and use a mixture of engines to enrich QA pair collection. Fourth, due to resource limitations, including the availability of language-specific annotators, we have successfully annotated QA pairs for the test sets. Annotation for the development and training sets would be a part of our ongoing efforts.
Finally, our study is currently limited to benchmarking various open and closed models. Future research will focus on fine-tuning and training new models.

## Ethics and Broader Impact

The proposed *NativQA* framework does not involve collecting any personally identifiable information. Additionally, the proposed dataset does not include any information that can offend or harm any individual, entity, organization, or society. Therefore, we do not foresee any issues that may lead to potential risks. Human annotators were paid through external companies at standard payment rates applicable to their region. Information about human annotators is not part of the dataset, and their identities remain confidential.
The proposed framework and dataset will be released publicly for non-commercial research purposes. Therefore, we strongly believe that they will be beneficial for the research community.

## Appendix

## Appendix A Query on Search Engine

In Figure [6](#A1.F6), we show an example of a query to a search engine, that demonstrates related queries under “People also ask”, which we have also considered as queries in the several iterations of QA pair collection.

Figure: Figure 6: An example of search interface showing search response with “people also ask” option.
Refer to caption: /html/2407.09823/assets/figures/google_search.png

### A.1 Guideline for Collecting Seed Queries)

The purpose of this study is to collect natural basic Question Answer (QA) pairs to evaluate and enhance LLMs. Our approach to collecting such QA pairs is to use widely used search engines with natural queries to find relevant QA pairs. We intended to find a diverse set of questions; therefore, we selected different topics as listed below.

Topics: Education, Travel, Events, Food and Drinks, Names and Persons, Animals, Religion, Business, Language, Sports and Games, Clothes, Tradition, Weather, Geography, General, Literature, Plants, Science, and Immigration.

For each topic, the task was to collect seed queries. While collecting the seed queries, we needed to ensure language-specific and main-city-centric information as naturally as possible—information we typically ask on search engines. For example, “Does Qatar have beaches?” or “Do I need a visa to visit Qatar?”

These examples are based on Qatar; however, for each language, the questions will be specific to the local (main city/country) context.

### A.2 Domain Reliability (Detail Annotation Guideline)

##### General Characteristics

Below are the characteristics that we have considered as criteria for a domain to be more reliable:

##### Overall Design:

- •
The domain has a professional, polished, and attractive design. It has interactive features, is well organized, easy to navigate, loads fast, and has good response speed.
- •
There are no errors or broken links.
- •
It might have paid access to information.
- •
The domain name suffix is considered trustworthy (e.g., “.gov”).
- •
Absence/limited advertising. If advertising is present, they are good quality ads for reputable and decent products and organizations.
- •
The domain might be sponsored by or shows links to reputable organizations.
- •
Presence of privacy and security policies section or page. Presence of an About page, contact info, and address.
- •
If videos, images, and graphics are used on the website, they are high-quality and professional.

##### Content Quality:

- •
Author/entity names, qualifications, credentials, and contact information are present, and they are relevant to the topic of the content.
- •
Author/entity is reputable.
- •
Contains date stamp
- •
Presents information that is current and up to date.
- •
Has citations, especially to scientific data or references, and shows links to external authorities.
- •
Content is relevant to the target topic and current events.
- •
Professional-quality, clear writing, good formatting of text.
- •
Content appears accurate, lacks bias, factually correct, plausibility, and uses appropriate objective language.
- •
Free of misspellings, grammar mistakes, etc.
- •
The information provided is at an appropriate level, not too generic or elementary.

##### General Instructions:

We also provided the following general instructions to guide the annotation process.

- •
Do not spend more than five minutes per given Web domain.
- •
Explore/observe/look at ALL elements in the domain’s home page from top to bottom.
- •
Repeat points 1-2 on other pages from the same domain, and look at their content, structure, design, author, etc. You are not required to read these pages in full, reading the first 1-2 paragraphs is enough.
- •
During annotation, consider the annotation criteria mentioned in this guideline, and evaluate each source based on those aspects. A “reliable website” might not meet all those criteria. It is your job to measure the website’s reliability guided by these elements.
- •
You should evaluate a source based on that source only and what it presents. You should not navigate or search for outside sources even if some are linked inside the given domain/page.
- •
Please use “Not sure” very sparingly in rare cases when you are extremely unsure. It is preferable to always choose one of the other three labels.
- •
For social media websites (e.g., X, Facebook) choose: Very Reliable.
- •
For shopping websites, use the criteria listed in this guideline to decide. Some shopping websites are very reliable.
- •
For famous people’s websites, use the criteria listed in this guideline to decide.
- •
Websites that are in any other language ONLY (for example, only in En when you are working on a Bangla queries), for such cases choose: Not Sure.

### A.3 Definitions of Answer Categorization

Below we provide the definition of the categories that are defined for the answer categorization task.

- •
Correct answer: When the answer aligns with the information provided by the source. Note that the answer must be complete and address all parts of the question, but it does not need to match the source webpage verbatim. The answer can be a long, detailed response or a short snippet.
- •
Partially correct answer: When the answer does not address all parts of the question. In this case, the goal is to edit the answer using information from the source page. This involves directly copying text from the source webpage. Minimal editing may be needed to make the answer more comprehensive.
- •
Incorrect answer: When the answer text does not address the question. In this case, the goal is to edit the answer using information from the source page.
- •
Cannot find answer: When the answer is not available on the provided link/page.

## Appendix B Annotation Interface

In Figure [7](#A2.F7), we present an example of domain reliability checking, which consists of a URL of the domain, annotation guidelines, and four different options associated with the four categories we defined for this annotation task. Annotators select one of these options and submit.

In Figure [8](#A2.F8) and [9](#A2.F9) we demonstrate the two steps of question selection and answer editing and categorization tasks, respectively. Depending on the type of question selected, the annotator will be able to choose whether to edit the answer or not.

Figure: Figure 7: An example of the annotation interface for domain reliability checking.
Refer to caption: /html/2407.09823/assets/figures/domain_reliability.png

Figure: Figure 8: Annotation interface for Question selection.
Refer to caption: /html/2407.09823/assets/figures/qa_annotation_step1.png

Figure: Figure 9: Annotation interface for Answer editing and answer categorization.
Refer to caption: /html/2407.09823/assets/figures/qa_annotation_step2.png

## Appendix C Dataset: Additional Details

In Figure [10](#A3.F10) and [11](#A3.F11) we present the topic-wise data distribution for different datasets associated with various languages. Starting with the Arabic dataset, the predominant topic is names, comprising 10.2% of the data, a trend that also holds true for Assamese (8.2%). For Bangla, whether from Bangladesh or India, the major topic is general, representing 8.8% and 10.0% respectively. In Bangladesh, religion (10.4%) is the major topic for English, whereas in Qatar, general dominates at 26.6%. For Nepali, the leading topic is Business (22.9%), for Hindi it is Travel (8.2%), and for Turkish, names is the primary topic at 8.6%.

Figure: (a) Arabic
Refer to caption: /html/2407.09823/assets/figures/arabic_graph.png

Figure: (a) Nepali
Refer to caption: /html/2407.09823/assets/figures/nepali_graph.png

## Appendix D Data Release and License

The *NativQA* dataset will be publicly released under the Creative Commons Attribution Non Commercial Share Alike 4.0: [https://creativecommons.org/licenses/by-nc-sa/4.0/](https://creativecommons.org/licenses/by-nc-sa/4.0/).

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: none
soundness: 2
strengths: The collection of regionally relevant QA data is very important for expanding the capabilities of QA models. Even high resource languages contain lower resource regional variants and topics which need to be studied independently to improve user experiences in those regions; lower resource languages will benefit from the additional data provided here. The human validation steps taken in this work add non-trivial value to the automatically collected data.
summary: This paper introduces a semi-automatic framework for collecting multilingual QA data and provides a new dataset of questions in 7 languages from 9 regional language communities covering 18 topics. The languages in the new data include higher, medium, low, and extremely low resource languages. The proposed framework involves collecting human authored seed queries and finding related question/answer pairs from a search engine, then expanding this set iteratively. There are manual filtering mechanisms to ensure data quality: domain verification removes questions from untrusted domains, and a human annotation process verifies question quality, region relevance, and answer accuracy. The resulting dataset consists of ~64k verified datapoints. The paper provides some analysis of this data
weaknesses: The evaluation metrics used in this work are poor measures of QA system performance. The authors should consider modern metrics such as LLM-as-judge (https://huggingface.co/papers/2306.05685) or PEDANTS (https://arxiv.org/pdf/2402.11161) to give a better sense of existing model performance on the new data.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: * Why is the inter annotator agreement for English (line 347) so low? Any justification for it? 
* Figure 1: caption can be a bit more informative. Where are these examples from? 
* How do you decide whether a language is low resource or high resource? There can be multiple definitions, it’d be good to clear how you define “extremely low” to “high” resource language, that you argue as included in the part of the dataset. 
* How do you define “domain”? The domain reliability check part should be written a bit more clearly. How many domains were manually checked, and what is the rate of untrustworthy domains?
* Providing example instances from the dataset across the wide range of languages (with gloss) in the appendix will be helpful. 

Minor comments:
Table 5,...13.. - the Arabic questions should come with English gloss for readability. 
Missing related work:
* TyDiQA: https://arxiv.org/pdf/2003.05002
- this is one of the earlier work that collected multilingual questions for factoid questions with native speakers. 
* GooAQ: https://arxiv.org/abs/2104.08727 
- this paper also uses google to collect QA dataset.
soundness: 3
strengths: The paper presents a useful artifact, MultiNativQA dataset, that contains QA pairs in 7 languages, totaling 64K question answering datasets. The paper proposes human-in-the-loop data collection approach that leverages existing tools (e.g., Google's search outputs) smartly. 
The paper is easy to follow and writing is relatively clear.
summary: The paper presents a pipeline for collecting QA data, that leverages functions and outputs from Google Search (find relevant questions, providing answer to the questions) that can be applied to generate questions in a wide range languages. Then, they manually filter the generated QA pairs. Their new pipeline enables large-scale data collection. Paired with this framework, they also present 64K QA datasets on 7 languages. 

They evaluate LLM's performance on their dataset mainly with lexical based metrics (F1, BLEU, Rouge), showing that (1) performance drops as we move from high-resource to low-resource languages, and (2) the performance of closed-sourced models (GPT-4o and Gemini) outperforms that of open-sourced models (Llama-3.1, Mistral). 

While multilingual dataset, and question answering dataset, can be helpful for ICLR community, I find it hard to recommend the paper in its current form. The novelty from existing datasets is unclear, documentation of dataset is lacking, and I'm concerned about the quality of the dataset as well. Please see weaknesses section for details.
weaknesses: The contribution is not substantial enough. For a dataset paper / dataset generating framework paper, I think few key questions are: 

These questions are related to the novelty of the dataset:
* What are new capabilities of models that we weren't able to evaluate before that we can now evaluate with this benchmark? How does the new dataset qualitatively or quantitatively differ from existing datasets? 
  - More comparison with existing work would strengthen this work. Beyond the scale, how does the dataset differ from CaLMQA (Arora et al 24)? 
* What do we learn about models through this dataset?
   - The result discussion in section 6 can be improved. How does the findings in this paper differ from other findings in the prior work? contextualizing will be helpful. 

The data collection should be documented more carefully.
* The annotator statistics (e.g., how many annotators per language, how they are compensated, etc). 
* Data collection cost: what's the estimated cost for collecting dataset through this framework on other languages, and how much did it cost to collect MultiNativQA? Is this framework viable for other researchers without access to your inhouse annotator pool?
* Data statistics: How long are the queries and answers typically?
* Over how many examples did you compute inter-annotator agreement? If on a subset, how did you select a subset?

The validity of the dataset/evaluation quality:
* I think key concern that I have is the quality of the answer that is automatically extracted and included in this dataset. The paper presents filtering step (Section 3.3) and agreement statistics (4.4) but it's not very convicing. What does is the matching of 66.04% mean in answer editing task? Can we compare this to some existing datasets?
* Is lexical based evaluation metric sufficient for this dataset? 

I find the introduction to be vague. It’d be good to summarize concrete experimental findings (e.g., what are the result from benchmarking 2 open and 2 closed LLMs, as well as the fine-tuned LLM)?

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: N/A
soundness: 3
strengths: Originality

Through NativQA, the authors address a previously underexplored area: the creation of QA datasets that are culturally and regionally relevant, especially for languages with limited digital resources. This contribution moves beyond standard methods that rely on translations from English datasets, which often miss the nuanced needs of native speakers and culturally specific knowledge. The integration of a semi-automated process combining native query input with machine-assisted QA generation is particularly novel, reducing the dependency on purely manual or translation-based methods. This approach not only mitigates costs but also opens up new avenues for evaluating and fine-tuning LLMs in low-resource, culturally rich settings, setting it apart from existing QA datasets.

Quality

The quality of this work is supported by a rigorous, three-part framework, which includes query collection, QA pair extraction, and quality validation. The authors demonstrate a well-structured and scalable pipeline that ensures the QA data is both culturally relevant and diverse. The design of the MultiNativQA dataset, with over 64,000 annotated pairs across seven languages and nine regions, provides robust benchmarking capabilities. The authors also benchmark the performance of both open and closed-source LLMs on this dataset, offering a thorough evaluation. Additionally, the paper's use of parameter-efficient fine-tuning (PEFT) is well-justified and strategically chosen to address computational costs, a thoughtful addition to the framework's design.

Clarity

The paper is largely clear and methodical in its presentation. The breakdown of the framework’s three modules—Query Collection, QA Collection, and QA Validation—offers a logical flow and helps readers understand each step of the process. The choice of examples from different languages further elucidates the framework’s application across diverse contexts. Tables and figures, such as those illustrating dataset distribution and evaluation metrics, are well-designed and aid in comprehending the results. However, a more detailed discussion of the potential limitations in the methodology (e.g., reliance on a single search engine) could further improve clarity in terms of the framework’s scope.

Significance

The significance of NativQA is considerable, both in terms of immediate applications and future impact. Culturally and regionally relevant datasets are essential for deploying LLMs in real-world applications that involve diverse linguistic groups. By making both the NativQA framework and the MultiNativQA dataset publicly available, the authors enable the broader NLP community to explore and develop culturally aware models, especially for underrepresented languages. This dataset can potentially become a standard for future evaluations of LLMs on low-resource languages, encouraging the development of models that are better aligned with regional and cultural nuances. Additionally, the fine-tuning experiments with PEFT demonstrate a practical, resource-efficient method for enhancing LLMs with the MultiNativQA data, suggesting paths for further research in adaptive fine-tuning strategies.
summary: The paper introduces NativQA, a framework designed to create culturally and regionally aligned multilingual QA datasets for large language models (LLMs), addressing the lack of region-specific datasets generated by native users in their languages. This gap has limited the effectiveness of LLMs in real-world applications, particularly in low-resource and culturally rich languages. The framework includes a scalable, language-independent approach that combines human and machine collaboration for QA collection, enabling minimal human intervention. Key contributions of the paper are:

- NativQA Framework: A semi-automatic approach to build culturally specific QA datasets that enhances LLM inclusivity.
- MultiNativQA Dataset: A multilingual QA dataset with approximately 64,000 annotated pairs across seven languages, representing various resources (high to extremely low) and covering 18 topics from nine regions.
- Benchmarking: Evaluation of both open and closed-source LLMs on the MultiNativQA dataset.
- Parameter-Efficient Fine-Tuning: The use of PEFT techniques demonstrated the dataset's efficacy for enhancing LLMs' cultural and regional awareness.

The authors highlight that the dataset and framework fill a significant gap in creating LLM benchmarks that align with diverse linguistic and cultural contexts​
weaknesses: 1. Limited Scope in Query Collection Sources
Weakness: The paper relies heavily on a single search engine (Google) for QA collection, which may not always yield the culturally diverse content required for all languages and regions, particularly for low-resource languages. This reliance introduces potential biases, as Google’s indexing and search result prioritization may skew towards more widely available or commercial content, possibly missing nuanced, locally relevant knowledge.

Recommendation: Expanding the query sources to include regional search engines, local knowledge databases, or even social media platforms specific to target cultures would enhance the cultural alignment of the collected QA pairs. Alternatively, a comparison of QA pairs from different sources could strengthen the dataset’s diversity and reduce source bias, offering insights into the efficacy of different sources for culturally rich QA data.

2. Inconsistent QA Pair Quality in Low-Resource Languages
Weakness: The quality of QA pairs in extremely low-resource languages like Assamese and Nepali showed significant variability, with a notable decrease in usable QA pairs post-annotation. This limitation could reflect inherent challenges in capturing native content or the translation issues arising when the search engine retrieves results in more dominant languages.

Recommendation: To address this, a two-step QA collection approach could be beneficial: first, gather region-specific web content from trusted local sources in each language, then apply language detection tools to filter out non-native responses effectively. The use of pre-trained language-specific filters or bilingual annotators could improve the consistency and relevance of collected QAs, especially in low-resource settings where non-native responses are common.

3. Narrow Evaluation of Fine-Tuning Efficacy
Weakness: The fine-tuning evaluation, while demonstrating the dataset’s utility, is conducted on a single LLM (Llama-3.1-8B-Instruct) using parameter-efficient fine-tuning (PEFT) techniques. This limited evaluation scope might not provide a comprehensive view of how effectively MultiNativQA can adapt models of varying architectures and training paradigms to cultural contexts. For instance, closed-source models could behave differently compared to open-source models after fine-tuning.

Recommendation: Extending the fine-tuning experiments to a broader set of LLMs, including both transformer-based and non-transformer architectures, would enhance the robustness of the findings. Alternatively, conducting ablation studies that isolate the effects of PEFT could offer insights into the dataset’s specific impact on language and cultural alignment. If computational costs are prohibitive, an experiment with a smaller subset of the dataset across diverse models could still provide valuable insights.

4. Absence of Qualitative Cultural Alignment Analysis
Weakness: Although quantitative metrics (e.g., BLEU, ROUGE, F1 BERTScore) are valuable for measuring accuracy and fluency, they fall short in assessing how well the fine-tuned models respond to culturally nuanced or sensitive queries. An evaluation based solely on standard metrics might miss subtle cultural misalignments in the generated responses.

Recommendation: Incorporating a qualitative analysis where native speakers or cultural experts evaluate the responses for cultural appropriateness, sensitivity, and relevance could offer an additional layer of validation. This approach would complement the quantitative results, particularly for languages with rich cultural nuances that are difficult to measure with traditional metrics. An example might include a sample of questions from the MultiNativQA dataset, evaluated through a Likert scale for cultural alignment, with native speakers providing specific feedback on cultural appropriateness.

5. Ambiguity in Defining “Reliable” Sources for QA Pairs
Weakness: The paper mentions a domain reliability check, but it does not provide detailed criteria for how reliability was assessed or examples of sources that were classified under each reliability label. This lack of transparency could lead to inconsistency in determining the factuality of answers, especially for culturally specific knowledge, where conventional definitions of reliability may not apply.

Recommendation: Providing a clearer definition of reliability criteria and examples for each category (e.g., “very reliable,” “partially reliable”) would enhance reproducibility and make it easier to apply the framework to new languages and cultures. Additionally, discussing specific challenges or biases encountered in assessing domain reliability in low-resource languages would add depth to the paper’s contributions and offer a guide for future researchers working with culturally sensitive data.

6. Absence of Error Analysis for Model Limitations on Low-Resource Languages
Weakness: While the paper mentions qualitative error types (e.g., numerical errors, temporal inaccuracies), it lacks an in-depth error analysis that connects these errors with the limitations of the dataset or fine-tuning approach, especially for low-resource languages. Understanding where the models fall short could help guide future iterations of the dataset or model improvements.

Recommendation: A detailed error analysis that categorizes mistakes by language, cultural content, or query type (e.g., temporal questions, region-specific facts) would provide actionable insights. For instance, if numerical inaccuracy is prominent in certain language pairs, the analysis could propose dataset improvements or filtering strategies for factual consistency. This analysis could also help identify specific areas where the fine-tuned model struggles, offering valuable direction for future research in adapting LLMs to culturally specific information.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: * Why only deduplicate synthetic queries with exact matches?
* Was Google always accessed from the region in question or using VPN, to ensure geographic consistency?
* Many of the questions in your example tables are time sensitive. Do you consider keeping time stamps with all these questions, in case the answers change?
soundness: 2
strengths: * The paper is well written and has clear contributions—especially their method of scalably collecting QA pairs, by leveraging Google’s “People also asked” feature.
* The authors take care to verify the correctness of the domains, the questions, and the answers, in a multi-stage verification process. This attention to detail is notable.
* The authors address a key problem in collecting better training and evaluation data for languages that have less resources.
* The manual qualitative analysis of errors from major models is quite useful, and an important set of insights.
summary: The authors present a strategy to scalably collect QA pairs for lower resource languages, by leveraging Google’s “People also asked” feature. MultiNativQA spans 64k QA pairs that undergo manual verification, and are used as a resource for culturally-relevant training and evaluation.
weaknesses: * Invariably, the naturalness of the questions may be skewed by the seed set, which is constructed by annotators who are asked to think of natural questions for their region. To some extent this is artificial, but also a difficult hurdle to circumvent. The authors could strengthen the work by listing how many seed authors there were for each language, to illustrate the extent that there was diversity of thinking.
* It isn’t clear the extent to which the information in the dataset is novel to the tested models, because BERT-Score, ROUGE, and BLEU are fairly unreliable scores, especially multilingually. It would have been stronger to conduct human reviews of the answers and calculate the correlations with automatic scores, to give an idea of whether they are reliable.

Meta-review: All the reviewers pointed to the quality of the paper in terms of the presentation and point to usefulness of the dataset in general sense of having an extra multilingual benchmark. However, I agree with some of the reviewers that it is not clear what this new dateset is evaluating that give us more insight on weakness of LLMs. Specially since some of the models score 80%+ F1 score across all languages on the provided benchmark already.

Decision: Reject