Cross-lingual Open-Retrieval Question Answering for African Languages
Authors: Odunayo Ogundepo
Tajuddeen Gwadabe
Clara E. Rivera
Jonathan H. Clark
Sebastian Ruder
David Ifeoluwa Adelani
Bonaventure F. P. Dossou
Abdou Aziz DIOP
Claytone Sikasote
Gilles HACHEME
Happy Buzaaba
Ignatius Ezeani
Rooweither Mabuya
Salomey Osei
Chris Chinenye Emezue
Albert Kahira
Shamsuddeen Hassan Muhammad
Akintunde Oladipo
Abraham Toluwase Owodunni
Atnafu Lambebo Tonja
Iyanuoluwa Shode
Akari Asai
Anuoluwapo Aremu
Ayodele Awokoya
Bernard Opoku
Chiamaka Ijeoma Chukwuneke
Christine Mwase
Clemencia Siro
Stephen Arthur
Tunde Oluwaseyi Ajayi
Verrah Akinyi Otiende
Andre Niyongabo Rubungo
Boyd Sinkala
Daniel Ajisafe
Emeka Felix Onwuegbuzia
Falalu Ibrahim Lawan
Ibrahim Said Ahmad
Jesujoba Oluwadara Alabi
CHINEDU EMMANUEL MBONU
Mofetoluwa Adeyemi
Mofya Phiri
Orevaoghene Ahia
Ruqayya Nasir Iro
Sonia Adhiambo
Venue: EMNLP 2023 Findings (2023)
Keywords: African Languages
Question Answering
Information Retrieval
Low-resource Languages

Abstract: African languages have far less in-language content available digitally, making it challenging for question answering systems to satisfy the information needs of users. Cross-lingual open-retrieval question answering (XOR QA) systems -- those that retrieve answer content from other languages while serving people in their native language---offer a means of filling this gap. To this end, we create Our Dataset, the first cross-lingual QA dataset with a focus on African languages. Our Dataset includes 12,000+ XOR QA examples across 10 African languages. While previous datasets have focused primarily on languages where cross-lingual QA augments coverage from the target language, Our Dataset focuses on languages where cross-lingual answer content is the only high-coverage source of answer content. Because of this, we argue that African languages are one of the most important and realistic use cases for XOR QA. Our experiments demonstrate the poor performance of automatic translation and multilingual retrieval methods. Overall, Our Dataset proves challenging for state-of-the-art QA models. We hope that the dataset enables the development of more equitable QA technology.

AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languages
Odunayo Ogundepo + Tajuddeen R. Gwadabe" Clara E. Rivera", Jonathan H. Clark?, Sebastian Ruder", David Ifeoluwa Adelani Bonaventure F. P. Dossou" 4,5,6,
Abdou Aziz DIOP"*, Claytone Sikasote 10,, Gilles Hacheme Happy Buzaaba 16,x
Ignatius Ezeani Rooweither Mabuya Salomey Osei", Chris Emezue 13,+
Albert Njoroge Kahira 17.*, Shamsuddeen Hassan Muhammad 18,31, 18,31,+
Akintunde Oladipo' Abraham Toluwase Owodunni", Atnafu Lambebo Tonja 12,6,, Iyanuoluwa Shode
Akari Asais, Tunde Oluwaseyi Ajayi ,Clemencia 20,+ Steven Arthur
Mofetoluwa Adeyemi +t, Orevaoghene Ahia 8* AAuoluwapo Aremu", Oyinkansola Awosan", Chiamaka Chukwuneke', Bernard Opoku 22.x Awokoya Ayodele 23,x, Verrah Otiende
Christine 2wase25, Boyd Sinkala 10.. Andre Niyongabo Rubungo 26,+, Daniel A. Ajisafe
Emeka Felix Onwuegbuzia 23,-, Habib Mbow22.* Emile Niyomutabazi Eunice Mukondel0,
Falalu Ibrahim Lawan 30,+ Ibrahim Said Ahmad 31,* Jesujoba O. Alabi
Martin Namukombo Mbonu Chinedu Mofya Phiri Neo Putini Ndumiso Mngoma 3l, Priscilla A. Amuok', Ruqayya Nasir Iro Sonia Adhiambo
Masakhane NLP, University of Waterloo, Canada, "Gongle Rescarch, University College London,
Mila Quebee AI Institute, McGill University, "Lelapa AI, 'GalsenA,,
University Washington, "Ai4Innov, University of Zambia, Montclair State University,
Institutoo Politécnico Nacional, Mexico, Toeechnica University of Munich, LLancaster University.
16 RRIKNN Center for AIP, 18 South African Centre for Digital Language Resources, !7 JJulich Supercomputing Centre,
University of Porto, Insight Centre for Data Analyties, University of Amsterdam, Accra Institute of Technology.
22 Kwame Nknumah University of Science and Technology, University Thadan, 2Toom Mhoya University
25 Fudan University." 26 University Electronic Science and Technology China, The University British Columbia,
28 African Master in Machine Intelligence," 29 College de Rebero. 30 Kaduna State University. 31 Bayero University Kano
Saarland University, Germany, University of Edinburgh." 94 Kenyatta University." 2 Nnamdi Azikiwe University
Abstract
lation and multilingual retrieval methods. Over- all, AFRIQA proves challenging for state-of- the-art QA models. We hope that the dataset enables the development of more equitable QA technology.
African languages have far less in-language content available digitally, making it challeng- ing for drosstio-ansseerig systems satisfy the information needs of users. Cross-lingual open-retrieval question answering (XOR QA) systems -yste that retrieve answer content from other languages while serving people in their native language-offer means of fill- ing this gap. To this end, we create AFRIQA, the first cross-lingual QA dataset with fo- African languages. AFRIQA includes 12,000+ XOR QA examples across 10 African languages. While previous datasets have fo- cused primarily languages where cross- lingual QA augments coverage from the tar- get language. AFRIQA focuses on languages where cross-lingual answer content is the only high-coverage source of answer content. Be- cause of this, we argue that African languages are one of the most important and realistic use cases for XOR QA. Our experiments demon- strate the poor performance of automatic trans-
Introduction
Question Answering (QA) systems provide access to information (Kwiatkowski et al., 2019) and in- crease accessibility in range of domains, from healthcare and health emergencies such as COVID- 19 (Möller et al., 2020; Morales et al., 2021) to legal queries (Martinez-Gil, 2021) and financial questions (Chen et al., 2021). Many of these appli- cations are particularly important in regions where information and services may be less accessible and where language technology may thus help to re- duce the burden on the existing system. At the same time, many people prefer to access information in their local languages or simply do not speak
'The data is available at: https://gtthh..c wttpgghhhhh..oo//aaaaaaaaeeeeeee http://aaaaaaaaaaaiiiiii and https://huggingfice htt.e///ttttttttssaaaaaaaaaaaaaa
Equal contribution.

---

Dataset
QA? CLIR?
Open Retrieval?
Languages
African Languages
XQA (Liiu et al., 2019)
9
XOR QA (Asai et al., 2021)
7
Nil
XQuAD (Artetxe et al., 2020)
X
X
11
Nil
MLQA (Lewis et al., 2020)
X
X
7
Nil
MKQA (Longpre et al., 2021)
X
/
26
Nil
TyDi QA (Clark et al., 2020)
X
J
11
AmQA (Abedissa et al., 2023)
X
X
1
KenSwQuAD (Wanjawa et al., 2023)
X
X
AFRIQA (Ours)
(sce Table 3)
Table 1: Comparison of the Dataset with Other Question Answering Datasets. This table provides comparison of the current dataset used in the study with other related datasets. The first, second, and third columns, "QA", "CLIR", and "Open Retrieval", indicate whether the dataset is question answering, cross-lingual or open retrieval, respectively. The fourth column, "# Languages", shows the total number of languages in the dataset. The final column lists the African languages present the dataset.
language supported by current language technolo- gies (Amano et al., 2016). To benefit the more than three billion speakers of under-represented languages around the world, it is thus crucial to enable the development of QA technology in local languages.
language (either English or French) based on its prevalence the region corresponding to the query language. AFRIQA includes 12,000+ examples across 10 languages spoken in different parts of Africa. The majority of the dataset's questions are centered around entities and topics that closely linked to Africa. This is advantage over simply translating existing datasets into these languages. By building dataset from the ground up that is specifically tailored African languages and their corresponding cultures, we are able to ensure better contextual relevance and usefulness of this dataset.
Standard QA datasets mainly focus on En- glish (Joshi et al., 2017; Mihaylov et al., 2018; Kwiatkowski et al., 2019; Sap et al., 2020). While some reading comprehension datasets are available in other high-resource languages (Ruder and Sil, 2021), only few QA datasets (Clark et al., 2020; Asai et al., 2021; Longpre et al., 2021) cover ty- pologically diverse of languages-ann and very few datasets include African languages (see Table 1).
We conduct baseline experiments for each part of the open-retrieval QA pipeline using different translation systems, retrieval models, and multilin- gual reader models. We demonstrate that cross- lingual retrieval still has a large deficit compared to automatic translation and retrieval; we also show that hybrid approach of sparse and dense retrieval improves over either technique in isolation. We highlight interesting aspects of the data and discuss annotation challenges that may inform future an- notation efforts for QA. Overall, AFRIQA proves challenging for state-of-the-ar QA models. We hope that AFRIQA encourages and enables the de- velopment and evaluation of more multilingual and equitable QA technology. The dataset will be leased under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.
In this work, we lay the foundation for research on QA systems for one of the most linguistically diverse regions by creating AFRIQA, the first QA dataset for 10 African languages. A focuses on open-retrieval QA where iitormattoonscekinn questions" are paired with retrieved documents in which annotators identify an answer one is avail- able (Kwiatkowski et al., 2019). As many African languages lack high-quality in-language content online, AFRIQA employs a linss-lingual setting (Asai et al., 2021) where relevant passages retrieved in high-resource language spoken in the corresponding region and answers are trans- lated into the source language. To ensure utility of this dataset, we carefully select a relevant source
In summary, we make the following contribu- tions:
These questions are information-seeking in that they are written without seeing the answer, as is the case with real users of QA systems. We contrast this with the reading com- prehension task where the question-writer sees the answer passage prior to writing the question; this genre of questions tends to have hoth higher lexical overlap with the question and clicit questions that may not be of broud interest.
We introduce the first cross-lingual ques- tion answering dataset with 12,000+ ques- tions across 10 geographically diverse African languages. This dataset directly addresses

---

lang
Question QL
Relevant Passage Pa
Answer Agt
(Franslation Que)
(Transiation Ar.)
Jahohi kasar Malaysia? banga? (How many states are there in Malaysia?)
The and federal territories of Malaysia the principal administrative divisions of Malaysia. Malaysia federation of 13 states (Negeri) and 3 foderal territories.
hau
13 (13)
Mwanawasa stadium ingisha abantu banga? (What is the capaciry Mwanawasa Stadium?)
The Levy Mwanawasa Stadium is multi- purpose stadium Ndolu. Zambia. used mostly for football matches. The stadium has capacity of 49,800 people.
49,800 people (Abantu 49800)
Man po moo niroo ag powum Softbal? (Quel sport ressemble beaucoup softball?)
Ce sport est un descendant direct du baseball (afin de différencier les deux) mais diffère de ce dernicr par différents uspects dont cinq principaux sont les suivants.
bascball (Bas-bal)
wol
Table 2: Table showing selected questions, relevant passages, and answers in different languages from the dataset. It also includes the human-trane translated versions of both questions and answers. For the primary XOR QA task, systems are expected to find the relevant passage among all Wikipedia passages, not simply the gold passage shown above.
the deficit of African languages in existing datasets.
mon question words in Bemba include "cinshi" (what), "waoni"(woo), "liisa" (when), "mulandun- shi" (why), "ciisa" (which), "kwi/kwiisa" (where), and "shaani" (how).
We conduct an analysis of the linguistic prop- erties of the 10 languages, which is crucial to take into account when formulating questions in these languages.
Fon is isolating language in terms of morphol- ogy typology. Common question wh-words in Fon Eté(what), Me (who), Hweténu (when), Aniwú (why), de te (which) and Fitz (where).
Finally, we conduct comprehensive evalua- tion of the dataset for each part of the open- retrieval QA pipeline using various translation systems, retrieval models, and multilingual reader models.
Hausa is the only Afro-Asiatic language in A ARIQA. typically makes use of indicative words for changes to the grammatical forms within a sentence, such as negation, tenses, and plural- ity. For example, "hula" (cap) "huluna" (caps), "mace" (girl) "mataye" (girls). Typical question wh-words used are "melya" (what), "wa"(who), "yaushe" (when), "dan me/akan me" (why), "wanne" (which), "ina/ ina" (where), and ""yayaqqaq"" (how).
2 AFRIQA
AFRIQA is a cross-lingual QA dataset that was created to promote the representation and cover- age of under-resourced African languages in NLP research. We show examples of the data in Table 2. In $2.1, we provide an overview of the 10 lan- guages discussing their linguistic properties, while $2.2 and $2.3 describe the data collection proce- dure and quality control measures put place to ensure the quality of the dataset.
Igbo is morphologically rich language and most changes in grammatical forms (negations, ques- tions) can be embedded a single word or by varying the tone. Question words are often pre- ceded by "kedu" or "gini" like "kedu/gini" (what), "onyee/kedu onye" (who), "kedu mgbe" (when), "gini mere/gini kpatara" (why), "kedu nke" (which), "ebee" (where), and "kedu ka" or "kedu etu" (how).
2.1 Discussion of Languages
African languages have unique typologies, gram- matical structures, and phonology, many of them being tonal and morphologically rich (Adelani et al., 2022b). We provide high-level overview of the linguistic properties of the ten languages in AFRIQA that are essential consider when craft- ing questions for QA systems.
Kinyarwanda is morphologically rich language with several grammatical features such negation, tenses, and plurals that are expressed changes to morphemes in a word. Question words typically used are "iki" (what), "nde/inde" (who), 'ryari"" " (when), "ikihe/uwuhe" (which), "hehe" (where), and "gute" (how).
Bemba, a morphologically rich Bantu language, uses affixes to alter grammatical forms. Com-
Swahili a morphologically rich language that

---

typically has several morphemes incorporate changes grammatical forms such as negation, tenses and plurality. A question word can be placed at the beginning or end of the question sentence, for example, "amekuja nani?" (who has come?) and "nani amckuja?" (who has come?). Other question words often used are "nini" (what), "nani" (who), "lini" (when), "kwanini" (why), "wapi", (where), and "vipi" (how).
2.2.1 Question Elicitation
The TyDi QA methodology (Clark et al., 2020) was followed to elicit locally relevant questions. Team members were presented with prompts in- cluding the first 250 characters of the most popular Wikipedia articles in their languages, and asked to write factual or procedural questions for which the answers were not contained in the prompts. Annotators were encouraged to follow their natu- ral curiosity. This annotation process avoids ex- cessive and artificial overlap between the ques- tion and answer passage, which can often arise in data collection efforts for non-informmation-sceinng QA tasks such reading comprehension." For languages like Fon and Bemba without dedi- cated Wikipedia, relevant prompts from French and English Wikipedia were used to stimulate na- tive language question generation. For Swahili, unanswered TyDi QA questions were curated for correctness. The inability of the original TyDi QA team locate suitable Swahili paragraph to an- swer these questions necessitated their inclusion. Simple spreadsheets facilitated this question elici- tation process.
Twi is dialect of the Akan language and AFRIQA includes the Asante variant. A few common ques- tion words used in Twi are "wdeen"(what), "hwan" (who), "daben" (when), "aden" (why), "deehen" (which), "ehenfa" (where), "sen" (how).
Wolof is an agglutinative language, and unlike other Bantu languages, it utilizes dependent words rather than affixes attached to the headwords for grammatical modifications. Common ques- tion words in Wolof include "ian" (what), "kan" (who), "kafi" (when), "lu tax", "ban" (which), "fan" (where), and "naka" (how).
Yorùbá has a derivational morphology that en- tails affixation, reduplication, and compounding. Yorùbá employs polar question words such as "nje", "se", "abi", "sebi" (for English question words "do" or "is", "are", "was" or "were") and con- tent question markers such as "tani" (who), "kini" (what), "nibo" (where). "elo/meloo" (how many). "bawo" (hhw""(owwinn is), "kilode" (why), and "igba/nigha" (when). Negation can be expressed with "kò".
Before moving on to the second stage, team co- ordinators reviewed elicited questions for grammat- ical correctness and suitability for the purposes of informatiin--seeking QA.
2.2.2 Question Translation
Zulu is very morphologically-ricc language where several grammatical features such as tense, negation, and the plurality of words are indi- cated through prefixes or suffixes. The most com- monly used question words in Zulu are "yini" (what), "ubani"" (who), "nini" (when), "kungani" (why), "yiliphi" (which), "kuphi" (where), "kan- jani" (how), and "yenza" (do).
Blicited questions were translated from the original African languages into pivot languages following Asai et (2021). English was used as the pivot language across all languages except Wolof and Fon, for which French was used. Where possible, questions clicited by one team member were allo- cated to different team member for translation to further ensure that only factual or procedural questions that grammatically correct make it into the final dataset. This serves as an additional validation layer for the elicited questions.
2.2 Data Collection Procedure
For each of the 10 languages in AFRIQA, a team of 2-6 native speakers was responsible for the data collection and annotation. Each team was led by coordinator. The annotation pipeline consisted of 4 distinct stages: 1) question elicitation in an African language; 2) translation of questions into a pivot language: 3) answer labeling in the pivot language based on set of candidate paragraphs; and 4) an- swer translation back to the source language. All data contributions were compensated financially.
Phttps: //ww. www.pedia.aooomm org/
'Reading comprehension differs from information-seekian QA as dnestion-wwitere see answer prior to writing the question and thus tests understanding of the answer text rather than the general ability provide correct answer. FFench is widely used in the regions where Fon and Wolof are spoken, so there may be a higher prohability of finding answers in French than in other pivot languages.

---

2.2.3 Answer Retrieval
Each language team comprised minimum of three members, with Fon and Kinyarwanda teams as exceptions, hosting two members each. This structure was designed to ensure different team members handled question elicitation and trans- lation, enhancing quality control. Non-factual or inappropriate questions were flagged during the translation and answer labeling phases, leading to their correction removal. 'eam coordina- tors meticulously reviewed all question-and-anse pairs alongside their translations, while central managers checked translation consistency. Post- annotation controls helped rectify common issues such as answer-span length and incorrect answer selection.
Using the translated questions as queries, Google Programmable Search Engine was used to retrieve Wikipedia paragraphs that are candidates to con- tain an answer in the corresponding pivot language. The Mechanical Turk interface was employed- annotations were carried out by team members. used to show candidate paragraphs to team members who paragraph that contains an answer and 2) the exact minimal span of the answer. the case of polar questions, team members had to select "Yes" or "No" instead of the minimal span. In cases where candidate paragraphs did not contain the answer to the corresponding question, team members were instructed to select the "No gold paragraph" option. As with question elicitation, team members went through phase of training, which included group meeting where guidelines were shared and anno- tators were walked through the labeling tool. Two rounds of in-tool labeling training were conducted.
then asked to identify 1) the
2.4 Final Dataset
The statistics of the dataset are presented in Table 3, which includes information on the languages, their corresponding pivot languages, and the total num- ber of questions collected for each language. The final dataset consists of a total 12,239 questions across 10 different languages, with 8,892 corre- sponding question-answer pairs. We observed a high answer coverage rate, with only 27% of the to- tal questions being unanswerable using Wikipedia. This be attributed the lack of relevant infor- mation Wikipedia, especially for Africa related entities with sparse information. Despite this spar- sity, we were able to find answers for over 60% of the questions in most the languages in our collection.
2.2.4 Answer Translation
To obtain answers in the African languages, we translated the answers in the pivot languages to the corresponding African languages. We allocated the task of translating the answers labeled by team members to different team members in order to ensure accuracy. Translators were instructed to minimize the span of the translated answers. In cases where the selected answers were incorrect or annotators failed to select the minimum span, we either removed the question, corrected the answer, or re-annotated the question using the annotation tool.
3 Tasks and Baselines
As part of the evaluation for AFRIQA, we fol- low the methodology proposed in Asai et al. (2021) and assess its performance on three diffcrent tasks: XOR-Retrieve, XOR-PivotLanggageeSen and XOR-Ful1. Each task poses unique challenges for cross-lingual information retrieval and QA due to the low-resource nature of many African lan- guages.
2.3 Quality Control
We enforeed rigorous quality control measures throughout the dataset generation process to as- certain its integrity, quality, and appropriateness. Our strategy involved recruiting native language speakers as annotators and team coordinators. Each annotator underwent initial training question elic- itation via English prompts, with personalized feed- back focusing on factual question generation and avoiding prompt-contained answers. Annotators were required to achieve at least 90% accuracy to proceed to native language clicitation, with ad- ditional one-on-one training rounds provided as necessary.
Experiments
4.1 Translation Systems
A common approach cross-lingual QA is to trans- late queries from the source language into target language, which is then used to find an answer in a given passage. For our experiments, we explore the use of different translation systems as baselines for AFRIQA. We consider human translation, Google
'https://developeer wow.ee....e...ee.eee..oooooooo.. com//custon-searcc/

---

Source
Pivot
African
Native Speakers
Unanswerable
ISO
Script
Train
Dev
Test
Language
Language
Region
Questions
Bemha
bem
English
South, East & Central
Latin
4M
502
503
314
0.41
Fon fon
French
West
Latin
2M
427
428
386
0.22
Hausa
hau
Finglish
West
Latin
63M
435
436
300
0.36
Igbo
English
West
Latin
27M
417
409
0.18
Kinyarwanda
Fingishh
Central
Latin
15M
407
409
347
0.26
Swahili swa English
East Central
Latin
98M
415
302
0.34
Twi
English
West
Latin
451
490
0.12
Wolof wol French
West
Latin
503
0.38
Yorūbá
yor
English
West
Latin
42M
360
0.21
Zulu
zul
English
South
Latin
27M
387
388
325
0.26
Total
292M
4333
4346
3560
0.27
Table 3: Dataset information: This table contains key information about the AFRIQA Dataset
Translate, and open-source translation models such NLLB (NLLB Team et al., 2022) and finetuned M2M-100 models (Adelani al., 2022a) in zero- shot settings.
is classic term- cerm-frequency-baat retrieval model that matches queries relevant passages using the frequency of word occurrences both queries and passages. We use the BM25 implementation provided by Pyserini (Lin et al., 2021) with default hyperparameters kl = 0.9, b = 0.4 for all languages.
Google Translate. We use Google Translate be- cause it provides out-of-the-box translation for 7 out of 10 languages in our dataset. Although Google Translate provides strong translation base- line for many of the languages, we cannot guaran- tee the future reproducibility of these translations as it is product API and is constantly being up- dated. For our experiments, we use the translation system of February 2023? ?
mDPR. We evaluate the performance mDPR, a multilingual adaptation of the Dense Passage Re- triever (DPR) model (Karpukhin et al., 2020) using multilingual BERT (mBERT), We finetuned mDPR the MS MARCO passage ranking dataset (Bajaj et al., 2018) for our experiments. Retrieval is per- formed using the Faiss Flat Index implementation provided by Pyserini.
NLLB. NLLB is an open-source translation sys- tem trained on 100+ languages and provides trans- lation for all the languages in AFRIQA. At the time of release, NLLB provides ofte--ofttheart translation in many languages and covers all the languages in our dataset. For our experiments, we use the 1.3B size NLLBB models.
Sparse-Dense Hybrid. We also explore sparse- dense hybrid baselines, a combination of sparse (BM25) and hybrid (mDPR) retrievers. We use a linear combination of both systems to generate reranked list of passages for each question.
4.3 Answer Span Prediction
To benchmark models' answer selection capabili- ties on AFRIQA, we combine different translation. extractive, and generative QA approaches.
4.2 Passage Retrieval (XOR-Retrieve)
We present two baseline retrieval systems: translate-retrieve and cross-lingual baselines. In the rretrevt bascline, we first translate the queries using the translation systems described in 1..1. The translated queries are used to retrieve relevant passages using different retrieval systems outlined below. Alternatively, the cross-lingual baseline directly retrieves passages in the pivot lan- guage without the need for translation using mul- tilingual dense retriever.
Generative QA on Gold Passages. To evalu- ate the performance of generative QA, we utilize mT5-base (Xue et al., 2021) finetuned on SQuAD 2.0 (Rajpurkar et al., 2016) and evaluate it using both translated and original queries. The model was provided with the queries and the gold pas- sages that were annotated using template prompt and generates the answers the questions.
Extractive QA on Retrieved Passages. For XOR-Pi XOR-PivotL votLanguageSSann baselines, we employed an extractive QA model that extracts the answer span from the retrieved passages produced by the
BM25. BM25 (Robertson and Zaragoza, 2009)
'Noe that while Google Translate supports 133 languages, it does not include Bemha, Fon, nor Wolof.

---

Human Translation
GMT BM25
NLLB
M2M-100
Crosslingual
lang
BM25
mDPR
Hybrid
mDPR
BM25
mDPR
BM25
mDPR
mDPR
Recall 10
bem
55.7
67.5
72.3
52.2
59.8
14.7
fon
66.3
69.4
70.7
43.9
48.7
39.9
43.3
28.5
hau
58.0
65.7
72.7
53.3
60.3
52.0
59.7
36.7
44.3
13.7
igb
70.4
74.3
82.9
65.5
71.2
64.8
68.0
62.1
67.5
25,4
kin
59.1
66.3
75.5
53.6
61.1
53.0
58.8
15.6
46,0
61.9
67.6
45,0
60.9
43.1
58.3
39.1
$4.6
20.9
twi
61.8
66.7
75.3
56.1
58.0
50.4
54.1
45.7
49.4
21.4
wol
61.4
67.7
68.6
35.0
36.5
34,4
35.0
13.8
yor
55.1
66.6
71.7
52.1
59.0
50.9
57.5
36.8
35.5
21.4
zul 59.7
70.2
76.3
57.2
66.2
51.5
64.6
45.5
60.0
14.2
avg 59.4
67.6
73.4
54.7
62.4
49.7
56.6
42.5
48.7
19.0
Table 4: Retrieval Recall @ 10: This table displays the retrieval recall results for various translation types the test set of AFRIQA. The table shows the percentage retrieved passages that contain the answer for the top-10 retrieved passages. The last column represents crosslingual retrieval, where skip the translation step and the original queries. We boldface the best-performin model for each language within the human translation oracle scenario and within the real-world automatic translation scenario.
GMT
NLLB
Crosslingual EM
HT
GMT
NLLB
Crosslingual
F1
EM
FI
EM
F1
EM
F1
EM
FI
EM
F1
EM
F1
EM
bem
48.8
41.7
38.5
32.0
38.2
29.5
30.0
21.9
0.4
0.4
41.4
28.5
23.4
15.3
fon
53.8
40.4
37.5
26.7
13.4
6.0
hau
58.5
49.0
53.5
45.7
50.9
25.8
22.3
hau
52.7
47.7
50.9
43.7
27.7
23.7
66.6
59.2
$9.8
60.2
ibo
55.0
62.8
S6.2
29.2
24.7
60.8
43.8
57.3
40.9
58.8
42.9
kin
36.0
36.6
22.7
17.9
swa
52.3
42.6
48.9
40.8
49.2
41.2
29.4
swa
37.9
45.2
38.1
31.6
24.6
55.4
45.3
42.0
33.7
40.1
33.1
twi
41.8
31.1
34.3
30.0
3,4
2.5
wol
44.6
36.1
21.8
16.9
3.9
wol
45.2
33,9
33.2
26.0
1.8
0,9
yor
54.9
49.8
48.9
45.1
47.9
43.0
11.9
yor
36.0
31.7
32.3
28.0
3.8
zul
60,2
50.8
57.4
48.9
55.6
46.5
24.7
20.9
zul
59.1
49.2
56.0
48.6
53.6
45.8
17.0
13.5
avg
54.5
44.7
46.0
38.6
44.6
36.7
17.6
13.9
avg
52.4
42.4
42.9
36.0
43.1
35.3
15.3
11.8
Table 5: Generative Gold Passages Answer Predic- Comparison of Fl and Exact Match Accuracy scores for generative answer span prediction on test set using m-s-bbse (Xue et al., 2020) the backbone.
Table Extractive Gold Passages Answer Predic- tion: Comparison of Fl and Exact Match Accuracy scores extractive answer span prediction on the test using AfroXLMR- (Alabi et al., 2022) as the backbone.
tion:
various retrieval baselines outlined in $4.2. The model is trained to extract answer spans from each passage, along with the probability indicating the likelihood of each answer. The answer span with the highest probability is selected as the correct an- swer. We trained multilingual DPR reader model, which was initialized from mBERT and finetuned on Natural Questions (Kwiatkowski et al., 2019).
using different translation and retrieval systems. We also report the performance with both original and
queries. The table shows that hybrid retrieval using human translation yields the best results for all languages, with an average recall 10 of 73.9. In isolation, mDPR retrieval outperforms BM25 for all translation types. This table also enables to compare the effectiveness of different translation systems in locating relevant passages for cross -ingull QA in Aftican languages. This is illustrated in Figure 1, showing retriever call rates for different translation types at various
5 Results and Analysis
5.1 XOR-Retrieve Results
We present the retrieval results for recall @ 10 in Table 8, The table includes retriever results
Foor recall k retrieval results, we assume that there is only one gold passage despite the possibility of other retrieved
passages containing the answer.

---

Pivot Language Span F1
Query
Average
Translation
Retrieval
bem
fon
hau
ibo
kin
twi
wol
zul
F1
EM
HT
BM25
29.2
11.4
31.4
43.0
33.8
24.3
38.4
15.4
28.9
32.8
28.9
19.9
HT
mDPR
32.5
11.0
35.8
44.8
35.4
28.2
40.7
14.7
31.7
36.5
31.1
21.5
HT
Hybrid
34.7
11.3
35.5
46.1
39.2
27.5
41.8
16.2
32.4
34.6
32.0
21.9
GMT
BM25
21.0
38.6
28.3
24.7
27.7
21.7
31.6
27.7
21.2
GMT
mDPR
31.5
39.3
35.3
29.1
31.1
22.9
36.0
32.2
22.3
NLLB
BM25
23.8
3.6
24.6
37.6
29.3
25.2
25.7
4,4
17.3
26.8
19.8
13.8
NLLB
mDPR
24.1
5.1
27.2
39.6
33.3
25.9
28.2
5.2
21.4
30.4
24.0
16.0
Table 7: Fl scores on pivot language answer generation using extractive multilingual reader model with different query translation and retrieval methods.
Translation
XOR-Full F1
Average
Query
Answer
Retrieval
bem
fon
hau
ibo
kin
swa
wol
yor
EM
BLEU
GMT
GMT
BM25
20.4
30.4
24.2
18.1
14.9
16.1
19.7
20.5
12.1
18.3
GMT
GMT
mDPR
21.7
33.0
26.5
21.9
16.5
14.2
20.4
21.1
23.0
14.2
20.7
NLLB
NLLB
BM25
13.6
2.6
17.5
26.5
19.9
19.2
18.4
3.2
12.7
12.5
14.6
7.5
12.9
NLLB
NLLB
mDPR
13.3
4.3
19.3
29.9
22.4
20.3
19.5
3.5
17.6
13.1
16.3
8.3
14.3
Table 8: XOR-Full Fl results combining different translation and retriever systems.
cutoffs using mDPR.
We observe that human translation yields better accuracy than all other translation types, indicating that the current state-of of-the-art machine translation systems still have long way go in accurately translating African languages. Google Translate shows better results for the languages where available, while the NLLB model provides better coverage. The cross-lingual retrieval model that retrieves passages using questions in their origi- nal language is the least effective of all the model types. This illustrates that the cross-lingual rep- resentations learned by current retrieval methods are not yet of sufficient quality enable accurate retrieval across different languages.
0.7
Crosslinual GHT
Recall
0.5
0.4
20
passages
Figure 1: Graph of retriever recall @ k for different trans- lation systems. The scores shown in this graph are from mDPR retrieval.
5.2 XOR-PivoLLannuugeepan Results
top-10 retrieved passages for each question using an extractive multilingual reader model (see $4.3). The model assigns a probability to each answer span, and we select the answer with the highest probability as the final answer.
Gold Passage Answer Prediction. We first evalu- ate the generative QA setting using gold passages. We present F1 and Exact Match results using differ- ent methods translate the query in Table 5. Hu- man translation of the queries consistently outper- forms using performs using queries their original language.
Our results show that hybrid retrieval using human-translatee translated queries achieves the best perfor- mance across all languages on average. Using Human-translaie queries generally outperforms using translations by both Google Translate and NLLB, regardless of the retriever system used. In terms of retrieval methods, mDPR generally per- forms better than BM25, with an average gain of 3 Fl points across different translation types. These
queries, which out-
Retrieved Passages Answer Prediction. We now evaluate performance using retrieved passages from 1. We present F1 and Exact Match results combinations in Table 7. We extract the answer spans from only the
with different

---

results highlight the importance of carefully select- ing translation rettreee combinations to achieve the best answer span prediction results in cross- lingual QA.
data sources for low-resource languages, cross- lingual datasets (Liu al., 2019; Asai al., 2021) emerged that translate between low-resource and high-resource languages, thus providing access to a larger information retrieval pool which decreases the fraction of unanswerable questions. Despite these efforts, however, the inclusion of African languages remains extremely rare, shown in Ta- ble I, which compares dataset to other closely related QA datasets. TyDi QA features Swahili the sole African language out of the 11 languages covers.
5.3 XOR-Full Results
Each pipeline consists of components for question translation, passage retrieval, answer extraction, and answer translation. From Table 8, we observe that Google machine translation combined with mDPR is the most effective. This is followed by a pipeline combining NLLB translation with mDPR.
6 Related Work
7 Conclusion
Africa NLP. In parallel with efforts to include more low-resource languages in NLP research (Costa- jussà et al., 2022; Ruder, 2020), demand for NLP that targets African languages, which rep- resent more than 30% of the world's spoken lan- guages (Ogueji et al., 2021) is growing. This has resulted in the creation of publicly available mul- tilingual datasets targeting African languages for a variety of NLP tasks such as sentiment analy- sis (Muhammad et al., 2023; Shode et al., 2022), language identification (Adebara et al., 2022), data- to-text generation (Gehrmann et al., 2022), topic classification (Adelani et al.. 2023; Hedderich et al., 2020), machine translation (Adelani et al., 2022a; Nekoto al., 2020), and NER (Eiselen, 2016; Ade- lani et al., 2021, 2022b).
In this work, we take a step toward bridging the in- formation gap between native speakers of many African languages and the vast amount of digi- tal information available on the web by creating AFRIQA, the first open-retrieval cross-lingual QA dataset focused on African languages with 12,000+ questions. We anticipate that AFRIQA will help improve access relevant information for speak- ers of African languages. By leveraging the power of cross-linguus QA, we hope to bridge the infor- mation gap and promote linguistic diversity and inclusivity in digital information access.
Limitations
Our research focuses on using English and French Wikipedia as the knowledge base for creating sys- tems that can answer questions in 10 African lan- guages. While Wikipedia is comprehensive source of knowledge, does not accurately reflect all societies and cultures (Callahan and Herring, 2011), 1). There a limited understanding of African contexts with relatively few Wikipedia articles ded- icated to Africa related content. This could poten- tially limit the ability of a QA system to accurately find answers to questions related to African tradi- tions, practices, entities. Also, by focusing English and French and pivot languages, might introduce some translation inaccuracies ambigu- ities which might impact the performance of QA system.
Datasets for QA and Information Retrieval tasks have also been created. They are, however, very few and cater to individual languages (Abedissa et al., 2023; Wanjawa et al., 2023) or a small subset of languages spoken in individual coun- tries (Daniel et al., 2019; Zhang et al., 2022). Given the region's 's large number of linguistically diverse languages, multilingual and
and information cross-lingual datasets are encouraged to catalyze research efforts. To the best of our knowledge, there are no publicly available cross-lingual open- retrieval African language QA datasets.
Comparison to Other Resources. Multilingual QA datasets have paved the way for language mod- els to simultaneously learn across multiple lan- guages, with both reading comprehension (Lewis al., 2020) and other QA datasets (Longpre et al., 2021; Clark al., 2020) predominantly utilizing publicly available data sources such as Wikipedia, SQUAD, and the Natural Questions dataset. To ad- dress the information scarcity of the typically used
Addressing these limitations requires concerted efforts to develop more localized knowledge bases or improve existing sources such as Wikipedia by updating or creating articles that reflect the diver- sity and richness of African socicties.

---

References
Rijhwani, Sebastian Ruder, Stephen Mayhew, Is- rael Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nikkatumba-Nabendee Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yi- mam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Ruhungo Andre Niyongabo, Jonathan Mukiibi, Ver- rah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi. Gerald Muriuki, Emmanuel Anebi, Chiamaka Chuk- wuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akin- ode, Deborah Nabagereka, Maurice Katusiime, Ayo- dele Awokoya, Mouhamadane MBOUP, Dibora Ge- breyohannes, Henok Tilaye, Kelechi Nwaike, De- gaga Wolde, Abdoulaye Faye, Blessing Sibanda, Ore- vaoghene Abia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Sa- lomey Osei. 2021. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics. 9:1116-1131.
Tilahun Abedissa, Ricardo Usbeck, and Yaregal Assabie. 2023. Amqa: Amharic question answering dataset. arXiv preprint (xXiv:23033032900
Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Alcoba Inciarte. 2022. Afrolid: A neural language identification tool for african languages. arXiv preprint orXiv:2210.1774
David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajud- deen Gwadabe, Freshia Sackey, Bonaventure P. Dossou, Chris Emezue. Colin Leong, Michael Beuk- man, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Ben- jamin Ajibade. Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, An- uoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe. Derguene Mbaye. Allahsera Auguste Tapo, Victoire Memd- jokam Koagne, Edwin Munkoh-Buabeng, Valen- cia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda, Andiswa Bukula, and Sam Manthalu. 2022a. A few thousand trans- lations go a long way! leveraging pre-trained mod- els for African news translation. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies. pages 3053-3077 0053-0070. Seattle, United States. Association for Computational Linguistics.
David Ifcoluwa Adelani, Marek Masiak, Israel Abebe Azime. Jesujoba Oluwadara Alabi, Atnafu Lam- bebo Tonja, Christine Mwase, Odunayo Ogun- depo, Bonaventure F. Dossou, Akintunde Oladipo, Doreen Nixdorf, Chris Chinenye Emezue, Sana Sabah al azzawi, Blessing K. Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tunde Oluwaseyi Ajayi. Tatiana Moteu Ngoli, Brian Odhiambo. Abraham Toluwase Owodunni, Nnae- meka C. Obiefuna, Shamsuddeen Hassan Muham- mad, Saheed Salahudeen Abdullahi, Mesay Gemeda Yigezu, Tajuddeen Gwadabe, Idris Abdulmumin, Mahlet Taye Bame, Oluwabusayo Olufunke Awoy- omi, Iyanuoluwa Shode. Tolulope Anu Adelani, Habiba Abdulganiy Kailani, Oodul-Haaeem Omo- tayo, Adetola Adeeko, Afolabi Abeeb, Anuoluwapo Aremu, Olanrewaju Samuel, Clemencia Siro, Wan- gari Kimotho, Onyekachi Raphael Ogbu, Chinedu E. Mbonu, Chiamaka I. wuneke, Samuel Fanijo, Jessica Ojo, Oyinkansola F. Awosan, Tadesse Kebede Guge, Sakayo Toadoum Sari. Pamela Nyatsine, Freedmore Sidume, Oreen Yousuf, Mardiyyah Odu- wole, Ussen Kimanuka. Kanda Patrick Tshinu. Thina Diko, Siyanda Nxakama, Abdulmejid Tuni Johar, Sinodos Gebre, Muhidin Mohamed, Shafie Abdi Mohamed, Fuad Mire Hassan, Moges Ahmed Mehamed, Evrard Ngabire, and Pontus Stenetorp. 2023. Masakhanews: News topic classification for african languages.
David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen- Michel, Constantine Lignos, Jesujoba Alabi, Sham- suddeen Muhammad, Peter Nabende. Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe. Derguene Mbaye, Amelia Taylor, Fatoumata Ka- bore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez. Ogayo, Catherine Gitau, Edwin Munkoh- Buabeng. Victoire Memdjokam Koagne, Allah- sera Auguste Tapo, Tehogo Macuewa, Vukosi Mari- vate. Mboning Tchiaze Elvis. Tajuddeen Gwad- abe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabende: Neo Lerato Mokono, Ig- natius Ezeani, Chiamaka Chukwuncke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022h. MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition. In Proceedings of the 2022 Conference Empirical Methods in Nat- ural Language Processing, pages 4488 4588,450, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguisties.
Jesujoba Alabi, David Iffoluwa Adelani, Marius Mos- bach, and Dictrich Klakow. 2022. Adapting pre- trained language models to african languages via multilingual adaptive fine-tuning In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336-4349.
David Ifeoluwa Adelani, Jade Abbott, Graham Neu- big. Daniel D' souza. Julia Kreutzer, Constantine Lig- nos, Chester Palhn-Michel, Happy Buzaaba, Shruti
Tatsuya Amano, Juan P Varzá, and William J

---

Sutherland. 2016. Languages are still major barrier to global science. PLoS biology. 14(12):2000933.
EMNLP 2020. pages 1193-1208, Online. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transfcrability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623 4637, Online. Association for Computational Linguistics.
Roald Eiselen. 2016. Government domain named entity recognition for south african languages. In Proceed- ings of the Tenth International Conference on Lan- guage Resources and Evaluction (IREEC pages 1344-3348
Sebastian Gehrmann, Sebastian Ruder, Vitaly Nikolaev, Jan A Botha, Michael Chavinda, Ankur Parikh, and Clara Rivera, 2022. Tata: A multilingual table-to- text dataset for african languages. arXiv preprint Sifi:2211.00144.
Akari Asai, Jungo Kasai, Jonathan Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021. XOR QA: Cross-lingual open-retrieval question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- lional Linguistics: Human Language Technologies, pages 547-564, Online. Association for Computa- tional Linguisties.
Michael A Hedderich, David Adelani, Dawei Zhu. Je- sujoba Alabi, Udia Markus, and Dietrich Klakow. 2020. Transfer learning and distant supervision for multilinguul transformer models: A study on african languages. arXiv preprint arXiv:2010.03119.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, An- drew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. MS MARCO: A Human Gen- erated MAchine Reading COmprehension Dataset. 1.Xiv:661.0026630
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zeullemyee. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume I: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Ewa S. Callahan and Susan C. Herring. 2011. Cultural bias in wikipedia content on famous persons. J. Am. Soc. Inf. Sci. Technol., (2)))): (100:001)999990 1999-1915.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 1769-6781 Online. Association for Computational Linguistics.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A dataset of nu- merical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 3697-3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Ilaia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki, 2020. TyDi QA: A benchmark for cettrmmmttionseeeiine question answering in typo- logically diverse languages. Transactions of the As- sociation for Computational Linguistics, 8:454-470.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evalu- ating cross- mooss-liggual extractive question answering. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 15- 7330, Online. Association for Computational Lin- guistics.
Marta R Costa-jussă, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi. Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.
Jimmy Lin, Xueguang Ma, Sheng-Chiel Lin, Jheng- Hong Yang. Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), I). pages 2356-2362
Jeanne E Daniel, Willie Brink, Ryan Eloff, and Charles Copley. 2019. Towards automating healthcare ques- tion answering in a noisy multilingual low-resource setting. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 948-953.
Martin d' d'Hoffschmidt, Wacim Belblidia, Quentin Heinrich, Tom Brendlé, and Maxime Vidal. 2020. FQuAD: French question answering dataset. Find- ings of the Association for Computational Lingaistics:
Jiahua Liu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2019. XQA: A cross-lingual open-domain question answering dataset. In Proceedings of the 57th An- nual Meeting of the Association for Computational

---

Linguistics, pages 2358-2368, Florence, Italy, Asso- ciation for Computational Linguistics.
Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. Small data? no problem! exploring the viability of pretrained multilingual language models for low- resourced languages. In Proceedings of the 1st Work- shop on Multilingual Representation Learning, pages 116-126, Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics.
Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. MKQA: A linguistically diverse benchmark for mul- tilingual open domain question answering. Transac- tions of the Association for Computational Linguis-
tics, 9:9399-1406.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints. page urXiv16606.05250.
Jorge Martinez-Gil. 2021. A survey on legal arXiv preprint
question answering systems. arXiv:2110.07333.
Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends in Information Re- trieval, 344:3333-38.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can suit of armor conduct elec- tricity? new dataset for open book question swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium. Association for Computational Linguisties.
Sebastian Ruder. 2020. Why you should do nip be- yond english. Sebastian Ruder htpp:://rude.. io/nlp- beyond-english.
Timo Möller, Anthony Reina, Raghavan Jayakumar, and Malte Pietsch. 2020. COVID-QA: A question answering dataset for COVID-19. In Proceedings of the 1st Workshop NLP COVID-19 at ACL 2020. Online. Association for Computational Linguistics.
Sebastian Ruder and Avirup Sil. 2021. Multi-domain multilingual question answering. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing: Tutorial Abstracts. pages 17-21.
Maarten Sap. Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. 2020. Commonsense reason- ing for natural language processing. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 27-33, Online. Association for Computational Lin- guistics.
Hugo MP Morales, Murilo Guedes, Jennifer S Silva, and Adriano Massuda. 2021. Covid-19 in brazil- preliminary analysis of response supported by artificial intelligence in municipalities. Frontiers in Digital Health, 3:648585.
Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, Nedjma Ousidhoum, David Ife- oluwa Adelani, Seid Muhie Yimam, Ibrahim Sa'id Ahmad, Meriem Beloucif. Saif Mohammad, Sebas- tian Ruder, et al. 2023. Afrisenti: A twitter sentiment analysis benchmark for african languages. arXiv preprint arXiv:2302.0800
Tyanuoluwa Shode, David Ifeoluwa Adelani, and Anna Feldman. 2022. yosm: new yoruba sentiment corpus for movie reviews.
Manveer Singh Tamber, Ronak Pradeep, and Jimmy Lin. 2023. Pre-processing matters! improved wikipedia corpora for open-domain question answering. In 45th European Conference on Information Retrieval, ECIR 2023, page Forthcoming.
Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Tajudeen Kolawole, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsud- deen Hassan Muhammad, Salomon Kabongo, Sa- lomey Osei, et al. 2020. Participatory research for low-resourced machine translation: A case study in african languages. arXiv preprint arxiv:2010.02353.
Barack W. Wanjawa, Lilian D. A. Wanzare, Florence Indede, Owen Mconyango, Lawrence Muchemi, and Edward Ombui. 2023. Kenswquad a question an- swering dataset for swahili low resource language. ACM Trans. Asian Low-Resour, Lang. Inf. Process. Just Accepted.
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman. Semarley Jarrett, Kaushik Ram Sadagopan. Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale. Sergey Edunov. Angela Fan. Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.1192
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre- pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line, Association for Computational Linguistics.

---

Parameters
Value
B
Training and Evaluation Details
backhone
multilingual-bert
B.1 mDPR Reader:
train epochs
25
# warmup steps
500
We train multilliguul DPR reader model using pretrained
# GPUs
# gradient accumulation
-nccssd the model backbone. The model was trained to predict the correct answer span for a question given set of relevant passages. We trained our model using the DPR retriever output"o on the training and de- velopment set of Natural questions and evaluated on the test set of AFRIQA in zero-shot manner. The model was trained on 4 A6000 Nvidia GPUs with batch size of 16 and gradient accumula- tion steps. We used initial learning rate of 5e-5 and 500 warmup steps. The full list of training hyperparameters can found in Table 9.
learning rate
5.0c-05
1.0c-08
batch size
16
weight decay
0.01
max gradient norm
1.0
seed
42
max sequence length
256
Table 9: DPR Reader Training Configurations
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelle Xi- aoguang Li, Qun Liu, Mehdi Rezagholizadch, and Jimmy Lin. 2022. Making miracl: Multilingual in- formation retrieval across continuum of languages.
B.2 AfroXLM-R Reader
To extract answer spans from the gold passages. we train extractive reader models on the training set of Squad 2.0 (Rajpurkar et al., 2016) and fQuad (d' Hoffschmidt et al., 2020) using AfroXLM-R as backbone. We evaluated the models on the test queries and the annotated gold passages. The models were trained for 5 epochs using a fixed learning rate of 3e-5 and batch size 16 on a single A A100 Nvidia GPU.
A Preparing Wikipedia Passages
Wikipedia is a popular choice as a knowledge base for open-retrieval iments, where articles are usually divided into fixed-length passages that are indexed and used for retrieval and reading comprehension, as seen in previous works such as (Karpukhin et al., 2020; Asai et al., 2021). However, Tamber et al. (2023) highlighted that splitting articles into fragmented and disjoint passages can negatively impact down- stream reading comprehension performance. In- stead, they proposed sliding window segmenta- tion approach to create passages from Wikipedia articles. In line with this methodology, we used the same approach to create passages for our cross- lingual question-ann answering experiments.
(QA) exper-
B.3 mT5 Reader
We finetuned multilingual pretrained text-to-text transformer (mT5) (Xue et al., 2020) Squad 2.0 (Rajpurkar al., 2016) dataset to generate answers from the gold passages. We trained the model for 5 epochs with learning rate of 3e-5 and batch size of 32 on a single A100 Nvidia GPU.
C Machine Translation BLEU Scores
To create our passages, we downloaded the Wikipedia dumps dated May 01, 2022, for En- glish Wikipedia and April 20, 2022, for French Wikipedia. We then applied the sliding window approach to generate fixed-length passages of 100 tokens each from these dumps. These passages serve as our knowledge base for retrieval and an- swer span extraction. By adopting the sliding win- dow segmentation approach for creating Wikipedia passages, we aim to improve downstream read- ing comprehension performance. The fixed-length passages enable efficient indexing and retrieval of relevant information for given question while reducing the impact of disjoint and fragmented in- formation that may occur when arbitrarily splitting articles.
Table 10 shows the BLEU of the different translation systems on the test of AFRIQA, eval- uated against the human- suman-translated queries. Google Translate performs the best on the languages it sup- ports while NLLB 1.3B achieves slightly poorer performance with broader language coverage.
D Additional Experiments D.1 Retrieval Top-20/100 Accuracy
We present top-20 retriever accuracy results in Ta- ble 11.
hhttps:://ugginggacc bert bert-bbse-multee b "https:///ithhb http:///booookorseerrc/////

---

Source Target
GMT NELB M2M-100
lang lang
bem
eng fon Fre
24.4
16.6
8.7
hau eng
55.2
44.6
26.3
ibo
elgg
48.3
46.3
34.1
kin
eng
44.9
43.1
eng
54.0
53.2
34.7
twi
eng
33.0
30.1
15.7
free
16.6
yor
eng
32.7
30.6
7ul
eng
50.2
45.4
33.3
45.5
35.1
22.0
Table 10: Translation BLEU Scores: BLEU score of some translation systems on the test set for the answer translation task. Note that Google Translate is not yet available in all languages, due to their very low-resource nature.
This further highlights the downstream effect of translation quality on retriever effectiveness with human translations showing better accuracy than other machine translation systems.
D.2 XOR-Full Results
Table 12 presents the Exact Match Accuracy and BLEU scores of the XOR-Full task. The table con- tains downstream results of different translation- retriever pipelines to extract the answer span and translate it back to the same language as the ques- tion.
E Summary of Language Linguistic Properties
Table 13, we provide a structured breakdown of the typologies, grammatical structures, and phonol- ogy of the 10 languages in AFRIQA.

---

Human Translation
GMT
NLLB
M2M-100
Crossliguul
BM25
mDPR
Hybrid
BM25
mDPR
BM25
mDPR
BM25
mDPR
mDPR
lang
Recall @ 20
bem
64.3
72.6
76.8
60.2
65.3
22.0
fon
71.5
72.2
74.6
49.6
52.3
46.5
46.9
30,3
hau
64.3
73.3
78.0
60.0
70.0
59.3
68.7
43.3
51.7
20.0
igb
75.3
78.7
87.8
72.4
76.0
70.2
73.4
67.2
74.3
34.0
kin
67.4
72.6
80.1
63.1
68.6
62.0
65.7
19.3
swa
54.6
67.6
72.5
52.7
66.9
50.3
64.6
47.0
61.3
26.8
twi
69.0
71.4
78.4
61.0
63.7
55.9
58.6
49.8
53.9
26.3
wol
68.6
73.1
72.2
42.8
43.7
41.0
40.4
18.0
yor
62.7
72.6
77.7
58.4
66.9
58.1
65.7
41.9
41.9
31.3
zul
68.6
76.6
83.7
66.5
71.7
62.2
69.2
53.2
64.9
18.2
Recall 100
bem
76.8
81.9
84.7
70.4
74.2
37.3
fon
79.3
80.1
60.3
59.3
59.6
46.9
hau
77.7
83.3
77.7
79.3
75.0
77.7
58.3
34.3
igb
89.7
84.8
83.9
82.4
83.4
50.1
kin
78.1
81.3
87.0
75.2
78.1
74.1
77.0
30.3
swa
70.9
80.5
82.1
68.1
79.8
68.2
77.2
64.2
76.2
40.1
twi
78.4
82.9
85.7
71.6
83.7
70.0
72.5
61.8
63.1
38.4
wol
82.6
82.6
84.7
56.0
55.1
57.2
53.6
31.1
yor 78.6
83.4
87.1
73.2
79.2
71.1
78.3
59.6
55.4
46.7
zul 86.2
86.2
91.1
83.1
72.0
77.0
80.6
71.1
74.8
28.9
avg 79.5
83.1
86.2
76.4
79.9
70.8
73.6
64.3
38.4
Table Retrieval recall @ 20/100: This table presents the retrieval recall @20/100 results for different translation types on the test set of AFRIQA. This shows the percentage of the top 20/100 retrieved passages that contain the answer. Crosslingual retrieval skips the translation step
Translation
Query
Answer
Retrieval
XOR-Full BLEU
Average
bem
fon
hau
ibo
kin
twi
wol
zul
BLEU
GMT
GMT
BM25
19.4
28.2
21.1
16.0
11.7
13.8
18.1
18.3
GMT
GMT
mDPR
20.1
30.3
23.3
19.9
13.2
18.6
19.6
20.7
NLLB
NLLB
BM25
11.4
1.7
15.9
24.8
16.8
16.9
16.6
2.9
10.9
10.7
12.9
NLLB
NLLB
mDPR
10.9
3.3
17.0
27.2
18.8
18.3
17.5
3.1
15.3
11.4
14.3
XOR- Full EM
EM
GMT
GMT
BM25
16.3
21.0
12.3
10.9
4.0
8.0
12.0
12.1
GMT
GMT
mDPR
15.7
22.7
15.0
14.6
4.9
12.7
14.2
14.2
NLLB
NLLB
BM25
6.7
0.5
11.7
15.4
7.8
10.0
10.6
2.4
5.1
4.3
7.5
NLLB
NLLB
mDPR
5.4
0.2
10.7
17.6
8.6
15.3
10.8
2.4
7.2
4.9
8.3
Table 12: XOR-Full results

---

Table 13: Language Linguistic Features: This table provides a breakdown of the typologies, grammatical structures, and phonology of the 10 languages in AFRIQA
W When: "yaushe", Why: "dan molskan me.
W When: daben. Why: aden. Which: derhen Where: rhonfa, How:
What: "yini" Who: "uhani" When: "nimi" Why: "Kungani" Which: "yiliphi",
What: kedu/gini Who: onye/keda onye
When: kodu mghe, Why: gini mere/gini
What: "nii" Who: "nani" When: "lini"
When: "lisa", Why. "mulandunshin Which: "clisa", Where: "kwi/kwiisa"
Which: "wanne", Where: "insl ina"
When: "ryari", Which: "ikihe/hwahh"
What:" "Et"""EE"" Who: "Me" When: "Hweréno", Why: "Aniwd" Which: "de "de te " Where: "Fite"
W When: "iga nighu", Why: "kilode" Which: "owo", Where: "Nibo"
Which: kedu nke, Where: ebee
Why: "kwanini", Which- "upi". Where: "upi", How: "vipi"
What ian Who: kan, When: kañ
Where: "Kuphi" How: "kanjani"
What: "cinshi" Who: "naani"
What: "iki" Who: "nde/inde"
What: "rdern" Who: "hwun",
What: "melya", Who: "wa"
Where: "hehe", How: "gute"
What: "Kimi", Who: "Tani"
How: kedu ka or kedu ctu
Why: tax. Which: bun.
Where: fan. How: naka
WH-questions
No suffix. Count is often specified
Dependent word: plurality. "yi" or "uy" attached before or after the word
Suffix with vowel deletion. E.g.: "hula" (cap). "huluma" (caps) "mace" (girl), "mataye" (girls)
Indicated by changes to the prefix according to noun class
Indicated by replacing the first two letters a root word with "mm" or "nn"..
Count is specified with word
Changes to morphemes
Indicated by morpemes "aba", "izzi" "imi """"
Affix to the steam of the word depending on noun class
New word added: "16"
after the word
Plurality
Indicutive form. Words used indicate negation: ba/ha a" (mst) and "banda" (except)
Keyword "ul" is added at the end of the verb g nekk -; nekkul
"n" added the word
Afix to head word: "ta", "shi", and "kaana"
Changes morphemes in word
New word added: "8"
Keywords such ko. máa, nile
Typically indicated by the prefix "nga-
Suffix "ghi"
Negation
To indicate present tense, keyword "n". Past tense is indicated with "ti" with or without a time period
Indicative form Words used indicate tenses: post: "tsohon" (was) present: "yanzu" (is)
Present: affix after subject comcord (c.g. "ya" or "sa") Past: suffix (e.g. "c" or "ile")
Present: "ni" (is), Past: "alikuwa"" (was/former)
Future: "atakuwa" (will be)
Dependent word: past tense, "oon" attached to the end of the verb
Afiix to head word present "ali", post "nali"
Changes morphemes in a word
New word added: past "x6x6"
Tenses
Cgger-Congo None
twi Niger-Congo None
bem Niger- Congo
Niger- Congo
Afro- Asiatic
Niger- Congo
Niger- Congo
wol Niger- Niger-Congo Congo
Niger- Congo
Niger-Cong Congo
lang Family
fon
hau
igb
kin
swa
yor
zul


Reviews:
[{"review_id": "kqyuM8oNF0", "reviewer": "Reviewer_u1CQ", "summary": "", "questions": "1 Can you better clarify how is quality of the translations monitored and evaluated?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback! We are encouraged that you find our work beneficial to the study of low-resource languages. \nWe have addressed the different points raised below:\n\n- Better Paper Structure:  Thank you for pointing out the issue with the paper structure. As suggested we will move the related work section before the experiments when we prepare the final version of the paper.\n\n- Monitoring and evaluating the quality of translations: In Section 2.3, \"Quality Control\", we detailed the multi-step process employed to ensure the quality of translations. Initially, annotators identified potential errors in both the local and pivot languages. These flagged translations were then reviewed by our language coordinators for accuracy in both the questions and answers. Finally, a central project manager further validated these translations for consistency across the dataset. It is worth noting that we did not aim for perfect translation but for naturalness to reflect how any speaker would pose a question in the language, hence why professional translators were not engaged in the translation phases.  We will make sure to improve the clarity of this section in the final submissions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ new dataset for xor-qa and low resources languages (10 africans languages)\n+ clear experiments and clear results\n+ comparison of human translations and machine translations\n+ large set of proposed experiments\n+ dataset released with cc license", "reasons_to_reject": "+ better paper structure (e.g: related work section need to be moved before experiments for a better clarification )\n+ quality control section is not completely clear\n", "questions_to_authors": "1 Can you better clarify how is quality of the translations monitored and evaluated?", "paper_topic_and_main_contributions": "The paper is about a new dataset focused on low resource languages for cross lingual retrieval and question answering. In this case it is focused on 10 African languages.\nThe authors uses and evaluated manual translations and automatic translations to generate the dataset.\nExperiments were conducted using sota retrieval approaches as BM25, DPR and hybrid systems.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2YyTwScWgr", "reviewer": "Reviewer_Fe2K", "summary": "", "questions": "- How would one expand this dataset for new languages?\n- Have you done any quality checks to mitigate possible toxic cases?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback! \nWe have addressed the different points raised below:\n\n- How would one expand this dataset for new languages: In the final version of our paper, we'll include a link to a repository outlining our methodology which can be extended to create the dataset for new languages. This repository contains reproducible code for running baseline retrieval and reading comprehension experiments. We've also made our annotation tools available as open source, complete with comprehensive documentation. This will facilitate anyone interested in extending the dataset using the tools we developed. We removed the link in the current paper version due to anonymity.\n\n- Have you done any quality checks to mitigate possible toxic cases: Yes, In Section 2.3 Quality Control from line 317 we try to explain the quality control process. Annotators flagged inappropriate questions when translating or during answer annotation. Additionally, language coordinators further checked the question and answer pairs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Valuable resource creation\n- Good benchmarking\n- Well-presented.", "reasons_to_reject": "None", "questions_to_authors": "- How would one expand this dataset for new languages?\n- Have you done any quality checks to mitigate possible toxic cases?", "paper_topic_and_main_contributions": "The main contribution of this work is the creation of the first open-retrieval cross-lingual question answering dataset focusing on African languages. The provide collection procedure for this dataset, analysis on the features, as well as benchmarking on cross-lingual retrieval task. The paper is well-written and the dataset creation is well-motivated and supported.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qmEhWASrCA", "reviewer": "Reviewer_aGgm", "summary": "", "questions": "Question A: It is unclear why the linguistic properties of each of the languages are crucial when formulating the questions. \n\nQuestion B: I somehow miss an explanation on why precisely these 10 languages are included and not others.\n\nQuestion C: Section 2.2. What is the exact financial compensation provided to each contributor?\n\nQuestion D: Up to 40% of the questions in certain languages lack any question (at least 12%). Is this imbalance reasonable/representative? Could the authors justify why this large amount of unanswerable questions make the dataset realistic?\n\nQuestion E: That google does not cover a certain language could simply mean that the company is not necessarily interested in it commercially. Therefore, I suggest to dim the claim in footnote 6.\n\nQuestion F: The authors should make it clear what exactly the models are. Is BM25 the only monolingual one? Why not DPR beside mDPR? Thsi is confusing in Section 4.2 and in 5.1 as well.\n\nQuestion G: Line 463. Could the authors give some context? What is the situation with other languages? Is it similar?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback! We have addressed the different points raised below:\n\n-  Linguistic Properties: We talk about the linguistic features of these languages to highlight their diversity. This is important because our work focuses on translating questions from one language to another and understanding these unique features helps in the translation process. That said, when it came to actually creating the questions in our dataset, we gave our annotators full freedom. We didn't enforce any specific linguistic guidelines. So, the linguistic properties are useful for the bigger picture but were not directly used in making the questions. I hope this clarifies our approach.\n\n- Selection of 10 languages: Thank you for pointing out the need for clarity on our language selection. The primary reason we focused on these 10 African languages is because we had access to native speakers who could effectively handle the tasks of creating, translating, and annotating questions in these languages. Finding annotators who are proficient speakers of many African languages can be a challenge, and it was pivotal for our research to have individuals with a good grasp of their respective languages. We will ensure that this is explicitly stated in the updated version of our manuscript.\n\n- Financial Compesation: The total budget for each team is $3600 including the team coordination stipend. However, as mentioned in Section 2.2 Data Collection Procedure, team size ranges from 2-6 native speakers.\n\n- Unanswerable  Questions:     To address this, it's crucial to first refer to our methodology for question generation: We asked our annotators to ask questions based on their genuine curiosity, rather than being restricted to specific passages. Afterward, we tried to find answers to these questions from Wikipedia, a common source used in well-known QA datasets like Natural Questions and TyDi QA, in English or French depending on the African language. The high rate of unanswerable questions, particularly for the 10 African languages in our collection, comes from Wikipedia's limited coverage of topics directly related to Africa. In the paper, we discuss this point specifically in lines 337-339. \n\n- Google Translate coverage:      We appreciate the reviewer's insight. Indeed, Google's decision not to cover certain languages might be tied to commercial interests, not just the low-resource status of these languages. We'll adjust the footnote accordingly.\n\n- Retrieval Models: Thank you for bringing this to our attention. In our study, we used three main retrieval methods:\n    - BM25: This is a keyword-based search method. We had separate indexes for both French and English texts making it a monolingual search.\n    - mDPR (multilingual Dense Retriever): Unlike the typical DPR that relies on an English-only BERT model, mDPR uses a multilingual BERT model. This was crucial for our experiments, which looked at both English and French texts and accepted questions in 10 different languages.\n    - Hybrid Retriever: This method combines the results from BM25 and mDPR to improve performance.\n  We opted for mDPR over the standard DPR specifically because mDPR can work well with multiple languages, aligning more closely with the goals of our research. In the revised version of our paper, we'll be sure to spell out these details more clearly to avoid any confusion.\n\n- Could the authors give some context? What is the situation with other languages? Is it similar?: Line 463 states that “cross-lingual representations learned by current retrieval methods are not yet of sufficient quality to enable accurate retrieval across different languages.” This refers to using a single multilingual model to perform retrieval across multiple languages. This is still an active area of research and even for high-resource languages, translation-based approaches (query/document translation) still yield better results compared to cross-lingual dense retrieval methods. I hope that gives a clearer picture of our point. Thanks for allowing us to explain further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors have developed a good new resource and include reasonable experimentation. ", "reasons_to_reject": "Some of the details of the corpus construction, as well as of the experiments, are missing. Some decisions lack support.", "questions_to_authors": "Question A: It is unclear why the linguistic properties of each of the languages are crucial when formulating the questions. \n\nQuestion B: I somehow miss an explanation on why precisely these 10 languages are included and not others.\n\nQuestion C: Section 2.2. What is the exact financial compensation provided to each contributor?\n\nQuestion D: Up to 40% of the questions in certain languages lack any question (at least 12%). Is this imbalance reasonable/representative? Could the authors justify why this large amount of unanswerable questions make the dataset realistic?\n\nQuestion E: That google does not cover a certain language could simply mean that the company is not necessarily interested in it commercially. Therefore, I suggest to dim the claim in footnote 6.\n\nQuestion F: The authors should make it clear what exactly the models are. Is BM25 the only monolingual one? Why not DPR beside mDPR? Thsi is confusing in Section 4.2 and in 5.1 as well.\n\nQuestion G: Line 463. Could the authors give some context? What is the situation with other languages? Is it similar?", "paper_topic_and_main_contributions": "The manuscript introduces a new dataset for cross-language question answering focusing on African languages. The main contribution is the dataset, which includes 10 African languages plus two supporting languages, English and French. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The authors say \"we demonstrate\" (l. 88). They do not demonstrate. They show\n\nFigure 1 should be larger. The font size should be at least that of the footnotes.\n\nevaluate performance -> the performance\n"}}]

Author rebuttal:

Meta-review: The main contribution of this paper is a cross-lingual question answering/retrieval dataset for 10 African languages (+ English and French), generated through automatic and human translation. The dataset is claimed to be the first of its kind for these languages.

The reviewers agree that the corpus building corpus and the benchmarking and analyses are generally clearly presented. The resulting resource is expected to be very valuable. On the other hand, the reviewers found the structure of the paper a bit unintuitive and were missing some details. Also, some decisions should be better justified. However, these issues mostly concern the presentation of the paper and can be addressed in the camera-ready version.
Decision: Accept-Findings