Question-Answering in a Low-resourced Language: Benchmark Dataset and Models for Tigrinya
Year: 2023

Abstract:
Question-Answering (QA) has seen significant advances recently, achieving near human-level performance over some benchmarks. However, these advances focus on high-resourced languages such as English, while the task remains unexplored for most other languages, mainly due to the lack of annotated datasets. This work presents a native QA dataset for an East African language, Tigrinya. The dataset contains 10.6K question-answer pairs spanning 572 paragraphs extracted from 290 news articles on various topics. The dataset construction method is discussed, which is applicable to constructing similar resources for related languages. We present comprehensive experiments and analyses of several resource-efficient approaches to QA, including monolingual, cross-lingual, and multilingual setups, along with comparisons against machine-translated silver data. Our strong baseline models reach 76% in the F1 score, while the estimated human performance is 92%, indicating that the benchmark presents a good challenge for future work. We make the dataset, models, and leaderboard publicly available.1

heading: 1 Introduction
text: Question Answering (QA) and Machine Reading Comprehension (MRC) have seen significant advances in recent years, achieving human-level performance on large-scale benchmarks (Rajpurkar et al., 2016, 2018). The main factors driving the progress are the adaption of large pre-trained large language models and the proliferation of QA datasets (Rogers et al., 2022). However, most studies focus on high-resourced languages, while the task remains unexplored for most of the World’s diverse languages. The primary challenge for nonEnglish QA is the lack of native annotated datasets. In particular, there is little to no study done on scarcely resourced languages such as Tigrinya that
∗ Coressponding author 1 TiQuAD: https://github.com/fgaim/tiquad
are markedly different from English in terms of linguistic properties including syntax, morphology, and typography.
This work presents TiQuAD, the first publicly available Question-Answering Dataset for Tigrinya; see Figure 1 for an example entry. We collaborate with native Tigrinya speakers to collect documents and annotate the dataset, yielding a total of 10.6K question-answer pairs with 6.5K unique
11857
questions over 572 paragraphs gathered from 290 news articles.
We assess the quality of annotations and explore strong baselines by fine-tuning TiRoBERTa and TiELECTRA (Gaim et al., 2021) as monolingual models of Tigrinya and XLM-R (Conneau et al., 2020) and AfriBERTa (Ogueji et al., 2021) as representative multilingual models. In addition to the monolingual QA setup, we perform three scenarios of cross-lingual and multilingual experiments. First, we translate SQuAD1.1 to Tigrinya and evaluate the performance in conjunction with the native TiQuAD. Second, we assess a zero-shot crosslingual transfer learning approach (Artetxe et al., 2020; Lewis et al., 2020) by evaluating on the new dataset. Third, we explore the performance of a multilingual setup by jointly fine-tuning the models on English and Tigrinya datasets. The experimental settings are depicted in Figure 2. The bestperforming baseline model achieves up to 76% in F1 score in the multilingual setup, while the estimated human performance is 92%. Considering the challenges of constructing annotated datasets for under-represented languages, we believe this work could serve as a reference case for similar languages. In particular, the TiQuAD benchmark is an important milestone in the advancement of question-answering for the Tigrinya language.
The contributions of this work are summarized as follows: (1) We build the first questionanswering dataset for Tigrinya and make it publicly available. (2) We present an in-depth analysis of the challenges of question answering in Tigrinya based on the dataset. (3) We apply transformer-based language models to the question-answering task in Tigrinya and compare it with datasets of other languages. (4) We investigate various resourceefficient cross-lingual and multilingual approaches to QA and assess the utility of the native dataset.

heading: 2 Related Work

heading: 2.1 Tigrinya Language
text: Tigrinya (ISOv3: tir) is a Semitic language, part of the Afro-Asiatic family with over 10 million native speakers in the East African regions of Eritrea and Northern Ethiopia. Tigrinya is closely related to Amharic and Tigre languages that are also spoken in similar regions and share the same ancestor, the now extinct Ge’ez language. In recent years, there is a growing research body and interest in Tigrinya. Gasser (2011) developed HornMorph,
a morphological analysis and generation framework for Tigrinya, Amharic, and Oromo by employing Finite State Transducers (FSTs). Later, Tedla and Yamamoto (2018) employed a manually constructed dataset to train a Long ShortTerm Memory (LSTM) model for morphological segmentation in Tigrinya. Osman and Mikami (2012) proposed a rule-based stemmer for a Lucene based Tigrinya information retrieval. Tedla et al. (2016) presented a part-of-speech (POS) corpus for Tigrinya with over 72K annotated tokens across 4.6K sentences. A few studies explored statistical and neural machine translation, between English and Tigrinya, by exploiting morphological segmentation (Tedla and Yamamoto, 2016; Gaim, 2017; Tedla and Yamamoto, 2018) and data augmentation via back-translation (Öktem et al., 2020; Kidane et al., 2021). More recent studies applied pretrained language models to various downstream tasks such as part-of-speech tagging, sentiment analysis, and named entity recognition (Tela et al., 2020; Gaim et al., 2021; Yohannes and Amagasa, 2022). Moreover, Gaim et al. (2022) presented a dataset and method for the automatic identification of five typologically related East African languages that include Tigrinya. However, despite the recent progress, Tigrinya still lacks basic computational resources for most downstream tasks with very limited availability of annotated datasets.

heading: 2.2 Question-Answering beyond English
text: Native reading comprehension datasets beyond the English language are relatively rare. Efforts have been made to build MRC datasets in Chinese, French, German, and Korean, among others, all of which are designed following the formulation of SQuAD. The SberQuAD dataset (Efimov et al., 2020) is a Russian native reading comprehension dataset made up of 50K samples. The CMRC 2018 (Cui et al., 2019) dataset is a Chinese reading comprehension dataset that gathers 20K question and answer pairs. The KorQuAD dataset (Lim et al., 2019) is a Korean native reading comprehension dataset containing 70K samples. On the end of low-resourced languages, Mozannar et al. (2019) developed ARCD for Arabic with 1.3K samples. Keren and Levy (2021) presented ParaShoot, a reading comprehension dataset for Hebrew with a size of 3.8K question-answer pairs. More recently, Kazemi et al. (2022) built PersianQuAD, a native MRC dataset for Persian with over 20K samples.
Cross-lingual Question Answering Languagespecific datasets are costly and challenging to build, and one alternative is to develop cross-lingual models that can transfer to a target without requiring training data in that language (Lewis et al., 2020). It has been shown that unsupervised multilingual models generalize well in a zero-shot cross-lingual setting (Artetxe et al., 2020). For this reason, crosslingual question answering has recently gained traction with the availability of a few benchmarks. Artetxe et al. (2020) built XQuAD by translating 1190 question-answer pairs from the SQuAD1.1 development set by professional translators into ten other languages.
Multilingual Question Answering The MLQA dataset (Lewis et al., 2020) consists of over 12K question and answer samples in English and 5000 samples in six other languages such as Arabic, German and Spanish. More recently, Clark et al. (2020) presented TyDiQA, a dataset particularly designed to address information-seeking and natural questions covering 11 typologically diverse languages with a total of 204K samples. Longpre et al. (2021) presented an open domain dataset comprising 10K question-answer pairs aligned across 26 typologically diverse languages, yielding a total of 260K samples. Hu et al. (2020) presented XTREME, a multi-task benchmark for nine prominent NLP tasks including question-answering across 40 languages. Ruder et al. (2021) further extended the benchmark to XTREME-R, covering ten tasks across 50 typologically diverse languages. Xue et al. (2021) proposed a large multilingual pretrained model that handles 101 languages. Note that none of the aforementioned datasets and models include the Tigrinya language.
Translated QA datasets Another relatively inexpensive alternative to building a native annotated QA dataset is translating an existing English dataset to the target language. Carrino et al. (2020) explored this by proposing the Translate-AlignRetrieve (TAR) method to translate the English SQuAD1.1 dataset to Spanish. Then the resulting dataset was used to fine-tune a multilingual model achieving a performance of 68.1/48.3% F1/EM on MLQA (Lewis et al., 2020) and 77.6/61.8% F1/EM on XQuAD (Artetxe et al., 2020). Similar approaches were also adapted for the Japanese and French languages (Asai et al., 2018; Siblini et al., 2019), where a multilingual version of BERT
(Devlin et al., 2019) is trained on the English SQuAD1.1 and evaluated on the small translated corpus, reaching promising scores of 76.7% in F1 and 61.8% in EM.

heading: 3 Dataset Annotation
text: TiQuAD is designed following the task formulation of SQuAD (Rajpurkar et al., 2016), where each entry in the dataset is a triple consisting of a paragraph, a question, and the corresponding answer. The answer is a contiguous span of text in the paragraph, a typical setup of extractive questionanswering.
The dataset was constructed in four stages: First, a diverse set of articles are collected from which we extract paragraphs that will serve as contexts. Second, the initial question and answer pairs are annotated for all the extracted paragraphs. Third, additional answers are annotated for all the questions in the development and test sets. Fourth, we post-process the annotations for quality control and remove noisy examples. The final dataset contains over 10.6K question-answer pairs across 572 paragraphs. While the size is on the smaller end compared to the English datasets, it reflects a realistic amount of data that researchers of low-resourced languages can acquire with a limited annotation budget.2 The dataset characteristics are presented in Table 1. In the following sections, we present the data collection and annotation processes.

heading: 3.1 Collecting Articles
text: In the absence of sufficient Tigrinya content on Wikipedia3, the Haddas Ertra4 newspaper provides a large body of professionally edited Tigrinya text, covering diverse domains and has been used as the main source in previous research (Tedla et al., 2016;
2 We paid an hourly salary of 150 ERN (10.00 USD, at the time of writing) to the annotators.
3 Tigrinya Wikipedia https://ti.wikipedia.org 4 The newspaper is made available at www.shabait.com,
the official website of the Eritrean Ministry of Information.
Gaim et al., 2021). We collected 550 Tigrinya articles from Haddas Ertra covering a wide range of topics, including science, health, business, history, culture, and sports, published in a period of seven years, 2015-2021. The articles that contain at least 500 characters of plain text are kept after filtering out images and tabular content. We split the dataset randomly into training, development, and test sets of 205, 43, and 42 articles, respectively.

heading: 3.2 Annotating Question-Answer Pairs
text: In the first round of annotation, we recruited eight native speakers of Tigrinya [4 female, 4 male] with ages ranging from 20 to 46. Each annotator is presented with a random paragraph from the collection and tasked to write questions that can be explicitly answered by a contiguous segment of text in the provided context. The annotators were encouraged to phrase questions in their own words instead of copying words from the context and to highlight the minimal span of characters that answer the respective question. The annotators were asked to spend on average one minute for each question and answer pair. The end result of this stage is the set of 6,674 unique questions across all documents.

heading: 3.3 Collecting Additional Answers
text: In the second round of annotation, we asked four of the original annotators to provide a second answer to questions in the development and test parts of the dataset. Our annotation tool ensures that annotators cannot give a second answer to the questions they contributed already in the second stage. Finally, we recruited two new annotators to provide a third reference answer to all the questions in the evaluation
sets. These annotators were not involved in the first round of the annotation; with no prior exposure to the task, they are expected to show less bias towards the question formulation. We ensure that all entries in the test and development sets have at least three answers from different annotators, resulting in 6,205 answers for 2,056 questions.

heading: 3.4 Post-processing Annotations
text: Throughout the annotation campaign, we collected over 6,674 unique questions and 10,600 answers, i.e., 2,056 of the questions had at least three groundtruth answers by different annotators. From these annotations, we discarded 166 entries (2.5%) that either contained apparent errors, were incomplete, unanswerable by the context, or had a wrong question formulation such as verification (yes/no) and cloze type. For instance, the question “]≈nÛs ³M³y ¿b ƒsŒ• ƒs³t 28 ¾‹ °r nX¶Ω ___ rJc tr¼b ~ [Lower Shmangus is located about 28 km from Asmara on the ___ side.]" is in cloze format, hence deleted. We also removed outlier entries that had answers with more than 250 characters.

heading: 4 Dataset Analysis
text: To assess the quality and diversity of the dataset, we perform various analyses of the annotations.

heading: 4.1 Question-type Analysis
text: We clustered all questions in the development set into nine types using a manually curated list of question words. As presented in Table 2, the top three types are which [ƒÎ¶y²/ƒÎ¹y], how many/much [knÔy], and what […n³y], accounting
for≈72% of all the questions. These types of questions also make up the largest proportions in other datasets (Keren and Levy, 2021; d’Hoffschmidt et al., 2020). Question types that lead to named entity answers such as who [Œn], when [ŒAs], and where [ƒ¤y] comprise 24%. While there are only 3.2% of the why [s†mn³y], how [bˆŒy], and Other types, they generally necessitate more complex reasoning and are challenging to create during annotation.

heading: 4.2 Question-Context Lexical Overlap
text: The degree of lexical overlap between questions and paragraphs might affect the difficulty of a dataset. To assess this behavior in TiQuAD, we analyzed 100 random samples from the development set and assigned them to four categories of question-context-answer linguistic relationships proposed by Rajpurkar et al. (2016): (1) Synonymy implies that key terms in the question are synonyms of words in the context; (2) World knowledge implies that the question requires world knowledge to find the corresponding answer in the context; (3) Syntactic/Morphological variation implies a difference in the structure between the question and the answer in the context; (4) Multi-sentence reasoning implies that answering a question requires combining knowledge from multiple sentences in the context. We observe that syntax and morphology variations are the most common type in TiQuAD.
The results of our findings are presented in Table 3.

heading: 4.3 Answer correctness and length
text: We randomly selected 100 question-answer pairs from the validation set to assess the accuracy and length of the answers manually. We specifically check whether each annotated answer is correct and has a minimal length in answering the corresponding question. We observe that 74% of the answers are accurate and with a minimum span length, while a significant minority, 23%, contain extra information and are longer by a factor of 1.5 on average than the desired span. Only 3% were shorter than the optimal span length, such as partial annotation of the answer.

heading: 4.4 Sequence Lengths
text: The lengths of paragraphs in TiQuAD range between 39-278 words or 198-1264 characters. Around 60% of the questions have 5-10 words, but we observe some verbose examples such as ƒb gµ≈t ff‘e∆ …g” Ô  °§‘etÓ, ƒb² ƒb Œnß Á†¥ KlK†n Á†¥ µ¤§tn ≈M¤•Ä s¶fl°tn ‘AÖ‘yMn z¶¤ gµm Á†¥ µ¤§tn ≈M¤•Ä s¶fl°tn ‘AÖ‘yM knÔy Xµ³t ƒŒzÜ§? [In the men’s soccer match between Halhale College and Adikeih College of Arts and Social Sciences, how many goals did Adikeih College of Arts and Social Sciences score in the match?]. The shortest questions have three words,
for example, å— ƒ¤y tr¼b? [Where is Foro located?]. Over 57% of the answers have three or fewer words, but there are cases with up to 32 words that typically constitute a list of items.

heading: 4.5 Estimating Human Performance
text: We assess the human performance on TiQuAD’s development and test sets, where each question has at least three answers. In SQuAD, Rajpurkar et al. (2016) use the second answer as the prediction and the rest as ground truths; while in FQuAD, d’Hoffschmidt et al. (2020) compute the average by successively taking each of the three answers as the prediction. For TiQuAD, the third answer is regarded as a prediction, and it is annotated by a control group who had no prior exposure to the task, as elaborated in Section 3.3. We obtain scores of 84.80% EM and 92.80% F1 in the development set, and 82.80% EM and 92.24% F1 in the test set, which are comparable to those of the SQuAD and FQuAD benchmarks.
We analyzed the cases where the human annotators failed to agree and observed that they are mainly due to extra tokens in the answer spans rather than fundamental differences. For instance, the question ¡®n±s n˜” ƒ bknÔy °ŒrN ƒ‰? [With how many [points] is Juventus leading the Serie A?] has three different annotations: (1) 10 ¶µ¦ [10 points]; (2) 10 ¶µ¦ flly [10 points difference]; (3) b¹y 10 ¶µ¦ flly [with a 10 points difference], resulting in zero EM agreement.

heading: 5 Experiments

heading: 5.1 Model Training
text: Given a question Q and a context paragraph P from an entry in a QA dataset, the training objective is to predict the start and end positions of the answer span within the paragraph. Following Devlin et al. (2019), we set the input to the transformer model as a concatenation of Q and P , separated by a special delimiter token, SEP. Two linear layers, S and E, are introduced to learn the starting and ending positions of answer spans, respectively. Then the probability distributions of token i being the start or the end of an answer span with respect to all tokens in the context can be computed as follows:
P start(i) = exp (S · T i)∑ j=1 exp (S · T j) , (1)
P end(i) = exp (E · T i)∑ j=1 exp (E · T j) , (2)
where T is the model’s output of the context sequence, and T i is the hidden state of the i-th token.
The score for a candidate span (i, j) is defined as the product of the start and end position probabilities, and then the highest-scoring span where j ≥ i is used as the final prediction.
Score(i, j) = P start(i) · P end(j). (3)
The loss function L is the sum of the negative log-likelihoods of the ground truth start and end positions, denoted as i∗ and j∗, respectively.
L = − logP start(i∗)− logP end(j∗) (4)
During training, a gradient-based optimizer minimizes the loss and gradually enables the model to accurately predict the answer spans in the context.

heading: 5.2 Evaluation Metrics
text: We use the standard Exact Match (EM) and F1 metrics for evaluation. EM is the percentage of predictions that exactly match the ground truth. F1 score is the average overlap between the predicted tokens and the ground truth, hence rewards partial matches. For both metrics, when there are multiple ground truth answers for a given question in the test set, the final score represents the highest overlap between the prediction and all the reference answers. To improve the robustness of the evaluation, SQuAD (Rajpurkar et al., 2016) removes the English punctuation and articles before computing the scores. Other non-English datasets have also adapted the metrics (d’Hoffschmidt et al., 2020; Möller et al., 2021). In the case of TiQuAD, we remove Tigrinya’s articles, common functional tokens, and the punctuation set of its writing system, the Ge’ez Script (Gaim et al., 2022).

heading: 5.3 Experimental setup
text: We designed six experimental configurations and evaluated each on four models of varying sizes, ranging from 14 to 278 million parameters. Details of the models are presented in Table 4. The experiments can be grouped into three setups, based on the language of the training data: (1) Monolingual setting: We train and evaluate models using the native and machine translated datasets, separately and in combination; (2) Zero-shot cross-lingual setting: We investigate transfer learning by training models on an English dataset and evaluating them on Tigrinya – treating QA as a language-independent task; and (3) Multilingual setting: We investigate models trained on combined Tigrinya and English QA datasets and evaluated in a native setup. Figure 2 illustrates the experimental settings.
In all experiments, we use AdamW (Loshchilov and Hutter, 2019) as the optimizer with the weight decay parameter set to 0.01 and a learning rate of 3e−5. We set the mini-batch size to 16 and finetune for 3 epochs, except in the Native settings, where only the small native dataset is used, the batch size and number of epochs are set to 8 and 5, respectively. In the settings where only the small native dataset is used for training, we use a minibatch size of 8 and fine-tune for 5 epochs; in all other settings, the batch size and the number of epochs are set to 16 and 3, respectively. The experiments were implemented using the HuggingFace Transformers library (Wolf et al., 2020) and ran on a single NVIDIA V100 GPU.
Translation of English dataset For the experiments, we machine translated the training part of SQuAD v1.1 to Tigrinya. The positional information of the answer spans needs to be computed as it is generally lost during translation, making it difficult to retain the original data size. As a remedy,
we applied two machine translation services5 and aggregated the aligned entries, while discarding the spurious ones. This resulted in 46.7K questionanswer pairs that we use for model training in our experiments.

heading: 6 Results and Discussions
text: In this section, we present and discuss the results of the proposed experimental setups.

heading: 6.1 End-to-end Tigrinya QA
text: In this setup, we train all models on the native and translated Tigrinya datasets then evaluate on the TiQuAD development and test sets. The experimental results are presented in Table 5.
5 Bing Translator API; Google Translate API
Native vs. Translated QA Datasets For models TiRoBERTaBASE and XLM-RBASE, we observe significant gains when training on the native dataset over the translated one, despite the latter being 10 times larger. The performance of TiRoBERTaBASE increases by 5 and 3 points in EM and F1 scores on the test set, respectively. However, we observe that the smaller models TiELECTRASMALL and AfriBERTaBASE perform better when trained on the translated data than on the native one. More consistent performance benefits are observed in all models when the two datasets are combined. For instance, TiRoBERTaBASE gains up to 10 points in EM and F1 than when it is trained on the datasets separately. Overall, our experiments show: (1) A small native dataset can make a positive impact when augmented with larger low-quality data; (2) Machine translated datasets are useful augmentation but can be suboptimal when used alone depending on the quality; and (3) A native dataset could be a vital resource in the evaluation process.
Monolingual vs. Multilingual QA Models When comparing models of comparable sizes, we observe that the monolingual models achieve better performance than their multilingual counterparts. As shown in Table 5, TiRoBERTaBASE is consistently better than AfriRoBERTaBASE, with gains of 6-15 points in F1 score. Conversely, the larger multilingual model, XLM-RBASE, outperformed all models despite not being exposed to Tigrinya during its pre-training. While TiELECTRASMALL trailed in performance in all settings, confirming the impact of model size on the QA task.
Training Sample Efficiency To assess the impact of data size, we fine-tuned the TiRoBERTaBASE and XLM-RBASE models on subsets of the TiQuAD train set gradually increased by 10% of randomly selected samples and evaluated every step on the test set. We observe a promising trajectory where the models do not show signs of saturation and can potentially benefit from a larger dataset. The progress in F1 score performance is depicted in Figure 3, and a similar trend was observed for the EM score.

heading: 6.2 Zero-shot Cross-lingual QA
text: We investigate the transferability of QA models in a zero-shot setting by training on the high-resource language English and evaluate them on Tigrinya. The multilingual models, AfriBERTaBASE and XLM-RBASE, trained on the English SQuAD1.1 achieve 32-34% in F1 score on the TiQuAD test set and outperform their monolingual counterparts. While the models show promising results in transferring the task between two linguistically distant languages, those trained on the small native dataset remain vastly superior. Table 6 presents the results of the cross-lingual and multilingual experiments.

heading: 6.3 Multilingual QA
text: In this setup, we train the models on combined English and Tigrinya training datasets, exposing the models to both languages, then evaluate on the native TiQuAD. We observe a consistent improvement in performance across all models in contrast to the previous setups. For instance, XLM-RBASE in the multilingual setup obtains an increase of over three points in F1 score, setting the state-of-the-art on the TiQuAD test set at 68.06% EM and 76.58% F1 score. Our experiments show that the transfer of models from high to low resourced languages is a viable approach to mitigate the scarcity of annotated datasets. In our case, the benefit emerges when the native and translated Tigrinya datasets are combined with their English counterpart.

heading: 7 Conclusion
text: In this work, we presented the Tigrinya Question Answering Dataset (TiQuAD). The context paragraphs were collected from high-quality News articles of diverse genres, and we collaborated with native speakers to annotate over 6.5K unique questions and 10.6K answers. The development and test sets were further enriched with additional answers to enable a robust evaluation. We conducted comprehensive experiments in monolingual, crosslingual, and multilingual settings. The estimated human performance on the test set is 81.3% EM and 92.1% F1 score, while the top performing model achieves 68.06% EM and 76.58% F1, leaving a room for future improvements.
Limitations
There are two known limitations of the SQuADlike annotation approach we used in this work: (1) It can result in higher lexical-overlap between the context and question pairs. (2) It leads to proportionally fewer truly information-seeking questions (Gururangan et al., 2018; Kaushik and Lipton, 2018). The main reason is that the annotators create questions after reading a paragraph, which can induce bias towards recycling words and phrases observed in the context. Our annotation guidelines advise against this, but it is difficult to avoid entirely. Several approaches have been proposed to mitigate this issue, such as Natural Questions (Kwiatkowski et al., 2019) and TyDiQA (Clark et al., 2020). However, they tend to be expensive, and comparatively, the SQuAD-like method is resource efficient and a more suitable starting point for low-resourced
languages such as Tigrinya. Finally, the current dataset does not include adversarial examples to measure the capability of models to abstain from providing an answer when it does not exist in the context; this extension is left for future work.
Ethics Statement
This research adheres to the academic and professional ethics guidelines of our university. Our annotation task was approved by the Institutional Review Board (IRB)6. All the data collection and annotation procedures were conducted with respect and the informed consent of the participants, and best effort was made to ensure their privacy and autonomy. All participants of annotation tasks indicated their understanding of the procedure for the annotation and acknowledged their agreement to participate. The data sources are published News articles, and for our dataset, we have made an effort to ensure that (1) no personally identifying sensitive information is included, and (2) there exists a fair representation of various genres of news. Furthermore, we ensure that the dataset is available for public use. There may exist inaccuracies or inconsistencies in the questions or answers that could be misleading or ambiguous, potentially due to mistakes and subjective decisions made by the annotators. Furthermore, a bias in the dataset could lead to wrong answers or answers that are only applicable to specific groups of people. We have made the best effort to avoid such issues, but these types of limitations are difficult to detect and remove entirely and potentially present in all similar datasets. The dataset and models released in this work are for research purposes only and may not be suitable for production services without further scrutiny.

heading: Acknowledgements
text: We would like to thank all the annotators who participated in this work and the anonymous reviewers for their time and constructive feedback. This work was supported by Institute for Information and communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No. 2018-0-00582, Prediction and augmentation of the credibility distribution via linguistic analysis and automated evidence document collection).
6 Approval number: KH2018-080

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7, after the conclusion section.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 7, after the conclusion section.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3
3 B1. Did you cite the creators of artifacts you used? 3
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 3

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 3

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4
C 3 Did you run computational experiments? 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 5.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 5.3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3.2

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? 3

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 3

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? 3
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? 3

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: 3.2

heading: Future Work
text: 58% F1, leaving a room for future improvements. Limitations
There are two known limitations of the SQuADlike annotation approach we used in this work: (1) It can result in higher lexical-overlap between the context and question pairs. (2) It leads to proportionally fewer truly information-seeking questions (Gururangan et al. , 2018; Kaushik and Lipton, 2018). The main reason is that the annotators create questions after reading a paragraph, which can induce bias towards recycling words and phrases observed in the context. Our annotation guidelines advise against this, but it is difficult to avoid entirely. Several approaches have been proposed to mitigate this issue, such as Natural Questions (Kwiatkowski et al. , 2019) and TyDiQA (Clark et al. , 2020). However, they tend to be expensive, and comparatively, the SQuAD-like method is resource efficient and a more suitable starting point for low-resourced
languages such as Tigrinya. Finally, the current dataset does not include adversarial examples to measure the capability of models to abstain from providing an answer when it does not exist in the context; this extension is left for future work