HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language
Year: 2023

Abstract:
This paper presents HaVQA, the first multimodal dataset for visual question-answering (VQA) tasks in the Hausa language. The dataset was created by manually translating 6,022 English question-answer pairs, which are associated with 1,555 unique images from the Visual Genome dataset. As a result, the dataset provides 12,044 gold standard EnglishHausa parallel sentences that were translated in a fashion that guarantees their semantic match with the corresponding visual information. We conducted several baseline experiments on the dataset, including visual question answering, visual question elicitation, text-only and multimodal machine translation.

heading: 1 Introduction
text: In recent years, multidisciplinary research, including Computer Vision (CV) and Natural Language Processing (NLP), has attracted many researchers to the tasks of image captioning, cross-modal retrieval, visual common-sense reasoning, and visual question answering (VQA, Antol et al., 2015; Goyal et al., 2017). In the VQA task, given an image and a natural language question related to the image, the objective is to produce a correct natural language answer as output (Kafle and Kanan, 2017; Ren et al., 2015a). VQA is one of the challenging tasks in NLP that requires a fine-grained semantic processing of both the image and the question, together with visual reasoning for an accurate answer prediction (Yu et al., 2019b). The general approaches followed by existing VQA models include i) extracting features from the questions and the images, and ii) utilizing the features to understand the image content to infer the answers.
Recently, research on improving visual questionanswering systems using multimodal architectures and sentence embeddings (Kodali and Berleant, 2022; Urooj et al., 2020; Gupta et al., 2020; Pfeiffer
et al., 2021) has seen tremendous growth. However, most of the VQA datasets used in the VQA research are limited to the English language (Kafle and Kanan, 2017). Although the accuracy of the VQA systems for English improved significantly with the advent of Transformer-based models (e.g., BERT, Devlin et al., 2018), the lack of VQA datasets has restricted the development of such systems in most languages, especially the low-resource ones (Kumar et al., 2022).
The availability of original datasets for state-ofthe-art natural language processing tasks has since been appreciated, especially on the African continent. While some of the efforts to create such datasets for African languages are supported by funding such as Facebook’s Translation Support for African Languages, the Lacuna Fund1, and many others, including the dataset in this work are driven by the enthusiasm for developing quality NLP solutions for African languages that are useful to the local communities.
Contributions: The main contribution of this work is building a multimodal dataset (HaVQA) for the Hausa language, consisting of question-answer pairs along with the associated images and is suitable for many NLP tasks. As per our knowledge, HaVQA is the first VQA dataset for Hausa language, and will enrich Hausa natural language processing (NLP) resources, allowing researchers to conduct VQA and multimodal research in Hausa.

heading: 2 Related work

heading: 2.1 Datasets for African NLP
text: African languages are low-resourced; many do not have any datasets for everyday NLP tasks. While some datasets exist for some African languages, they are often limited in scope or lack the necessary quality (Kreutzer et al., 2022). For Visual
1https://lacunafund.org/
10162
Question Answering, no publicly available dataset exists in any African language. Luckily, there has been a recent surge in efforts by researchers to create datasets for African languages. In this section, we provide an overview of some of the most recent examples of such efforts.
Abdulmumin et al. (2022) created the multimodal Hausa Visual Genome dataset for machine translation and image captioning. Adelani et al. (2022a) created the MAFAND-MT2 collection of parallel datasets between 16 African languages and English or French. The HornMT3 dataset was created for machine translation in languages in the Horn of Africa. Akera et al. (2022) created about 25,000 parallel sentences between 5 Ugandan languages and English, covering topics such as agriculture, health and society.
Muhammad et al. (2022) classified about 30,000 tweets in each of the four major Nigerian languages as either positive, negative, neutral, mixed or indeterminate for sentiment analysis task. Subsequently, the dataset was expanded to include 14 African languages, resulting in the largest sentiment dataset for African languages (Muhammad et al., 2023a,b). Aliyu et al. (2022) collected about 4,500 partially code-switched tweets for detecting hate against the Fulani herdsmen in Nigeria.
Adelani et al. (2022b) created MasakhaNER 2.0, the most extensive corpus for named-entity recognition tasks. Wanjawa et al. (2022) created the multipurpose Kencorpus, a speech and text corpora for machine translation, text-to-speech, question answering, and part-of-speech tagging tasks for three Kenyan languages. KenPOS (Indede et al., 2022) corpus was created for part-of-speech tagging for Kenyan languages. KenSpeech (Awino et al., 2022) is a transcription of Swahili speech created for text-to-speech tasks.

heading: 2.2 Visual Question Answering Datasets
text: Researchers have created several visual questionanswering datasets for different purposes, including for medical research (He et al., 2021; Lau et al., 2018; Ben Abacha et al., 2021), improving reading comprehension (Li et al., 2019; Sharma and Jalal, 2022), among others.
DAQUAR (Malinowski and Fritz, 2014) was the first attempt at creating and benchmarking a dataset for understanding and developing models
2https://github.com/masakhane-io/lafand-mt/ tree/main/data/text_files
3https://github.com/asmelashteka/HornMT
in visual question answering. The dataset consists of 1,449 real-world images and 12,468 synthetic and natural question-answer pairs. Ren et al. (2015b) then created a substantially larger dataset called the COCO-QA dataset using images from the Microsoft COCO dataset (Lin et al., 2014). The dataset consists of 123,287 images with each having a corresponding question-answer pair.
Gao et al. (2015) reused the images from COCOQA to create the FM-IQA dataset. The authors instruct the annotators to create custom questions and answers, through a crowd-sourcing platform. This resulted in 250,560 question-answer pairs from 120,360 images. Other similar datasets include the Visual Madlibs (Yu et al., 2015), VQA (Antol et al., 2015), Visual7W (Zhu et al., 2016), Visual Genome (Krishna et al., 2016), CLEVR (Johnson et al., 2017a). Others, such as VQA-HAT (Das et al., 2016) and VQA-CP (Agrawal et al., 2018) datasets, extended the VQA to solve specific problems associated with the original dataset.
In the medical area, Ben Abacha et al. (2021) and Lau et al. (2018) created the VQA-Med and VQARad datasets for radiological research, respectively. The VQA-Med dataset was created from 4,200 radiology images and has multiple-choice 15,292 question-answer pairs. The VQA-Rad data consists of 3,515 questions of 11 types that were crafted manually, where clinicians on radiological images gave both questions and answers. He et al. (2021) created the PathVQA, a datasets that consists of 32,795 mostly open-ended questions and their answer pairs that were generated from 4,998 pathology images.
One common theme across all these datasets is that they are all in English. For the majority of other languages, there exists only a few or no datasets for visual question answering tasks. Some of the few in other languages include the original FMIQA which was created in Chinese before being manually translated into English. Another is the Japanese VQA (Shimizu et al., 2018), where the authors used crowdsourcing to generate. It consists of 99,208 images, each with eight questions, resulting in 793,664 question-answer pairs in Japanese. For African languages, however, there is no dataset for visual question answering tasks. This is the gap that we are trying to fill with the creation of the HaVQA dataset.

heading: 3 Focused Language
text: Hausa is a Chadic language and is the largest indigenous African language that is spoken as the first or second language by about 79 million people, mainly in northern Nigeria, Niger and northern Cameroon, but also in Benin, Burkina Faso, Central African Republic, Chad, Congo, Côte d’Ivoire, Gabon, Sudan and Togo.4 The language has welldocumented literature and is studied at various local and international institutions. Several international radio stations, including BBC,5 VoA,6 and DW7 run Hausa broadcasting service.
In the early days, the Hausa language was written in “Ajami”, using Arabic scripts, mainly because of the earlier contacts between the Arabs and the Hausa people (Jaggar, 2001). Nowadays, the language is predominantly written in Latin script. This resulted from the colonial influence that began in the early 19th century. Hausa text is written using the English alphabet except for p, q, v and x, with some additional special letters:  , ¡, ¨ and ¯. Nowadays, especially on social media, writing in Hausa has experienced some form of distortion, such as the use of characters p for f, v for b, q or k for ¨ and d for ¡.

heading: 4 The HaVQA Dataset
text: This section provides a detailed description and analysis of HaVQA Questions and Answers.
4https://www.ethnologue.com/language/hau 5https://www.bbc.com/hausa 6https://www.voahausa.com/ 7https://www.dw.com/ha/labarai/s-11605

heading: 4.1 Data Collection and Annotation
text: We extracted the images along with the questionanswer (QA) pairs8 from the Visual Genome dataset (Krishna et al., 2017). The Visual Genome dataset was created to provide a link between images and natural text, supplying multimodal context in many natural language processing tasks.
We used 7 Hausa native speakers to generate the translations of the QA pair manually. The annotation process was done using a web application developed by integrating the Hausa keyboard to provide easy access to Hausa special characters and restrict access to the unused characters, as shown in Figure 1. During the translation exercise, a set of instructions was provided to annotators, including (i) “the translation should be manually generated without using any translation tool” and (ii) “the on-screen keyboard or any other keyboard that supports the Hausa special characters should be used.” These simple and easy-to-remember instructions were adopted to ensure data authenticity and the quality of the dataset. Importantly, the picture has always been presented to the translators.

heading: 4.2 Data Validation
text: After the annotation, each question and the answer were validated to ensure the quality and consistency of the translations, including the basic check that the translations were done using the correct alphabet and special characters. We validated the translations by relying on 7 Hausa language experts. A separate interface was created for the validators to be able to see the images, the original English Question-Answer pair, and the translations of all the pairs that were generated by the annotators in the first phase at once.
8https://visualgenome.org/static/data/dataset/ question_answers.json.zip
The common problem was that the annotators mixed up the choice of words when translating objects that did not have a clear masculine or feminine grammatical gender. Examples of such cases are: “Ina ¯ar tsana ruwan hoda?” (gloss: Where is the pink teddy bear?), where “¯ar” (feminine) was used, but “yake” (masculine) was used in “Me teddy bear ¡in yake sanye dashi?” (gloss: What is the teddy bear wearing?).
Another problem was that the annotators were still using b, d, k and y instead of the special characters  , ¡, ¨ and ¯. Using plain ASCII instead of accented symbols introduces ambiguities that can be sometimes resolved only by consulting the original English questions or answers or the associated image. An example is the different meanings of the words “kare” (dog) and “¨are” (finish). Some annotations included ’y for ¯ (e.g., “Ina ’yar tsanar dabbar?”, instead of “Ina ¯ar tsanar dabbar?”).

heading: 4.3 Data Analysis
text: The HaVQA consists of questions and their corresponding answers. For each image, at least one and at most five questions were asked and answered; see the distribution in Figure 2. We used the punkt9 tokenizer in the NLTK toolkit (Bird et al., 2009) for tokenization. Some relevant statistics in the created HaVQA dataset are shown in Table 1.
9https://www.nltk.org/api/nltk.tokenize.punkt. html

heading: 4.3.1 Questions
text: A variety of question types were included in the original English data; they start with the question words: what, why, how, when, where, and who. In the created HaVQA, these words were translated based on the context in which they appeared. In the Hausa language, these question types vary according to usage, gender, and dialect. For example, the word “who” is translated as “wanne” if it is associated with the male gender, or “wacce” or “wace” for female. The statistics of the different question types (based on the words that start the question) are shown in Table 2. The Hausa questions vary, from as short as 2 to as long as 16 words. The length distribution is shown in Figure 3.
In Figure 4, we show the distribution of Hausa words used when asking a question based on the English question tags of the original dataset. While most of the question tags are used at the beginning of the sentence, as in English, “nawa/nawane” (how much) is mostly used when the subjects that need counting are mentioned, e.g., 2nd in the example “Jirgin leda nawa ne a jikin wannan hoton?” (How many kites are there in this picture?) and 3rd in the example “Maza nawa ne akan dusar ¨an¨arar?” (How many men are there in the snow?).

heading: 4.3.2 Answers
text: Based on the questions, various answers are included, ranging from a single word (or number) to a short description. The distribution of answer lengths is shown in Figure 3. The distribution of answers for the question types is shown in Figure 10.

heading: 5 Sample Applications of HaVQA
text: We tested the HaVQA data by experimenting with the following NLP tasks: i) Questions, Answers, and Images for visual question answering, multimodal machine translation, ii) Questions, and Images for visual question elicitation, and iii) Questions and Answers for text-only machine translation.

heading: 5.1 Visual Question Answering
text: We used multimodal Transformer-based architecture for VQA consisting of three modules: i) feature extraction module—which extracts features from the image and question, ii) fusion module— which combines both textual and image features, and iii) classification module—which obtains the answer (Siebert et al., 2022).
Similarly, we used Visual Transformer (ViT) for image feature extraction (Dosovitskiy et al., 2020)
and multilingual BERT for Hausa, i.e., Hausa BERT10 for extracting features from the Hausa questions. The classifier which obtains the answer is a fully connected network with output having dimensions equal to the answer space. This architecture is illustrated in Figure 5. We used the Wu and Palmer metric for VQA evaluation (Wu and Palmer, 1994b).

heading: 5.2 Machine Translation
text: We performed text-only and multi-modal translation using the HaVQA dataset. We partitioned the dataset into train/dev/test sets in the ratio of 80:10:10 as shown in Table 4.

heading: 5.2.1 Text-Only Translation
text: We used the questions and answers in English and Hausa for text-only translation. We utilized two approaches for training the Transformer model (Vaswani et al., 2018): training from scratch and fine-tuning a pre-trained multi-lingual model. We evaluated the models’ performance using SacreBLEU (Post, 2018) for the dev and test set.
Transformer Trained from Scratch We used the Transformer model as implemented in OpenNMT-py (Klein et al., 2017).11 Subword units were constructed using the word pieces algorithm (Johnson et al., 2017b). Tokenization is handled automatically as part of the pre-processing pipeline of word pieces.
We jointly generated a vocabulary of 32k subword types for both the source and target languages, sharing it between the encoder and decoder. We used the Transformer base model (Vaswani et al., 2018). We trained the model on the Google Cloud Platform (8 vCPUs, 30 GB RAM) and followed
10https://huggingface.co/Davlan/ bert-base-multilingual-cased-finetuned-hausa
11http://opennmt.net/OpenNMT-py/quickstart. html
the standard “Noam” learning rate decay,12 see Vaswani et al. (2017) or Popel and Bojar (2018) for more details. Our starting learning rate was 0.2, and we used 8000 warm-up steps. The model was trained using 200k training steps and 3k validation steps, and the checkpoints were saved at 3k steps.
Fine-tuning We also employed fine-tuning a large pre-trained model on our domain. This approach has been shown to leverage monolingual data and multilingualism to build a better translation model (Adelani et al., 2022a). We used the M2M-100 pretrained encoder-decoder model (Fan et al., 2022). The model was built to translate between 100 language pairs, including Hausa and 16 other African languages. Specifically, we finetuned the 418 million parameter version of M2M,13 for three epochs. We used a maximum of 128 tokens for both the target and source sentences and a beam size of 5 during decoding. We trained the model on Google Colab (1 GPU, Tesla T4).

heading: 5.2.2 Multi-Modal Translation
text: We used the QA pairs of English and Hausa and the associated images for multimodal machine translation (MMT). Multimodal translation involves utilizing the image modality and the English text for translation to Hausa. It extracts automatically
12https://nvidia.github.io/OpenSeq2Seq/html/ api-docs/optimizers.html
13https://huggingface.co/facebook/m2m100_418M
learned features from the image to improve translation quality. We take the MMT approach using object tags derived from the image (Parida et al., 2021).
We first extract the list of English object tags for a given image using the pre-trained Faster RCNN (Ren et al., 2015c) with ResNet101 (He et al., 2016) backbone. We consider up to top 10 object tags for each image based on their confidence scores. The object tags are concatenated to the English sentence, which needs to be translated into Hausa. The concatenation uses the special token ‘##’ as the delimiter, followed by comma-separated object tags. Adding object labels enables the otherwise textonly model to utilize visual concepts which may not be readily available in the original sentence, and to supply context information for easier disambiguation. The English sentences and object tags are fed to the encoder of a text-to-text Transformer model, as shown in Figure 6.
We used the pre-trained M2M-100 Transformer. We trained the model on the Google Cloud Platform (1 GPU, NVIDIA T4). For comparison, we keep the dataset division the same as the text-only translation, as shown in Table 4.

heading: 5.3 Visual Question Elicitation
text: Similar to image captioning, we used the images and associated questions to train an automatic visual question elicitation (VQE) model. We extracted visual features using the images and fed them to an LSTM decoder. The decoder generates the tokens of the caption autoregressively using a greedy search approach (Soh, 2016). Trained to minimize the cross-entropy loss on the questions from the training data (Yu et al., 2019a) was minimized. The architecture is illustrated in Figure 7.
Image encoder All the images were resized to 224×224 pixels, and features from the whole image were extracted to train the model. The feature
vector is the output of the final convolutional layer of ResNet-50. It is a 2048-dimensional feature representation of the image. The encoder module is a fixed feature extractor and, thus, non-trainable.
LSTM decoder A single-layer LSTM, with a hidden size of 256, was used as a decoder. The dropout is set to 0.3. During training, for the LSTM decoder, the cross-entropy loss is minimized and computed using the output logits and the tokens in the gold caption. Weights are optimized using the Adam optimizer (Kingma and Ba, 2014) with an initial learning rate of 0.001. Training is halted when the validation loss does not improve for ten epochs. We trained the model for 100 epochs.
VQE Dataset We used the Hausa Visual Genome (Abdulmumin et al., 2022) and HaVQA datasets to build our Hausa vocabulary, resulting in 7679 Hausa word types for question generation. The question elicitation experiment was carried out using the 1,555 images present in the HaVQA dataset. For training and evaluation of the visual question elicitation, we have considered images and questions as shown in Table 5. As in VQE, we only considered images and their associated questions ignoring answers which are not necessary for the question generation, the statistics of the dataset differ from the multimodal translation tasks.

heading: 6 Results and Discussion
text: This section presents the results obtained after implementing the experiments described in Section 5.

heading: 6.1 Visual Question Answering
text: We employed state-of-the-art language and vision models for Visual Question Answering to report a viable baseline. Table 6 presents the different image encoders we use to obtain our experiment results in combination with the Hausa BERT-based text encoder. The text encoder remains the same across all these experiments. The WuPalmer (Wu and Palmer, 1994a) score was chosen as the metric
to evaluate the baselines. The WuPalmer metric measures semantic similarity between words based on their depth in a lexical hierarchy and the depth of their common ancestor. The metric ranges from 0 to 1, with higher values indicating greater similarity. It is widely employed in tasks such as word sense disambiguation, information retrieval, and semantic relatedness estimation.
From the results reported, the Data-Efficient Image Transformers (DeiT, Touvron et al., 2021 model proposed by Facebook yielded the best results in our architecture. It reached a score of 30.85 and became our best-performing baseline. ViT-base and BEiT Large yielded scores of 28.90 and 27.75, respectively. ViT Large reported a WuPalmer score of 29.67. The DeiT models utilize a distillation token to transfer knowledge from a teacher to a student model through backpropagation. This transfer occurs via the self-attention layer, involving the class token (representing the global image representation) and patch tokens (representing local image features). The distillation token interacts with these tokens, assimilating important information from the teacher model and effectively transferring its knowledge. As a result, the student model trained with the distillation token demonstrates improved performance compared to models trained solely with supervised learning.
In our study, we conducted manual validation of the results generated by the Visual Question Answering (VQA) model. Our analysis revealed that the model exhibited higher performance when tasked with answering questions that required oneword answers. In these cases, the model consistently provided precise answers for the majority of questions and achieved a very good for the remaining ones.
The training dataset used for training the VQA model consisted of 5500 instances, while the test dataset comprised 520 instances. To provide further insight into the distribution of answers, we
.
presented Figure 3, which plots the distribution of word counts in the answers.
By focusing on questions that necessitate oneword answers, our study aimed to explore the extent to which the VQA model can excel in a more restricted task akin to classification. The choice to emphasize single-word answers allowed us to investigate the model’s capabilities within a specific context and assess the potential impact of this narrowed scope on its performance. The observed errors were mainly associated with cases where there is a dominant object in the picture. The dominant object is returned as the answer regardless the question, see the example prediction in Figure 8. More sample VQA outputs are provided in Figure 11 in the Appendix. Example 4 in Figure 11 illustrates the same problem when answering the question “Wacce dabba ce ta fito?” (gloss: What animal is shown?). Some systems respond with the word “ciyawa” (grass), because it is the dominant element in the picture.

heading: 6.2 Machine Translation
text: The text-to-text and multi-modal translation model results are shown in Table 7. For the text-only translation, fine-tuning the Facebook M2M-100 model on the questions and answers for English→Hausa
translation delivers a score by about +8.4 BLEU points better than training a Transformer model, and +11.6 for Hausa→English. The multimodal translation model achieved a decent performance comparable to the text-only translation (−0.8 BLEU points).
The better performance by the text-only translation model is expected because, unlike in the Visual Genome dataset (Abdulmumin et al., 2022), the sentences in HaVQA are mostly unambiguous and, hence, do not require the context that was provided by the images. Also, it is possible that the text captions extracted from the image brought different synonyms than what the single reference translation in Hausa expects. This situation would lead to a comparably good translation quality when assessed by humans but a decreased BLEU score.

heading: 6.3 Visual Question Elicitation
text: Because it is difficult to measure the quality of the generated questions using automatic evaluation metrics, we manually evaluated the samplegenerated questions, relying on a native Hausa speaker. We sampled about 10% of the elicited questions and subjected them to manual evaluations. We categorized each of the sampled questions as either “Exact”, “Correct”, “Nearly Correct” and “Wrong”. We present the distribution of these classes in Figure 9, and provide some samples in Figure 12 in the Appendix.
All the generated predictions were valid (reasonable) questions, with all but 3 (99.5%) having the question mark (‘?’) appended at the end of the question. The distribution of the question types are: “menene/me/mene” (what)–64.1%, “ina/a ina” (where)–26.4%, “yaushe/da yaushe” (when)–2.8%, “waye/wacce/wana/wanne” (who)–3.7%, “meyasa” (why)–1.5%, “nawane/nawa” (how much)–1.32% and “yaya” (how)–0.2%.

heading: 7 Conclusion and Future work
text: We present HaVQA, a multimodal dataset suitable for many NLP tasks for the Hausa language, including visual question answering, visual question elicitation, text and multimodal machine translation, and other multimodal research.
The dataset is freely available for research and non-commercial usage under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License at: http://hdl.handle.net/11234/ 1-5146. We released our experimental code through Github.14
Our planned future work includes: i) extending the dataset with more images and QA pairs, ii) providing ground truth for all images for image captioning experiments, and iii) organizing a shared task using HaVQA.
Ethics Statement
We do not envisage any ethical concerns. The dataset does not contain any personal, or personally identifiable, information, the source data is already open source, and there are no risks or harm associated with its usage.
Limitations
The most important limitation of our work lies in the size of the HaVQA dataset. However, substantial further funding would be needed to resolve this. For the baseline multimodal experiments, we did not use the image directly but resorted to extracting textual tags and including them in the text-only translation input. A tighter fusion technique may give better performance.
14https://github.com/shantipriyap/HausaVQA/ tree/main

heading: Acknowledgements
text: The HaVQA dataset was created using funding from the HausaNLP research group. This work is supported by Silo AI, Helsinki, Finland. This work has received funding from the grant 19-26934X (NEUREM3) of the Czech Science Foundation and has also been supported by the Ministry of Education, Youth, and Sports of the Czech Republic, Project No. LM2018101 LINDAT/CLARIAH-CZ.

heading: A Annotators and Validators Recruitment
text: We recruited Hausa natives from the team of experienced translators at the HausaNLP research group as annotators and validators. The team of annotators consisted of 4 females and 3 males, while the validators included 3 females and 4 males. Each member of the annotation/validation team have at least an undergraduate degree. They all reside in different parts of Northern Nigeria.

heading: B Annotation Guidelines
text: The following guidelines were provided to the native Hausa annotators and validators:
1. Remember to read the Hausa typing rules. Before starting annotation, test it once and report for any issues.
2. The annotator must be a native speaker of the Hausa language.
3. Look at the image before annotating.
4. Try to understand the task, i.e., translate the questions and answers into the Hausa language.
5. Do not use any Machine translation system for annotation.
6. Do not enter dummy entries for testing the interface.
7. Data will be saved at the backend.
8. Press the Shift Key on the virtual keyboard for complex consonants.
9. Contact the coordinator for any clarification/support
C Question-Type Answer Distribution
wani ka la guda na wa
0 1 8 0
3 6 0
5 4 0
hudu daya
3 4
hu¡u 2
babu
uku
1
biyu
¡aya
azurfakorayefara ba¨a koriya
rawaya
ja
ba¨i
kore
shu¡i
fari
A.
menene (s.) waye menene (pl.)
0 6 0
1 2 0
1 8 0
dutse benci tanis
bishiyoyi katako
dawakai tsayuwa ¨asa ruwa
ciyawa
tafiya
mutum yarinyar
mace
yaro
bakowa
matar
namiji
mutane
mata
mutumi
mutumin
tabarau
ba¨i
shinge
ciyawa
dutse
hula
bishiyoyi
bishiyu
kore
fari
katako
B.
ina wani iri yay a wace da bba wani iri /yanayi
0 8
1 6
2 4
firisbi
kwala-kwale
doki
daisies
bambaro
hotdog
sanwic
gilashi
karnuka
jeans
mage
karkanda
yak
saniyoyi
kwa¡o
kuliya
tunkiya
agwagwa
doki
saniya
kare
mage
duhu
suya
rufe
linzami
fanko
shu¡i
gajimare
washe
haske
fari
bayyananne
tasi
waje
katako
nisan
kofi
shingled
ayaba
ba¨i
bas
fiza
kare
keyboard
gini
falon
gado
falo
titi
ruwa
kamara
fili
¨asa
bango
C.
D Visual Question Answer Sample Predictions
S/N Example Model Instances Prediction Gloss.Test Train
E Visual Question Elicitation Sample Predictions

heading: F Glossaries
text: Hausa Gloss Hausa Gloss Hausa Gloss
a in da yaushe when kore green (s.) a benci on bench daya one koriya green a bishiya on tree ¡aya one kuliya cat a gona on farm doki horse kwa¡o frog a hagu on left duhu dark kwala-kwale canoe a hanya on way (road) dutse stone kwalekwale canoe a ina where (is/are) falo parlour la’asar evening a jaka in bag falon the parlour linzami bridle akan (me) on what fanko empty lu‘u-lu‘u diamond a kamara on camera fara white (she) mace woman a kirji on chest fari white (he) mage cat a ruwa in water fata skin magen the cat a sama in air fika-fikai wings mai yasa why is a yaushe when fili field mata woman adadin the quantity firisbi frisbee matar the woman almakashi scizzors fiza pizza me what agogon the clock fuska face me aka/ya/ta what does agwagwa duck gado bed me yake da what has ayaba banana gajimare cloud me yasa why azurfa silver gilashi glass mene what babu nothing gine-gine buildings menene (s.) what is [it] babur motorcycle gini building menene (pl.) what are bacci sleep giwa elephant meyasa why [did] ba¨a black (she) giwar the elephant meye what ba¨i black (he) guda nawa how many mutane people bakowa nobody gudu run murabba’i square/quarter bambaro straw haske light mutum person bango wall hoton the image mutumi person (m.) barci sleep hudu four mutumin the man bas bus hu¡u four namiji male basbal baseball hula cap namijin the man bayyananne clear ina where (is) nawa how much/mine benci bench inane where is nawane how much [is]/it’s mine bishiyoyi trees iyo swimming na wane how much [is]/it’s mine bishiyu trees ja red raktangula rectangle biyu two jaririn the baby rana day/sun bulo block/brick kaka autumn rawaya yellow ciyawa grass kamara camera rufe close [it] daga (me) from what kare dog ruwa water daga ina from where karkanda rhinoceros saniya cow da’ira round karnuka dogs saniyoyi cows da me with what ¨asa sand saukowa coming down dame with what katako wood sanwic sandwich damisa tiger keke bicycle shago store dare night kofi cup shinge wall dawakai horses koraye green (pl.) shu¡i blue
Continued on Next Page. . .
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7.2
7 A2. Did you discuss any potential risks of your work? We couldn’t find any potential risk in our work.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sections 5 and 6

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Sections 4 and 5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4, Appendix E

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix F

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix E
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No personal data was used in this work
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We annotated an open source data and used it in this work

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix E

heading: Future Work
text: 14
Our planned future work includes: i) extending the dataset with more images and QA pairs, ii) providing ground truth for all images for image captioning experiments, and iii) organizing a shared task using HaVQA