Assessing Massive Multitask Language Understanding in Arabic (2024)

Abstract: The focus of language model evaluation has transitioned towards reasoning and knowledgeintensive tasks, driven by advancements in pretraining large models. While state-of-the-art models are partially trained on large Arabic texts, evaluating their performance in Arabic remains challenging due to the limited availability of relevant datasets. To bridge this gap, we present ArabicMMLU, the first multi-task language understanding benchmark for the Arabic language, sourced from school exams across diverse educational levels in different countries spanning North Africa, the Levant, and the Gulf regions. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) and is carefully constructed by collaborating with native speakers in the region. Our comprehensive evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models. Notably, BLOOMZ, mT0, LLaMA2, and Falcon struggle to achieve a score of 50%, while even the top-performing Arabic-centric model only achieves a score of 62.3%.1

heading: 1 Introduction
text: Although large language models (LLMs) such as GPT-3.5 (Ouyang et al., 2022), BLOOMZ (Muennighoff et al., 2022), and Jais (Sengupta et al., 2023) have been pretrained with substantial coverage of Modern Standard Arabic (MSA), their reasoning and knowledge assessments are primarily conducted using datasets translated from English to Arabic (Sengupta et al., 2023; Huang et al., 2023), which means there is limited capacity to evaluate
1Data and code can be accessed at: https://github. com/mbzuai-nlp/ArabicMMLU
content specific to Arabic. This reliance on translation systems not only demonstrates an Anglocentric approach (Ramesh et al., 2023; Talat et al., 2022) but also potentially introduces errors and biases. Given that Arabic is one of the most widely-spoken languages in the world, with a speaker population of over 400 million people (Shoufan and Alameri, 2015; Diab et al., 2017), it is critically important that datasets be constructed for the language that are regionally- and culturally-localized.
The evaluation of language models has increasingly shifted from linguistically-centric tasks, such as part-of-speech (POS) tagging and named entity recognition (NER), towards reasoning and knowledge evaluation. This shift is evidenced in evaluations of models like GPT-4 (OpenAI, 2023), LLaMA2 (Touvron et al., 2023), and LLM360 (Liu et al., 2023) on various commonsense reasoning datasets (Zellers et al., 2019; Huang et al., 2019;
5622
Koto et al., 2022, 2024), mathematical problems (Amini et al., 2019; Cobbe et al., 2021), coding challenges (Chen et al., 2021; Austin et al., 2021; Yu et al., 2024), and school exams (Hendrycks et al., 2021; Li et al., 2023; Koto et al., 2023). One notable dataset for knowledge evaluation is MMLU (Massive Multitask Language Understanding) (Hendrycks et al., 2021), which comprises multiple-choice questions across various subjects based on the US education system. In recent Arabic-centric LLMs like Jais (Sengupta et al., 2023) and AceGPT (Huang et al., 2023), knowledge evaluation was carried out using MMLU translated from English to Arabic.
To comprehensively evaluate the reasoning and knowledge capabilities of Arabic LLMs in local Arabic-speaking contexts, we introduce ArabicMMLU, styled around MMLU and sourced from school exams across Arabic-speaking countries spanning North Africa, the Levant, and the Gulf regions. ArabicMMLU was constructed through collaboration with native Arabic speakers from Jordan, Egypt, UAE, Lebanon, and Saudi Arabia (KSA), ensuring rich local context, particularly in the subject areas of history, geography, law, civics education, and driving tests. Figure 1 summarizes the distribution of education levels and corresponding subjects in ArabicMMLU. The proportion of primary school, middle school, high school, and university level questions in ArabicMMLU are 22.2%, 12.2%, 34%, and 6.1%, respectively, with the remaining questions categorized as “NA”.
Our contributions can be summarized as follows:
• We introduce the first Arabic MMLU-style dataset in Modern Standard Arabic (MSA), featuring 40 tasks covering various subject areas and educational levels across eight countries. Over 50% of the questions in our dataset are tailored to Arabic-specific contexts.
• We evaluate 22 open-source multilingual models, 11 open-source Arabic-centric models, and 2 closed-source models. GPT-4 achieves the best performance, while the open-source models struggle to achieve scores above 60%.
• We conduct a thorough analysis of the topperforming open-source models across various dimensions, encompassing: (1) individual subject areas, education levels, countries, and Arabic-specific topics; (2) few-shot inference performance; (3) model confidence; and (4) the influence of negation.

heading: 2 Related Work

heading: 2.1 Language Models in Arabic
text: Early Arabic pretrained language models typically had less than 2 billion parameters and were primarily monolingual. These models can be classified into three main categories: encoder-only, decoder-only, and encoder–decoder models. The encoder-only models, such as AraBERT (Antoun et al., 2020), CAMeLBERT (Inoue et al., 2021), AraELECTRA (Antoun et al., 2021a), and ARBERT & MARBERT (Abdul-Mageed et al., 2021), are mainly from the BERT family. AraGPT2 (Antoun et al., 2021b), on the other hand, is a decoderonly model available in different sizes ranging from 135M to 1.4B parameters. Examples of encoder– decoder models include AraT5 (Nagoudi et al., 2022) and AraBART (Kamal Eddine et al., 2022).
Jais (Sengupta et al., 2023) and AceGPT (Huang et al., 2023) are two recent Arabic-centric decoderonly models with parameter sizes of up to 30B and 13B, respectively. Jais is pretrained on a corpus of 72 billion Arabic tokens, while AceGPT builds upon LLaMA2 and is enhanced with reinforcement learning from AI feedback (Lee et al., 2023) to localize the model to Arabic values and culture. Both models are bilingual (English and Arabic), and were fine-tuned on various instruction datasets.
Arabic is also present in multilingual models. This includes earlier models such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020), and more recent LLMs such as BLOOMZ (Muennighoff et al., 2022), mT0 (Muennighoff et al., 2022), Falcon (Penedo et al., 2023), GPT-3.5 (Ouyang et al., 2022), and GPT-4 (OpenAI, 2023). In the original papers, only GPT-4 was evaluated in Arabic in terms of its reasoning and knowledge capabilities, using the English–Arabic translated MMLU dataset, reporting an accuracy of 80%.

heading: 2.2 Arabic Benchmarks for Evaluating Language Models
text: Arabic is included in various multilingual benchmarks for natural language understanding and generation, such as XGLUE (Liang et al., 2020), XTREME (Hu et al., 2020), XTREME-R (Ruder et al., 2021) and GEM (Gehrmann et al., 2021). In recent years, several Arabic-centric benchmarks have been released, such as Dolphin (Nagoudi et al., 2023), OCRA (Elmadany et al., 2023), and LAraBench (Abdelali et al., 2024). Many tasks in these benchmarks involve classification, such as natural lan-
guage inference (Conneau et al., 2018), POS tagging (Darwish et al., 2017), named entity recognition (Pan et al., 2017), and summarization (Ladhak et al., 2020). There are three notable question answering datasets: TyDiQA (Clark et al., 2020), Arabic-SQuAD (Mozannar et al., 2019), and MLQA (Lewis et al., 2020). These datasets primarily focus on reading comprehension and question answering, unlike the MMLU dataset (Hendrycks et al., 2021) which evaluates reasoning and knowledge in realworld settings, in the form of multiple-choice questions. Related, EXAMs (Hardalov et al., 2020) is a dataset based on multilingual school exams, which contains a subset of about 500 Arabic questions.

heading: 3 ArabicMMLU
text: In the Middle East, the education system mostly follows the K12 system, consisting of six years of primary school, three years of middle school, and three years of high school.2,3 Many education systems in countries within the region, such as Egypt
2https://www.pwc.com/m1/en/ industries/education/publications/ understanding-middle-east-education.pdf
3With the exception of the UAE, which follows a 4-4-4 structure for primary, middle, and high schools.
and KSA, prioritize Islamic studies alongside subjects like mathematics, natural science, social science, and geography.4 In public schools, Arabic is commonly used for teaching and assessment, while in international schools, English is the predominant language of instruction for most subjects, following either the UK or USA curriculum. When designing ArabicMMLU, we excluded questions in English and only included questions in Arabic. ArabicMMLU is an Arabic multiple-choice question-answering dataset comprising 40 tasks spanning a wide range of subjects and education levels. The questions are sourced from eight different countries in North Africa (Morocco and Egypt), the Levant (Jordan, Palestine, and Lebanon), and the Gulf (UAE, Kuwait, and KSA). Each question has 2–5 candidate answers, with one correct answer. Table 1 provides details of the subjects in ArabicMMLU. The subjects are drawn from different education levels (primary school, middle school,
4https://www.tabahfoundation. org/wp-content/uploads/2018/12/ TabahFuturesInitiative-Islamic-Education_En.pdf

heading: Group Subjects
text: high school, university, and professional) and are categorized into STEM, social science, humanities, language, and other fields.
Figures 2 and 3 showcase various examples of ArabicMMLU questions, with some focusing on history, driving tests, social science, and bar exams, all of which are pertinent to Arabic-specific norms and cultures. Notably, Arabic multiple-choice questions sometimes use Arabic-script characters (
@, H. ,
h. , X, è) rather than Latin-script characters (e.g. A, B, C, D, E). This differs from many other languages, where the answer options are strictly in Latin script (even if the local writing script is not Latin, as with Mandarin Chinese). In prior work (Hendrycks et al., 2021; Koto et al., 2023; Li et al., 2023), answering these multiple-choice questions has relied on the probability of the alphabetic options. We experiment with both Arabic and Latin script outputs in Section 4.

heading: 3.1 Data Construction
text: The data construction process involved a total of 10 Arabic native speakers from different countries: 6 internal workers (1 Jordanian, 1 Egyptian, 1 Lebanese, 1 from UAE, and 2 from KSA) and 4 external workers (3 Jordanian and 1 Egyptian).
During the first stage of data collection, the internal workers were tasked with collecting relevant
sources for data collection. These sources were URLs containing the questions, which needed to be publicly available.
In the second stage, all workers were asked to manually scrape the data within a 2-month period. The task was to collect metadata, including the source (URL of the source document), country, subject, level, question, multiple-choice options, and the correct answer key. Each external worker was assigned to gather 2,000 questions, while internal workers were tasked with gathering 1,000–2,000 questions each. Our internal workers are Master’s students and Research Assistants in Computer Science, while the external workers hold Bachelor’s degrees. We ensured competitive compensation for the workers, exceeding the monthly average wage in each respective country.
During manual data scraping, workers were instructed to include only questions accompanied by an answer key, and to discard questions containing multi-modal information (e.g., images, videos, or tables). If a question had additional contextual information (e.g., a passage referenced by several questions), the context was required to be included with each corresponding question.

heading: 3.2 Quality Control
text: While our workers are native speakers of Modern Standard Arabic with at least Bachelor’s degrees, we maintain the quality of our dataset construction through meticulous steps. Firstly, we conducted a 1-hour workshop before the data collection stage to clarify the process. Secondly, we automatically filtered out repetitive questions and those without an answer key, reducing the initial set of over 15,000 questions to 14,575 unique questions. Finally, we assessed the accuracy of our data collection by having two native Arabic speakers annotate 100 randomly sampled questions. They were provided with all metadata, including the answer key, and tasked with verifying the correctness of each sample using any available resources (e.g., search engines). We found that 96% of the questions and answer keys match on average, while the remaining could have incorrect answer keys. This 96% score is considered to represent the maximum score meaningfully achievable for ArabicMMLU.

heading: 3.3 Data Statistics
text: Table 2 presents detailed statistics of ArabicMMLU, categorized by education level and subject area. The distribution of questions across education lev-

heading: Group # Questions # Chars

heading: Country STEM Social Hum. Lang. Other TOTAL
text: Jordan 1086 2163 1579 362 863 6053 Egypt 1012 581 335 324 254 2506 Palestine 860 585 600 2 – 2047 Morocco – – 317 – – 317 Lebanon – – – – 239 239 UAE – – 56 – 128 184 Kuwait – – – – 111 111 KSA 67 – – – 37 104 Other 195 211 768 973 867 3014
TOTAL 3220 3540 3655 1661 2499 14575
Table 3: The distribution of ArabicMMLU sources by country, categorized according to subject areas. “Social”, “Hum.”, and “Lang.” denote social science, humanities, and Arabic language, respectively.
els varies, with primary school having the largest number, around 4.9K, followed by high school with 3.2K. Questions and candidate options are generally longer at the high school and university levels. Additionally, we observe that questions in the “NA” (other) category are four times longer (in characters) than those in school exams. This is expected since this category includes subjects like Arabic language (General) and Arabic language (Grammar), where questions typically involve lengthy paragraphs as context. For a detailed breakdown of questions for each subject in each education level, please refer to the Appendix (Table 7).
For subject areas, they are reasonably evenly distributed, particularly for STEM, social science, and the humanities, each consisting of roughly 3.2K to 3.5K questions. There are only minor differences in question length between these three subject areas. However, for the language category, the average question length (in characters) is 10 times longer than other categories.
Table 3 further shows the distribution of questions across the eight countries from which questions were collected, with Jordan, Egypt, and Palestine being the top three sources. Various subjects within the social sciences, humanities, and other categories (such as driving tests) often include Arabic-specific content, representing 57.7% of the dataset. While STEM questions are more aligned with the English MMLU, it is worth noting that differences in curriculum between North Africa, the Levant, the Gulf regions, and the USA may influence variations in assessment question design.

heading: 4 Experiments

heading: 4.1 Set-Up
text: Our experiments focus on zero-shot and fewshot settings across 35 models. This includes 22 open-source multilingual models (XGLM (Lin et al., 2022), BLOOMZ (Muennighoff et al., 2022), mT0 (Muennighoff et al., 2022), Falcon (Penedo et al., 2023), and LLaMA2 (Touvron et al., 2023), across various sizes), 11 open-source Arabiccentric models (AraT5 (Nagoudi et al., 2022), AraGPT2 (Antoun et al., 2021b), AceGPT (Huang et al., 2023) and Jais (Sengupta et al., 2023), also across various sizes), and 2 closed-source models (GPT-3.5: gpt-3.5-turbo (Ouyang et al., 2022) and GPT-4: gpt-4-0613 (OpenAI, 2023)).
We initially conducted experiments with four settings: (1) Arabic prompt and Arabic alphabetic output, (2) Arabic prompt and English (i.e. Latin script) alphabetic output, (3) English prompt and Arabic alphabetic output, and (4) English prompt and English alphabetic output. Figure 4 illustrates the Arabic and English prompts. The placeholders [SUBJECT], [LEVEL], and [COUNTRY] are replaced with the corresponding Arabic and English words, while the placeholders [INPUT] and [OPTION] are in Arabic. The choice of the al-
phabetic output (English vs. Arabic) is adjusted in [OPTION]. See Appendix B (Figure 10) for examples of the full input in both English and Arabic.
Following previous studies (Koto et al., 2023; Li et al., 2023), for open-source models, we determine the answer based on the highest probability among all possible options. In the case of English alphabetic output, we measure the probability of the first generated token being A, B, C, D, or E. For Arabic, we measure the probability of the first generated token being
@, H. , h. , X, or è. For closed-
source models, we determine the answer based on the first token generated in the text using a regular expression. If there is no match, we assign a random answer.

heading: 4.2 Results
text: To evaluate the influence of prompt language, we initially benchmarked the open-source models using all four prompt settings (Section 4.1), as depicted in Figure 5. We observe that the optimal configuration across all models is to use an English prompt and English alphabetic output. Predictably, the Arabic-specific LLMs — Jais-chat (30B) and AceGPT-chat (13B) — demonstrate the greatest robustness when employing Arabic alphabetic output. Please refer to Appendix for complete results of all prompt settings across the open-source models. For the remaining experiments, we will report based on the setting of English prompt and English alphabetic output.
Results across all models Table 4 shows the full results of all models, grouped by subject area. As expected, the Arabic-centric model Jais-chat (30B) emerges as the top-performing open-source model, boasting an average score of 62.3%, surpassing GPT-3.5 by 4.6 points. Compared to AceGPT-chat (13B), both Jais-chat models (13B and 30B) exhibit substantially higher accuracy in areas including STEM, Social Science, Humanities, and Others. For multilingual models such as BLOOMZ (7B) and mT0 (13B), their performance lags behind Jais, with a disparity of more than 14 points. XGLM, LLaMA2, and Falcon perform at a level close to random, suggesting their limited proficiency in Arabic. GPT-4 achieves the highest accuracy, with a score of 72.5%, surpassing Jais-chat (30B) by 10 points. It is noteworthy that in the GPT-4 technical report (OpenAI, 2023), the accuracy of the EnglishArabic translated MMLU dataset is reported as 80%, which is 8 points higher than our ArabicMMLU results. One possible explanation for this difference is that our ArabicMMLU presents a greater challenge due to its inclusion of a higher proportion of Arabicspecific content.
Furthermore, we notice a trend of increasing accuracy with larger models, with the exception of XGLM. For example, BLOOMZ (7B) achieves an accuracy 15.9 points higher than BLOOMZ (560M), while mT0 (13B) shows a 13.8-point increase compared to mT0 (300M). This trend is also evident in AceGPT and Jais, although it is less pronounced in LLaMA2 and Falcon, which are

heading: Country # Q. BLOOMZ AceGPT Jais
text: English-centric models.
Results across education levels Figure 6 depicts the average scores of the top-performing models (BLOOMZ, AceGPT-chat, Jais-chat, GPT-3.5, and GPT-4) across different education levels. We observe that ArabicMMLU questions are more challenging at the high school level compared to the primary and middle school levels. Specifically, for high school questions, GPT-4 achieves a score of only 61.7%, while Jais-chat scores 51.2%. Interestingly, we notice that the model accuracy at the university level is higher than for high school. This could be attributed to the relatively small portion (i.e., 6%) of university-level questions in ArabicMMLU, which potentially skews the results.
Results by country We present the performance of open-source models on selected subjects that potentially contain Arabic-specific contexts. These subjects include history, geography, civics, political
science, law, and driving tests, grouped by country in Table 5. We observe that BLOOMZ performs less well on questions sourced from the UAE and Morocco compared to other countries, while Jais performs best overall except in questions sourced from Morocco.

heading: 4.3 Analysis
text: We focus our more detailed analysis in this section solely on the best open-source models, namely BLOOMZ, AceGPT, and Jais, providing researchers and the community with insights to better understand these models and opportunities for future improvements.
Few-shot performance While all the results in Section 4.2 were based on zero-shot learning, we observe in Figure 7 that when we move to few-shot learning, results for base models improve but those for instruction-tuned models deteriorate. Specifically, AceGPT and Jais show an improvement of 2–10 points when using few-shot learning, but the results for BLOOMZ and Jais-chat drop. These findings are consistent with prior research over IndoMMLU (Koto et al., 2023) and CMMLU (Li et al., 2023).
Model confidence We analyze whether BLOOMZ, AceGPT, and Jais are well-calibrated in answering ArabicMMLU questions by comparing the probability of the correct answers with the actual accuracy for each task (i.e., subject and level combination). The answer probability is obtained through softmax normalization across the available candidate answers. In Figure 8, we observe that
the three open-source models are well calibrated with correlation scores r > 0.9.
Additionally, we investigate the correlation between model confidence and question length in Figure 9. We find no correlation between the length of the questions and the model confidence for either Jais or AceGPT.
Impact of negation Despite negation being an absolutely foundational linguistic phenomenon, LLMs have been shown to be worryingly insensitive to its effects in English (Kassner and Schütze, 2020; Hosseini et al., 2021; Truong et al., 2023). We thus perform an analysis of LLM performance over questions in ArabicMMLU with and without negation to determine whether this observation ports across to Arabic. We utilize specific negation phrases to identify questions containing negations in Arabic. These include: B (no), Ë (is not), I Ë (is not), ÕË (did not), Q « áÓ (without),

heading: Model W/o negation W/ negation
text: ZA J J AK. (excluding), and àðX (without). To prevent ambiguity, the term AÓ is omitted, as it can also mean “what”. After applying this filtering, we obtain 816 questions. We randomly inspected 100 random samples and found the detection accuracy for negation to exceed 95%.
Table 6 presents the accuracy of the LLMs in answering questions with and without negation in the top three subjects containing negation (Geography, Biology, and Economics). Overall, negated questions generally exhibit slightly lower accuracy, particularly in Biology and Economics. However, for Geography, the models actually achieve higher accuracy.

heading: 4.4 Discussion
text: Our experiments show that open-source LLMs perform poorly on ArabicMMLU questions, particularly multilingual models. Furthermore, the Arabiccentric LLMs still struggle to capture Arabic cultural knowledge across all education levels. This highlights a significant need for improvement in Arabic LLMs. In contrast, GPT-4 demonstrates remarkable performance across all tasks, surpassing all other models. However, it remains unclear whether the success of GPT-4 results from scaling up the dataset and model size or simply from memorization (given that all questions were taken from public sources).

heading: 5 Conclusion and Future Work
text: We introduce ArabicMMLU, the first large-scale multi-task language understanding dataset designed to evaluate real-world knowledge in Arabic. Through experiments with over 14K multiplechoice questions spanning various subjects and education levels, we observed that Arabic-centric LLMs outperform multilingual LLMs, albeit with lower accuracy than GPT-4. We envision ArabicMMLU as a valuable resource for tracking the real-world knowledge and reasoning capabilities of future Arabic LLMs. For future work, ArabicMMLU can be extended to include short-answer or essay questions, different modalities (i.e., images, audio, video), larger region coverage, and more questions in professional settings. This will enhance the evaluation to better reflect real-world scenarios.

heading: Limitations
text: Although we believe our benchmark will significantly contribute to the advancement of Arabic LLMs, it is important to acknowledge limitations that need to be addressed in future work. We outline these limitations as follows:
Limited diversity ArabicMMLU does not represent all Arabic countries equally. For example, we have collected over 6K multiple-choice questions from Jordan, while other countries are represented with only 100 questions or, in some cases, not at all. This is largely due to the availability of publiclyaccessible exams in each country; some countries have digitized their exams, but not others. Additionally, our search for relevant Arabic content across the internet was not exhaustive.
Dialectical Arabic is excluded The dataset primarily focuses on Modern Standard Arabic (MSA). However, multilingual and Arabic LLMs are often exposed to both MSA and dialectical Arabic.
Text-based questions only ArabicMMLU is focused solely on text-based assessment, and the exploration of multimodal questions is left for future work.

heading: Ethical Considerations
text: It is important to emphasize that our experimental results do not provide conclusive answers regarding the performance of LLMs in Arabic. This issue becomes even more vexing when discussing the GPT-4 results, which outperformed all models, due
to a lack of sufficient information about its training regimen. As such, we cannot assert that the model’s pretraining data is free from contamination.

heading: Acknowledgements
text: We extend our gratitude to all collaborators from Jordan, Egypt, Lebanon, UAE, and Saudi Arabia who participated in the data collection process. We also acknowledge the contributions of Samta Kamboj, Sarah Al Barri, and Onkar Pandit from Core42, who assisted in collecting the Arabic Language question dataset.

heading: A Data Statistics
text: Table 7 presents the distribution of ArabicMMLU data categorized by subject across different education levels.

heading: B Examples
text: Figure 10 illustrates a complete example of prompts used in this study. This example features a Natural Science question with prompts provided in both Arabic and English.

heading: C Detailed Experiment Results
text: Table 8 presents the detailed zero-shot results across subjects and education levels, while Table 9, Table 10, Table 11 display the results with different prompts and alphabetic outputs (complementing the main result at Table 8).

heading: Other / NA

heading: University and Professional

heading: High School

heading: Middle School

heading: Model (#parameters) STEM Social Humanities Arabic Other AverageScience Language

heading: Model (#parameters) STEM Social Humanities Arabic Other AverageScience Language

heading: Model (#parameters) STEM Social Humanities Arabic Other AverageScience Language

heading: D Model Artifacts
text: Table 12 lists the sources of pre-trained models used in this study. All models are sourced from Huggingface (Wolf et al., 2020).

heading: Future Work
text: Although we believe our benchmark will significantly contribute to the advancement of Arabic LLMs, it is important to acknowledge limitations that need to be addressed in future work. We outline these limitations as follows:
Limited diversity ArabicMMLU does not represent all Arabic countries equally. For example, we have collected over 6K multiple-choice questions from Jordan, while other countries are represented with only 100 questions or, in some cases, not at all. This is largely due to the availability of publiclyaccessible exams in each country; some countries have digitized their exams, but not others. Additionally, our search for relevant Arabic content across the internet was not exhaustive. Dialectical Arabic is excluded The dataset primarily focuses on Modern Standard Arabic (MSA). However, multilingual and Arabic LLMs are often exposed to both MSA and dialectical Arabic. Text-based questions only ArabicMMLU is focused solely on text-based assessment, and the exploration of multimodal questions is left for future work