Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU
Authors: Fajri Koto
Nurul Aisyah
Haonan Li
Timothy Baldwin
Venue: EMNLP 2023 Main (2023)
Keywords: large language models
Indonesian school exam problems
evaluation
local languages and cultures
TL;DR: We evaluate large language models on school exam questions across different education levels and subjects in Indonesia, and we find that the best model only passes primary school exams, with limited knowledge of local languages and cultures.

Abstract: Although large language models (LLMs) are often pre-trained on large-scale multilingual texts, their reasoning abilities and real-world knowledge are mainly evaluated based on English datasets. Assessing LLM capabilities beyond English is increasingly vital but hindered due to the lack of suitable datasets. In this work, we introduce IndoMMLU, the first multi-task language understanding benchmark for Indonesian culture and languages, which consists of questions from primary school to university entrance exams in Indonesia. By employing professional teachers, we obtain 14,981 questions across 64 tasks and education levels, with 46% of the questions focusing on assessing proficiency in the Indonesian language and knowledge of nine local languages and cultures in Indonesia. Our empirical evaluations show that GPT-3.5 only manages to pass the Indonesian primary school level, with limited knowledge of local Indonesian languages and culture. Other smaller models such as BLOOMZ and Falcon perform at even lower levels.

Large Language Models Only Pass Primary School Exams in Indonesia:
A Comprehensive Test on IndoMMLU
Fajri Koto' Nurul Aisyah* Haonan Li'
Timothy Baldwin
*Department Natural Language Processing, MBZUAI
Quanticc School of Business and Technology The University of Melbourne
fajri. kotikkkotmmbbzi...mm .a..aaaaccaaa. ac, nurulaisyah. www.hhiiinmmm..ccmmm com, (haonan 1i, timothy. waaothy..bllliiiii bal.winjjnbzzaii. aww.:baaaaaiaa...aa... acc
Abstract
150)
Although large language models (LLMs) are of- ten pre-trained on large-scale multilingual texts, their reasoning abilities and real-world knowl- edge are mainly evaluated based on English datasets. Assessing LLM capabilities beyond English is increasingly vital but hindered due to the lack of suitable datasets. In this work, we in- troduce IndoMMLU, the first multi-task language understanding benchmark for Indonesian cul- ture and languages, which consists of questions from primary school university entrance ams in Indonesia. By employing professional teachers, we obtain 14,981 questions across 64 tasks and education levels, with 46% of the questions focusing on assessing proficiency in the Indonesian language and knowledge nine local languages and cultures in Indonesia. Our empirical evaluations show that GPT-3.5 only manages pass the Indonesian primary school level, with limited knowledge of local Indone- sian languages and culture. Other smaller mod- els such BLOOMZ and Falcon perform even lower levels.'
High school
Junier
Social
Hum
LOOT
Indo
(SMP) 20%
STEM
STEM
Indo
Sociel
MMLU ONI
Local
Indo
STEM
Hum
Social
Sociat
Ham
University
STEA
Local
entrance test
14%
Senior High school (SMA)) - 32%
Figure 1: Distribution subject areas and education levels in IndoMMLU. "Hum", "Social", "Indo", and "Lo- cal" refer to Humanities, Social Science, Indonesian Language, and Local Languages and Cultures, respec- tively.
edge of LLMs, given that these tests are meticu- lously designed by expert educators, drawing upon the principles of learning science. At various educa- tional levels, school exams function as assessment tools, evaluating not only language proficiency but also higher-order cognitive skills such as compre- hension, analytic abilities, and the application of real-world knowledge across diverse scenarios (No- vak, 1988).
1 Introduction
The evaluation of large language models (LLMs)
has predominantly relied on English datasets to assess language proficiency (Wang et al., 2018;
Baradaran al., 2022), reasoning abilities (Zellers et al., 2019; Huang et al., 2019; Bisk et al., 2020;
Hendrycks al. (2021) proposed MMLU, mas- sive multitask language understanding benchmark in English that compiled from different exams, covering topics including US history, computer sci- ence, and high school subjects. Recent progresses on LLMs such as LLaMA (Touvron et al., 2023) and GPT-4 (OpenAI, 2023) use MMLU one of the evaluation datasets. In the GPT-4 technical report, automatic evaluation is further extended encom- pass various standardized exams, including SAT. GRE, and bar exams.
Talmor et al., 2019), and real-world knowledge (Hendryeks et al., 2021). LLMs such as GPT-3.5 (Ouyang al.. 2022), Falcon (Penedo et al., 2023), and BLOOMZ (Muennighoff et al., 2022), however, are trained on large-scale multilingual data, and thus is critical to evaluate what knowledge they capture and their reasoning abilities languages beyond English.
School exams serve as powerful means to as- sess the reasoning abilities and real-worke knowl-
While there has been plethora of work on LLM evaluation for English (OpenAI, 2023; Katz et al.,
'Code and datasct can be found at htt::: httss:///itttub.oomm com/ fajri91/IndoMLJI

---

2023; Choi al., 2023; Ryznar, 2023; Chalkidis,
under-represented languages but also incor- porate specific cultural content, such as art, poetry, and daily life. For Lampungic (1jp) and Makassarese (mak) in particular, this is the very first NLP resource to be released.
2023), there has been comparatively little work in other languages (Li et al., 2023b; Sengupta et al., 2023). Recent work by OpenAI (2023) evaluated GPT-4 using a translated version of MMLU, and re- ported strong performance. While encouraging, us- ing translations of English evaluation datasets has serious shortcomings, including translation noise, a complete lack of content that is sensitized to the local language/culture (esp. as most English evalua- tion datasets are highly US centric), and conversely, the existence of content that irrelevant to the lo- cal language/culture (e.g. questions relating to US law or customs) and incongruent with the language- specific evaluation (Liu et al., 2023a).
We evaluate various multilingual LLMs, including GPT-3.5 (Ouyang et al., 2022). XGLM (Lin et al., 2021), Falcon (Penedo et al. 2023), BLOOMZ (Muennighoff et al., 2022), mTO (Muennighoff et al., 2022), LLaMA (Touvron et al., 2023), and Bactrian- X (Li et al., 2023a), across different model sizes. We find that only GPT-3.5 passes the highest primary school level exam, and no models demonstrate familiarity with local In- donesian languages and culture.
In this paper, we ask professional teachers (of Indonesian nationality) to collect exam ques- tions from various educational levels in Indonesian schools (i.e. primary school, junior high school, senior high school, and university). We categorize the collected questions into different subject areas, including: (1) STEM (Science, Technology, Engi- neering, and Mathematies); (2) Social Science; (3) Humanities; (4) Indonesian Language: and (5) Lo- cal Languages and Cultures. Figure presents overview of the distribution of the resulting dataset, IndoMMLU, across different subject areas and edu- cation levels. It is worth mentioning that 21 % of the questions specifically focus on the Indonesian language, and 25% encompass nine distinct local languages and cultures that are specific to Indone- sia.
2 Related Work
Evaluating Large Language Models Various benchmarks have been released to evaluate En- glish pre-trained LMs (Devlin et al., 2019; Con- neau et al., 2020). Early benchmarks such as GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) consist of various natural language under- standing (NLU) tasks of different types with vary- ing training data sizes. XGLUE (Liang et al., 2020), XTREME (Hu et al., 2020), and XTREME-R (Ruder et al., 2021) serve as multilingual benchmarks of more than 20 languages. For natural language generation (NLG), the GEM benchmark (Gehrmann et al., 2021) is a collection of machine translation, summarization, and generated descriptions in many languages.
Our contributions can be summarized as follows:
We introduce the first Indonesian MMLU dataset, namely IndoMMLU, which comprises 64 tasks across different subject areas and education levels in Indonesia.
As LLMs have become larger in size and im- proved over the standard benchmarks, there has been a shift in evaluation practice to focus on reasoning abilities (Zellers et al., 2019; Huang et al., 2019; Bisk et al., 2020; Talmor et al., 2019; Koto et al., 2022a), and real-world knowl- edge (Hendrycks et al., 2021). In GPT-4 (Ope- nAI, 2023), for instance, commonsense reason- ing is evaluated using HellaSwag (Zellers et al., 2019) and WinoGrande (Sakaguchi et al., 2021). while real-world knowledge is evaluated based on school exams including MMLU (Hendrycks et al., 2021), ARC (Clark et al., 2018), and GSM-8K (Cobbe et al., 2021). Similarly, LLaMA (Touvron et al.. 2023) was evaluated using school exam problems, in addition to closed-book question answering (K wwiatkowskii et al., 2019; Joshi et al., 2017) and
Our dataset includes exam questions from school grades to 12, as well as university entrance exams. This comprehensive cover- age allows us to perform fine-grained assess- ment of the Indonesian language proficiency of existing L.Ms.
Approximately 25% of our data encompasses nine distinct local languages and cultures in Indonesia, namely Lampungic (1jp), Ba- linese (ban), Makassarese (mak), Banjarese (bjn), Madurese (mad), Sundanese (sun), Ja- vancse (jav), Dayak Ngaju (nij), and Mi- nangkabau. These questions are not only in
Forr Minungkabau culture, the Indonesian language is
used in teaching and exams.

---

Group
Question
Answer
Minangkabau subject, Mid-term exam, (SMP)
Primary school
99.6
65.5
Terjadinya hubungan induak bako dengan anak pisang. karena adanya ...
anak pisang
a relationship in Minangkabau
Junior high school
188.3
105.9
family because ...
Senior high school
167.7
130.3
University Entrance Test
204.9
186.2
Perkawinan
Marriage
B. Satu suku
B. One tribe relationship
Satu nagari
village relationship
STEM
133.7
102.4
D. Kaum
D. One sub-tribe relationship
Social science
136.2
131.9
Humanities
113.2
104.4
Minangkabba culture sutject, Mid-term exam, class (SD)
Local languages and cultures
88.4
68.0
Tari rantak kudo berasal dari
Rantak kudo dance originally comes from
Indonesian language
307.4
161.8
Tanah datar
Pasisir Selatan
A. Tanah datar
Table 1:
Average question and answer length (in charac- ters) for each education group and subject area.
Pasaman beral
B. Pesisir Selatan
D. Solok
C. Pasaman barat
D. Solok
Sundanese subject, Mid-term exam, (SD)
the RACE reading comprehension benchmark (Lai et al.,,001. 2017).
Hal - hal anu ditepikeun dina kawih, disebutna ....
Message that we want express Kawih
A. rasa
feeliig
Indonesian Pre-trained Language Models and Benchmarks Several monolingual pre-trained language models have been released for Indone- sian, including IndoBERT (Koto et al., 2020b; Wilic et al., 2020), IndeBERTweet (Koto et al., 2021), and IndoBART (Cahyawijaya et al., 2021). These models have been evaluated on NLU (e.g. In- doLEM and IndoNLU) and NL. (e.g. IndoNI.G) benchmarks. Component tasks include sentiment analysis (Koto and Rahmaningtyas, 2017; Purwari- anti and Crisdayanti, 2019), emotion classification (Saputri et al., 2018), hate speech detection, sum- marization (Koto et al., 2020a, 2022b), and trans- lation (Cahyawijaya et al., 2021; Koto and Koto, 2020).
B. amanat
advice
C. nada
tone
D. tema
theme
Figure 2: The first question focuses on the family re- lationship between anak pisang "children" and induak bako "aunt on the father's 's side". Both terms are com- monly used in Minangkabau but not in the Indonesian language. The second and third questions pertain to traditional art. Kawih in the third question means a song set to a distinctive beat in Sundanese culture. Left is the original text and right is the English translation for illustrative purposes. The bold options are the correct answer keys.
Indonesian education curriculum, and has more fine-grained education levels than MMLU.
In contemporaneous work, Cahyawijaya al. (2023) evaluated several LLMs using existing In- donesian datasets. This collection includes the re- cent NusaX dataset (Winata et al., 2023), which is a parallel sentiment analysis dataset in 10 In- donesian local languages, created through human translation. The collection also includes several question-answering datasets, such as FactQA (Pur- warianti et al., 2007), IDK-MRC (Putri and Oh, 2022), and TyDiQA (Clark al., 2020), over news and Wikipedia documents. IndoMMLU is different that explicitly evaluates reasoning, language. and cultural abilitics in fine-graine manner from the perspective of education science.
In Indonesia's curriculum, schools are catego- rized into three levels: (1) six years of primary school (Sekolah Dasar =SD"), (2) three years of junior high school (Sekolah Menengah Pertama = "SMP"), and (3) three years of senior high school (Sekolah Menengah Atas "SMA"") At primary school, pupils all grades are taught the Indone- sian language, civics, mathematics, art, sports, and religion. From grade to and in junior high school, pupils additionally learn a foreign language, local language/culture, science, and social sci- ence." 3 In senior high school, pupils study more spe- cialized natural science and social science subjects, including physics, chemistry, biology, geography, sociology, economy, and history. In IndoMMLU, we explicitly exclude mathematics because the ques- tions typically consist primarily of symbols with lit-
3 IndoMMLU
IndoMMLU multiple-choice problem in 64 subjects from different education levels, follow- ing the format of English MMLU (see Figure and Figure 3). IndoMMLU, however, is based on the
'In a recent curriculum change, science and social science have heen added from grade 3.

---

subject, Final-term exam, class (SMA)
Group
Subjects
STEM
Chemistry (SMA, UE), Biology (SMA, UE), Physies (SMA, UE). Science (SD, SMP)
Ketentuan mengenal menteri diatur dalam UUD NRI Tahun
Provisions regarding ministers regulated in the 1945
yakni dalam Bab tentang kementerian negara lepalnya pada pasal...
Constitution of the Republic of Indonesia, namely Chaptar concerning state ministries to be precise the article...
Social science
Geography (SMA, UE), Sociology (SMA, UE), Economy (SMA, UE), Civies education (SD, SMP, SMA), Social science (SD, SMP)
A. 17
B. A
B. 17
C. 17B
C. 17
D. 18
D. 18
Humanities
History (SMA, UE), Art (SD, SMP, SMA), Sports (SD, SMP, SMA), Islam religion (SD, SMP, SMA), Christian religion (SD, SMP. SMA). Hindu
E.
19
Chemistry subject, University entrance test exam
Asam okssiat adalah asam berbasa dua. Sebanyak 10 mL larutan asam oksalat diencerkan dengen air sampal volumenya 100 ml., Larutan ini digunakan untuk menitrasl 20 mL, larutan NaOH 0,2 M dongan indikator bromtimol biru, Bila titi akhir titrasi diperolah seat volume asam oksalat mencapal mL. maka konsentrasi larutan asam oksalst adalah...
Oxalic acid is two basic acid. total 10 mL of oxalic acid solution was diluted with water to volume
religion (SD, SMP. SMA)
Local languages and cultures
Lampungic (SD, SMP, SMA), Balinese (SD, SMP, SMA), Makassarese (SD, SMP, SMA), Banjarese (SD, SMP, SMA), Madurese (SD, SMP, SMA). Minangkabau culture (SD, SMP). Duyak Ngaju (SD), Sundunese (SD, SMP, SMA), Javanese (SD, SMP, SMA)
solution was used to titrate 20 NaOH solution bromthymal blue indicator. the end point of the titration obtained when the volume of oxalic acid reaches 25 mL., then the concentration of the initial oxalic acid solution
Indonesian language
Indonesion language (SD, SMP, SMA, UE)
M
A. 0,08 M
B. 0,40 M
B. 0,40 M
C.
C. 0,80 M
D. 1,60 M
1,60
Table 2:
Subject areas in IndoMMLU. "SD", "SMP", "SMA", "UE" indicate that questions in the subject are are available in primary school, junior high school, se- nior high school, and university entrance exams, respec- tively.
E. 3,20 M
Figure 3: Examples of civics and chemistry exam ques- tions. Left is the original text and right is the English translation for illustrative purposes. The bolded options are the answer keys.
only include exams that had accompanying answer keys, and to exclude problems that contained im- ages. Additionally, we organized an 1-hour work- shop to discuss the data collection procedure with all the teachers, addressing any questions or con- cerns they had. All teachers are paid competitively, higher than the Indonesian average monthly wage. 5
tle language content, and there are existing datasets for mathematical reasoning such as GSM-8K (Cobbe et al., 2021) and NumGLUE (Mishra et al., 2022).
The local language/culture subjects vary across provinces in Indonesia and depend on the local government policy. For example, in West Sumatra, Minangkubau culture is taught using the Indonesian language, while in West Java, pupils are exposed to the Sundanese language and culture. Figure 2 illustrates two exam questions for Minangkabau culture, and one exam question for Sundanese.
3.2 Quality Control
To ensure the accuracy of the data entry pro- cess, we randomly checked questions collected by each teacher. We manually verified the questions. multiple-choice options, and the corresponding an- swer keys based on the given URL, and found that each teacher conducted the work accurately. We also additionally performed automatic filtering remove repetitive questions, and remove questions that have no answer key.
3.1 Data Construction
We asked seven professional teachers with at least a bachelor's 's degree in education to gather publicly- available school exam questions in Indonesia from web sources.4 They were tasked with gathering problems for specific subject areas and educational levels, as well as metadata such as the source (i.e. URL of the source document), school level, class level, question, multiple-choice options, and the correct answer key. We instructed the teachers to
3.3 Data Statistics
After data cleansing, we obtained total of 14,981 questions, distributed over school levels and sub- jects as detailed in Figure 1; the details of each subject area are in Table 2 and the Appendix.
'The work for single teacher was equivalent to 5-day full-time job.
"The seven teachers were selected from 70 applicants.

---

200
answer protability
the probability of the full generated answer; and (2) the probability of the first token in the generated answer. For the first, we select the answer with the highest normalized log likelihood, and for the sec- ond, we simply select the key token (e.g., C) with the highest probability among all possible keys.
40-
First token probability
35
25
20-
4.2 Results
15
Figure 4 presents the zero-shot accuracy when us- ing: (1) the full answer probability; and (2) the probability of the first token in the generated an- swer. Among the open-sourced language models (LLMs) including XGLM (7.5B), Faleon (40B), BLOOMZ (7.1B), ml (13B), LLaMA (65B), and Bactrian-X X (13B), we find that estimating the answer based on the probability of the first token in the generated answer generally performs best, with the notable exception of XGLM. Thus, we report results under this configuration in the remaining sections; the full results for both settings can be found in the Appendix.
10.
XGIM
Faboo
mio
Bactrien-X
Figure 4: LLM performance (% accuracy) based on: (1) the probability of the full generated answer, and (2) the probability of the first token in the generated answer.
IndoMMLU consists of 30% primary school, 24% junior high school, 32% senior high school, and 14% university entrance exam questions. Table 1 shows the average question length for each educa- tion level and subject area. We can observe that primary school questions tend to be shorter and uni- versity entrance exam questions are longer. Indone- sian language questions have the highest average length, while local languages and culture questions are around 88 characters on average.
Results across all models Table 3 shows the av- erage accuracy for each subject area across the 24 models. To compute the scores, disregard the education level of the questions, and average scores based on the subject (e.g. Biology), and finally com- bine the scores across all subject areas (e.g. STEM). The random performance varies between 20% to 27% due to the differing number of multiple-choice options (i.e. three to five).
4 Experiments
4.1 Set-Up
We evaluate 24 multilingual LLMs of different sizes in zero-shot and few-shot settings. This includes GPT-3.5 (Ouyang et al., 2022), XGLM (Lin et al., 2021), Falcon (Penedo et al., 2023), BLOOMZ (Muennighoff et al., 2022), mTO (Muen- nighoff et al., 2022), LLaMA (Touvron et al., 2023) and Bactrian-X (Li et al., 2023a). We add sim- ple prompt in the Indonesian language Ini adalah soal [subject] untuk [level]. Pilihlah salah satu jawaban yang dianggap benar! "This is a [subject] question for [level]. Please choose the correct an- swer!" prior to the question and multiple-choice options.
Overall, we found that GPT-3.5 attains the high- est overall accuracy, albeit low at 53.2%. GPT- 3.5 is also notably the highest in each subject area, except in local languages and culture sub- jects. Among the open-source models, we observe that mT00xi (13B) achieves an average accuracy of 42.5%. Falcon (40B) performs worse than mTO,xi (13B) and BLOOMZ (7B).
Performance based on model size varies, with smaller models such as BLOOMZ (7B) and mT0 being better than Falcon (40B) and LLaMA (65B). We suspect that this is due to the absence of the Indonesian language in Falcon and LLaMA'S pre- training data. The poor performance of the 13B and 30B LLaMA models might imply that any "emer- gent abilities" of LLMs generally appear in the same or closely-related languages. This is fur- ther supported by Bactrian---LLaMe (13B), a LLaMA model fine-tuned on instruction datasets in 52 languages (including Indonesian), which obtain a +5% average increment, compared to LLaMA
For closed source models, we evaluate questions by comparing the first generated tokens (e.g., A, B. C) and the answer key using a regular expres- 7 7 For open-sourced models, we benchmark two strategies. Given a question and the corre- sponding multiple-choice options, we calculate: (1)
"At the time this research was carried out, we did not have access to the GPT-4 API, and thus leave it future work. 'In cases where there is no match, we assign random answer.

---

Social Science
Indonesian Language
Local languages and Caltures
Model (#parameters)
STEM
Humanities
Average
Random
21.9
23.4
23.5
24.4
26.6
24.4
GPT-3.5 (175B)
54.3
62.5
64.0
62.2
39.3
53.2
XGLM (564M)
22.1
23.0
25.6
25.6
27.5
25.2
XGLM (1.7B)
20.9
23.0
24.6
24.8
26.6
24.4
XGLM (2.9B)
22.9
23.2
27.2
25.2
XGLM (4.5B)
21.8
23.1
25.6
25.8
27.1
25.0
XGLM (7.5B)
23.6
24.5
27.5
24.5
Falcon (7B)
22.1
22,9
25.5
25.7
27.5
25.1
Falcon (40B)
30.2
34.8
34.8
34.9
32.1
BLOOMZ (560M)
22.9
23.6
23.2
24.2
25.1
24.0
BLOOMM (I.1B)
20.4
21.4
21.1
23.5
24.7
22.4
BLOOMZ (1.78)
31.5
39.3
38.3
42.8
29.4
34.4
BLOOMZ (3B)
33.5
44.5
39.7
46.7
29.8
36.4
BLOOMZ (7.18)
37.1
46.7
28.2
38.0
mTmmmal (300M)
21.8
21.4
25.7
25.1
27.6
24.9
mT0 (580M)
22.6
22.6
25.7
25.6
26.9
25.0
mT0 (1.2B)
22.0
23.4
25.1
27.3
27.6
25.2
mTO, (3.7B)
31.4
41.0
35.7
38.2
mT0 (13B)
33,5
46.2
47.9
52.6
39,6
42.5
LLaMA (7B)
22.8
25.1
26.7
25.3
LLaMA (13B)
24.1
23.0
24.4
29.5
26.7
25.3
LLaMA (30B)
25.4
23.5
25.9
28.4
28.7
26.5
LLaMA (65B)
33.0
37.7
40.8
41.4
32.1
35.8
Bactrian- Bactrian-X-LLaMa X-LLaMA (7B)
23.3
24.0
26.0
26.1
27.5
25.7
Bactrian X- X-LLaMA (13B)
28.3
29.9
32.8
35.2
29.2
30.3
Table 3: Zero-shot performance (% accuracy) of LLMs, combined across education levels. "Average" means the average across all subject areas IndaMMLU.
(13B).
90
GPT-3.5 (1758)
mT0,w (13B)
Results across education levels As illustrated in Figure 1, IndoMMLU includes detailed education level metadata, which enables us to gain a deeper understanding of the capabilities of LLMss in terms human education levels. In the Indonesian con- text, the minimum passing score for exams varies across subjects and typically ranges between 65 and 70. By setting the passing score at 65, assess GPT-3.5
80
BLOOMZ (7B)
70
Accuracy
60
50
40
30
real-world knowledge capabil- ities, shown in Table 4. Green indicates that the model has successfully passed the subject, while red indicates it has failed. This reveals that GPT-3.5 generally performs well on primary school exams for general subjects, but exhibits a lack of under- standing of local languages and culture. In sub- jects that require less analytical thinking, such as civics and religion, GPT-3.5 tends to achieve higher scores in high school exams.
13
Education level
Figure Fine-grained accuracy (%) of GPT-3.5, mT0,s.. and BLOOMZ in the Indonesian language sub- ject area. The horizontal depicts the passing score of 65, and the education level of refers the univer- sity entrance exam.
and education levels, allowing to assess the In- donesian language proficiency of LLMs. Figure 5 illustrates that GPT-3.5 achieves its highest accu- racy in grade approaching 90%. However, as the education level increases, the model's performance gradually declines. For grades 3 and above, the
Indonesian language proficiency of LLMMs As discussed in Section 3, IndoMMLU specifically in- cludes Indonesian language exams for all grades
"This refers to Curriculum 2013 Indonesia.

---

Subject
SD
SMP
SMA
UE
mT0..
Dayak Ngaju
GPT-3.5
Science
76.3
67.8
52.8
43.7
Minangkabau
Social science
84.6
73.1
63.5
48.2
Madurese
Indonesian language
74.7
61.8
55.1
42.3
Civics
64.6
65.2
65.4
Lampungic
Sports
66.7
44.7
62.0
Banjarese
Art
73.9
71.2
58.7
Islam religion
78.6
59.9
67.7
Balinese
Christian religion
83.7
77.6
62.0
Hindu religion
62.0
55.1
Javanese
Sundanese
50.0
45.1
37.9
Sundanese
Javenese
46.1
36.1
36.1
Balinese
32.2
38.5
36.1
Makassarese
33.7
48.8
38.3
Figure 6: 7ero-shot performance of mT0 and GPT-
Banjarese
50.0
44.4
28.6
3.5 in local languages and cultures subjects.
Lampungic
40.0
30.0
33.3
Madurese
41.0
28.3
35.0
50
Minangkabau culture
38.0
52.2
Falcon (40B)
BLOOMZ (7B)
Dayak Ngaju
31.1
mT0 (13B)
LLamA (65B)
45
Table 4: GPT-3.5 performance (% accuracy) across dif- ferent education levels. "SD", "SMP", "SMA", "UE"
Accuracy
40
indicate primary school, junior high school, senior high school, and university entrance tests, respectively. Red indicates that the score is below the minimum passing threshold of 65, while green signifies score above this minimum.
35
30
0-shot
1-shot
2-shot
3-shot
scores fall below 75, and for classes and above, GPT-3.5 fails to pass the exams. We observe that this trend is similar for mT0xx1 and BLOOMZ, which only pass grades 2, and 3. This fine- grained evaluation provides a valuable benchmark for LLM proficiency in Indonesian.
Figure Few-shot performance (% accuracy) of mT0 BLOOMZ, Falcon, and LLaMA, averaged across all subject areas.
ported to improve model performance (Karimi Ma- habadi et al., 2022; Hendrycks et al., 2021). We run similar experiments with our top-4 best open- source models and observe mixed outcomes in Fig- ure 7.9 Few-shot inference does not yield improve- ments in instruction-tuned models like mTO and BLOOMZ, evidenced by decrease in accuracy. In contrast, the pure LLMs Faleon and LLaMA show better performance with few-shot inference compared to zero-shot. These findings align with those of Liu et al. (2023b); Li et al. (2023b), where few-shot prompts may lead to unnatural inferences for instruction-tuned models.
LLM performance on local languages and cul- tures is interesting to observe in Table that de- spite having only 13B parameters, mTTxsl achieves the highest accuracy on local languages and cul- tures. On the other hand, GPT-3.5 with 175B pa- rameters achieves competitive accuracy, just 0.3 absolute points lower than m1 To further in- vestigate this, Figure displays the accuracy scores of each local language and culture subject, reveal- ing that both mTOxa and GPT-3.5 excel in different subject areas. mTO shows greater familiarity with Javanese and Sundanese, with a disparity of +10 for both subjects compared to GPT-3.5. GPT- 3.5 performs better in Dayak Ngaju, Banjarese, and Minangkabau culture.
Model confidence Given the top three models Table 3, we assess whether their confidence predic- tions (i.e. the predicted likelihood of the predicted answer being correct) corresponds to the actual accuracy across 64 tasks. This uncertainty cali- bration gives us hints about the model's reliability
4.3 Analysis
Few shot performance Providing several ques- tions and the answer key in prompts has been re-
Refer to the Appendix for details of the prompts.

---

(13,8
mT30g (126) cor- 0.04
BLOOME (7B) 0.03
80
BLOOMZ (7B)
WE + +W
x
(1758)
0.8
60
Accuracy
Confidence
0.6
0.7
20
Confidence (%)
40
60
80
0.0
Question length (characters)
500
1000
1500
2000
Question longth (charactors)
500
1000
1500
2000
Figure 8: Zero-shot calibration of mT BLOOMZ. and GPT-3.5 across 64 tasks. The average standard deviations of the confidence scores across all data points are 36.5, 26.4, and 43.9, respectively
Figure 9: Correlation between question difficulty and question length.
Model
W/ negation
W/o negation
Indonesian language GPT-3.5 (175B)
and how to use them appropriately in real-world settings. For mTO and BLOOMZ, the confidence score is determined through softmax normalization over probabilities of the multiple-choice options. For GPT-3.5, we adopt the approach described by Si et al. (2022); Wang et al. (2022), using a high- temperature value (0.7) during decoding. For each question, we generate different outputs and mea- sure self-consistency. The probability of multiple- choice option is calculated based on the output fre- quency. In this experiment, we use n = 7, and choose the most frequentl--ccurring answer as the final prediction.
58.0
62.7
m'T0 (13B)
47.9
53.1
BLOOMZ (7B)
39.3
50.1
Social science GPT-3.5 (175B)
66.2
63.0
m'TO (13B)
48.2
47.1
BLOOMZ (7B)
43.3
48.2
Table 5: Accuracy (%) for questions with and without negation in the Indonesian language and social science subject areas.
We average the confidence scores across the 64 tasks, and display the calibration of mTO, BLOOMZ, and GPT-3.5 in Figure 8. We observe that all three models are well-calibrated, with cor- relation scores of 0.85.
within each subject area. 10 We then break down the accuracy for the top three models (GPT-3.5, mTOO, and BLOOMZ)) based on the presence absence of negation. Among the subject areas, Indonesian lan- guage and social science are the most prevalent in employing negation, accounting for approximately 10% in each group. Through manual observation of 100 random samples, we verified that 85% of these questions indeed contained negation.
Additionally, we examine the relationship be- tween confidence scores and question length, as depicted in Figure 9. We found very weak cor- worth
relation for both m'TO and BLOOMZ. noting that the confidence score can also be inter- preted as a measure of question difficulty, based on which question length appears to have no bearing on difficulty.
Table 5 shows the effects of negation IndoMMLU accuracy. For the Indonesian language subject area, negated questions prove be more challenging, with a decrease accuracy ranging from -4 to -10. In social science, mTO and BLOOMZ are similarly more accurate over ques- tions without negation. Compared to mTO, how- ever, BLOOMZ is less robust to negation, as indi- cated by the -5 accuracy drop.
Impact of negation In Indonesian school exam questions, the use of negation is common to en- hance question difficulty and assess students' rea- soning abilities. Similarly, in the field of NLP, nega- tion is known increase the difficulty of NLP tasks (Truong et al., 2022). To investigate the impact negation, we employ a simple string-matchin matching strategy to identify questions that contain negations
UTo identify negation, we use strings kecuali "except", yang bukan "which is not", and yang tidak "which is not"

---

5 Discussion
questions as they are already well covered by ex- isting English math reasoning benchmarks. We suggest that essay questions enable deeper assess- ment of comprehension and critical thinking, but that methods for evaluating essay quality across education levels in languages other than English are severely lacking.
If LLMs are to be deployed in diverse contexts, it is critical to have more work on evaluation for different languages and cultures. In Table we ob- served that the models struggle to answer questions that pertain to local languages and cultures across all levels of education in Indonesia. Minangkabau culture in particular is taught and assessed in the In- donesian language, and yet the limited performance in answering questions relating to it underscores a lack of cultural knowledge, despite reasonable results for the Indonesian language.
Ethical Considerations
The IndoMMLU dataset used in our study is collected from publicly- web resources. In compli- ance with the Indonesian Copyright Law number 28 year 2014, specifically article 44, the use, trieval, reproduction, and/or modification of works and/or related rights products, in whole or in sub- stantial part, not considered a copyright infringe- ment the source is fully cited or mentioned for educational and research purposes." |I
We also argue that education science should play a more central role in the future evaluation of LLMs. Current NLP work has mostly focused on developing larger models with different tech- niques and architectures, and evaluation has pri- marily been in terms of specific NLP tasks. Educa- tion science has decades of experience in designing assessments to evaluate student progress through painstakingly-designedd comprehensive tests, which the NLP community should better engage with. With IndoMMLU, we have shown that exam ques- tions across fine-grained educational levels offer a more profound comprehension of model profi- ciency in the Indonesian language, while also re- vealing potential areas for improvement.
Regarding our experimental results, it is impor- tant to note that they do not provide definitive answer as to the relative abilities of LLMs, and we caution readers against overinterpreting the find- ings. While we conclude that GPT-3.5 demon- strates proficiency in passing primary school exams in Indonesia based on IndoMMLU, U, it is essential to consider potential contamination in GPT-3.5's pre- training data, which could impact the results. Fur- thermore, it is worth noting that real-world student assessments encompass not only multiple-choice questions but also practical exams, laboratory work. and essay writing.
6 Conclusion
In this paper, we presented IndoMMLU, multi-task language understanding benchmark for real-world evaluation of knowledge in the Indonesian con- text. By leveraging education level metadata, we found that current LLMs like GPT-3.5 are only able to pass primary school exams in Indonesia, while smaller models struggle across nearly in all education levels. Notably, none of the 24 evalu- ated models perform well in the domain of local languages and cultures, highlighting the need for further research in this direction.
References
Razieh Baradaran, Razieh Ghiasi, and Hossein Amirkhani. 2022. A survey machine reading com- prehension systems. Natural Language Engineering, 28(6):683-32.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et 2020. PIQA: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pages 7432-7439.
Limitations
Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Indra Winata, Bryan Wilie, Fajri Koto, Rah- mad Mahendra, Christian Wibisono, Ade Romad- hony, Karissa Vincentio, Jennifer Santoso, David Moeljadi, Cahya Wirawan, Frederikus Hudi, Muham- mad Satrio Wicaksono. Ivan Halim Parmonangan, Ika Alfina, Ilaa Firdausi Putra, Samsul Rahmadani, Yulianti Oenang, Ali Akbar Septiandri, James Jaya,
Despite being the largest question-answering dataset in the Indonesian context, IndoMMLU still has some limitations, in that it lacks: (1) multi- modal questions; (2) arithmetic reasoning tasks; and (3) essay-style questions. First, IndoMMLU is comprised solely of text-based questions, and ques- tions with tables and figures are discarded to sim- plify data collection. We specifically exclude math
res.eex--eeeeeee res. rww. http///doos//exxxxxxeeee waww/en/id/aeee http://dddddd4eee.ddcm

---

Kaustubh Dhole, Arie Suryani, Rifki Afina Putri, Dan Su, Keith David Stevens, Made Nindyatama Nityasya, Muhammad Farid Adilazuarda, Ryan Ig- natius Hadiwijaya, Ryandito Diandaru, Tiezheng Yu, Vito Ghifari, Wenliang Dai, Yan Xu, Dyah Inastra Damapuspita, Haryo Akharianto Wihowo, Cuk Tho, Ichwanul Muslim Karo Karo, Tirana Noor Fatyanosa, Ziwei Ji, Graham Neubig, Timothy Baldwin, Sebas- tian Ruder, Pascale Fung. Herry Sujaini, Sakriani Sakti, and Ayu Purwarianti. 2023. NusaCrowd: Open source initiative for Indonesian NLP resources. In Findings of the Association for Computational Linguistics: ACL. 2023.
the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khy- athi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Eisin Durmus, Ondiej Dušck, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jham- tani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bod- hisattwa Prasad Majumder. Pedro Henrique Martins, Angelina McMillan-Man Major. Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachin, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimo- rina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Longuage Generation, Evaluation, and Metrics (GEM 2021), pages 96-120. Online. Association for Computational Linguistics.
Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio. Xiaohong Li, Adhiguma Kun- coro, Sebastian Ruder, Zhi Yuan Lim, Syafri Ba- har, Masayu Khodra, Ayu Purwarianti, and Pascale Fung. 2021. IndoNLG: Benchmark and resources for evaluating Indonesian natural language generation. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Ilias Chalkidis. 2023. ChatGPT may pass the bar exam soon, but has a long way go for the LexGLUE benchmark. arXiv preprint arXiv:2304. 12202.
Jonathan H Choi, Kristin E Hickman, Amy Monahan, and Daniel Schwarez. 2023. ChatGPT goes to law school. Available at SSRN.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki, 2020. TyDi QA: benchmark for information-sseking question answering in typo- logically diverse languages. Transactions of the As- sociation for Computational Linguistics, 8:454-470.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In International Conference on Learning Representations.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018 Think you have solved question an- swering? Try ARC, the A12 reasoning challenge. arXiv preprint urXiv:1803.0544
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual general- ization. In Proceedings of ICML 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. 2021. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14168.
Lifu Huang. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA Machine reading comprehension with contextual commonsense rea- soning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 2391-2401, Hong Kong, China. Association for Com- putational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- une I: Long Papers), pages 1601-1611, Vancouver. Canada. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of

---

Rabech Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, and Majid Yazdani. 2022. Prompt- free and efficient few-shot learning with language models. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 3638-3652, Dublin, Ireland. Association for Computational Linguistics.
Conference on Asian Language Processing (IALP), pages 391-394. IEEE
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wai Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452 466.
Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2023. GPTT-4 passes the bar exam. Availuble at SSRN 4389233.
Fajri Koto. Timothy Baldwin, and Jey Han Lau. 2022a.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy, 2017. RACE: Large-scale ReAd- ing comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785- 794, Copenhagen, Denmark. Association for Compu- tational Linguistics.
Cloze evaluation for deeper understanding of com- monsense stories Indonesian. In Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022), pages 8-16, Dublin, Ireland. Association for Computational Linguistics.
Fajri Koto, Timothy Baldwin, and Jey Han Lau. 2022b. LipKey: A large-scale news dataset for absent keyphrases generation and abstractive summariza- tion. Proceedings of the 29th International Con- ference on Computational Linguistics, pages 3427- 3437, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Haonan Li, Fajri Koto. Minghao Wu, Alham Fikri
Aji, and Timothy Baldwin. 2023a. Bactrian- X: A multilingual replicable instruction-followe model with low-rank adaptation. arXiv preprint arXiv:2305. 250.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Bald- win. 2023b. CMMLU: Measuring massive multitask language understanding in chinese. arXiv preprint 2922222006.02222.
Fajri Koto and Ikhwan Koto. 2020. Towards computa- tional linguisties in Minangkabau language: Studies on sentiment analysis and machine translation, In Proceedings of the 34th Pacific Asia Conference on Language, Information and Computation, pages 138- 148, Hanoi, Vietnam. Association for Computational Linguistics.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi. Ming Gong, Linjun Shou, Daxin Jiang, Guibong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new benchmark dataset for cross-lingual pre-training, un- derstanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008-601 6018, Online. Association for Computational Linguistics.
Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2020a. Liputan6: A large-scale Indonesian dataset for text summarization. In Proceedings of the 1st Confer- ence of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th Interna- tional Joint Conference on Natural Language Pro- cessing, pages 998-608, Suzhou, China. Association for Computational Linguistics.
Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. IndoBERTweet:: A pretrained language model for Indonesian Twitter with effective domain-specific vo- cabulary initialization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 10660-10668. Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal. Shruti Bhosale, Jingfci Du et al. 2021. Few-shot learning with multilingual language models. arXiv preprint arxi 12. 10668.
Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2023a. Are multilingual Ilms culturally- diverse reasoners? an investigation into multicultural proverbs and sayings. arXiv preprint orXiv22309.0859)
Fajri Koto, Afshin Rahimi, Jey Han Lau, and Timo- thy Baldwin. 2020b. IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 757-770, Barcelona, Spain (Online). Interna- tional Committee on Computational Linguistics.
Chuang Liu, Renren Jin, Yugi Ren, Linhao Yu, Tianyu Dong, Xiaohan Peng. Shuting Zhang, Jianxiang Peng. Peiyi Zhang, Qingging Lyu, et al. 2023b. M3KE: A massive multi-level multi-subject knowledge evalu- ation benchmark for chinese large language models. arXiv preprint ar0iv:2305. 10263.
Fajri Koto and Gemala Y Rahmaningtyas. 2017. Inset lexicon: Evaluation of word list for Indonesian sen- timent analysis in microblogs. In 2017 International

---

Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan, 2022. NumGLUE: A suite of funda- mental yet challenging mathematical reasoning tasks. In Proceedings of the 60th Annual Meeting of the Association for Cumputational Linguistics (Volume
Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Margaret Ryznar. 2023. Exams in the time of ChatGPT. Washington and Lee Law Review Online, 80(5):305.
1: Long Papers), pages 3505-3523, Dublin, Ireland. Association for Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: An adver- sarial Winograd schema challenge at scale. Commu- nications of the ACM, 64(9)999-106.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf, et al. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint arXiv:221 1..07866
Mei Silviana Saputri, Rahmad Mahendra, and Mirna Adriani. 2018. Emotion classification Indonesian Twitter dataset. In 2018 International Conference on Asian Language Processing (IALP), pages 90-95, IEEE.
Joseph Novak. 1988. Learning science and the science of learning. Studies in Science Education, 15(1):77- 101.
Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. 2023, Jais and jais-chat: Arabic-centric foundation and instruction-tunce open generative large language models. arXiv preprint arXiv:2308. 16149.
OpenAI. 2023. GPT-4 technical report. ArXiv, abs/2303.08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Neural Information P Processing Systems, 35:27730-2778
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang. Jianfeng Wang. Jordan Boyd-Graber, and Li- juan Wang. 2022. Prompting GPT-3 be reliable. arXiv preprint arXiv:2210.09150.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon L.LM: Outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:230.00116.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Pupers), pages 4149-4155 4158. Minneapolis, Minnesota. Association for Computational Linguistics.
Ayu Purwarianti and Ida Ayu Putu Ari Crisdayanti. 2019. Improving bi-LSTM performance for Indonesian sen- timent analysis using paragraph vector. In 2019 Inter- national Conference of Advanced Informatics: Con- cepts, Theory and Applications (ICAICTA), pages 1-5. IEEF.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and ef- ficient foundation language models. arXiv preprint ar 2rXiv:2302.13971
Ayu Purwarianti, Masatoshi Tsuchiya, and Seicchi Nak- agawa. 2007. A machine learning approach for In- donesian question answering system. In Artificial Intelligence and Applications, pages 573-578.
Thinh Truong, Timothy Baldwin, Trevor Cohn, and Karin Verspoor. 2022. Improving negation detection with negation-
pre-training. In Proceedings of the 2022 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, pages 4188-4193, Seattle, United States. Association for Computational Linguistics.
Rifki Afina Putri and Alice Oh. 2022. -RKM Unan- swerable questions for Indonesian machine reading comprehension. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6918 6933, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stick- ier benchmark for general- purpose language under- standing systems. Advances in neural information processing systems, 32.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Sid- dhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu. Dan Garrette, Graham Neubig, and Melvin John- son. 2021. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10215-10245,
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE:

---

A multi-ttsse benchmark and analysis platform for ural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNI.P, Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.1177:
Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020, IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. In Proceedings of the 1st Conference of the Asia-Pacijp Chapter of the Association for Computational Linguistics and the 10th International Joint Conference Natural Lan- guage Processing, pages 843-857, Suzhou, China. Association for Computational Linguistics.
Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawi- jaya, Rahmad Mahendra, Fajri Koto, Ade Romad- hony, Kemal Kurniawan, David Moeljadi, Radi- tyo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, and Sebastian Ruder. 2023. NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 815-834, Dubrovnik, Croatia. Association for Com- putational Linguisties.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of of -ft---f---hheeee art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, Association for Computational Linguistics.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can chine really finish your sentence? Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4791 7791-4800, Florence, Italy, Association for Computational Linguistics.

---

A Data Statistics
Category
#question
Table 6, Table 7, and Table 8 provide detailed statis- tics of the question distribution in IndoMMLU.
STEM
2995
Social science
2772
Humanities
2301
Subjects
SD
SMP
SMA
UE
Total
Indonesian language
3213
Science
488
680
1168
Local languages and cultures
3700
Physics
297
297
Chemissry
287
398
685
Total
14981
Biology
457
388
845
Social science
300
299
599
Table 8: Total number of questions based on subject areas.
Goography
196
Sociology
Economics
296
192
History
300
198
498
B
Few-shot Prompt
Civics
99
300
300
699
Indonesian language
1125
850
857
381
3213
Balinese
200
123
148
471
Makassarese
98
41
47
186
Ini adalah beberapa contoh soal [SUBJECT]
These are several examples of [SUBJECT] question.
Banjurese
10
144
Lampungic
30
[Еxample-1]
[Example-1]
Madurese
93
102
295
Jawabar: [Answer-1] 1]
Answer: [Answer-1]
Sundanese
718
294
145
1157
Javanese
396
298
298
992
[Example-2]
[Exampie-2]
Jawaban: [Answer-2]
Answer: [Answer-2]
Dayak Ngaju
109
109
Minangkabau culture
153
46
199
[Example-3]
[Example-3]
Art
200
200
201
601
Jawaban: [Answer-3]
Answer: [Anewer-3]
Sports
49
49
148
[QUESTION]
[QUESTION]
Islam religion
201
300
703
Jawaban:
Answer:
Christian religion
50
49
102
Hindu religion
49
52
Total
4548
3616
4765
2052
14981
Figure 10: Illustration of our few-shot prompt template. The English tramslation on the right is solely for illus- trative purposes. our experiments, we used up three examples within the prompt. The placeholders [SUBJECT]] Example-i, Answer-i. and QUESTION cor- respond the subject, the i-th question example, the answer key for the i-th question example, and the main question, respectively.
Table 6: Total number of questions for each subject area and education level. "SD", "SMP". "SMA", "UE" indicate primary school, junior high school, senior high school, and university entrance tests, respectively.
Class #questions
200
150
195
4
187
197
7
282
291
277
10
295
11
288
12
274
12+
381
3213
Table 7: Total number of questions in the Indonesian language subject, including those designated for univer- sity entrance tests (12+).

---

C Zero-shot Performance Based the Probability of the Full Generated Answer
Social Science
Indonesian Language
Local languages and Cultures
Model (#parameters)
STEM
Humanities
Average
Random
21.9
23.4
23.5
24.4
26.6
24.4
XGLM (564M)
24.2
23.9
27.2
29.0"
27.8
26.3
XGLM (1.78)
23.7
25.4
27.1
28.4
28.9
26.9
XGL M (2,98)
23.6
25.4
28.3
28.8
28.8
26.9
XGLM (4.58)
23.9
25.5
29.4
27.9
28.1
27.2
XGLM (7,5B)
23.5
26.0
28.6
27.6
Falcon (7B3
22.2
25.8
28.4
50.1
27.9
26.8"
Falcon (40B)
28.4
32,9
BLOOMZ (560M)
23.0
24.4
23.7
27.2
24.9
BLOOMZ (1.1B)
22.9
25.8
26.6
28.3
26.2
BLOOMZ (1.7B)
23.7
29.8
29.7
32.8
28.1
28.3
BLOOMZ (3B)
27.6
32.5
32.6
35.0
27.4
30,0
BI.OOMZ (7.18)
26.8
32,9
33.5
36.5
28.1
30.5
_ (50DM)
24.0
26.1
27.0
29.8
30.8
27.8
mi (580M)
23.9
25.5
27.6
30.1
30.5
27.7
uT00 (1.28)
25.1
27.5
27.9
33.6
29.6
28.2
mT00 (3.7B)
28.5
36.1
35.3
40.7
34.3
34.2
mT (13B)
30.1
38.1
40.9
43.2
36.4
LLamA (7B)
23.7
25,6
28.0
29.0
27.0
(13B)
25.4
24.3
26.4
29.5
L. (65B)
26.7
29.3
32.4
32.9
29.7
Bactrian X X-LLamA (7B)
23.8
25.4
28.7
29.8
28.0
27.0
Bactrian-X-LLan (13B)
25.6
27.4
29.2
30.7
27.9
27.8
Table 9: Zero-shot performance (% accuracy) of large language models based on the probability of the full generated answer, aggregated across education levels. "Average" means the average across subject areas IndoMMLU.
D Model Artifacts
Models (#parameters)
Source
XGLM (564M)
facebook/xgllm-544M
(1.7B)
facebook/xglnm-1.7
XGLM (2.9B)
faccebook/xxllm22.999
XGLM (4.5B)
facebook/xglIn-4.5B
XGL.M (7.5B)
facebook/xxll--7.56
Falcon (7B)
tiiuae/falcon-76
Falcon (40B)
tiluse/falcon-400
BLOOMZ
bigscience/bloonz-1b1
BLOOMZ (1.7B)
bigsclenceebloowzz11u
BI BI.OOMZ
bi rigsci secciiencc/bblommzoi-a
(7.1012
bigsc bigscience/bloowzz
inT (300M)
bi eigiciennii/ine soncee/ntt-snni
mTT (580M)
digscience/nte-bba bi bigscience/mtt--111
mTTugge (1.2B)
mT0, (3.7B)
bigsciancee/nt@-xl
mT00g (13B)
bigarience/ntt-
LLamA (7B)
LLamA (13B)
decapod-resserrc//1lmee1
LLamA (30B)
decaos@reseerdd/11mm-----
1.LamA (65B)
huggyllama/11mma--e 6mm-6bb
Bactrian-X-11 amm. (7B)
Bactrian-X-Le X-LLamA (13B)
#BzUtT/bactriannx-1lea
Table 10: With the exception of GPT-3.5 (Ouyang et al., 2022), all the models used in this study sourced from Huggingface (Wolf et al., 2020).

---

E Full Results in Each Subject and Education Level in GPT-3.5, m'TO, and BLOOMZ
m20,0
3.5 BLOOMZ
History
Economy (UE)
Sociology (UE)
Geography (UE)
Blology (UE)
Chemistry (UE)
Indonesian language (UE)
Hindu religion (SMA)
Isiam religion (SMA)
Christian religion (SMA)
Javanese
Sundanese (5MA)
Lampungic (5MA)
Banjarese
Makastarese (5MA)
Balinese (5MA)
Art (SMA)
Sport (5MA)
Indonesian language (5MA)
Civic education (SMA)
History
Econamy (SMA)
Saciology
Geography (5MA)
Biology
Chemistry (SMA)
Physiss
Hindu relgion [SMP)
Islam religion (SMP)
Christian relgion (SMP)
Minangkabau culture (SMP)
Javanese (SMP)
Sundanete
Madurese (SMP)
Lampungi:
Banjarese (SMP)
Makcassarese (SMP)
Balinese (SMP)
(SMP)
Sport (SMP)
Indonesian language (SMP)
Civic education (SMP)
Social science (SMP)
Science (SMP)
Hindu religion (5D)
Islam religion (5D)
Christian religion
Minangkabau culture (50)
Dayak anguage (5D)
Javanese (5C)
Sundanese (5D)
Madurese (50)
Lampungic (5D)
Banjarese (SD)
Makassarese (SD)
Balinese (5D)
Art (SD)
Sport (5D)
Indonesian anguage (50)
education
Social sc science (5D)
(SD)
Figure Performance (% accuracy) breakdown across the 64 tasks. "SD", "SMP", "SMA", "UE" indicate primary school, junior high school, senior high school, and university entrance tests, respectively. The red vertical line denotes random performance.


Reviews:
[{"review_id": "djgwfwh1bi", "reviewer": "Reviewer_Qs8E", "summary": "", "questions": "Question A: (Section Discussion) Why should education science play a more central role? The experiments and results of this paper do not provide persuasive proof for this argument. To strengthen the standpoint of evaluating models on a learning curve, the author(s) should explain why the knowledge of LLMs in grade 12 should be disregarded if they fail in grade 4. For example, if an LLM fails the test about basic fractional operations, we can reasonably disregard the performance of this LLM on calculus. However, this approach does not appear to be reasonably applied to subjects such as Social Science or Humanities  where the progression of knowledge is not consistently cumulative. \n\nThis direction may be linked to the idea in procedural understanding of scientific facts in paper Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension (Dalvi et al., NAACL 2018). Besides, the discussion of Creating Dependency Between Questions in paper Benchmarking Machine Reading Comprehension: A Psychological (Sugawara et al., EACL 2021) may also offer some insights.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback. \n\n> The benefits of evaluating LLMs on a learning curve (from 1 to 12) still need to be further explored\n\nA: Our focus is not on the “learning curve” and we do not make any strong claims about the monotonicity of results across grade levels (as in, a score of x at a lower grade level must mean a score of <= x for a higher grade level), but on (1) understanding how well a given large language model embeds knowledge across different subjects and pegged against different education levels, to get more motivated and fine-grained insights into model performance; and (2) creating a high-quality dataset that is multi-task (with good coverage across different tasks), crafted by education experts, and covers the Indonesian language in addition to 10 local languages. The Lampungic and Makassarese questions in IndoMMLU are the very first NLP resources to be released for these languages.\n\n> Why should education science play a more central role? \n\nA: As stated in L.479-483, we draw on education science to evaluate knowledge embedded in large language models. School assessments (exam questions) are not designed in an ad hoc way by crowd workers or drawn from arbitrary web sources, but are expertly based on a curriculum in education science, to evaluate the degree of mastery of the curriculum content by a student of a given grade level. In terms of knowledge acquisition by large language models, this provides fine-grained insights into model limitations that can be used to drive further improvements.\n\n> Paper references from NAACL 2018 and EACL 2021.\n\nA: These papers introduce reading comprehension tasks, which are not directly related to the primary focus of our work in evaluating the knowledge of LLMs in an Indonesian context, but we can certainly add them to the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tIndoMMLU is a valuable resource for evaluating the reasoning abilities in Indonesian languages and real-world knowledge about Indonesia. \n\n2.\tThe comprehensive analysis in Section 4 uncovers many significant insights about different LLMs' abilities and real-world knowledge.", "reasons_to_reject": "1.\tThe benefits of evaluating LLMs on a learning curve (from 1 to 12) still need to be further explored in the current version of the paper. \n\nRefer to the Question A.", "questions_to_authors": "Question A: (Section Discussion) Why should education science play a more central role? The experiments and results of this paper do not provide persuasive proof for this argument. To strengthen the standpoint of evaluating models on a learning curve, the author(s) should explain why the knowledge of LLMs in grade 12 should be disregarded if they fail in grade 4. For example, if an LLM fails the test about basic fractional operations, we can reasonably disregard the performance of this LLM on calculus. However, this approach does not appear to be reasonably applied to subjects such as Social Science or Humanities  where the progression of knowledge is not consistently cumulative. \n\nThis direction may be linked to the idea in procedural understanding of scientific facts in paper Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension (Dalvi et al., NAACL 2018). Besides, the discussion of Creating Dependency Between Questions in paper Benchmarking Machine Reading Comprehension: A Psychological (Sugawara et al., EACL 2021) may also offer some insights.", "paper_topic_and_main_contributions": "The paper introduces IndoMMLU, a multi-task language understanding for Indonesian. This dataset includes 14,906 exam questions from grades 1 to 12 and the university entrance exam. The questions within IndoMMLU encompass various subjects taught in Indonesian educational institutions, including STEM, Social Science, Humanities, local languages and cultures, and the Indonesian language. The construction process of IndoMMLU is meticulously documented, ensuring transparency and reliability. Subsequently, this paper also comprehensively analyzes the performance of 24 large language models (LLMs) on IndoMMLU.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rCm8xQvY2l", "reviewer": "Reviewer_AUi2", "summary": "", "questions": "Some of the questions include questions about Indonesian culture and geography. I think it is of important interest to the reader whether the inclusion of questions about Indonesian culture and geography is the cause of the poor performance of the LLM or whether the problem is simply that there are too few Indonesian language resources. I believe that asking questions about Indonesian culture and geography in English would yield suggestive results in this regard. What would be the contribution of this study for this point?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and constructive feedback.\n\n> There is no evaluation using GPT-4\n\nA: We agree that it would be interesting to add GPT-4 evaluation. However, we do not have access to GPT-4 prior to submission. Furthermore, we argue that evaluating a closed-source model such as GPT-4 provides less benefit to the research community, in part because of the difficulty in reproducing the results because of behind-the-scenes model updates. Nonetheless, we can certainly add GPT-4 to the camera-ready paper, as we now have access to the model.\n\n> I believe that asking questions about Indonesian culture and geography in English would yield suggestive results in this regard. What would be the contribution of this study for this point?\n\nA: We agree that it will be a valuable discussion. However, the questions for Indonesian culture subjects are in the local languages, which are extremely challenging to translate into English because they are extremely low resource. Moreover, our focus in this paper is to evaluate knowledge embedded in Indonesian models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The attempt to investigate the performance of large-scale language models in Indonesian language. It can alleviate the English-centricity of natural language processing.", "reasons_to_reject": "There is no evaluation using GPT-4, which seems to perform the best, even though the main result is the claim \"Large Language Models Only Pass Primary School Exams in Indonesia,\" which suggests performance limitations.\n\"We do not have access to the GPT-4 API, thus leave it for future work. However, even if we do not have access to the API, it should be possible to ask the GPT-4 about 200 questions and collect the answers via a form. If the main claim is the limitation of performance, an evaluation using the method that is considered to have the best performance should be added, even if it is only a small-scale experiment.", "questions_to_authors": "Some of the questions include questions about Indonesian culture and geography. I think it is of important interest to the reader whether the inclusion of questions about Indonesian culture and geography is the cause of the poor performance of the LLM or whether the problem is simply that there are too few Indonesian language resources. I believe that asking questions about Indonesian culture and geography in English would yield suggestive results in this regard. What would be the contribution of this study for this point?", "paper_topic_and_main_contributions": "This study proposes a benchmark to investigate the performance of large-scale language models using school exams in Indonesian.\nIt is an excellent attempt to investigate the performance of large-scale language models in languages other than English.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZoFxTtf8fZ", "reviewer": "Reviewer_49YF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback. \n\nWe completely agree that applicability analysis across languages is interesting. One way to do it is by translating IndoMMLU to English via translation. However, 25% of IndoMMLU uses 10 Indonesian local languages. Finding good human/machine translators is extremely challenging for these local languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This study effectively developed a benchmark for LLMs in Indonesian languages by using K-12 exams. Particularly, this benchmark demonstrated that LLMs encountered difficulties in answering questions related to local languages and cultures, resulting in inconsistent performances across the board.", "reasons_to_reject": "This study could have verified the proposed benchmark's applicability across languages, such as English and Indonesian languages. This expansion would lead to a more comprehensive evaluation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduced a massive multitask language understanding benchmark for Large Language Models (LLMs) in Indonesian languages. The proposed benchmark included exam questions for K-12 school students covering multiple subjects. The primary contribution of this study lay in the creation of a benchmark for minor languages. The proposed benchmark was thoroughly examined with 24 multilingual LLMs, and the results were compared across the models and education levels. Additionally, this study analyzed the few-shot performance of LLMs on this benchmark. The findings aligned with previous studies, indicating that state-of-the-art LLMs perform poorly in Indonesian languages.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a benchmark using K-12 exams for LLMs in Indonesian languages. It demonstrated that LLMs encountered difficulties in answering questions related to local languages and cultures. The reviewers agree that the dataset, i.e. IndoMMLU, is a valuable resource for evaluating the reasoning abilities in Indonesian languages and real-world knowledge about Indonesia. The comprehensive analysis showed interesting insights about different LLMs' abilities. Another good aspect is that this research is on promoting AI democratization, i.e. drawing attention to the LLMs' limitation on low-resource languages.
Decision: Accept-Main