Venue: NAACL 2025

# Cracking the Code: Multi-domain LLM Evaluation on

Real-World Professional Exams in Indonesia

Fajri Koto

Department of Natural Language Processing

MBZUAI, Abu Dhabi, UAE

fajri.koto@mbzuai.ac.ae

###### Abstract

While knowledge evaluation in large language models has predominantly focused on academic subjects like math and physics, these assessments often fail to capture the practical demands of real-world professions. In this paper, we introduce IndoCareer, a dataset comprising 8,834 multiple-choice questions designed to evaluate performance in vocational and professional certification exams across various fields. With a focus on Indonesia, IndoCareer provides rich local contexts, spanning six key sectors: (1) healthcare, (2) insurance and finance, (3) creative and design, (4) tourism and hospitality, (5) education and training, and (6) law. Our comprehensive evaluation of 27 large language models shows that these models struggle particularly in fields with strong local contexts, such as insurance and finance. Additionally, while using the entire dataset, shuffling answer options generally maintains consistent evaluation results across models, but it introduces instability specifically in the insurance and finance sectors.1

Footnote 1: Data can be accessed at [https://huggingface.co/datasets/indolem/IndoCareer](https://huggingface.co/datasets/indolem/IndoCareer).

## 1 Introduction

The evaluation of large language models (LLMs) has shifted from traditional natural language processing (NLP) tasks Mikheev et al. (1999); Straka and Strakova (2017) to more complex, knowledge-intensive, and reasoning-based challenges. One of the key datasets used to assess these abilities is the massive multitask language understanding (MMLU) Hendrycks et al. (2021). Initially introduced in English, MMLU datasets have also been developed in other languages, including Indonesian Koto et al. (2023), Chinese Li et al. (2024), and Arabic Koto et al. (2024). These datasets consist of school exam questions across various subjects and education levels, tailored to local curricula.2 However, they primarily focus on academic subjects, often overlooking vocational and professional expertise, which are more relevant to real-world applications.

Footnote 2: The English MMLU is based on the U.S. curriculum, while the Indonesian MMLU follows the Indonesian curriculum.

Due to the recent widespread adoption of LLMs across various domains, including health Zhang et al. (2024), education Weijers et al. (2024); Srivastava and Kochmar (2024), and finance Lee and Soon (2024), evaluating a model's knowledge across professional fields has become crucial. For instance, in healthcare, the model must adhere to ethical standards Gundersen and Baroe (2022) and possess expertise in prevalent regional diseases. We should not trust AI-based health recommendations from models that have not passed a competency exam. Similarly, in education, the model needs to understand and align with local government teaching guidelines. Despite the importance of certification exams in professional fields, such exams have been largely excluded from prior work Koto et al. (2023).

Figure 1: Distribution of professions in IndoCareer.



In this paper, we introduce IndoCareer, a dataset comprising 8,834 multiple-choice questions collected from various Indonesian competency exams, certification exams, and vocational school exams. Our focus on Indonesian addresses the limitations of prior work Koto et al. (2023) and aims to enrich language diversity and local context nuances in NLP datasets, which are predominantly English-centric Liu et al. (2024). Figure 1 shows the distribution of IndoCareer, which covers 22 different professions across 6 categories: (1) healthcare, (2) insurance and finance, (3) creative and design, (4) tourism and hospitality, (5) education and training, and (6) law. Additionally, we demonstrate that IndoCareer is generally robust to option shuffling Zhou et al. (2024) when using the entire dataset, but it specifically introduces instability in insurance and finance professions.

## 2 Related Work

Indonesian Language ModelsIndoBERT Koto et al. (2020); Wilie et al. (2020), IndoBERTweet Koto et al. (2021), IndoGPT Cahyawijaya et al. (2021), and IndoBART Cahyawijaya et al. (2021) are among the earliest transformer-based language models developed from scratch for Indonesia. These models have been widely adopted by industry and academia across various applications. For models exceeding 1 billion parameters, no foundational models have been pre-trained exclusively on Indonesian text. Instead, research has focused on adapting multilingual models through fine-tuning techniques. Notable examples include Bactrian-X Li et al. (2023), which employs LoRA Hu et al. (2022) for fine-tuning LLama-1 Touvron et al. (2023a), and Merak Ichshan (2023), Cendol Cahyawijaya et al. (2024), and Komodo Owen et al. (2024), which are fine-tuned adaptations of LLama-2 Touvron et al. (2023b). Despite growing interest in deploying Indonesian LLMs across various domains and job sectors, there remains a lack of suitable benchmarks tailored to evaluate their performance. To address this gap, we introduce IndoCareer.

Benchmarks for Evaluating Language ModelsNusaCrowd Cahyawijaya et al. (2023) represents a significant effort to consolidate scattered datasets for Indonesian NLP. While most high-quality datasets focus on classical NLP tasks such as sentiment analysis, summarization, and text classification, benchmarks for knowledge-intensive and reasoning tasks have been notably limited until very recently. The introduction of IndoMMLU Koto et al. (2023), COPAL-ID Wibowe et al. (2024), and IndoCulture Koto et al. (2024b) marks a step forward in this direction. COPAL-ID and IndoCulture focus on cultural commonsense reasoning, while IndoMMLU evaluates exam questions across different education levels in Indonesia, from primary to high school.

Despite recent advancements, a significant gap remains in evaluating LLMs on professional tasks in the Indonesian context, as IndoMMLU does not include questions from professional exams. This limitation is not unique to Indonesia; professional exam coverage is also limited in similar benchmarks for other languages. For example, English MMLU Hendrycks et al. (2021) and Chinese MMLU Li et al. (2024) include professional exam questions in only 20% of their datasets, while Arabic MMLU Koto et al. (2024a) has an even lower coverage of just 4%.

As LLMs are increasingly applied across various domains Zhang et al. (2024); Lee and Soon (2024), there is a pressing need for a benchmark that evaluates their readiness for professional job sectors. IndoCareer addresses this gap, offering a comprehensive benchmark of professional exams spanning 22 professions, making it the first of its kind in Indonesia.

## 3 IndoCareer

IndoCareer comprises 8,834 multiple-choice questions compiled from Indonesian competency exams, certification exams, and vocational school exams across 22 professions. In Indonesia, competency exams are commonly required in healthcare professions by the government. Certification exams, on the other hand, focus on specific skills within a profession, such as tax accounting in finance. At the high school level, vocational schools offer specialized training in areas like tourism, culinary arts, and fashion design. In Figure 1, Table 1, and Table 2, we present detailed statistics for the 22 professions covered in IndoCareer. In this dataset, we exclude engineering-related professions, as their certification exams are generally conducted in English.

Data ConstructionWe manually collected exam questions from publicly available sources across 22 professions. A majority (78%) of the questions 

were sourced from Scribd,3 a document-sharing platform, while the remaining were obtained from local government websites4 and shared Google Drive folders. We ensured that all collected questions were relevant to their respective professions and suitable for distribution for research purposes. Importantly, 99% of the exam questions were retrieved from file formats, such as PDFs and Word documents, rather than directly from web pages, minimizing the risk of overlap with training data used by LLMs.

Footnote 3: [https://www.scribd.com/](https://www.scribd.com/)

Footnote 4: For example: [https://badanbahasa.kemdikbud.go.id](https://badanbahasa.kemdikbud.go.id)

To extract the questions and answers, we hired three professional teachers with Bachelor's degrees in Education for a one-month period. Their task focused exclusively on text-based questions, excluding any questions containing images (see Figure 2 for examples). Each worker was responsible for extracting approximately 3,000 questions. To ensure ethical practices, they were compensated above the minimum wage in Indonesia, with the total workload equivalent to five full-time workdays.

Quality ControlWe ensure the high quality of our dataset through a rigorous and multi-step quality control process. Although we employ "expert" workers who are native Indonesian speakers with at least a Bachelor's degree, additional measures are implemented to maintain and verify quality. First, all data sources are manually checked and validated by the author before being distributed to the workers. Workers also participate in a 1-hour workshop prior to data collection, ensuring they fully understand the guidelines and the expected data standards.

After the workers complete their tasks, we apply automated filtering to eliminate repetitive questions and entries without answer keys. To further validate the dataset, we conducted a manual review of 300 randomly selected samples (3.3% of the dataset), performed by the authors of this paper. During this review, we verified the accuracy of the questions, answer options, and answer keys. The manual review achieved an accuracy rate of 99%, demonstrating the dataset's reliability and representing the highest meaningfully achievable score for IndoCareer.

Data StatisticsTable 1 summarizes the distribution of questions in IndoCareer across 22 pro

\begin{table}
\begin{tabular}{l l c r} \hline \hline
**Field** & **Professions** & **Exam Type** & **\#Q** \\ \hline  & Medical Doctor & Competency Exam & 805 \\  & Pharmacist & Compotency Exam & 598 \\ Healthcare & Multivefivery & Competency Exam & 680 \\  & Nurse & Competency Exam & 497 \\  & Clinical Psychology & Other & 95 \\ \hline  & Life Insurance & Certification Exam & 476 \\  & Sharia Life Insurance & Certification Exam & 558 \\  & CFP & Certification Exam & 96 \\ Insurance \& CPA & Certification Exam & 663 \\ Finance & CPMA & Certification Exam & 169 \\  & CITA & Certification Exam & 253 \\  & Risk Management & Certification Exam & 194 \\ \hline Tourism \& Security & Vocational School & 222 \\ Hospitality & Hospitality & Vocational School & 367 \\  & Culinary Art & Vocational School & 294 \\ \hline Creative \& Design & Vocational School & 423 \\ Design & Fashion Design & Vocational School & 267 \\  & Broadcasting & Vocational School & 422 \\ \hline Law & Advocate & Certification Exam & 591 \\  & Police & Other & 280 \\ \hline Education \& Teacher Competency Test & Certification Exam & 538 \\ Training & Office Administration & Vocational School & 346 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Number of questions in IndoCareer across different professions. CFP stands for Certified Financial Planner, CPA stands for Certified Public Accountant, CPMA stands for Certified Professional Management Accountant, and CITA stands for Certified Indonesian Tax Accountant.

Figure 2: Example of questions in IndoCareer. The English translation is only for illustrative purposes.

 

fessions, organized into six main fields: Healthcare, Insurance & Finance, Tourism & Hospitality, Creative & Design, Law, and Education & Training. Each profession corresponds to specific exam types, including competency exams, certification exams, vocational school exams, and others. Healthcare encompasses five professions, such as Medical Doctor and Pharmacist, contributing a total of 2,675 questions. Insurance & Finance, the largest category with seven professions, includes fields like Life Insurance, Certified Public Accountant (CPA), and Risk Management, with 2,409 questions. Tourism & Hospitality covers three professions--Tourism, Hospitality, and Culinary Art--comprising 883 questions, while Creative & Design features 1,112 questions. The Law field includes Advocate and Police exams, with a total of 871 questions, while Education & Training, with Teacher Competency Tests and Office Administration, adds another 884 questions.

According to Table 2, healthcare questions are the longest, averaging 2 to 3 times the length of those in tourism and hospitality, and creative and design. The number of multiple-choice options is generally consistent across professional fields, averaging 4 options. However, the total character count of the options varies, with insurance and finance, and education and training having the longest options, exceeding 160 characters.

Additionally, we manually examined 300 random samples to assess whether answering the questions required local context.5 Our analysis revealed that 34% of the questions incorporated Indonesian local context, with a notable concentration in the fields of insurance and finance, tourism and hospitality, and law.

Footnote 5: The 300 random samples are the same as those used for the manual review. Given the 99% accuracy rate from the initial review, we included an additional 1% of randomly selected correct samples for the local context assessment.

## 4 Experiments

Pezeshkpour and Hruschka (2024); Zhou et al. (2024) demonstrated that LLMs are highly sensitive to the order of options in multiple-choice questions. To ensure a more robust evaluation, we report the average performance across three evaluations for each model: one using the original order of options and two with the options shuffled.6 We evaluated one closed-source model (GPT-4o) and 26 open-weight LLMs, comprising 18 multilingual models (BLOOMZ Muennighoff et al. (2022), mT0 Muennighoff et al. (2022), Gemma-2 Team et al. (2024), Aya-23 Ustun et al. (2024), LLAMA3.17) and 8 Indonesia-centric models (IndoGPT Cahyawijaya et al. (2021), BactrianID Li et al. (2023), Merak Ichsan (2023), Komodo Owen et al. (2024), SeaLLM Nguyen et al. (2023), SEA-LION Singapore (2023), and Cendol Cahyawijaya et al. (2024)). Details for each model can be found in the Appendix.

Footnote 6: For reproducibility, we also release two versions of IndoCareer with shuffled options, available at [https://huggingface.co/datasets/indolen/IndoCareer](https://huggingface.co/datasets/indolen/IndoCareer).

Footnote 7: [https://github.com/meta](https://github.com/meta)–llama/llama3

Our focus is on zero-shot experiments using the Indonesian prompt: _Ini adalah soal [subject] untubk [exam type]. Pilihah salah satu jawaban yang dianggap benar!_.8 For evaluation, we use the LM-Harness package Gao et al. (2024), selecting the answer based on the highest probability of the first token (i.e., A, B, C, D) in the generated output. Specifically, for GPT-4o, we used the gpt-4o model from OpenAI,9 selecting the answer based on the first letter generated in the output.10

Footnote 8: The English translation is “This is a {subject} question for [exam type]. Please choose the correct answer!”

Footnote 9: [https://openai.com/](https://openai.com/)

Footnote 10: For GPT-4o, we slightly adjusted the prompt, instructing the model to output only one of the options as the answer.

### Results

Table 3 summarizes the zero-shot performance of various large language models (LLMs) across professional fields in IndoCareer, highlighting significant differences in their ability to handle Indonesian professional exams. GPT-4o and LLaMA-3.1 (70B) emerge as the top-performing models, with GPT-4o achieving the highest overall accuracy at 72.3%, followed closely by LLaMA-3.1 (70B) with 68.5%. This 4-point gap demonstrates GPT-4o's superior capability in handling complex tasks across diverse professions. In contrast, other multilingual models show significantly lower accuracy, ranging

\begin{table}
\begin{tabular}{l r r r} \hline \multirow{2}{*}{**Field**} & \multicolumn{1}{c}{\multirow{2}{*}{**\# Questions**}} & \multicolumn{2}{c}{**\# Chars**} \\ \cline{3-4}  & & **Question** & **Answer** \\ \hline Healthcare & 2675 & 277.3 & 95.9 \\ Insurance and Finance & 2409 & 156.3 & 165.3 \\ Tourism and Hospitality & 883 & 99.8 & 96.2 \\ Creative and Design & 1112 & 101.0 & 100.5 \\ Law & 871 & 130.7 & 141.2 \\ Education and Training & 884 & 159.5 & 165.9 \\ \hline \end{tabular}
\end{table}
Table 2: Average question and answer length (in characters) for each profession fields.

 

between 38.0% and 60.0%, indicating their struggles with Indonesian-specific professional exams.

Indonesian-centric models, including SEALION, Komodo, and Cendol, underperform dramatically, with results close to random guessing in some fields. These findings suggest that existing Indonesian-centric models are not yet optimized for professional exam tasks, limiting their utility in practical applications. Notably, the SEA-LION (7B) and Komodo (7B) models achieve only 23.8% and 28.5% average accuracy, respectively, underscoring the gap between local adaptations and the more capable multilingual models.

Healthcare stands out as the most challenging professional field, with an average performance across all models at only 37.2%.11 This poor performance underscores the limitations of current off-the-shelf LLMs as reliable health advisors in the Indonesian context. These findings highlight the critical need for robust model adaptations and fine-tuning specifically tailored to Indonesian professional tasks to enhance performance and to ensure applicability in high-stakes domains such as healthcare.

Footnote 11: This figure is calculated by averaging all values in the Healthcare column of Table 3.

### Analysis

Shuffling the multiple-choice options leads to unstable results in insurance and finance.Table 4 lists the top 10 professions with the highest standard deviation (\(\sigma\)) in performance across three evaluation runs. While the standard deviations are relatively low, ranging from 1.5 to 3.0, they indicate minor instabilities in model predictions when the multiple-choice options are shuffled. For certain professions, such as Certified Financial Planner, Certified Indonesian Tax Accountant, and Certified Professional Management Accountant, the average rank correlation (\(\tau\)) drops below 0.9, indicating reduced consistency in model performance across evaluation runs. Although their deviations are not severe, they highlight areas where models are less robust to option shuffling, particularly in domains requiring nuanced reasoning. Across the entire

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Model (#parameters)** & **Healthcare** & \begin{tabular}{c} **Insurance** \\ **\& Finance** \\ \end{tabular} & \begin{tabular}{c} **Tourism \& R** \\ **Hospitality** \\ \end{tabular} & **Law** & \begin{tabular}{c} **Creative** \\ **\& Design** \\ \end{tabular} & 
\begin{tabular}{c} **Education** \\ **\& Training** \\ \end{tabular} & **Average** \\ \hline Random & 20.6 & 25.8 & 20.0 & 24.1 & 20.1 & 22.8 & 22.5 \\ \hline BLOOMZ (560M) & 17.9 & 23.9 & 19.3 & 27.5 & 17.6 & 24.9 & 21.3 \\ BLOOMZ (1.7B) & 28.2 & 34.7 & 40.2 & 32.6 & 39.0 & 35.9 & 33.7 \\ BLOOMZ (3B) & 29.8 & 39.2 & 42.2 & 37.3 & 44.2 & 40.8 & 37.3 \\ BLOOMZ (7B) & 32.9 & 41.7 & 47.1 & 40.3 & 48.9 & 45.1 & 40.7 \\ mT0xman (300M) & 22.3 & 26.2 & 21.7 & 23.5 & 22.2 & 19.5 & 23.1 \\ mT0base (580M) & 23.3 & 26.5 & 24.8 & 24.3 & 23.0 & 24.0 & 24.4 \\ mT0large (1.2B) & 25.0 & 26.8 & 25.3 & 24.2 & 24.3 & 23.3 & 25.2 \\ mTOd (3.7B) & 27.7 & 38.9 & 43.8 & 36.0 & 42.4 & 43.3 & 36.6 \\ mT0xl (13B) & 29.4 & 41.1 & 44.3 & 40.0 & 46.1 & 44.1 & 38.7 \\ Gemma-2 (2B) & 35.7 & 51.0 & 55.5 & 44.4 & 55.0 & 52.1 & 46.8 \\ Gemma-2 (9B) & 54.3 & 62.2 & 68.0 & 56.9 & 68.1 & 60.8 & 60.5 \\ Gemma-2 (2TB) & 58.3 & 64.2 & 71.7 & 60.2 & 71.7 & 62.6 & 63.5 \\ Aya-23 (8B) & 37.0 & 46.1 & 51.7 & 44.3 & 51.7 & 47.5 & 44.6 \\ Aya-23 (35B) & 43.9 & 52.9 & 59.0 & 50.4 & 61.8 & 53.3 & 51.7 \\ LLaMA-3.1 (8B) & 35.9 & 46.7 & 51.9 & 41.2 & 53.0 & 45.3 & 44.1 \\ LLaMA-3.1 (1nstruct (8B) & 44.8 & 53.6 & 61.1 & 47.7 & 63.3 & 54.9 & 52.4 \\ LLaMA-3.1 (70B) & 61.4 & 65.0 & 69.4 & 64.0 & 72.3 & 61.4 & 64.8 \\ LLaMA-3.1 (70B) & **64.4** & **69.3** & **74.2** & **68.1** & **75.1** & **65.3** & **68.5** \\ \hline Bacteria-ID (7B) & 20.5 & 29.0 & 22.7 & 26.6 & 25.5 & 25.1 & 24.7 \\ IndoGPT (117M) & 21.5 & 26.6 & 24.5 & 23.2 & 18.1 & 23.6 & 23.2 \\ Merak (7B) & 37.2 & 45.6 & 49.7 & 43.8 & 50.8 & 46.9 & 44.1 \\ SeaLLM (7B) & **41.1** & **54.7** & **56.0** & **44.7** & **61.3** & **50.8** & **50.1** \\ SEA-LION (7B) & 19.2 & 28.9 & 20.0 & 27.6 & 20.9 & 27.3 & 23.8 \\ Komodo (7B) & 25.5 & 29.7 & 27.4 & 30.5 & 29.8 & 31.8 & 28.5 \\ CendolaTS+val (13B) & 20.8 & 24.8 & 22.9 & 22.9 & 21.8 & 21.4 & 22.5 \\ CendolaTS+val (13B) & 23.3 & 28.6 & 22.7 & 24.7 & 24.0 & 25.2 & 25.1 \\ GPT-do & **68.3** & **73.5** & **75.7** & **75.4** & **78.3** & **67.4** & **72.3** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Zero-shot LLM performance (% accuracy), combined across professional fields. “Average” means the average across all questions in IndoCraeer.

 

dataset, however, the rank correlation remains high, with an average of 0.98. This indicates that while minor instabilities exist at the profession level, the overall dataset maintains stable performance.

LLMs perform well in life insurance certification but struggle with finance-related certifications.Figure 3 illustrates the performance of LLMs across the top 5 and bottom 5 professions in terms of accuracy relative to the passing scores. The passing scores for each exam, represented by red horizontal lines, were sourced from publicly available information. The figure highlights that while GPT-4o, LLaMA-3.1 (70B), and Gamma-2 (27B) achieve passing scores for professions such as life insurance, sharia life insurance, graphic design, midwgery, and teacher competency, they fall significantly short for finance-related certifications.

None of the models evaluated pass the exams for Certified Financial Planner (CFP), Certified Professional Management Accountant (CPMA), Certified Public Accountant (CPA), fashion design, or office administration. Notably, GPT-4o, the best-performing model overall, falls over 20 points below the passing score for CFP, emphasizing the difficulty of finance-related tasks. The results suggest that finance-related certifications, which often require domain-specific reasoning and detailed calculations, remain a challenge for current LLMs. On the other hand, professions with more straightforward knowledge requirements, such as life insurance or midwery, align better with the strengths of existing LLMs. These findings highlight the need for targeted fine-tuning and adaptation to improve performance in specialized and calculation-heavy fields like finance.

Questions with local context and numerical analysis pose greater challenges.We conducted an error analysis on the best-performing open-weight model, LLaMA-3.1 (70B), by examining 100 incorrectly predicted samples and 100 correctly predicted samples for comparison. These samples were drawn from the original questions, without applying option shuffling. The analysis showed that questions with Indonesian local context were more common among the incorrectly predicted samples, with 50% of the incorrect predictions containing local context, compared to only 22% among the correct predictions. Considering that IndoCareer contains 34% local context overall, as discussed in Section 3, this suggests that questions incorporating local context are particularly challenging for language models. This finding aligns with prior research [6], indicating that questions grounded in local context often introduce cultural or situational nuances not well-captured in the models' pretraining data.

In addition to local context, questions involving numerical analysis also posed significant challenges for LLaMA-3.1 (70B). Among the incorrectly predicted samples, 43 required numerical reasoning, compared to only 29 among the correctly predicted ones. Numerical questions often involve calculations or logical reasoning steps, which many LLMs are not explicitly optimized to handle. These results reveal two key areas where model performance could be improved: understanding and addressing culturally specific content and enhancing their capabilities for numerical reasoning.

\begin{table}
\begin{tabular}{l r l} \hline \hline
**Profession** & \multicolumn{1}{c}{\(\sigma\downarrow\)} & \multicolumn{1}{c}{\(\tau\uparrow\)} \\ \hline Clinical Psychology & 3.00 & 0.93 \\ Cert. Financial Planner & 2.91 & 0.68 \\ Cert. Professional Management Accountant & 2.00 & 0.90 \\ Fashion Design & 1.98 & 0.93 \\ Advocate & 1.96 & 0.91 \\ Police & 1.86 & 0.95 \\ Cert. Indo. Tax Accountant & 1.81 & 0.85 \\ Sharia Life Insurance & 1.81 & 0.97 \\ Risk Management & 1.76 & 0.97 \\ Tourism & 1.63 & 0.96 \\ \hline All & 1.57 & 0.98 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Top 10 professions with the highest standard deviation (\(\sigma\)). \(\tau\) represents the average rank correlation across three runs. The red cells are the three worse score. The scores are based on evaluations across 27 models.

Figure 3: Top 5 and bottom 5 professions based on the model’s accuracy disparity relative to the passing score.

 

Conclusion

We introduce IndoCareer as the most comprehensive dataset of professional exams across various job sectors in Indonesia. The dataset encompasses 22 professions, categorized into healthcare, insurance and finance, creative and design, tourism and hospitality, education and training, and law. Evaluations across different LLMs show that most off-the-shelf models demonstrate vocational and professional expertise below the passing scores. We believe IndoCraeer will be valuable in supporting LLM adaptation for various job sectors in Indonesia.

### Limitations

There are three main limitations to our work: (1) IndoCareer excludes multimodal data such as tables, audio, images, and videos. Including these would make the benchmark more comprehensive and reflective of real-world scenarios. However, since our focus is on LLM evaluation, we only include text-based questions; (2) Engineering-related professions are excluded from IndoCraeer because the language used in these exams is primarily English, while our focus is on the Indonesian language; (3) The evaluation is limited to multiple-choice questions and does not include text generation tasks. We follow prior work in using the multiple-choice format as an initial step to address the lack of professional and vocational exam benchmarks in Indonesian.

### Ethical Considerations

IndoCraeer is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License12 and is intended solely for academic research. The questions included in IndoCraeer are sourced from publicly available materials. We collected these questions in compliance with Indonesian Copyright Law No. 28 of 2014, specifically Article 44. This article states that the use, reproduction, and/or modification of works or related rights, in whole or in part, is not considered copyright infringement, provided the source is properly cited and the purpose is for education or research.13

Footnote 12: [https://creativecommons.org/licenses/by-nc-sa/4.0/](https://creativecommons.org/licenses/by-nc-sa/4.0/)

Footnote 13: [https://wipolex-res.wiopo.int/edocs/lexdocs/laws/en/id/id664en.pdf](https://wipolex-res.wiopo.int/edocs/lexdocs/laws/en/id/id664en.pdf)

## References

* Cahyawijaya et al. (2024) Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Winata, Bryan Wilie, Fajri Koto, Rahmad Mahendra, Christian Wibisono, Ade Romadohony, Karissa Vincentio, Jennifer Santoso, David Moeljadi, Cathya Wirawan, Frederikus Hudi, Muhammad Satrio Wickassono, Ivan Parmonagan, Ika Alfina, Ilhan Firdausi Putra, Samual Rahmandi, Yulianti Otenang, Ali Septantirad, James Jaya, Kaustubh Dhole, Arei Suryani, Rifki Afna Purti, Dan Su, Keith Stevens, Made Nindystama Nitysayas, Muhammad Adilazuarda, Ryan Hadiwijaya, Ryandito Diamaru, Tiezheng Yu, Vito Ghifari, Wenliang Dai, Yan Xu, Dyah Damapuspita, Harroy Wibowo, Cuk Tho, Ichwanl Kuro Karo, Tirana Fatyanosa, Ziwei Ji, Graham Neubig, Timothy Baldwin, Sebastian Ruder, Pascale Fung, Jerry Sujanii, Sakriani Sakti, and Ayu Purwarianti. 2023. NusaCrowd: Open source initiative for Indonesian NLP resources. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 13745-13818, Toronto, Canada. Association for Computational Linguistics.
* Calwayajaya et al. (2024) Samuel Calwayajaya, Holy Lovenia, Fairi Koto, Riffki Patri, Pawan Cenggoro, Jhonso Lee, Salasabil Akbar, Emmanuel Dave, Nuurshadieq, Muummadal Mahendra, Kr Putri, Bryan While, Genta Winata, Albamaji, Ayu Purwarianti, and Pascale Fung. 2024. Cendol: Open instruction-tuned generative large language models for Indonesian languages. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 14899-14914, Bangkok, Thailand. Association for Computational Linguistics.
* Chayanwijaya et al. (2021) Samuel Chayanwijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Khodra, Ayu Purwarianti, and Pascale Fung. 2021. IndoNLG: Benchmark and resources for evaluating Indonesian natural language generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 8875-8898, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Gao et al. (2024) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonnam Li, Kyle McDonnell, Niklas Muennighoff, Chris Occiega, Jason Phanga, Laria Reynolds, Halisey Schoeklopf, Aviya Skowron, Linntag Sutwikai, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.
* Gundersen and Berese (2022) Torbjorn Gundersen and Kristine Berese. 2022. The future ethics of artificial intelligence in medicine: making sense of collaborative models. _Science and engineering ethics_, 28(2):17.
* Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.

 

2021. Measuring massive multitask language understanding. In _International Conference on Learning Representations_.
* Hu et al. (2022) Edward J Hu, yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In _International Conference on Learning Representations_.
* Ichshan (2023) Muhammad Ichshan. 2023. Merak-7b: The llm for bashaa indonesia. _Hugging Face Repository_.
* Koto et al. (2023) Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMALU. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 12359-12374, Singapore. Association for Computational Linguistics.
* Koto et al. (2021) Fabiri Koto, Joy Han Lau, and Timothy Baldwin. 2021. IndoBERTweet: A pretrained language model for Indonesian Twitter with effective domain-specific vocabulary initialization. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 10660-10668, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Koto et al. (2024a) Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahaman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Ajayafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024a. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 5622-5640, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Kato et al. (2024b) Fajri Koto, Rahmad Mahendra, Nurul Aisyah, and Timothy Baldwin. 2024b. Indoculture: Exploring geographically-influenced cultural commonsense reasoning across eleven indonesian provinces. _arXiv preprint arXiv:2404.01854_.
* Koto et al. (2020) Fajri Koto, Afshin Rahimi, Jey Han Lau, and Timothy Baldwin. 2020. IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 757-770, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Lee and Soon (2024) Meisin Lee and Lay-Ki Soon. 2024. 'finance wizard' at the FinLLM challenge task: Financial text summarization. In _Proceedings of the Eighth Financial Technology and Natural Language Processing and the 1st Agent AI for Scenario Planning_, pages 153-158, Jeju, South Korea. -.
* Li et al. (2023) Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-X: A multilingual replicable instruction-following model with low-rank adaptation. _arXiv preprint arXiv:2305.15011_.
* Li et al. (2024) Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. CMMLU: Measuring massive multitask language understanding in Chinese. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 11260-11285, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Liu et al. (2024) Chen Cecilia Liu, Iryna Gurevych, and Anna Karbonen. 2024. Culturally aware and adapted nlp: A taxonomy and a survey of the state of the art. _arXiv preprint arXiv:2406.03930_.
* Mikheev et al. (1999) Andrei Mikheev, Marc Moens, and Claire Grover. 1999. Named entity recognition without gazetteers. In _Ninth Conference of the European Chapter of the Association for Computational Linguistics_, pages 1-8, Bergen, Norway. Association for Computational Linguistics.
* Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xir Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. _arXiv preprint arXiv:2211.01786_.
* Nguyen et al. (2023) Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Living Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, et al. 2023. SeCallms-large language models for southeast asia. _arXiv preprint arXiv:2312.00738_.
* Owen et al. (2024) Louis Owen, Vishesh Tripathi, Abhay Kumar, and Biddwan Ahmed. 2024. Komodo: A linguistic expedition into Indonesia's regional languages. _arXiv preprint arXiv:2403.09362_.
* Pezeshkpour and Hruschka (2024) Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In _Findings of the Association for Computational Linguistics: NAACL 2024_, pages 2006-2017, Mexico City, Mexico. Association for Computational Linguistics.
* Singapore (2023) AI Singapore. 2023. Sea-lion (southeast asian languages in one network): A family of large language models for southeast asia. [https://github.com/aisingapore/sealion](https://github.com/aisingapore/sealion).
* Srivaras and Kochmar (2024) Kv Aditya Srivastava and Ekaterina Kochmar. 2024. What makes math word problems challenging for LLMs? In _Findings of the Association for Computational Linguistics: NAACL 2024_, pages 1138-1148, Mexico City, Mexico. Association for Computational Linguistics.
* Straka and Strakova (2017) Milan Straka and Jana Strakova. 2017. Tokenizing, POS tagging, lemmatizing and parsing UD 2.0 with UDPipe. In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_, pages 88-99, Vancouver, Canada. Association for Computational Linguistics 

* Team et al. (2024) Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Rame, et al. 2024. Geamma 2: Improving open language models at a practical size. _arXiv preprint arXiv:2408.00118_.
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacadr, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. _CoRR_, abs/2302.13971.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Ustun et al. (2024) Ahmet Ustun, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D'souza, Ghemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction finetuned open-access multilingual language model. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15894-15939, Bangkok, Thailand. Association for Computational Linguistics.
* Weijers et al. (2024) Ruben Weijers, Gabrieille Fidelis de Castilho, Jean-Francois Godbour, Reihaneh Rabbany, and Kellin Pelrine. 2024. Quantifying learning-style adaptation in effectiveness of LLM teaching. In _Proceedings of the 1st Workshop on Personalization of Generative AI Systems (PERSONALIZE 2024)_, pages 112-118, St. Julians, Malta. Association for Computational Linguistics.
* Wibowo et al. (2024) Haryo Wibowo, Erland Fuadi, Made Nityasaya, Radityo Eko Prasaojo, and Albham Aji. 2024. COPAL-ID: Indonesian language reasoning with local culture and nuances. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 1404-1422, Mexico City, Mexico. Association for Computational Linguistics.
* Wilie et al. (2020) Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_, pages 843-857, Suzhou, China. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remul Louf, Morgan Funowiczic, Joe Davison, Sam Sheifeer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Zhang et al. (2024) Kai Zhang, Yangyang Kang, Fubang Zhao, and Xiaozhong Liu. 2024. LLM-based medical assistant personalization with short- and long-term memory coordination. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 2386-2398, Mexico City, Mexico. Association for Computational Linguistics.
* Zhou et al. (2024) Wenjie Zhou, Qiang Wang, Mingzhou Xu, Ming Chen, and Xiangyu Duan. 2024. Revisiting the self-consistency challenges in multi-choice question formats for large language model evaluation. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_, pages 14103-14110, Torino, Italia. ELRA and ICCL 



## Appendix A Models

### Full Results

Table 6 presents the accuracy of each model across various professions. The passing scores for each exam were sourced from publicly available information. We found that GPT-40 passes most of the exams, with the exceptions being Certified Financial Planner, Certified Public Accountant, Certified Professional Management Accountant (CPMA), and Office Administration. LLaMA-3.1 and Gemma-2 also pass some Indonesian exams, but no Indonesian-centric model has yet passed the professional and vocational exams in Indonesia.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Models (#parameters)** & **Source** \\ \hline BLOOOMZ (560M) & bigscience/blomoz=56m \\ BLOOMZ (1.1B) & bigscience/blomoz=1b1 \\ BLOOMZ (7.1B) & bigscience/blomoz=1b7 \\ BLOOMZ (3B) & bigscience/blomoz=1b \\ BLOOMZ (7.1B) & bigscience/blomoz=7b1 \\ nTon-2 (300M) & bigscience/nt+smail \\ mTbon\_nost (850M) & bigscience/net/base \\ nTlong\_(1.2B) & bigscience/nt+large \\ nTO\_tQ (1.3TB) & bigscience/ntq-xl \\ mTol\_tQ (13B) & bigscience/ntq-xl \\ Genma-2 (2B) & google/gemna-2-2b-it \\ Genma-2 (9B) & google/gemna-2-2b-it \\ Genma-2 (27R) & google/gemna-2-2rb-it \\ Ayz=23 (8B) & Corber/Carl/Aya-23-bs \\ Ayz=23 (3SB) & Corber/Carl/Aya-23-bs \\ LiLAMA3:1 (8B) & nete-lima/Meta-Llama-3.1-lb8 \\ LLAMA3.1-Instruct (BB) & meta-llama/Meta-Llama-3.1-lb8-Instruct \\ LLAM3.1-OutB (7bB) & nete-lima/Meta-Llama-3.1-lb8 \\ LLAMA3.1-clat (7OB) & nete-lima/Meta-Llama-3.1-lb8 \\ Backrim-ID (7B) & haonomi-1/hactin-id-liama-7b-lora \\ IndoBART (132M) & indomethacin/indobert/uiz \\ IndoGPT (17M) & indohenchark/indoget \\ Merak (7B) & Ichshan285V/Merak-7B-v->ProductYPF1 \\ SealLM (7B) & SealLMUs/SeadLus-37B-Chat \\ SEAL-LION (7B) & higasinger/seai-lm-7b \\ Komoda (7B) & yellow-Al-NLP/Komodo-7b-base \\ Cendoln\_trSSL (13B) & indonly/cendol\_nt5-rxrl-nerged-inst \\ Candid\_tLAME (13B) & indonly/cendol\_11-mana-13-bp-nerged-chat \\ \hline \hline \end{tabular}
\end{table}
Table 5: With the exception of GPT-40, all the models used in this study were sourced from Huggingface (Wolf et al., 2020).





\begin{table}
\begin{tabular}{l l c c c c|c c c c c c c}
**Profession** & **P.Score** & **BLOOMZ** & **mT0** & **Aya-23** & **Germa-2** & **LLaMA3.1** & **Merak** & **SeaLLM** & **SEA-LION** & **Komodo** & **Cendolol** & **GPT-do** \\ \hline
**Healthcare** & & & & & & & & & & & & & & \\ Medical Doctor & 66.0 & 33.4 & 27.3 & 45.1 & 61.6 & 70.9 & 39.6 & 42.9 & 19.9 & 24.1 & 23.1 & 74.8 \\ Pharmacist & 57.0 & 30.0 & 24.4 & 41.2 & 55.8 & 62.9 & 36.2 & 40.3 & 19.4 & 23.0 & 23.0 & 64.5 \\ Midwifery & 40.0 & 29.4 & 30.9 & 42.2 & 52.2 & 57.9 & 33.1 & 37.4 & 20.5 & 22.8 & 22.2 & 63.3 \\ Nurse & 60.0 & 35.3 & 33.9 & 45.5 & 58.4 & 60.8 & 36.2 & 44.7 & 23.8 & 25.6 & 25.7 & 66.4 \\ Clinical Psychology & 70.0 & 53.3 & 50.7 & 62.7 & 73.6 & 71.4 & 53.3 & 62.7 & 26.4 & 33.7 & 34.1 & 74.3 \\ \hline
**Insurance \& Finance** & & & & & & & & & & & \\ Life Insurance & 70.0 & 48.7 & 48.1 & 61.3 & 71.7 & 76.5 & 49.0 & 59.3 & 27.3 & 35.0 & 30.2 & 77.3 \\ Sharia Life Insurance & 70.0 & 45.9 & 47.1 & 61.6 & 77.8 & 81.7 & 51.7 & 64.1 & 32.2 & 30.0 & 31.8 & 85.3 \\ Cert. Financial Phanner & 70.0 & 29.4 & 27.6 & 36.9 & 41.9 & 49.5 & 25.4 & 38.0 & 24.0 & 22.6 & 22.2 & 49.8 \\ Cert. Public Accountant & 75.0 & 37.9 & 36.7 & 44.7 & 58.4 & 61.7 & 42.8 & 48.2 & 25.9 & 27.0 & 26.6 & 67.8 \\ Cert. Indo. Tax Accountant & 60.0 & 37.5 & 39.1 & 41.7 & 47.3 & 50.4 & 34.3 & 45.5 & 32.3 & 33.2 & 31.5 & 60.7 \\ CPMA & 75.0 & 32.7 & 28.7 & 40.6 & 55.2 & 62.0 & 38.6 & 40.8 & 26.9 & 25.9 & 24.1 & 61.4 \\ Risk Management & 70.0 & 41.7 & 37.9 & 51.1 & 64.0 & 67.2 & 45.0 & 53.2 & 26.9 & 29.3 & 32.1 & 70.9 \\ \hline
**Tourism \& Hospitality** & & & & & & & & & & & \\ Tourism & 70.0 & 51.3 & 53.6 & 58.1 & 72.6 & 74.3 & 45.8 & 58.8 & 21.8 & 30.1 & 24.2 & 76.6 \\ Hospitality & 70.0 & 43.0 & 43.4 & 54.8 & 67.2 & 69.0 & 47.6 & 55.4 & 23.4 & 25.1 & 26.4 & 71.7 \\ Culinary Art & 70.0 & 47.0 & 45.2 & 62.0 & 73.5 & 76.7 & 50.1 & 60.4 & 22.8 & 28.3 & 22.2 & 79.5 \\ \hline
**Creative \& Design** & & & & & & & & & & & \\ Fashion Design & 70.0 & 34.8 & 35.2 & 47.1 & 59.0 & 62.4 & 36.4 & 49.1 & 20.6 & 25.3 & 21.7 & 66.2 \\ Graphic Design & 70.0 & 52.9 & 53.8 & 65.3 & 76.7 & 80.6 & 54.8 & 65.0 & 23.6 & 29.3 & 28.0 & 82.2 \\ Broadcasting & 70.0 & 51.6 & 49.2 & 63.9 & 75.3 & 77.3 & 54.7 & 63.9 & 23.4 & 30.7 & 25.0 & 79.8 \\ \hline
**Law** & & & & & & & & & & & & \\ Advocate & 70.0 & 34.6 & 39.9 & 47.1 & 59.9 & 68.7 & 36.7 & 41.9 & 26.4 & 27.3 & 26.4 & 72.9 \\ Police & 60.0 & 44.2 & 37.4 & 47.7 & 56.2 & **64.0** & 45.4 & 47.5 & 21.7 & 27.0 & 26.5 & 67.6 \\ \hline
**Education \& Training** & & & & & & & & & & & \\ Teacher Competency & 55.0 & 46.6 & 44.1 & 53.4 & 61.5 & 62.1 & 45.5 & 48.8 & 27.5 & 29.3 & 27.7 & 65.6 \\ Office Administration & 70.0 & 38.9 & 42.1 & 52.1 & 64.0 & 68.2 & 42.0 & 50.0 & 24.2 & 27.7 & 23.0 & 69.9 \\ \hline \end{tabular}
\end{table}
Table 6: Zero-shot LLM performance (% accuracy) across professions for each model. “P.Score” indicates the passing score for each exam. The models used in this table include BLOOMZ (7B), mT0xxl, Aya-23 (35B), German-2 (27B), LLaMA-3.1Instruct, Merak (7B), SeaLLM (7B), SEA-LION (7B), Kondo (7B), Cendolol\({}_{\text{LLaMA2}}\) (13B) and GPT-4o. Green cells indicate that the model meets or exceeds the passing score.