MedAraBench: Large-scale Arabic Medical Question Answering Dataset and Benchmark
Authors: Mouath Abu Daoud
Leen Kharouf
Omar El Hajj
Dana El Samad
Mariam Al-Omari
Jihad Mallat
Khaled Saleh
Nizar Habash
Farah E. Shamout
Venue: ICLR 2026 Poster (2026)
Keywords: Dataset Benchmark
Large Language Models
Arabic Natural Language Processing
Medical Question Answering
Primary area: datasets and benchmarks

Abstract: Arabic remains one of the most underrepresented languages in natural language processing research, particularly in medical applications, due to the limited availability of open-source data and benchmarks. The lack of resources hinders efforts to evaluate and advance the multilingual capabilities of Large Language Models (LLMs). In this paper, we introduce MedAraBench, a large-scale dataset consisting of Arabic multiple-choice question-answer pairs across various medical specialties. We constructed the dataset by manually digitizing a large repository of academic materials created by medical professionals in the Arabic-speaking region. We then conducted extensive preprocessing and split the dataset into training and test sets to support future research efforts in the area. To assess the quality of the data, we adopted two frameworks, namely expert human evaluation and LLM-as-a-judge. Our dataset is diverse and of high quality, spanning 19 specialties and five difficulty levels. For benchmarking purposes, we assessed the performance of sixteen state-of-the-art open-source and proprietary models, such as GPT-5, Gemini 2.0 Flash, and Claude 4-Sonnet. Our findings highlight the need for further domain-specific enhancements. We also explore QLoRA fine-tuning on LLaMa-3.1-8B-instruct to assess our dataset's viability. We release the dataset and evaluation scripts to broaden the diversity of medical data benchmarks, expand the scope of evaluation suites for LLMs, and enhance the multilingual capabilities of models for deployment in clinical settings.

Published as a conference paper at ICLR 2026
AEDERABENCH: LARGE-SCALE ARABIC MEDICAL QUESTION ANSWERING DATASET AND BENCHMARK
Mouath Abu-Daoud', Leen Kharouf", Omar El Hajj', Dana El Samad', Mariam Al-Omari', Jihad Mallat Khaled Salch Nizar Habash", Farah E. Shamout
1 Engineeriig Division, New York University Abu Dhabi, UAE "Science Division, New York University Abu Dhabi, UAE *Cleveland Clinic Abu Dhabi, UAE {
{mma9138, 1k2713, ose6432, dae7168, ma8058, nh48, 1s999) 1s999)enyu.cm edu {mallatj,
ABSTRACT
Arabic remains one of the most underrepresented languages in natural language processing research, partieularly in medical applications, due the limited availability of open-source data and benchmarks. The lack of resources hinders efforts evaluate and advance the multilingual capa- bilities of Large Language Models (LLLss). In this paper, we introduce MedAraBench, largo-scale dataset consisting of Arabic multiple-choice question-answer pairs across various medical specialties. We constructed the dataset by manually digitizing a large repository of academic materials created by medical professionals in the Arabic-speaking region. We then conducted extensive preprocessing and split the dataset into training and test sets to support future research efforts in the area. To assess the quality of the data, we adopted two frameworks, namely expert human evaluation and L.LM-as-ajjudee. Our dataset is diverse and of high quality, spanning 19 specialties and five difficulty levels. For benchmarking purposes, we assessed the performance of sixteen state-of-the-a prietary models, such as GPT-5, Gemini 2.0 Flash, and Claude 4-Sonnet. Our findings highlight the need for further domain-specific enhancements. We also explore QLoRA fine-tuning on LLaMa-33.1 dataset's viability. We release the dataset and evaluat ion scripts to broaden the diversity of medical data benchmarks, expand the scope of evaluation suites for LLMs, and enhance the multilingual capabilities of models for deployment in clinical settings.
open-source and pro-
to assess our
1 INTRODUCTION
The emergence of Large Language Models (LLMs) has driven transformative progress in Natural Language Processing (NLP) in recent years. They have demonstrated exceptional performance across various renowned benchmarks due to their powerful understanding and reasoning abilities, grounded the vast amount of knowledge in their training corpora
(Brown et al. 2020 Bommasani et al. 2022 Chowdhery et. al. 2022 This includes general
and domain-specific benchmarks (Wang el al. 2021 Stahlberg 2020
However, performance improvements remain variable across underrepresented languages and domains, particularly in high-stakes applications like medicine (Jiang et al. 2025 Yang et al. 2025). For example, Arabic is among the most spoken languages in the world, with over 400 million speakers across the globe. However, it remains underrepresented in the medical
domain, mainly due to limited expert-annotated resources (Habash) 2005). Arabic NLP generally presents inherent language-specific linguistic challenges (Habash 2005 Farghali
& Shaalan, 2009 AlMoaiad 2024), making the availability of such resources essential for

---

Published as a conference paper at ICLR 2026
Table Comparison of MedAraBench with existing medical QA benchmarks, including estimated datasct sizc.
Benehmark
Langugge(s)
Type
Size
Expert Annotation
Difficully Mapping
Specially Coverage
Arable
Public
McdQA
English. Chinese
Me1MCQA
Eaglish
MCQ=
153,000
MMLU (USMLE)
English
MOQ=
1,500
MMLU Transaaion
Azabie
MCQs
AzsMed
Asable
QA
270,000
x
MedArabiQ
Azabie
QA and MCQs
700
/
MedAraBench (Ours)
MCQ+
assessing the performance of LLMs, especially as they are being deployed in diverse medical contexts.
Several benchmarks have been recently introduced in the medical domain, as summarized in
Table 1 However, most of them focus almost exclusively on English (Jin et al.
2021 2019)
Recent work began to address this need but remains limited in scope and size [Abu Daoud
ct.al al. 2025). Thus, there is pressing need for large-scale benchmarks to assess and improve 1.MS for Arabic-language medical reasoning.
To address those gaps, in this paper, we present MedAraBench, a comprehensive benchmark for evaluating and advancing LLMs on Arabic medical tasks. The dalaset consists of curated Kultiple-Choiee Questions (MCQs) spanning different specialties and difficulty tiers aligned with stages of medical education. We propose a standardized development and evaluation protocol to enable reproducible and clinically meaningful assessment of LLMs. Our key contributions are as follows:
We introduce MedAraBench, large-scale Arabic medical benchmark featuring 24,883 MCQs across 19 medical specialties and five difficully levels. The bench- mark includes standardized training and test sets to enable systematic evaluation and advancement of LLMs.
We perform extensive quality assessment via human expert evaluation, focusing on question clarity, clinical relevance, and medical correctnoss, well automated LLM-as-a-judge analysis.
We benchmark 16 state-of-the-art proprietary and open-source LLMs across three categories (general purpose, Arabic-centric, medical) on the Aranche test set in the zero-shot setting to establish baseline performance for future research.
2 RELATED WORK
In recent years, several benchmark datasets have been developed to assess the capabilities of LLMs the medical domain, driven by the expanding demand for applications that can streamline clinical workflows. Despite this progress, Arabic remains underrepresented in clinical NLP, mainly due to the lack of high-quality data to support building clinical ap- plications in Arabic (Abdelaziz et al. 2025 As such, most existing benchmarks focus on English. For instance, the Massive Multitask Language Understanding (MMLU) benchmark includes question-ant qustiion-answer pairs from the US Medical Licensing Exam (USMLE) (Hendrycks) el. al. 2021). Jin et. al. (2020) introduce MedQA, a multilingual benchmark dalaset con- sisting of multiple-choice questions sourced from medical licensing exams Fnglish and Chinese. Pal et al. (2022) extend these benchmarks in MedMCQA to multilingual evalu- ation framework but remains limited in Arabic.
Recent work has introduced new resources for medical evaluation in Arabic. Translations of existing datasets, such as of MMLU into 42 languages, including Arabic, provide valuable data but lack the necessary nuances for proper integration into clinical practice (Singh 2025). AraMed presents an Arabic medical corpus and an annotated Arabic QA dataset sourced from online medical platforms (Alasmari et al. 2024). Ardbia presents of
2

---

Published as a conference paper at ICLR 2026
BiMedie-il Source
Panar 6.78 Open Source
Apello ided
Open Scurco
Cizude Sennet-4 Proper etory
ModGemma
M2 Med42
Sample
Preproeasiin
0PT-4
Pippretor
Prrprietryy
Lima-
Llama 5-1
Deepsook
Propristary
Processing Pipeline
tropekiarr
MedAraBench
Specialties
Results
Plysnn
Figure 1: Overview of MedCraBenchh.
the first Arabic medical MCQ datasets, yet lacks specialty coverage, difficulty mapping, and expert evaluation (Abu Daoud et al. 2025).
Several evaluation frameworks have been proposed to evaluate the performance of clinical AI models. Kanithi et (2024) introduce 'MEDIC", framework for evaluating LLMss from medical reasoning and ethics, to in-context learning and clinical safety. [Wang et al. (2024) propose testing models on real-world input noise, dialogue interruptions, and reasoning jus- tifications. Despite the growing interest in multilingual evaluation, there remains a critical gap in comprehensive, high-quality, and clinically relevant benchmarks for under-served languages. We aim to address this gap by introducing comprehensive Arabic beuchmark. Table provides a structured comparison between MedAraBench and other existing medical benchmarks, covering key dimensions such language coverage, specialty diversity, expert annotation, and public availability.
METHODOLOGY
Here, describe the steps pertaining to data collection, processing, evaluation, bench- marking, fow-shot lcarning, and finetuning, to facilitate proper reuse and fair comparison, aligning with best practices for benchmark construction and evaluation. An overview is provided in Figure
3.1 DATA COLLECTION PRE-PROCESSING
We compiled large repository of scanned paper-based exams, hosted on student-led social platforms of regional medical schools. The dataset did not include any personal or real patient data, anonymization was not necessary, and data collection complied with privacy and ethical standards. Considering the nature the documents, we recruited pro- fessional typists to digitize the data. We then aggregated the paper-based exam documents to build a single MCQ datasct.
Upon manual inspection by NLP researchers, observed that several documents exhib- ited issues such as missing or malformed correct answers, incomplete or duplicated answer choices, non-standard formatting misaligned fields, and ambiguous answer keys extra-
3

---

Published as a conference paper at ICLR 2026
neous non-MCQ content. To ensure dataset quality and model compatibility, we applied strict filtering criteria to remove any questions with such issues. The filtering process was performed manually by five NLP researchers. While data acquisition and preprocessing quired extensive effort, it highlights that the dataset is not publicly accessible in structured formats, thus reducing the likelihood of data contamination.
Each question is associated with several annotations: (i) number of answer choices (i.c., ABCD (4 choices), ABCDE (5 choices), and ABCDEF (6 choices)), (ii) difficulty level cor- responding to five years of study (Y1 - Y5), and (iii) medical specialty, The questions fall under 19 medical specialties: Anatomy, Anesthesia, Biochemistry, Cell and Molecu- lar Biology. Chemistry and Physics, Embryology, Emergency Medicine, Internal Medicine, Medical Ethics, Microbiology, Ophthalmology, Pathology, Pediatrics, Pharmacology, Phys- iology, Preventive Medicine, Psychiatry, Statistics, and Surgery. The scanned documents collected from the repositories were originally categorized by specialty, which allowed us to directly inherit the medical specialty classification for each question.
We started by omitting all questions with 6 answer choices due to the small size of this subset (9 questions), allowing us lo calegorize our data into two categories (4 answer choices and answer choices). We then performed a stratified random split of the dataset into training (80%) and test sets (20%). This was to ensure that the medical specialties are represented evenly across the training and test sets (i.e., if the dataset contains 100 Cardiology questions, 80 would be randomly included in the training set, while the remaining 20 would be in the test set). A sunmary of the dataset in terms of token length distribution, medical specialty distribution, and difficulty level distribution is presented Appendix [A]
It is important lo note that the homogeneity regarding medical lerminology consistency is essential in assessing the capabilities of LLMs on standardized Arabic medical tasks. As it pertains to MedhraBench, we did not perform any external standardization across the dataset as we acknowledge that this may not be a major issue from benchmarking perspective since real-world medical QA may not necessarily be standardized in terms of terminology.
We acknowledge that Arabic medical terminology standardization remains an ongoing chal- lenge, with unified vocabulary frameworks still under development across Arabic-speaking regions [Amar 2022), Atlii 2024). The expert clinician evaluations described in Section 3.2 serve as quality control, with reviewers achieving high agreement on all metrics despite the absence of standardization.
3.2 QUALITY ASSESSMENT
To further assess the quality of our dataset, we conducted two analyses: human expert evaluation and using LLM-as-a-uudge.
3.2.1
HUMAN EXPERT EVALUATION
We designed our expert evaluation protocol to assess the data according to the following criteria:
1. Medical Accuracy:
the extent to which the question, options, and correct answer reflect current, evidence- based medical knowledge (Scale: high or low) (Olatunii
ct al. 2024 Iskander et al. 2024 Rejeleene et al., 2024)..
2. Clinical Relevance:
the practical importance and applicability of the question content to real-world medical practice education (Scale: high or low) Iskander
ct al. 2024 Olatunji et al. 2024).
3. Question Difficulty: the complexity required to answer the question correctly (Scale: high or low) (Iskander al. 2024).
4. Question Quality: assessment of the MCQ construction quality (Scale: high or low) following established medical education standards (Al-Rukban 2006
Clarity: question is clear, complete, and unambiguous.
4

---

Published as a conference paper at ICLR 2026
Option Homogeneity: all distractors plausible and of similar type.
Single Best Answer one clearly correct option exists.
No Clueing: options do not provide clues to other answers.
We selected samples for review from the test set and determined the sample size based Cochran's 8 formula (Cochran, 1977), to create a representative sample size (Hosseini (024). We estimated single proportion at 95% confidence with a #5 percentage-point margin of error, using p = 0.5 as a consorvative assumption when the true quality rate is unknown because it maximizes variance and therefore yields safe upper bound on sample size. We first calculated an estimated sample size assuming an infinile population using
z² z²p(1 p))
n0
with = = 1.96, p= 0.5, and e 0.05. However, since the dataset is finite, we then applied the finite population correction using
no
71
1+ n
We recruited two board-certifidd clinicians specializing in Ancsthosiology and Internal Medicine with Arabic clinical fluency and over 20 years of experience cach. The reviews were couble-blinded to model outputs and data provenance, and were conducted independently with pre-registered instructions web-based survey platform allowing for standardized presentation of evaluation criteria and independent response collection across both reviewers.
Quallrics (Provo, UT, https://w www.wualwrree.eeeooooooooooooooe come,,
3.2.2 LLM-AS-A-JUDGE
Considering that only a subset of the test set was considered for the human expert evaluation, further introduced an LLM-as-a-judge evaluation protocol as additional rating of data quality. Motivated by previous literature (Abu Daoud et al. 2025), we prompled four of best-performing SOTA LLMs (gpt-03, gemini-2.0-llashh, and claude-4-sonnet) medical education experts and to evaluate the MCQs along the same metrics used in our expert quality evaluation: Medical Accuracy, Clinical Relevance, Question Difficulty, and Question Quality, on binary (0 or 1) scale for the entire test set. Additionally, we calculated Pcarson Correlation cocfficients for each model and the expert reviewers on the 378-question dataset evaluated by medical experts. This approach gives us the advantage of providing a more nuanced evaluation across broader set of our data, while also providing insights into the efficacy of LLMs in evaluating the quality of Arabic medical data relative to expert annotators.
act
3.3 BENCHMARKING PROTOCOL
To introduce new benchmark results, we evaluated 16 proprietary and open-source models on the test set of the MedAraBench benchmark. We set the models' temperature as to ensure stable outputs as shown in previous work for classification of MCQs (Abu Daoud We selected one letter response per question.
et al. 2025
Open-source models: Llama-3, 3-770--insttret (Grattafiori et al. 2024a). Llama-
3.1-Sb-instruct (Grattafiori et al. 2024b). seepseek-chat-v3--03
DeepSeek-At Al
2024h.
Allam-7b-instruct (Bari 2024). Cohere 2at.mmmmmann----aabb------22222
Alnumay 2025). Medgemma- 4b-it (Sellergren 2025), Apollo-7b
(Wang 2024).
Fanar-C-T
8.7b (Abbas 2025 BiModix-Bi-27B
Pieri et al. 2024). and Mcd42-8b
Christophe 2024).
Proprictary models: Claude-sonnet
(Anthropic 2025 Gemini-22 2.0
flash (Google Cloud, 2025). GPT-4.1 (OenAA 2025a), G
OpenAi 2025b
GPT-03 (OpenAT 2025ch, and Qwen-plus Yang & et al. 2024

---

Published as a conference paper at ICLR 2026
Our prompt is shown in Appendix B below. Answers extracted via post-processing, whereas models were prompted to output the answer choice letter (A-D) directly, and we parsed their text responses using pattern matching to extract the answer. No set language parameters in API and model calls were used since models automatically detect Arabic text. from the input. The results of our benchmarking experiments shown in Table 4 below.
3.4 FEW-SHOT LEARNING PROTOCOL.
Building upon zero-shot benchmarking protocol, conducted few-shot experiments 1--Maa-.1--B-inttree to assess in-context learning capabilities. We provided 3 high-quality sample questions that were rated highly across all evaluation metrics (question quality, clin- ical relevance, difficulty, and medical accuracy) by expert evaluators. These exemplar ques- tions were carefully selected from the training split and omitted from the test to ensure fair evaluation. The few-shot examples covered diverse medical topics including anatomy, biochemistry, and physiology, formatted with questions, options, and correct answers in Arabic to maintain linguistic consistency with the test items. The chosen questions and few-shot prompt are provided Appendix B below.
3.5 LOW-RANK ADAPTATION
We further investigated parameter-efiee efficcien fine-tuning using QLoRA (Quantized Low-Rank Adaptation) on the and trained on the MedAraBench training split, formatted Arabic prompt-response pairs for multiple-choice question answering. LoRA adapters were applied to key attention mod- ules (q-proj, k-proj, v-proj, o-proj) with standard hyperparameters, training for up to 800 steps with batched gradient accumulation. This approach enabled efficient adaptation to Arabic medical terminology and reasoning patterns while preserving the models' general ca- pabilities. The fine-tuned models were evaluated the same test set used in zero-shot and few-shot experiments to directly measure the impact of MedA Bench data on model performance.
Baaaa-..----iittttee model. The model was loaded in 4-bit precision
RESULTS
In this section, we present summary of our dataset and the results of our expert quality evaluation, LLM-as-a-judge experiments, and benchmarking experiments.
4.1 DATASET SUMMARY
The initial dataset consisted of 34,333 MCQs. The manual filtering process resulted in reduction of approximately 29% of the initial dataset, yielding 24,883 samples overall. The training set consisted of 19,894 samples, and the test set of 4,989 samples. An overview of the dataset shown in Figure AT in Appendix [ below, along with additional statistical sunmaries of the dataset.
4.2 EXPERT QUALITY ASSESSMENT
At 95% confidence and =5% margin, Cochran's formula initially yielded no 384 questions. The final sample size was 378 after adjusting for finite sample. Hence, our two annotators completed review of 378 questions
representative sample of the entire dataset. The results of our data quality assessment and the inter-annotator agreement are summarized in Table 2 Our results show slight to fair levels of agreement across all metrics, with Medical Accuracy having the highest level of agreement with a Cohen's 's Kappa score of 0.555 and percentage agreement of 82%.
Additionally, we provide a detailed per-specialty breakdown of the evaluation results in Appendix C. To better assess the results, we investigate individual average and agreement scores for each specialty. Specifically, Figures CT per specialty for each metric, while Tables
C2
C3 and [hohow the average accuracy
and C4 show the average accuracy

---

Published as a conference paper at ICLR 2026
Table 2: Expert quality assessment results for the representative data subset.
Metric
Average (standard deviation]
Percent Agreement Cohen's Kappa
Medical Accuracy
0.722 [0.448]
82.0%
0.555
Clinical Relevance
0.653 [0.476
65.6%
0.275
Question Difficutyy
0.669 00.471
65.6%
0.233
Question Quality
0.767 [0.423)
68.3%
0.152
Table 3: Evaluation results of LLM-as-a-judde applied to the test set.
Model
Evaluation Metric (average)
Medical Accuracy
Clinical Relevance
Question Difficulty Question Quality
CPT--3
0.673 [0.469)
0.827 [0.378]
0.588 [0.492]
0.841 [0.366]
Gemini 2.0 Flash
0.717
00450
0.565 [0.496
0.815 00.388
0.774 00.36
Claude-4-Sonnet
0.711
0.453
0.749
0.434
0.576
0.494
0.764 0.425
GPT-5
0.533
0.499
0.610 [0.488]
0.597
00.490
0.476 (0.499)
and agreement results per specialty for each metric. All in all, our expert quality evaluations indicate that the dataset is of high quality with fair levels of agreement across random sample of our test set.
Table 4: Benchmark accuracies, model sizes, and training dataset size for all evaluated LLMs.
Type Model Category Model
Model Training
Overall Accuracy
Unkrownn
Unknown
0.RM
gemini-2.0 2.0llall
Unknown
Unknowa
0.654
Proprietary Geoersl-pprroeee
Unknown
Unkuowa
gut-5
Uniuown
Uaknowa
0.764
gpt-o3
Unknown
Unkcnwnn
0.765
8B parsuneters
urilionn
Geoeral-purrpoee
qwen-pius
1lamn-3.3322 3-50--insttre
70B poramcterss
SB parameters
15 trillion
trillion
0.618
0.547
Domu-3 1-80-instruel
8B parsneters
triliion
Open-sonze
Gouar-e-1-8 1-1777
parzuneer
Irillion
0.498
Arabic-centzic
allazn 7b-instruct
7B parametets
5.2 trillion
0.147
Shil-command----anntiee 222225
7D parmmeters
Unkcoww
0.081
medgenmma-10 spollo- 7b
permneters
Irillion
0.390
Medical
me 8b
SB parameters
7B parametets
15T
Unkaown
0.318
0.238
bimedix--h
paremeterrs
4.3 LLM-AS-A-JUDGE ASSESSMENT
The results of our LLM-as-a-judee experiments are summarized in Table 3 and Table D1 in Appendix[Dacrmm all four evaluation metrics. Our results show moderate agreement among dilferent LLMs and comparable results with the expert evaluation scores. To better under- stand the results in comparison with expert evaluations, we provide a detailed breakdown of the LLMdas-a-judge results in Appendix D
4.4 BENCHMARKING SOTA LLMs
We report all benchmarking results in Table 4 Our results show that general-purpose and reasoning-optimized models consistently outperform specialized smaller open-source models on MedAraBench. GPT-o3 and GPT-5 both demonstrate top overall accuracy (0.765 and 0.764, respectively). while models like med42-8h 8b and apollo-7b achieve the lowest scores (0.318 and 0.238, respectively). Tables E1 and E2 in Appendix showcase a more detailed breakdown of the accuracy per specialty and level for each of the 15 evaluated models.

---

Published as a conference paper at ICLR 2026
4.5 FEW-SHOT LEARNING AND QLORA
The results of our few-shot learning and QLoRa protocols on 1a-ma-3.--8B-nsstte are shown in Table below.
Table 5: Few-shot, and QLoRA fine-tuning performance compared to baseline zero-shot accuracy for 1aana--.--BB-insst
Model
Baseline Accuracy
Few-shot Accuracy QLoRA Accuracy
llama-3. 1-0e--3.---b-instrucc
0.170
0.191
0.320
The few-shot and QLoRA fine-tuning experiments demonstrated substantial improvements over the baseline zero-shot performance. Few-shot learning provided a modest gain of 12.4% (from 0.170 0.191), while QLoRA fine-tuning, boosted accuracy by 88.2% 0.320 nearly doubling the model's performance.
DISCUSSION
Overall, in this study, we present MedAraBench, a new 25k question dataset consisting of both training and test sets. We report performance bascline results for SOTA LLMs and highlight critical differences in model capabilitics under zero-shot scttings. By exposing gaps in Arabic medical understanding, MedAra offers useful insights for the development of more inclusive, multilingual, and domman--ppecialized language models.
The expert quality assessment and LLM-as-a-judde experiments provide valuable insights into the quality of our data and the plausibility of using LLMs to evaluate the quality of medical datasets. Namely, the expert quality evaluation yields average scores ranging from 0.653 0.767 across all 4 evaluation metrics, with percent agreements and Cohen's Kappa scores ranging from 0.656 0.820 and 0.152 0.555, respectively, indicating slight to fair levels of agreement across all metrics. The average evaluation metric scores indicate moderate to high-quality data and fair agreement across annotators, but they indicate the need for the curation of more benchmark datasets of higher quality and clinical relevance to properly assess the readiness of LLMs for clinical deployment. Generally, we see weak to moderate alignment between LLM-as-a-judge evaluation and expert evaluation of the quality of our dataset on all metrics, except for question difficulty. Specifically, GPT- o3 has the highest Pearson Correlation scores with both experts A and B across Medical Accuracy (0.577 and 0.505, respectively). Clinical Relevance (0.252 and 0.377, respectively), and Question Quality (0.407 and 0.336, respectively). As for Question Difficulty, we see weak to no aligument across all models, with Gemini-2.06 Flasshhh scoring highest with Expert scoring higher with Expert B (0.039). Our results indicate that LLMs are yet to be reliable for proper evaluation of Arabic medical data, and highlight the need for further training to better align with expert. human standards.
A (0.019) and
Our benchmark evaluation results coincide with prior rescarch on LLM benchmarking in the medical domain, whereas proprictary models typically outperform open-access models in structured tasks such as multiple-choice QA. This was previously demonstrated by Chen et. al. (2025) and Alonso et al. (2024h, who demonnttraed superior accuracy performance by proprietary models relative to open-source models in medical QA tasks across multiple languages. This was further shown by Abu Daoud et al. (2025). who demonstrated supe- rior performance by proprietary models such as Gemini 1.5 Pro, Claude 3.5 Sonnet, and GPT-4 in Arabic medical MCQ. Our results reinforce those findings, with all proprictary models performing at significantly higher or similar accuracy scores to open-source models. Namely, the proprietary models GPT--3 and GPT-5 achieved the highest overall accuracies on MenAraBennc 70.765 and 0.764, respectively), with Claude-Sonnett also performing strongly (0.694). By contrast, even the largest open-source general-purpose models such as 1itma--.3-70b-instrru and qwen-plus only reached 0.547 and 0.618, while domain-focused Arabic-centric and medical models, such as allam-7b 7b-instruct (0.447), fanar-c-1-8.70 (0.498), and medgemma-4bb-tt. (0.390), remained below 0.5 accuracy. While training dataset size and
8

---

Published as a conference paper at ICLR 2026
model scale correlate with performance (e.g., larger models like 1amma--.3-7b--insrrre and deepseek-chat- chat-v3-022- outperformed smaller Arabic-centric or medical-only models), this was not sufficient for open-source models to match proprietary LLMs. We also observe that general-purper models fine-tuned for reasoning consistently outperform Arabic-focused or medical-specifcc models, underscoring the importance of both architectural and data scale, as well as transfer learning capabilities honed in proprietary labs.
The superior performance by proprietary models possibly due to the larger training cor- pora, stronger pretraining on structured datasets, more extensive instruction tuning, and specialized reinforcement learning pipelines that proprictary models undergo relative their open-source counterparts. Additionally, our results show significantly higher performance by reasoning models (gpt-5 and gpt-o3) relative to other models, showing the promise and importance of incorporating reasoning and explainability into medical NLP as whole and Arabic medical NLP specifically.
The few-shot learning and QLoRA experiments in Table 5 above revealed substantial im- provements over the baseline zero-shot performance for Llama-3.1 1-8-.-1---sst,e,e, though with notable differences in elfectiveness between the two approaches. Few-shot learning provided a modest gain of 12.4% (from 0.170 to 0.191), suggesting that in-context examples helped the model better understand the medical question format and reasoning patterns. Addition- ally, QLoRA fine-tuning, boosted accuracy by 88.2% to 0.320, nearly doubling the model's performance. The dramatic superiority of QLoRA fine-tuning highlights that parameter- efficient adaptation using domain-specifie medical data is substantially more effective than in-context learning alone for adapting general-purpont models to specialized medical QA tasks in Arabic. These results emphasize the importance of targeted training protocols over prompt engineering for achieving competent performance in specialized medical language understanding tasks.
We further evaluate the evolution of models across generations by comparing contempo- rary and legacy model performance on the MedArabiQ benchmark (Abu Daoud et al. 2025). Our results, presented in Table G1 in Appendix G below, show that all contempo- rary models outperform legacy models on the MCQ task. Additionally, this analysis allows us to compare the merit of both the AedAra and MedAraBench benchmarks, with 1emmini-2.0-flsstt gpt-4.1, gpt-5, gpt-o3, and qwen-plus performing better on the MedAra- biQ benchmark, while saaude-soneet-4222222
and 3.0--i-s----tt perform better on Med This indicates that the MedAraBence benchmark is more challenging overall, highlighting its value in advancing medical Arabic NLP.
However, despite the significant improvement across generations, the highest performing model achicved an accuracy score of 0.765, which does not match expert-level performance and indicates clear headroom for improvement before being ready for deployment in clinical settings. Furthermore, this allows us to raise important questions about what exactly LLMs are learning. High accuracy does not necessarily indicate deep understanding or clinical rea- soning. Instead, models may be leveraging statistical associations and lexical patterns to eliminate implausible answers. For example, frequent exposure to certain discase-treatment pairs during pretraining may allow models to make educated guesses without reasoning through symptom progression or differential diagnosis. This distinction is crucial, particu- larly in high-stakes applications such as medical education or decision support. Future work should consider evaluating not just answer correctness, but also the rationale behind model choices, possibly through explanation based tasks clinician scoring of model justifications.
6 LIMITATIONS AND FUTURE WORK
While our study provides a comprehensive evaluation of LLMs on Arabic medical MCQs and represents a substantial advancement in benchmarking capabilities, several limitations exist. First, the dataset is limited in its capability to evaluate LLMs on classification tasks only due to the nature of the MCQ dataset, preventing the evaluation of LLMs in generative tasks. Additionally, although the data source was not available in a structured digital format and required extensive digitization and cleaning efforts, we cannot certainly rule out contamination. Furthermore, our data assumes fluency in Modern Standard Arabic, which,
9

---

Published as a conference paper at ICLR 2026
despite being common in formal settings, may not fully align with linguistic realities. This can affect the generalizability of MedAraBench to learners or practitioners accustomed to dialectal or mixed- language instruction.
Another limitation emerges from our expert quality evaluation experiments. Although our dataset was reviewed by two expert clinicians, observed occasional inconsistencies across their assessments. We acknowledge that expert disagrecment and inherent subjectivity are common in clinical judgment, but recognize the need for broader consensus in future vali- dation efforts. Furt hermore, our data is largely skewed toward easier difficulty levels (Y1 at 56% and Y2 at 22.89%) relative to more advanced levels (Y3 at 12.04%, Y4 at 3.38%, and Y5 at 5.14%. This distribution is due to the availability of source materials collected from regional medical repositories, which contained disproportionately more early-year content. While this evaluation limits the benchmark's capacity to assess advanced clinical reasoning, is representative of the present educational resource landscape in Arabie, and provides value by establishing baseline performance for Arabie medical understanding across the present difficulty levels and specialties, revealing that even basic medical reasoning remains challenging for current LLMs. Morcover, our data is text-only in its format, limiting its ap- plicability to accommodating image- based reasoning required in specialties such as radiology and dermatology, and warranting expansions to other data modalities in future work.
In this study, we primarily focused on zero-shot evaluation of model performance, providing assossment without further adaptation. While this is an important evaluation framework, future work could explore the impact of few-shot and chain-of- tfthouhht prompting strategies, as well as fine-tuning as an opportunity to improve model performance. Furthermore, future work could warrant the incorporation of dialectal data to enhance model adaptability across diverse Arabic clinical settings. Another important area of futuure work is investigate the use of Arabic- based lecture notes to advance medical Arabic NLP beyond MCQs.
7 CONCLUSIONSS
To conchude, we introduced a large-scale Arabic medical benchmark designed evaluate the zero-shot performance of LLMs on curated MCQs. Covering 19 medical specialties and span- ning five difficulty levels, MedAraBench provides a comprehensive and fine-grained lens for assessing Arabic medical reasoning in LLMs. Our benchmark provides developing benchmarks the Arabic language and exposes limitations in the performance of current LLMs in low-resource language tasks and the need for robust multilingual train- ing strategies. Future work should explore fine-tuning strategies and the curation of larger and higher-quality datascts tailored to Arabic medical contexts. We release MedAraBench in hopes of supporting downstream clinical applications, and we hope that it serves as a catalyst for continued research at the intersection of Arabic NLP and medical AI.
advancement for
ETHICS STATEMENT
The authors disclose the use of generative AI vools to assist with LaTeX code cleanup and formatting only, with all content, analyses, and conclusions authored and verified by the researchers involved in this project.
REPRODUCIBILITY STATEMENT
The MedAraBench dataset available at http:://gthhah...mmmyaaaaaaaaaa
ACKNOWLEDGEMENTS
This work was supported by the Meem Foundation, the NYUAD Center for Artificial In- telligence and Robotics, funded by Tamkeen under the NYUAD Research Institute Award CG010, the Center for Cyber Security (CCS), funded by Tamkeen under the NYUAD Re- search Institute Award G1104, and the Center for Interdisciplinary Data Science & AI (CIDSAI), funded by Tamkeen under the NYUAD Research Institute Award CG016. The research was carried out on the High Performance Computing resources at New York Uni- versity Abu Dhabi (Jubail).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Umar Abbas. Fanar: An arabic-centric multimodal generative ai platform. arXiv preprint arXio:2501.1394 2025.
Mariam Essam Abdelaziz, Mohanad A. Deif, Shabbab Ali Algamdi, and Rania Elgohary. A benchmark arabic dataset for arabic question classification using aafaq framework. Scientific Data, 2025.
Mouath Abu Daoud, Chaimae Abou Zahir, Leen Kharouf, Walid AlEisawi, Nizar Habash, and Farah Shamout. Medarabiq: Benchmarking large language models on arabic medical tasks. PMLR, 2025.
Mohammed Al-Rukban. Guidelines for the construction of multiple choice questions tests. Journal of Family Community Medicine, 2006. URL httpss//pmcc ncbi nlm. nih. nov.
articlee/PPCC4400060
Ashwag Alasmari, Sarah Alhumoud, and Waad Alshammari. Aramed: Arabic medical question answering using pretrained transformer language models. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT), pp. 50-56, 2024.
Yazeed AlMoaiad. Challenges in natural arabic language processing. Edelweiss Applied Science and Technology, 2024.
Yazeod Alnumay. Command r7b arabic: A small, enterprise focused, multilingual, and culturally aware arabic Ilm. arXiv preprint arXiv:2503. 14609v1, 2025.
Inigo Alonso, Maite Oronoz, and Rodrigo Agerri. Medexpqa: Multilingual benchmarking of large language models for medical question answering. Artificial Intelligence in Medicine, 102988,,,, September 2024. ISSN 0993-3657, doi: 10.1016/1 att.:/.aatmm..224.....
URL
http://áx. http:://xxxdoi..rgg.... owo.1..rg11... 1016/J artmed. 2024. 102938
Khatra Amar. Arabic morphological productivity in the translation of medical terminology.
PhD Thesis UNIVERSITI SAINS MALAYSIA, 2022.
Anthropic.
Claude opus 4 & claude somnet 4
system
card.
https:/ http://wwwwwttopc.oooo haap://ooooomoddd-ssssteeemee 4-system-card, 2025.
Yasser Mohamed Attia. The challenges of achieving dynamic equivalence in the arabic translation of medical terminology for heart and brain diseases: Difficulties and recom- mendations. Journal of Faculty of Arts, Port Said University, 30(30):14-31, 2024.
M Saiful Bari. Allam: Large language models for arabic and english. arXie preprint arXiv:2407. 159900 2024.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, An- nie Chen, Kathleen Creel, Jarod Quiney Davis, Dora Demszky, Chris Donahue, Moussa Dounbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei- Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitrion,, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshay Santhanam, Andy Shih,
11

---

Published as a conference paper at ICLR 2026
Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramer, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2022. URL https://arxiv org/ abs/ abs//108.... 07258
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Danicl M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https::/arxiv.ot hwtgg/bb//005..... 2005 14165
Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions, 2025. URL https:
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Hen- ryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanu- malayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Frica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL bttps://aaxiv..oom ab//aabb//22222. 02311
Clement Christophe. Med42-v2: A suite of clinical llms. arXie preprint arXiv:2408.061.2 2024.
William Gemmell Cochran. Sampling techniques. john wiley 8 sons, 1977.
DeepSeck-AI. Deepseck-v3 technical report. arXiv preprint arXiv:2412.19459 2024.
Ali Farghali and Khaled Shaalan. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 2009.
Google Cloud. Gemini 2.0 flash http://clloudgoogle..ooo.. wwwteeeeeeeeeeeeeeeeeeeeeee http:///woooeeeeeeeee///eereaaaa
generative ai on vertex / May
2025.
A. Grattafiori et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024a.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and Akhil Mathur et al. The llama 3 herd of models. arXio preprint arXiv:2407.21788 2024b.
Nizar Habash. Introduction to arabic natural language processing. ACL Tutorial, 2005.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv http://rrr//abb//..00033
Pedram Hosseini. benchmark for long-form form medical question answering. arXiv preprint, 2024.
12

---

Published as a conference paper at ICLR 2026
Shadi Iskander, Nachshon Cohen, and Zohar Karnin. Quality matters: Evaluating synthetic data for tool-using llms. orXiv preprint arXiv:2409.1663 16341, 2024. URL hhttps://arxiv. org/abs/2409 16341
Luyi Jiang, Jiayuan Chen, Lu Lu, Xinwei Peng, Lihao Liu, Junjun He, and Jie Xu. Bench- marking chinese medical Ilms: A medbench- based analysis of performance gaps and hier- archical optimization strategies, 2025. URL htttps://arxie wwteeiivvorr///e org/
07306
Di Jin, Fileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams, 2020. URL httpss///axxiv. www.:/rgg//bbbbaeeeee. 1000/
Di Jin, Yansong Pan, Tao Ouyang, and Pascale Fung. What disease does this patient have? large-scale open domain question answering dataset from medical exams. arXiv preprint ar Xix::2009.1308, 2021. URL https: wttp://.wrrgg////////////////// 13081 13081 Submitted to AAAI 2021.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2567-2577, 2019.
Praveen K. Kanithi, Clément Christophe, Marco A. F. Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnic Rajan, and Shadab Khan. Medic: Towards comprehensive framework for evaluating Ilms in clinical applications. Xixie preprind arXiv:2409. 07314, 2024.
Tobi Olatunji, Charles Nimo, Abraham Owodunni, Tassallah Abdullahi, Emnanuel Ayo- dele, Mardhiyah Sanni, and Chinemelu Aka. Afrimed-qa: A pan-african, multi-specially, medical question-answering benchmark dataset. arXiv preprint arXiv:2411.166/ 2024.
URL htttt::tt//////t/// http://aaaaa....ogat/////111111.
OpenAL. Gpt-4 technical report.
April 2025a.
OpenAl. GPT-5 system card. http:///dchhoggeeni.c.oo////a/as 2025b.
OpenAl. Openai o3 and o4-mini system card. httpps//cchaaappnaacccc//////// system-card. pdf, 2025c.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large- scale multi-subject multi-choice dataset for medical domain question answering. In Con- ference health, inference, and learning, pp. 248 260. PMLR, 2022.
Sara Pieri, Sahal Shaji Mullappilly, Fahad Shahbaz Khan, Rao Muhammna Anwer, Salman Khan, Timothy Baldwin, and Hisham Cholakkal. Bimedix: Bilingual medical mixture of experts llm. arXiv preprint Xxx:2200211255, 2024.
Rick Rejeleene, Xiaowei Xu, and Johm Talburt. Towards trustable language models: Inves- tigating information quality large language models. arXie preprint arXiv:2401.13086 2024. URT. htttps//arxxxi org/ abs/ 2201. 13086
Andrew Sellergren. Medgemma technical report. arXiv preprint orXiv:2507.0520. 2025.
Shivalika Singh. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025.
Felix Stahlberg, Neural machine translation: A review and survey, 2020. URL https:
7/axxiv.. www. 11tpp///991.02477
Bin Wang, Chengwei Wei, Zhengyuan Liu, Geyu Lin, and Nancy Chen. Resilience of large language models for noisy instructions. EMNLP, 2024.
13

---

Published as a conference paper at ICLR 2026
Cunxiang Wang. Pai Liu, and Yue Zhang. Can generative pre-trained language models serve as knowledge bases for based-book qa?, 2021. URL https://arxiv.. org/abs/ F156.
Xidong Wang, Apollo: A light multilingual medical Ilm towards democratizing med- ical ai to 6b people. arXiv preprinl arx:::2403.0033 03.00340, 2024.
An Yang and et al. Qwen2.5 technical report. ar preprint arXi::2412.15111 2024. URL httpps:/arxxvvorrg/ wtw.v//.rrrrbb//////2/ ab2// 15115
Han Yang, Mingchen Li, Huixue Zhou, Yongkang Xiao, Qian Fang, and Rui Zhang. Large language model synergy for ensemble learning in medical question answering: Design and evaluation study. J Med Internet Res, 2025.
14

---

Published as a conference paper at ICLR 2026
A DATA ANALYSIS
A.1 DATA OVERVIEW
An overview of the MedAraBench dataset. according to the distribution of questions per difficulty level and specialty is shown in Figure Al below.
Questions per Difficulty Level
20000
100
Number of Questions
17500
15000
Cumulative
12500
10000
7500
5000
2500
Y3
Y4
Y5
Difficulty Level
Questions Specialty
Numbor Questions
Specialty
(b)
Figure A1: Overview of dataset according to difficulty level and specialties.
In terms of specialty, the largest subsets of the dataset falls under Anatomy (6100 questions 24.61% of the datasct) and Physiology (3302 13.32 (2%,,, while the smallest subset falls under Pathology (56 - 0.23%). A more detailed brcakdown of the distribution of questions according specialty can be shown Table A1 and Figure A1 (b) above.
On the other hand, in terms of difficulty level, the largest subset of the dataset falls under Y1 (15095 questions- 60.80% of the dataset) while the smallest subset falls under Y4 (1033 4.17%). A more detailed breakdown of the distribution of questions according to level can be shown in Table A2 and Figure A1 (a) above. Additionally, Figure A2 shows the the composition of the five levels across the 16 specialties included in MedAraBenchh.
15

---

Published as a conference paper at ICLR 2026
Table Al: Distribution of questions per medical specialty.
Medical Specialty
Number of Questions Percentage
Anatomy
6100
24.61%
Anesthesia
405
1.63%
Biochemistry
1387
5.59%
Cell and Molecular Biology
3155
12.73%
Chemistry and Physics
3894
15.71%
Embryology
184
0.74%
Emergency Medicine
120
0.48%
Internal Medicine
762
3.07%
Microbiology
750
3.03%
Ophthalmology
1318
5.32%
Pathology
56
0.23%
Pharmacology
329
1.33%
Physiology
3302
13.32%
Statistics
1795
7.24%
Surgery
357
1.44%
Table A2: Distribution of questions per difficulty level.
Difficulty Level Number of Questions Percentage
Y1
15095
60.89%
Y2
4954
19.98%
Y3
2313
9.33%
Y4
1033
4.17%
Y5
1396
5.63%
Lesdl Cumpostion witli Each Specially
3008
2009
300
Naber Serwer
Figure A2: Level composition of the 5 different difficulty levels (Y1 Y5) within each specialty in the MedAraBench dataset.
16

---

Published as a conference paper at ICLR 2026
A.2 TOKEN LENGTH DISTRIBUTION
There are 24,791 questions in our dataset. The questions are moderate in length, with a total average length of 37.86 characters. The answers vary in formal, whereas 18554 questions have lour answer choices (A, B, C, and D) and 6228 questions have five answer choices (A, B, C, D, and E). The average
length across the datasets is 171.09 characters. Figure A3 below gives a detailed breakdown of the distributions of text lengths of the entire dataset.
Distribution Text Lengths
Option
Option
11:0
509-
250
1:6
200
214
301
169
109
Option E Length
Option Eangth
Option R Length
106
Figure A3: Distribution of text lengths in the MedAraBench dataset: (a) Distribution of question length; (b) Distribution of answer length; (c) Distribution of Option A length; (d) Distribution of Option B length; (e) Distribution of Option C length; (f) Distribution of Option length; and (g) Distribution of Option E length.
17

---

Published as a conference paper at ICLR 2026
B BENCHMARKING AND FEW-SHOT LEARNING PROMPTS
B.1 BENCHMARKING PROMPT
Listing 1: Benchmarking prompt used for medical MCQ evaluation
"You are an expert medical virtual assistant. Please provide the correct answer letter (A, B, C, D) for the following Arabic medical multiple- question. Question:
{question_texx text in inaxt_ii_Arabici
Options: B:{option_B_ii C: Arabic}
D:(option_D_in Arabic) Answer:"
B.2 FEW-SHOT LEARNING PROMPT
Listing Few-shot prompt for medical MCQ evaluation
"You are an expert medical assistant. You will be provided with few Arabic medical sample questions and their correct answers, then a new question that you must answer.
Sample Questions: ceample_questions
Now, the new question:
Question: {question)
Options:
{options_tens
(Please answer with the letter choice (alloved_ ONLY): :"
The sample questions are as follows:
الفاصرة من المرججال المقن الشمل النولل
المرن :t الجاابي -ي الألبب بي البفي البااات
إباي الإ
ماعيا عميع بي vا في الخدمي المص تنص فياا النولل
الد الطاير فاترن -يج المدررة ال5بب راسي in السعد بدسل بي المغديةة الغفررة الأسيي البيل من ينياا البلااتت
تابعة العحاة بصب مs الممم uن الأنبية الاية قايعة وز رحني بكوة ;s الملريي الشبيي الهاني المريان مققة المقد
الأنسية الرسة
بلاييية الإ
الاركلاية الدم سكك تقمص أساب من الوال:
سor be 1f الكية Yill 'ي للووننة الشساية بن اطفي اليم سعك قصي الياراتة
الإاااة
18

---

Published as a conference paper at ICLR 2026
C EXPERT QUALITY ASSESSMENT DETAILS
In this appendix, we provide a detailed breakdown the expert evaluation results introduced in Section 2 While the main text summarizes overall averages and agroement levels across all specialties, here we report per-specialty results to give a more fine-grained view of model performance and annotator consisteney.
present the distribution of annotalor scores across specialties for each of the four evaluation metrics (Medical Accuracy, Clinical Relevance, Question Difficulty, and Question Quality). The corresponding Tables C1 C4 report the average scores, number of evaluated questions, percentage agreement, and Cohen's Kappa values for each specialty, Together, these results highlight the variability across domains and provide context for interpreting the aggregate quality metrics shown in the main text.
Figures
Accuracy by Specialty
ACCURIS
0.6
Becherarry
Gal and
More degg opea speranniig
Fot dege
marmisc
Specially
Figure C1: Overall annotator accuracy scores per specialty.
Overal Relevance by Specialty (Mean SD)
1.4
12
-0.2
Weaing
200p
Spec diy
Figure C2: Overall annotator relevance scores per specialty.
19

---

Published as a conference paper at ICLR 2026
Overall Difficulty by Sperialty (Mean 5D)
1.0
123
Dffiadts
0.0
-0.25
Payi
Specialty
Figure C3: Overall annotator difficulty scores per specialty.
Overall Quality by Specislty 1Mean 501
0.6
De:30
Specially
Figure C4: Overall annotator quality scores per specialty,
Table Cl: Annotator accuracy scores per specialty.
Specialty
Average [standard deviation)
Number of Questions Percent Agreement Cohen's Kappa
Anatomy
[0.464
Auestbesia
0.600
01.503
10
0.600
0.167
Biochetissty
0.625
0.489
0.750
0.471
Cell and Molecular Biology
0.613
00.889
0.774
0.529
Chemistry and Physies
0.708
01.457
48
0.917
0.798
Embryology
01.000
Emergeney Merlicine
1.000
0..00
1.000
Internal. Medicine
0.917 0.280
0.944
0.640
Microbiology
0.100
00.155
0.733
0.135
Opithhlmology
0.820
11.388
25
0
0.603
Patlaology
01.18
Phartnnoology
1.000
0.000
1,000
Physiology
0.895
0.30S
0.884
0.380
01.226
20

---

Published as a conference paper at ICLR 2026
Table C2: Annotator relevance scores specialty.
Specialty
Average jstandard deviation)
Number of Questions Percent Agreement Cohen's Kappa
Austony
[0:179
0.225
Aneshhesia
1.000
00.000
10
1.000
Biochemistry
0.667
00.191
0.708
0.400
Cell and Molecular Biology
0.726
00.501
0.435
0.083
Chemistry and Plassis
0.333
01.170
48
0.688
0.286
Embryology
11.000
Emergenccy Medicine
1.000
01.000
1.0000
Internal Medicine
1.000
[1 280
0.833
0.000
Microbinlogy
00.407
Ophatanlmooooy
1.000 01.303
25
0
0.000
Pathologg
163
Phartncooooy
1.000) 01.000)
1.0000
Physiology
0.930 0.275
0.881
0.221
Statistics
01.504
0.021
Table C3: Annotator difficulty scores per specialty.
Specialty
Average (standard deviation]
Number Questions
Agrecment
Kappa
Anatomy
0.816 [0.464
103
0.650
0.240
Anesthesia
01.510'
0.074
Biochemistay
0.708 00.176
24
0.750
0.412
Call and Molecular Biology
0.661
00.663
0.710
0.320
Chemistey and Plysies
0.958
00:108
0.625
0.027
Embryology
1.000
11.000
1.0000
Emergencc Medicine
0.377
Internal Medicine
0.722 0.487
0.611
0.182
Microbiology
0.933 0.3015
0
41.098
Oplithumoloogy
01.18
Pattaology
0.500 01.335
1.0000
Pharnacology
0.518
Physiology
0.372
0.494
0.651
0.281
Statisties
0.316 00.00
0.368
-0.009
Table C4: Annotator quality scores per specialty.
Specialty
Average [standard deviation]
Number of Questions
Percent Agreement
Coben's Kappa
Anatomy
0.573 00.451
103
0.573
0.044
Anesthesia
0.000
01.889
10
0.700
0.348
Biochemistry
01.449
Cell and Molecular Blology
0.548
00.488
0.484
40.120
Chemistry and Physies
0.708
01.55
0.792
0.496
Embeyology
1.000 0..000
1.000
Emergency Medicine
1,000
0.000
1.000
Internal Medicine
01,280
Microbiology
0.533 0..90
0.533
0.037
Opirhulmooooy
0.780
00.370
0,840
0.432
Pathology
0.750 0..11
Pharmsoology
1,000 0.000
1,000
Physiology
0.884
01.202
0.850
0.178
Staristies
0.895 00.273
0.842
-0.075
21

---

Published as a conference paper at ICLR 2026
D LLM-AS-A-JUDGE ASSESSMENT
The models were provided with the full test set of MCQ (stem, options, and correct answer) and instructed to return only valid JSON output. The exact prompt
Listing Prompt provided to LLMs
You are medical education expert. Evaluate the following multiple-choie question (MCQ) a binary scale (0 1) for each of the following metrics:
1. Medical Accuracy (1=high, 0=1ow)
2. Clinical Relevance (1=high, 0=low)
3. Question Difficulty (1=high, 0=low)
4. Question Quality (1=high, 0=low) Important: Return ONLY valid JSON. No explanations, no markdown no text.
The response must be exactly like this:
"medical "medical_accurac"" <0-1>,
"clinical remariccereelevaceeee <0-1>,
<0-1>,
"question_quality"" <0-1>
Question stem: [row[ '
Options: {options_tene
Correct answer:
Answer']]
This setup ensured that LLM outputs standardized and machine-readable. By ag- gregating scores across thousands of test questions, we obtained descriptive statistics and model-wise distributions that enabled more fine-grained analysis than binary human rat- ings alone.
To examine agreement between models and expert evaluators, Pearson correlation coeffi- cients were calculated on per-question basis. The results of this analysis are shown in Table D1 below. Our Pearson Correlation results show GPT-o3 was the best performing model in terms of alignment with expert evaluations. As such, we opted to use GPT-o3 to complete L.M--as-aJuddo evaluations over the entire training set of ModAraBench,, allowing us to have full evaluation of the entire dataset. Additionally, GPT-o3 scores on the training and test sets and found no significant difference on all 4 eval- uation metrics, indicating consistent quality across splits. The results of this experiment are shown in table D2 below.
ran t-tests to compare
Table D1: Pearson correlation coefficients between model and expert ratings.
Expert A
Expert
Model
CPT-c3
Cluido-4-Sonet
Ceniis221-Fhch
GPT-5
GPT-63
Chandlo4-SSomet
Cemmuss-20-Phelll
CPT-5
Medical Accutacy
0.577
0.055
0.023
0.013
0.505
0.053
0.05
0.068
Clinical Relevanoe
0.252
0.165
0.176
0.071
0.377
0.13)
0.131
0.104
Question Quality
0.407
0.336
Diliculty
0.019
22

---

Published as a conference paper at ICLR 2026
Table D2: GPT-03 LLM-as-a-judge scores on training and test sets (mean [std]) with t-test p-values.
Dataset
Medical Accuracy Clinical Relevance Question Quality Question Difficulty
Training set
0.638 [0.48]
0.821 0.383
0.839 00.367]
0.561 (0.96)
Test set
0.673 [0.469]
0.827 [0.378]
0.588 [0.492]
0.841 [0.366]
P-value
0.0001
0.0362
0.0001
0.0001
23

---

Published as a conference paper at ICLR 2026
E PERFORMANCE PER SPECIALTY AND LEVEL
To complement the overall evaluation, we analyze model accuracy across different medical specialties and difficulty levels. This breakdown highlights domain-specifi strengths and weaknesses, as well as how performance varies across the five curriculum levels (Y1-Y5). We provide detailed breakdown of model accuracies per specialty and levels in Tables and below.
In terms of specialty performance, our analysis reveals clear superior performance across models, with GPT-5 and GPT--3 dominating most domains while apollo-7b and med42-8b consistently underperformed. Notably, the smaller medgemma-4b-it excelled in Pathology (0.574), demonstrating that specialized training can somctimes overcome scale limitations.
As for difficulty level performance, all models exhibited an accuracy drop from Y1 Y3 before recovering in Y4-Y5. GPT-5 maintained the highest performance across four of five levels, while weaker models showed steeper declines al intermediate levels. This consistent. pattern indicates that Y3 questions, which likely require more complex clinical reasoning, present the greatest challenge across all model architectures.
The significant performance variation between difficulty levels highlights critical limitations in current LLMs for medical applications. The average 15-20% performance drop from Y1 to Y3 suggests that while models excel at factual recall in early currieulum years, they struggle with the integrative climical reasoning required at intermediate levels. This difficulty scaling effect was most pronounced for smaller models, indicating that scale and sophisticated training are crucial for handling complex medical reasoning tasks. These findings suggest. I.I.ss may be better suited for foundational knowledge and specialized domains than for complex clinical decision- makinn..
24

---

Table El: Overall Model Accuracy per Specialty,
Published 8s conference paper at ICLR 2026
Specialty
Emergencc Medicine
Internal Medicine
Model
Amatomy
Biochenistry
Embeyylog
Microbiology
Ophntalanoooy
Pathology
Pharmacclogy
Physislogy
Statistics
Surgery
allaen-Tb-iinn Tb-instrues
0.402
0.460
0.438
0.333
0.478
0.351
0.3338
0.320
0.151
0.463
0-409
0.454
0.624
0.430
upollo-7h
0.315
0.195
0.237
0.000
0.435
0.175
0.192
0.178
0.076
0.259
0.106
0.183
0.255
0.100
clni-command--7h-ne
0.303
claulo-enmee-4422222000
0.601
0.741
0.723
0.303
0.739
0.784
0.483
0.691
0:206
0.722
0.727
0.747
0.774
0.703
deepseek-
-rrppeek-llat--300322
0.596
0.583
0.652
0.533
0.565
0,619
0.401
0.568
0.191
0.630
0.967
0.895
0.750
0.502
fanar-o-1-8.77
geminn-22 D-fiish
0.645
0.679
0.808
0.500
0.652
0.701
0.483
0.048
0:193
0.630
0.636
0.702
0.772
0.648
gpt-1.1
0.681
0.588)
0.712
0.333
6522
0.711
0,401
0.653
0. 0.1177
0.722
0.561
0.757
0.794
0.635
gpt-5
ggt-o33
0.797
0.802
0.792
0.500
0.783
018335
0.680
0.772
0.215
0.750
0.864
0.824
0.835
0.727
[omm-33 1-S0-inntruct.
0.335
0.2216
0.332
0.167
0.345
0.175
0.225
0.288
0. 0.116
0.315
0:227
0.349
0.478
0.328
laama-3 3.3-70b-inntruct
mod42-8h
0.304
0.235
0.348
0.167
0.043
0.309
0.185
0.220
0.107
0.380
0.242
0.321
0.475
0.297
medggnnmn--l-i
0.365
0.272
0.380
0.500
0.391
0.247
0.305
0.375
0. 0.126
0.574
0:273
0-424
0.5-49
0.320
qwea- plus
Table E2: Overall Model Accuracy per Difficulty Level
Lansd
choale-conet---3310 despaskeclatt
Xpk-5
80c-13
[eene-.--8b-astmet
Bann--55-76-iiittee:
Y1
0.475
0.311
0.415
0.711
BJ645
0,485
0.567
11639
0.769
0,773
1.366
0.566
0.330
0.416
[.644
V2
0.15)
6:215
0.350
0.561
0.515
0:133
0.515
01835
0.735
0.730
0.331
0.510
0.322
0.357
(1581
Y3
0.307
0.172
0.215
0.545
0.430
0.341
0.544
0.855
0.054
0.002
0.213
0.418
0.244
0
0.20
0.199
0.547

---

Published as a conference paper at ICLR 2026
F ANSWER CHOICE DISTRIBUTION BALANCE
After constructing the dataset, observed small but notable imbalances in answer distri- butions after constructing our dataset.
This could lead to model bias toward any specific answer position (e.g., always selecting "A"). As such, to address this, we analyzed and adjusted the distribution of correct answer choices across all subsets and splits by making minimal targeted adjustments (e.g., reorder- ing options when possible) to bring the correct answer frequencies closer to uniformity. This refinement done independently for the training and test sets across cach answer format group (ABCD, ABCDE, ABCDEF). This adjustment ensures balanced representation of correct answers and helps reduce the likelihood that models learn position-based heuristics.
Figures
visualize the resulting distributions.
Correct Answer Distribution (ABCD)
Split
3500
Text
3000
Number Questions
2500
2000
1500
1000
500
Correct Option
Figure Fl: Answer Choice Distribution for ABCD format (Train/Teen )estt)
Correct Answer Distribution (ABCDE)
1000
Trann
Number Questers
600
200
Option
Figure F2: Answer Choice Distribution for ABCDE format (Train Test))
We further validated this balance using chi-square goodness-of-ftt tests. Chi-square goodness-of-fit tests confirmed no significant deviation from uniformity in ABCD and ABCDE splits (p > 0.97), indicating that the observed answer distributions do not sig- nificantly deviate from uniform distribution. The ABCDEF format excluded from
26

---

Published as a conference paper at ICLR 2026
Correct Answer Distribution (ABCDEF)
2.00
Split
1.75
Text
1.50
Number Questions
1.25
1.00
0.25
0.00
Answer Option
Figure F3: Answer Choice Distribution for ABCDEF format (Train/Tesh]
this analysis the test set due to insufficient sample size (n = 2). All tests support the conclusion that the distributions do not significantly differ from uniformity.
The resulting x² values and p-values were as follows:
ABCD (Train): x² = 0.099, p= = 0.992
ABCD (Test): x2 = 0.212, p = 0.976
ABCDE (Train): X² 0.422, p 0.981
ABCDE (Test): x² 0.226, p 0.994
ABCDEF (Train): x² 0.7144, 0.982
ABCDEF (Test): x² = 6.02, p 0.304
This adjustment ensures balanced representation of correct answers and helps reduce the likelihood that models learn position-based heuristics.
27

---

Published as a conference paper at ICLR 2026
G CONTEMPORARY VS LEGACY MODEL ASSESSMENT
In this section, we evaluate the evolution contemporary models against their legacy coun- terparts by ovaluating them the MedArabiQ Abu Daoud et al. (2025) dataset and com- paring their accuracy scores. Additionally, this analysis allows us to compare model scores on the MedArabiQ and MedAraBench benchmarks, providing valuable insights into the value of each benchmark.
The results shown in Table G1 below. Our experimental results the Arabii benchmark show that all contemporary models outperform legacy models on the MCQ task. Additionally, our results show that gemini-2.0-flase gpt-4.1, gpt-5, gpt-o3, and qwen-plus perform better on the MedArabiQ benchmark, while claude-somnet--22255555 3.3-70b-instruct perform better on ModAraBench. This indicates that the MedAraBench benchmark is more challenging overall, and thus provides larger value for the advancement of medical Arabic NLP. It important to note that legacy models gemini-1.55 1.5-pro and claude- 3.5-somnet were deprecated, which prevented us from evaluating them on MedAraBench to provide a better comparison between benchmarks.
and llama-
Table G1: Legacy vs contemporary model performance on MedArabiQ and MedAraBench
MedArabiQ
MedAraBench
Legacy Model
Accuracy
Contemporary Model
Accuracy
Contemporary Model
Accuracy
claude-sonnet33.5
0.535
sonuee-oonuet4-202255
0.6869
claude-ounet-4222225
0.6937
gemini-L.5-pro
0.575
gemini-2.0-HE f.a
0.7273
gemini-2.0-faas.
0,6539
gpt-4.1
0.8081
gpt-1.1
0.6733
gpt-4
0.535
gpt-5
0.8586
gpt-5
0.7612
gpt-o3
0.8384
gpt-o3
0.7652
Ilama-3.1-80
0.262
Ilama-3.3 3-70-insttret
0.49-19
llama-3.3- 3-mbiiistert
0.5466
gwen-2.5-7b
0.380
qwen-plus
0.6566
qwen-plus
0.6177
28


Reviews:
[{"review_id": "DCOrVbbaTi", "reviewer": "Reviewer_ratc", "summary": "The paper presents a benchmark for medical knowledge in Arabic. The dataset comprises multi-choice questions in 19 medical specialities, extracted and refined manually from exams and lecture notes of medical schools in the Arabic-speaking world, then quality-checked by experts and LLM-as-a-judge scheme. The paper presents benchmarking results for a number of SOTA LLMs.", "questions": "- How is it that Arabic is underrepresented in the medical domain because of its rich morphology and dialectal variation? And why the uneven linguistic landscape calls for medical LLMs? The motivation of this work in Sec.1 should be revised.\n\n- The reference to Appendix A is missing in Sec. 3.1\n\n- MedArabiQ is missing from Table 1\n\n- How were lecture notes converted to meaningful MCQs? This is an important detail that deserves to be discussed.\n\n- The LLMs chosen for benchmarking do not include any Arabic-focused model. Why not? The differences in the benchmarking results could be due to the models' inherent limitation in Arabic itself rather than in medical knowledge, where bigger, proprietary models have an edge, but this angle is not explored.\n\n- Line 331 (Sec 4.3) is truncated.\n\n- How big were the subsets of MCQ? Guessing from Fig. E3. subset ABCDEF has only 2 questions in the test set and 9 question overall, so it might be better not to be considered at all as a separate category.\n\n- In Sec. 5 (Discussion) and Table D1, how can MCQ scores of different benchmarks be at all comparable?\n\n- How did the authors homogenize medical terminology in their dataset given the lack of standardization in the Arabic medical domain?\n\n- In Appendix A, how comes the average question length is just over 8 characters?\n\n- How was the MCQ benchmarking implemented? e.g. using logit ranking or post-processing of model answer for choice character?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a useful resource for Arabic medical understanding, and benchmarking results for a number of SOTA models.", "weaknesses": "Some missing details about the construction of the dataset and the implementation of the benchmarking are mentioned in Questions below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FfPDIwGxcn", "reviewer": "Reviewer_m6wR", "summary": "This paper introduces MedAraBench, a benchmark for evaluating LLMs on Arabic medical MCQ task. The dataset contains about 24k questions across 19 medical domains and 5 difficulty levels, manually digitized from Arabic medical school materials. The authors conduct both clinical expert evaluation and LLM-as-a-judge assessments to measure data quality, finding moderate agreement and generally acceptable accuracy. They then benchmark open-source and proprietary models like GPT-5, Gemini 2.0, and Claude 4, showing that proprietary models outperform open-source ones but still fall short of expert level accuracy. The paper provides a structured resource for testing Arabic medical tasks but is mainly limited to multiple-choice formats and zero-shot evaluations.", "questions": "Please address the above weaknesses. \n- How were the medical specialties determined for the data samples? Was this an automated process or done with the help of clinical experts?\n- Line 197: “No Cueing: options do not provide clues to other answers.” Is it meant to be “Clueing”?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The manual digitization and expert validation of data from non-digital academic sources shows significant effort and ensure the dataset’s authenticity and reliability.\n- The dataset spans 19 medical specialties at various difficulty levels, offering a structured framework that supports fine grained evaluation of LLM performance across various domains of medical knowledge for the Arabic language.", "weaknesses": "- For validating data quality using LLM-as-a-judge, the authors employ GPT-4, Gemini 1.5 Pro, and Claude 3.5 Sonnet. However, there is no justification provided for selecting these specific models, Are they known to outperform others in Arabic understanding? Moreover, the prompt instructs the models to act as medical education expert but does not account for the Arabic language aspect of the task. The capability of these models in Arabic medical understanding needs further evaluation.\n- The literature review and experimental comparisons are not comprehensive. Open-source models like BiMediX [1] have benchmarked Arabic medical tasks and have released translated and verified datasets. Additionally, compare with more medical open-source models like Apollo [2], Med42 [3], Meditron [4]. Proprietary models like Gemini 2.5 Pro [5] and Flash are also missing from the evaluations.\n- If LLM as a judge is an automated framework to evaluate the quality of the proposed dataset. Extending this validation to the training set could help further filter and enhance the quality of the data. Currently, the validation is limited only to the test set.\n\n[1] *Pieri, Sara, et al. \"Bimedix: Bilingual medical mixture of experts llm.\" arXiv preprint arXiv:2402.13253 (2024).*\n\n[2] *Wang, Xidong, et al. \"Apollo: A lightweight multilingual medical LLM towards democratizing medical AI to 6B people.\" arXiv preprint arXiv:2403.03640 (2024).*\n\n[3] *Christophe, Clément, et al. \"Med42-v2: A suite of clinical llms.\" arXiv preprint arXiv:2408.06142 (2024).*\n\n[4] *Chen, Zeming, et al. \"Meditron-70b: Scaling medical pretraining for large language models.\" arXiv preprint arXiv:2311.16079 (2023).*\n\n[5] *Comanici, Gheorghe, et al. \"Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.\" arXiv preprint arXiv:2507.06261 (2025)*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gW5UQZbnvw", "reviewer": "Reviewer_BRmF", "summary": "This paper introduces MedAraBench, a large-scale Arabic medical QA benchmark of 24K multiple-choice questions across 19 specialties and five difficulty levels. The dataset was digitized from Arabic medical exams, manually filtered, and evaluated by human experts and through an “LLM-as-a-judge” framework. Eight LLMs (open-source and proprietary) are benchmarked in zero-shot settings. The goal is to provide a specialty-diverse Arabic medical QA benchmark for evaluating LLMs.", "questions": "Questions:\n\n1. Did you test Arabic or medical-specialized LLMs as potential judges?\n\n2. How is the “Average (Fraction of 5)” metric in Table 3 calculated?\n\n3. How do explain the fact that several models reach perfect (1.0) accuracy in the ABCDEF configuration?\n\n4. What does Table D1 intend to represent, benchmark comparison or model evolution?\n\n5. Could you share the prompts used to evaluate the benchmarked models, including input format, language setup, and answer extraction method?\n\nRemarks:\n\n1. Misplaced or unclear citation (line 099).\nThe citation to the GPT-4 technical report (Achiam et al., 2023) does not logically connect to the preceding sentence. If the authors meant to refer to GPT-4 being evaluated on translated benchmarks, the sentence should be rephrased for clarity.\n\n2. Missing cross-reference (lines 170–171)\nIn Section 3.1, methodological details are discussed without referencing the appropriate subsection (Section 4.1), reducing readability.\n\n3. Lack of explanation for evaluation platform (line 215)\nThe authors mention that expert evaluations were conducted using Qualtrics, yet they do not explain what it is nor provide a footnote or citation.\n\n4. Incomplete sentence (line 331)\n\n5.  Invalid link (line 452): \nThe repository link (https://anony-mous.4open.science/r/medarabench-3BE4/) is inaccessible.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. Significant manual data cleaning and digitization effort, which adds credibility and quality to the dataset.\n\n2. Diverse specialty coverage and structured annotation across difficulty levels, ensuring representativeness within the medical domain.\n\n3. Inclusion of a human expert evaluation component, which is commendable and adds qualitative depth to the study.\n\n4. Contributes to Arabic NLP, a domain with limited existing benchmarks and resources.", "weaknesses": "1. Unjustified selection of evaluator LLMs (Section 3.2.2)\nThe paper provides no justification for the selection of the three LLMs used as evaluators in the LLM-as-a-judge setup. There is no discussion of why these particular models were chosen, nor any rationale for excluding medical or Arabic-specialized LLMs, such as BiMediX (arabic+medical) or Fanar(arabic) or medgemma(medical+multilingual) etc .\n A more rigorous approach would have been to compare multiple candidate evaluators and measure their correlation with human expert scores (e.g., Pearson or Spearman coefficients) to identify which LLM aligns best with human judgment.\n\n2. Missing mention of existing Arabic medical benchmarks (Table 1, lines 058–059). \nThe comparison table omits BiMediX (arXiv:2402.13253), which already provides Arabic translations of MedQA and MedMCQA.\n\n3. Unsupported validation and overstated conclusions (lines 360–362, Table D2)\nThe conclusion that “ the potential of LLMs to be used for data quality evaluation in the medical domain” (lines 362–363) is overstated and empirically unsupported. The only evidence presented is a superficial similarity in average scores between human and LLM evaluations (Table D2). However, this does not constitute proof of agreement or reliability.\nSeveral methodological issues invalidate the comparison:\n- Different evaluation scales: Human experts used a binary high/low rating, while LLMs used a 1–5 Likert scale, making numerical averages non-comparable.\n- Different samples: Humans and LLMs did not evaluate the same subset of data\n- No agreement metrics: No statistical measure of correlation or agreement between human evaluators and the LLM judges (e.g., Pearson, Spearman, or Cohen’s κ) is reported. \n- Moreover, the statement in lines 360–362 that : “While they are not directly comparable due to varying evaluation scales, we note that the results of LLM-as-a-judge and expert quality evaluation are comparable.” is internally contradictory comparability cannot be claimed if the scales and samples differ.\nThe clause “pending further alignment with medical standards” implicitly acknowledges this weakness, but does not substitute for empirical validation.\n\n4. Unjustified use of the Likert 1–5 scale \nThe 1–5 scale is applied without defining intermediate values (2–4) in the prompt (Appendix C), and no rationale is provided for using a 5-point scale instead of a binary one matching the human evaluation. This undermines comparability and interpretability.\n\n5. Absence of medical Arabic LLM baselines (Section 3.3)\nAlthough the work benchmarks several proprietary and open models (GPT-5, Gemini, Claude, etc.), no Arabic or medical LLMs are tested, despite the availability of models like BiMediX (Arabic and medical), medgemma (medical and multilingual) etc.\n\n6. Invalid cross-benchmark and cross-model comparisons and flawed analysis of model progress (discussion lines 384–394; Table D1)\nIn the discussion (lines 384–394), the authors claim to observe “evolution of model performance across generations”.\n This analysis is methodologically invalid, as it compares different models on different benchmarks (MedArabiQ vs. MedAraBench).\n Because neither the models nor the datasets are constant, performance differences cannot be attributed to either factor.\nTable D1 seems intended to show that MedAraBench might provide a more informative or challenging evaluation, but the comparison is not correctly designed, and the caption does not clarify what the table represents.\n The two columns correspond to different model generations, making them not directly comparable.\nThis is a missed opportunity:\n If the authors had evaluated the same models on both MedArabiQ and MedAraBench, they could have shown whether the new benchmark is more challenging and thus more valuable.\n Alternatively, testing different generations of the same model family (e.g., GPT-4 vs GPT-5) on MedAraBench would have allowed a valid analysis of progress over time.\n As it stands, the setup conflates dataset variation with model advancement, so conclusions about “model evolution” are unsupported.\n Both the discussion and Table D1 should be revised: either clarify that the comparison is descriptive or conduct controlled, same-model evaluations.\n\n7. Invalid comparison of models (lines 365–367)\nThe authors conclude that “proprietary models outperform open-source models,” yet the proprietary models are orders of magnitude larger than open ones. Such comparisons are meaningless without controlling for scale.\n\n8. Dataset imbalance (Figure 2a, Table A2)\nOver 56 % (Figure 2a, Table A2) of questions are Year-1 level and only 5 % Year-5, resulting in a dataset dominated by basic-science items. This imbalance likely makes the benchmark less challenging and limits its capacity to assess advanced reasoning.\n\n9. Suspicious perfect accuracies without explanation (Table 3)\nIn Table 3, several models report perfect accuracies (1.0) for the ABCDEF configuration, while scores on other subsets remain between 0.55–0.77.\nThis sudden jump to perfect accuracy across models is highly suspicious and atypical for medical QA tasks.\nNo explanation or investigation is provided. The authors should have clarified whether the ABCDEF subset:\n- contains very few items (inflating accuracy), \n- includes only Year-1 questions (simpler), or \n- whether the addition of letter choices (A–F) helped models guess the correct answer (e.g., positional or formatting cues).\nWithout such clarification, the results appear unreliable and raise concerns about evaluation validity.\n\n10. Limited novelty and under-utilization of the dataset\nWhile the dataset is valuable for Arabic medical NLP, the contribution is incremental rather than conceptual, there is no new evaluation framework or modeling insight beyond prior work (MedArabiQ). The paper advertises ~24K MCQs, yet only ~4.9K test items are actually used in experiments; the ~20K training split is never explored (no fine-tuning, few-shot, or human/LLM evaluation on train). As a result, the empirical scope is limited to the test set, leaving the benchmark largely under-utilized. The most tangible contribution remains the digitization and manual curation of Arabic medical MCQs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)