ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models (2024)

Abstract: The rapid advancements in Large Language Models (LLMs) have led to significant improvements in various natural language processing tasks. However, the evaluation of LLMs’ legal knowledge, particularly in nonEnglish languages such as Arabic, remains under-explored. To address this gap, we introduce ArabLegalEval, a multitask benchmark dataset for assessing the Arabic legal knowledge of LLMs. Inspired by the MMLU and LegalBench datasets, ArabLegalEval consists of multiple tasks sourced from Saudi legal documents and synthesized questions. In this work, we aim to analyze the capabilities required to solve legal problems in Arabic and benchmark the performance of state-ofthe-art LLMs. We explore the impact of incontext learning and investigate various evaluation methods. Additionally, we explore workflows for generating questions with automatic validation to enhance the dataset’s quality. We benchmark multilingual and Arabic-centric LLMs, such as GPT-4 and Jais, respectively. We also share our methodology for creating the dataset and validation, which can be generalized to other domains. We hope to accelerate AI research in the Arabic Legal domain by releasing the ArabLegalEval dataset and code: https://github.com/Thiqah/ArabLegalEval

heading: 1 Introduction
text: The development of LLMs has revolutionized various fields by enhancing natural language understanding and generation capabilities. However, the applicability and performance of these models in specialized domains, such as legal contexts, specially in low- and medium-resource languages such as Arabic, remain active research areas. In this paper, we report on ongoing work for evaluating the proficiency of large language models in understanding and processing Arabic legal texts. Given the complexity and richness of legal language, especially in Arabic, it is crucial to develop benchmarks
that accurately assess the models’ capabilities in this domain in order to guide model development. One of the key motivations for this work and benchmark is to find out the current state of LLMs in the Arabic Legal domain. Thus, we benchmark a wide range of LLMs from proprietary multilingual LLMs, such as GPT-4 (OpenAI, 2024a), to open-source Arabic-centric LLMs, such as Jais (Sengupta et al., 2023).
There are perhaps two broad categories of evaluation criteria that are useful for assessing the performance of legal LLMs. The first category is the ability of a model to use specific regulations, facts, and data that are relevant to a particular conversation. This can be achieved by having the LLM memorize specific facts, presumably by finetuning it on particular legal corpora, or more conveniently by using a Retrieval-Augmented Generation (RAG) system to retrieve information relevant to the context at hand. The second broad category of assessment criteria is related to the model’s ability to exhibit logical reasoning, understand relationships between entities and events, and apply these skills to answer
225
questions. In this initial release of ArabLegalEval, we include tasks to assess the legal reasoning capabilities in Arabic as well as tasks to measure the ability to recall and use legal knowledge embedded in the finetuned models. Much like ArabicMMLU (Koto et al., 2024) was designed to test the general reasoning capabilities of Arabic LLMs but includes a mix of tasks with some requiring previous knowledge, the ArabLegalEval benchmark seeks to develop Arabic legal tasks and questions derived from original Arabic legal sources, in consultation with legal professionals, to test the legal reasoning capabilities of Arabic LLMs. The benchmark also includes some high quality translations, verified by legal experts, of tasks from English legal benchmarks (LegalBench) (Guha et al., 2023).
The benchmark includes Arabic legal Multiple Choice Questions (MCQs), Question & Answer (QA) pairs where the relevant Saudi regulations are included in the question, as well as Arabic translations of tasks from LegalBench. The benchmark does not assess the ability of models to retrieve specific facts and laws relevant to a particular context from external knowledge bases. While important for the successful deployment of a legal LLM, these retrieval abilities will be assessed in future tasks.
The primary contributions of this paper are: the development of a novel methodology for generating a legal QA dataset that can be adapted to other domains, such as finance; and ArabLegalEval, an initial dataset specifically designed for assessing Arabic legal knowledge in large language models.

heading: 2 Related work

heading: 2.1 Arabic and multilingual reasoning capabilities in LLMs
text: The success of LLMs, such as GPT-4 and GPT-4o (OpenAI, 2024a,b), Claude-3 (Anthropic, 2024), Command R and Command R Plus (Cohere, 2024a,b), and Llama3 (LLaMA, 2024), in exhibiting general purpose reasoning abilities - when queried in English - have naturally led to the development of models trained on Arabic content as well as benchmarks to evaluate the quality of reasoning in Arabic. Jais (Sengupta et al., 2023), for example, trained on native Arabic and quality translations, has shown better performance than other open-weight models on an Arabic version of the MMLU multiple-choice school exam questions benchmark (Koto et al., 2024). Arabic content and
questions are also part of many broadly used multilingual benchmarks for assessing general reasoning. For example, MLQA (Lewis et al., 2020) assesses reading comprehension and question answering capabilities in languages including Arabic.
While substantial experimentation is ongoing for evaluating reasoning and knowledge-based tasks in multilingual models, there are some model scale effects that appear to be emerging. On one hand, smaller models and models trained with limited data do not generally perform very well. Even GPT-3.5 (Ouyang et al., 2022) performs worse when responding to queries in Arabic and NonEnglish languages as compared to English on various tasks (Koto et al., 2024; Jin et al., 2023). On the other hand, the performance generally improves steadily with increasing model scale (Shi et al., 2023). GPT-4, for example, exhibits remarkable performance on all sections of ArabicMMLU surpassing all other models including Arabic-centric ones, even when few-shot prompts are used. It appears that multilingual reasoning is an emergent ability of large language models. Beyond a certain (task-dependent) scale, LLMs have evidently strong multilingual reasoning abilities.

heading: 2.2 LLMs and evaluation benchmarks in legal domains
text: Given the importance of natural language to law, the advancing capabilities of large language models have been very quickly recognized and used for performing legals tasks. Current efforts are ongoing to explore whether current LLMs can be used as legal assistants for producing background research, drafting initial documents, summarizing contracts, answering questions about reports, and handling related tasks (Nay et al., 2023; Perlman, 2022; Goth, 2023). The announcement that GPT-4 has “passed the bar exam” (Katz et al., 2024) has shown potential that intelligent legal advisors are not too far off. In the area of tax law for example, LLMs, particularly when combined with prompting enhancements and the correct legal texts, can perform at high levels of accuracy (Nay et al., 2023). Systems for Arabic court rulings and QA related to Legal Palestinian cooperative maters were presented in (Ammar et al., 2024; Maree et al., 2024).
However, closer inspection has revealed that certification exams or narrow domains are not always representative of the actual use-cases in law practice or for LLMs (Fei et al., 2023). As a result, a number of efforts have been ongoing to develop
models, agents, and benchmarks that use tasks and general conceptual frameworks similar to those used in the legal profession. LegalBench and a Chinese counterpart, LawBench (Fei et al., 2023), have sought to collect and categorize a broad set of tasks that appear in law practice, from simple tasks requiring rule-recall or issue recognition to sophisticated tasks that require interpretation, drawing conclusions, and multi-hop reasoning. As many of the LegalBench tasks, particularly those requiring interpretation, appear to involve reasoning skills that are not particular to US law, it seems reasonable to use translated versions of these tasks for testing LLMs in different languages.

heading: 3 Data Sources

heading: 3.1 Raw Arabic data sources
text: Obtaining comprehensive Arabic legal data is challenging. After consulting Saudi legal experts, we identified key sources that we have started to incorporate (Figure 1). These include scanned documents from the Ministry of Justice (MoJ, ÈYªË@ èP@ Pð) and the Board of Experts (BoE, Z @P PñË@ Êj. Öß. Z @Q . mÌ'@ é JJ ë). MoJ documents cover 67 regulations (5720 subjects) and 388 circulars, while BoE documents include 448 rules (14134 subjects). See Appendix B for samples.
Both sets of documents contain regulations and statutes, they differ primarily in the topics they cover. MoJ documents specialize in topics issued by the Ministry, providing a comprehensive database of judicial regulations and legislation essential for legal research and practice.
Note that all data sources used are open and publicly available and scraped from official websites with no confidential information. These were the raw data sources used to generated the benchmark

heading: 3.1.1 Preparation Steps
text: For both BoE and MoJ documents, we followed a systematic data preparation process to ensure that the data is rich and easy to work with. The data was scraped from the web to capture all the regulations while preserving all metadata to allow for careful filtering later.
An example of a processed, structured, MoJ data document is shown in Figure 10.

heading: 3.1.2 Frequently asked Questions (FAQs)
text: In addition to these raw sources, we also rely on human-written FAQs that are publicly available.
The questions and answers in this data are generally available in the BoE and MoJ documents, and we use them to build an open-ended question answering task in the benchmark, which we call NajizQA. A sample of this data can be found in Figure 11.

heading: 3.2 LegalBench
text: In addition to native Arabic sources, we rely on the translation of English legal documents. Legalbench is a benchmark for legal reasoning in English LLMs (Guha et al., 2023). We selected four datasets from it and translated them from English to Arabic. These were specifically chosen because they have fewer localization requirements and are more universal. This makes them ideal for assessing the ability of LLMs to understand and interpret legal clauses and contracts in Arabic. Consumer Contracts QA: this dataset consists of 400 yes/no questions about the rights and obligations outlined in online terms of service agreements. Contracts QA: this dataset consists contract clauses and questions about these contracts. It has 88 examples, with 80 examples for testing and 8 examples for training. Privacy Policy QA: the dataset consists of questions and corresponding clauses from a privacy policy. It consists of a total of 10,931 instances, with 8 examples for training and 10,923 for testing. Privacy Policy Entailment: this dataset has 4385 examples in training and testing, each example is a privacy policy clause and a description. The goal is to determine if the description for the clause is correct or not.

heading: 3.3 ArabicMMLU
text: ArabicMMLU (Koto et al., 2024), an Arabic knowledge evaluation benchmark constructed from human-written school exams from Arabic-speaking countries, served as one of the inspirations for this work. With a subset of its samples focused on the legal domain, ArabicMMLU provided a valuable starting point for us to generate our MCQs.

heading: 4 Benckmark Tasks
text: In this section, we describe the three broad task categories in the benchmark, including 10,000+ MCQs from the native Arabic MoJ and BoE documents, a set of QA from these documents, and a quality translation of a subset of the English
LegalBench benchmark related to consumer contracts and privacy policies. See Figure 1. We believe that the mixture of questions from native Arabic documents and translated questions gives us a somewhat diverse set of tasks and allows the benchmark to test a broader set of capabilities. In addition, this allows us to test the observation that, with increasing model scale, multilingual LLMs can display reasoning abilities and semantic judgment in Arabic as well as in they do in English (Shi et al., 2023). The quantitative details and human performance baseline are presented in Appendix J.

heading: 4.1 MCQs
text: One standard method of benchmarking reasoning and memorization capabilities in neural networks are MCQs, such as MMLU (Hendrycks et al., 2021). It is easy to verify the correctness of the answer using exact matching or regular expressions making MCQs ideal for automatic evaluation.
Given the availability of a large collection of raw legal documents from the MoJ and BoE, we aim to generate synthetic MCQs from them, using them the documents as context. Generating MCQs poses two main challenges: formulating questions and generating options (correct answers and plausible distractors). We approached this using a robust LLM and experimented with three methods: 1. QA to MCQ, 2. Chain of Thought (CoT), and 3. Retrieval-based in-context learning. See Figure 2.
In all cases, we prompt the model to synthesize questions in the same format and style as MMLU.

heading: 4.1.1 QA to MCQ
text: Here we use a two-step prompt. Given a legal document, the model is prompted to generate both a question and its answer, then a follow-up prompt to convert the question into a Multiple Choice Question (MCQ) by rephrasing the answer and generating appropriate distractors, f(ci) → qi, ai; g(qi, ai) → Di where f is a language model that is given some legal context ci, and then prompted to generate a QA pair (qi, ai), and g is another instance of the model that is given the QA pair and is tasked to generate a set of distractors Di for the question.

heading: 4.1.2 QA to MCQ with CoT
text: CoT is a relatively recent method of prompting LLMs where instead of directly producing the answer, the model is given space to reason and “think
out loud” before answering, essentially providing itself with more context. This simple technique has led to relatively huge gains in performance, in a variety of tasks (Wei et al., 2023). We want to utilize this idea to generate reasoning-based plausible distractors. This approach can be formulated as: f(ci) → qiai; gCoT(qi, ai) → Di where f is a language model prompted with some context c from the scraped MoJ documents to produce a question q and its answer a, these are then fed to another instance of the model g with CoT prompt to produce a set of plausible distractors D for the question.

heading: 4.1.3 Direct MCQs generation with in-context examples
text: Since the inspirations of this work are popular knowledge evaluation benchmarks such as MMLU and ArabicMMLU, our goal is to generate questions in a similar format. We begin by taking a subset of ArabicMMLU questions that have ‘Law’ as their subject tag (about 300 examples). For each document in the MoJ dataset, we perform a semantic similarity search (Risch et al., 2021) to retrieve the top k examples from these questions. These examples are then added to the prompt to guide the model in generating questions of a similar style.
f(ci, Ek) → qi, ai, Di
The MoJ context is augmented with a set of ArabicMMLU examples Ek that are fed to a model f to generate a question q, where k is the number of retrieved examples. These examples provide context for the model for in-context learning so that the generated answer has the same style and format as the ArabicMMLU examples. From this, the model generated a question a and all of the distractors D in one go.

heading: 4.1.4 MCQ filtering and curation
text: Each of the above techniques was tested and then had the results manually reviewed and inspected by the legal experts according to the metrics in Appendix 3.1
It was concluded that the best method for generating MCQs was in-context examples with k = 3. Based on this, we decided to use this technique to generate all the MCQs.
After generating the MCQs dataset (approximately 12k samples), we did some automatic filtering using GPT-4 (Chiang and yi Lee, 2023), where it was prompted to check if each sample satisfied our criteria (see appendix subsection 3.1). Any sample that failed to satisfy all of the above were removed, which left us with 10,583 MCQs for our benchmark. At the end, we extracted a random subset of the dataset for a final manual inspection.

heading: 4.1.5 Models for Generation
text: It has been observed that models tend to perform better on synthetic data generated by themselves as apposed to another model (Huang et al., 2024). To mitigate this unfair advantage, we split our documents and alternate between two state of the art models: Claude-3-opus and GPT-4. We make sure a model isn’t evaluated on questions generated by itself.

heading: 4.2 QA
text: This dataset includes filtered legal QAs from a publicly available human written set of FAQs. The questions include only those referencing specific statues and regulations and articles in the MoJ and BoE Arabic documents, and are therefore particularly valuable for evaluating Arabic legal LLMs. See figure 4. A sample of the data shown in the Appendix B.
Embedding techniques were used in the semantic similarity matching phase (Figure 4). Specifically, we utilized text-embedding-3-small (OpenAI, 2024c) to generate embeddings for both the questions and contextual information. Subsequently, cosine similarity was employed to identify relevant texts corresponding to each question. This methodological framework was selected to evaluate the models’ reasoning capabilities with and without context.

heading: 4.3 Arabic translation of LegalBench

heading: 4.3.1 Translation Strategy
text: We evaluated three different machine translation models, Azure Translation Services (Microsoft, n.d.), Google API Translation (Google, n.d.), and Opus MT (Tiedemann and Thottingal, 2020) along with GPT-4 to determine the best translator for a publicly available dataset with legal context. The dataset used in these experiments is the United Nations Parallel Corpus (Ziemski et al., 2016), which consists of UN documents in the six official UN languages. We focused solely on the English and Arabic datasets, which comprise 20 million rows. However, to expedite the experimentation process, we utilized a subset of 14,000 examples from this dataset. This subset was used to evaluate the performance of the selected models without any preprocessing. Rouge metrics were used in evaluating the translation quality.
By comparing the results obtained from the different models, we aimed to identify the one that consistently produced the best translation from En-
glish to Arabic in the legal domain. The results of our experiments on this subset of data, without any prepossessing, are shown in Table 3 in Appendix D. Overall, the Opus model outperformed the other models across all the metrics. The Opus model achieved a ROUGE-1 score of 0.52, a ROUGE-2 score of 0.3, and a ROUGE-L score of 0.51. In addition to this, we had a sample of the translations manually reviewed by legal experts.
These findings established a basis for selecting Opus as the translator for our main task: translating selected datasets from LegalBench from English to Arabic.

heading: 4.3.2 Evaluation of Translated Output
text: To ensure the highest translation quality, we employed the three models from our initial experiments to translate the “Consumer Contract QA” dataset. We conducted a manual inspection of the overall results and, to further enhance the quality assessment, engaged both GPT-4 and legal experts as evaluators for the translated content.
GPT-4 Evaluation We tasked GPT-4 with evaluating a sample of the translated text alongside the original text, asking it to rate the translation quality on a scale from 1 to 5. Interestingly, GPT-4 consistently rated the translations as either 4 or 5 out of 5, indicating a high level of perceived quality, see Appendix D for GPT-4 evaluation prompt and result example.
Human Evaluation We selected five examples from the translated texts, each consisting of a contract text and a corresponding question. To evaluate the performance of the three translation models — Azure Translation Services, Google API Services, and Opus MT — we asked legal experts to assess each contract and question separately. The experts provided scores for each model’s based on the translation quality and how well the legal context was preserved in the translation. The overall scores for all three models were relatively close, but Opus MT consistently achieved the highest score among them:
• Google API: average score of 3.3 out of 5 • Azure: average score of 3.6 out of 5 • Opus MT: Average score of 4 out of 5
In addition to the overall scores, we asked the experts to rank the best model for each example. Opus MT was chosen as the best model 60% of the time, while Azure Translation Services was selected 40%
of the time. Interestingly, Google API Translation was never selected as the best model for any of the examples.
These results suggest that while all three models performed reasonably well, Opus MT demonstrated superior translation quality for the given legal texts and questions, as determined by the expert evaluations. See Figure 20 in Appendix D for a translated example.

heading: 5 Evaluation and results

heading: 5.1 MCQ Evaluation
text: In this section, we evaluate the performance of language models on our synthetic MCQs dataset using tailored prompts, where the instructions in the prompt are provided in English for each model.

heading: 5.1.1 Experiment Setup
text: We aim to improve the models’ capabilities by modifying the given prompt. Different parts of the prompt can be optimized according to a given metric, and in this evaluation, we started with optimizing the instruction and few-shot examples to determine which method is more effective. Unfortunately, instruction optimization yielded no significant performance gain. On the other hand, fewshot optimization boosted the performance of many models. Hence, we decided to focus on few-shot optimization. ArabicMMLU is a benchmark to assess the capabilities of models, similar to MMLU benchmark, but with localized data in Arabic. A subset relevant to the legal domain of ArabicMMLU was sampled resulting in a total size of 524 questions after filtering questions that require context. Out of those 524 samples, 314 and 210 are law and political science, respectively. We optimize the prompts on this subset to use it for evaluating our generated MCQs.
DSPy (Khattab et al., 2024) is a Language Model (LM) programming framework to optimize LM prompts and weights automatically by recompiling the entire pipeline to optimize it on a specific task. We relied on this framework for prompt optimization. Initially, all of the models were given a zeroshot prompt with an English answer instruction and the input-output format. Then, this zero-shot prompt is optimized for each model to achieve a higher performance by augmenting it with either plain few-shot examples or few-shot with reasoning demonstrations using CoT. Teacher and student models were used to create few-shot examples with
CoT demonstrations, where the teacher is either a clone of the student or another model. Figure 5 demonstrates DSPy’s bootstrapped few-shot optimization.

heading: 5.1.2 MCQ Results Analysis
text: The MCQ dataset contains 10,583 questions, with 5,544 generated by GPT-4 and 5,039 by Claude-3-opus. We evaluated several LMs on this dataset, including Command R, Command R Plus, and Llama3 (8B and 70B). However, GPT-4o was tested only on Claude-3-opus subset to mitigate potential bias towards its own generated questions (Panickssery et al., 2024). Our evaluation metric assesses LM performance by comparing the selected answer with the correct one for each question. For further ArabicMMLU detailed results with optimized prompts, refer to appendix G.
Table 1 summarizes the performance of LMs on our generated MCQs using prompts optimized with DSPy on ArabicMMLU’s legal subsets. Interestingly, many of the optimized few-shot prompts shared identical examples, suggesting that certain examples play a more significant role in improving LMs’ performance than others. In addition, few-shot examples coupled with CoT reasoning boosted the capabilities of the models. For further
testing, we employed GPT-4 as a teacher model for smaller LMs in both plain and CoT few-shot prompting. Surprisingly, these smaller LMs demonstrated greater performance in few-shot CoT when the teacher was a clone of themselves, rather than the more advanced GPT-4. This unexpected result suggests that LMs may have a better grasp of their own reasoning.
We observed that the choice of language plays a crucial role in the reasoning abilities of smaller models. In many cases, these LMs generated answers without providing accompanying reasoning. Figure 6 shows the differences in reasoning language for Command R Plus, revealing a degradation of the model’s reasoning capability when the language choice is Arabic.
Table 1 shows that GPT-4o demonstrated superior performance across all prompting methods, achieving 79.10% accuracy with few-shot prompting. Llama3 (8B) achieved a 5% increase using few-shot prompting with CoT reasoning. Similarly, the other LMs, except Command R, obtained better results when the prompt included their own CoT reasoning. Our findings reveal potential for improving LMs’ question-answering capabilities through task-specific optimized prompts.
Table 1: Experimental results on Generated MCQs
Original Few-shot Few-shot (GPT-4 Teacher) CoT Few-shot CoT Few-shot (GPT-4 Teacher) GPT-4o 76.80% 79.10% - 76.50% -
Command R 68.10% 71.40% 72.30% 71.00% 70.09% Command R+ 68.10% 71.40% 72.30% 73.10% 69.86% Llama3 (8B) 68.20% 71.20% 72.30% 73.60% 70.45% Llama3 (70B) 68.17% 71.47% 72.34% 71.93% 70.53%

heading: 5.2 Open ended QA Evaluation

heading: 5.2.1 Experiment Setup
text: In open-ended QA—contrary to closed-set QA where the possible answers are fixed—the answer can be expressed in natural language, which is harder to evaluate since it relies on semantics and meaning. We observe that English prompts for the exam-taker LLM and the judge-LLM outperform Arabic prompts, so we use those English prompts for the instructions.
Traditional evaluation of QA models uses metrics like Exact Match (EM), F1-score, and top-naccuracy, focusing on lexical matching. These metrics often miss semantically similar answers that use different words.
We use use an LLM-as-a-judge (Huang et al., 2024), (Kim et al., 2023), (Zheng et al., 2023) to rate the answer similarity on a scale from 0 to 5 given the generated answer and the reference ground truth answer. In our case, GPT-4 is the judge, and we refer to the output score as the answer similarity metric, see figure 23 for details. We notice that the judge LLMs give a lower score to answers that are in a different language than that of the reference answer even when the content is correct. To mitigate this, we prompt the models to output answers in the same language as the reference answer (which is Arabic in our dataset).
We run the evaluation for each LLM with two cases: one where it is given the question and the needed context, and one where it is given only the question. We also add a "golden model" that always answers the perfect ground truth answer, just so that we can compare against the upper bound of what the judge-LLM is going to score.

heading: 5.2.2 QA Results
text: We run our experiments on the 79 NajizQA pairs that have been filtered and verified by legal experts.
From the original 1358 QA pairs, we chose the filtered 79 QA + context sets. We run them with and without context and show result in Figure 7.
We notice that all models perform significantly better when the context is provided, and on closer inspection (Figure 11), we can see that the answer can easily be extracted from the context, making the context a giveaway of the answer which only measures the model’s ability to recall and extract information and not legal reasoning.

heading: 5.3 Arabic LegalBench Evaluation

heading: 5.3.1 Experiments Setup
text: In total, we carried out 96 unique experiments with different prompting techniques to assess the performance of the models in legal reasoning. All prompts were in English, as this yielded better model performance. Appendix I provides prompt examples for each technique used.
For the Contract QA and Consumer Contract QA datasets, we utilized the entire testing data in our experiments. However, for the larger datasets, we selected a representative sample. For each technique and dataset, we created tailored prompts. The training examples in one-shot and few-shot learning were fixed across all models for each specific dataset. We then benchmarked the performance of all models using the following four approaches: zero-shot learning with simple prompts where models are asked straightforward questions without extensive instructions; zero-shot learning with detailed instructions in the prompt; one-shot learning; and few shot learning.

heading: 5.3.2 Results Analysis
text: A comprehensive overview of model performance across all tasks and learning techniques is presented in Table 6 in Appendix H. This table provides a summary of F1 scores for each model and task combination, offering a comparative analysis of the various approaches evaluated in this study.
The consumer contract QA task can assess the models’ ability to answer questions based on a long context, in this case consumer contracts. GPT-4 with one-shot learning, and Llama3 (70B), with a zero-shot basic prompt, achieved the highest F1 score of 90%. This suggests that both models can extract relevant information from consumer contracts and provide answers, even when training examples are not available or limited. We observed that most of the models performed well on the Contract QA task, which assessed the models capabilities to answer questions related to contracts. Command R Plus, using few-shot learning, achieved the highest F1 score of 99%. This high score indicates that the model can accurately understand and respond to questions about contracts when provided with a small number of training examples. However, this task proved to be the least challenging among the four tasks. On the other hand,the privacy policy entailment task proved to be the most challenging for the LLMs across all techniques, highlighting the complexity of this task. Command R Plus, using one-shot learning, achieved the best F1 score of 66%. This result suggests that while the models struggle with this task, Command R Plus is more capable of understanding of privacy policies when given a single training example.
In the final task, Privacy Policy QA, which evaluated the models’ ability to answer questions based on privacy policies, GPT-4 with one-shot learning achieved the highest performance. This result demonstrates GPT-4’s strong capability in extracting relevant information from privacy policies and providing accurate answers when given a single training example.
Overall, one-shot learning achieved the best results for most of the models across the various tasks. This finding suggests that providing a single example can significantly improve the models’ performance in understanding and responding to questions related to legal documents such as consumer contracts and privacy policies.

heading: 6 Limitations
text: The ArabLegalEval benchmark currently relies heavily on Saudi Arabian legal documents, with some tasks translated from universal benchmarks. Including documents from more Arabic-speaking countries would improve geographic representation. Our study did not evaluate all models, which limits generalizability; future work should include a broader range of models. Limited access to legal experts affected validation depth; involving more experts would improve quality control. The dataset lacks granular categorization, such as taskspecific prior knowledge, document origin, and AI-generated content labels. Adding more granular metadata and task categories would aid nuanced model training and evaluation.

heading: 7 Conclusions and Future Work
text: We are developing an Arabic benchmark to evaluate LLMs’ legal reasoning, using Saudi regulations and translated LegalBench problems. Future plans include adding more KSA regulatory documents, court cases, and judicial decisions to enhance this benchmark and promote advancements in Arabic legal AI.

heading: Acknowledgments
text: We would like to express our sincere gratitude to Yusra Alonaizan for her contributions in envisioning this project, and dedicated team of six members—Lama Alsaif, Alanoud Alangari, Alaa Bazaid, and Faisal Alessa for gathering and curating the data sources. Additionally, we extend our thanks to the legal team at THIQAH for their valuable time in reviewing the labeled data, we are grateful to Norah AlHussein, an external legal expert, for her meticulous double-review of the labeled data and the translated LegalBench dataset, as well as for her clarifications and support in explaining the legal data.

heading: B Examples of Data Sources

heading: A LMs descriptions

heading: C MCQs Generation

heading: 3.1 MCQs Evaluation Criteria

heading: 3.2 MCQs Example
text: 3.3 Prompts

heading: D Translation

heading: 4.1 ROUGE scores for UN English-Arabic translation

heading: 4.2 GPT-4 Translation Evaluation Prompt

heading: 4.3 GPT-4 Translation Result Evaluation
text: 4.4 Sample of the Translated Data
E QA

heading: F Data statistics

heading: G MCQs Evaluation

heading: 7.1 DSPy Signatures

heading: 7.2 ArabicMMLU Results with DSPy optimization
text: Table 5 displays the performance of models on ArabicMMLU’s law and political science subsets with optimized prompts. The prompts were optimized with 15 examples as labeled inputs from law and political science subsets, respectively. Therefore, the accuracies were computed using the remaining 299 and 195 samples from the respective law and political science sets.
H Details for Arabic LegalBench Experiments Results

heading: I Araic LegalBench Prompts

heading: 10.1 MCQs

heading: J Dataset Quantitative Analysis

heading: 10.2 QA

heading: 10.3 Arabic LegalBench

heading: 10.4 Experts Performance on the Tasks
text: To establish a human performance baseline and provide a meaningful comparison for the LLMs’ performance, a sample of the data was presented to legal experts for evaluation. These experts, with their specialized knowledge and experience, were tasked with solving the same problems that were presented to the LLMs. Table 9 summarize the results of the experts assessment.

heading: Future Work
text: Future plans include adding more KSA regulatory documents, court cases, and judicial decisions to enhance this benchmark and promote advancements in Arabic legal AI