LEXam: Benchmarking Legal Reasoning on 340 Law Exams
Authors: Yu Fan
Jingwei Ni
Jakob Merane
Yang Tian
Yoan Hermstrüwer
Yinya Huang
Mubashara Akhtar
Etienne Salimbeni
Florian Geering
Oliver Dreyer
Daniel Brunner
Markus Leippold
Mrinmaya Sachan
Alexander Stremitzer
Christoph Engel
Elliott Ash
Joel Niklaus
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
legal reasoning
long-form
multiple-choice
question answering
Primary area: datasets and benchmarks
TL;DR: We introduce LEXam, a comprehensive benchmark derived from real law exams, highlighting substantial challenges for LLMs in structured legal reasoning tasks and providing a scalable evaluation method beyond simple accuracy metrics.

Abstract: Long-form legal reasoning remains a key challenge for large language models (LLMs) in spite of recent advances in test-time scaling. To address this, we introduce ***LEXam***, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels. The dataset comprises 7,537 law exam questions in English and German. It includes both long-form, open-ended questions and multiple-choice questions with varying numbers of options. Besides reference answers, the open questions are also accompanied by explicit guidance outlining the expected legal reasoning approach such as issue spotting, rule recall, or rule application. 
Our evaluation on both open-ended and multiple-choice questions present significant challenges for current LLMs; in particular, they notably struggle with open questions that require structured, multi-step legal reasoning. Moreover, our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities. 
Deploying an ensemble LLM-as-a-Judge paradigm with rigorous human expert validation, we demonstrate how model-generated reasoning steps can be evaluated consistently and accurately, closely aligning with human expert assessments. Our evaluation setup provides a scalable method to assess legal reasoning quality beyond simple accuracy metrics. Project page: https://lexam-benchmark.github.io/.

Published as a conference paper at ICLR 2026
LEXAM:BENCHMARKING LEGAL REASONING ON 340
LAW EXAMS
Yu Fan F Jingwei Ei
Jakob Merane F.L.M
Yang Tian* +2
Yoan Hermstrüwer Yinya Huang
Mubashara Akhtar Etienne Salimbeni
Florian Geering
Oliver Dreyer Daniel Brunner
Markus Leippold
Mrinmaya Sachan E Alexander Stremitzer Christoph Engel M
Elliott Ash Joel Niklaus
FTH Zurich University of Zurich University of Lausanne
M Max Planck Institute for Research on Collective Goods Ommilex
University of St. Gallen
Swiss Federal Supreme Court NNiklaus.a ai
Eqqual Contribution
ABSTRACT
Long-form legal reasoning remains a key challenge for large language models (LLMs) in spite of recent advances in test-time scaling. To address this, we intro- duce LEXAM, novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels. The dataset comprises 7,537 law exam questions in English and German. includes both long-form, open-ended questions and multiple-choice questions with varying numbers of op- tions. Besides reference answers, the open questions are also accompanied by explicit guidance outlining the expected legal reasoning approach such as issue spotting, rule recall, or rule application. Our evaluation on both open-ended and multiple-choice questions present significant challenges for current LLMs; in par- ticular, they notably struggle with open questions that require structured, multi-step legal reasoning. Moreover, our results underscore the effectiveness of the dataset differentiating between models with varying capabilities. Deploying an ensemble LLM-as-a-Judge paradigm with rigorous human expert validation, we demonstrate how
reasoning steps can be evaluated consistently and accurately, closely aligning with human expert assessments. Our evaluation setup provides a scalable method to assess legal reasoning quality beyond simple accuracy metrics. Project page: https:://// //lexam benchmark, gi github.... i
INTRODUCTION
Legal reasoning a critical frontier for large language models (LLMs) specifically and artificial intelligence (Al) at large, requiring specialized domain knowledge and advanced reasoning abilities such as precedent interpretation, statutory analysis, and legal inference. Despite progress in general reasoning, legal reasoning remains difficult and under-assessed in NLP research. Moreover, the legal domain is inherently high-stakes and a failure to thoroughly examine the capabilities and limitations of models could lead to serious real-world consequences (Kleinberg et al. 2018 Medvedeva & Mcbride 2023 Mahari et al.) 2023). As LLMs are used increasingly for various legal tasks, such as legal judgment prediction (Wang et al.
2024a Shui al. [003 Jiang & Yang 2023 Deng et al. 2023). legal summarization (Bauer et al. 2023 Ash et al. 2024), legal interpretation (Savelka et al.
2023 Engel McAdams 2024 Dugac & Altwicker 2025) Luo et al. 2025). legal case retrieval
(Ma al. [2024] Zhou et al. [023 Mahari et al. (2024), legal argument classification (Thalken et al.)
2023 Chlapanis et al. 2024), and contract drafting (Lam et al. 2023), is imperative to assess and
understand their capabilities and limitations.
However, LLM benchmarking on deterministic tasks has so far mainly concentrated in STEM domains, such math Olympiad problems (Zheng et al. 2022] Liu et al. 2023] Tsoukalas al. 2024 Ren et al. (2025) and physics problems (Feng et al. |025] Zhang et al. 20.5|
[2022]CCho. Chow et al.] 2025). Reasoning models such OpenAl o3 (OpenAl 2025 2025 and DeepSeek-RI (DeepSeek-Al) 2025a have

---

Published as a conference paper at ICLR 2026
6AB
A=
Process- &
Raw Data
Open Questions
Open Questions with Domain
Outcome-
LLM-as-A-Judee AJudee
Open Questions
Test: 2,541
based Evaluation
O
Metadata
Human Judge
-= T/Fs D= MC Questions
Domain Expert
Question
P6 Permutation Augmentation
Outcome-
MC Questions
Extraction
and Check
Domain Metadata
MCQs
Evaluation
Accuracy Assessment
Test:
Collection
Dataset Curation
Evaluation
Figure 1: Process for creating LEXAM, a comprchensive legal reasoning benchmark derived from real law school exams. Created through careful expert extraction and curation, LEXAM contains 2,841 open-ended and 4,696 multiple-choide Open-ended questions support both process- and outcome-bee based evaluation by -aaMs-as--JJudge and human judges, while MCQs provide clear, outcome-basee assessments.
questions (MCQs), each with detailed domain metadata.
demonstrated impressive performance in tasks dominated by deduction and established scientific rules, Nevertheless, their abilities more nuanced and context-dependent domains, such as the law, remain poorly understood (Engel et al. 22025] NNklaus et al. 2025). Legal reasoning requires not only rigorous deductive and inductive logic but also their application to complex practical scenarios, often involving ill-defined problems. This domain offers an ideal testbed for non-formal reasoning, yet it remains underexplored the context LLM performance. In contrast, STEM benchmarks allow for straightforward validation-for example, by checking final numerical answers or employing formal verifiers. Adopting this outcome-basee evaluation paradigm, prior legal reasoning benchmarks (Guha et al. 2023 Fei et al. [203) Chlappais al. 2024; Zheng et al. (025) have primarily assessed the correctness of a model's 's final outputs, without explicitly evaluating the intermediate steps necessary for reaching a legally sound conclusion. Consequently, these benchmarks often overlook the reasoning process itself, leaving it unclear why I.LMs fail to reach correct conclusions in some cases. This limitation raises concerns of tangible harm when LLM-based systems are deployed in practice.
Furthermore, while jurisdictions differ in their authoritative sources and treatment of precedent, they share a common foundation in legal reasoning, though with notable distinctions between common law and continental law systems. Common law systems (e.g. the United States, United Kingdom, and Australia) emphasize case precedent and judge-made madee law, whereas continental law systems (prevalent in most of Europe, Latin America, and parts of Asia) rely primarily on comprehensive legal codes and statutes. Across traditions, legal reasoning involves identifying the legal issue from the facts, determining the relevant rules, and applying those rules to the case. It also depends on diverse linguistic expressions that vary across systems. Prior studies (Ryan et al. 2024 Jin et al. 2025
Fan et al. 2025: Tian al., 2026) show that LLMs exhibit cultural biases tied language use. Evaluating LLMs on multilingual legal data is therefore essential to align them with real-world needs.
To address these gaps, introduce LEXAM, multilingual legal reasoning benchmark designed to assess the process-base based and outcome- based correctness of the answers generated by LLMs, as illustrated in Figure I Based on exams in Swiss, European, and international law, LEX AM provides a comprehensive framework for evaluating the legal reasoning capabilities of LLMs through problem sets available in both English and German. The dataset comprises 7,537 law exam questions in English and German. It includes both long-form, open-ended questions and multiple-choice questions (MCQs) with varying numbers of options. Each open question paired with reference answers and clear normative guidance, outlining the expected legal reasoning chain. This allows for thorough benchmarking of an LLM's legal reasoning capabilities beyond merely evaluating final answers. Questions are sourced from 340 law exams across 116 courses at the Bachelor's and Master's levels taught by leading law professors at the University of Zurich Faculty of Law, one of Europe's 's largest and most prestigious law schools (see Section B for further details). As a result, LEXAM spans wide range of legal areas both in domestic and international law, thus providing a robust benchmark for the evaluation of legal reasoning skills (see T more details).
Experimental results show that state-of of-the-an art (SOTA) models struggle with multi-step reasoning and applying legal rules and principles to novel scenarios. LLMs particularly struggle with open questions requiring structured legal reasoning. By designing and deploying an ensemble LLM-as-a- Judge evaluation framework, we demonstrate that model-generated reasoning steps be evaluated
2

---

Published as a conference paper at ICLR 2026
Open Question
MCQ
Question:
Question:
(domicledd in Switzeriandi, (domicled Switzerland) and (domiclleed France) enter contract The contract contains an arbitrtioon cause: "Any dispute [-I] shall be resolved either by artitration Swizarland French counts" is the PIL applccabl determine the validity of arbitration agreement inbates arbiraaion against
Madeliine, lawyer special knowladge insurances, planning to open an insurancs broker boutique In order to start her business actvities, she needs advice which business association sho should choose. Which answers correct?
Reference Answer (excerpf):
company limited by shores makes business dependemt the person of Madereine than
178(1) PILA:
labilit comoany.
Seat In Switzorian"" not speaifi may lead to invalidity I..]
1. In company limited by shares the company's Iampiyy is Emiled share capvisl.
2. Donicile: and B both in CH Chapter 12 PILA not applicable (5FT view)...
With company
shares Madeleine's business sooms
LLM Answer (excerpt):
more trustwrrthy than with a iniltd labilly company.
Since the seat is Switzerland, validity governed by Swiss law (Art. 198 PILA).
order
company limited by shares Madeleine needs
Conckusion apples. Arcitration agmeament Legal Provisions: Artt 193 PILA
Swiss
CHF 100/000.
A. I. H., and
LLM Judgment (excerpt):
B. and ii
Model incorrectly Art. PILA (wrong chapter), epply
the statements
Falls assess whether Chopter 12 apples (ail parties CH),, ignores SFT jurispradence, and
D. i
does not address ambigulty dauuse.
Correctness Score: [[0.2])
Reference Answer::
Illustration of long- forg-formmo open question (left, abbreviated example for illustration purposes. The full version is provided in Appendix (E33) and a MCQ-4 with a set of candidate statements (right).
Figure 2:
consistently and accurately, closely aligning with human expert assessments. This approach provides a scalable method for assessing legal reasoning quality beyond simple accuracy metrics. Our work provides both resource for NLP research in the legal domain and a practical tool for educators evaluating AI capabilities.
Our contributions be summarized follows: (1) We introduce novel legal reasoning benchmark, addressing challenges such as long-context understanding, multilingualism, complex reasoning, and hallucinations. (2) We demonstrate baseline performance and challenges related our benchmark by evaluating wide range of SOTA models, providing a solid foundation for future research. (3) We perform expert analyses of model outputs, offering detailed insights into the dataset characteristics and current LLMs' limitations in legal reasoning. (4) Finally, we conduct a rigorous validation with human experts to establish scalable and reliable LLM-based evaluation for open-ended questions. The resulting annotation can be reused to validate any backbone LLM for LEX AM evaluation.
2 DATASET
Figure I overviews our dataset construction process, detailed in the following sections.
2.1 CONSTRUCTION
The raw dataset is derived from 340 law exams that span 1 116 courses taught the University of Zurich between 2016 and 2023 and was downloaded from their publicly accessible website. It contains a total of 3,628 questions, comprising 2,867 open questions, 398 MCQs, and 363 true/false questions (TFQs). All open questions, MCQs, and TFQs from the included courses were retained without filtering. Courses were included if they fell under one of four high-level legal areas, i.c., private law, public law, administrative law, or interdisciplinary. and were subsequently grouped by three legally trained authors into 78 subdomains based on doctrinal similarity, following the University's official course descriptions. The use and publication of the dataset was approved by the dean of the law school. F provides a sample open question and a MCQ.
Although the exams originate from Swiss institution, the dataset is not limited to Swiss liv doctrine. As detailed in the course metadata (Table (). many of the courses cover international, comparative, and interdisciplinary topics that are central to both civil-law and common-lav law curricula. This diversity broadens the dataset beyond a single legal tradition and enables evaluation across multiple legal systems. Expanding the benchmark include additional jurisdictions, in particular in common-law contexts, is an important future direction outlined in Appendix K
Moreover, human performance data are unavailable due institutional restrictions. In addition, the MCQ format does not appear in prior exams. We therefore report human performance in a separate experiment, summarized in Appendix
3

---

Published as a conference paper at ICLR 2026
Open and MC Questions
Open Questions
Legal Area
Language
Jurisdiction
1245
2500
2325
1905
1200
Private
1750
1751
Internetional
Numbbe beccnds
1000
Interdisc Criminal
Number Nacoods
Records
1500
800
1500
1250
600
582
583
1,00
1000
750
000
25
500
51.6
500
200
140
Public
Private
Interdise.
Criminal
de
Swiss
Interaaionall
Generic
Figure 3: Distribution of open questions and MCQ-4 by legal area, language, and jurisdiction across development and test datasets, Open questions (solid) and MCQs (hatched).
2.1.1 OPEN QUESTIONS
We divided the open questions into test and development (dev) sets to support few-shot learning. Specifically, for each course containing least open questions, we randomly selected five for the dev set and assigned the rest to the test set. This procedure yielded 300 questions from 60 courses in the dev set, and 2,541 the set.
2.1.2 MULTIPLE- CHOICE QUESTIONS
The MCQs were manually parsed into question stems and statements by three authors, where the stem provides contextual information to evaluate each statement. For each stem, we randomly generate MCQs that contain two to five statements extracted from the original MCQ. Each question includes one correct answer and 3 7 / 15 31 distractors, randomly selected from the set of all possible incorrect answers where applicable. The set of incorrect answers comprises all answer combinations where at least one statement is incorrect. This ensures a uniform number of answer choices per question and baseline accuracy for random guesser.
To unify the evaluation metric across formats, we convert all TFQs into multiple-choice format. First, each TFQ manually parsed into a question stem (if applicable) and a statement. For questions that share the same stem, we group them and randomly generate MCQs with two to five associated statements. For stemless questions, we group them by domain and design templates to generate MCQs with two to five randomly selected statements. As with the original MCQs, each converted question includes four answer choices: one correct option and 7 / 31 randomly selected incorrect options from the entire pool of possible answers. This process yields total of 4,696 MCQs. sample MCQ-4 can be found in Figure 2
While evaluating LLMs using MCQs has become common practice due to the ease of interpreting results, previous research has demonstrated that LLMs can exhibit unstable and sometimes unpre- dictable behavior in response changes in question formatting, phrasing, the order of answer choices (Wang et al.) 2024b: Pezeshkpour & Hruschka 2023) Alzahrani et al.) 2024). To further examine this issue, we constructed auxiliary MCQ dataset comprising 385 questions, each con- taining five statements. This subset is determined solely by the structural requirement of having five statements, with additional filtering applied, and the questions originate from courses (see Table & for the distribution). We systematically vary the number of answer choices (4, 8, 16, and 32) while keeping both the question stem and the order of the statements constant. This perturbation setup aims to diagnose whether models select the correct answers based on genuine domain understanding or merely guess due to insufficiently challenging distractors.
2.2 DATASET STATISTICS AND STRUCTURED FEATURES
The LEXAM dataset comprises 2,841 open questions, split into dev (10.6%) and test (89.4%) sets, and 4,696 MCQs with vary numbers of options (4 to 32). Detailed per-course statistics and textual descriptions in Section B.2 Figure iillustrates the data distributions.
We stored several features as metadata for our analysis. The area feature groups questions into four distinct legal categories: private law, public law, criminal law, and interdisciplinary. The course
4

---

Published as a conference paper at ICLR 2026
feature represents course information, while the jurisdiction feature classifies questions into one of three legal jurisdictions: Swiss, international, or generic. In addition, the language feature indicates the language of the exams, either English German. The English and German questions are independent items drawn from distinct exams and are not parallel translations. We deliberately avoided translation due the complexity and risk of semantic drift in legal contexts (see also (Niklaus) iet al. 2. The binary label none_as as _a___nn_ppp__ ion identifies MCQs where the option "None of the statements" ("Keine der Aussagen") appears; this attribute is applicable exclusively MCQs. Another feature, n n_staat attemttsss..... specifies numerically how many statements each MCQ includes, again relevant only for MCQs. furthermore, the binary label negative weerueeetttteeeeeeeeeeeeeeeeesee t i ion indicates whether the question is phrased negatively (e.g. "Which of the following statements are incorrect?"), which is once again relevant only for MCQs. Lastly, the feature year specifies when the exam was administered.
3
EXPERIMENTAL SETUP
In this section, we cover the experimental details used to benchmark models on LEXAM questions.
3.1 OPEN QUESTIONS
Section E. exxhibits the prompt to answer open questions. first informs about the relevant knowledge domain with course name, and then provides standard guidance that law students follow when answering exam questions.
Evaluation Metric.
Evaluating open questions in specialized domains like law is challenging. Lexical or shallow semantic metrics, such as BLEU (Papineni et al.) (002). ROUGE (Lin) (004). BERTScore (Zhang et al. 2020), and AlginScore (Zha et al. 2023). may accurately reflect answer quality, as high lexical or semantic overlap does not necessarily indicate correct legal reasoning, and
vice versa. Legal reasoning often involves low lexical similarity (Zheng et al. (025). LLLM-as-a-Judee (Zheng et al. 2023) may help, but whether a general-domain LLM is capable of judging the quality of legal responses remains an open question.
To develop more suitable LLM judge and verify its reliability, we proceed as follows: First, two of the authors, both with formal doctoral-level legal training, draft specialized judging prompt, as presented in Section E.2 Then, they conduct pilot study to iteratively optimize the prompt (with GPT-4o as the judging LLM) until they were satisfied with the judging performance. This pilot was conducted on diverse sample of courses and key challenge was to calibrate penalties. Through this process, several refinements were also made to the prompt.Rather than encoding an abstract theory of legal reasoning, the prompt is tuned to track adherence the doctrinal structure present in professor-written model answers and to penalize domain-specific errors such as fabricated or incorrect statutory citations. In this way, legal expertise enters the system both through these model answers and through the domain-specific prompt design, which distinguishes our setup from general-purpose reasoning evaluation.
The next step is to identify capable LLMs for executing the expert-verified prompt. We rigor- ously evaluate various LLMs against human experts using the Alternative Annotator Test (Alt-test) (Calderon et al. 2025). detailed in Section 5 We find that only proprietary LLMs (e.g., GPT-4o, and Gemini-2.5-Pro and extremely large reasoning models (e.g., DeepSeek -1) consistently outperform human judges. Exclusive reliance on such models for evaluation, however, undermines the acces- sibility of LEXAM. Encouragingly, we find that carefully constructed ensembles of open-source LLMs-fol for instance, taking the minimum score of Qwen3-32B and DeepSeek-V3-at surpass human judges. Our analyses further demonstrate that minimum-scone ensembles of LLM judges consistently enhance evaluation performance.
In our evaluation, we adopt an minimum- score ensemble of GPT-4o, Qwen3-32B, and DeepSeek- V3 to grade open questions. Practitioners may instead rely any LLM judge ensemble that passes the Alt-test.
Importantly, ensemble design avoids dependence any single model, whether open closed. The adopted judge includes two open-source models (Qwen3-32B and DeepSeek-V3. V3) and one closed model (GPTT-4o), ensuring robustness across different architectures and providers. This
5

---

Published as a conference paper at ICLR 2026
diversity is crucial for reducing self-bias and family-bias, which are known risks in LLM-as-a- Judge setups. By aggregating pointwise minimum scores, our ensemble suppresses overly favorable evaluations for answers produced by the same model or model family (see Appendix (.6). We also report performance for fully open-source judge ensembles in Appendix (.44, enabling reproducible evaluation pipelines without reliance proprietary systems. Additional robustness checks for L.LM judgese are presented in Appendices H and While we do not present the ensemble LLM-as-a-Judge approach as novel evaluation paradigm, we argue that it provides practical, empirically validated solution for reliable evaluation of open-ended legal reasoning. The ensemble is designed specifically to mitigate concerns around self self-referentiality and bias, as supported by both human comparisons (Alt-test) and consistency checks.
3.2 MULTIPLE-CHOIE QUESTIONS
Appendix shows the prompt used for answering MCQs. It first provides the course title indicate the relevant domain, then guides the LLM through standard legal reasoning steps such as clarifying facts, identifying issues, and explaining rules.
Evaluation Metric. We use accuracy scores for the MCQ evaluations. Since the choice label distribution is balanced through permutation, accuracy reflects unbiased performance of L.Mss. For example, random predictor will achieve approximately 25% accuracy in MCQ dataset with four answer choices. To assess the stability of the aggregated accuracy, we also compute its standard error by bootstrapping across different sampling of subsets. As robustness check, we also evaluate subset of frontier models on perturbed MCQs (see Section 2.1.2).
3.3 LLM SETTINGS
Model Selection.
For open questions, we evaluate 35 LLMs across three representative categories:
(1) Reasoning LLMs include DeepSeek- V..raaooee (Liu al.
2025), DeepSeek- V3.2- Exp
(DeepSeek-Al 2025), DeepSeek- R1 (DeepSeek-Al 2025a). GPT-5, -mini, and -nano (OpenAl
2025b), -.5mni-2.5-PPr (Gemini 2025),
Claude-3.7- (Anthropic 2025
and 4.5-Sonnet (Anthropic 2025), GPT-OSS-20B and -120B (Agarwal al. 2025). 03-mini
OpenAAl 2025d). QwQ-32B (Qwen 2024). Qwen3 32B and 235B (Qwen) 2025). and Qwen
Next (Al 2025ap; (2) Large models include GPT-4.1 (OpenAT, (025a). GPT-4o (OpenAl 2024a),
DeepSeck-V3.11
(Liu et al. 2025). DeepSeek-Ve V3 (DeepSeck-Ae All (025b). Llama-4-
(Meta AI 2025), and -lama-3-Instruccee
with 70B and 405B parameters (Grattation al. 2024);
and (3) Small models include GPT-4. I-mii, and -nano (OpenAl 2025a). GPT-4o-mini (OpenAl
2024b). Gemma-3-12BB B-12-it (Gemmal 2025), Gemma-2-9p 2-9--i (Gemma) (0024). Phi-4 (Abdin al.
2024). EuroLLMM-9Biit (Martins et al. 22024. Qwen-2.5-7B-it (Qwen) 2025). Ministral-8B-i (Mistral
All 2024), Llama-3. 1-8B--i (Meta-Ilama] 2024)), and Apertus-70B and -8B (Apertus al. 2025). We
select well well-performing flagship LLMs from diverse families (c.g.. Llama, GPT) and also prioritize those with strong multilingual abilities (c.g., EuroLLM, Gemma-3) due to LEX AM'ss multilingual nature. For MCQ-4 we selected subset of the aforementioned models; for MCQ-16. we selected subset of the aforementioned models and several newer models, including GPT5.2 (OpenAl 2025c Claude-4.6-Sonnet (Anthropic 20226), Qwen3.5-plus (Al 2025b), GLM-5 [Z.Al) 2026), Kimi-k2.5
(Moonshot Al 2026), Gemma-3-27B and MiniMax-m2.55 (MiniMaxAl 2026).
Inference Hyperparameters. For conventional LLMs, we set temperature to 0 and max length to 4096, which sufficient for complete answers. We adopt the lighteval framework (Fourrier al.) 2023) to standardize the evaluation of conventional LLMs with different endpoints. Reasoning models have more diversified sets of hypepparameters and are not supported by lighteval at the time of writing, so we follow the official recommended settings. We document detailed settings in Section
RESULTS
4.1 RESULTS ON OPEN QUESTIONS
https://deepm/ goog lemgeee/mmccoo e/model stteel////eeiiiieeeeeee/ee /npro//

---

Published as a conference paper at ICLR 2026
Table 1: Performance of models on long- form open questions with bootstrapped stan- dard error. All questions are graded by en- semble LLM judges (GPT-40, DeepSeek- V3, & Qwen3-32B), Results are sorted by Judge Score in descending order.
We show our main results open questions in Table] [ SOTA reasoning models, such as GPT-5 and Gemini- 2.5-Pro, achieve the best performance, with GPT-5 showing the highest average score (70.20). Other rea- soning models, namely -laude,3.7-Sonnet, Claude-4.5- Sonnet, GPT-S-mini, and DeepSeek reasoning mod- els. are also competitive among all the models, high- lighting the strength of models explicitly optimized for reasoning tasks. Among the large conventional LLMs, GPT-4.1 and GPT-40 perform strongly. with 57.50 and 56.93, respectively, followed by DeepSeek models and Llama-4-Maveriick Small LLMs gener- ally exhibit the lowest overall performance. However, GPT-4.1-mine stands out with average score of 54.58. Interestingly, Gemma-3-12-it achieves a score of 41.22 41.29, comparable to Llama-3.1 1lama-3.1-405B--1 (43.14) and Llama- 3.3-70B-it (41.27), despite being approximately 33x and 6x smaller in size, respectively. This strong per- formance may be attributable to the model's partic- ular focus on multilinguality. Moreover, smaller or older-generation LLMs, including reasoning models such as QwQ-32B and O3-mini, as well as conven- tional models like GPT-4o-mini, 1aama-3.14055P and 1aama-3.3-70B-.
Model
Open Questions
Judge Score (4 S.E)
GPT-5
70.20 (= 0.41)
Gemini-2.5-Pro
67.40 (# 0.51)
Claude-3.--Somee
62.86 (+ 0.51)
Reasoning
62.76 (2: 0.43)
GPT-5-mini
60.32 (# 0.45)
DeepSeek-
57.42 0.45)
DeepSeck-V33 V3.2-reasoner
56.53 (a 0.45)
DeepSeek-R
55.91 (+55)
Gemini-3- P-epni---rr--eeeeee P-oppeeee
55.38 (1 (1.6)
GPT-OSS-120B
51.74
03-mini
(=
Qwen3 235B
47.25 (1466)
QwQ-32B
(4
Qwen3-Next
43.37
Qwen3-32B
(2.00)))
GPT-OSS-20B
32.12 (= (.337)
GPT---naao
27.25 (+ 0.63)
GPT-4.1
$7.50 (# 0.51)
GPTT-4
56.93 (+ 0.48)
generally perform worse than flagship LLMs. Notably, bootstrapped standard errors remain low across all groups, suggesting that the LLMs perform consistently across different subsets of the dataset. We provide a more in-depth qualitative analy- sis of common error patterns in Appendix
Large
DeepSeck-V3.22 -3.22cck
(s
DeepSeck-VS
Llama-4-Maverce
(+
-Iama-3.1-405B-it
43.14
Llama-3.3-70B-it
41.27 (+41)
Apertus-70B
34.70
GPT-4.1-mini
54.58 0.44))
GPT-4.1-nano
43.68 0.41)
Figure 4 shows average LLdeJ scores across three dimensions: Language, Legal Area, and Jurisdiction. Additional features appear in Appendix C.I (Figure D). Reasoning models consistently outperform large and small models universally. For Language, all groups perform better on English than German courses, with small models showing the largest gap. The Hnglish- German performance gap likely reflects both linguistic and legal reasoning differences. Since the questions are not parallel translations, disentangling these factors would require high-quality legal translation, which an important but out-of-scope direction we discuss in Appendix K In Legal Area, interdisciplinary and public law topics score higher than criminal and private law. For Jurisdiction, generic and international law tasks achieve higher accuracy than Switzerland-spe specific ones.
GPT-4o-mini
(2
Gemma-3-12B-it
41.29 (+4.8
Small
Phi-4
38.54 (+42))
Gemma-2-99-ii
27.41 0.37)
EuroLLM-99B
22.95 0.35)
Apertus-8B
22.44 0.41)
Qwen2.5.7BB
16.67 (+ 0.29)
Ministral-8B-it
14.88 (# 0.32)
I.lama-311-8
10.00 (+ 0.26)
4.2 RESULTS ON MULTIPLE-CHOICE QUESTIONS (16 CHOICES)
As shown in Table 9 reasoning models lead in accuracy, with GPT-5.2 (52.53%) and Claude-4.6- Sonnet (52.42%) achieving the top two scores. These are closely followed by Qwen3.5-plus (43.77%) and DeepSeek V3ea-neas 151.),%) which also demonstrates strong competitive performance. Among large and small models, 1lama-3.3-70B-- (26.07%) and Phi-4 (21.40%) outperform the rest, while all others in the two categories score below 20%. Across all groups, bootstrap standard errors are low (typically below 1.5%), indicating stable performance estimates.
Figure S shows model accuracy across five dimensions: Language, Legal Area, Jurisdiction, Num- ber of Statements (legal propositions preceding multiple choice options), and Question Polarity (affirmative vs. negative phrasing). Further results on MCQ-4 appear in Appendix Figu I0 Reasoning models consistently outperform others, with the largest gap in English-language courses
We will present results from future models https: //1 www.eeexaaeeeeeeeeeee wwam-benchharkk.ithhhee
1leaderboard
7

---

Published as a conference paper at ICLR 2026
(A), international jurisdictions (C), and affirmative questions (E). All models show accuracy drops for German courses, Swiss law and negative questions, while public law yields lower accuracy than private or interdisciplinary law (B). When the number of statements increases from 4 to 5 (D), reasoning models maintain comparatively stable performance while large and small models degrade, indicating sensitivity to prompt length. Interestingly, all models exhibit significant performance decrease when the MCQ is phrased negatively. This effect is particularly pronounced for reasoning models, which is counterintuitive. The performance of small models on negatively phrased questions is approximately random. All bars indicate bootstrapped standard errors.
Language
Logal Area
Jurisdiction
# Statements
Question Polarity
Actunty
40
Reasoning Large
Small
Reasoring Large
Small
Reasoning ing Large
Small
Reasaning Large
Small
Reasoning Large
Small
(C)
(D)
Figure 5: Model performance on MCQ-16 grouped by various metadata.
4.2.1 ROBUSTNESS CHECK ON CULTIPLE-CHOICE QUESTIONS WITH PERTURBATIONS
The experiments on MCQs with perturbations were conducted using seven selected models: DeepSeek- RI, DeepSeck-v3, Gemini-2.5-PPr 03-mini, GPT-4.1, and GPT-4o. The models were tested using the same combinations of ques- tion stems and statements, with a vary- ing number of choices: 4, 8, 16, and 32. The accuracy results, summarized in Table 2 and illustrated in Figure clearly indicate consistent trend across all settings. Specifically, the accuracy decreases significantly as the number of answer choices increases.
Language
jurisdiction
50300/ing
Large
Small
Ressoning
Large
Small
Reasoning
Large
Small
Figure 4: Model performance on open questions grouped by metadata.
In all settings tested, the models show significant performance differences, further confirming the dataset's effectiveness in distinguishing between models with different capabilities. Further details are summarized in Appendix D These results indicate substantial variability in robustness and accuracy among the evaluated models, highlighting risks associated with performance sensitivity stemming from reliance spurious signals and guessing behavior. Moreover, these findings urge caution when benchmarking LLMs using standard MCQs, as this approach may produce overly optimistic results.
EXPERT EVALUATION FOR OPEN QUESTIONS
Evaluating open-ended legal questions presents a complex challenge. Prior work has increasingly turned to LLM-based eval- uators ("LLM-as-a-judgee"" yet concerns remain about their reliability in high- stakes domains such as law. To address this, we introduce an ensemble LLM judge (see Section 3.1 and Appendix G.4 for more details) based on GPT-4o, Qwen3-32B, and DeepSeek-V3. V3. To rigorously validate its use, conduct expert evaluation. This analysis both benchmarks human annotator agreement and assesses whether our LLM judge can serve as a statistically grounded proxy for expert annotation using the Alt-test (Calderon al. 2025).
Table 2: Model Accuracy (Acc.) and Bootstrap Standard Error (S.E.) across varying context lengths as percentages.
Model
4 Choices
8 Choices
16 Choices
32 Choices
Gemini-2.5-Ph
68.61 (+ 2.37)
51.56 (+ 2.55)
45.24 (# 2.43)
35.62 (+ 2.30)
Claude-3.. 3.7-So
60.92 (#: 2.48)
48.59 (#: 2.44)
40.38 (# 2.43)
33,02 2.31)
DeepSeek-R1
57.54 2.49)
44.11 2.45)
36.94 2.36)
24.93 2.21)
GPT-4.11
58.02 (1: 2.53)
42.31 (1: 2.51)
33.23 2.38)
26.30 2.13)
GPT-4o
$3.73 (..56)
36.42 (.37)
22.55
(= 2.09)
21.81 (# 2.18)
V3epSeek-V3
58.57 (+ 2.57)
36.07 (+ 2.40)
28.92 2.28)
16.03 1.88)
o3 63-mini
50.02 2.55)
33.54 2.34)
24.46 2.16)
17.01 1.89)
8

---

Published as a conference paper at ICLR 2026
5.1 INTER-ANNOTATON AGREEMENT
We first measured the consistency of three legal experts in rating 50 randomly drawn question-anne answer pairs on a 0-10 scale. All three hold Swiss Master's degree law and are pursuing or have completed doctorate in Law. The experts scored all 50 items blindly and independently, following only brief calibration instructions (see Appendix G.1 for details).
The three legal experts achieve an average Pearson correlation of r = 0.70. indicating strong linear consistency in their ratings. The mean quadratic weighted x 0.49 further confirms substantial agreement. A mean absolute error of 1.95 points on the 0-10 scale implies that expert judgments deviate by less than two points on average. For the full pairwise breakdown, see Appendix G.3
5.2 ALTERNATIVE ANNOTATOR TEST
To rigorously assess whether our ensemble LLM judge can replace human experts, we applied the Alt-test (Calderon al., 2025 with ==0 (i.e. no bonus tolerance margin for LLM judges). Following their guidelines, we used a leave-one-out design with m = 3 experts and n=50 items. We reused the exact same 50-item random sample and three expert annotators described in Section 5.1
In each of three leave-one-out trials, we compared our ensemble LLM judge's ' distance to the mean of the other two experts against that expert's distance, counted "wins" for the LLM versus the human, formed difference vector d, E +-1, and ran one-sided t-test on d < E. We then corrected the three raw p-values via Benjamini Yekutieli (BY) a=0.55.
Benjamini-Yekutillii
Our ensemble LLM judge sur- passes the recommended thresh- old of w 2 0.5. As Table 3 shows, it significantly exceeds the three experts, yielding win- ning rate w=1.00. Additional Alt-test results using other LLMs are provided in Section G.4
Table 3: Results of the Alternative Annotator Test with e 0.
LLM wins
Human wins
BY p
Adv. Prob
Legal Expert
36
14
0.001
0.002
0.72
Legal Expert 2
34
16
0.005
0.009
0.68
Legal Expert
13
0.000
0.001
0.74
Winning Rate
Taken together, our expert evalu-
ation establishes that annotators agree substantially among themselves (Pearson r=0.70,, quadratic weighted k=0.49, MAE=1.955 and that our ensemble LLM judge matches (w=1.00) three anno- tators even under the strictest no-bonus criterion (==0). These results validate our ensemble LLM judge as statistically justified replacement for open questions.
5.3 FURTHER ROBUSTNESS ANALYSES OF LLM JUDGES
We conducted additional quantitative and qualitative analyses to assess the robustness of judge models across languages and legal domains using four models (GPT-40, Gemini-2.5-Pro. DeepSeek-R 1, and Claude-4-Sonnet), Correlation tests on expert-annotated samples show that all models align closely with human scores in both English and German, with only minor and statistically insignificant language effects, indicating trnss-lnnguistic consistency (see T3bles[33anm of LLM answers revealed recurring failure modes, including flawed legal reasoning, references to incorrect or fictitious authorities, and weaker German or multilingual proficiency in smaller models, which often resulted in incoherent outputs. We also observed systematic differences in leniency: GPT-4o was the most generous, Claude-4-Sonnet and Gemini-2.5-PH5 were stricter, and DeepSeek-BR 1 fell in between. Notably, the largest gaps between model and human evaluations appeared in cases with higher human disagreement, underscoring the inherent difficulty of borderline legal judgments. These results further motivate the use of ensemble judge with minimum-soo score integration. More details are available in Appendices H and
Qualitative inspection
6 RELATED WORK
6.1
INTERNATIONAL BENCHMARKS
Multiple international benchmarks for legal reasoning have been established, comprising various individual datasets, primarily sourced from China and the United States. Examples are LegalBench

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of LEXAM prior legal QA datasets. Answer type is either true-false (TF), multiple-choice (MC), or long-form (LF), where we specify the average answer length in whitespace words in brackets. CH is short for Switzerland. Int. is short for International.
Benchmark
License
Jurisdiction
Languages
Legal Domain
# Examples
Answer Type
Housing Statutes
Unknown
English
Statulory Housing Law
6,853
TF
Unknown
English
Liability
Brazilian
Unknown
Pariuguese
2,130
COLIEE
CCBY 1.0
Japan. Canada
Japanese, English
Japanese Civil Law, Canadian Cass Law
1,778 MC
GLOBALCIT
CC BY 4.0
International
English
International Chitennshp Lam
9,310 MC
JncQA
CC BY-NC-ND
China
Chinese
Chinose Law
21,072 MC
MMLU Subwer)
BY
Primarity
General Legal Knowlodge
1,970 MC
Exum
Unknown
questions
1.195 MC
PrivacyQA
MIT
Global
English
Privauy Law & Duta Protection
1,750 (F4(40)
LEXAM (ours)
CCBY 4.0
CH. EU & Int.
German, English
78 Subdomains
4,886
1F (248) & MC
Guua al. 2023 LawBench (Fei et al. 2023). LvxEval (Li et al. 2024), LEXTREME (Niklaus et al. 2023b or LexGLUE (Chaliddis et al. 2022 While they propel the field forward, they still fall short in the complexity of the tasks evaluated. In contrast, we introduce very challenging law exam questions, often requiring detailed, long-form answers.
In Table we compare LEXAM to prior legal QA datasets. We demonstrate that LEXAM a) covers a wide array of legal disciplines (78 subdomains), b) is the first general legal reasoning dataset containing challenging long-form ground-truth answers, and c) is the only dataset with an expert-tuned L.LM-as-a-judge system for evaluation.
Because existing legal NLP benchmarks focus on narrow, task-specific formulations, whereas LEXAM evaluates holistic exam-style reasoning, direct aligned comparisons would require sub- stantial reformatting and are not meaningful. We therefore provide descriptive comparisons and position LEXAM as complementary to these task-oriented datasets.
6.2 SWISS BENCHMARKS
Research on NLP benchmarks around Swiss legal data started with the task of legal judgment prediction, where models predict the case outcome based facts NNillaus et al. 202 2022).
Subsequently, many other tasks have been studied on Swiss court rulings, such as explainabilit
for judgment prediction (T....ss. et al., 2024), negation scope resolution Christen al. (2024).
sentence boundary detection (Brugger et al. [2023), citation extraction (Rasiah et al.) 2023). case
importance prediction (Stem al. 2024), law area prediction (Rasiah al. 2023). summarization of leading decisions (Rolshoven et al. 2024). drafting of considerations (Rasiah et al. 2023). translation
(Niklaus et al.) 2025). and the analysis of gender distribution in court decisions (Merane) (202I).
Additionally, prior work studied the anonymization (Niklaus et al. (202a)) and risk of re-identification of involved persons court rulings (Nyffenegger et al. 2024).
Most prior tasks focus on content from court rulings and, to some extent, legislation. To our knowledge, no Swiss legal dataset from law school exams exists. In much prior work, labels have been extracted through regexes instead of manual human annotation (Merane & Geering 2023). While regexes allow for very large-scale datasets, the covered tasks may not as natural compared to fully -annotatdd datasets. In this work, collect large dataset of worl-world university law school exams, including detailed reference responses written by leading law professors.
7 CONCLUSION
We introduced LEXAM, a comprchensive legal reasoning benchmark composed of 340 law exams, specifically designed to challenge the performance of LLMs through both outcome- and process- based evaluations. Our analysis reveals substantial variability and limitations LLM capabilities for addressing MCQs and especially complex open questions; notably, increasing the number of MCQ options consistently reduces model accuracy. Our evaluation framework offers a scalable approach for assessing legal reasoning quality beyond simple accuracy metrics, thereby facilitating future research aimed at enhancing the reliability and robustness of LLMs on challenging legal tasks. Plans for further data collection are detailed in Appendix
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work introduces a benchmark for evaluating LLMs on legal reasoning tasks using law school exam data. All exam questions and reference answers obtained with permission from the University Zurich Faculty of Law and are used exclusively for research purposes. The dataset consists solely of publicly available academic materials, namely professor-written exam questions and model solutions, and excludes any personal, confidential, or sensitive content, such as student-writeen answers.
Because the dataset contains no personal or human subject data, no anonymization procedures were necessary, and no institutional ethics board review was required under Swiss research regulations. This assessment was confirmed through consultation with multiple legal experts.
Our analyses focus exclusively on model performance and do not constitute legal advice. Moreover, while the benchmark reflects authentic legal reasoning tasks, it is intended purely as research and educational resource, not as a tool for any real-world legal practice. Given the high-stakes nature of the legal domain, we strongly emphasize that current LLMs must not be relied upon for legal decision- making without qualified human oversight.
REPRODUCIBILITY STATEMENT
To facilitate reproducibility and future research, we release both the full LEXAM dataset and the complcte evaluation pipeline. Specifically, (a) the datasct-which includes all exam questions, metadata, and reference materials and (b) the evaluation code are provided in the anonymous repository and uploaded supplementary materials. Together, these resources enable replication of our results and support extensions of the benchmark new models, tasks, and evaluation methods.
ACKNOLEDGEMENT
We thank Benjamin Chen, Alexander Hoyle, Luka Nenadic, and Uriel Stettner for their very helpful comments throughout this research.
REFERENCES
Marah Abdin et al. Phi-4 technical report, 2024. URL
08905
Sandhini Agarwal, Lama Ahmad, Jason Ai. Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.1009 2025.
Qwen AI. Latest advancements. Blog post, October 23 2025a. URL https
tecearch__attess-caaaaccceeeeeee //tpe/.aa/loog0223077/3333333313 029333555000 71183f9cbdd2oooooo 22d777dd.. from-
Available at https://qwen.at [aggg:40022000311102228803333132 Fronrerearrddrrtrreeeeeeeeea.. Latest-adlvancements--111
wwnn AI. Qwen3.5: Towards native multimodal agents. 2025b. URL
Blog post, February 15
Available at https
629836 weceeccbd2ddece 7d27cds romm
research. 1.....attee latest wwwwwadwwwwwaaaaaa a 1ist
Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13787-13805, 2024.
Anthropic. Claude 3.7 sonnet and claude code,
2025. /
com/ com/news/ news//wwwww/eeeeee/
c aude 7- sonnet Accessed: 2025-04-30.
11

---

Published as a conference paper at ICLR 2026
Anthropic. Claude sonnet 4-5: System card.
http:://asett..attrroo.... www.c/cooommmmmmmmmmmmm/ 2025.
Accessed: 2025-10-23.
Anthropic. Claude sonnet 4-6: System card.
https://www-coh www..cc...ntrrii..co....
pdf] 2026. Accessed: 2025-02-27.
Project Apertus, Alejandro Hernández-Camg Alexander Hägele, Allen Hao Huang, Angelika Ro- manou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Durech, al. Apertus: Democratizing open and compliant Ilms for global language environments. arXiv preprint arXiv:2509. 14233, 2025.
Elliott Ash, Aniket Kesari, Suresh Naidu, Lena Song, and Dominik Stammbach. Translating legalese: enhancing public understanding of court opinions with legal summarizers. In Proceedings of the Symposium Computer Science and Law, pp. 356,157,5 2024.
Emmanuel Bauer, Dominik Stammbach, Nianlong Gu, and Elliott Ash. Legal extractive summariza- tion of us court opinions. arXiv preprint S.Xiv:2305.08422 2023.
Tobias Brugger, Matthias Stürmer, and Joel Niklaus. MultiLegalSBD: A Multilingual Legal Sentence Boundary Detection Dataset. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL '23, pp. 42-51, New York, NY, USA, September 2023. Association for Computing Machinery. ISBN 9798400701979, doi: 00.1145/3594533.3555133
httt:t/:: /dl acm. wwtpmmo/g///o///ooooo 1111115555555 11453359533.33511333 3595132
Nitay Calderon, Roi Reichart, and Rotem Dror. The alternative annotator test for llm-as-a-judge: How to statistically justify replacing human annotators with llms. arXiv preprint arXiv:2501.10970, 2025.
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 4310 4330--333, 2022.
Odysseas S Chlapanis, Dimitrios Galanis, and Ion Androutsopoulos. Lar-echr: A new legal argument reasoning task and dataset for cases of the european court of human rights. arXiv preprint 1333:22410.13352, 2024.
Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physbench: Benchmarking and enhancing vision-language models for physical world understanding. CoRR, abs/2501.16411, 2025. doi: 10.48550/ARXXV22511.646 18550/arXiv wtt.aaiwii........... 16411
URL
Ramona Christen, Anastassia Shaitarova, Matthias Stürmer, and Joel Niklaus. Resolving Legalese: Multilingual Exploration of Negation Scope Resolution in Legal Documents. In Nicoletta Calzolari, Min-Yen Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 13992-14004, Torino, Italia, May 2024. ELRA and ICCL. URL
2022 2024. Irec ma.i.. 1220
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning,
2025a.
22999 12948
DeepSeck-AL. Deepseck-v3 technical report, 2025b. URL bttts::/aaxiiii: /RL_/tpps////axxvv..r//////////
19437
Dee Deepseek-v3.2-eyg Boosting long-context efficiency with deepseek sparse at- 2t.
tention. https: DeepSeek V3 2. pdf 2025. Accessed: 2025-10-23.
Wentao Deng, Jiahuan Pei, Keyi Kong, Zhe Chen, Furu Wei, Yujun Li, Zhaochun Ren. Zhumin Chen, and Pengjie Ren. Syllogistic reasoning for legal judgment analysis. Proceedings of the 2023 Conference Empirical Methods in Natural Language
ProsieePPorr Processing, pp. 13997-14009, 2023.
12

---

Published as a conference paper at ICLR 2026
Gaspar Dugac and Tilmann Altwicker. Classifying legal interpretations using large language models. Artificial Intelligence and Law, pp. 1-19, 2025.
Christoph Engel and Richard H McAdams. Asking gpt for the ordinary meaning of statutory terms. U. IIl. JL Tech. & Pol'y, pp. 235, 2024.
Christoph Engel, Yoan Hermstrüwer, and Alison Kim. Human realignment: An empirical study of IIms as legal decision-aids in moral dilemmas. MPI Collective Goods Discussion Puper, (2025/3), 2025.
Yu Fan, Yang Tian, Shauli Ravfogel, Mrinmaya Sachan, Elliott Ash, and Alexander Hoyle. The medium is not the message: Deconfounding text embeddings via linear concept erasure. arXiv preprint arXiv:2507.01234 2025.
Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. Lawbench: Benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309.16289 2023.
Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, and Arman Cohan. Physics: Benchmarking foundation models on university-level physies problem solving. arXiv preprint arXiv:2503.21821, 2025.
Clémentine Fourrier, Nathan Habib, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. Lighte- val: A lightweight framework for IIm evaluation, 2023. URL https://githuttt com/
huggingface/ lighteval
Gemini 2025,
Gemini 2.5: Our most intelligent AI model.
Gemini 2.5 Pro.
URL https::/blog. wwwg:///eeooooeeeoeeoeooooeeeeee Btteer//bccc..ooe////eeeoooo/o// http://cchcoooroooooooeeeeeeeeee
gemini sondel-thinking wtinking--ppateee...... updates
2025/ -
Prompt Description. Gemini 2.5 Pro. Google, April 30, 2025.
Gemma. Gemma 2: Improving open language models at a practical size, 2024. URL https
//axxiv.org/abbt sttii...rr/abs//448... 00118
Gemma. Gemma technical report, 2025. URL htr/:/tpp///aaa...o//////////1
Aaron Grattafiori et al. The llama 3 herd of models, 2024. URL. httppss//aaxx...rr//////
24C7.21783 21783 arXiv:2407.21783
Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wooke Austin Peters, Brandon Waldon, Daniel Roekmore, Diego Zambrano, et al. Legalhench: collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36:44123-44279, 2023.
Cong Jiang and Xiaolei Yang. Legal syllogism prompting: Teaching large language models for legal judgment prediction. In Proceedings of the nineteenth international conference on artificial intelligence and law, pp. 417-421, 2023.
Zhijing Jin, Max Kleiman Kleiman-Weiner, Giorgio Piatti, Sydney Levine, Jiarui Liu, Fernando Gonzalez Adauto, Francesco Ortu, András Strausz, Mrinmaya Sachan, Rada Mihalcea, Yejin Choi, and Bernhard Schölkopf. Language model alignment multilingual trolley problems. In The Thirteenth International Conference Learning Representations, 2025. URL htttp:://oppeeeeeeeeee
Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovee, Jens Ludwig, and Sendhil Mullainathan. Human decisions and machine predictions. The quarterly journal of economics, 133(1):237-22) 293, 2018.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposiun on Operating Systems Principles, 2023.
13

---

Published as a conference paper at ICLR 2026
Kwok- Yan Lam, Victor CW Cheng, and Zee Kin Yeong. Applying large language models for enhancing contract drafting. In LegalAllA@ ICAIL, pp. 70-80, 2023.
Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun Liu. Lexeval: A comprehensive chinese legal benchmark for evaluating large language models, 2024. URL
fitps://arxiv. 2ttp:///////////022222223 20288
Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pp. 74 81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
TtRLhttps:// acc www.eaannnhohoooooooooooooooeeee 20eoo/////411000 W04/
Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseck-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.0255:
2025.
Chengwu Liu, Jianhao Shen, Huajian Xin, Zhengying Liu, Ye Yuan, Haiming Wang, Wei Ju, Chuanyang Zheng, Yichun Yin, Lin Li, Ming Zhang, and Qun Liu. FIMO: A challenge formal dataset for automated theorem proving. CoRR, abs/2309.04299 2023. doi: 10.48550/ARXIV. 2309.04295. htt.t::://do..oggg/ett 48w.://............/////////../.
04295
Kangcheng Luo, Quzhe Huang, Cong Jiang, and Yansong Feng. Automating legal concept inter- pretation with llms: Retrieval, generation, and evaluation. arXiv preprint arXiv:2501.0174. 2025.
Shengjie Ma, Chong Chen, Qi Chu, and Jiaxin Mao. Leveraging large language models for relevance judgments in legal case retrieval. arXiv preprint arXiv:2403.18400 2024.
Robert Mahari, Dominik Stammbach, Elliott Ash, and Alex'Sande Pentland. The law and nlp: Bridging disciplinary disconnects. arXiv preprini arXiv:2310.143444 2023.
Robert Mahari, Dominik Stammbach, Elliott Ash, and Alex 'Sandy' Pentland, LePaRD: A Large- Scale Dataset of Judges Citing Precedents, October 2024. http://////ax....coooo wwwpwwiwwiorrggaoa..o org/abs///// arXiv:231 1.09356 [es].
2311. 09356
Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski. Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. Martins. Eurollm: Multilingual language models for europe, 2024. URL https::tttaatttt.e org/ abs /205/2409922 16235
Masha Medvedeva and Pauline Mcbride. Legal judgment prediction: If you are going to do it, do it right. In Proceedings of the Natural Legal Language Processing Workshop 2023, pp. 73-84, 2023,
Jakob Merane. Computergestützte Ermittlung der Geschlechterverteillngg im bundesgerichtliceen Verfahren: Resultate einer Studie. Schweizerische Juristen- Zeitung, (10):511-516. 2021.
Jakob Merane and Florian Geering. Bundesgerichtsurteile nutzen, Das Swiss Federal Supreme Court Dataset (SCD). Jusletter, 2023.
Meta AI. The llama 4 herd: The beginning of new of natively mul- timodal innovation. April 2025.
URL https: meta. cww/blog///oooooooooooooooooooo. Accessed: 2025-04-30.
1lama- mi-muii:in
Meta-llama. Llama 3.2: Revolutionizing
edge ai and vision with open,
customizable models, 2024.
URL https://al
/log/
MiniMaxAI. Minimax-m2.55 model card, Februrary 2026. URL/tttps./hugiiggacccee
Accessed: 2026-02-27.
Mistral AI. Ministral-Sb--instuut--21100
maristral-- 88--888-88 - 2410 2024. Accessed: 2025-09-24.
Moonshot AI. Kimi k2.5 model card, January 2026.
URL http::/hugiinfacc.cc
moonshotai - Accessed: 2026-02-27.
14

---

Published as a conference paper at ICLR 2026
Joel Niklaus, Ilias Chalkidis, and Matthias Stürmer. Swiss- Judgment-Predicceeeee A Multilingual Legal Judgment Prediction Benchmark. In Proceedings of the Natural Legal Language PringgeePrroee Processing Computational Linguistics. Utt.ttps htt.httts://ccct antho I ogy wow..orgg..oooo................. www.ogg..........o org/2021
Workshop 2021, pp. 19-35, Punta Cana, Dominican Republic, November 2021. Association for
n n1lp-1..1...
Joel Niklaus, Matthias Stürmer, and Ilias Chalkidis. An Empirical Study Cross-X Transfer for Legal Judgment Prediction. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 32 46, Online only, November 2022. Association for Computational Linguistics. URL https: http://aaaaatttoooo...oo. org/
a222.aaclmmain
Joel Niklaus, Robin Mamié, Matthias Stürmer, Daniel Brunner, and Marcel Gygli. Automatic Anonymization of Swiss Federal Supreme Court Rulings. In Daniel Preotrextcommaaeloiiue
Pietro, Catalina Goanta, Ilias Chalkidis, Leslie Barrett, Gerasimos Spanakis, and Nikolaos Aletras (eds.), Proceedings of the Natural Legal Language Processing Workshop 2023, pp. 159- Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 10.1833311/202. htt.ttps::
nl1p-1.16. https: a wallllnnthoolog www.orrg/222 111.:.nn1pp11111
Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. LEX- TREME: A Multi-Lingual and Multi- Benchmark for the Legal Domain. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3016-3054, Singapore, December 2023b. Association for Computational Lin- guistics. doi: 10. 10.118633/11203.. http:////1212222/nnnnnnneepppppp findings-emmlp.200, URL hhttps::/aclantholoooy. org/ 2023. findings emnlp.20..
Joel Niklaus, Jakob Merane, Luka Nenadic, Sina Ahmadi, Yingqiang Gao, Cyrill A. H. Chevalley, Claude Humbel, Christophe Gösken, Lorenzo Tanzi, Thomas Lüthi, Stefan Palombo, Spencer Poff, Boling Yang, Nan Wu, Matthew Guillod, Robin Mamié, Daniel Brunner, Julio Pereyra, and Niko Grupen. SwiLTra-Bench: The Swiss Legal Translation Benchmark, March 2025. URL http://arxiv. hwwp:/wwwwrrreeee abs /203 01372 arXiv:2503.01372 [cs].
Alex Nyffenegger, Matthias Stürmer, and Joel Niklaus. Anonymity at Risk? Assessing Re- Identification Capabilities of Large Language Models in Court Decisions. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2433-2462, Mexico City, Mexico, June 2024, Association for Computational Lin- guistics. doi: 10.466555//0004mmmmmmmm
naall.s- URL /tt.s////.ttt///o.oooooooooo.//
org/
2024. 12224.findins-maacll
OpenAl. Hello gpt-4o, May 2024a. /ttpp//el////eeeaa..c//////eeeea 40/ Accessed: 2025-04-30.
OpenAl. mini: Advancing coste
http///ooooooooooiooooeooooeoenn ogg.::/wwp....aaaaa..oooooooaaaa 2025-04-30.
2024b. Accessed:
OpenAl. Openai o3 and 04-mini system card. -ns
2221c875- 0255-002c-77------ 728
pdf 2025.
OpenAl Introducing gpt-4.1 in the api, April 2025a. URL /tLppttpp///geeaa.ccm/nneee
4-t-4417 Accessed: 2025-04-30.
OpenAl. Gpt-5 system card, August 2025b. URL httpp://cch..peeaal.coom gpt 5- systtm-cardddo
Accessed: 2025-09-15.
OpenAl. Update to gpt-5 system card: Gpt- December 2025c. URL URLhttts:/oppeaa.cooo Accessed: 2026-02-27.
hnte:///p---sssyttttoooooeeueeee 5-2/
OpenAl. Openai 03-mini. January 2025d.
URL http:://peeat.coommmm
openai -o3-mini/ Accessed: 2025-04-30.
15

---

Published as a conference paper at ICLR 2026
Arjun Panickssery, Samuel Bowman, and Shi Feng. Lim evaluators recognize and favor their generations. Advances in Neural Information Processing Systems, 17766772-68802 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. ACL '02. Pp. 311-318, USA, 2002. Association for Computational Linguistics. doi:
1073135. URL utthttps:: /oo. 1073135 event-place: Philadelphia, Pennsylvania.
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. arXiv preprint arXiv:2308. /483, 2023.
Qwen. Qwq-32b: Embracing the power of reinforcement learning. https://qwen.n. github 1o/blog/ qwq 32b/ 2024. Accessed: 2025-04-30.
Qwen. Qwen2.5 technical report, 2025. URL Ttt.ttppp./aarr..oor/a////22221.
Qwen. Qwen3 technical report, 2025. BBlp:/ttsss/aaxxxx.rr//////////2
Vishvaksenan Rasiah, Ronja Stern, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, Daniel E. Ho, and Joel Niklaus. SCALE: Scaling up the Complexity for Advanced Language Model Evaluation, June 2023,
092377 arXiv:2306.09237 [es].
ZZ Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, al. Deepseek- prover-v2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition. arXiv preprint arXiv:2504.21801, 2025.
crepseek-prover-22
Luca Rolshoven, Vishvaksenan Rasiah, Srinanda Brügger Bose, Matthias Stürmer, and Joel Niklaus. Unlocking Legal Knowledge: A Multilingual Dataset for Judicial Summarization Switzerland, October 2024. URLhtttp://arxx www..oog/aas.ss...e.ae.... /obbss220 200 13456 arXiv:2410.1 134456 [cs].
Michael J. Ryan, William Held, and Diyi Yang. Unintended impacts of LLM alignment on global representation. In Lun-We Mei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 16121-16140. Association for Computational Linguistics, 2024. doi: 10. 853
URL |RLhttps
7/doi.org/I00 1ttp://1...1153/11/2///000
Jaromir Savelka, Kevin D Ashley, Morgan A Gray, Hannes Westermann, and Huihui Xu. Explaining legal concepts with augmented large language models (gpt-4). arXiv preprint arXiv:2306.099 09525, 2023.
Ruihao Shui, Yixin Cao, Xiang Wang, and Tat-Seng Chua. A comprehensive evaluation of large language models on legal judgment prediction. arXiv preprint arX.v:2310.1176. 2023.
Evangelia Spiliopoulou, Riccardo Fogliato, Hanna Burnsky, Tamer Soliman, Jie Ma, Graham Hor- wood, and Miguel Ballesteros. Play favorites: A statistical method to measure self-bias in llm-as-a-judge. arXiv preprint arXiv:2508.06709, 2025.
Ronja Stern, Ken Kawamura, Matthias Stürmer, Ilaas Chalkidis, and Joel Niklaus. Breaking the Manual Annotation Bottleneck: Creating Comprehensive Legal Case Criticality Dataset through Semi-Automated Labeling, October 2024. URL http:/tp//aaaxi.orrra/////////// 12Xiv22410.13460 [cs].
Rosamond Thalken, Edward H Stiglitz, David Mimno, and Matthew Wilkens. Modeling legal reasoning: Lm annotation at the edge of human agreement. arXiv preprint arXiv:2310.1844M 2023.
Yang Tian, Yu Fan, Liudmila Zavolokina, and Sarah Ebling. Investigating disability representations in text-to-image models. arXiv preprint arXiv:2602. 04687, 2026.
16

---

Published as a conference paper at ICLR 2026
George Tsoukalas, Jasper Lee, John Jennings, Jimmy Xin, Michelle Ding, Michael Jennings, Ami- tayush Thakur, and Swarat Chaudhuri. Putnambench: A multilingual competition-mattinn mathematics benchmark for formal theorem theorem-proving. In Al for Math Workshop @ ICML 2024, 2024. URL htttps://pperret
net/ forum? i V= VRFeVP
Santosh T.y.s.s.. Nina Baumgartner, Matthias Stürmer, Matthias Grabmair, and Joel Niklaus. Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset. In Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 16510, Torino, Italia, May 2024, ELRA and ICCL. URL httttp:://aaaaatool...too org/ feg/2024.22 -rec main. 1434
Xuran Wang, Xinguang Zhang, Vanessa Hoo, Zhouhang Shao, and Xuguang Zhang. Legalreasoner: A multi-stage framework for legal judgment prediction via large language models and knowledge integration. IEEE Access, 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024b.
Yongxuan Wu, Runyu Chen, Peiyu Liu, and Hongjin Qian. Livelongbench: Tackling long-context understanding for spoken texts from live streams. arXiv preprint arXiv:2504 2025.
Z.AL. Glm-5 model card, Februrary 2026. URL Accessed: 2026-02-27.
GLM-5
Yuheng Zha, Yichi Yang. Ruichen Li, and Zhiting Hu. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 11340- 11355. Association for Computational Linguistics, 2023. wttttssss//clleeeeeeeeeee
634
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Evaluating Text Generation with BERT. arXiv:1904.00 09675 [cs), February 2020. URL http: arXiv: 1904.09675.
httgg//bs//904.005500 09675
Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, and Jun Liu. Physreason: comprehensive benchmark towards physics- doi: 10.48550/RXIV.250222222
based reasoning. CoRR, abs/2502. 12054, 2025.
URL
https://dot http:://0oo..rg/110.88555///////
48550/ 2502. 12054
Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. minif2f: a cross-system benchmark for formal olympiad-level mathematics. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL net htttps://openevite wwt/ wtr.ooorumuuoooooo/ Pw
Lianmin Zheng, Wei Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin. Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging alm-as-a-judge with mt-bench and chatbot arena, 2023. URL httt:/s:///rxiiv.orgg abs/23306 2.366 05685
Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D Manning, Peter Henderson, and Daniel Ho. A reasoning focused oecusee legal retrieval benchmark. In Proceedings of the Symposium on Computer Science and Law on ZZZ, pp. 169-193, 2025.
Youchao Zhou, Heyan Huang, and Zhijing Wu. Boosting legal case retrieval by query content selection with large language models. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pp- 184, 2023.
17

---

Published as a conference paper at ICLR 2026
A USE OF AI ASSISTANTS
We used Al assistants, including GPT-4o, GPT-4.5, and GPT-5, for coding, shortening texts and editing LaTeX more efficiently. The Al tools are not directly used in writing, but assisting human through criticizing, grammar checks, etc..
B DATASET DETAILS
B. INSTITUTIONAL BACKGROUND
Switzerland's legal system belongs to the civil law tradition and shares many characteristics with other continental European jurisdictions (though with the important distinction that Switzerland is not a member of the European Union). Legal interpretation Switzerland focuses primarily on statutory texts. However, case law also plays supporting role, especially decisions from higher courts, which contribute to legal clarity and consistency. Nevertheless, legal education in Switzerland emphasizes deductive reasoning (applying general legal rules specific facts) rather than reliance precedent.
Switzerland has several law faculties, with the University of Zurich hosting the largest. Legal education in Switzerland follows the Bologna model, consisting of three-year Bachelor of Law (BLaw) and a two-year Master Law (MLaw). The Bachelor program is typically highly structured, with fixed curriculum covering the foundational areas of Swiss law. In contrast, the Master program allows for greater specialization and elective choices, enabling students to focus on specific legal domains. Upon graduation, students may pursue the bar exam, which organized the cantonal level. This exam places more emphasis on practical legal skills, such as drafting legal documents, applying procedural law, and demonstrating familiarity with cantonal and professional regulations.
B.2 DATASET STATISTICS
Open Questions (Test: 2,541, Dev: 300): This set contains 2,541 questions (17.2% English, rest German), averaging 317.6 questions/ycar (2016-2023). Key distributions include: Private Law (43.5%), Public Law (38.4%), Criminal Law (9.0%), and interdisciplinary (9.1 (9.1%); and Swiss jurisdiction (62.8%), international/non-Swiis (2).1%), and generic (17.1%). Average question/answer lengths are 174.3/246.6 words.
Open Questions Dev (300 questions): Sampled per course, this set of 300 questions mirrors the test set's 's distributions for legal area, jurisdiction, language, and question/answer length.
MCQs (4,696 questions): These questions are 62.8% German and 37.2% English. 91.5% target Swiss jurisdiction. Topics include: interdisciplinary (32%), Public Law (28.6F (2,6%) Criminal Law (27.0%). and Private Law (11.5%). Questions average 3.7 choices. Unlike open questions, MCQ distribution by year is uneven, peaking in 2019 and 2023 (approx. 500 each year).
MCQs Extended (385 questions): All 385 questions target Swiss jurisdiction. Distributions similar to the main MCQs: 64.4% German; Public Law (27.5%), Criminal Law (34.8%), Interdisciplinary (35.6%), with most samples from 2019 and 2023.
18

---

Published as a conference paper at ICLR 2026
Table Number of questions by Jurisdiction, Area, and Language
(a) Dev Dataset
Criminal
Interdisc.
Private
Public
Total
Jurisdiction
de
en
de
en
de
en
de
en
Generic
0
0
10
10
10
5
20
0
55
International
5
5
25
90
Swiss
20
0
0
80
5
50
0
155
Total
25
5
10
10
105
35
80
30
300
(b) Test Dataset
Criminal Interdisc.
Private
Public
Total
Jurisdiction
de
en
de
en
de
em
en
Generic
0
193
37
20
21
164
0
435
International
18
26
0
0
56
211
67
132
510
Swiss
185
0
0
1
789
8
613
()
1596
Total
203
26
193
38
865
240
844
132
2541
19

---

Published as a conference paper at ICLR 2026
Table 6: Number of records by Jurisdiction, Area, Course and Year on Open Questions (test)
Fusisdiction
Area
Course
2016
2018
2019
2420
2021
2022
2023
Totul
Gesede
Iusendisiplinarry
Juistische Zeitgeschichee
Lagal Sociology
Legal
Rachitigeschicbee
12
12
Privase
Antike Rochisgescichle
10
10
Hasory of Business Law
12
21
10
10
Public
Gegenwan repeolllee der poltischen Ethik
0
Kirchemechsgesschchee und Kirchenrecht Reeht und Religios
13
17
20
13
84
10
14
24
Reclitpphhoiippii
12
44
Nrrfasunggeschicee der Neuzeix
Internatoona
Criminal
Internatonal Crimital Law Rechishille Sunfsachhen
11
10
26
Fumpliszhes Stafrecht
Private
Comparative Corporate
Comporative
Europlisches Privtroctt
Foundations Trists
Internaional Commercial Acbitration
Internaional Sales Las
16
Intemationlls Piisttecct
10
36
Interntionals Zholberfshmmnoeccch
10
Introduction Sports Law
11
Principles of Corporate Lawe US Business Law
0
II
18
87
Public
Comporative Concinutioal Low
Duropean Economic Low
23
Intemational
International Human Righis
Interational Diganisatons
10
Interationals Sicueticele
Recht der Gevoltrrending und Homanitires Velkernechi Transnational Public Seeurity Law
Swis
Criminal
zeg und Sanktioneseehtt
22
15
21
22
80
Nebenstra frecht
9
21
47
Strafrecht und Strafverfshreeseec..
11
7
40
Wimschubbattffreeh
0
18
cipliaayy
(Iecrduccion Swiss Law
Privste
Aktieneccht
21
Altemative Streicheselligung
22
Arbeitsrcht
Bunkzechs
Grsellichulivehht
Grundbscrrecte
and Edbrccht
Halptlicht. Verichenungereett
Handels- Winschoftsrecht
Immaerialgiteemech
12
smmobliarschcurell
10
49
Inlocmations- und Kommarkktiasseeci
26
Kepitalmrkmmah
42
Kollequium zum Allgemeinen Teil des Obligationen rechts
0
O
O
11
Kund- und Kalturrchh
14
1I
12
18
64
Lizenzvertnigg und Lizenkkrteellrehhe
25
10
20
14
97
Nachhasplaming
IS
II
56
Notariaszecha
Wertaaieeecht
Demokratic
Gesundheissechi und Binrthik
Migratinsmech
IS
14
Raumplanings und Bourecht
10
45
Rechnsetnngslches
14
21
13
19
71
Sicherheits- Polizet,, und Memschenrechi Sodiahesicheengsseeet
7
9
9
12
10
9
68
10
16
14
12
12
13
14
91
Steuenecht
0
9
10
16
58
Enwwhhech
11
32
Untenehmennseeurmmch
63
Offentiches Nefahremsecht
20

---

Published as a conference paper at ICLR 2026
Table 7: Number of records by Jurisdiction, Area, Course and Year on Open Questions (dev)
Jarisdiction
Anzz
Coursc
2016
2017
2018
2019
2070
2021
2022.
2023
Textl
Gesedo
Iusediicipllnury
Legal Sociology
Legal Theory
Privase
History Rochnggschtchtt
Public
Kirchmnehtsgesshichhe Kirchennechi
Recht und Religioa
Rechnnghiloophic
Verfasongggeschhcche der Newraä
Internatonall
Criminal
Interational Criminal Law
Interaaionales und Europlisches Strufrecht
Privsse
Cove Privni Law
Puropsiacces Privatrecht
Poundations Trusts
Interntional Commercial Acbitration
Pricutrcht
Interaaionalss Zivibverahrensseect
Public
Europeиn Eosnnmic
Interntional Ecunomic Law
Internaional Finunce Law
Internaional Finsncial Law
Intemitional Hamna Rights
Internaioonl Organisations
Intemationalss Siguerieche
Recht der flowaltarwendingg und Humanitizes Vilkerechi
Swiss
Criminal
Ingendttrafrcch und Sankrionemrccht
Nebenstrufrecht
Strafrecht Strafeerfahreesrecht
Winschaatsstraliss
Privste
Aktienrecht
Altemative Strasbeeelligung
Arheltsrechi
Bankrecte
Gesellichotuecht
Grundbachreczz
Gilze and Edivecht
Halppfiict-- and Verschenungecele
Immohhlarsachenech
0
Infocmations und Kommunkatoeseechh
0
Kopitalmurktnclle
Kollequium zum Allgemeisen Teil des Cbligationcneehis
Kune- und Kalturreht
Lizendaatellreeee
Nachlsssplaung
Zevilvefiireemmeeh Burdevverwhingseeeit
Migrationenzht
Raumplanings und Baurecht
Rechneiruuplehrre
Sicherhelts. Polizii, and Memschnnechie
Sorishaaidde Snnisheniideemeesecc
Stevenecht
0
Umwelinecht
0
Intenchoenssseerace
Offentiihee Verfahrenseecht
Grand Total
21

---

Published as a conference paper at ICLR 2026
Question Length Diemibutioo
Length Distrbbtiion
Oed Moar: 222 Tue Mear: 1763
2er
2oe Roar 207
Tul-Maa 2416
0.305
0.330
0.00
0.310
0.331
0.392
0.300
Narder
Figure 6: Distribution of the number of words per question and answer in the open-ended question dataset.
Distribution of Areas by Year (test)
Area
Interdiocpiinryy
Private
125
Public
146
300
Number Racands
105
1:3
200
23c
155
144
100
:05
107
107
125
38
22
39
2016
2017
2016
2019
2020
2021
2022
2023
Year
Figure 7: Yearly distribution of the number of question in the Open question test dataset, per legal area.
C SUPPLEMENTARY RESULTS ON MODEL PERFORMANCE
C.1 OPEN QUESTIONS
This section presents additional analyses to complement the main results in Figure including Challenging Course, Top vs. Bottom Courses, Year, Question Length and Answer Length. The feature Challenging Course represents courses taught in German, focusing on Swiss law, and lacking an interdisciplinary approach. Following manual review conducted by legal experts from the author team, this feature serves as a proxy for legally intensive and advanced-level courses, which, while not all directly relevant to the bar exam, generally include fewer introductory courses and exhibit higher overall difficulty. The Top vs. Botton Courses feature constructed by computing the mean model score for each course and identifying the ten courses with the highest and lowest average performance, respectively. This allows for a contrastive view where models excel versus where they struggle most.
22

---

Published as a conference paper at ICLR 2026
Distribution of Areas by Year (mccqa_tess
Area
Interisciplinryy Wriste Public
500
400
Numbar nacands
302
S44
203
102
33
2026
2018
2019
2021
2022
2023
Figure 8: Yearly distribution the number question the MCQs dataset, per legal area.
Table 8: Number of the 385 MCQs with perturbations per course.
Course
Questions
(Introduction to) Swiss Law
137
Nebenstrafrecht
58
Steuerrecht
56
Unternehmmensstuerreeett
33
Strafrecht und Strafverfahhrensrecht
32
Jugendstrafrecht und Sanktionenrecht
24
Wirtschaftsstrafreehtt
20
Rechtsetzungslehre
16
Lizenzvertrags- und Lizenzkartellrecht
7
Raumplanungs- und Baurecht
Nachlassplanung
For Challenging Course (D) and Top vs. Bottom Courses (E), we observe a similar pattern to the main features, where reasoning models outperform large and small models. Additionally, accuracy is consistently higher for non-challenging courses, and all three model groups achieve significantly higher scores on the top 10 courses, while struggling on the bottom 10.
The feature Year (F) tracks model performance over time from 2016 to 2023, where lines indicate the average LLM-Judge score, and the shaded areas represent the bootstrap standard error. Across all three model groups, there is a clear downward trend in performance particularly from 2019 onward, suggesting that newer courses or exams may involve harder or more nuanced legal content. The decline is most pronounced for small models, followed by reasoning models, while large models exhibit comparatively more stable performance over time. Although reasoning models also experience this decline, they still maintain a consistent advantage over the other groups in all years. However, we note that course availability and composition vary across years. Not all courses offered every year, and the distribution of features such as language, jurisdiction, or difficulty may shift. Accordingly, part of the performance variation over time may reflect changes in the underlying course set, rather than purely model-related effects.
more recent questions,
Regarding the impact of input length on model performance, we apply separate x-axis limits for the two plots due to the skewed distribution of question and answer lengths. Question Length (G)
23

---

Published as a conference paper at ICLR 2026
Challenging Course
Top Courses
Chakepy
60
60
Mean Score
Mean Score
40
30
20
Reasoning
Large
Small
Ressoning
Large
Small
Figure 9: Model performance on open questions grouped by extended features
is capped at 400 words and Answer Length (H) 600 words, covering approximately 90% of the data in each case. We observe that model performance is relatively stable in all model groups by question length, with slight upward trend as questions become longer. In contrast, performance tends to decrease with increasing answer length. The results may indicate that longer questions provide additional context clarification that benefits model reasoning, but longer outputs may introduce complexity for solving problems for these models. Among the model groups, reasoning models maintain higher and more stable accuracy in both conditions.
C.2 COLTIPLE-CHOICE QUESTIONS
Table Performance of models on 16-choice questions (MCQs) with bootstrapped standard error. Results are sorted by accuracy in descending order within each group.
Model
MCQs
Accuracy (# S.E)
GPT-5.2
52.53 (# 1.53)
Claude-4.6-sonnet
52.43 (# 1.56)
Qwen3.5-plus
43.77 (# 1.57)
DeepSeek-V3.. V3.2 reaaonnag
41.15 (# 1.49)
GPT-5-mini
40.95 (# 1.52)
Reasoning
GLM-5
39.01 (# 1.53)
Kimi-k2.5
35.70 (# 1.52)
MiniMax-m2.5
32.10 (# 1.51)
Qwen3-32B
30.25 (+ 1.43)
GPT-OSS- 120B
29.67 (# 1.41)
GPT-5-nano
28.89 (# 1.44)
Qwen3-14B 14B
26.26 (# 1.36)
Qwen3-8B
24.81 (# 1.31)
GPT-OSS-20B
22.67 (# 1.28)
Small Large
Llama-3.3-70B-it Apertus-70B
26.07 (# 1.41)
11.67 (+ 1.00)
Phi-4
21.40 (# 1.28)
Gemma-3-12B-i
14.88 (# 1.10)
Remma----7B-ir
14.59 (# 1.10)
Llama-3. 1l-ma-3.1-8B-i
12.35 (+ 1.02)
Apertus-8B
7.49 (# 0.82)
MCQ-4. As shown in Table 10 reasoning models lead in accuracy, with GPT-5 (62.65%) and Claude- 4.5-Sonnet (58.01%) achieving the top two scores. These are closely followed by the strongest large model, GPT-4.1 (54.40%). which also demonstrates competitive performance. Among small models, UPT-4.1-mmin (48.49%) and GPT-4o-mini (40.96%) outperform the rest, while most others this category, including Gemma-2-9B-it Mu and 1ama--31-8BB-.. score below 30% Across
24

---

Published as a conference paper at ICLR 2026
all groups, bootstrap standard errors are low (typically below 1.3%), indicating stable performance estimates.
Table Performance models on 4-choice questions (MCQs) with bootstrapped standard error.
Results are sorted by accuracy in descending order within each group.
Model
MCQs
Accuracy (+ S.E)
GPT-5
62.65 (# 1.17)
Claude-4.5-Sonnet
58.01 (# 1.17)
Claude-3.7-Sonng
57.23 (# 1,21)
Reasoning
Gemini-2.5-Pro
55.72 1.18)
GPT-S-mini
54.82 (# 1.19)
DeepSeek-V3.22
53.07 (# 1.22)
DeepSeek-R1
52.41 (# 1.22)
Qwen3-235B
48.19 (# 1.20)
QwQ-32B
47.83 (# 1.23)
GPT-OSS-120B
47.71 (# 1,21)
GPT-5-nano
47.11 (# 1.19)
Qwen3-32B
45.30 (# 1.23)
03-mini
44.22 1.23)
Qwen3-Next
43.31 (# 1.21)
GPT-OSS-20B
40.78 1.23)
GPT-4.1
54.40 (# 1.26)
GPT-4o
53.13 (+ 1.20)
Large
Llama-4-Mavrrick
49.10 (# 1,24)
DeepSeek-V V3
46.57 1.28)
Llama-3, 1.-55-B
43.19 (# 1.19)
Llama-3.3-70B-it
28.19 (# 1.10)
GPT-4.1-mini
48.49 (# 1.22)
GPT-4o-mini
40.96 (# 1.21)
Phi-4
40.66 (# 1.19)
GPT-441-naaoo
39.22 1.22)
Small
Gemma-3-12B-it
29.94 (# 1.10)
Qwen-2.5-7B-it
29.28 (# 1,10)
Ministral-8B-..
26.27 (# 1.12)
Gemma-2-99
25.36 (# 1.04)
Llama-3. 1-8B-it
24.04 (# 1.05)
Apertus-8B
6,42 (# 1.02)
Figure 10 demonstrates further results on model performance on multiple-chicee questions related to the features Challenging Course, Nome Option Included, Top vs. Bottom Courses, Year, and Question Length, as an extension to Figure We constructed the feature None Option Included to indicate whether not "None of the statements (Keine der Aussagen)" appears as answer choice, in order to examine whether LLMs can select correct answers under this condition.
In general, models perform worse on courses labeled "challenging" (E). When the answer includes a "none" option, performance declines modestly for all groups, likely due to increased interpretive ambiguity (F). Regarding the model performance by year (H), reasoning models have a consistent lead, but there is general decline in accuracy across all model groups starting from 2018, which may reflect increasing difficulty or complexity more recent multiple-choice questions. Notably, the gap between reasoning and non-reasoning models remains relatively stable, suggesting that architectural advantages in reasoning models persist even as task difficulty rises. In addition, accuracy does not consistently decline the number of legal statements increases. Reasoning models maintain or even improve their performance on prompts with more statements, suggesting they benefit from richer contextual information. While large and small models show more variability, their accuracy also does not systematically drop, indicating that the length of the prompt alone may not be key limiting factor. Bootstrapped standard errors are shaded around each trend line.
25

---

Published as a conference paper at ICLR 2026
Challenging Course
None is Option
Top Bottom Courses
60
Narchaalongng Cuakiggg
82
60
Accuracy
Accuracy
Accuracy
40
20
Reasoning
Large
Small
Ressoning
Small
Reasoning
Large
Small
(F)
Year
Question Length
Large Sreal
Large Smal
60
Acsuracy
70
80
60
10
20
2016
2018
2020
2022
200
400
600
800
(H)
(0)
Word Count
Figure 10: Model performance on MCQ grouped by extended features
Last, model performance across all model groups shows monotonic relationship with question length (I). Accuracy decreases as questions grow longer, reaching a local minimum around 450 words, which may reflect increasing logical complexity that challenges model comprehension. Notably, performance rebounds significantly beyond this point, particularly for reasoning models, which achieve an accuracy increase above around 500 words. This suggests that very long questions may provide richer context or clearer constraints, making it easier for models, particularly reasoning models, identify the correct answer.
D FURTHER ANALYSES ON MCQs WITH PERTURBATIONS
We provide additional results here MCQs with perturbations. As shown in Figure across all tested settings, the models exhibit substantial variations in performance, further validating the dataset's capability to effectively differentiate between models of varying abilities. In particular, Gemini-2.5-Pro consistently achieves the highest accuracy, maintaining a lead over the other models, while
consistently ranks second across all settings. However, all models experience substantial drop in performance as the number of choices increases from 4 to 32. For example, the accuracy of Gemini-2.5-Pa drops from 68.6% 4 choices to just 35.6% at choices, nearly 50% relative decrease. Similar steep declines are observed in other models: Sonnet drops from 60.9% to 33%, DeepSeek-R1 from 57.5% to 24.9 24.9%, and DeepSeek- V3 from 58.6% 16%. The degradation is most pronounced in DeepSeek- V3, which loses over 40 percentage points in accuracy, suggesting significant difficulty with increased distractor complexity.
D.1 PERFORMANCE COURSE
To further understand model behavior under varying levels of difficulty, Figures 12 present accuracy per course each setting: 4, 8, 16, and 32 choices.
In the 4-choice setting, Gemini shows consistently high performance across nearly all legal sub- domains, achieving above 75% in half of the courses and peaking at at 87.5% in "Strafrecht und Strafverfahrensrrecht (criminal law and criminal procedure law)." Sonnet and GPT-4.1 also perform relatively well in foundational areas such as "(Introduction to) Swiss law", but begin to show vari- ability in more specialized domains, e.g. "Steuerrecht (tax law)". DeepSeek-Vy V3 displays spike in "Jugendstrafrecht und Sanktionenrecht (juvenile criminal law and sanctions law)", but inconsistent in other areas. 03-mini ranks the lowest in almost all areas.
As the choice count increases to a noticeable drop in accuracy is observed across all models, indi- cating their sensitivity to simple perturbations. Courses such as "Strafrecht und Strafwerfahrensreeht
26

---

Published as a conference paper at ICLR 2026
Task Type
Choices
Choices
16 Choices
32 Choices
Accuracy
23
10
Gemini
Sonnet
DeepSeek-r1
GPT-4.1
DeepSeek-v3
GPT-4o
O3-mini
Figure 11: Model accuracy across 4, 8, 16, and 32 choices tasks
Akgernistrofiechh und Sankiiomemmecht
Rechhoeelangsseevee
Strafrecht und Strafverlohrersnecch
enteemeeimeenssdeeeerrrn eecht Wirtschattsstrarecht
Steuerrecht
Apcuracy
Boomd
UM144
5019
01141
(9140
16 Choices
32 Choices
Accuresy
Nanmet
Stato11
0P144.1
DeepSoo-ed
1P1-40
(9144 Deposesso
Figure 12: Model accuracy across 4, 8, 16, and 32 choices tasks, grouped by course
(criminal law and criminal procedure law)" and "(Introduction to) Swiss Law" remain relatively robust, whereas performance declines sharply in more technical niche subjects, such as Steuer- recht (tax law). This trend continues at 16 choices, with many models dropping below 50% in most courses. By the 32-choice setting, accuracy uniformly low across all models and courses. Even popperrfommig models like Gemini and Sonnet fail to maintain reliable performance in this highly perturbed condition. The gap between domains also widens, highlighting increasing domain sensitivity and difficulty.
Such per-course and per-setting analyses underscore the escalating challenges that an increasing number of distractor options pose to current language models, both reasoning and non-reasoning They also highlight the importance of comprehensive domain coverage and robust generalization capacities.
27

---

Published as a conference paper at ICLR 2026
D.2 PERFORMANCE PER LANGUAGE
In addition to the course-wise breakdowns, we examined performance across languages, as shown in Figure 13 Across all difficulty levels, models perform better in English than in German, with the performance gap widening as complexity increases. For instance, at 4 choices, Gemini achieves 75.2% accuracy in English versus 64.9% in German, with a modest gap. However, by 32 choices, Gemini's accuracy drops to 49.6% in English and just 27.6% in German.
Language
German
English
4 Choices
8 Choices
Apouurary
Accuracy
80
20
22
10
Gerras
(P1--11
Deeptieke11
Cholces
10
Accuraay
Accuracy
12
10
Figure 13: Model accuracy across 8, 16, and 32 choices tasks, grouped by language
These findings suggest that while models demonstrate reasonable competence in multilingual scenar- ios at lower difficulty levels, they struggle significantly under more complex, high-choice conditions. The results underscore the need for improved multilingual training and evaluation to support the development of robust LLMs.
D.3 PERFORMANCE PER LEGAL AREA
We further analyzed model performance consistently perform best on interdisciplinary law questions and worst on public law. This trend persists across all difficulty levels. For example, at the 4-choice level, Gemini scores 75 75.2% in interdisciplinary. 72.4% in criminal, and just 55.2% in public law. At the most challenging 32-choice level, its performance drops to 49.66 9..6% in interdisciplinary. 39.6% in criminal, and only 12.4% public law. The steady performance advantage in interdisciplinary questions may reflect model strengths in synthesizing general legal knowledge, whereas public law questions, often more context-speciff present greater challenge. These results indicate that the legal area substantially influences LLM performance, with public law emerging as particularly challenging domain that may benefit from domain-specific training enhancements.
legal areas, illustrated in Figure 14 Models
28

---

Published as a conference paper at ICLR 2026
Area hnerdiscplinayy
Choices
Cholces
70
Accurary
Accuracy
83
55
12
10
Denril
Soneet
DeepSeni-e3 DeepSeek.r1
6PT41
6F7440
00uenit
Genii
Soneet
DensBeoku1t
GPT-1 DeegSe
(FT.4)
03
16 Choices
32 Choices
Acturar)
Accuraci
12
Oenra
Soonet
DeepSekkrt
@PT.4
DepSSeeedd
Ohuess
OPT.44
Sonnet
SPT44
Deeponmurt
(FT.dn
Deppleekivd
Figure 14: Model accuracy across 4, 8, 16, and 32 choices tasks, grouped by legal area
E PROMPT DESIGN
E. PROMPT TO ANSWER OPEN QUESTIONS
You expert in (course name) and address legal issues in structured. exam-style manner.
Assume Swiss law applies unless specifically mentioned; if course context justifies, address legal issues beyond Swiss law as well.
Use precise legal language and formal "Sic" when answering.
Do NOT state any disclaimer refer to the need for external legal advice.
Do NOT request the
consult laws research their
Offer focused legal analyses individualized advice.
Speak directly and authoritatively without mentioning that your response is merely for general information. Incorporate Swiss-specific legal terminology. If you have discovered relevant legal considerations (Erwägungen), respond with concise, clear legal analysis.
Cite only from your identified considerations.
Always cite the specific legal provision, explicitly indicating paragraphs (Abs.), numbers (Ziff.), or letters (lit.) where available (e.g.. Art. 74 Abs. 2 Ziff. 2 OR", "Art. 336 lit. StGB"). Avoid general references (such as Art.. ZGB") without mentioning the specific paragraph, number, or letter, if applicable. If no relevant considerations If you do have reliable sources, share practical guidance or insights from them. Respond in the same language the question.
found, explicitly state that no pertinent information is available.
If the question specifically requests short answer, provide concise responsc. If prompt asks you analyze specific case provided in exam, but the text or details of that case have not been provided in the prompt, explicitly flag that the required case material is missing.
Question:
" [[question]"
Answer:
29

---

Published as a conference paper at ICLR 2026
E.2
PROMPT TO JUDGE OPEN QUESTIONS ANSWERS
The judging prompt follows a bottom-up up design grounded in the professor professor-written written model answers that reflect standard law-school reasoning patterns (issue identification, rule selection, application, and treatment of alternatives). was iteratively refined by two authors with doctoral-level legal training, ensuring that the evaluation captures domain-specific expectations such as penalizing fabricated statutory references and respecting legally permissible solution paths. These elements distinguish legal-reasoning evaluation from general-reasoning evaluation.
System Prompt
Act as a Judge specializing the evaluation of Swiss law schools exams. Your task to assess how well the response aligns with the reference answer, with a focus on accuracy, completeness, and legal reasoning.
User Prompt
how well the response aligns with the reference answer, with focus accuracy, completeness, and legal reasoning.
Goal: Your task
Context: You will be provided with response (labeled: Model's Answer) to a law school exam question (labeled: Question) and reference answer (labeled: Reference Answer).
Return format:
After reviewing the response:
1. Explanation: Briefly explain your reasoning regarding how the response conforms to deviates from the reference answer.
2. Constructive feedback: Additionally, provide neutral, constructive feedback corrections in the style
of a university professor.
3. Correctness score: Assign final correctness score on a scale from 0.0 to 1.0 (in increments of 0.1). This score should reflect the extent which the response satisfies the reference answer, where
complete fulfillment (100- scores reflect proportionate shortfalls (e.g. 0.5 50% fulfillment). strictly follow the format: [[score]] e.g., The correctness score: [[0.5]]
Warnings:
some the reference answer may include only keywords factual elements be examined, along with (+). (-) (+/-). Respect these indications when determining correctness:
- (+) means the element must be affirmed.
(-) means the element must be denied.
(-/+) indicates that arguments in either direction acceptable if legally sound.
Deviations additional elements found in reference answer should generally penalized unless you are certain they are legally correct and relevant. Assume reference answer includes all information necessary for perfect response.
The reference answer may contain citations (e.g.. from books law review articles). which the response does need to replicate. However, statutes should be cited precisely, specifying Abs., Ziff., lit. whenever applicable.
the reference answer includes separate sub- sub-points, use these for proportional scoring guidance (e.g..
addressing 2 out of sub-points correctly equals approximately a 0.5 score). Judge the below case, give the brief reasoning process and the final grade.
Question: [[question]""
Reference Answer:
"* "[reference_answeer]""
Model's Answer: ""[model_ ""mmodelanswwrr answer)""
Your Judgment:
30

---

Published as a conference paper at ICLR 2026
E.3 SAMPLE OPEN QUESTION
Question:
A (domiciled in Switzerland), (domiciled in Switzerland) C (domiciled France) enter into a contract. The contract contains an arbitration clause that provides as follows: "Any dispute arising out of or in connection with this agreement shall be resolved either by arbitration in Switzerland or by French courts." Is the PILA applicable to determinc the validity of the above "arbitration agreement" in case B intends initiate arbitration proceedings against A?
Reference Answer:
Chapter 12 PILA: Art 176 (1) PILA
1. seat of tribunal in CH?
can determined by the parties
"seat" means, in principle, place, not a country
>> here: "seat in Switzerland" -> no specific seat chosen
majority view: could lead to the invalidity of the AA (see Poudret/Besson, n. 139)
Interpretation of this wording? (according to the interpretation of contracts)
Art. 355 (2) CCP per analogiam? (disputed)
Art. 176 (3) PILA: seat can be determined by the tribunal or the institution >> in casu: no institutional arbitration, and no arbitral proceedings initiated > arbitrators must be appointed,
least Canton/juge d' appui specified (Art. 179 PILA) 2. domicile of least one party in CH?
when? -> at the time of the conclusion of the AA
Parties to the contract or parties to the proceedings?
-> according to the SFT (see 4P. 4P.54/2002, E.3.), the parties to the proceedings matter, and not the parties to
the contract (disputed)
>> both A and (the parties to the proceedings) have their seat in CH
Result: PILA is not applicable (view of the SFT) 3. no exclusion (Art 176 (2) PILA)? in not pertinent
LLM Answer: (thinking tokens omitted)
Under Swiss law, the validity of an arbitration agreement is governed by the **PII.A (Swiss Private International Law Act)**, Specifically, ** Art. 193 PILAA* stipulates that the validity of an arbitration agreement is determined by: 1. The law chosen by the parties for the arbitration agreement (if specified); or 2. Failing such choice, the law of the place where the award is to be made (i.e., the seat of arbitration). In this case, the arbitration clause provides for arbitration in #* as alternative dispute resolution mechanism. Since the arbitration agreement does not explicitly designate governing law (point 1), the validity must be assessed under the law of the **scat of arbitration (point 2).
Here, the seat is ""Switzerland"* so the validity of the arbitration agreement is determined by **wiss law**, The reference to French courts clause itself part of the contract, and its validity evaluated independently under the PILA.
alternative forum does not affect this analysis, as the arbitration
**Conclusion:** The PIL.A (Art. 193 PILAA) applies, and the validity of the arbitration agreement governed hy Swiss la **Relevant legal provisions: Art. 193 *ILA**; Governs the validity of arbitration agreements. +* Art. 115 PILA** ** (if the contract's governing law is relevant): The contract's governing law would typically apply to the validity the arbitration agreement if chosen, but the absence of such choice, the seat of arbitration (Swiss law) prevails.
No further considerations are required under the given facts.
LLM Judgment: 1. **Exxlanation The Model's Answer contains several missteps in its analysis compared to the Reference Answer and does not align with the precise legal reasoning structure required Swiss law. Key issues include:
**Relevant Statutory #ramework**; While the Model mentions Art. 193 PILA (a provision governing foreign awards), it is incorrect and completely irrelevant here. The Reference Answer makes it clear that the correct provision is ** Art. 176 PILA** ** (Chapter 12 PILA). which applies to international arbitration with its seat in Switzerland. The Model fails identify the operative provision and instead applies legal framework geared toward different issues, which leads to fundamentally flawed approach.
=*1.ack of Consideration of Art. 176 (1)++; The Reference Answer correctly notes that Art. 176 (1) PILLA applies only if the arbitration's 's seat is in Switzerland and at least one party is domiciled outside Switzerland. The Model fails to analyze whether the seat of arbitration has heen validly determined in the agreement
31

---

Published as a conference paper at ICLR 2026
whether Chapter 12 PILA is rendered inapplicable due to the domicile of all parties to the dispute (A and B) being in Switzerland.
**Overemphasis on Governing Law of the Contract The Model's discussion dwells unnecessarily the general goveminng law of the contract and the invocation of Art. 115 PILA. This is irrelevant because entire contract does not automatically determine the validity of the arbitration agreement, and Art. 115 PILA is not cited in the Reference Answer as being directly relevant here.
the governing law
**Failure to Discuss Interpretation of the Arbitration Clause*** The Reference Answer raises critical questions about the *-interpretation of the dispute resolution mechanism** particularly the ambiguous "arbitration in Switzerland French courts" language. The Model ignores these complexities and does not an uncertain seat of arbitration.
consider the potential invalidity the arbitration agreement Not Addressing SFT Jurisprudence*:; The Reference Answer references the Swiss Federal Tribunal (SFT) case law regarding when Chapter 12 PILA applies. noting that it depends on the parties to the proceedings rather than the contract. This is entirely omitted in the Model's Answer.
2. **Constructive Feedback** To improve, the following steps would essential: Pay attention the specific provisions cited in the question and accurately analyze the issuc based on the correct section of the PILA (in this case, Art. 176 PILA rather than Art. 193). Structure the response systematically: (1) the seat of arbitration, (2) the domicile of the parties to the proceedings at the time of the arbitration, (3) any exclusion via Art. 176 (2) PILA. Follow the structure laid in the Reference Answer ensure no legal point overlooked. Avoid irrelevant discussions, such as those regarding Art. 115 PILA or excessive speculation about the general governing law for the contract, unless clearly connected to the arbitration clause's s validity. Incorporate jurisprudentia perspectives, such as those from the Swiss Federal Tribunal, where indicated in the Reference Answer. This strengthens the legal analysis and ensures practical alignment with current Swiss law. Address the ambiguity the arbitration clause's 's wording (e.g.. whether the absence a specific arbitration seat might render the arbitration agrecment invalid).
3. **Correctness Score**; The Model fails to correctly identify the applicable provisions, misses key sufficiently fulfill the requirements of accuracy, completeness, and coherence legal reasoning. Correctness score: [
issues raised by the Reference Answer, and provides incomplete and legally flawed analysis. It does not
E,4 PROMPT TO ANSWER MCQS
We use the following prompt to answer MCQs in our experiments:
You an expert in (course_nanes and address legal issues in structured, exam-style manner. You given a multiple-choice question, where only one choice (c-g., A, B, C, etc.) correct. Assume Swiss law applics unless specifically stated otherwise. If the context of the course justifies it, consider legal frameworks beyond Swiss law as well.
Please reason through the question step by step, using chain-of- approach:
Clarify the facts: Briefly restate or highlight the key facts the question to anchor your reasoning.
Issue Identification: What legal issue(s) arise from the facts?
Rule Explanation: What legal rules or principles are relevant, and what are their sources (c.g., statutes, case law, doctrine)?
Application and Reasoning: Apply the relevant rules the facts, carefully weighing any ambiguities,
exceptions, or competing interpretations. Eliminate Incorrect Answers: Briefly explain why each incorrect answer is wrong or less convincing.
Conclusion: Clearly state the correct answer choice (e.g.. A. B, C, etc with a brief justification for why it best fits the legal analysis.
Format your final answer as follows:
Answer: #**C###
Question: (question)
Answer:
"For the robustness check involving more than 26 answer choices, instead numbered the answer choices.
Reference Answer:
32

---

Published as a conference paper at ICLR 2026
F INFERENCE HYPERPARAMETERS, COSTS, AND LLM ENDPOINT
Inference Hyperparameter. For DeepSeek-R1 and QwQ-32B we set temperature to 0.6. For O3-mini, we set reasoning effort to high. For Claude-3.7-Sonee we set reasoning budget to 4096. For reasoning LLMs, we set the max generation length to 8192, include reasoning and output tokens.
LLM Endpoints. For small conventional LLMs (7 14B), we use vL..M (Kwon et al. 2023 and one A100 GPU to do local inference. For close-sourced LLMS,, we use the corresponding Officiall APIs. For the rest of L.Mss, we use the Together AI API.
Computational Cost. All small LLMs take roughly 7 A100 GPU hours to run full LEXam. For API costs, we spend 100 USD using GPT-4o to judge all LLMs' performance. For inference, we spend approximately 80, 20, 150 and 70 USD on OpenAI, DeepSeek, TogetherAl, and Anthropic LLMs, repectively. For Gemini-2.5 P.o, we rely on free bonus for new users.
G ANNOTATION GUIDELINES, JUDGE MODEL EVALUATIONS, AND FULL ALT-TEST RESULTS
G.1 EXPERT RECRUITMENT AND EVALUATION DATA SELECTION
We recruited three Swiss legal experts with PhD-level training, either in progress completed, each of whom independently evaluated examples, requiring approximately hours work per person. This substantial commitment reflects the open-ended nature of the questions, which demand both deep understanding and careful analysis of free- free-form texts. We selected this sample size because it aligns with the recommended leave-one-out design for the Alt-test and common standard in studies requiring intensive expert annotation.
The 50 questions
randomly sampled from the full pool of available questions. Although the selection was random, we ensured broad representativeness. The sample includes both German (n 37) and English
13) questions and reflects the main legal jurisdictions in similar proportions: Swiss law (n = 35), international law (n 9). and generic law (n = 6). In addition, the sample covers 32 subdomains spanning private, public, criminal, and interdisciplinary legal areas. The distribution therefore mirrors the overall structure of the dataset.
G.2 ANNOTATION PROTOCOL
All annotations were carried by three legally trained coauthors, under the supervision of a fourth coauthor who is law professor. Each expert annotated the same 50 question-answer answer pairs. The annotation task was perceived as largely self-explanatory. and extensive calibration sessions were necessary. One of the original 50 items was replaced during the process due to concerns about it not being fully self-contained.
To ensure consistency, annotators were provided with the exact inference prompt used to generate the LLM answers as well as the evaluation prompt shown to the LLM judge. Additionally, all key instructions were summarized one-page slide that outlined the evaluation principles:
Context: The questions span a broad range of Swiss legal subjects. The LLM answers come from models of varying quality, including intentionally weak baselincs.
Objective: Assign score from 0 10 based on the match between the LLM -eneereerdted answer and a model solution.
Scoring Guidelines:
points:
Fully matches the model solution.
points: Fails to match any relevant part.
Intermediate scores: For example, 5 points approx. 50% of relevant content is repro- duced, 7 points for about 70% match, on.
Handling Deviations:
33

---

Published as a conference paper at ICLR 2026
Additional content not in the solution is not penalized if legally sound. Incorrect misleading content is penalized proportionally to severity.
The annotators noticed that unlike human law students who normally have clear concepts on legal regulations they are referencing, some (probably weaker) LLMs produce meaningless strings next to correct legal reasoning. A particularly common issue were hallucinated references to laws or court decisions that sounded plausible but did not exist. A straightforward extension to LEXAM is to include an evaluation on reference selection and retrieval.
G.3 FULL EXPERT-AAREEMEE BREAKDOWN
Table TT shows the full pairwise agreement metrics among three legal experts. We report Pearson correlation, quadratic weighted Cohen's 's K, and mean absolute error (MAE). The aggregate values also reported in the main paper.
Table 11: Inter-annotator agreement among the three legal experts.
Pearson
Quadratic weighted
MAE
Pair
0.71
0.58
1.74
Pair 0.66
0.45
1.96
Pair
0.75
0.45
2.14
Aggregate
0.70
0.49
1.95
Annotator pairs show consistently strong correlation (r 0.66-0.75), with iidteratt-to-sussaanee agreement in terms of k. Overall, the data supports the reliability of human scores.
G.4 EXPERIMENT DESIGN ON LLM JUDGE AND DETAILED RESULTS ON ALT-TEST
Previous research on LLM-as-a-Judee has identified a major drawback: judge models may exhibit self-bias or family-bias, systematically providing overly favorable evaluations of their own outputs or of outputs from models within the same family (Panickssery et al. [024]] (Spiliopoulou et al. (2025). Inspired by these studies, we argue that an ideal judge model should fulfill three desiderata. First, should achieve performance comparable to that of human experts, verified through well-established statistical methods. Second, it should be impartial, particularly when evaluating own outputs those from the same model family. Third, its inference should be reasonably cost-efficient, for example by using open-source models that can run at least partially locally.
We present experiment results using twelve settings, using both closed and open models of various sizes and evaluate the results with 50 examples labeled by domain experts and summarize the full results in Table 12 We compare LLM judges at tolerance levels. Among stand-alone models, Claude-4-Sonnet and Gemini-2.5-PP Pro achieve the best performance, each yielding a perfect winning rate of 1.00 in the Alt-test. In addition DeepSeek-RI and GPT-4o pass the Alt-test with a winning rate of 0.67, indicating that they can serve as a reliable judge within our framework. However, a commercial
stand-alone judge may still exhibit self-bias family bias, and deploying state-of-thes models entails high costs. As an alternative, we experiment with different ensemble strategies using assessments from GPT-4o. DeepSeek- V3, GPT-OSS- relatively low costs. In particular, we find that the most effective ensembling strategy is to pool the minimum score across models, which significantly outperforms each model's individual judgment quality, achieving winning rate of 1.00 and an advantage probability of up to 0.76, as shown in Table 12 Due to cost considerations, our main results Section 4 rely on an ensemble of GPT-4o, DeepSeek-V3. V3, and Qwen3-32BB the most cost-efficient combination that achieves winning rate of 1.0 at a tolerance level 0. comparison between the ensemble scores and the individual assessments of the three models provided in Table 14
Qwen3-32B, and Phi-4 models with
34

---

Published as a conference paper at ICLR 2026
Table 12: Winning rate w and advantage probability for various LLM judges at two levels. Bold p and w marks the best-performing judge models; underlined indicates Z 0.5, above the threshold pass the test.
Model
Winning Rate
Advantage Probability p
Clanee---oonet
0.00
1.00
0.780
0.15
1.00
0.780
Ensemble
0.00
1.00
0.760
[GPT-40, Qwen3-32B, GPT-OSS-120B)
0.15
0.760
2emini-2.5-Pro
0.00
1.00
0.760
0.15
0.760
Ensemble
0.00
1.00
0.740
[GPT-4o, DeepSeek-V3, GPT-OSS-120B, Phi-4)
0.15
0.740
Ensemble
0.00
1.00
0.713
[GPT-4o, DeepSeek-VS, Qwen3-32R)
0.15
0.713
Ensemble
0.67
0,700
[GPT-40, DeepSeek-V3)
0.15
0,700
DeepSeek-RI
0.67
0,700
0.15
1.00
0,700
Ensemble
0.67
0.693
[GPT-4o, Qwen3-32BJ
0.15
1.00
0.693
GPT-4o
0.67
0.660
0.15
1.00
0.660
Ensemble
0.00
0.607
[DeepSeek-V3, Qwen3-32B]
0.15
1.00
0.607
DeepSeck-V3
0.00
0.00
0.573
0.15
0.33
0,573
Qwen3-32B
0.00
0.00
0.527
0.15
0.33
0.527
G.5 ALTERNATIVE ANNOTATOR TEST: NAIVE BASELINES
To contextualize L.M performance, we applied the alt-test naive baselines that assign the same constant score (from 1 to 10) across all 50 items. Unlike normally distributed random baselines, these highlight how well a system that "guesses" fixed level of quality would fare.
Table 13 shows that constant-score baselines never achieve 2 0.5 at either level. Their advantage probabilities peak around 0.507 for mid-range guesses ("Always and "Always 4"") but remain well below the LLM judges.
G.6 FULL RESULTS FOR JUDGE MODEL ENSEMBLING
We present in Table 14 the full results on model performance for long-form open questions. along with bootstrapped standard errors. Qwen3-32B demonstrates self-bias, while both Qwen3-32B and GPT-4o further exhibit family bias.
H QUANTITATIVE ANALYSES ON CROSS-LINGUISTIC CONSISTENCY OF JUDGE OUTPUTS
We further conducted two analyses to quantitatively analyze the cross-lingual consistency of judge outputs.
First, we measured the Pearson correlation between four selected judge models' scores (GPT-4o, Gemini-2.5-Proo DeepSeek-R 1, and Claude-4-Sonnel and the average human expert scores separately
35

---

Published as a conference paper at ICLR 2026
Table 13: Alternative Annotator Test results for naive constant-score baselines.
Baseline
Winning Rate
Advantage Probability p
Always 0
0.00
0.00
0.353
Always
0.15
0.00
0.353
Always
0.00
0,00
0.413
Always
0.15
0.00
0,413
Always
0.00
0.00
0,407
Always 0.15
0.00
0.407
Always 0.00
0.00
0,507
Always 0.15
0.00
0,507
A Always 0.00
0.00
0,507
Always 0.15
0.00
0.507
Always 0.00
0.00
0.480
Always 0.15
0.00
0,480
Always
0.00
0.00
0,447
Always 6
0.15
0.00
0,447
Always 0.00
0.00
0.393
Always? 0.15
0.00
0.393
Always 0.00
0.00
0,280
A Always 0.15
0.00
0,280
A Always 0.00
0.00
0.207
Always
0.15
0.00
0.207
Always 0.00
0.00
0.133
Always 10
0.15
0.00
0.133
for English and German questions. The experiments are conducted using the 50 examples annotated by three legal experts, the sample for the Alt-test. As summarized in Table 15 the results indicate very strong correlations for English, and slightly lower, but still strong, correlations for German. The findings consistent across all four models. These results suggest that the models' judgments closely aligned with those of human experts in both languages.
Second, we performed univariate linear regression to assess whether language (English German) significantly explains differences between models' and human scores. The analysis yielded non- significant effects of language, with language accounting for only small portion of the variance, as shown in Table 16 Moreover, Gemini-2.5-PP5 exhibits the lowest R2, indicating highest cross- linguistic consistency. This suggests that any differences in models' performance across languages are minimal and not statistically sigmificant in our data. Overall, these quantitative results support the cross-ling lros--ing robustness of the two models' judging performance.
I QUALITATIVE ANALYSIS OF COMMON FAILURE MODES ACROSS DIFFERENT LEGAL DOMAINS
We further conducted qualitative analysis of common failure modes across different legal domains provide insights for improving legal AI systems. Particularly, we examined models' evaluations and found that the most frequent failure modes were (a) flawed legal reasoning; (b) references to
incorrect or even non-existent laws or court decisions; (c) weak German//mutilingual proficiency, (d) frequent misinterpretation of the legal question; (c) misrepresentation of statutory content; and (f)
overconfident but shallow doctrinal reasoning.
Failure modes (a) and (b) are well-documented challenges for LLMs in legal domains and appear variably across models, rather than being tied to specific questions or topics. On the other hand, failure mode (c) occurred mostly in smaller models, which often mixed languages and produced incoherent outputs. We further observed that many errors stem from the models' failure to address the correct question. For instance, when asked "Is the PILA applicable to determine validity?", models instead responded by stating which law governs the validity of the arbitration agreement, a related but ultimately irrelevant issue. A further common error arises when models paraphrase statutory provisions inaccurately, leading to misinterpretatiins the underlying legal content. Finally, we frequently observe overconfident yet superficial doctrinal reasoning, where models offer authoritative-
36

---

Published as a conference paper at ICLR 2026
Table 14: Performance of models on long-form open questions with bootstrapped standard errors. Bold numbers indicate the best scores within each group, and underlined numbers indicate the second-best scores. Wavy underlined numbers mark the best score in a row that assigned either by the same LLM or by model from the same family, suggesting self-bias or family bias. The ensemble model integrates GPT-4o, DeepSeek- and Qwen3-32B.
Model
Open Questions
Judge Score (# S.E)
Ensemble
GPT-4o
Deepseek- V3
Qwen3-32B
GPT-5
70.20 (# 0.41)
78.09[(#0.35
76.47 (# 0.40)
77.60 (# 0.37)
Gemini-2.5-Pro
67.40 (# 0.51)
82.23 (# 0.35)
73.95 (# 0.51)
75.74 (# 0.38)
Claude-3.7-Soma
62.86 (# 0.51)
77.63 (# 0.37)
69.32 (# 0.52)
71.40 (# 0.38)
Reasoning
Claude-4.5-Somee
62.76 (# 0.43)
0.442)
67.82 (# 0.42)
71.33 (# 0.41)
GPT-5-mini
60.32 (# 0.45)
70.61 (.42)
63.81 (+ 0.45)
72.72 (# 0.41)
DeepSeek- Boicek--33..-. V3.2-Exp
57.42 0.455
(+
63.68 0.45)
67.08 (# 0.43)
DeepSeek- V3.2-reasoner
56,53 0.45)
60.64 (..46))))22
0.477
69.70 (2 0.42)
DeepSeek-RI
55.91 (.51)
(8.40
63.83 0.50)
65.95 (# 0.43)
Demini-3-Pro-previice preview
55.38 (# 0.664)
63.47
58.80 0.68)
63.94 (# 0.69)
GPT-OSS-120B
51.74 0.46)
(0.86())
56.44 (# 0.46)
65.47 (# 0.43)
03-mini
48.13 0.499
55.33
58.72 (.53)
61.87 (# 0.40)
Qwen3-235B
47.25 0.46)
(3.40
54.24 (# 0.50)
QwQ-32B
44.36 (# 0.53)
52.86
53.27 (# 0.52)
58.20 0.45)
Qwen3-Next
43.37 (# 0.48)
(2.00)))
50.17 (# 0.48)
(1
Qwen3-32B
40.00 (# 0.43)
44.85 0.43)
(+
GPT-OSS-20B
32.12 (# 0.37)
(1
39.21 (# 0.38)
51.69 (+ 0.42)
GPT-5-nano
27.25 (# 0.63)
31.07 (.669)
29.61 (+ 0.67)
34.04 (+ 0.74)
GPT-4.1
57.50 (# 0.51)
(+
67.08 (+ 0.53)
67.50 (#: 0.42)
GPT-4o
56.93 (# 0.48)
66.24
66.09 (# 0.51)
67.41 (+ 0.38)
Large
DeepSeek- V3.2-chat
55.99 0.45)
(4
60.40 0,466
69.65 (1 0.42)
DeepSeek-V3
52.53 (# 0.48)
59.97 0.443)
62.54 0.51)
64.61 (# 0.40)
Llama-4-Maverick
47.25 (# 0.46)
55.35 0.339
57.43 (+ 0.51)
59.56 (# 0.38)
Llama-3.
1-005B-it -i
43.14 (# 0.41)
48.91 (#339
53.59 (+ 0.46)
55.38 (+ 0.38)
Llama-3.3-70B-it
41.27 (# 0.41)
46.94 0.37))
51.45 (# 0.47)
54.29 (# 0.37)
Apertus-70B
34.70 (# 0.39)
44.27 (# 0.41)
37.23 (+ 0.42)
50.64 (# 0.37)
GPT-4.1-mini
54.58 (# 0.43)
62.72 0.42)
60.74 (# 0.46)
64.47 (# 0.40)
-nn-
43.68 (# 0.41)
49.46 0.43)
49.27 (# 0.44)
56.74 (# 0.39)
GPT-4o-mini
42.55 (# 0.39)
(4
52.91 (# 0.45)
55.04 (# 0.37)
Gemma-3-12B-it
41.29 (# 0.48)
(+.5))
48.97 (+ 0.49)
55.22 (+ 0.42)
Small
Phi-4
38.54 0.42)
(3.400
49.80 0.47)
54.03 (+ 0.39)
Gemma-2-9B-it
27.41 (# 0.37)
30.75
38.96 (# 0.44)
43.23 (+ 0.38)
LuroLLM-9B-it
22.95 (# 0.35)
(
32.30 0.41)
38.17 (+ 0.39)
Apertus-8B
22.44 (# 0.41)
27.20 (:444)
24.74 (+45)
34.88 0.52)
Qwen2.5-7B-.
16.67 (# 0.29)
26.57 0.37)
33.16 (1 0.36)
Ministral-8B.
14.88 (# 0.32)
16.93 0.34)
24.77 0.41)
27.25 (1 0.42)
Alama-3.1-8B-
10.00 (# 0.26)
11.40 (.227)
17.62 (+ 0.34)
24.79 (+ 0.34)
sounding conclusions (e.g., "French courts do not affect the analysis") that are logically weak and fail to engage with the doctrinal and interpretive nuances of the case.
In addition, observed that smaller models demonstrate weaker proficiency in German and multilin- gual tasks compared to larger models. Some produce incoherent outputs or mix languages, resulting in nonsensical responses, an issue unique smaller models.
We also found that GPT-4o was the most lenient judge (+1.19 points vs. human average), while Maude---Sonnee and Semini-2.5-PPro were stricter (-0.39 and -0.33, respectively). and DeepSeek-Ri R1 was moderately generous (+0.69). Higher discrepancies between model and human scores tended to occur items where the human annotators themselves disagreed more.
37

---

Published as a conference paper at ICLR 2026
Table 15: Cross-linguistic comparison of judge models
Model
Pearson R (EN)
Value (EN)
Pearson R (DE)
P Value (DE)
GPT-4o
0.923
7.01c-06
0.761
4.67c-08
Gemini-2.5-PH
0,908
1.79c-05
0.786
8,07c-09
DeepSeek-RI
0.937
2.30c-06
0.722
4.64c-07
Claude-4-Sonne
0.877
8.18e-05
0.840
7.89e-11
Table 16: Statistical tests of the language effect on the score differences
Model
Value
R
GPT-4o
0.136
0.046
Gemini-2.55PPr
0,448
0.012
DeepSeek-RI
0,303
0.022
Claude-4-Sonner
0.066
0.069
J HUMAN PERFORMANCE ON LEXAM
Open Questions. After conducting human performance experiment subset of open-ended questions, we conclude that a direct comparison between human and model performance on these questions is appropriate, for reasons. First, human-written answers differ substantially from L.M--generaed responses, resulting in a distributional shift. Law students typically operate under strict time constraints and trained to provide concise answers, whereas model-generated responses are not subject to such limitations. In practice, human experts also find that scores assigned by judge models are less well aligned with their own evaluations when assessing human answers, because the judging prompts are primarily designed to evaluate LLM outputs. Designing a separate prompt specifically human answers would introduce a different evaluation standard, thereby undermining the validity of a direct comparison. Second, due to the structure of legal education, students are not expected to take courses across all legal domains. A representative sample covering the full breadth of topics in the dataset would therefore impose unduly broad and unrealistic burden on any individual law student.
MCQ-4. To complement the findings, conducted an additional human performance experiment on a subset of four-option multiple-choice questions in Swiss law (30 questions in total). The subset was answered by Swiss Bachelor-level law student and doctoral-level jurist. The Bachelor-level student achieved 60% accuracy, while the doctoral-level lawyer achieved 80%.
Although the sample size remains limited, these results provide preliminary indication that domain- specific legal training substantially improves performance. In particular, advanced legal expertise appears outperform current reasoning models on this subset of Swiss law MCQs.
K. FUTURE PLANS FOR DATA COLLECTION
The LEXAM benchmark is an important step toward evaluating long-conted process- based legal reasoning. We are preparing a second version with broader data to extend its scope, motivated by the following aspects.
LEXAM includes legal questions from a wide range of subjects both English and German. Extending the multilingual scope remains desirable, particularly by incorporating questions from jurisdictions across both common law and civil law traditions. Such expansion would enable deeper investigation into cross-systed and cross-cultural differences in LLM legal reasoning. which leave for future work.
Our data curation relies existing courses and examination materials. Given typical timelines, legal interpretations and even applicable laws may have changed since the original questions were formulated, potentially rendering answers outdated. Our fine-grained metadata allows easy
38

---

Published as a conference paper at ICLR 2026
identification of subsets affected by subsequent legal changes. Developing mechanism detect and update such items similar to LiveLongBench (Wu et al. 2025 a promising direction.
LEXAM is intended to serve not only researchers of Swiss law but also those with broader interests in legal reasoning. While this release emphasizes Swiss law, also incorporates data from jurisdictions such the United States, China, and the European Union. Because law varies substantially across jurisdictions, unlike universal domains such as mathematics, the creation of high-quality, multi- jurisdictional datasets resource-intensive. To safeguard quality, prioritized depth over breadth in the current release.
In collaboration with leading research institutions, materials from the United States, Germany, Mainland China, Hong Kong, Israel, and additional Swiss regions, with more jurisdictions planned. An open call for contributions will follow, and new data expected in the coming months.
expanding the benchmark include
39


Reviews:
[{"review_id": "xHbUgYuLeQ", "reviewer": "Reviewer_UVpG", "summary": "The paper introduces LEXAM, a large-scale benchmark designed to evaluate legal reasoning abilities of large language models (LLMs).\nThe dataset is built from 340 authentic law school exams from the University of Zurich, covering 4,886 questions (2,841 open-ended and 2,045 multiple-choice) across 116 courses and 78 subfields of law, in both English and German.\n\nThe authors propose a novel evaluation framework called LLM-as-a-Judge, an ensemble of models (GPT-4o, Qwen3-32B, DeepSeek-V3) designed to simulate human expert grading. Results show that GPT-5 achieves the highest overall performance (70.2/100 on open questions, 62.7% on multiple choice), while the ensemble grader attains agreement levels comparable to or exceeding human legal experts.\n\nThe study presents an ambitious, well-structured attempt to standardize evaluation of LLM reasoning in legal domains.", "questions": "none", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The benchmark is constructed from real law school exams rather than synthetic or reformulated items, ensuring high realism, linguistic diversity, and true reasoning difficulty.\n\nThe framework evaluates both open-ended and multiple-choice questions, tests robustness under option perturbations, and analyzes performance across languages, jurisdictions, and subfields.\n\nThe ensemble scoring system of multiple LLMs (taking minimum scores) is well-motivated and empirically validated against human expert grading. It represents a meaningful step toward scalable, consistent evaluation of open-ended legal reasoning.\n\nIncluding both English and German questions, spanning Swiss and international law, greatly enhances the generalizability and research utility of the benchmark.", "weaknesses": "1. All data originate from Swiss civil law contexts; thus, generalization to common law systems (e.g., US/UK) is uncertain.\n\n2. Although “LLM-as-a-Judge” performs well, it introduces dependence on closed models (GPT-4o, DeepSeek-V3) and may inherit their biases.\n\n3. The paper reports scores but lacks detailed qualitative analysis or case studies illustrating where models succeed or fail in reasoning.\n\n4. While the dataset is said to be authorized, the anonymization and ethical review process is not discussed in depth.\n\n5. Quantitative comparison with existing benchmarks (LegalBench, LexGLUE, LawBench, COLIEE) is limited to descriptive discussion rather than aligned experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FkDCRmTIS3", "reviewer": "Reviewer_PxoQ", "summary": "This paper introduces LEXAM, a comprehensive benchmark designed to evaluate the long-form legal reasoning capabilities of LLMs. LEXAM is constructed from 4,886 law exam questions collected from 340 exams across 116 law school courses, covering a wide range of subjects and degree levels in English and German. The dataset includes 2,841 open-ended questions and 2,045 multiple-choice questions. Each open-ended question is accompanied by detailed guidance outlining the expected reasoning process, such as issue spotting, rule recall, and rule application.\nThrough extensive evaluation, the authors find that current LLMs perform poorly on tasks requiring structured reasoning, highlighting persistent challenges in legal understanding. The study further demonstrates the discriminative power of LEXAM in distinguishing models with different reasoning strengths. By employing an ensemble LLM-as-a-Judge framework validated by human experts, the authors provide a scalable and consistent evaluation methodology that aligns closely with expert assessments and goes beyond traditional accuracy-based metrics, offering a new paradigm for measuring legal reasoning quality.", "questions": "(1) Are the English and German questions parallel translations, or are they completely independent question sets? If they are translations, were they verified by qualified legal professionals to ensure semantic and legal-effect equivalence?\n(2) The results show a substantial English–German performance gap. How do the authors disentangle legal reasoning ability from multilingual or translation-related effects in this evaluation?\n(3) How do the authors attempt to address or analyze the potential circularity of using LLMs to evaluate other LLMs’ answers—beyond the reported human consistency checks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(1) Comprehensive Real-World Dataset. The paper presents a comprehensive real-world dataset for legal reasoning, derived from 340 actual law exams across 116 courses at the University of Zurich. The dataset contains 4,886 questions in both English and German. Notably, it includes long-form, human-written reference answers accompanied by explicit guidance on the expected reasoning process—resources that remain rare in existing legal NLP benchmarks.\n(2) Broad Empirical Evaluation. The authors conduct an extensive empirical study covering 26 diverse large language models, including reasoning-oriented systems such as GPT-5, DeepSeek-R1, Gemini-2.5, and Claude-3.7.\n(3) Evaluation Design and Reliability. For open questions, the paper adopts an LLM-as-a-Judge framework that ensembles multiple models. This ensemble approach appears more robust than traditional single-model judging methods. For multiple-choice questions, the authors further design perturbed variants by varying the number of answer options, in order to test model robustness and sensitivity to superficial cues.", "weaknesses": "(1) Limited Scope and Jurisdictional Coverage. The dataset is almost entirely derived from Swiss and European law exams at the University of Zurich, using only English and German. Although the authors briefly mention cross-system differences, the dataset design remains heavily biased toward the civil-law tradition. It lacks representation of common-law jurisdictions such as the United States and civil-law contexts like China’s legal system. As a result, while the contribution is valuable, its applicability and generalizability across broader legal systems are clearly limited.\n(2) Question Equivalence and Multilingual Validity. LEXAM claims to be a “multilingual legal reasoning benchmark,” but it does not clarify whether the English and German questions are parallel translations or independent items. Nor does it specify whether bilingual legal experts verified semantic and legal-effect equivalence between the two languages. The authors should explicitly state this point, as it directly affects the validity of cross-language comparisons.\n(3) Evaluation Framework and Conceptual Limitations. While the proposed ensemble-based LLM-as-a-Judge approach is more robust than using a single LLM judge, it does not introduce a fundamentally new or thought-provoking evaluation paradigm. Moreover, the logic of using LLMs to generate answers, then using other (or similar) LLMs to grade those answers, and finally using those grades to infer the reasoning capability of the same class of models, raises methodological concerns about circularity and self-referential validation.\n(4) Entanglement Between Legal Reasoning and Multilingual Processing. The models exhibit strong language sensitivity. As shown in Figure 5, performance in English is consistently and significantly higher than in German. This suggests that the benchmark’s evaluation signal mixes legal reasoning ability with multilingual processing and translation competence. Since the benchmark does not attempt to disentangle these two components, it becomes difficult to interpret the reported results as a clean measure of “legal reasoning” alone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l38RGZOzhM", "reviewer": "Reviewer_ASGZ", "summary": "The paper presents a new dataset consisting of ~4500 examples with multi-choice and long form questions taken from law exams at the University of Zurich. The LEXam benchmark covers English and German languages and multiple jurisdictions from Switzerland, Europe and international. It also provides an extensive array of baseline experiments covering multiple open and closed weight LMs of different sizes and families. Results show that the questions are challenging (GPT-5 achieving best performance ~70% for open and ~63% for multi-choice).", "questions": "- See weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Valuable resource that will enable further research in LLM capabilities in a very critical domain such as law\n- Extensive benchmarking and analysis of results.", "weaknesses": "- Not clear what the human performance is on the dataset. It might be a good idea to include it in Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CGnZrLdIZM", "reviewer": "Reviewer_zGey", "summary": "The paper introduces LEXAM, a benchmark evaluating the legal reasoning capabilities of large language models (LLMs). The dataset comprises 4,886 law exam questions sourced from 340 law exams across 116 courses at the University of Zurich, covering a range of legal subjects in both English and German. The authors propose an ensemble LLM-as-a-Judge evaluation paradigm with expert-crafted prompts to evaluate the reasoning quality of legal reasoning and validated it against human expert assessments.", "questions": "See in the weak points.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea of evaluation legal reasoning is important.\n2. The authors have conducted rigorous validation of LLM-as-judges’ outputs against expert assessments.\n3. The inclusion of both English and German questions", "weaknesses": "1. Some details about the data collection are missing. For example, how did we select the 385 questions for the auxiliary MCQ dataset? Randomly sampled from the 1660 MCQs? How did we get the distribution of questions? Did you apply any filtering process to the question selections or simply use all the questions from the courses? Then how did we select the courses?\n2. The paper acknowledges the inherent complexity of legal reasoning, but, according the prompt shown in E.2, it seems that no special guideline based on legal expertise is incorporated in the LLM-as-judge system. If so, what’s the differences between judging general reasoning and judging legal reasoning? The experiments show that this paradigm can outperform human annotations on the 50 randomly selected questions. This means that either the question samples could be biased (e.g., overly simple) or legal experts don’t have a common agreements on the judgment of legal reasoning itself and thus a well-crafted prompt for general reasoning evaluation could outperform legal expert annotations directly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)