Title: LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models

Benchmark: LexEval
Summary: LexEval is a benchmark for evaluating LLMs' legal capabilities in Chinese. It includes 23 tasks spanning six cognitive levels and tests both general and legal-specific models.

Contribution: Scale and some novel expert-annotated datasets, but mostly regrouping of existing ones.

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Legal understanding and reasoning in Chinese

Phenomenon defined by authors: Yes
Phenomenon definition: Legal understanding is defined through six cognitive ability levels (e.g., Memorization, Logic Inference, Ethics), each mapped to concrete tasks.

Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: 23 tasks grouped under six legal cognitive abilities, including classification and generation.

Task item definition: Each task provides a legal input (e.g. case description or statute) and asks for a multiple-choice or text-generated response.

Task definition detail: Tasks are derived from real law exams, curated datasets, and expert annotations, unified in format.

Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Modified from another benchmark (e.g. translation into another language)
Task source detail: Data includes CAIL, JEC-QA, LeCaRD, legal exams, and 18 legal experts contributing 6,250 examples.

Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 14150
Dataset metadata available: Yes
Dataset metadata detail: Task ID, task type, ability category, source, mean input length

Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Response format detail: Most tasks are MCQ; generation tasks require structured legal text output. (But still freeform, more 'free response' than 'structured response')

Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: Basic comparisons across models; no significance tests

Metric subscores: Yes
Metric subscores detail: By ability level and task

Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Law / 
New taxonomy: Specific Application / Law