Title: LawBench: Benchmarking Legal Knowledge of Large Language Models
Benchmark: LawBench
Summary: LawBench tests 21 models on 20 Chinese legal tasks (500 instances each), which are classified along Bloom's taxonomy into knowledge memorization, understanding, and application. It is the first benchmark for the Chinese legal domain, and the first for civil law (vs. common law) jurisdictions.
Contribution: Most of these tasks are compiled/sampled from existing benchmarks, notably JEC-QA and the CAIL series. However some tasks are created originally - eg. asking legal students to choose suitable questions or scraped from a legal Q&A website.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: legal knowledge memorization, understanding, and application
Phenomenon defined by authors: Yes
Phenomenon definition: LawBench is the first evaluation benchmark developed for the Chinese legal domain. It defines the phenomenon in terms of legal knowledge capabilities mapped to cognitive levels from Bloom’s Taxonomy.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: Bloom's taxonomy for task grouping
Task definition: Perform 20 specific legal functions using text-based input and return a defined output (of various forms, including classification label, summary, number)
Task item definition: Varies strongly between the 20 tasks, but generally: a legal input (fact description, question, judgement) and a required output of various forms.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Most tasks adapted from existing legal datasets: CAIL, JEC_QA, and LEVEN.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Validity varies strongly between tasks. Memorization tasks (2/20) do not reflect real-world human work. Most others are taken from benchmarks in QA format. Some are "partial real tasks" eg. answering legal questions scraped from a legal QA site.
Dataset size: 10000
Dataset metadata available: Yes
Dataset metadata detail: Task ID, blooms taxonomy level (used to indicate difficulty), task type, metric
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Response format detail: Response format varies by task. Dataset sampling above: mostly "convenience sampled"/rehashed from existing benchmarks.
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric definition detail: Metrics are task-specific.
Metric aggregation: Simple Mean
Metric statistics: Simple means and macro-averaging (mean across tasks, which is identical here because each task has same # of instances)
Metric subscores: Yes
Metric subscores detail: By task (each of 20), by blooms taxonomy level (each of memorization, understanding, application), by zero-shot vs. one-shot
Metric access: Outputs alone
Authorship: Mostly academia, 1 research institute, 1 high school
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Law / 
New taxonomy: Specific Application / Law