Title: AGB-DE: A Corpus for the Automated Legal Assessment of Clauses in German Consumer Contracts
Benchmark: AGB-DE
Summary: In this paper, the authors introduce AGB-DE, a corpus of 3,764 clauses from German consumer contracts that have been annotated and legally assessed by legal experts. Together with the data, they present a first baseline for the task of detecting potentially void
clauses, comparing the performance of an SVM baseline with three fine-tuned open language models and the performance of GPT-3.5. The results show the challenging nature of the task, with no approach exceeding an F1-score of 0.54. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon:  consumer contract understanding
Phenomenon defined by authors: No
Phenomenon definition: It's not defined 
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Not defined
Task definition: The task is to predict whether a given contract clause is valid or not.
Task item definition: A contrac clause (paragraph)
Task source: Expert-crafted task examples (e.g. hand-written examples)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: Train:3004, Test: 755
Dataset metadata available: Yes
Dataset metadata detail: Topic, language, number of sentences
Sampling method: Unknown
Split reviewed: Test, Train
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Unknown
Metric statistics: Unkown 
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Authors discuss validity: No
Taxonomy: Law / 
New taxonomy: Specific Application / Law