Title: PatentEval: Understanding Errors in Patent Generation
Benchmark: PatentEval
Summary: PatentEval, a benchmark annotated by human experts, tailored for assessing language models of different sizes and capacities. This includes pairwise comparisons and detailed analysis of error types in each output.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: patent text evaluation
Phenomenon defined by authors: Yes
Phenomenon definition: Ability to generate high-quality patent text for two distinct tasks in machine-generated patent texts: claims-to-abstract generation, and the generation of the next claim given previous ones.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: "Evaluating two distinct tasks in machine-generated patent texts: claims-to-abstract generation, and the generation of the next claim given previous ones"
Task item definition: Patent claims as input, paired with two outputs (machine-generated or human-written claims) that are evaluated through pairwise comparison by human experts.
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: Real subtasks within the patent drafting process, though they don't cover the complete patent drafting workflow.
Dataset size: 400
Dataset metadata available: Yes
Dataset metadata detail: patent domain, pairwise comparison, claim dependency
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric aggregation: Win and Draw Rates
Metric statistics: Win and draw rates from pairwise comparisons. For automated metrics and human judgment evaluation: Kendall's Tau 
Metric subscores: Yes
Metric subscores detail: Error types, model, domain
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail:  (1) Expert annotation from patent lawyers; (2) Correlation analysis between human judgments and automated metrics; (3) Statistical evaluation of pairwise annotations; (4) Detailed error analysis
Taxonomy: Law / 
New taxonomy: Specific Application / Law