Title: TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models
Benchmark: TRIGO
Summary: The paper proposes TRIGO, a benchmark evaluating the ability of generative language models to formally reduce trigonometric expressions using the Lean proof assistant. It includes manually annotated real-world problems and procedurally generated problems of controlled difficulty.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Formal mathematical reasoning with number and formula manipulation
Phenomenon defined by authors: Yes
Phenomenon definition: The task requires models to reduce trigonometric expressions through formal proofs, manipulating and combining numbers and formulas in the Lean formal language."
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Reduce a given trigonometric expression step-by-step into a simplified form using Lean proof tactics.
Task item definition: Each item consists of a trigonometric expression (the goal) and the requirement to produce a formal proof sequence leading to its simplification.
Task definition detail: Problems come from real-world high school math questions and synthetic transformations, annotated step-by-step and formalized into Lean.
Task source: Human exam questions (e.g. GRE questions), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: TRIGO-real samples from real textbooks and exams; TRIGO-web curated online; TRIGO-gen synthetically generated via rule-based expansions.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 86 (TRIGO-real test split)
Dataset size notes: Train: 299, Validation: 42 (TRIGO-real)
Dataset metadata available: Yes
Dataset metadata detail: proof length, difficulty
Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Success is defined as completing the proof by applying correct Lean tactics until no goals remain.
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: No, no link is provided
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors discuss construct validity by showing that TRIGO captures both formulaic and numerical challenges not previously explored in other benchmarks.
Taxonomy: Reasoning / Mathematical
New taxonomy: General Capability / Mathematical Reasoning