Title: SciCode: A Research Coding Benchmark Curated by Scientists
Benchmark: SciCode
Summary: SciCode is a benchmark consisting of multi-step scientific code generation problems. Scientists curate code implementations from published research in their field and write test cases for python implementations of these problems. Frontier reasoning models evaluated on SciCode struggle to achieve double-digit accuracy on the most "realistic" evaluation setup.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Coding for scientific research problems
Phenomenon defined by authors: Yes
Phenomenon definition: "Solving the main problems requires deep scientific background knowledge, strong analytical capabilities to decompose complex problems into simpler ones and correctly solve each, and the ability to integrate partial into complete solutions" (2)
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: A problem is presented from a natural scientific field, and the model must compose python functions for each subproblem and integrate their behaviour into the solution of the main problem. Automated test cases run against the proposed answer.
Task item definition: Models receive a main problem and each subproblem one at a time, with potential scientific context. A subproblem might be "Write a Haldane model Hamiltonian on a hexagonal lattice," asking for a python function.
Task definition detail: Different variants of the task subtly change the phenomenon under study (the authors address this). E.g., an evaluation mode that removes scientific background is testing "inherent scientific knowledge," whereas included background shifts focus to "coding and instruction-following capabilities" (6).
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 288
Dataset size detail: Subproblems decomposed from 80 main problems (65 test, 15 train)
Dataset size notes: 50
Dataset metadata available: Yes
Dataset metadata detail: field, subfield
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), Unit test cases
Metric aggregation: Subproblem accuracy aggregated by main problem
Metric statistics: mean/sum, where problem correct means all subproblems must be correct
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: "Realistic and current problems sourced from scientists’ everyday research tasks or influential papers. This ensures SciCode’s relevance to real-world applications"
"Problems curated to have zero overlap with publicly available datasets to prevent potential data contamination" (2)
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation
Procedural notes: Most impressive setup for expert annotation I have seen in my batch so far