Title: SCIBENCH: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
Benchmark: SciBench
Summary: SciBench is a dataset of ~1000 college-level scientific questions from maths, physics and chemistry
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: College level scientific reasoning
Phenomenon defined by authors: No
Phenomenon definition: Distinct from existing benchmarks, all of the problems are
open-ended, free-response questions that demand multi-step
reasoning abilities, the understanding of scientific concepts,
the retrieval of domain-specific knowledge (e.g., equations
and theorems), and complex numeric computation capabilities (e.g., calculus or differential equations)
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Colege level science questions collected from textbooks. Short (1-2 sentences) question with short (~20 characters) free form response. 
Task item definition: Problem (fund)
Two charged particles are fixed to an x axis: Particle 1 of charge q1 = 2.1 × 10−8C is at position x = 20 cm and particle 2 of charge q2 = −4.00q1 is at position
x = 70 cm. At what coordinate on the axis (other than at infinity) is the net electric field produced by the two particles equal to zero?
Answer: −30 cm
Task source: Human exam questions (e.g. GRE questions), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples)
Task source detail: Sourced from questions in textbooks
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: As with any science exam, it only tests one component of being a scentist.
Dataset size: 986
Dataset size detail: 869 text questions + 117 multimodal
Dataset metadata available: Yes
Dataset metadata detail: Breakdown by subject area
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Breakdown by topic: physics, chemistry etc
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / 
New taxonomy: Specific Application / Physical Sciences