Title: SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading

Benchmark: SciEx
Summary: SciEx is a multilingual, multimodal benchmark of university-level computer science exams. It includes freeform questions with expert grading that contain both text and images. It compares to a baseline of human students, as all questions are real exam questions.

Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: scientific reasoning, problem solving

Phenomenon defined by authors: Yes
Phenomenon definition: "Solve scientific tasks" via exam-style problem solving, including reasoning, proof generation, and algorithmic thinking

Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Task definition: Answer university CS exam questions, some of which are multimodal and freeform, with human-graded performance.

Task item definition: Each item consists of a CS exam question (text and possibly images), and the LLM must generate a free-text or structured response.

Task definition detail: Questions span various formats, languages (English and German), and topics (e.g. AI, databases, algorithms)

Task source: Human exam questions (e.g. GRE questions)
Task source detail: CS exams from Karlsruhe Institute of Technology (2022–2024), authored by instructors

Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Reflects typical LLM use cases in academic or tutoring settings, but the authors specifically claim the benchmark is for scientific work, which makes QA on exam questions representative at best

Dataset size: 154
Dataset metadata available: Yes
Dataset metadata detail: max score, student average, gold reference answer, difficulty level, language, modality

Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric definition detail: Expert graders score each answer; LLM-as-a-judge methods are evaluated via correlation with these expert scores but not used in the benchmark

Metric aggregation: Simple Mean
Metric statistics: Pearson correlation, RMSE, differences to student baselines

Metric subscores: Yes
Metric subscores detail: Grouped by difficulty, modality, and language
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Comparison of expert vs LLM grading; examined factors like image use, question difficulty, language; measured grader bias. Detailed comparison to student performance, though number of students is not listed.

Taxonomy: General Science / 
New taxonomy: Specific Application / Physical Sciences