Title: SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
Benchmark: SUPER
Summary: SUPER evaluates LLM agents ability to reproduce ML experiments. They provide both end-to-end examples as well as smaller masked examples. The models are tested both on the outcome (e.g., reaching a score) and reaching different landmarks.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: ML Research reproduction
Phenomenon defined by authors: Yes
Phenomenon definition: A successful agent would need to set up the experiment by installing dependencies, downloading the provided data, and making code changes to load it (...), then execute the training script while responding to unexpected issues such as an incompatible dependency (...), and finally report the result metrics. 
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: The end-to-end task is to deterministically reproduce an experiment based on instructions, e.g., "train a classifier using REPO and DATASET and report accuracy". There are also subtasks for more fine-grained evaluation. 
Task item definition: For the end-to-end, a task consists of instructions (e.g., task and hyperparams), a repository, a metric to be reported,  and a dataset. For the subtasks, 
Task source: Expert-crafted task examples (e.g. hand-written examples), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: The LLM-generated tasks are only used for the dev set - not the actual benchmark. 
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: Reproducing code is a quite realistic task - however, it is slightly more specified in the benchmark than it would normally be.
Dataset size: 45
Dataset size detail: The main benchmark is for end-to-end, the masked is based on simpler subtasks, and the 
Dataset size notes: Masked: 152, automatic (dev): 604
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Response format detail: It is evaluated on the _outcomes_ of executable code. 
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: The main outcome is a reported metric that comes from successfully reproducing the results. 
Metric aggregation: Simple Mean
Metric statistics: mostly mean - variance only reported for best model. 
Metric subscores: Yes
Metric subscores detail: There are both scores for end-to-end and landmark (i.e., partial) solutions.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Non-profit
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: No
Author validity detail: Since the authors use human annotators for gold solutions, 100% is implicitly the human baseline. 
Taxonomy: Agents / 
New taxonomy: General Application / Coding Agents