Title: MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
Benchmark: MLAgentBench
Summary: MLAgentBench benchmarks the ability of LLM agents to perform machine learning experiments. The benchmark comprises different tasks from canonical classification to code optimization. A success is beating the baseline by more than 10%
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: ML Experimentation
Phenomenon defined by authors: No
Phenomenon definition: competence in accomplishing the task, i.e., the fraction of time that the agent was able to improve the performance metric
Definition scope: Subset
Definition integrity: Authors' description is unclear
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Purpose notes: While the definition is very high-level (i.e., "ML experimentation"), the authors make no claim that their benchmark is comprehensive.
Task definition: A task is broadly to improve on some starter code either in terms of performance of the trained model (e.g., classification accuracy) or code efficiency (e.g., clock speed). Each task has a description with instructions and goals as well as a set of starter files.
Task item definition: A dataset (e.g., CIFAR), a starter model (defined in a `train.py`), and a metric (e.g., `test accuracy`). 
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Task ecology detail: The task of improving an existing codebase/doing a kaggle challenge has a degree of gamification but is still quite realistic. 
Dataset size: 13
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: functioning code (i.e., a .py script or model artifacts)
Metric definition: Score improvement of script
Metric definition detail: On a high-level, all metrics are "did the model improve $SCORE by more than 10%?" averaged  over 8 trials. 
Metric statistics: mean over 8 runs. 
Metric subscores: Yes
Metric subscores detail: Measure for each task
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Agents / Coding
New taxonomy: General Application / Coding Agents