Title: InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback
Benchmark: InterCode
Summary: A standard coding benchmark with an interactive environment. An early agentic coding benchmark.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Interactive code generation
Phenomenon defined by authors: No
Phenomenon definition: (Implied definition) - Involves writing code over many steps to achieve some task an a code environment
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Given a coding instruction in natural language, the agent has to issue code over multiple steps to achieve the goal. They can choose to submit their work at any time. 
Task item definition: A single set of natural language instructions and a docker-based environment. Each question has a gold standard solution.
Task source: Modified from another benchmark (e.g. translation into another language)
Task source detail: Adapt text-to-code datasets (NL2Bash, Spider, MBPP) to an agentic setting.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 1351
Dataset metadata available: Yes
Dataset metadata detail: Difficulty level, programming language
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Extended interaction (e.g. conversation, calling an API and processing the response), Structured response (e.g. valid JSON, API call alone)
Metric definition: Execution-based evaluation (unit tests)
Metric aggregation: Simple Mean
Metric statistics: Mean, standard errors
Metric subscores: Yes
Metric subscores detail: Problem difficulty, programming language
Metric metascoring: None, Minor evaluation of success after n actions (agentic setting). Not main metrics.
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Limited set of programming languages
Taxonomy: Agents / Coding
New taxonomy: General Application / Coding Agents