Title: SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents
Benchmark: SWT-Bench
Summary: A benchmark for generating code tests (unit tests) from natural language user GitHub issues.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Automatic code test generation (i.e. generating unit tests for issues)
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to generate valid tests to reproduce an issue in a codebase.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a GitHub issue in natural language, you must write tests to reproduces the described issue.
Task item definition: A GitHub issue (taken from SWE-Bench), code that contains the issue and code with a 'golden patch' that has the issue fixed. The goal is to write unit tests that fail on the faulty code but pass after the patch is added.
Task definition detail: Very comprehensive details about task definition.
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task source detail: SWE-bench, which originates from real GitHub issues
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 1900
Dataset metadata available: Yes
Dataset metadata detail: Length of the GitHub issue in tokens, original GitHub repository
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Whether the faulty code fails on the test and the gold-standard code passes it.
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Description length in tokens, original GitHub repository
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Limitations in how the phenomenon was operationalised - all problems are in Python.
Taxonomy: Agents / Coding
New taxonomy: General Application / Coding Agents