Title: R2E: Turning any Github Repository into a Programming Agent Environment
Benchmark: R2E-Eval1
Summary: We present Repository to Environment (R2E), a framework that can turn any GitHub repository into a test environment to evaluate the performance of code-generating systems, both static and interactive. 

We instantiate our framework to build the first large-scale benchmark, R2E-Eval1, for building realistic environments for AI coding assistants. 

Our results demonstrate that even when SOTA models cannot generate correct solutions with advanced prompting techniques, they can effectively use environment feedback highlighting the need to move from static functional coding to interactive programming paradigm. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Code generation
Phenomenon defined by authors: Yes
Phenomenon definition: The ability of LLM coding agents to solve real-world software engineering tasks by 
 modifying codebases and using test outcomes to guide code generation. 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The benchmark evaluates LLM coding agents for their ability to interact with GitHub repositories and do test generation, code repair, and code validation.
Task item definition: A single item consists of a GitHub repository, a target task for the LLM agent to solve (e.g, implement a function or fix a bug) and an evaluation outcome.
Task source: Real task examples (e.g. GitHub issues), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 1000 coding-related tasks across 300 repositories.
Dataset metadata available: Yes
Dataset metadata detail: repository, type of task, programming language
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: No
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Agents / Coding
New taxonomy: General Application / Coding Agents