Title: RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
Benchmark: REPOBENCH
Summary: Evaluating whether models can do repository-level code generation.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Repository-level coding tasks.
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Three sub-tasks. (1) the ability to retrieve the most relevant code snippets from a repo (2) predict the next line of code and (3) do both simultaneously.
Task item definition: Retrieval: identify the most relevant code snippets to predict the next line given an in-file context. Generation: predict the next line of code based on a given in-file context.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 1669
Dataset size notes: 25301
Dataset metadata available: Yes
Dataset metadata detail: Task, difficulty level, different code masking settings
Sampling method: Random sample (creators defined a task space and sampled from it), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric definition detail: Not the best. Most programming benchmarks use some kind of execution base metric. Not applicable for the retrieval task here but would be applicable for the generation task.
Metric aggregation: Simple Mean
Metric statistics: mean,
Metric subscores: Yes
Metric subscores detail: Task, programming language,
Metric metascoring: accuracy@k
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation