Title: EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations
Benchmark: EvoCodeBench
Summary: Code generation benchmark with evolving questions (updated every 6 months)
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Code generation
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Given a requirement and a repository, LLMs are tasked to generate the code for the repository.
Task item definition: An LLM has to generate a function given requirements and the rest of a repo (e.g. repo-level code generation). 
Task definition detail: Fairly unclear.
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 275
Dataset metadata available: Yes
Dataset metadata detail: Domain of codebase
Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Execution-based / functional correctness. Pass unit tests.
Metric aggregation: Simple Mean
Metric statistics: Mean, 
Metric subscores: Yes
Metric subscores detail: Domain of the codebase
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes, but fairly poor task definition.
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Only covers Python code generation.
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation