Title: ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?
Benchmark: ECCO
Summary: Evaluating the efficiency of LLM generated code.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: LLM-generated code efficiency.
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: (i) Code generation from natural language instructions and (ii) editing existing programs. 
Task item definition: A natural language description or existing program and the task is either to generate the program in an efficient way or refactor the program to make it more efficient.
Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 48
Dataset size notes: 1262, 69
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Code "Speedup" and "Memory Reduction" versus reference solutions.
Metric aggregation: Simple Mean
Metric statistics: Mean, variance
Metric subscores: Yes
Metric subscores detail: The two tasks in the benchmark
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Only uses Python problems and competitive coding exam questions, therefore "results may not be comprehensive enough to reflect the quality of model-generated programs".
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation