Title: EffiBench: Benchmarking the Efficiency of Automatically Generated Code
Benchmark: EffiBench
Summary: The paper introduces EffiBench, a benchmark of LeetCode problems designed to assess the time and memory efficiency of LLM-written programs. Problems are filtered from HuggingFace to problems corresponding to conventional algorithmic problem types (DFS, binary search, greedy, ...).
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: writing efficient code
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Generate a snippet of python code to solve a LeetCode problem, matching the desired output behaviour. Solutions are checked against hidden unit tests for correctness and later efficiency.
Task item definition: A LeetCode problem looks like a prompt, following by explicit "Input" and "Output" descriptions, and then a correct example. The dataset contains canonical human-written solutions which are the most upvoted on LeetCode forums.
Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues)
Task source detail: The authors filter by LeetCode interview frequency, meaning task instances are "real examples" by being coding problems encountered in technical interviews.
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 1,000
Dataset metadata available: Yes
Dataset metadata detail: difficulty level, algorithm type
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Execution time and memory usage efficiency; unit test correctness
Metric definition detail: metric_face_validity is "No" because efficiency scores are computed on only the questions LLMs answer correctly, meaning the worst LLMs get the highest efficiency scores because they can only answer simple questions with smaller variation in possible solutions.
Metric aggregation: Weighted Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: Scores by algorithm problem type as well as pass@1 accuracy
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: No
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation