Title: Mercury: A Code Efficiency Benchmark for Code Large Language Models
Benchmark: Mercury
Summary: Introduces the first code efficiency benchmark for Code LLMs. Benchmark functional correctness and code efficiency simultaneously
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Code Efficiency
Phenomenon defined by authors: Yes
Phenomenon definition: Code efficiency refers to the performance measure of time and space complexity to accomplish a specific task (they explicitly say they focus on the time dimension only)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: Define code efficiency over time and memory elements. Just focus on the time element in this benchmark.
Task definition: Code generation problems from Leedcode. Natural language to code tasks.
Task item definition: A Python Leetcode question.
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 256
Dataset size notes: 1,633
Dataset metadata available: Yes
Dataset metadata detail: Difficulty level.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: runtime percentile of the LLM-generated code on the runtime distribution supported by corresponding solutions (the Leetcode solutions)
Metric definition detail: The average question has 18.4 reference solutions (to form the runtime distribution)
Metric aggregation: Weighted Mean
Metric subscores: Yes
Metric subscores detail: Difficulty level.
Metric metascoring: Mean score @ k
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation