Title: Multi-lingual Evaluation of Code Generation Models
Benchmark: MBXP and Multilingual HumanEval (two benchmarks)
Summary: Measures code generation capabilities across 10 programming languages (Java, JavaScript, TypeScript, Go, Ruby, Kotlin, PHP, C#, Scala, C++, Swift, and Perl). Transforms existing Python benchmarks into other languages.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Code generation
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Generating code to complete a function given a docstring.
Task item definition: Each example contains a function signature and a docstring. The docstring is detailed and contains examples of the desired behaviour.
Task definition detail: Fairly limited discussion given it was a transpiled from existing benchmarks.
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset metadata available: Yes
Dataset metadata detail: Programming language.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Accuracy when the generated function is executed.
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: Programming language
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Discuss this briefly in the limitations. Say that they assume this is representative of all code completion problems.
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation