Title: XCODEEVAL: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval
Benchmark: XCODEEVAL
Summary: A large multilingual, multitask coding benchmark. Includes classification, generation, translation and retrival tasks.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Coding
Phenomenon defined by authors: No
Phenomenon definition: [Explanation] The seven tasks are defined but there is no global definition of the top phenomenon.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Two classification, three generative, and two retrieval tasks. All fairly well defined e.g. the "Code Compilation" task is a binary classification task to establish whether code is compilable or not.
Task item definition: It includes a language description of the problem, input/output (i/o) description, and a few i/o examples. The task is then to e.g. generate code. 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: Pretty unclear exactly where it all came from.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 1354
Dataset size detail: Extremely unclear how many examples in each set + multiple different types of test set.
Dataset size notes: Unclear
Dataset metadata available: Yes
Dataset metadata detail: Task, language, difficulty, release date, 
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), Execution-based evaluation. Fairly comprehensive.
Metric aggregation: Simple Mean
Metric statistics: Mean
Metric subscores: Yes
Metric subscores detail: Task, coding language, difficulty level
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Link in paper is broken but benchmark is still available on GitHub
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Author validity detail: Discuss the risk of prior data leakage (somewhat related to task validity)
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation