Title: CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
Benchmark: CodeScope
Summary: Evaluation of code understanding and generation capacities. "An execution-based, multilingual, multitask, multidimensional evaluation benchmark"
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: LLM capabilities on coding tasks
Phenomenon defined by authors: Yes
Phenomenon definition: Defines the high-level phenomenon as consisting of eight different sub-tasks (code summarization, code smell, code review, automated testing, Program synthesis, Code translation, Code repair, Code optimization). Each of these has a clear definition in the paper.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: They explicitly make big claims about the phenomenon. "We built the first-ever comprehensive benchmark for evaluating LLMs on code understanding and generation tasks"
Task definition: Each of the eight sub-tasks has a clear operationalisation e.g. "The input is a programming scenario described in natural language, including sample inputs and outputs of the problem, while the expected output is code that can solve the corresponding problem"
Task item definition: Eight different categories, but all are some coding task with instructions in natural language e.g. generate a function which specific characteristics.
Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task source detail: Many different sources.
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 13390
Dataset metadata available: Yes
Dataset metadata detail: Task, Type of task (Summarisation, problem solving, efficiency)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Execution-based metrics.
Metric aggregation: Simple Mean
Metric statistics: Mean, standard deviation
Metric subscores: Yes
Metric subscores detail: Tasks, Task type
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Mixed. For the execution-based tasks, yes, but for the code summarisation tasks they use BLEU/CodeBLEU
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation