Title: Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks
Benchmark: SAFIM
Summary: Create a "Fill-in-the-Middle" code benchmark for LLMs and uses it to make claims about effective pretraining in code LLMs.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Fill-in-the-middle coding tasks (specific form of code generation)
Phenomenon defined by authors: Yes
Phenomenon definition: Generating code to fill in a 1-5 tokens-long gap.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: A model is presented with code with a 'code block' masked. The model then has to generate code to complete the  function based on natural language instructions.
Task item definition: A natural language description of the function along with an incomplete function (some of it is masked). 
Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues)
Task source detail: Codeforces problems (coding exam questions) and GitHub (real-world cases)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 17,720
Dataset metadata available: Yes
Dataset metadata detail: Code topic 
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Execution-Based Evaluation (unit tests)
Metric aggregation: Simple Mean
Metric statistics: Mean, 
Metric subscores: Yes
Metric subscores detail: Code topic 
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Author validity detail: The benchmark has high construct validity because they essentially define the phenomena as the task e.g. don't call it a 'code generation' benchmark as many others do.
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation