Real-FIM-Eval Benchmark

Real-FIM-Eval is a benchmark to evaluate Fill-in-the-Middle (FIM) capabilities of code LLMs in scenarios that reflect real-world code completion. Real-FIM-Eval is built using data from GitHub commits between Jan. 2025 and Feb. 2025. These commits originate from 228 permissively licensed GitHub repositories with 10,000+ stars, spanning the top 12 widely-used programming languages.

• Paper: https://arxiv.org/abs/2506.00204

We process git commits using diff_match_patch (https://github.com/google/diff-match-patch) to identify line-level changes. The commits are then categorized into two splits: Add and Edit.

• Add (17,879 examples): This split uses git commits where a developer added a new segment of code into an existing file. To create the FIM prompt, we treat the added code segment as the "middle" part that the language model needs to predict. The code surrounding the addition forms the prefix (code before) and the suffix (code after).

• Edit (13,922 examples): This split uses git commits where a developer modified existing code by removing a segment and replacing it with a new one. We present this task to LLMs in a conflict-merge format. The prompt includes the code context (prefix and suffix) and marks the original code segment (to be removed). The model is asked to infill the updated code segment.

Evaluation Metric

The evaluation metric is character-level perplexity (lower = better), calculated as:

$$\exp (-\frac{1}{\mathrm{n}\mathrm{\_}\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{s}(\mathrm{m}\mathrm{i}\mathrm{d})}\sum _{i\in \mathrm{m}\mathrm{i}\mathrm{d}}\log p_i)\backslash exp\backslash left(-\backslash frac\{1\}\{\backslash mathrm\{n\backslash \_chars\}(\backslash mathrm\{mid\})\}\; \backslash sum\_\{i\backslash in\backslash mathrm\{mid\}\}\; \backslash log\; p\_\{i\}\backslash right)$$

In this formula, p_i represents the probability the model assigns to the i-th token of the ground truth (only the "middle" part that the model is tasked to infill), and n_char(mid) is the total number of characters in that ground truth middle segment.

Statistics of Programming Languages

Copyright Information

The dataset is a verbatim snapshot of source‑code files from public GitHub repositories. Each data example retains the exact license chosen by its original author(s). The repository of origin for every file is recorded in that data example’s repo field. To reuse or redistribute any code sample, you must comply with the license found in the upstream repository (see the LICENSE file in that repo for full terms). The dataset organization and the README.md file are licensed under Creative Commons Attribution 4.0 International (CC‑BY‑4.0).