Gistify: Codebase-Level Understanding via Runtime Execution
Authors: Hyunji Lee
Minseon Kim
Chinmay Singh
Matheus Pereira
Atharv Sonwane
Isadora White
Elias Stengel-Eskin
Mohit Bansal
Zhengyan Shi
Alessandro Sordoni
Marc-Alexandre Côté
Xingdi Yuan
Lucas Caccia
Venue: ICLR 2026 Poster (2026)
Keywords: codebase-level understanding
runtime code execution
coding agent benchmark
Primary area: datasets and benchmarks
TL;DR: We introduce Gistify, a task where a coding LLM must create a single, minimal, self-contained file that can reproduce a specific functionality of a codebase.

Abstract: As coding agents are increasingly deployed in large codebases, the need to automatically design challenging, codebase-level evaluation is central. We propose Gistify, a task where a coding LLM must create a single, minimal, self-contained file that can reproduce a specific functionality of a codebase. The coding LLM is given full access to a codebase along with a specific entrypoint (e.g., a python command), and the generated file must replicate the output of the same command ran under the full codebase, while containing only the essential components necessary to execute the provided command. Success on Gistify requires both structural understanding of the codebase, accurate modeling of its execution flow as well as the ability to produce potentially large code patches. Our findings show that current state-of-the-art models struggle to reliably solve Gistify tasks, especially ones with long executions traces.

Published as a conference paper at ICLR 2026
GISTIFY! CODEBASE-LEVEL UNDERSTANDING VIA RUNTIME EXECUTION
Hyunji Lee
Minseon Kim?
Chinmay Singh?
Matheus Pereira"
Athary Sonwane
Isadora White*
Elias Stengel-Eskin'
Mohit Bansal
Zhengyan Shi?
Alessandro Sordoni
Mare-Alexandre Côté
Xingdi Yuan?
Lucas Caccia
'University of North Carolina at Chapel Hill
Microsoff Research
"Comell University
*University of California San Diego
UUniversity of Texas at Austin
hyunjil jiccs. wel.cc..unc.edu
debug- wwwpsegygmmmmmmmmmmmommm
ABSTRACT
As coding agents are increasingly deployed in large codebases, the need to au- tomatically design challenging, codebase-level evaluation is central. We pro- pose GISTIFY, task where coding LLM must create single, minimal, self- contained file that can reproduce a specific functionality of a codebase. The cod- ing LLM is given full access a python command), and the generated file must replicate the output of the same command ran under the full codebase, while containing only the essential compo- nents necessary to execute the provided command. Success on GISTIFY requires both structural understanding of the codebase, accurate modeling of its execution flow as well as the ability to produce potentially large code patches. Our findings show that current sfate-of-the-art models struggle to reliably solve GISTIFY tasks, especially ones with long executions traces
codebase along with specific entrypoint (e.g..
INTRODUCTION
Large language models (LLMs) are increasingly being used in code-related tasks, powering appli- cations debugging (Yuan et al. 2025) and agentic code generation (Yang et al. (0024] Liang et al.) 2025). Thus, the ability handle isolated snippets and reasoning across entire codebases, includ- ing complex file and module relationships, is becoming increasingly essential. Yet, the evaluation toolkit for assessing such capabilities has lagged behind. Recent evidence shows that widely-adopted repository-level benchmarks such as SWE-bench (Jimenez et al. (024) and RepoBench (Liu et al., 2023b) still do not require full reasoning over the whole execution and could be solved through heuristic shorteuts or retrieval of localized patches (Aleithan et al., 2024, Liang et al.) 2025). More- over, because many of these datasets rely on GitHub issues or pull requests for construction, they not easily generalizable arbitrary repositories. At the same time, coding agents are increas- ingly deployed in large, real-world codebases, highlighting the need for automatically constructed, broadly applicable, and more challenging repository-level evaluation.
To fill this gap, we introduce the GISTIFY task, which is deliberately inspired by a common prac- tice of how developers navigate and understand unfamiliar repositories. Rather than reading files in isolation, they start from a concrete exccution point such as test command or entry script often mentioned in READMES. Then, they iteratively reason over the runtime behavior such identi- fying dependencies, following control paths to uncover the codebase's 's structure and functionality. GISTIFY formalizes this practice by requiring an (agentic) coding model extract the gist of a given command, i.e. to generate a single, self-contained, minimal, and executable gistified file that faith- fully reproduces the runtime behavior of a given command as when using the original full codebase (Figure T). In addition to serving as challenging coding task, such gistified repositories might give
Denotes equal contribution
Our code is available http://gtthhh.commminooooigggg

---

Published as a conference paper at ICLR 2026
compact.py
authhy
cookot impott
gimttied fii.py
del Brsk jusermmm
from ceoki import Moodd
test requeelss py
def Juck Josemoned
fra requ compatt Importhomn sdipta Import Ridiipor
adapters.pp
from suth Inport, JOBR aith
morial Marsel)
authie
bark (arfirane)
cea MiT Pidtpptttbaaskkkaaat (w+l)
clacc Teaeiennnn morell Matonn)
Teathlossdl
umoumnt ITPRicapte (0.00
cuth
Gistify
dethest peodede
Codebese
Testhlrsel))
Command
pytest test requestspy.ttest coolde
Figure I: The GISTIFY task: given codebase and a command of entrypoint, the goal is to generate a minimal, self-contained gistified code file that faithfully reproduces the original runtime behavior using code from the given codebase.
human coders a better understanding specific functionality a given codebase, or even way to export the single functionality of interest without inheriting heavy dependencies.
To perform well in GISTIFY, an agent should generate single gistified file that satisfies four key requirements: it should be self-contained, including all necessary components from the codebase so that it can be executed independently; it should ensure execution fidelity, producing the same outputs the original codebase under the given command; should satisfy minimality, retaining only the essential code required for execution without redundant or extraneous lines; and it should guarantee faithful preservation, avoiding hallucinated or fabricated code and relying solely on con- tent from the original codebase. To assess model performance, we introduce evaluation metrics that align with these requirements, providing systematic way to measure codebase-leve understand- ing. GISTIFY requires agents to follow the execution path through the codebase without bypassing modules, i.e., understanding how relevant objects are modified along the way, and identifying which classes or functions can be simplified or removed. Since even moderately sized codebases exceed the context window of current L.Mss, success also requires effective search capabilities.
The advantages that GISTIFY brings are multiple: first, it provides direct insight into the ability of models to reason the codebase level with an understanding of runtime execution, rather than on isolated code snippets. Second, it is lightweight and broadly applicable: it requires only the repository and a test suite (or any other collection of entrypoints with a well-defined expected output) and does not require issue logs pull requests. This allows automatic construction of challenging tasks for arbitrary repositories, including private ones. Finally, gistified files themselves are valuable outputs: by compressing specific feature of large codebase into minimal file, they can be applied to various downstream tasks, including automated debugging or error localization.
We conduct experiments across variety of frameworks (mini-SWE-agemi SWE-agent, and Copi- lot) and models (GPT-5-mini, GPT-5, 3.nuee,,,-Sonnet, and Claude-Sonnet-44) and uncover several interesting findings. First, even widely used, neghhoerff frameworks and models struggle to create a successful gistified file, especially when execution traces are long and have high coverage on the repositories. Second, faithfully reproducing the test function in the generated file is strong indicator of gistified performance, as it serves as the starting step for reasoning about execution traces. Third, enabling execution tools yields small but consistent performance gains, and addition- ally providing global code context and runtime information further boosts performance. Finally, agentic models benefit from dynamically deciding what to read and refine their reasoning through multi-step trajectories, outperforming static approaches.
2 RELATED WORKS
2.1
CODERASE-LEVE. UNDERSTANDING BENCHMARK
Previous work has introduced a variety benchmarks to evaluate LLMs on codebase-level code understanding. These generally fall into three categories: question answering, code synthesis,
2

---

Published as a conference paper at ICLR 2026
and mapping natural language specifications to the entire codebase. Several benchmarks introduce codebase-lewe ceestion-answering (Strich et al. 2024; Li et al. 2024b Sahu et al. 2024 Chen et al, 2025 Hu et al. 2024, Fu et al., 2025). In these settings, the model must correctly answer questions that require an understanding of the codehase. The questions are drawn from various sources, in- cluding real-world GitHub issues and queries resembling those asked of tools like Copilot. Another line of work evaluates whether models can synthesize code by leveraging information distributed across multiple files in the codebase (Zhang et al., 2023 Liu al., 2023b Ding et al., 2023 Li et al., 2024a; Yu et al. 2024). These benchmarks include tasks such as retrieval- -augeeeed comple- tion, cross-hle refactoring, and more specialized settings such as sketch- basedd coding or codebase evolution. Moreover, there is a line of benchmark that maps natural language specifications to entire code repositories, leveraging hierarchical or multi-stage representations to capture fnter-file relation- ships and maintain consistency across a codebase (Tang et al. [023] 7an et al. 20224) Ni et al. 2025) Our work tackles a more complex setting, where models must reason over full execution traces and examine multiple files, making the task challenging, and even widely used agentic models struggle alongside static ones
2.2 RUNTIME EXECUTION
Various works have introduced benchmarks to evaluate LLMs ability to reason over code execution
runtime (Gu et al. 2024 Chen et 2024 Xie et al., 2025 Beger & Dutta 2025 Hu al. 2025)
These benchmarks typically test whether models can predict execution traces or intermediate states such as variable values, control flow, or data dependencies-giivee code and inputs, or alternatively, infer inputs from code and outputs. Some benchmarks further extend this paradigm by leveraging execution traces to construct new problems through program composition, thereby varying complex- ity in principled way. Beyond evaluation, execution traces have also been incorporated into training pipelines to strengthen models' runtime reasoning abilities (Liu et al.] 2023a Ding et al. 2024). By augmenting pre-training and fine-tuning with execution states, paths, and coverage signals, these methods help models capture program dynamics and generalize to execution- tasks. At infer- ence time, several frameworks leverage runtime feedback to iteratively guide models in debugging or completing partial programs, thereby improving performance on execution-driven tasks (Zhong al. 2024 Xue et al. 2024). In this work, we extend prior approaches by going beyond reasoning over execution traces to also reformulate programs; the model not only tracks execution but also identifies how to compress and organize code into concise, coherent file. We further show that this capability serves as a useful tool at inference time, helping models better structure and complete execution-driee drrienn---d-ennnooe tasks.
3 GISTIFY
3.1 TASK DEFINITION
As shown in Figure I when given a codebase and command as input, the coding agent must generate single gistified file that reproduces the runtime behavior of the original codebase under the given command. Specifically, the gistified file must satisfy the following requirements.
Celf-Contained: All necessary components from the given codebase must be included so that the gistified file can be executed standalone, i.c. without relying on the codebase. The model must iden- tify all relevant modules and dependencies, demonstrating understanding of fillr-file relationships.
Execution Fidelity: Executing the gistified file must replicate the original codebase's runtime be- havior. ensuring the model captures the dynamic execution, not just static code patterns.
Minimalism: Only the code essential to reproducing the runtime behavior should be preserved, with unused functions and objects pruned. This requires ane-grained understanding of the code to identify which lines are actually executed and essential for the task.
Grounded Preservation: No hallucinated code may be introduced. All content must be derived directly from the original codebase. This ensures the task evaluates the model's understanding of the codebase, rather than its ability to generate arbitrary code that happens to satisfy the command.
"ee Appendix A.1 for related works regarding "Methods for Codehasc-level Understanding"
3

---

Published as a conference paper at ICLR 2026
3.2 EVALUATION PROTOCOL
There are two imputs to GISTIFY task: i) docker image containing the target codebase, for consistent evaluation; ii) an entrypoint, such as pytest command on one of the tests in the codebase. Test cases existing entrypoints one can easily leverage. but broadly, any command that the user would want to use to run a functionality of the existing codebase allowed.
All models are prompted to generate gistified file for the entrypoint. We can programmatically verify whether the expected behavior is preserved when the ground-truth test is run within this setup. Here, focus on comparing outputs of test commands. Once the model generates the gistified file, to ensure that execution for evaluation is based on the original test, we integrate the test code from the original codebase to the gistified file and execute This ensures that the model does not cheat by modifying the test.
3.3 METRICS
Once a gistified file is generated, we evaluate it using the given execution command. The evaluation considers three dimensions, aligned with the task requirements, to provide comprehensive measure of a model's s ability to reason over an entire codebase and understand its execution behavior. See Appendix B.1 for more details.
Execution Fidelity is binary metric where 1 means the gistified file runs successfully and produces the same output as the original codebase when executed under the given command; otherwise, is O. Failures include cases where the file is not runnable or yields different outputs. The comparison checks for tests pass/fail consistency and stdout/stderr matching.
Formally, let denote the given command, C a given codebase, and G gistified file. Define runs(c, C) as an indicator of whether executes without crashing when running over C, and out(c, C) returns the set of outputs and error traces from running with C. Then, execution fidelity is defindd as
1 [runs(c, 9) A out(c, 9) out(c,C)].
where [.:] the indicator function.
Line Execution Rate measures minimality by calculating the fraction of lines in the gistified file that are actually executed under the given command. A 100% execution rate means all lines are essential, indicating a focused and concise file. This metric is only computed for files that run successfully, since the execution trace is required to determine which lines are run.
Formally, let (g) be list of executable lines (i.e., no comments) in G. Then, the Line Execution rate is defined as
1[ is executed).
LLexec (9)
Loguo
Line Existence Rate measures the proportion of code in the gistified file that is directly preserved from the original codebase. Specifically, lines of code are grouped into blocks (classes, functions, or top-level units), and matches are computed block by block while respecting the code hierarchy. This helps avoiding false matches from common lines appearing unrelated parts of the codebase. To ensure robustness, we normalize across common variations such indentation, multi-line state- ments, and imports. A 100% existence rate indicates full fidelity to the original codebase without hallucination.
Formally, let Bg and Be be the sets blocks in the gistified file and the original codebase, respec- tively. For block b, let L(b) represent its set of lines. Then, the existence rate is defined
Σ bedo tEC(E)
Σ
i(lE Lc(b))
(3)
EbeBo
[(b)]
where 1 {fe E Lc(b)) 0. if matching block exists in Bc.
4

---

Published as a conference paper at ICLR 2026
Table I: Average Performance over three agentic frameworks with four models. We evaluated over 25 tests over 5 repositories. Execution Fidelity is shown w/o exec, and w execution tools. Line Existence and Execution are averaged across the two settings for clarity.
Framework
Model
Execution Fidelity
Line Existence
Line Execution
(wo exec w. exoc)
GPT-5-mini
17.1 /24.0
44.9
61.2
GPT-5
51.0
56.8
83.1
mini-SWE--ggenl
Claude-3.7
38.7 /43.3
66.0
69.2
Claude-4
54.0/55.3
67.0
75.7
GPT-5-mini
30.9 /45.3
47.9
74.8
GPT-5
30.7 /4.00
48.3
81.7
SWE-agent
Claude-3.7
40.7 /46.0
66.8
69.9
Claude-4
56.7 /7.3
66.3
72.9
GPT-5-mini
55.0
62.4
77.8
GPT-5
60.7
66.9
81.4
Copilot
Claude-3.7
56.00
63.0
74.4
Claude-4
58.7 61.3
69.6
80.3
EXPERIMENTS
4.1 SETTING
We conduct experiments using three widely adopted open-sourced frameworks. SWE-Agent (Yang) ctt al. (024) and GitHub Copilot (Microsoft) 2025) provide a rich scaffolding to LLM-based agents, enabling them autonomously perform software engineering tasks. This includes set of tools for creating and editing code files, navigating repositories, and executing tests. These frameworks also offer the LLM controllable cache management, and LLMs follow the standard tool-calling format. We also experiment with Mini-SWE-AAent (Yang et al. 2024), lightweight framework where LLMS only have access bash terminal solve the task. Commands are parsed from the agent output and executed directly. As the task objective for the model to use reasoning over the exe- cution flow rather than ability of tool usage, for the agentic models, we exclude the execution tools ("python", "pytest") in the default setting where execution is disabled. For all three frameworks, unless specified otherwise, hyperparameters and configurations (e.g. system prompts, cache man- agement, tools) are kept to the default values. Please see Appendix
Our evaluation spans four leading LLM variants: GPT-5 (OpenAl 2025a), GPT-5-mini (OpenAl, 2025b), Claude-3.7-Sonnet (Anthropic) 2025a), and Claude-Sonnete (Anthropic 2025b), offering different cost / performance tradeoffs. For ease or reading, we will refer to the last two models as Claude-3.7 7 and Claude-4. We use 128K token limit for all models. All experiments ran capped at 50 steps, after which whatever is generated at this moment in the gistifed file is submitted for evaluation.
the data side, we experiment over with widely used GitHub repositories which are present in SWE-Bench (requests, pylint, flask, sciki kitkit- learn, seaborn). We also explore an additional repository, debug-gym (Yuan et al. 2025 This library is relatively new and impor- tantly does not overlap with SWF-Bench. We extract and filter test sets for each repository. Namely, we remove tests whose execution is dependent on the test's file location. For the main experiment, evaluate over 25 tests for each of the 5 repositories" More details regarding the evaluation setup and prompt can be found in the Appendix C
4.2 RESULTS
We begin by giving an overview of the main results presented in Table We report results for our main evaluation protocol, where the model does not have access to execution tools (e.g. "python" and "pytest" commands), as well as the alternative. Examples of gistified files are in Appendix D.I
provide link to all the GitHub repositories used this work in Table
Wee include results larger evaluation set in Appendix D.6 and observe that they are consistent with those obtained the smaller subset.
5

---

Published as a conference paper at ICLR 2026
Table Average error rates (%) of different failure reasons when running SWE-agent across models. Error cases are categorized into four groups. The numbers in parentheses indicate the number of errors for each category.
Models
Import Error
File Creation Failure
Missing Test Function
Pytest Runtime Error
GPT-S-mini
2.1 (2)
11.3 (1T)
76.3 (72)
10.3 (10)
GPT-5
5.2 (4)
10.4
77.9 (60)
6.5
Slaude-Sonnet-3.
20.0 (10)
20.0 (10)
2.0 (1)
58.0 (29)
Claude-Sonnet-4
32.5 (13)
10.0 (4)
7.5 (3)
50.0 (20)
Claude-4 shows the most robust performance. Across all frameworks and configurations. Claude- 4 consistently provides the best performance, reaching a 54-60% average solve rate. Moreover, the model shows the highest values of Line Existence, meaning that it the most successful model at faithfully extracting code from the original codebase. We note however that GPT-5 produces the most concise outputs, with Line Execution markedly higher than other models.
Frontier models (GPT-5/ Claude-4) are strong bash users. When looking performance on mini-swe-agent, where the models only have access to a bash terminal to solve the task, both models perform relatively well, solving over half of the tasks. Importantly, this is not the case for smaller and previous-generee generation models.
Execution tools are not silver bullet. Overall, when comparing performance with and without execution in Table [T, we note that in most cases we observe only a small performance gain. We expected that current coding LLMs could better leverage execution tools: indeed, using tools specif- ically for runtime execution analysis, such as a debugger, could significantly help solving gistify seeing this behavior emerge,
task. However, we sharp decrease in performance for the GPT-5 model when evaluated on SWE-Agent without execu- tion tools. We performed visual inspection and noticed formatting issues when rewriting the input test function. A detailled discussion can be found in Appendix D.2
from frontier models. We observed
Small(er) models perform well with scaffolding. We note that GPT-5-mini's performance varies significantly across different evaluation settings, from 17% in a bash-only setup to 58% when pro- vided with large inventory of tools from the Copilot framework (see Appendix [D3 for full list). We note that this performance increase also reflected in the quality of the generated gist, where see notable increase in line existence and line execution.
4.3 ERROR ANALYSIS OVER EXECUTION FAILURE
We proceed with an analysis of the underlying failure causes, in order to understand which aspect of the GISTIFY task different models struggle with. Table that each model tends to fail for different reasons. See Appendix D.4 for detailed examples of each error case.
Import Error occurs when the model incorrectly imports the original codebase (e.g., import requests) instead of inlining the required modules into the gistified file. We note that this error occurs even as coding LLMs are explicitly prompted not to import the specific packages in question. Perhaps surprisingly, the best performing model, Claude-4, commits this seemingly innocuous error the most out of all four models.
File Creation Failure errors arise when the model fails to generate the gistified file. This can happen two ways: the model exceeds the maximum step limit, or the model terminates the task without any file being generated.
Missing Test Function errors occur when the generated gistified file does not contain the function implementation for the test specified in the given command, or implements the test a different structure. This happen when the model strips out the content of the test and executes it outside of the pytest wrapper, under e.g. if name = main :. Claude models tend to avoid this mistake, while this is the main source of error for GPT-5 models, specifically under the SWE-agent

---

Published as a conference paper at ICLR 2026
Table 3: Analysis of the effect of different strategies and tools (global information, execution) on the GISTIFY task. We evaluate SWE-Agent with Claude using 50 test instances from the pylint codebase. Max Steps Reached (%) indicates the percentage of runs that terminated because the maximum step limit was reached.
Max Steps Reached (%)
Ablation
Type
Execution Fidelity
Line Existence
Execution
Base GISTIPY
42.0
65.0
58.3
14.6
Tracing
48.0
75.4
62.8
Prompted Strategies
Reading
50.0
77.6
62.6
3.9
RepoGraph Tracing
52.0
76.1
60.1
6.0
Global Info (Tool)
56.0
75.1
65.1
0.0
Bash
52.0
73.1
64.2
16.0
Execution (Tool)
Edit And Execute
56.0
74.3
64.2
10.0
framework. Importantly, we observe that this error does not happen at random, but rather alongside other execution errors; we attempted to add the missing test function, and it in most cases the test fails to i.c it results in runtime emrrr This aligns with the analysis in the next section, showing a strong correlation between the task's ' success and the fidelity between the original and the generated tests.
Pytest Runtime Error occurs when the execution of the generated file fails, either due runtime етror or because the gistified output does not match the output from the original codebase. The results indicate this is the most common cause of error for the best performing model, Claude-4.
4.4 IMPORTANCE OF FAITHFULLY PRESERVING THE TEST FUNCTION
We observe that models frequently modify the test function, despite being provided with explicit instructions to copy without modification, except for unavoidable adjustments (e.g.. removing im- ports). Again, to ensure consistent evaluation, we replace the test function in the gistified file with the original version before evaluation.
To measure such modifications, we define the Test Fi Score the line-level overlap between the test code of the original file and the gistified version. High Test Fj Score indicates that the model has successfully identified and copied the correct test function the gistified file. We observe strong correlation between Test Fi Score and execution fidelity (correlation00.76 p=0.01); test instances with higher Fi scores are substantially more likely to produce successful gistified file. We hypothesize that this arises because in the GISTIFY task, models often reason backwards from the test file, thereby if the model fails from identifying or copying the test function, the subsequent reasoning process is highly likely to fail.
To better understand the impact of the first step ssep-searching. viewing, and copying the test func- tion-we conduct ablation study where remove potential failure this stage. Specifically, explicitly provide the correct test function body and signature in the prompt. so the model no longer needs to locate copy it. This isolates the effect of errors in identifying the test function. In this set- ting, we observe that Test F1 Score improves highly from the base GISTIFY 68.4 to 85.3, along with execution fidelity (from 42.0% to 60.0%). This suggests that accurately handling the test function is a critical first step to do the GISTIFY task successfully. Detailed results are in Appendix D.5
ANALYSIS
In this section, we analyze how different strategies and tools affect performance on the GISTIFY task, identify factors that contribute to its difficulty, and experiment with the use of a static coding LLM to gain deeper understanding of the task. For all experiments, evaluate 50 test instances drawn from the pylint codebase, a setting where the model generally exhibited modest performance. We use SWR-Agent paired with Claude-Sonnet-4.

---

Published as a conference paper at ICLR 2026
5.1 EFFECT OF VARIOUS STRATEGIES AND TOOLS
In this section, we analyze how different strategies and sources of information affect model perfor- mance. We begin with the simplest approach, modifying the prompt to guide the model (Prompt- Based Guidance), and then move to more explicit approaches that rely on additional tools: providing global context (Global Information via Tools) or feedback from code execution (Execution- Bassed Tools). Detailed descriptions of prompts and tools, along with examples, provided in the Ap- pendix E.
Prompt-Based Guidance We first begin with the simplest approach: modifying the prompt to provide explicit task guidance. We experiment over two settings. In the former, we prompt the model to perform step-by-step reasoning, by first predicting the execution traces and then going over them, adding relevant code snippets along the way (tracing). In the latter, similar approach is used, with explicit instructions on how to recursively determine the execution traces: starting from the test, identify the relevant components and read the files where they are defined, and repeat until the end (reading). As shown in Table 3 we observe that adding such strategies tends to enhance overall metries, giving both better execution fidelity and more faithful code extractions, as measured by line existence.
Global Information via Tools Building the above observation, we next assess the effect of explicitly providing global context through external tools, rather than predicting it. We examine two tools: (1) RepoGraph (Ouyang et al., 2024), which constructs a graph of the codebase where each node represents a line of code and edges capture connections between lines, enabling graph- based search over the entire codebase; and (2) a Tracing tool that exposes gold execution traces obtained from running the given test command. Results in Table 3 show that both tools improve performance, with the Tracing tool yielding the largest gains. This finding suggests that access to the global context, especially the gold tracing information, substantially strengthens the model's ability to perform runtime reasoning, as it can easily identify which file to look at.
Execution- Breeetion-Based Tools In Section 4.2 we saw that enabling execution tools resulted in small but consistent gains overall. this section, we examine whether having unrestricted access to a bash terminal is really necessary to observe these gains, or whether simply having access to execution logs of the generated file is enough. For this experiment we compare Bash access with a simple method that executes and prints the output of the gistified file whenever it is edited (Edit And Execute). No other execution tools are available to the agent, including runtime information about the ground truth test. The results surprising: having access fewer tools actually increases performance. Indeed, we note that when give access to a full set of bash commands, the coding LLM tends to explore more tools, increasing the overall trajectory length, and potentially reaching the maximum step limit.
5.2 TESTS WITH HIGH COVERAGE ARE HARDER TO GISTIFY
In this section, we investigate what properties makes a given test hard GISTIFY. We hypothesize that tests generating a longer and more complex execution trace would entail a harder task for the coding LLM. To this end, we investigate how two axes to measure a runtime execution's 's difficulty affect performance: the length of trace, as measured by the number of function calls executed, and the number of unique files touched by the tracing procedure. While these metrics correlate with one another, they will differ when, for example, function is looped over many times or when the location of the relevant functions is in a single file versus across multiple files.
For this experiment, we use again the same configuration as prior analysis, namely Claude-4 with 50 tests sampled from the pylint codebase. In Figure 2a] we see a clear correlation between the difficulty of a given GISTIFY task, and how complex the execution traces are, according to both metrics considered. We leverage this insight to create a GISTIFY-hard subset, where we select the 30 most difficult examples according to each. We end up with 57 unique datapoints (30 from pylint, 28 from skleamn from seaborn). On this subset, performance drops to 21%, as compared to 43%, the baseline weighted performance average following the same distribution over repositories. Overall, this selection criteria offers a promising direction for designing challenging evaluation scenarios with GISTIFY.
8

---

Published as a conference paper at ICLR 2026
Performance according to Exec. Trace Difficulty
static coding LLM
SWE-Agent
1.00
Length
mini SWE
Copliot
Number of Flles Covered
Execution Fidellty (%)
80
60
60
Scores
40
20
20
10.0. 0.21
10.2.0.4
10.4.
10.6, 0.8) 0.81
jo.8. 1.1
Execution Fidelity
Line Existence
Line Execution
Binned Quantiles according to difficulty metric
(a) Difficulty of the Gistify task measured as a func-
(b) Performance of a static coding LLM and vari-
tion of the execution trace difficulty of the underlying test.
ous agentic coding LLMs (mini-SWE-Agent, SWE- Agnet, Copilot).
5.3 STATIC CODING LLM
In this section, we experiment over how models perform static setup, where they have no access to tools and cannot iterate on the generated solution. As such static coding LLMs do not have tools, they cannot search or view files dynamically. Thereby, measure a possible upper bound for non-agentic approaches, we provide as input all files that were accessed during the original program execution (gold files). Also, as they cannot iterate over multiple steps, they have to output everything at once and are therefore restricted by the context window of the LLM. Since solving the GISTIFY task involves touching multiple files, we observe in many cases that the inputs exceed the model's maximum sequence length. Thus, sample a subset of test examples where the combined content fits within the 128K token limit of the L.M.. As shown in Figure 2b agentic models outperform static ones even when the latter receive all relevant files. This suggests that selecting files dynamically over multiple iterations is more effective than providing everything at once, which can overwhelm the mode However, interestingly, the static coding LLM setup achieves the highest Line Existence score. This is likely because the model can copy lines directly from input, yet it performs worse on Line Execution and Execution Fidelity, suggesting that models do not have a good understanding of the codebase, often copying lines that are incomplete incorrect. Additional details and chain-of- thought results are provided Appendix E.2
DISCUSSION AND CONCLUSION
In this paper, we introduced the GISTIFY task which a coding LLM extracts specific funtionality of codebase into single, self-contained file. Beyond serving standalone evaluation task that easily applicable to arbitrary repositories with a test suite, the gistified file itself also opens sev- eral promising directions for research and practical applications. Large codebases often overwhelm automated agents due to their complex dependencies, and they especially struggle when tasked with fixing bugs that span multiple files (Ganhotral 2025). In such scenarios, gistified file would greatly reduce this challenge, and enable a more efficient reasoning about the codebase without navigating through unrelated code. In other words, this file could be leveraged in other downstream tasks such as code refactoring or debugging, or even as a way to extract and share a minimal implementation of specific codebase functionality. Lastly, current limitation of the results presented the reliance on an existing test suite for given repository. We believe that the GISTIFY task can be extended arbitrary entrypoints, although issues stemming from non -on-deteriiiistic execution will need be carefully addressed. We defer proper exploration of this to future work.
In summary, with coding L.MSs increasingly being deployed in real-world software development, the need for automatically constructing evaluation setups that require codehase-leve understanding of arbitrary repositories is growing. Through extensive experiments across a range of models and frameworks, we found that state-of-the-are art LLMs still face challenges the GISTIFY task, espe-
SSee Appendix E.3 for detailed statistics the usage of various tools.

---

Published as a conference paper at ICLR 2026
cially when faced with long, complex execution traces. Our analysis shows that incorporating global code context or execution-awar tools improves performance, and agentic coding LLM tend to han- dle the task more effectively by reasoning about which files to inspect using various tools. Beyond serving as a benchmark, the gistified files themselves are valuable artifacts. They distill the essential functionality of complex systems into compact, executable form, making them easier to inspect and understand. Such files could support a range of practical applications, including debugging, refactoring, and code review, which we leave this for future work.
REFERENCES
aider. Ai pair programming in your terminal. 2025. URL UKChttpss//iithhb..om//iiiie a didertaaa= ?tab= -o--ooveeeee
Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. Swe-bench+: Enhanced coding benchmark for Ilms. arXiv preprini arXiv:2410.06992. 2024.
Anthropic. Claude sonnet 3.7.
claude- 1onnde---7- sonnet 2025a. Hybrid reasoning model; accessed: 2025-09-25..
Anthropic. Claude sonnet 4.
https://wwis pttps:/ww..ntthropi...oo
2025b.
Improved version over Sonnet 3.7; accessed: 2025-09-25.
Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D C, Arum Iyer, Suresh Parthasarathy, Sriram Rajamani, Balasubramanyan Ashok, and Shashank Shet. Codeplan: Repository-level coding using Ilms and planning. Proceedings of the ACM on Software Engineering, I(FSE): 675-698, 2024.
Claas Beger and Saikat Dutta. Coconut: Structural code understanding does not fall out of a tree. In 2025 JEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), pp- 128-136. IEEE, 2025.
Jialiang Chen, Kaifa Zhao, Jie Liu, Chao Peng, Jierui Liu, Hang Zhu, Pengfei Gao, Ping Yang, and Shuiguang Deng. Coreqa: uncovering potentials language models code repository question answering. arXiv preprint arXiv:2501.03444 2025.
Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. Reasoning runtime behavior of program with IIm: How far are we? arXiv preprint urXiv:2403.16477, 2024.
cursor. cursor. 2025. URLptttss//currot com/
Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. Crosscodecval: A diverse and multilingual benchmark for cross-file code completion. Advances Neural Information Processing Systems, 36:46701-46723, 2023.
Yangruibo Ding, Benjamin Steenhoek, Kexin Pei, Gail Kaiser, Wei Le, and Baishakhi Ray. Traced: Execution-awain aware pre-training for source code. In Proceedings of the 46th IEEE/ACM Interna- tional Conference on Software Engineering, pp. 1-12, 2024.
Lingyue Fu, Hao Guan, Bolun Zhang, Haowei Yuan, Yaoming Zhu, Jun Xu, Zongyu Wang, Lin Qiu, Xunliang Cai, Xuezhi Cao, al. Corecodebench: A configurable multi-scenario repository-Ievel benchmark. arXiv preprint arXiv:2507.0528// 2025.
Jatin Ganhotra. Do swe-agents solve multi-file issues like humans? deep dive into swe-bench ver- ified, January 2025, URUattss: ht100///wee-Deeccchmmmtppppp----
Blog post.
Dhruv Gautam, Spandan Garg, Jinu Jang, Neel Sundaresan, and Roshanak Zilouchian Moghaddam. Refactorbench: Evaluating stateful reasoning in language agents through code. arXiv preprint arXiv:2503.090 07782, 2025.
10

---

Published as a conference paper at ICLR 2026
Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. CRUXEval: A benchmark for code reasoning, understanding and execution. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scar- lett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 18568-166211 PMLR, 21-27 Jul 2024.
Ruida Hu, Chao Peng, Jingyi Ren, Bo Jiang, Xiangxin Meng. Qinyun Wu, Pengfei Gao, Xinchen Wang, and Cuiyun Gao. Coderepoqa: A large-scale benchmark for software engineering question answering. arXiv preprint arXiv:2412. 14764, 2024.
Wenhao Hu, Jinhao Duan, Chunchen Wei, Li Zhang, Yue Zhang, and Kaidi Xu. Dynacode: A dynamic complexit--wware code benchmark for evaluating large language models in code gener- ation. arXiv preprint arXiv:2503. 10452, 2025.
Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, and Robby T Tan. Codejudgebench: Benchmarking -lm-as-a-judge for coding tasks. arXiv preprint arXiv:2507.10535 2025.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 2023.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. Evocodebench: An evolving code gen- eration benchmark aligned with real-world code repositories. arXiv preprint M.Nv:22404000599, 2024a.
Linyi Li, Shijie Geng, Zhenwen Li, Yibo He, Hao Yu, Ziyue Hua, Guanghan Ning, Siwei Wang, Tao Xie, and Hongxia Yang. Infibench: Evaluating the question-answent capabilities of code large language models. Advances Neural Information Processing Systems, 37:128668- 128698, 2024b.
Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam. The swe-bench illusion: When state-of-ther of-the-art art lims remember instead of reason. arXiv preprint arXiv:2506. /2286, 2025.
Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sun- daresan, and Nan Duan. Code execution with pre-trained language models. In Anna Rogers, Jordan Graberr and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 9984-4999, Toronto, Canada, July 2023a. Association for Computa- URL
tional Linguistics. doi: 10. 000816653311223 h2t.://///.12.....nnnnaa....... org/2023. findi ngs acl 30087
308/]
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository- code auto-completion systems. arXiv preprint arXiv:2306.03001,, 2023b.
Xiangyan Liu, Bo Lan, Zhiyuan Hu, Yang Liu, Zhicheng Zhang, Fei Wang, Michael Shieh, and Wenmeng Zhou. Codexgraph: Bridging large language models and code repositories via code graph databases. arXiv preprint arXiv:2408..0 03910, 2024.
Qinyu Luo, Yining Ye, Shihao Liang Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong. Yankai Lin, Yingli Zhang, al. Repoagent: An llm-powered open-source framework for repository-level code documentation generation. arXiv preprint arXiv:2402.16666 2024.
Microsoft. Github copilot in code. 2025. URL https: //ta://ccoee visaaeeviillssse www.suuduss.oo... com/ Bocs/copil tttes//cooooooooerr////o
Ziyi Ni, Huacan Wang, Shuo Zhang, Shuo Lu, Ziyang He, Wang You, Zhenheng Tang, Yuntao Du, Bill Sun, Hongzhang Liu, et al. Gittaskbench: A benchmark for code agents solving real-world tasks through code repository leveraging. arXiv preprint arXiv:2508. 18993, 2025.
11

---

Published as a conference paper at ICLR 2026
OpenAl. Gpt-5 technical overview.
http:://paactom..pppaaa.ccmo// 2025a. Ac-
cessed: 2025-09-25.
OpenAl. Gpt-5 mini http:///ooaa////a.a.oaaaaaaaa//a 2025b. Compact variant of GPT-5; accessed: 2025-09-25.
Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hong- ming Zhang, and Dong Yu. Repograph: Enhancing ai software engineering with repository-Ievel code graph. arXiv preprint 2rXiv:2410.144: 14684, 2024.
Surya Prakash Sahu, Madhurima Mandal, Shikhar Bharadwaj, Aditya Kanade, Petros Maniatis, and Shirish Shevade. Codequeries: A dataset of semantic queries over code. In Proceedings of the 17th Innovations in Software Engineering Conference, pp. 1-11, 2024.
Disha Shrivastava, Denis Kocetkov, Harm De Vries, Dzmitry Bahdanau, and Torsten Scholak. Re- pofusion: Training code models to understand your repository. arXiv preprint arXiv:2306. 10998, 2023.
Jan Strich, Florian Schneider, Irina Nikishina, and Chris Biemann. On improving repository-leve code QA for large language models. In Xiyan Fu and Eve Fleisig (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pp. 209-244, Bangkok, Thailand, August 2024, Association for Computational Lin- guistics. ISBN 1999-8899176-9977 doi: 10. 122
Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, et al. MI-bench: Evaluating large language models and agents for machine learning tasks on repository-level code. arXiv preprint vrXiv:231..09833 2023.
Huacan Wang, Ziyi Ni, Shuo Zhang, Shuo Lu, Sen Hu, Ziyang He, Chen Hu, Jiaye Lin, Yifu Guo, Yuntao Du, et al. Repomaster: Autonomous exploration and understanding of github repositories for complex task solving. arXiv preprint arXiv:2505.21577 2025.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407. 16741, 2024.
Danning Xie, Mingwei Zheng, Xuwei Liu, Jiannan Wang. Chengpeng Wang, Lin Tan, and Xiangyu Zhang. Core: Benchmarking Ilms code reasoning capabilities through static analysis tasks. arXiv preprint arXiv:2507.00266 2025.
Zhipeng Xue, Zhipeng Gao, Shaohua Wang, Xing Hu, Xin Xia, and Shanping Li. Selfpico: Self- guided partial code execution with llms. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Softwrre Testing and Analysis. pp. 1389-1401, 2024.
John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thihty-eighth Annual Conference on Nearal Information Processing Systems, 2024.
Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianx- iang Wang, and Tao Xie. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1-12, 2024.
Xingdi Yuan, Morgane M Moss, Charbel El Feghali, Chinmay Singh, Darya Moldavskaya, Drew MacPhee, Lucas Caccia, Matheus Pereira, Minscon Kim, Alessandro Sordoni, et al. debug-gym: A text-based environment for interactive debugging. arXiv preprint arXiv:2503.2/557, 2025.
Daoguang Zan, Ailun Yu. Wei Liu, Dong Chen, Bo Shen, Wei Li, Yafen Yao, Yongshun Gong, Xiaolin Chen, Bei Guan, et al. Codes: Natural language to code repository via multi-layer sketch. arXiv preprint ar0iv:240.. /6443, 2024.
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository-Level code completion through iterative retrieval and generation. arXiv preprint 13X.:22033./25000 2023.
12

---

Published as a conference paper at ICLR 2026
Li Zhong, Zilong Wang, and Jingbo Shang. Debug like human: A large language model debugger via verifying runtime execution step-by-step. arXiv preprint 1rXi::2402.16906, 2024.
13

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
A.1 METHODS FOR CODEBASE-LEVEL UNDERSTANDING
Recent work on autonomous agents for codebase-level code understanding has focused on improv- ing code navigation, reasoning, and generation through structured representations and planning. Approaches leverage structural information of code for function-call graphs, module-dependency graphs, and hierarchical code structures to provide models with core components of reposito- ries (Wang et al. 2025 Liu et al. (024).. Another line of work integrate multi-step reasoning and state update policies to enable more effective planning over complex tasks (Bairi et al.; 2024 Gau- tam et al. 2025). Additional methods combine various agents with multiple tools streamline
codebase-level exploration and task solving (Luo et al. 2024 Zhang et al.) [2033; Shrivastava et al,
2023
Wang et al.
2024 Yang et al. 2024
Tang al.
2023 aider 2025 Microsoft 2025 cursor
2025
B GISTIFY
B.1 METRICS
Execution Fidelity Exccution fidelity measures whether the generated gistified file reproduces the same functional behavior as the original codebase under the given command. This includes produc- ing the same number of test passes or failures, as well as consistent outputs and error handling. the file's behavior matches the original codebase, it is assigned 100%; otherwise it receives 0%.
Line Execution Rate The line execution rate measures the proportion of lines in the gistified file that are actually executed when running it under the given command. We first analyze the gistified file to identify which lines executable (e.g., imports, function class definitions) versus not- executable (e.g., comments). Using a tracing function, we then determine which of the executable lines are touched during execution. The line exccution rate is computed as the fraction of executable lines that are executed. A rate of 100% indicates that the gistified file is concise and contains primarily necessary lines that are executed, while 0% indicates that non of the executable lines were touched. When calculating line execution rate, we exclude the tests where the self-containment is 0% as the goal of line execution rate is to evaluate the model's 's ability to construct concise, executable file, not to penalize failures in generating runnable code.
We classify each line of code into three categories: executable, potentially executable, and non- executable. Executable lines include imports and functional code that can be directly run. Potentially executable lines are those that may or may not be executed during a run, such as the except block of try-except statement or placeholders for classes and function definitions. Non-executable lines, such as comments, are those that have no effect on execution. To calculate the line execution rate, we first classify each line in the gistified file and then consider only the executable lines. Non-executable lines are ignored since their presence or absence does not affect execution outcomes, and potentially executable lines are excluded because they are often ambiguous (e.g., placeholders) and cannot be reliably judged as necessary or removable.
Line Existence Rate The line existence rate measures the proportion of lines in the gistified file that are directly preserved from the original codebase. We first parse both the gistified file and the original codebase into blocks, where each block corresponds to a class or function. Within classes, functions are nested under their parent class, forming a hierarchy. Lines outside of any block (e.g.. top-level statements) are treated as standalone units.
For each block the gistified file, we locate the corresponding block in the original codebase using its name and hierarchical position. If a matching block exists, compare the two line by line to determine which lines are preserved; whether the lines in the gistified block appear in the corre- sponding original block. If no match is found, all lines in that block are treated For lines outside any block, existence determined by direct comparison with top-level lines in the original codebase.
existentt
14

---

Published as a conference paper at ICLR 2026
Preprocess
Require: Gistified file G, original repository R. module name M, execution information E for lines in G
1: Preprocessing: Line Classification
2: for all lines in G do
3: assign type[{] [c] {comment, cont definition, exe.uttaabee. import blank)
4:
end for
Figure 3: Details Line Execution and Line Existence (Part 1): Preprocessing
Details Line Execution
Require: Preprocessed gistified file G, original repository module name M, execution information E for lines in G
Ensure: Line execution rate Texec
1: Line Execution Rate
2: S E G type[[] (executable, import] }}
3: Sexee f- {E É marked as executed in E}
4: Texoc Sexeel
Figure 4: Details of Line Execution and Line Existence (Part 2): Line Execution
An existence rate of 100% indicates perfect preservation of the original code without hallucinated content.
Normalization for Line-wise Code Matching Figure 3 and show the detailed procedure to compute the line existence rate and line execution rate for a gistified file. The process begins by classifying every line of the gistified file into one of the categories: comment, control flow, definition, executable, import, or blank (Figure This classification forms the basis for later filtering and comparison steps.
When calculating the line execution rate, consider only lines classified executable or import, because these are the lines whose execution status can directly observed in the execution infor- mation (Figure44. We deliberately exclude control-flow and definition lines from this rate: although they are crucial for the gistified file to behave like the original repository, whether such lines "exe- cute" not is often input- parameter-dependent and therefore not reliably captured by simple per-line execution count. The validity of control-flor and definition structures will be instead be indirectly assessed through our execution fidelity metric, which measures whether the gistified file and the original repository exhibit consistent overall execution behavior. In this way, the line exe- cution rate focuses directly measurable execution coverage, while execution fidelity provides higher-level signal about behavioral correctness.
When calculating the line existence rate, both the gistified file and the original repository are parsed into nested structural blocks such as functions or classes. Any lines that are not contained within structural block are treated separately as top-level lines (line 1-4 in Figure 5p. For all cases, when calculating the line existence rate, we do not consider comments or blanks. The algorithm then examines each structural block in the gistified file and attempts to find the matching block from the original repository. If no matching block is found, all lines in the block are considered missing (line 12-19 in Figure 5). Otherwise (line 20-40 Figure 5) each line undergoes a series normalization steps such as fixing whitespace, indents, removing trailing comments, etc. The normalized line is then evaluated according to its type. Based on the comparison, each line in the block is marked as either existing not. Once all lines have been processed, the line existence rate is computed by
15

---

Published as a conference paper at ICLR 2026
Details of Line Existence
Require: Preprocessed gistitied lile G, original repository R, module name M. execution information E for lines in G Ensure: Line existence rale Texcist
I: Structural
Parsing
2: parse G and R into nested structural blocks functions and classes)
Ba f- set blocks
4: Lrop <- lines G not contained any structural block
5: Initialization
6: for all lines in G do exists /] + undefined
8: end for
9: Block-Level Existence Analysis
10:
for all blocks Bc do
11:
orig best-matching
12:
if orig does not exist then do
13:
for lines if typee[] & (comment, blank] then existts((] +- false
14:
15:
16:
end if end for
17:
18:
continue next block end if
19:
20:
for all lines E do
21:
tyype(c] (comment, blank) then
22:
continue
23:
24:
normalize é fix spacing, remove trailing comments, remove module prefixes, and split compound ("V") statements stypeef ()] = cont ron.rollllooo then
25:
26:
if an if elif statement then
27:
set excists(2] by comparing conditional expression with curresooningg i ifelli stalements orig else if é el statement then exista[f] 4- true for else-body matches
28:
29:
30:
end If
31:
else type[e] = definition then
32:
verify existence of correspoodngg decorators (if any) in orig
33:
verity existence of each argument in the detinition sepurately orig assign exists 8] based these mutches
34:
35:
else type import then
36:
docompose into individual imports and set exists(f] by per-import matching
37:
38:
compare against lines in orig and assign exists/() [f]
accordingly
39:
40: end for
41:
end
42:
Line Existence Rate
43:
Louili {E type[t] (comment, blank]]
44:
Looxtss [E Levalid exists [2] = true)
45:
Texist Lexiat
Figure 5: Details of Line Execution and Line Existence (Part 3): Line Existence
Table 4: Line execution performance of various models in the SWE-Agent framework with Gemini- 2.5-pro as the LLM-as-Judee.
Model
Original Evaluntion
LLM-us-Judge
GPT-5-mini
GPT-5
81.7
81.3
Claude-3.7
69.9
69.1
Claude-4
72.9
71.8
how many lines are marked as existing in the original repository over all lines except for comments or blanks (line 43-45 in Figure 5).
Line Execution Performance when Adding LLM-based judge module Table 4 shows line ex- ecution performance when incorporating I.LMMas-Judge module. Adding such module allows us
16

---

Published as a conference paper at ICLR 2026
Table 5: Details of the GitHub repositories used as the test set.
Repository
URL
License
flask requests pylint scikit-leam
BSD 3-Clause
Apache-2.0
http:///ahhib...mm.ppmmm--eeee- learn
GPL 2.0
BSD 3-Clause
seaborn
Ettps: sttres//gghhab..cc///aaa///////
BSD 3-Clause
debug-gym
MIT
to evaluate not only executable and import lines but also control-flow and definition lines (e.g., try:, else:), which were skipped in the standard metric because they exist as structural lines rather than lines actually executed (Details in previous paragraph).
Given the tracing output for each line, the LLM determined whether these lines would be executed according to Python semantics and runtime control flow. line is marked as executed if the inter- preter enters it along any execution path, and not executed if it is unreachable. Following Jiang al. (2025), we use Gemini-2..5pro as the Judge model, as it shows strong performance on code-related reasoning tasks.
We re-evaluated the results using the SWE-Bench framework with the LLM-as- LLM-as-Judge aus-Judee module added. We observed that including these additional lines slightly decreases the line execution rate, which we attribute to rarely executed lines, such as except: being taken into account. The largest drop observed 1.3% (GPT-5-mini). Overall, however, the relative trends across models remain con- sistent. Further analysis shows that control-flov and definition lines make up approximately 4.1% of lines gistified file, suggesting why adding the LLM-based judge has minimal impact on the metric.
User Study on the Definition We conduct user study to assess whether our proposed metrics, line existence and line execution, successfully capture the task's intended notions of faithfulness and minimality, respectively.
We recruit three software/AI engineers annotators and provide them with 15 test cases. Each test case contains a pair of gistified files generated by different frameworks or different models. For each pair, annotators were asked choose the file that better satisfied the task's criteria of minimality and faithfulness. Annotators were given: (1) description of how the gistified files were constructed, (2) the definitions of minimality and faithfulness used in our task, (3) the two gistified files for comparison. and (4) execution -raccintrnnnggg information to help them understand the flow of each test run. 6 shows the instructions provided to annotators.
To measure alignment between human judgment and our metrics, we computed Cohen's kappa correlation between the annotators' selections and the rankings produced by line existence and line execution. We observed average Cohen's 's kappa of 0.61 (0.52, 0.72, 0.58) for minimality with line execution and 0.76 (0.81, 0.71, 0.77) for faithfulness with line existence, indicating that our metrics correspond well with human judgments of minimality and faithfulness.
C EXPERIMENTAL SETTING
C. FRAMEWORK
We evaluate experiments with three agentic frameworks: mini-SWEEAgent Yang et al. 2024). SWE-Agent (Yang et al.) 2024), and Copilot (Microsoft) 2025). Unless otherwise noted, all ex- periments are run in the default GISTIFY setup, where the model is restricted from executing any commands (e.g., python, pytest). SWE-Agent and Copilot Agent enable LLMs to interact with a codebase through suite of tools, including bash commands. These tools support capabilities such as viewing, searching, editing, and creating files or directories. In addition, Copilot Agent extends this functionality with browser integration, explicit reasoning, and API usage. mini-SWE-ageen is a simplified variant SWE-Agent that only supports bash commands. Despite its minimal design, it achieves strong performance the SWE-Bench Verified benchmark (Jimenez al. 2023). For
17

---

Published as a conference paper at ICLR 2026
User Study Instruction
You are given two code files produced by a coding L.M.. Both files attempt complete the same task: create single, minimal, self-contained file that reproduces specific functionality of codebase.
This is the prompt we provide to the model to describe the task:
[Figure C.3
Your task:
Evaluate the two files and determine which one more minimal and more faithful according to the criteria below.
1. A file is "minimal" if:
It contains only the code truly required to reproduce the runtime behavior.
Unused functions, classes, variables, imports should be removed.
The evaluator must understand the code well enough to identify which lines are actually executed and essential. However, do not penalize minor extra lines that exist solely because of formatting or structure constraints (e.g., keeping an unused method such as try-except because the format requires it).
2. file is "faithful" if:
No hallucinated code is introduced. Every piece of code must come directly from the original codebase.
The structure of the code must stay consistent with the original. For example, moving a method that originally lived inside class to the top level considered incorrect.
Simple changes such incorrect indentation or broken multi-line statements are not pe- nalized as long as the intended semantics are still clear. However, structural changes that alter the meaning of the code are penalized. For example: If the original code defines a class with inheritance, such as class ABC(DE): but the generated file changes it to class ABC:, then this is considered incorrect, because removing the parent class changes the tual operation of the code.
Figure 6: Instruction for User Study
18

---

Published as a conference paper at ICLR 2026
both Mlni-SWE-Ageent and AgE-Agent, set the maximum number of steps to 50 and run them in the same Docker environment, using the current version of the repositories.
C.2 EXPERIMENTAL TEST SET CONSTRUCTION
Table summarizes the repositories used in our evaluation. For each repository, we begin by ex- tracting all available test cases, including parameterized ones. For experimental test runs, we group tests that share the same base structure but differ only in parameterization, treating them a single test. During evaluation, however, we execute all parameterized instances and measure how many are passed, thereby assessing execution fidelity. Finally, we filter euvironmeent-dependeer tests, such as those requiring relative file paths or fixed module locations. In the main experiments, we used 25 test instances for each of the six codebases, and the analysis was conducted using 50 test instances from the pylint codebase.
C.3 PROMPT FOR GISTIFY
Figure shows the prompt used in the main experiments.
C.4 PROVIDING SPECIFIC PARAMETERS TO COMMANDS TENDS TO MAKE MODELS GENERATE PARAMETER C GISTIFIED FILES
We observe that when specific command-line parameters are provided, models often adapt the gen- crated gistified file those parameters rather than producing a fully general solution. Examples of this parameter-specific behavior are shown in Figures 8 and Accordingly, in our experiments, we group test cases based on the parameters provided to the command.
C.5 ADDITIONAL HYPERPARAMETERS
All experiments were executed with the same base set of hyperparameters. For framework specific hyperparameters (e.g. cache management, system prompts, tools), we use the default configuration made available by each framework, except for vscode-copilot-chat, where we have listed the tools used the experiments can be found in
All the models used in the were configured with maximum context window length of 128K. We used a default temperature of and performed a single run for each configuration.
D RESULTS
D.1 EXAMPLE OF GISTIFIED FILE
Figure 10 and Figure show two gistified files on the same test case with different models; each model succeed fail in generating gistified file with execution fidelity of 100% and 0%, spectively. In the successful case (Figure 10). the generated file handles both parameters correctly, achieving 100% line existence rate, 65.5% execution rate, and test Fi score of 100. In contrast, the failed case (Figure [T] cannot execute due to a missing import pytest statement. More- over, the hallucinated test function yields test F, score of O, and the file shows much lower line existence rate of 28%.
D.2 ERROR ANALYSIS OVER EXECUTION FAILURE
We categorize enrors into four types:
Import Error Figure T2 shows an example of Import Error. This occurs when the model incor- rectly imports the original repository (e.g., import requests) instead inlining the required modules into the gistified file.
We adopt this grouping design as observe that models often overfit to specific values when parameters are provided. See Appendix C.4 for details.
19

---

Published as a conference paper at ICLR 2026
Prompt for GISTIFY
I've uploaded python code repository in the directory (working dir).
There is original test invocation (the command that reproduces behavior want to preserve): problem statement Your job: create a single file named "concise.py" saved at (working dir} that *=seff-ootaiined* **minmal*** and * **xxcutable**, and when used in place of the original test run reproduces the same runtime behavior and outputs. Follow the instructions below when creating the file.
OUTPUT
Produce one file only: '(working dir /crocccciis..py..
The assistant must return only the contents of "concise.py" (no extra files, no analysis, no commentary).
HIGH-LEVEL RULES for creating 'concise.py'
1. Inline internal dependencies
Copy into 'concise.py' every function, class, top-level code from the files inside wworking dir) that is executed when running (problem statement).
Do not use 'import' statements for modules defined in (working dir).
2. Remove unexecuted lines
When copying lines 'concise.py'. keep only the lines that actually executed when running {problem statement).
Delete unused functions, classes, variables, if-else, imports, and unreachable branches. Ensure the file remains syntactically correct and minimal after removal.
Preserve original source lines
Do not rewrite reformat lines unless necessary to keep the files valid.
Do not arbitrary generate new lines that do not exist in the original [working dir} files. You may adjust indentation, remove empty "else" blocks, adapt 'try-except" structures only when required to preserve correctness.
Keep external imports
Leave imports to external libraries, frameworks, or standard runtime libraries unchanged.
Only remove or inline dependencies that come from {working dir).
5. No shortcuts or cheating
Do stub, fake, or monkey-patee saacc- external modules.
Do not reimplement or newly add third-party libraries.
Do not hard-code outputs
Do not replace test logic with simplified equivalents Preserve test behavior
The test function much remain unchanged, except for import adjustments needed to reference inlined code.
The output, exceptions, exit codes must match the original run of {problem statement).
7. Do not execute the code
Do not simulate the program (e.g., with 'pytest', 'python', any other tools)
Figure Base Prompt Template for GISTIFY Task.
File Creation Failure This error arises when the model fails to generate the gistified file. This happen in ways: (1) the model exceeds the maximum step limit (2) the model completes within the time limit but still fails to generate the new file using the tool.
Missing Test Function This occurs when the generated gistified file does not contain the modules for specified test in the given command. typically arises when the model fails to locate or copy the modules necessary for the test into the gistified file. Conceptually, this corresponds to 0% line existence rate for the test function. Since the presence the modules for the given test case is essential for validation, classify this error.
20

---

Published as a conference paper at ICLR 2026
epytest meyytest.mrrk.paae www.mm.aaaatterie.......... "value, expected",
("applicate watliic/ttio/m1 ("app1 catiiitioonn cattin/xm1", (1)),
"application/json charset-utf-8",
("application/json"" ("charset"; "utf-3"I),
} ("text/plain", ("text/plain", ()),
def Bat____ars_ccotee___eeee ceannt__tye__eetette (value, expected) assert carre__onteet__yeeee (value) = expected
(a) Original Test Case
def htt__prrrre_ccttt___tooo_eeeeeee ():
"**Test for the
function with application
json and charset-utf-$"".
value = "application/jsen
charset-utf-g"
expected = ("application/jsen ("charset": "utf-8"])
assert _wwrss _arse_contentttyyo__ooooe (value) expected
(b) Gistified File
Figure 8: Example of model generating parameter-specific gistified file when given command that includes a parameter.
We also observe an interesting behavior of GPT-5 where it tends to insert -.name... "__main_"" even though it is not provided in the original codebase and even though it is explic- itly mentioned that we will test on the provided command and expect the same output. They often remove the test function but move the lines in the test function under the main guard (e.g., Figure We hypothesize that this may be because they are more familiar with codebases follow- ing this pattern. We also observe cases where the model attempts "cheat" the task by injecting mock, in-memory version of the original codebase package to satisfy import dependencies, rather than copying the necessary code inline (e.g., Figure 23).
Pytest Runtime Error
This егror refers to failures that occur during pytest execution, such as syntax emrrs or fixture- related issues (e.g., Figure Although the absence test functions also one pytest failures, we explicitly separate those cases by first verifying the presence of the required test functions and running pytest only when they exist.
D.3 TOOLS AVAILABLE IN GITHUB COPILOT
Table shows the list of available tools in Github Copilot.
D.4 CHANGE TEST
even high performing models and frameworks (especially GPT-5 and GPT-5-mini) seems to modify
test codes even though explicitly mentioned not to. We observed three common modification: (1)
removing the test function but move the lines in the test function under the main guard (e.g..
Figure (2). (2) adding the " "__mai imm.in_." guard even though unnecessary (e.g.. Figure [2]. and (3) mocking minimal in-memory package to bypass missing dependencies and force the test to
(e.g., Figure
21

---

Published as a conference paper at ICLR 2026
epytest. petresst.mr..pprammttoo "url, expected",
('http://192. 168.0. 0.1....1:000 1::00..1::0011 True), ... ("http://googleeccmm wtwp:o/p........0.00.o.oo.ooo. False),
)
def test_shhutdbppppeppoorteee..nne
(url, expected, monkeypatch) "**Tests for function should_bypass__rooe to check if proxy
can be bypassed or not using the 'no_proxy' argument =**
no_prexy =
22..0.0.000... htw.:/oo..........oo.......ooee. localdomain,
Test proxy' argument
assert ahould_bypase prpxees
proxies (url, no vo__proxy=no_ppc
expected
(a) Original Test Case
def test_ sist_sshull___ bypass proxies no peepy (url, expected, monkeypatch) ""*Tests for function should bypass proxies to check if proxy can be bypassed or not using the 'no_proxy' argument ***
prpxies
no_prexy = "192. 168. 0224, 1ot.a110.......aaaootooooooaaaa
172.16.1....
Test 'no 'no_proxy' argument
assert should_bypase proxies (url, no no-proxy=no_proxy
expected
(b) Gistified File
Figure 9: Example of model generating parameter-specific gistified file when given command that includes a parameter.
D.5 ADDITIONAL METRICS
Table shows the result of additional evaluation metrics, including the Average Pylest Pass Rate, which is defined as the average test pass rate over cases with at least one successful run, and the Test F1 Score, which quantifies the line-wise Fj existence between the functions the original codebase and those in the gistified fie.
GPT-5 shows notably higher Average Pytest Pass Rate, indicating that among the ones they suc- cessfully generate, they tend to pass all pytest. For the Test F1 Score, Claude-4 shows the highest performance, aliging with the trend discussed in Section
D.6 PERFORMANCE ON LARGER SAMPLE SIZE
Table shows the result of GPT-5-mini with SWE-Agent framework when running over larger number of samples (16.7x more samples from the 6 repositories). We observe that these extended results align with our initial numbers, showing that our initial sample size provides a meaningful estimate of the performance for given configuration.
E ANALYSIS
E. EFFECT OF VARIOUS STRATEGIES AND TOOLS
Prompt-Based Guidance We experiment with two variants the prompt, Reading and Tracing, where, top of the base prompt (Figure 7 add specific instructions of How to Operate to encourage reasoning using a particular strategy. The addition prompt detail of Reading in Figure 15 and for Tracing is in Figure 16
22

---

Published as a conference paper at ICLR 2026
Licensed under the GPL: https:/ /tops://ww.. wtw.gntt..rg//ieeeeeeeeeeeeeeeee https////ooootccccceeeeeggg- 8.mmm
details: https://githin cww/po//ccc/ooo////.cc/oocp www/pylint-dev/eeeeooeooooo wett-de/pyllnnt///eeee http:///ttt//..o//////oeoee/eeee
Copyright (c) httpss//g comb..co//pp nt-dew/py hnt/://tt//////b
future import annotations
import
frem collections.abe import Sequence
typing import Any
import pytest
discover discover_package (modulepath: stz, sourre_rcote: Sequence (stz]) [ste])
Liscover package pash from one its mocules and source roota.
df 05-psth, real path (os-pthh..epaannuues (modulepath)) if not pa.path, isdir (diename)
dicname os os.path. wigth.iiznmm dirname (dirname)
Look for source root that contains the module directory for seurce_root scurce_roots: source_raot
peath.realppee... los. bos..pth...xpcdueeee (source_rcos)) 68 po.pptt..ommmnng return source_root
source_root, eirname]) in [dtrname, source_root]
Fall back legacy discovery looking for _nit_.. -y upwards
itf's the only way given that source zoot was not found or was not provided
while True:
If not exatt path. Join (di .Py")
reture dirname
old_eirname dirname
dizname os os.path.diraame
dirname (dirname)
rname =s dirname:
return cs.getcwd!)
Bpytest. marke...mark.n.m waa..k.aameteriee.... Le__eed__es__eeee init " "impl") "),
(*expl t-namespace", "Impl t-namespaoe"),
best seat_discover pwwoerr_ppccaagee.oooooooo httpppttt_corrorrro___oooo
py_mod_based _a__o____s_eameeee______e str,
imp_path: Any,
T**Tost discover pacckver_packkge.o path when source rost is a parent of the module." path temporary structure;
/tmp_path/
project/
package/
init
project_dir tmp_path "projee="
package_di> a project_dls / "nypackage"
backage_dir.akdl (parents-True)
(package_dir_ (p httpadd__a_eeaaseeeeaaeeeeeeeeee
Jest with project_dir result =
(perent of package)
wwwpppckaa..ppttteeeeppe.ppeee result == atr (protecs__ir))
[ste (ptojectc__ddr
Figure 10: Example of successful gistified file: the code correctly handles all parameters, achieving 100% line ereeexxitence rate, a test Fj score of 100, and an execution rate of 65.5%.
Global Information via Tools We experiment with two tools that provide global information: RepoGraph and Tracing. Details of the information provided to the model about each tool are shown in Figure
RepoGraph (Ouyang al. 2024) is plug-in module designed to help LLMs leverage the codebase- level structure. It parses code at the line level, extracts relationships, and constructs a graph where each node represents line of code and each edge encodes dependencies between code definitions and their references. Thereby, when given a specific module, it returns the relationship with other modules represented within the constructed graph.
23

---

Published as a conference paper at ICLR 2026
import ce
import sys
from cb import contextmenager
from gattlib Impozt Path
def discover_package.ue (path, _
en*
Kinimal version: accept a filesysten path (string or Path) and return a each directory. This matches the usage in the apecifia test where path to the tests data directory is passed.
isinstanoa (path, Path):
p p peth
else:
Feth (pethl
() and p.1s_dtr(1:
return
raise SelueError("Cannn (CCannet desermine package path")
deontextmenager
augmenteed_sya_pathipoe
Kinimal context manager: sec ays the parent reccoooiess of the provided patha (in ordec), restoring the original eys.path on exit.
original list (sys.path)
tzy:
new
for in patha:
pth Path(p)
Ensure the parent directory (the "scurce root" in the test) perent pth.parent
www.oppond
sys.peth[:] sew yield
Finallys
sys.path[:] original
mark..
*py_mon base_name", (" init innt "
wwtpppppetttttmeppppee t-namespace")) ida=("expl
def test piiscooer_ppcceeeppttttt sout___oureeroott_.ae pt.entt
this test's behavicr is preserved: it aasoovr__pcckaee
the
tests data directory and enaures gggmanted_oys_patt sets ays.path to its
project/teste dereotory). rEST_DATA_DD
aos.
(fll___ "tests", "data"))
FROJRCT_ROGTT __C___OTT_RRR os w.t.path.bapppp... los path (TRST_ SET__DATA__DD DTR, +..*)) with augmented_aye_ (ci.pptaahhcccceeaee...arrr (ttaaaee___attt_S___A_AA_A assert sys.path [PROUECT_ROCT_0]]
[111):
Figure 11: Example of failed gistified file: the code fails to import pytest. The model hallucinates (), resulting in
the function test dest.discover. weover-package.e. wttprrppaccageppaassooeeee test Fi score of O and low line-existence rate of 28.0%
Tracing is tool that the tracer provided from the sys module execute command and track which components of the codebase are accessed. When the model uses the tool with specific com- mand, the tool provides the model with the files and functions touched when running the command, in the order in which they are encountered.
Execution-Based Tools We experiment with two execution-bass tools: the Bash tool and the Edit and Execute tool.
The Bash tool is basic utility that allows the model to invoke any necessary Bash commands. In contrast, the Edit and Execute tool is designed specifically for working with the gistified file: it enables the model to create or modify the gistified file and optionally execute to verify changes.
The primary difference between the two tools is their scope of execution. The Bash tool commands on both the original codebase and the gistified file, whereas the Edit and Execute tool is restricted executing only the gistified file.
24

---

Published as a conference paper at ICLR 2026
Bolick. (0--aannn--lll---11 -mall-nood is ts_flag=Irue, helg="Show HEAD and OPTIONS
methods."
swith_appcontext
def routes_commant (sort, all a
"**ghow all regi stered routes with endpoints and methods. "** from flask import eurrent app
rules = list (current app.ur) url map, iitettttt_rrette rules ())
if not rules:
click.echo ("No routes were registered.")
return
Figure 12: Example of an Import Error: the gistified file imports from the original repository (e.g., from flask import curzent.app).
t. TypeVar ("T")
class Confi ConfigAttributer (t. (e))ric(T))
"""Makes attribute forward to the config"**
def init
self, name: str, get_converter: t.Callable[[t.Aog) T] None
None
None:
self. name name
converter get_ converter
(a) Original Test Case
class Confignttribute,
def _init_(
self, name: str, get_ cet_cceverre:: t.Callable[[t.a [[t.Any), T] | None =
None
None: name_ name
self.get_converten get_cenverter
(b) Gistified File
Figure 13: Example of Pylest Runtime Error: gistified file fails with error message
TypeError: type 'ConfigAttribute" is not subscriptable
We include an example the behavior observed when adding the execution tool in Figure 18 Common patterns we observe are: (1) the model first runs the provided command to identify which files are accessed and to gather execution feedback; (2) after creating a file, iteratively executes it to verify that the generated gistified file behaves as expected; and (3) it repeatedly compares the outputs of the gistified file and the original codebase under the given command. We also observe that, due to this iterative checking process, enabling the execution tool often leads the model to terminate because it reaches the maximum step limit.
E.2 STATIC CODING LLM
Figure 19 presents the performance of static and dynamic coding LLM, including static model augmented with a single-turn, multi-step thain-of-though prompt. To test whether explicit reason- ing and planning
mitigate the limitations of static coding LLMs, which must output the entire program a single turn, we compare this CoT-augmented static model against both the baseline static model and the dynamic models.
25

---

Published as a conference paper at ICLR 2026
Tool
Description
Returns the list of Notchook cells with id, types, line ranges, language, exccution info, and output mime types. Useful for getting cell IDs. execution order, and outputs.
odit notebook
an existing Notebook file the workspace. Supports inserting, deleting, or edit- ing cells while preserving whitespace and indentation.
apply patch
Edit text files using special dift/patch format. Do not use for Jupyter notebooks.
semantic search
Run a natural language search for relevant code or documentation comments the workspace.
create directory
Create new directory structure the workspace (like mkdir -P))
create file
Create new with specified content. Automatically creates directories they not exist.
file seurch
Search for tiles in the workspace by glob pattern (e.g.. **/*. js). Returns matching paths only.
lest_search
For source file, find the corresponding file, versa.
grep search
Fast text regex search in the workspace. Useful for exact string regex queries.
run notehook cell
Run a code cell in notehook file return the output. Avoid running Markdown cells.
read notebook cell output
Retrieve the latest output for notebook cell, even if not run in the current session.
get search view results github_repo
Returns results from the search view.
Scarch GitHub repository for relevant code snippets. Use only for external repos, not local workspaces.
insert edit into
Insert or code an existing file using minimal hints, avoiding duplication unchanged code.
install extension
Install extension in VS Code. Used only during workspace creation.
list dir
List the contents a directory (folders and files).
create_ned wecpeenewwj_uuueee
hupyter__otehone
Generate a Jupyter Notehook (ipynh) VS Code.
create create_new.we weweeeewwwwoo.. workspcee
Set up complete new project (scaffolding, dependencies, config. boilerplate).
get.projcct.sctp.inf read file
Provides project sctup information for VS Code workspace after workspace creation.
Read the contents of a filc. Supports offsets and limits for large files.
open simple browser brroser
Preview open
VS Code's Simple Browser.
lest_failure think
Include test failure information in the prompt.
Think deeply about a request and log structured reasoning (no execution). Useful for planning, debugging, and brainstorming.
get_yscode_api
Retrieve comprehensive VS Code API documentation and references for extension dovelopment.
run. vscode command wetph_wehpage
a VS Code command by with arguments. Used mainly workspace creation. Fetch main content from webpage for summarization analysis.
Table 6: Available tools and their descriptions. We note that many tools available to the agent are never used.
Table Average Pytest Pass Rate and Test Fj Score of different models using SWE-Agent on the main table (Table T test dataset.
Models
Execution Fidelity
Average Pytest Pass Rate
Test Fi Score
GPT-5-mini
30.9
47.9
GPT-5
30.7
88.8
45.0
Claude-3.7
40.7
61.9
55.9
Claude-4
56.7
72.2
60.0
Table 8: Performance of GPT-5-mini with SWE-Agent framework larger number of samples (2.5k).
# Test Instances
Exccution Fidelity (w/ exec)
Line Existenee
Line Exccution
150
45.3
47.9
74.8
2.5k
46.4
47.8
75.6
Overall, we observe a trend similar to the baseline static model without a chain-of-thougle adding chain-of-thougee yields only modest improvement the static model without it, and it
26

---

Published as a conference paper at ICLR 2026
epytest
woytest.mark.parree www..maaaaatte.ia.......... "value, expected",
('foc="is rish", bar-"as well", {"foo": "is a fish", "bar": well")),
("key_ peatthout_valleee ("key_withneg witywwwthout__aleee value": Nonel),
),
def test_parse_dict_heaee (value, expected); assert srsse_ddct_heaceee (value) expected
(a) Original Test Case
assert
(' too="is fish", bar="as well"() == {"Eoo": "is fish", "bar"; "as well"}
assert harse_dict_heade None)
--
(b) Gistified File
Figure 14: The test function test-parse ct st__eeder is simplified: in the original, it used @pytest.. ma rk parammttr ze to feed multiple input/expected pairs into one function; in the gistified version, this is replaced with two direct assert statements, one per case.
Behavior Reading
How to Operate:
Examine the test file and the test function used for {problem statement]
2. Identify which module used by these functions are defined in { (working dir)
3. Copy and inline the code from those modules into 'concise.py'
4. Check these modules for any internal functions classes and inline them needed.
5. Repeat this process recursively until all internal dependencies are inlined.
6. Do not forget to copy and paste external imports.
Figure 15: Prompt for Reading strategy.
Trace Reasoning
How to Operate:
1. Predict the execution traces.
2. Follow the traces and inline (copy) only the necessary executed lines into 'concise.pp py"
3. Repeat until all traces are fully handled.
Figure 16: Prompt for Tracing strategy.
Tool Information
* search repo (repograph): searches in the mentioned repository with a specific function or class, and returns the def and ref relations for the search term. trace info (tracing): returns the trace log when running the 'pytest' command for the test currently being debugged in the given repo
Figure 17: Details of the information provided to the model about each tool.
27

---

Published as a conference paper at ICLR 2026
"role": "assistant",
"tocl_calls"
"arguments": "{3 "rcoommad\""\""" /app 56 python - pytest conci se.
oy::lestieprecationoo onat :tett
teststtts___eteeee new Serat___ew__aaaee names -v)")"
"name": "bash"
"role"; "tool", "content";
"type": "text",
"text": "OBSERVATION: test session starts ..A.sertionBrrorww
pyplint::: error: unrecognized arguments: taap/rogrtest__at//eeeee
An FAILED cod.ee..wr.ettpppoeooooooooooose ttw.__se___e_eeeeeeeeeeee ...
},
...
(fix concise.py based on observation)
"role": "assistant",
"tool_calls";
"arguments"; "[\"commadd)" "ed /app python -r pytest tests/ http:/tsst__rpppre__ooooooo test_new_namee -v -s>">",
b, "name": "bash"
rroee"; "assistant",
"cntent"; "Excellent! The messages are identical, Let me veri fy my implementation removes all unused imports and only includes what's s needed:"
Figure 18: Trajectory when including execution tool. Models often iteratively execute the given codebase generated gistified file to ensure that it operates expectedly.
Table Analysis of tool usage during the GISTIFY task
Models
Avg. tool usage
view
search
execute
other
GPT-5-mini GPT-5
10.8
71.9
9.8
1.7
16.6
18.5
72.4
8.3
3.3
16.1
Shudee-Snmet-3.7
67.5
10.1
Claude-Sonnet-4
19.3
74.6
2.1
11.8
11.5
still performs worse than the dynamic models, underscoring the importance of multi-turn generation and tool use.
Relative to the baseline static model, the CoT-augmented version slightly improves line-execution rates but slightly decreases line-existence. We hypothesize that this is because the CoT procedure encourages the model to focus on minimal set of necessary lines.
Details of the prompt used in this experiment are provided in Figure 20
28

---

Published as a conference paper at ICLR 2026
static coding LLM
SWE-Agent
static coding LLM w/ COT mini-SWEEAgent
Copilot
80
60
Scores
40
20
0
Execution Fidelity
Line Existence
Line Execution
Figure 19: Performance over Static and Dynamic Coding L.M
Static Coding LLM with CoT
THINK STEP BY STEP, and SHOW YOUR REASONING. You must follow multi-step solution process: (1) determine what code executes during test, (2) determine the minimal set of lines required, (3) plan the final single-file layout, and finally (4) output the new single-file code.
Figure 20: Prompt added to the static coding LLM to enable chain-of-thought reasoning. A similar instruction is also included in the system prompt.
class TestGetNetrcAuth:
def test_works (self, tmp_path, monkeypatch): netre_path tmp_path ".netre" monkeypatcc. setenv ("NETRC", str (netro_path)) with open (netro_path, "w") as f:
f.write ("machin example. com login aaaa password bbbb)n"))
auth = get_netre netre aat___etrr_uatattttgap assert auth == ("aaaa", "bbbb")
(a) Original Test Case
name main
Reproduce tests/test_utils.ppe hhw.:::::teeeteeeeeechhhhhhtt.e with t remporaryOlrectory/ as tmpdir:
netrc_path os joi ".netre") os.environ "NETRC"" netrc_path
with open (netro_path "w") as f:
T.write example.com login aaaa password bbbb)n") auth get_netrc_auth auth ("aaaa", "bbbb")
Gistified File
Figure 21: Test Modification Case 1: The test Testteet.NetrrAtth.eees works is converted from a pytest unit into standalone script.
29

---

Published as a conference paper at ICLR 2026
Test class and method preserved unchanged class TestArgaassoptionss roviddidiixiii ***Testa for the argparse impl implemetttionn of OptionaProvi derMividdeiiii xIn.
The logger checker is used as example checker for this implementation.
@staticmethed
def test www.oooreegwhhutteeeeeeoo
() -> None:
"""Check that we raise messages we not supply any options
with raises raises (SystemExit) as ex:
Run (en[[[GGINGGTT)) ] assert value. 2
Main execution for pytest if
name___ -- ___ainn:
test -
test. hett.tes___ggger_ttthttttteeeee
Figure 22: Test Modification Case 2: Adding unnecessary "..main.." guard
requests_mod - in-memory 'rquesss'' package with required submodu es. (' requests')
Create a minimal
requests_modd Baatodd..pahhhe path []
compat med types. ModuleType requests compat')
structures mod types.Moouless.......oo ModuleTTpe requests
Populate cempat with only what's needed by this test suite import paths
compat__mod.apppae Mapping
compat_mood MutableMapping MutableMapping compat_mod.ure jrijoin urljoin
Populate structures with the classes.
structures_moc strueturee.noo..aeetteasastt Caseensensstiiiieii CaseInseessttiveDict struutuurs htt.tuure__mod..ooupppiile....
LockupDict
Wire the package hierarchy and register in sys moduled..
wreuueetssmoddcooo compat compat_mod structures_nt _nd eys. I' requessts = requests_mat mod
sys moculess requests. compat'] cempat_mod
sys. moculcs requests. structures') structures_mod
if name r_m__ain main
import pytest
raise SystemExit ( (I "testss/eettsstruuuurs..pp Battassttsssstttieeaeaetetere list' }))
Figure 23: Test Modification Case 3: Manually mocking minimal in-memory package to bypass missing dependencics and force the test to run.
E.3 TOOL USAGE RATES
Table shows the statistics on tool usage across models using SWE-bench. We group various tools into four categories: view, search, execute, and other, which includes all remaining tools. For all
30

---

Published as a conference paper at ICLR 2026
models, compute usage rates both with and without execution enabled, and then average the two settings.
Among all models, Claude-4 exhibits the highest average tool usage for each test cases, followed by GPT-5, Claude-3.7, and GPT-5-mini. terms of specific functionality, Claude-4 shows the highest rate of both view and execute tool usage, while Claude-3.7 shows the highest usage of the search tool. To generate a high-quality gistified file, a model must effectively view relevant files and copy only the necessary content. The strong performance of Claude-4 line existence may be related to its high usage of the view tool. Also, the execution tool tends to support correctness verification of the generated file, which would lead to high execution fidelity.
31


Reviews:
[{"review_id": "CMTANZchRT", "reviewer": "Reviewer_SdHh", "summary": "GISTIFY tests whether LLMs truly understand whole codebases—not just snippets—by requiring them to generate a single minimal Python file that exactly replicates a command’s runtime behavior (e.g., pytest output). Success demands four criteria: files must be self-contained (inline dependencies), execution-faithful (identical output), minimal (only essential code), and grounded (zero hallucinations, strictly sourced). Evaluating top models (GPT-5, Claude-4) via agent frameworks like SWE-Agent, the authors introduce three practical metrics: binary execution fidelity, Line Execution Rate (minimality), and Line Existence Rate (source fidelity). Results reveal even leading models struggle significantly with complex tasks, especially long dependency chains. Error analysis shows failures stem from incorrect inlining or over-pruning critical code. This benchmark exposes core weaknesses in current LLMs' cross-file reasoning.", "questions": "1. How does the normalization and block-matching algorithm handle semantically equivalent but syntactically different code changes, such as rewriting a for-loop as a list comprehension or rearranging independent class methods? Could you include the relevant pseudocode or a more detailed description of the matching logic?\n2. For handling non-deterministic program outputs (e.g., timestamps, memory addresses, or unordered dictionary iteration in legacy Python versions), what normalization methodology is applied to stdout/stderr prior to comparison?\n3. In this experiment, the static model receives a strong oracle. The observed conclusion indicates dynamic file selection achieves superior performance. It is hypothesized that the static model's suboptimal results may stem from context overload. Could implementing alternative prompting strategies—such as single-turn multi-step Chain-of-Thought (CoT) that directs initial identification of essential code segments before file synthesis—effectively mitigate this performance gap?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper addresses a critical gap in current LLM code evaluation, which leans heavily on static analysis. Requiring full runtime behavior replication forces models to grasp entire codebase execution flows—crucial as AI coding assistants scale toward industrial systems handling multi-file complexity.\n2. With just a repository and entrypoint command, GISTIFY generates evaluation cases automatically for any codebase without relying on manually labeled data such as GitHub issues. This sidesteps annotation biases and enables direct cross-project comparisons.", "weaknesses": "1. The main results in Table 1 are based on 25 tests from each of 5 repositories. While the analysis section uses a larger set of 50 tests from pylint, the primary claims about model performance would be more robust if supported by a larger and more diverse set of test instances in the main experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yhnt7ebjbV", "reviewer": "Reviewer_TALW", "summary": "This paper introduces GISTIFY, a benchmark for evaluating codebase-level understanding in large language models. Given a repository and a command, the model is required to generate a minimal, self-contained file that reproduces the same runtime output as the original codebase. The benchmark is automatically constructed from real GitHub projects and paired with execution traces that capture ground-truth runtime behavior.Evaluation metrics include execution fidelity, line existence rate, and line execution rate. Experiments with multiple LLMs and agent frameworks show that while models can handle simple single-module tasks, they struggle on multi-file or long-trace scenarios.", "questions": "See weakness please", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This work introduces a new approach to evaluate models' code repair ability from a more comprehensive and systematic perspective. The benchmark is built from real execution traces, collected via an automated pipeline, and the evaluation criteria are well thought out.", "weaknesses": "1. Since code execution inevitably produces different errors due to various factors, it is unlikely that data filtering alone can fully prevent them. How should such errors be handled during evaluating? Could there be missing special filtering or handling mechanisms?\n2. The benchmark is quite novel, but the overall size might be somewhat limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "96GWqtYL3F", "reviewer": "Reviewer_C2sw", "summary": "This paper introduces **GISTIFY**, a new repository-level evaluation benchmark where LLM agents must compress a command's behavior (e.g., a specific pytest) into a single, self-contained file. The authors define clear task requirements, propose metrics for execution fidelity and minimality, and benchmark four leading LLMs across three agent frameworks on five SWE-Bench repositories plus debug-gym. The evaluation includes error analysis, ablations on prompting strategies and tools, and a difficulty analysis based on execution coverage.", "questions": "1. Do you have plans to expand the benchmark beyond 25 tests per repo? What's the timeline, and will the expanded sets be publicly available?\n2. Have you considered validating the minimality metrics against human judgments? Even a small pilot study would strengthen confidence in your metrics.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Unlike prior repository-level benchmarks (SWE-Bench, RepoBench), GISTIFY uniquely requires runtime-guided single-file reconstruction. This frames an interesting and practical challenge.\n- The paper tests four models across three frameworks, includes thoughtful ablations on prompting strategies and tool usage, and extracts concrete insights about agent behavior and design trade-offs.\n- The task is well-explained, metrics are intuitive, and the error categorization (import errors, missing functions, etc.) provides useful diagnostic insights.\n- The finding that state-of-the-art agents still struggle on long execution traces is noteworthy and motivates future work on execution-aware reasoning.", "weaknesses": "- The main experiments cover only 25 tests per repository across 5 Python projects. It's unclear how well this generalizes to larger codebases, other languages, or more diverse domains. The repository selection criteria and expansion plans are not well explained.\n- The line execution/existence rates are intuitive but not validated. Without human reference gists or user studies, it's hard to know if these metrics truly measure \"minimal and faithful\" distillations or just correlate with them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CTV7H7fcqs", "reviewer": "Reviewer_WQmg", "summary": "This paper introduces GISTIFY, a novel task designed to evaluate code generation models on codebase-level understanding through runtime execution. Given a codebase and an entry-point command, the goal is to generate a single, self-contained, and minimal gistified file that reproduces the original runtime behavior. The authors formalize three evaluation metrics, Execution Fidelity, Line Execution Rate, and Line Existence Rate, to measure model performance. Experiments are conducted across multiple frameworks and state-of-the-art models on popular Python repositories. Results show that even advanced models struggle with complex execution traces, while agentic frameworks and access to execution-aware tools yield consistent improvements. The paper argues that GISTIFY offers a lightweight, reproducible benchmark that reflects real-world developer workflows and provides valuable distilled code artifacts for downstream applications.", "questions": "- Could partial credit be integrated (e.g., per-test-case success rate) to provide smoother performance gradients?\n- According to the definitions of Line Execution Rate, this metric is 100% when every line of the gistified file is executed. However, this may misalign with practical minimality when syntactically necessary code (e.g., unused branches in conditional statements) is retained for correctness but not executed under test inputs. For example, an if-else block where only one branch runs would penalize the metric score despite the code being functionally minimal.\n- Could the line existence rate penalize valid simplifications?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper formalizes a runtime-based approach to codebase understanding, which is underexplored in prior benchmarks.\n- The three metrics (Execution Fidelity Rate, Line Execution Rate, Line Existence Rate) together provide a comprehensive evaluation of the GISTIFY task, offering a well-rounded assessment of the model’s effectiveness.\n- Experiments cover multiple models and frameworks, providing comparative insights.", "weaknesses": "- The Execution Fidelity can only be signed to 1 or 0, which lacks granularity in capturing partially correct implementations.\n- While error types are categorized (Import Error, Missing Test, etc.), deeper causal insights (why import errors dominate, or which repository structures cause more failures) would make the analysis stronger.\n- Insufficient documentation of evaluation details. Critical hyperparameters, such as temperature settings and the number of repeated runs per task, are not specified.\n- The evaluation uses only 25 test instances per codebase, which is insufficient for drawing robust conclusions.\n- The benchmark exclusively evaluates Python codebases, ignoring other programming languages (e.g., Java). This narrow focus limits insights into cross-language performance and fails to address challenges unique to diverse paradigms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)