CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning
Authors: Monoshi Kumar Roy
Simin Chen
Benjamin Steenhoek
Jinjun Peng
Gail Kaiser
Baishakhi Ray
Wei Le
Venue: ICLR 2026 Poster (2026)
Keywords: Code Semantics
Benchmark
LLM
Primary area: datasets and benchmarks

Abstract: Understanding and reasoning about code semantics is essential for enhancing code LLMs' abilities to solve real-world software engineering (SE) tasks. Although several code reasoning benchmarks exist, most rely on synthetic datasets or educational coding problems and focus on coarse-grained reasoning tasks such as input/output prediction, limiting their effectiveness in evaluating LLMs in practical SE contexts. To bridge this gap, we propose CodeSense, the first benchmark that makes available a spectrum of fine-grained code reasoning tasks concerned with the software engineering of real-world code. We collected Python, C and Java software projects from real-world repositories. We executed tests from these repositories, collected their execution traces, and constructed a ground truth dataset for fine-grained semantic reasoning tasks.  We then performed comprehensive evaluations on state-of-the-art LLMs. Our results show a clear performance gap for the models to handle fine-grained reasoning tasks. Although prompting techniques such as chain-of-thought and in-context learning helped, the lack of code semantics in LLMs fundamentally limit models' capabilities of code reasoning. Besides dataset, benchmark and evaluation, our work produced an execution tracing framework and tool set that make it easy to collect ground truth for fine-grained SE reasoning tasks, offering a strong basis for future benchmark construction and model post training. Our code and data are located at \url{https://codesense-bench.github.io/}.

Published as a conference paper at ICLR 2026
CODESENSE: A REAL- WORLD BENCHMARK AND DATASET FOR CODE SEMANTIC REASONING
Monoshi Kumar Roy', Simin Chen?, Benjamin Steenhoek Jinjun Peng", Gail Kaiser?, Baishakhi Ray?, Wei L.e
'Iowa State University, (monoshi, weile (iaeeelasttteeeee
PColumiia University, (sc5687, jinjun, kaiser, rayb) 8cs. col umbia edu Miicosoot Data & A1, bewsteerhooeeemmcrossrr..
ABSTRACT
Understanding and reasoning about code semantics is essential for enhancing code LLMs' abilities to solve real-world software engineering (SE) tasks. Although sev- eral code reasoning benchmarks exist, most rely on synthetic datasets or educational coding problems and focus on coarse-graiie graiindd reasoning tasks such as input/output prediction, limiting their effectiveness in evaluating LLMS in practical SE contexts. To bridge this gap, we propose CodeSense, the first benchmark that makes available a spectrum of fine-grained code reasoning tasks concerned with the software engi- neering of real-world code. We collected Python, C and Java software projects from real-world repositories. We executed tests from these repositories, collected their execution traces, and constructed ground truth dataset for fine-grained semantic reasoning tasks. We then performed comprehensive evaluations on state-of-the-an LLMs. Our results show clear performance gap for the models handle fine- grained reasoming tasks. Although prompting techniques such as chain-of- and in-context learning helped, the lack of code semantics in LLMs fundamentally limits models' capabilities of code reasoning. Besides dataset, benchmark and evaluation, our work produced an execution tracing framework and tool set that make it easy to collect ground truth for fine-grained SE reasoning tasks, offering a strong basis for future benchmark construction and model post training. Our code and data are located at https:::s. htt.p:::/codeeess--bbeee ccodesense--ennce gi thub. 1o/
INTRODUCTION
Semantic code reasoning the capacity to understand and predict the behaviour of software is core requirement underpinning wide range of complex software engineering (SE) tasks, including test input generation, vulnerability detection, fault localization, bug repair, refactoring, and functional verification. Unlike syntactic pattern matching, which may rely token-level similarity statistical regularities, semantic reasoning ("codesense") entails a deep. execution-oricn oriientd understanding of how software operates. Although code semanties developers engage in semantic reasoning through tasks like predicting a function's 's input-output behavior, tracing variable values, analyzing control flow paths, identifying loop invariants, etc. This form of reasoning aligns with formal definitions from programming language Micory-patticularyy operational semantics, which models step-by-step execution, and axiomatic semantics, which uses logical assertions to describe program properties. Such reasoning tasks also reflect the real-world demands placed on developers and provide natural grounding for their day-to-day work.
be expressed in many ways, in practice,
Recent years have witnessed the emergence of numerous benchmarks for evaluating coding-related related tasks. However, the majority of these efforts have focused on code generation using synthetic or narrowly scoped data-for for example, HumanEval+ (Liu et al. 2023), LiveCode 2024), Bigcodebench (Zhuo et al. 2024), and CodeBenchGen (Xie et al.. from isolated competitive programming problems. Consequently, they fail to capture the complexity and structure of real-world software development. Other benchmarks that incorporate real-world code, such as WWE-Bench (Jimenez et al. 2024). WWE-PolyBench (Rashid et al., (025). and KGym (Mathai cet al. 2024), tend to evaluate only task specifce performance (e.g., patch generation for GitHub issues), making it difficult to assess whether models exhibit generalizable semantic understanding. Finally,
(Jain et al. extracted

---

Published as a conference paper at ICLR 2026
vold Foo inpat) ( int B input 23;
void bar (ine abito) (
FFTeoatext +5 mal.ccicizcot c)1s nbits 0913
22201113-222111
//dangezous codo need to sested
1400 2N
0150 POCUER 5J
(A) Test input generation and program execution rea- soning: to generate a test input that can lead to the ex- ecution of dangerous code at line 4, the model needs to find input that can satisty the branch condition 34652n222876m
(B) Vulnerability detection, fault localization and pro- gram repair: this code has a memory leak vulnerahility when the i condition line 3 is fal se. To detect this vulnerability, the model should know that calls to mailec and free are related to the vulnerability (se- mantics of the API calls), and should be paired along the program paths along both branches starting at line 3 (semantics of the branch statements and control flow). Similarly, ma lalooc and free can be located in various code contexts in different programs,
line 3. Understanding the semantics of operators **" at line 2 and 'z' at line 3 is needed to effectively generate an input that reaches the dangerous code, c.g., imput 120. The branches and arithmetic operations can be quite diverse dif- ferent programs, it is hard to generalize patterns regarding which code text should use what kind of test input to execute.
thus hard to generalize the patterns only from the code text, without semantic code reasoning.
FIGURE 1. Fine-grained code semantics are the keys for solving many SE tasks
reasoning-focused benchmarks, such as CruxEval (Gu et al. 2024), primarily target function-level input/output prediction over short and synthetic code fragments involving random string operations. Such settings neglect the fine-graines semantic reasoning about internal program behavior and properties, data dependencies, and control structures required to solve a variety of SE tasks for complex real-world software systems.
To this end, we propose CodeSense, a benchmark for fine tne-grained code semantic reasoning, con- structed from real-world GitHub projects in Python, C, and Java (see Table CodeSense introduces a spectrum of reasoning tasks at statement, code-block, and function levels, targeting essential semantic properties frequently needed across SE activities. For example, predicting loop iteration counts is critical for input/output prediction, performance analysis, and detecting infinite loops (e.g., denial-of-servee vulnerabilities). Branch condition prediction and reasoning about pointers in C code are important for test input generation and memory safety assurance. As illustrated in Figures] Ta and Tb fine-grained semantic reasoning about arithmetic operations, control flow, and API semantics is foundational for variety of SE applications. Prior work (Ding et al. 2023a 2024) has shown that incorporating semantic signals during training improves model performance on code generation, branch prediction, code clone detection, program repair and vulnerability detection tasks, motivating our design of CodeSenseto comprehensively evaluate models' capabilities for semantic reasoning.
TABLE 1. Optimal design space of code reasoning benchmarks (O denotes not support, denotes partial support, and denotes fully support)
Benchmark
Real-Wurid Wurld Projects Multi-linuual F Function I/O Fine- Grained Reasoning Exec Stegs API Understanding Multi-File Context
CrusEval Kuu
2024
0
CrusEval-X Xu
2025
4 0
CodeMind
0
O
CoRe Xie
0
O
0
CodeSSase
To construct CodeSense, we collected 544 Python, 100 C, and 100 Java projects. To enable supervised evaluation, we develop a framework that automatically extracts ground-truth annotations for these tasks using static and dynamic program analysis. We build, execute and log run-time execution values and traces of the woald projects. Our dataset includes total of 2125 Python, 876 C, and 875 Java unique functions, based on which, we curated 4483 samples with their ground truth in our benchmark. Please more details in Section 2.
2

---

Published as a conference paper at ICLR 2026
Using our benchmark, we evaluated 14 state-of-the-art (SOTA) LLMs and investigated six research questions regarding the models' code semantics reasoning capabilities. Previous work has shown that models did not perform well on code reasoning tasks such as input/output prediction (Gu et al. 2024) and vulnerability detection (Steenhoek et al. 2025); to understand why models fail and identity places for improvement, we investigate: RQT: Does increasing code size make semantic reasoning more difficult? RQ2: Which types of program statements are easier or harder for models to reason about? RQ3: How do models perform on code properties critical for SE tasks, such as predicting pointer aliasing. loop iteration counts, and branch conditions? RQ4: How effective are different prompting strategies in improving semantic reasoning? RQ5: Can models reason approximately when exact values or semantics are hard to infer? RQ6: Do models perform better on some programming languages than others?
Our results reveal that current LLMss including SOTA models like Claude 3.5 (Anthropic 2024). GPT- 4o-mini (OpenAT 2024), and Gemini 1.5 (Gemini Team et al.) 2024) struggle with fine-grained code semantics. They often fail to reason about even single statements from real-world code port-ppaticularlly arithmetic expressions and API calls-and perform poorly on tasks involving loop values and iteration counts. Basic chain-of-thougee prompting offers limited benefit, and few-shot prompting yields only modest improvements. In-context learning is most effective when prompts define new concepts or include highly relevant examples. Interestingly, models can correlate natural language code semantics questions with certain code patterns. For instance, when the code contains assignments like = models correctly respond to the prompt "do p and q at <line> alias the same memory address" even in zero-shot settings, Similarly, models reliably infer loop bounds in explicit cases such as for i range (100) :. Among the 14 models evaluated, Claude 3.5 consistently achieved the best performance. We also observe that Java and Python code are generally easier for models to reason about than C, and that input prediction (i.e., reverse semantic inference) remains among the most challenging tasks.
Contributions. This work introduces CodeSense, a realistic and comprehensive benchmark for evaluating LLMs' fine-grained code semantics reasoning in practical software engineering contexts. We advance the state-of-the-an code reasoning benchmarks by:
1. Defining diverse set of fine-graing semantic reasoning tasks grounded in real-world software engineering needs,
2. Developing a scalable open-source framework and toolchain to automatically generate execution traces and semantic annotations, enabling continuous benchmark expansion while mitigating data leakage,
3. Constructing benchmark dataset using real-world projects in Python, C, and Java,
4. Empirically analyzing six research questions across 14 state-of-the-arr L.LMs to assess their strengths and limitations in semantic reasoning, and
5. Launching a public leaderboard to support reproducibility and accelerate progress on semantic reasoning for code: https://codesette senteeseeseeeeenchhhee github. io/ leaderboard. html
BENCHMARK CONSTRUCTION
2.1 DEFINING SPECTRUM OF CODE REASONING TASKS
To design tasks for evaluating LLMs' capabilities of code semantic reasoning, we first considered the definition of code semantics. In programming languages and software engineering, code semantics "what is the meaning of this code" - are defined as what is the output value given the input of a code snippet. Such fine-grained reasoning tasks are directly related to end-tasks in software engineering. For example, previous work (Ding et al. (023a) shows that when fine-tuned with statement-level values, the performance of the models improved for vulnerability detection, branch prediction and code clone detection. Prior study (Steenhoek et al.) 2025) reported that although recent LLMS improved math reasoning and natural language reasoning significantly, they are still insufficient for handling end-tasks related to code reasoning. To help locate the weakness of models' code reasoning at a granularitu and help models to improve a variety of SE applications that are linked the fine-grained reasoning steps, we designed the following code reasoning tasks:
3

---

Published as a conference paper at ICLR 2026
Data Curation
Step 2. Build & Execute
Pinc Luce estdio.h
Step Data Collection
int main(void)
Step 3. Traces
BUILD
EXECUTE
Curste valuea, data typea names of lundton colls, memory acdresses (G-only) elatement using debuggere:
Lensiling...
int i, sun Lineiing
OSS-Fumt
9F-100 for Jara
for [1=11x 10;
9uslosucoeded
Pyönooper Pytnon
for Python for
744 real-world projects
("Sum tolue,, Sunt;
return 8;
Benchmark Creation
Step Extract Reasoning Ground-truth
Step 1. Filtering
Step 3. LLM Evaluation
Code operational semantes (value prodiction)
inputoutput prodistion differett granularties:
I lama famil
GPT-4omii
Collect fundioon-eevel
nck
DespSeek Coder
funcsore
family
Oeninii-1
Program properties
Owen family
Seek-R
Pointer allaiing predicttio
Grantlo family
family
3876 real-world functions
Branch prediciiun
Phi
properies prediction
FIGURE 2. CodeSense: Data curation and benchmark creation overview.
Task 1: Block-level code semantics (RQ1): To investigate whether a model understand a chunk of code, we give block of statements. We give input and ask the models to predict the execution output; also we give output, we ask the models to predict the input. Input/output prediction of function is a special case of probing block-level code semantics. In our evaluation, we sampled block of statements from the entry of the functions and increased the sizes of blocks, including the entire function.
Task Statement-level code semantics (RQ2): We classified program statements based on programming language semantics and evaluated models five common statement types, including arithmetic. boolean expression, API/Function call, variable assignment and constant assignment. Knowing code semantics at the statement level means that given input of the statement, the models are able to produce a correct output. In our evaluation, we studied output predictions in more depth. We randomly sampled statement from the program and asked the models to predict its output given input.
Task 3: Code properties within function (RQ3): Code property (property regarding particular code construct) is another aspect of code semantics. We focused on three important properties in this benchmark. Loops are related to code optimization and detecting bugs. Reasoning about pointers in C code are very important for assuring memory safety and detecting and repairing vulnerabilities. Knowing how to predict branch outcomes can help generate test inputs and parallelize code.
Task 3-1:
Loop property. Given an input of function, we asked models to predict the number of loop iterations, the values in the loop and the values after executing the loop. In our evaluation, we randomly sampled loop in a function and randomly sampled variables in and after the loop.
Task 3-2: Pointer property. Here, we give models a function and its input, and we ask models to predict whether the two pointers are aliased (pointing the same memory location) at given program point.
Task 3-3: Branch property. Given function and its input well as the location of conditional branch in the function, we ask the model to predict what is the outcome of the branch. our evaluation, randomly selected conditional branch in the function for prediction.
Task 4: Approximation code semantics (RQS): Reasoning about concrete values for the above tasks is very challenging. Sometimes, to solve an SE task, may only need approximate value of code semantics. For example, in Fig. [a] the models do not have to generate a concrete number like i iput=120: is sufficient for models to tell that integer input between 100-150 can trigger
4

---

Published as a conference paper at ICLR 2026
the dangerous code. We designed set of abstract values for different data types, following prior literature (Ding et al. 2023a) and evaluate the models can predict abstract values correctly.
The above tasks also used for studying RQ4 regarding prompting techniques and RQ6 com- paring different programming languages. We have included the prompts for the above tasks in Appendix/data package.
2.2 COLLECTING AND TRACING REAL-WORLD MULTI-LINGUAL SOFTWARE PROJECTS
Constructing ground truth for the set of code-semantics tasks is great challenge. We collected a total of 744 real-world software projects of Python, and Java from GitHub. We developed a framework and tool chain build the projects, run tests and collect the execution traces which contain values, data types, names of function calls and memory addresses (for C code) at each statement. We developed analysis tools to extract ground truth for the benchmark tasks from those fine-grained code semantics data. Please check our Appendix [A.4 for our language selection rationale and how our framework can be easily extended to other languages.
Python. We collected 1489 GitHub repositories from the PyPIbugs dataset (Allamanis et al. 2021). We removed projects that don't contain test cases or have not been updated the last four years, and obtained 544 projects. We first installed dependencies for each project and used pytest (Krekel cct al. 2004) to run tests, and Pysnooper (Rachum et al. 2019) for tracing.
C. We used 100 projects curated in OSS-Fuzz (Arya et al. 2023). We built and fuzzed the real-world projects using the OSS-Fuzz infrastructure the docker environment with project-wise fuzzing harnesses. We developed a tracing framework built on the GNU debugger (GDB) (Free Software Foundation).
Java. We collected 100 projects from the SF110 dataset (Fraser & Arcuri 2012). We used Evo- Suite (Fraser & Arcuri 2011) to generate and run test cases and developed our tracing tool on top of Java Debugger (Oracle Corporation 2025) to record the execution details of the projects.
2.3 DATA FILTERING
We collected whole program traces, from which curated unique functions based their entry and exit points in the execution trace logs. We excluded functions that only contain comments, too lengthy to fit into the models' context, and the functions which don't have meaningful functionality in their body. For example, some functions only contain one statement like "return 0", "printf("".. or some functions are just a wrapper for another function which we have tested, such as "void myfunc(f funcO:]". We obtained total of 2125 Python, 876 C and 875 Java unique functions, with the sizes ranging from to 516 lines of code. From these unique functions, we curated our task-specific datasets.
In real-world code, we face many complexities, e.g., the input of function and the values of variable can be complex types. As the first step of probing models to reason about fine-graines code semantics for real-world code, we focus on ground truth values of primitive data types in all tasks, including int, float, str, bool, list, pointer, double, dict ciiiary, tuple, etc. In the evaluation, we show that even for values primitive types, the models find it challenging to predict them. We collected total of 4483 samples from Python, C and Java, and constructed the ground truth for the above tasks and used for evaluation. See Table
TABLE 2. Number of Samples for Tasks
Task
Python
Java
Total Sumples
Task 1: Block
1860
731
2591
Task Function
308
94
476
Task 2/Tosk Statement
545
485
1030
Task 3- 1/T/TTsk 4: Loop
105
105
Task Pointer
49
49
Task 3:3: Branch
232
232
Total Samples
3050
1359
74
4483
5

---

Published as a conference paper at ICLR 2026
3 EVALUATION
for We evaluate 14 SOTA LLMs, 8 reasoning models and non-reasoning models (Table for full names and short IDs used in figures), including open-source models (Llama, phi), close-source/AP1 models (GPT-4.0 Mini, Claude 3.5 and Gemini 1.5) and distilled models (DeepSeek R1 series), with the model parameter sizes ranging from 7 to 14 billions. We utilized vLLM(v0.3.3) as our inference engine to run the models.
TABLE 3. Model Names and Their IDs in the figures
Full Model Name
Model ID the figures
openalgptt4o-mmni
GPT-4o (Reasoning)
CL-3.5 (Reasoning)
gemini-..5-ttahh0002
Gem-1.5 (Reasoning)
L-3.1
Diven/Qwen22S-14--Insttte
Q-2.5
Qwen2.5-C
DS-C
miccrowotf/Phh---miiin-mmnnuuuu
Phi-4
microwwff/Phii33..-miiiininntt
Phi-3.5
ibm
Gr-.22 (Reasoning)
I--Isst 1-Di--till-OOO --Disiill-wwen-77
DSRI-Q2 1-1--Q (Reasoning)
dlepseeck-ai/DeeppSeee -ii/DaepSeeeeee R DDDsstll---lamae
DSR1-L. -I. (Reasoning)
satereek--iiDDeeSeeeeeeee ---sstl Seaill-Qwen-lll 14B
DSRI-QU ---14B (Reasoning)
ihm- hatpiii/grrannn----------nteeeee
Granite-3.2 Pr (Reasoning)
We designed five different natural language prompt templates (see Appendix/data package), and ran them on sampled dataset for each model. We observed that prompt templates are model-sensitive, but not task-sensitive. So select template for each model for all the tasks. We prompted the models to give a response inside specific tags (<ans> </ans>) and considered the response inside that tag compare with the ground truth, done in (Gu et al. 2024). For our evaluation metrics, used accuracy (exact matching of the generated outputs of the models and the ground truth label).
In the following, we presented selection of interesting results. For clarity, we present results from one representative LLM per model family to ensure model diversity. Please refer to the Appendix for the complete set of experimental results.
3.1 RESULTS FOR RQI: BLOCK-LEVEL CODE SEMANTICS
Fig. 3 shows input/output prediction results for code blocks and functions (a special case of code block) across varying sizes. In Fig. 3a shows the results for three block sizes blocks containing one, two, and three statements, respectively.
Overall, we observe that model accuracy is low even for small code blocks. For example, in C dataset, models such as Claude 3.5 and GPT-4o-mini achieve under 30% accuracy on single-statement blocks. Python yields slightly better results, though
model exceeds 50% accuracy. Performance further declines as block size increases from 1 to 3 statements, with open-soures models performing significantly worse. This degradation stems from two primary challenges: models often fail to reason about individual statements, and they struggle to track variable state across statements. Notably, even Claude 3.5 achieves only 20% accuracy on 3-statement Python blocks, and less than 10% on However, in some cases, smaller blocks can be harder because they contain API calls.
similar trend is observed in Fig. 3b (right: Output Prediction), where models perform better on smaller functions than larger ones for output prediction. However, performance input prediction remains consistently poor (left figure) across all function sizes. This highlights a broader limitation: LLMs are even less capable of understanding the "reverse" of operational semanties, i.e., inferring inputs from outputs. Even the best performern model, Claude 3.5, achieves only around 12% accuracy in input prediction for small functions.

---

Published as a conference paper at ICLR 2026
Block Size
Black
Bize's
0.8
Accuracy
0.6-
Accurazy
L4.1
Q266
05-C
Phild
Gra DSR-LGem-- 16 CL-66 GPT-40
Python Program
Program
(A) Output prediction for different block sizes
1.0
6mi
Medium
Large
1.0
Smail
Accurary
04-
0.4
02-
0.-
0266
beRt-
Gasc
Input Prediction
Output Prediction
(B) In/Out prediction for different function sizes
FIGURE RQI: Does increasing the size of code increase the difficulty of code reasoning?
1.0
L-8.1
P*H DERI-L
Gem-1 0PT4:
L-0.1
0-2.6-
Phi- Srd.2
Gem-1.8
CL-.5
0.8
Accuracy
Accuracy
0.6-
0.2
010
Arthmmeic
API
Bookean
C Program
Python Program
FIGURE RQ2:
What types of program statements can the model understand well?
3.2 RESULTS FOR RQ2: STATEMENT-LEVES CODE SEMANTICS
As shown in RQI, models struggle with value prediction even for single statements (e.g., block size 1). In RQ2, we further analyze model performance by categorizing results based on statement types. Fig. 4 presents these results, with the left plot showing C and the right showing Python. Each plot groups model performance by statement type to highlight specific areas of strength and weakness.
We observe that arithmetic and API/calls are the most diffficult statement types, reasoning models like CPT440-mint and Claude 3. For APIs, we sampled frequently used third-party libraries, like os, sys, and math installed by pip, but the models do not have knowledge about their execution semantics. We also experimented with adding the API definitions in the prompt, but it didn't tt increase the performance significantly A.IT Models handle better for predicting Boolean values such as the output of a comparison statement, and also statements where constant assigned to variable. The models may understand the assignment operator "=" and have captured easy patterns like 3a=3 indicates has value after executing this statement". We did not observe significant advantage of reasoning models over non-reasoning models for this task.
for the best
3.3 RESULTS FOR RQ3: CODE PROPERTIES WITHIN FUNCTION
report results predicting the number of loop iterations, values in and after the loop, given the input of the function. We observe that models find it difficult to predict values after executing the loop. Loop iterations the easiest tasks among the three. Our intuition is that
In Fig. 5a
7

---

Published as a conference paper at ICLR 2026
sometimes certain patterns in the code text linked to the loop iterations. For example, Python code for i in zange (1 (100): :. implies that loop iteration is 100. Somehow, some models know these patterns and constants are linked to the loop iterations. We inspected the predicted loop values and did not find a trend that the models just use any constant numbers in the code text as their answers.
In Fig. and Fig. 5c show that given input, predicting pointer aliasing program location and whether branch can be taken is easier than loop properties. Here, the models only need to predict "yes"/"no". The models predict pointer aliasing better than branch execution. We believe that the models are able to connect code patterns such as "p=q" to the aliasing definition provided in our prompt "when two pointers store the same memory addresses, they are aliasing". Notably, some open- opencsource models perform below 50% on these binary classification tasks worse than random guessing.
InLeop
tenton
Porrel Mbains
131 0.84 05-
6e52 166 Pea dut demn d0 doLanbare
Gen Saane sthoern
Ressoiing Commetia
Open Soarcs Nos Rossceing
Down Source Sessoiing Commetcal
(A) Loop Properties
(B) Pointer Aliasing
(C) Branch Execution
FIGURE 5. RQ3: Can models reason about different program properties?
3.4 RESULTS FOR RQ4: DIFFERENT PROMPTING TECHNIIQES
In Fig. 6, we show results of different prompting techniques on statement prediction and loop property predictions (relatively difficult tasks in our list). Our results show that in both cases, models benefited from more shots in the prompt. When we prompt models and provide examples more relevant to the query (RAG style): that is, for statement prediction, we provide shots with the same type of statement, and for loops, we provide shots of different loops in the same function, models improved their performance. However, applying a simple COT by "asking models think step by step" the beginning of the prompt did not help much for statement prediction, but helped for loop property prediction for some models. Our intuition that compared to statement prediction, loop reasoning, e.g., predicting values after loop, may be more complex and can benefit from multi-step reasoning.
1.0
1-5ha
D-Shol
1.0
3-Shot
2-5ha
Accuracy
0.
3-Sha
Accuracy
0.4
L-811
G260
D&C
PH-
dra DeAt
Loom-1.5 cL-as GPT-40
1-1.1
0-2.0
0S-c
Ph-
Gra. bsa CL-as GPr-4o
Source Non Reasoning
Open
Reasaning
Commerical
Open Non-Reasoning
Open Source Reasoning
Commerical
(A) In-context Learning: (left) statement prediction, (right) post-loop prediction
00t
(B) RAG & COT: (left) comparing random shots and shots relevant to the query for statement prediction, (mid) COT for statement prediction. (right) COT for post-loop prediction
FIGURE 6. RQ4: Can different prompting strategies help?
8

---

Published as a conference paper at ICLR 2026
3.5 RESULTS FOR ROS: APPROXIMATION OF CODE SEMANTICS
In Fig. 7 we observed that the models reported better performance to predict an approximation of code semantics, for both statement (Fig. 7b) and loop (Fig. 7cp predictions. See also Table in Appendix A.8.3 for comparing with random baselines. Interestingly, when we provide only the definition of "abstract" values (mapping from range of concrete values an abstract value) in the prompt, without giving an example showing an "abstract" output for given input, the models cannot predict abstract values better than concrete values (Fig. 7ap. Most models failed to apply definitions directly to the query examples; however, when we provide 3-shots of examples in the prompt, all the models can predict abstract values better than concrete ones.
D2AC
Au 3033 OF7
9250 666
2007
(A) O-shot statement prediction
(B) 3-shot statement prediction
(C) 3-shot post-loop prediction
FIGURE 7. RQ5: Can models reason about approximation of code semanties? (Python results)
3.6 RESULTS FOR RQ6: DIFFERENT PROGRAMMING LANGUAGES
Using input/output prediction as a case study, we investigated models code reasoning capabilities for different programming languages. Fig. Sa shows that Java and Python performed better than when predicting output given input. Our intuition is that compared to the C code, Java and Python code are more high-level and closer to the natural languages than C; also probably models have seen less code than Python/Java code in the training data. However, the models reported the lowest accuracy for input prediction Python code (Fig. Sb).
1.0
Pytan
C/C+
Java
0.8
Pyheh
Jaca
CiDié
Accuracy
Accunaty
0.4-
0.2
La1 C25C DBCC
Pi1
drai D0R---R--00 1.8m-- CL.35 4P
cesc
D5-C
PHH
3.3.2 DERi-LGam-155 1. GPT-45
Open Source Nan-Reasoning
Open Source Reasoning
Commerical
Open Source Ran-Roasaning
Open Sourcs Reasoning
Commerical
(A) Output Prediction
(B) Input Prediction
FIGURE 8. RQ6: Any particular programming languages are easier for the models?
RELATED WORK
Code Reasoning Benchmarks. CruxEval (Gu 2024 assesses LLMs' performance synthetic
Python programs using the task of input-output prediction for a function. CruxEval-X (Xu et al. 2025) extends this work multilingual settings by translating synthetic Python programs CruxEval to other languages using LLMs. REval (Chen et al, (024) evaluated branch prediction tasks using ClassEval (Du al. 2024) and HumanEval (Chen 2021). CodeMind (Liu
2024 proposed
output prediction and code synthesis tasks existing code benchmarks (Chen et al.
2021 Du et al.)
2024 Gu et al., 2024 Ahmad et al., 2023: Austin et al. 202 Puri et al. 2021 They found that
LLM reasoning capabilities deteriorate as program complexity increases (Liu et al. 2024: Zhang
et al. 2024b). Our benchmark CodeSenseis the first that used real-world Python, and Java code to evaluate LLMs' reasoning capabilities. While most code reasoning benchmarks reason about function level execution semantics, we proposed and made ground truth available for spectrum of fine-grained reasoning tasks regarding program behaviors within function.

---

Published as a conference paper at ICLR 2026
Other Code Application Benchmarks. SWE-Bench (Jimenez et al. 2024 used the task of generat- ing patches to resolve a given GitHub issues for real-world Python projects. CVVE-PolyBench (Rashid)
2025) extends this work to other programming languages. KGym (Mathai et 2024 delivered a benchmark consisting of Linux kernel crash data and evaluated L.LMs' capabilities of resolving
al.
Linux kernel crashes. These benchmarks focus task-specific performance rather than fine-grained
code semantics understanding. There are also benchmarks for code generation (Li et al. 2024 Zhang
al. 2024a] Yu et al. 2024 Chen et al., 2025 Du al. 2024) and code completion (Izadı al.
2024 Ding et al. 2023b). However, most of these datasets such as BigCodeBench (Zhuo et al.
2024) and CodeBenchGen (Xie et al. 2024) are restricted single language (primarily Python)
and extracted from isolated competitive programming problems.
5
CONCLUSIONS
Code semantic reasoning is foundational for solving many software engineering applications. We propose novel code benchmark and dataset, CodeSense, extracted from 744 Python, C and Java real-world projects, for evaluating L.MSs capabilities of code semantic reasoning. We defined spectrum of fine-grained code reasoning tasks include value predictions at various granularities of the code and program properties prediction for important code constructs like loops, pointers and branches. We developed framework and tools that can build, test and trace software projects in different programming languages, and can automatically generate ground truth for fine finegrained code semantic reasoning tasks. We conducted a comprehensive study on SOTA LLMs. We found that models general lack the knowledge of code semantics and face challenges for reasoning about even single statements. In limited cases, models can establish the correlation of code semantics description natural language with some simple frequent code patterns. We hope our dataset and framework can enable further code semantic benchmarks and provide ground truth for future LLMs post-training.
6
ACKNOWLEDGEMMEN
This work was supported in part by NSF CCF-2313054, NSF CCF-2313055, NSF CNS-2247370, DARPA/NIWC Pacific N66001-21-C-4018 Any opinions, findings, conclusions or recommendations expressed herein are those of the authors and do not necessarily reflect those of the US Government, NSF, or DARPA.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai- Wei Chang. Avatar: A parallel corpus for java-python program translation, 2023. URL https: http:////w.xvv..r.ee.o abs/
2108 11590
Miltiadis Allamanis, Henry Jackson-Flux and Mare Brockschmidt. Sclf-supervised bug detection and repair. In NeurIPS, 2021.
Anthropic. Introducing claude 3.5 sonnet, June 2024. 3-
/ttp://ww/ttt.. htttss/www.anttrrppc..comm
Accessed: 2025-05-15.
Abhishek Arya. Oliver Chang, Jonathan Metzman, Kostya Serebryany, and Dongge Liu. Oss-fuzz, 2023. URLhttpp::///e thw.. wwt.tttwwt..ce.ttttooottotottttt wwtpm///ooll/e/eee
fuzz. Accessed: 2025-05-14.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: a arxiiiivvvv org atwpr/////////// / 2108 07732
Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. Reasoning runtime behavior of program with IIm: How far are we?, 2024. /RRLittts://aaxxi.oooo htap////ab/24000 2463..6437777 16437
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Bares,, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. BtLhhtts::/// httttt://aaxxi...oot org abs www.a22220 2110 03374
Simin Chen, Pranav Pusarla, and Baishakhi Ray. Dynamic benchmarking of reasoning capabilities in code large language models under data contamination. arXiv preprint arXiv:2503.041 2025.
Yangruibo Ding, Ben Steenhoek, Kexin Pei, Gail Kaiser, Wei Le, and Baishakhi Ray. Traced: awece pre-training for source code, 2023a. URL httpp::/arxiv..rt http://axxxv..../aaaa// /bss/ 2306. 07487
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Mu- rali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Crosscodecval: A diverse and multilingual benchmark for cross-file code completion, 2023b. URL https::/arxiis org 2310. 11248
Yangruibo Ding, Jinjun Peng, Marcus J. Min. Gail Kaiser, Junfeng Yang, and Baishakhi Ray. Semcoder: Training code language models with comprehensive semantics reasoning, 2024. URL
hettps:/arxiv.org www.//////////ooooo//oo///t..aot 2405. 01006
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference Software Engineering, ICSE '24, New York, NY. USA, 2024. Association for Computing Machinery. ISBN 97984000021744 doi: 10.1145/35975003333222 3597503.3639221
Gordon Fraser and Andrea Arcuri. Evosuite: automatic test suite generation for object-oriented the 19th ACM SIGSOFT Symposium and the 13th European Conference Foundations of Software Engineering, ESEC/FSE '11, pp. 416-419, New York, NY. USA, 2011. Association for Computing Machinery. ISBN 9781450304436. doi: 10.1145/20251 10251117
software. In Proceedings the
URL URL http:pss/ooo.orrr//.111550/11110

---

Published as a conference paper at ICLR 2026
Gordon Fraser and Andrea Arcuri. Sound empirical evidence in software testing. In Proceedings of the 34th International Conference on Software Engineering, ICSE '2, pp. 178-188. IEEE Press, 2012. ISBN 9781467310673.
Free Software Foundation. GDB: The GNU project debugger. htttps:/www.sourccss org/ grgg//d// Accessed: May 15, 2025.
Gemini Team al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05550, 2024. URL httptt:/axxii..orrtt aatpg//bb////3..//
05530
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution, 2024. URL
atttp::/aaxiis org/ abs 2401. 03065
abs
Maliheh Izadi, Jonathan Katzy. Tim Van Dam, Mare Otten, Razvan Mihai Popescu, and Arie Van Deursen. Language models for code completion: A practical evaluation. In Proceedings of the JEEF/ACM 46th International Conference on Software Engineering, ICSE' '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400702174, doi: 10.1 10.1145/3597503.
3639138. URL BRth:t:s: //tpi::://oi... org. . 1145/1145553 11122//....145557553333 3639113
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodehench: Holistic and contamination free evaluation of large language models for code, 2024.
org.
07974
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL
https://arxiv: wwtp://wwwrrrr///oo http:/bbb/231...0
Holger Krekel, Bruno Oliveira, Ronny Pfannschmidr, Floris Bruynooghe, Brianna Laugher, and Florian Bruhin. pytest x.y. http::::/gitiib..cm//ttttttsss dev/pytest] 2004. Version x.y. Contributors include Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt, Floris Bruynooghe, Brianna Laugher, Florian Bruhin, and others.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium Operating Systems Principles, SOSP '23 pp. 611-626, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702297. doi: 10.11457300000.36611666 UtLhttts:://db: I0.1145/360006.36.0
Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. Evocodebench: An evolving code generation benchmark aligned with real-world code repositories, 2024. URL httttss////xxii
org//bb/ 2404. 00599
Changshu Liu, Shizhuo Dylan Zhang, Ali Reza Ibrahimzada, and Reyhaneh Jabbarvand. Codemind: A framework to challenge large language models for code reasoning, 2024. URL https: 09664
//arxiv.org/abs
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023. URL
org/ 2a0os/2235...2 01210
Alex Mathai, Chenxi Huang, Petros Maniatis, Aleksandr Nogikh, Franjo Ivančić, Junfeng Yang, and Baishakhi Ray. Kgym: A platform and dataset to benchmark large language models on linux kernel crash resolution. Advances in Neural Information Processing Systems, 37:78553-887777 2024.
OpenAL. Gpt-4o mini: Advancing cost- intelliggeen July 2024. httpstts::/oppeaai com/ Sore--o-miin http:///wwo..aaaaaaaaaa/aaaaaaaa cessed: 2025-05-15.
Ac-
12

---

Published as a conference paper at ICLR 2026
Oracle Corporation. JDB The Java Debugger, 2025. URL https: wwwcce................oo...eo.e. com/
/nip:///unii//aaaaa html Part of the Java SE Develop-
ment Kit.
Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. Codenet: A large-scale
ai for code dataset for learning a diversity of coding tasks, 2021. URL https://arxiv.orrr
abs/2105. 12655
Ram Rachum, Alex Hall, lori Yanokura, et al. Pysnooper: Never use print for debugging again, jun 2019. URL htttp::/gittub.. cttp:cccocccoooooomooooo coo.l/cool-oo RR/ RR/PySnooper
Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, Anoop Deoras, Giovanni Zappella, and Laurent Callot. Swe-polybench: A multi-languaae benchmark for repository level evaluation of coding agents, 2025. htt.htttss///axxx http://awwwv..ogga////5////
08703
Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam, Hengbo Tong, Swarna Das, Earl T. Barr, and Wei Le. To err is machine: Vulnerability detection challenges
Ilm reasoning, 2025. URL https: / arxiv. org/ abs//bb////// abbs/2403. 2403. 17218
Danning Xie, Mingwei Zheng, Xuwei Liu, Jiannan Wang. Chengpeng Wang, Lin Tan, and Xiangyu Zhang, Core: Benchmarking Iims code reasoning capabilities through static analysis tasks, 2025.
URL http:///xxvvvo.///////./ abs/ 05269
Yiqing Xie, Alex Xie, Divyanshu Sheth, Pengfei Liu, Daniel Fried, and Carolyn Rose. Codebenchgen: Creating scalable execution-based code generation benchmarks, 2024. URL https://arxxx
org/abs/ 2404 00566
Ruiyang Xu, Jialun Cao, Yaojie Lu, Ming Wen, Hongyu Lin, Xianpei Han, Ben He, Shing- Chi Cheung, and Le Sun. Cruxeval-x: A benchmark for multilingual code reasoning, understanding and execution, 2025. /ttpt/tts:////tii.....
org/ //ag//bbss abs/ 2408. 13001
Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, and Shuiguang Deng. Codescope: An execution-basee multilingual multitask multidimensional benchmark for evaluating Ilms on code understanding and generation, 2024. URL
org abs 2311. 08588
Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. Codereval: A benchmark of pragmatic code generation with generative pre- trained models. In Proceedings of the IEEE/ACM 46th International Conference Software Engineering, ICSE '24, New York, NY. USA, 2024. Association for Computing Machinery. ISBN 9798400702174, doi: 10. 10.1145/35975503362255 0197503.3623333
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges, 2024a. URL
Yakun Zhang, Wenjie Zhang, Dezhi Ran, Qihao Zhu, Chengfeng Dou, Dan Hao, Tao Xie, and Lu Zhang. Learning-basser widget matching for migrating gui test cases. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE '24. ACM, February 2024b. doi: 10. (0.11145//5955033323332 3623322
htt.httt:///t /tdtt://dx..o doi wrg.oo..//110... wtt.oo......../99....0001 http:////oooo.o.oo1111...1.....
Terry Yue Zhuo Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng. Jiawei Liu, Qian Liu, Zijian Wang, David Lo. Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Hanm de Vries, and Leandro Von Werra. Bigeodebench: Benchmarking code generation with diverse function calls and complex instructions, 2024. ARRLhhtps::/aet
org/ abs abw////6..2... /2406 15877
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1
COMPUTATION RESOURCES AND INFERENCE TOOLS
All our experiments were conducted using the following computational resources:
GPU: NVIDIA RTX A6000 with 49GB VRAM
Memory Utilization: 0.9GB GPU memory during inference runs
Software Stack: vLLM Kwon et al. 2023 for optimised transformer inference
Operations: Inference-only experiments (no fine-tuning performed)
The inference parameters were controlled through the following configuration
TABLE 4. Inference Configuration Parameters
Parameter
Value
Description
temperature
0.8
Controls randomness: Lower more deter-
ministic
top_p
0.95
Nucleus sampling: Only top 95% probabil-
ity mass
max max_tokenn tokens
4096 (default)
Base context window size
16384 (For reasoning models)
Extended context for specific models
tp_size
No tensor parallelism
dtype
float16
Half- ceff--rreiiionn floating point
stop
\nn>> An$....
Generation stopping tokens
A.2 LIMITATIONS AND FUTURE WORK
In this work, we consider exact matching between the model's response and ground truth as correct. In future work, we would like explore other metrics such as pass@k. However, we did explore models' performance of computing abstract value prediction/approximaaiin of code semantics.
For RQs, only evaluated the models subset of tasks. For example, when comparing different programming languages in RQ6, we used input/output predictions. In the future, we will extend such evaluations to more tasks. When computing pointer alias in RQ3, we used languages. Future work can also include object aliasing detection for Python and Java.
Additionally, we would like to expand our framework support project tracing and task specific benchmark datasets beyond the scope of three languages. This includes adding other languages with diverse domains like (e.g., functional, system language), enable more comprehensive evaluation of L.LMs.
In future, will be also interesting to explore more advanced prompting techniques and even fine-tune the models to further evaluate them.
A.3 TRACE COLLECTION
In this section, will discuss the detailed process to generate the C, Python and Java real-world traces.
A.3.1 REAL WORLD PROJECT COLLECTION
1. C Project Collection:
For our research, we wanted to generate trace dataset real-world projects. We have selected real-world C projects that were curated the OSS- OSS-FUZZ repository. The primary reason behind selecting these projects is that they represent different domains to ensure diversity in software types. This choice also aligns with our research goal generate benchmark real-world C trace dataset.
14

---

Published as a conference paper at ICLR 2026
2. Python Project Collection: To collect real-world Python projects, adopted two ap- proaches. 1) We cloned all 1489 repositories from GitHub that appear in the PyPIBugs dataset, which was released in 2021 (Allamanis et al. 2021 2) To avoid missing popular projects after 2021, we use GitHub API to search for repositories that are marked as mainly written in Python and get the results according to the descending order of the number of stars. To maximise the probability that we can execute them easily with the pytest module, we only consider projects that seemingly have testing folder at the top or second level. For better compatibility and the reflection of the recent trend of programming styles, we further filtered out projects that have not been updated in the last four years. Finally, we got 544 projects.
3. Java Dataset Collection: For Java, we aimed to gather a diverse set of real-world projects have comprehensive trace analysis, which aligns with our trace dataset generation objective. We have used EvoSuite (Fraser & Arcuri 2011) for test suite generation, and the SF110 dataset has been used as it recommended by EvoSuite. This choice ensures compatibility and a high testing coverage rate.
A.3.2 HIGH-LEVEL STEPS OVERVIEW OF GENERATING TRACES
C Trace Collection
1. Building Projects:
Building projects before fuzzing is necessary to ensure that different project dependencies are correctly installed and configured, avoiding runtime errors during the fuzzing process. This helps to create a consistent and effective environment for the next fuzzing process.
2. Fuzzing: In the fuzzing phase, we executed the fuzzer on the already-built projects to generate the input data corpora. We configure the fuzzing tools with appropriate settings and parameters for each project. This includes specifying input seed files, maximum time for the fuzzer to run and kill delay to maximise code coverage. Throughout the fuzzing process, detailed logs are maintained to track the execution progress and other relevant information. These logs can aid in debugging, result interpretation, and fuzzing outcomes.
3. Tracing: Tracing the most crucial step to have the execution information of real-world projects. We use tracing framework with the GNU debugger to log the execution of the projects. With the help of the framework, we log function calls, variable values, and other states during the execution of the projects. We start the tracing by setting entry point for the program, and during the execution of the tracing, we record the different states of the program at various points by logging them into an XML-formatted file for further analysis. Additionally, we have added a tracer timeout to ensure the maximum running time of the tracer, as well an extra kill delay to ensure the safe exit of the tracer. This ensures the reliability and robustness of the tracing process if any unexpected events occur.
Python Trace Collection
1. Execution: We execute the collected projects get traces in best-effort approach: 1) We sean the common dependency files to install the dependencies into an independent Python enviromment for each project. 2) We use pytest execute the test cases in the projects and collect the outputs. 3) We analyze the outputs to identify missing dependency errors and try to install the missing dependencies several times.
2. Tracing: We use the PySnooper tool to trace the projects but made the following mod- ifications to it: 1) We only keep traces corresponding to source code files in the project source directories dependencies. 2) We expand the representation of user-defined class objects by showing the name and value pairs of their first-level attributes. 3) We save the types of variables in traces instead of just value representations to provide more information for the execution-awan source code modeling.
exclude traces happened in Python built-in functions or third-party
15

---

Published as a conference paper at ICLR 2026
Java Trace Collection
1. Tracing: To generate the tracing framework for Java, we integrated Java Debugger to record the execution details of the projects. We logged method invocation, variable values, and different program states during the execution cycle. For Java, we stored the raw trace in JSON format, which was stored in directories specific to each class within the project directories. This helps manage large amounts of data, consequently making it easier to retrieve, analyze and clean it up for further tasks.
TABLE 5. Trace Collection
Real-world projects
Testing Tools
Tracing
Tools
Python
PyPlhugs+Githuu (544)
Pytest
Pysnooper
C
OSS-Fuzz (100)
Fuzzing
GDB
Java
SF-110 (100)
Evosuite
JDB
A.4 LANGUAGE SELECTION RATIONALE AND EXTENSIBILITY
The three programming languages in our benchmark are important and representative for program- ming language features and real-world applications: C is low level programming language and useful for building systems, Java has object-oriented programming features, are widely used for building enterprise and web applications, Python is important for data science and Al applications. Our benchmark is extensible, third- shirddpaaties (as well as ourselves in future) can add more languages. Our scripts, prompts and methodologies can be adapted for new programming languages (1) select and download programs of a programming language from GitHub, (2) fuzz for generating test inputs, (3) trace and curate ground truth data (4) provide input and parse output when interacting with models. Instead of GDB (for C), Pysnooper (for Python) and Java Debugger Oracle Corporation (for Java), we will need to plug in debugging tools for new programming languages.
You may ask "we have compilers and code execution tools, why do we need models to predict dynamic information" required for many other downstream tasks. For example, in Fig. [T to generate test input that can exercise true branch, the models need to know how each operator in statements updates the values. The key difference is that: when using code execution tools, we give one input and ask for the output, but in other downstream tasks, we require models to first understand fine-grained semantics and then find inputs that can satisfy certain constraints. Even for compiling and executing programs, benefit from LLMs prediction, as compilation and execution can be time-consine consuming and hard to be configured, especially for legacy code. This ability of LLMs is particularly important when the user cannot run the code snippet, e.g., missing dependencies or unavailable resources. Predicting input/output values as code reasoning tasks have been established by prior research (Gu et al. 2024] (Chen et al. 202T), Yann et al.] 2024). Our work extended prior research by introducing real-world projects and fine-grained tasks supported only by our tracing framework.
Predicting dynamic values not only useful for executing programs, but
may
A.5 DETAILED DESCRIPTION TASKS
In this section, we will provide a detailed description of our tasks. A.5.1 provides comprehensive description of the statement-based evaluation we performed on LLMs. We sampled five types of statements, i.e, Assignment, Arithmetic, Constant, Boolean, and Function Call, and prompted the models about the value after execution of each type of statement given the variable states before executing that statement. For the block prediction task A.5.2 we sampled statements from the start of the code snippet, given the input of the code, we prompted the model to predict the output at the end of the 1st statement, the 2nd statement, and the 3rd statement. For Branch prediction A.5.3] promoted the model whether a specific branch will be taken or not of a code snippet, given the input of that code snippet.
In the case of A.5.4 we sampled loop statements from the code snippets. For each loop, we first collected the number of iterations of that loop ground truth, and queried the model about how
16

---

Published as a conference paper at ICLR 2026
many times the loop would be iterated. We also collected and prompted the model regarding the variable state inside the loop body after the n-th interaction. We have named this "In-Loop" prediction. Additionally, sampled variables after the execution of the whole loop and queried the model regarding the variable value after the execution of the loop body ("Post-Loop" prediction).
For input/output prediction A.55.5] used the approach similar to (Gu al.) (024). For output prediction, we give the entire code snippet and the input of the code snippet, and vice versa for input prediction.
17

---

Published as a conference paper at ICLR 2026
A.5.1 STATEMENT PREDICTION TASK
def xldate xidate_from_ datpdddtte_tppppee (date_tuple, datemode)
year, month, day = date_tuple data_iist 1ist (data_tuple)
if datemode not in (D, 1):
raise XuDateSadDatemode (datemode)
if year and month and day return 0.00
10C
if not (1900 <- <- 9999)
raise XiDatesadTupiee (trrrpppeeeraaaiii_eeeeasa Br & ((year, nonth, day),
if not (1 <- month <= 12)
raise XLDatesadruppe (" mo (yyaar, month, day),
if day <
or (day
(month) [month] and not (day 29 month and _eep (year)
raise XiDatesadTuple Invalid day: (year, month, day) )
Yp year + 4716
M month
if M <= 2:
Yp Yp -
Mp K. +
else:
Mp
jdn ifd(14 Yp, 19d Mp 16, 32) \
day 1364 ifdd:fd(XPp 184, 100) 3,
xldays jdn -_JDN_delta [datemode]
if xldays <- 0:
raise
menth, day) :8r" ((yaa,, month
d
if xldays < 61 and datemode -- 0:
raise XiDattembiguou
(yyaar, month, day)
[))
float (xldays)
xldate_from_datee (date_tuple= 7, datemode=0)
Assignment Prediction
What will be the value of the final output of the statement year, month, day
date cutee given ( date_tuple'; (1907, 7, after executing the statement?
Arithmetic Prediction
What will be the value of the final output of the statement Yp = year 4716 given [' l'year': 1907; after executing the statement?
Constant Prediction
What will he the value of the final output of the statement 0 given (' constant": 0. after executing the statement?
Boolcan Prediction
Will the true branch of the statement if datemode in (0, 1): be executed given datemode'
0}?
Function Call Prediction
What will be the value of the final output of the statement data 1ist si (data_tuple) given (' t"plee" (1907, 7, 3) after execut- ing the statement??
18

---

Published as a conference paper at ICLR 2026
A.5.2 BLOCK PREDICTION TASK
def exchange (a, i, ;): temp = [[i]
a[i] = a[il
a[i] temp
exchange (a- 100, 200, C, O, 0, O, 0, c, 01, 1-2, j-1)
I-Block Prediction
What will be the value of output the statement temp a [[]] after executing the statement given the function input a': 10, 100, 200, 0, 0, 0, 0, C, C, 01, 'i': 2, )):: 1?
2-Block Prediction
What will be the value of final output the statement a [i] a Ijj after exccuting the statement given the function input a': 10, 100, 200, 0, 0, 0, 0, D,
C, 01, 'i': 2, ' 1?
3-Block Prediction
What will be the value of final output the statement a [j] temp after exccuting the statement given the function input 'a': [0, 100, 200, o, 0, 0, 0, 0,
C, 01, 'i': 2, ' 1?
A.5.3 BRANCH TASK
def xldate xldate_from_dase letrro_ddat_ttupllie aidate_tupll. datemode)
3. year, month, day date_tupie
4.
5. if dasemode not in (0, 11:
6. raise XLDateradDatemodt (datemode)
7.
8. if year -- 0 and month -- 0 and day -- 0:
9. return 0.00
10.
11. not (1900 <- year 9999)
12. raise (attteaaddaplee"nvaiid
((yarr, month, day), 1)
14. not month 12)
XLDateBadTuple ( howaalddmoothhhe $ ((year, month, day),)
15. if day <1
16. or (day > days_in_month
and not (day == 29 and month == 2
and leap (year)
17.
raise XiDateBadTaple: wwtpaa/eeeniaaddnnnaaaa "I $ ((year, month, day),)
18.
19.
Yp year 4716
month
21.
22. Yp Yp
23.
Mp M
24 else:
25.
Mp - M 3
26. jdn = (1461 * Yp, 4) 19d x Mp 16, 32)
27. day 1364 (ffd + 184, 100) * 3, 4)
28 xldays jdn _DN [datemode]
29. xldays XiDateRsdTuplee
(year, month, day) :b- ((year,
month, day),
31. if xldays 61 and datemode 0:
32. raise XiDattembiguouu ("Beforel 900-03-01:8r" (yyear, month, day),))
33.
return float (xldays)
34.
35 xx.x11d0 bectie__rro__drr_ttppeeeeee
7, 3), C)
Brach Prediction
line 12, raise libateeaarupplee Invalid year: $r" (1year, month, day), executed when @ 7,
3), 0) is called?
19

---

Published as a conference paper at ICLR 2026
A.5.4 LoOP TASK
def
(vstr=None):
vatr None:
3. vatr _version_..
4. if vatr[0] == "y":
5. vstr vstr[1:]
6. components []
7. for component vstr.eplit [0]. split (". [
:):
8. try:
9.
components appena (int (component))
10.
except Valuekrror:
break
12.
components tuple (component.s)
13.
return components
14.
15.
make_ www.vveriinnttuleece 'v0.1.1")
Iteration Prediction
How many times will the loop on line execute when vekke__ersion_tur ple ('v0.1.1)) is called?
In-Loop Prediction
What is the value of components s in line 9 after 2nd iteration when make_-
versioa_tuple (' v0.1111) is called?
Post-Loop Prediction
What is the value of components in line 12 when make_version_tur
ple v0.1.11 is called?
A.5.5 INPUT-OUTPUT TASK
def cast_ (val, length None) if isinstance (val, list): val tuple (val)
output = val if isinstance (val, tuple) else ((val,) default length, 1))
exists (length):
assert len (output) == length
return output
Output Prediction
What will be the output the code given input (? vaval' :1, 'length': :4)?
Input Prediction
What will be the input of the code given output (1, 1, 1, 1)?
A.6 CONCRETE TO ABSTRACT MAPPING
For the approximation of code semantics, prompted the models to reason in abstract values, instead of reasoning about the concrete exact value. Table shows the mapping from concrete value abstract category, following the prior literature (Ding al. 2023a). When defining these mappings,
20

---

Published as a conference paper at ICLR 2026
carefully aligned the value ranges for each abstract category with the overall value distribution observed in our benchmark. We evaluated the abstract mapping results against a random baseline, where mapping rules were selected randomly from all available mapping categories.
TABLE 6. Concrete Value to Quantize Value Mapping
Type
Condition
Category
0 < u u 10
Positive Regular
10
Positive Large
== 0
Zero
Integer
- 10 < U <
Negative Regular
< -10
Negative Large
1.0 v < 10.0
Positive Regular
0.0 < u 1.0
Positive Small
10.0 <v
Positive Large
Float
===00 0.0
Zero
1.0 S 0 0.0
Negative Small
10.0 <10
Negative Regular
u << 10.00
Negative Large
len (s) 0
Empty String
len (s)
and s. issslphh()) ()
Alphabetic String
String
len (s) > 0 and s. s.isdigit()
Numeric String
len (s) > 0 and not (s (s.isalpha() () or s.isdigit ())
Mixed String
len (1st)
Empty List
List
len (1st) > 0
Non-Empty List
len (tup) = 0
Empty Tuple
Tuple
len (tup) 0
Non-Empty Tuple
len (dict) 0
Empty Dictionary
Dict
len (dict) > 0
Non-Empty Dictionary
len (set) 0
Empty Set
Set
len (set) > 0
Non-Empty Set
True
True
Boolean
False
False
NoneTType
None
None
21

---

Published as a conference paper at ICLR 2026
A.7 PROMPTING TECHNIQUES
The following is a subset of prompts we used to evaluate the models. The rest prompts are shown in our data package,
A.7.1 RQI PROMPT
Generalized Statement Execution Prediction Prompt
Here's some (lang) code. Each example highlights a single statement of (assignment, branch, or function calls) and shows you what the variable values look like just before runs.
Your goal? Figure out what the result will be right after that statement runs.
Here are (shot) examples to walk you through it:
Assignment Prediction Prompt
You' re given some (lang) code and one specific assignment line.
Here are the local variables just before that 1ine runs. Can you figure out what the value of the assignment will be af- terwards?
Code Snippet: (lang) (code)
Statement: (statement)
Before Values: (variables}
Answer using <ans></ans> tags, Do not include any extra in- formation.
Boolean Prediction Prompt
Here' s a branch (if) /Boolean statement in (lang), and the val- ues of the variables it uses.
Will the branch run? Answer YYes' or 'No.
Code: (lang) (code)
Branch Statement: (statement)
Condition Variables: (variables)
Answer using <ans></ans> tags, Do not include any extra in- formation.
Function Call Prompt
Here' s a function or APT call in (lang) with some parameters.
Based on the inputs, what will it return?
Code: (lang) (code)
Call: (statement)
Parameter Values: (variables)
Answer using <ans></ans> tags, Do not include any extra in- formation.
22

---

Published as a conference paper at ICLR 2026
A.7.2 RQ2 PROMPTS
Generalized Block Execution Prediction Prompt
Take look the (lang) code blocks. One statement is highlighted in each.
You'l1 also see the input values going into the function.
Based on those, try to figure out what the highlighted 1ine will do. Here are (shot) examples that show how it works:
Block Prediction Prompt
Here's s full function in (lang) and line of code inside it we care about.
Given the function's s inputs, what value will that line pro- duce?
Code: (lang) (code)
Statement: (statement)
Inputs: (inputs)
Answer using <ans></ans ans>> tags
Generalized input_outpli prompt
Here's s some (lang) code. You' 11 either get the inputs or the outputs, but not both. Your task is to fill in the missing part--predict the out- put if you know the input, or figure out what input must 'e produced the output.
Check out these {shot] examples for reference:
Output Prompt
Here's s some (lang) code and the inputs passed into it. What output do you expect from it?
Code: (lang) (code)
Inputs:
(input)
Answer using <ans></ans> tags
Input Prompt
You know the output of a piece of (lang) code. Can you fig- ure out what the input must/ ve been? Code: (lang) (code)
Output:
(output)
Answer using <ans></ans> tags
23

---

Published as a conference paper at ICLR 2026
A.7.3 RQ3 PROMPTS
Generalized Loop Prediction Prompt
Let's s explore some loops in (lang). You' 11 get the full loop structure along with the input values used in the code.
I'll ask you questions about how the loop body or post-loop values behave with those inputs. Here's s how it works with (shot) example (s)
Iteration Prediction
Take a look at this lang loop with some given inputs.
Question:
{ (question)
Code:
{lang)
(code)
Answer using <ans></ans> tags.
In-Loop Prediction
This is a (lang) loop and what the input the function looks like.
I'll ask you something about what happens inside the loop body.
Code:
(lang)
(code)
Question:
(question)
Answer using <ans></ans> tags
Post-Loop Prediction
This is a (lang) loop and what the input to the function looks like.
I'11 ask you something about what happens after the loop body.
Code:
(lang)
(code]
Question:
{question)
Answer using <ans></ans> tags
Branch Prediction Prompt
Here's s a branch (if) block statement. in (lang).
Will the branch run given the function call? Answer 'Yes' or 'No",
Code: (lang) (code)
Question: (question) Answer using bans></ans> tags, Do not include any extra in- formation.
24

---

Published as a conference paper at ICLR 2026
Alias Prediction
Here's s some (lang) code with two pointer variables: Pointer A: "[pointer_1]"
Pointer B: '(pointer_2)'
these pointers reference the same memory address? Answer "Yes" or "No"
Code: (lang) (code)
Function Input: (input)
Question:
(pointer_1j" and (pointer 2 (line (line_1)) point to the same memory location?
Put your answer in <ans></ans> tags.
A.7.4 RQ4 PROMPTS
Assignment CoT'
Let's s figure out the result the assignment: "Istatement You' ve got the current variable values: ivariables) Think through the right-hand side, then update the left-hand side with the result.
Boolean CoTT
Here's s the condition: (statement)' These are the variable values: (variables) Eva Evaluate the condition. Is it true or false? That tells you the branch runs.
Function Call CoT
This is the function call: (staateent]/ }
With these parameter values: [variables) Figure out what the function does and predict the return value.
Block Prediction CoT
First, trace the execution flow till the highlighted state- ment (statement) and (input) of the given input, Then identify the variables associated with the statement Next, use the trace execution flow to evaluate the statement What value does the statement produce?
Output Prediction CoT
We' re given inputs: (input) Walk through the code step by step.
Watch how the values change until we get the final output. Check that it matches what the function should return.
25

---

Published as a conference paper at ICLR 2026
Input Prediction CoT
know the output: (output)
Work wac input could' ve led to that?
Figure out what had to happen the code, and reverse it get the input.
Iteration Prediction CoT
Start the loop using the initial values. Check the condition, run the body, update, and repeat. Keep going until the loop ends.
Loop in- Value CoT
Look at the variables the start of this iteration. Go through each 1ine in the loop body. What happens the variables by the end?
Loop Post-Value CoT
See why the loop stopped (condition failed).
Check the final values of all changed variables.
What did the last iteration do before ending? What would be the variable value after loop termination?
26

---

Published as a conference paper at ICLR 2026
Overall Statement Prediction Prompt (1-shot) with CoT steps
Here's s some Python code. Each example highlights a single statement of (assignment, branch, or function calls) and shows you what the variable values look like just before runs. Your goal? Figure out what the result will be right after that statement runs.
Here are example to walk you through it:
EXAMPLE 1:
Here's function APT call in Python with some parameters. Based on the inputs/paramett values, what will it return?
Code:
Python
(in-context Code]
Function Call:
(statement with function call)
Parameter Values: (values)
Let's s think step by step:
This is the function call: (statement) With these parameter values: (variables)
Figure out what the function does and predict the return value.
Therefore the final answer is:<ans> (Ground Truth} </ans>
Now, please solve the following new problem.
You' re given some Python code and one specific assignment line. Here are the local variables just before that line runs. Can you figure out what the value of the assignment will be afterwards?
Code:
Python
(Query Code)
Statement: (selected statement)
Before Values: (values)
Answer using <ans></ans> tags, Do not include any extra in- formation.
27

---

Published as a conference paper at ICLR 2026
A.7.5 RQ5 PROMPTS
Statement Prediction Prompt with Abstract Mapping
You' re given some Python code and one specific assignment line. Here are the local variables just before that line runs. Can you figure out what the value of the assignment will be after- wards?
Code:
Python
(Query Code)
Statement:
(selected statement)
Before Values:
(values)
You have to give your value prediction using the given quantiza- tion rules: [rules_list]
Answer using <ana></ana> taga, not incluce any extra informa- tion.
A.8 ADDITIONAL RESULTS
A.8.1
RQ1
Fig. 9 and Fig. 10 depict each model's 's capability on individual statement types. Fig. and Fig. show the performance of all the models across five types of statements for languages Python and C.
12
DeepSeecCotentTA Bedrmttuct Desuiiies RJ distit Liana de
Dopshnee- 41 chat wtw---ll---------- Owen- ea
DeepSeec Suati--eenn11
Deer2 5 100 intruct IM
Qwent SCose-TBBirattuut
eettraak chude sparet 30243002
Farcton
FIGURE RQ1 Statement type accuracy across Models (Python)
A.8.2 RQ4
In Fig. 13 we show that for most of the models, adding the number of shots/in-contex. examples helps the models. Fig. d4demnsttts that selecting in-context examples in a more controlled way, for example, selecting the same function as in-context examples. helps the models reason better. Finally, Fig. [5] whether adding Chain of Thought (CoT) with the in-context examples can help improve the performance.
28

---

Published as a conference paper at ICLR 2026
Deepleell indturt
Desptawdort1-0 (11-0 at B-Jama-85
DespBeck Al st Owen 148
DespSeel--t: 81 at 8-2wan-70
Uime--12 10-intrust
mas extruet
genbii 4ash-002
grenite 3.2 2t-iiiittt
gravite- www.--/ettttrttrreeeeeo..
Foretie Cal
FIGURE 10. RQ1 Statement type accuracy across Models (C)
Type
Baxleen
Constore
Eurction Cell
Viriube
0.6
Reaccning Model
Proprittry
0.4
Llana-2 10 sitoeeeeee
antarrnntt 14
Cwwndd Bi Deter 76 AEEDa
Deseiied Caden Lie evereet
Pst lnstrret
prttempli dlatth soroa sto H1z22220
pembiid Blak
presiedd abs eotturs
Dhepdennh oivit Qhean 78
Despsion deali Laeaa
Deepltik Zowno
oreati db instracd presing
Rii
FIGURE 11. RQ1 Statement type accuracy across Models (Python)
29

---

Published as a conference paper at ICLR 2026
Coda: Acuaasy Motel and Siavement Typs
Arl vati
Constert
dreec 1a3
predt S-Bhopptott
dess
FIGURE 12, RQI Statement type accuracy across Models (C)
Abstract Statement Prediction Accuracy
Shot 0
1.0
Shot 2
Shot 3
0.8
Accurccy
0.2
0.0
Unma-3 3. 1-1-1-8-80 it-insttett
Qwnm2 4-148- Instinsttcctii IM
Comm22. 5 3-Coder-1 78 Instrict
Deeptieek Deeptiesk-Couer-/ee Coster Coutee--oootteeeeeeeeee Couter-/--Lt-eeeeee instructt
Phu- min-inittttt
Phi-3 S-mini instrectt
grante-33 2 in-iinttrcr
DesppSenk Rt Divifl Qwenn-2 2B
granite:2 2.22. --- www.-irstuuct-peee instrrud-preeiied preview
arthropic cleades 3. Sisannet 202200224400 4222
gemini 5. 4-fuch-Qr2
mnt-4o-mi 4o-iin
DeepSeek- R1 Gisttill Lianu-
Deep #1 (ittill Ownen-
FIGURE 13. RQ4 Models' Performance with increasing shots from to (Abstract Value Prediction).
30

---

Published as a conference paper at ICLR 2026
Abstract value accuracy comparision (3-shot)
1.0
Incortest: Semee Different
0.6
0.2
Usioa 5 1-58 mnstrrrttt
Ownzz KB Iotmrar 14
Ownst S-Caden 7B Rnittrree
Despdask Cosaer Lite entrortt
Phi mal-aminddoo inatuutt
Rt- 5-rt5----dooin 5-rird-nattuut anttiittttttt
granin: 48- abbinttrent
Derpfentt R1 Diiiiii Qenon 7B
granies 3,2 4D inbirtttutt previee
auttooppic classtee 2. sompel -044002--42222 g
gemiish 5. 002
Dawprlakk #1 Esstis Jame
Demptieet at Dissii Omann
gat
FIGURE 14. RQ4 Models' Performance with random and same function in-context Examples.
Abstract value accuracy comparision (CoT vs No-CaT)
1.
Rensoning Model Non Roasoning Model
0.6
0.
Usma--3 3.1 48thmamee
Gwent S. 28l instroet
OeemZ Casler 7B Instrrctt
arstroplic claucie Sarnaaa 2iteeett----0022----- 220
Wili mnnei
Pfi 5,5 messimiiiittttnn Instrrett
pemino 5 Bashhiaa
granit AB-inatrurr
Deendiekk Cremm
Desps 21 dlse Liama
Dieepiesk RL Dtiest Quer:
pratetec 3 2-th aesiemaaccc prrasiiee
Despdhesk Cuder
FIGURE 15. RQ4 Models' Performance CoT vs No-CoT
31

---

Published as a conference paper at ICLR 2026
A.8.3
RQ5
Fig. 16 we compare abstract value vs concrete value prediction for post-loop values. Though models struggle with concrete value prediction, they can improve the performance for predicting the range/approximation the concrete value.
Anstract Predition
Hodel
Concrid hediiciio
Oryyded Caler fostert
5-0 fosthor
Phi yot WeSTTo
Pis mma enthoot
nuttect
arthoggic cleade Fieemee 2014102
Beatrantt
pemeil Maydu
Despiied civell All
Devsllnti dt Basti Dhant- 78
gracie 38 sntttrd
gresse Revnatrsc previve
Desaiier dest
git
FIGURE 16. RQ5: Post-Loop value prediction abstract vs concrete values (3-shots)
A.9 EXTENDED EVALUATION ON FRONTIER MODELS
To address the concern regarding the performance of the latest frontier models, we added python statement prediction benchmark to include the DeepSeek-Ve V3, DeepSeek-RI and the recently released Qwen families. Table summarizes these additional results.
A.10 APPROXIMATION SEMANTICS ON RANDOM BASELINE
The random baseline is calculated by performing a uniform random selection across the possible mapping space for each specific code snippet. The Table g shows the models performance on the random bascline against O-shot and 3-shots.
TABLE 7. Statement Prediction Accuracy for Frontier and Reasoning Models
Model Name
Statement Prediction Accuracy
Proprietary/rronter Models
Authropic Claude 3.5 Sonnet GPT-4o-mini
0.65
0.60
Gemini 1.5 Flash-002
0.60
DeepSeek-V3 V3
0.58
Reasoning Models
DeepSeek-R1
0.53
DeepSeck-RI R1--istill----E
0.42
0.27
DeepSeck-RI-Diee -wenl-B
0.26
Qwen Family (New)
Qwen3-14B Qwen3-8B 8B
0.46
0.36
32

---

Published as a conference paper at ICLR 2026
TABLE 8. Model Performance Comparison on Abstract Value Prediction
Model Name
Random Baseline
0-shot
3-shot
DeepSeek-Coder-
0.084
0.247
0.504
DeepSet--RU-Distil-Laam
0.112
0.255
0.549
DeepSSek-RR--iisill--oe
0.169
0.316
0.784
DeepStek-RR--Ditiill--eee
0.113
0.238
0.614
Llama-3.1-8B
0.104
0.266
0.597
Phi-3.5-mini
0.077
0.121
0.577
Phi-4-mini
0.088
0.156
0.478
Qwen2.5-14B
0.176
0.218
0.618
Qwen2.5-Cov 5-CCode--B
0.099
0.222
0,509
granite-3.22 3,2-8b
0.152
0.396
0,528
granite-3.2-8b
0.147
0.367
0.535
A.11 API DEFINITION ABLATION STUDY
In Task 2, when predicting the output for output of an API call, we conducted additional experiments evaluate whether providing API definitions improves model performance. We tested two types of settings: (i) with No API definition and (ii) with API definitions.
We evaluation the open-source models and evaluated all the 248 function call prediction examples from our dataset. Table shows the results on the best-performing open-soure models:
TABLE 9. Accuracy of API prediction with different API definition strategies
Model
No API definitions
API implementation
Qwen 2.5-7B
0.206
0.226
Qwen 2.5-14B
0.182
0.194
Phi-4
0.125
0.089
Llama-3.1-8B
0.105
0.089
The results indicate that providing API definitions does not significantly improve performance, and in some cases slightly degrades it. This supports our hypothesis that the fundamental limitation for fine-grained code reasoning is not lack of API knowledge, but rather the models' inability to reason about statement-level and block-level semantics.
A. COMPARISON WITH EXISTING BENCHMARKS
We did partial evaluation of the output prediction task evaluation framework the three best-performing open-source models on CodeSense, and we present the results Table 10 Our result shows that the models perform significantly worse in CodeSense than CruxEval.
TABLE 10. Output prediction accuracy comparison: CruxEval vs. CodeSense
Model
CruxEval
CodeSense
Drop
DeepSeek-RR -i-itk----------I-OOO Seatll--Qeen-44BB 14B
0.75
0.37
0.38
Qwen2.5-14B
0.52
0.27
0.25
Qwen2.5-Codde-77B
0.50
0.30
0.20
A.13 VARIANCE ANALYSIS
We have run the statement prediction task, and the results in Table across three runs demonstrate that the variance across multiple runs is minimal, and this doesn't change findings.
33

---

Published as a conference paper at ICLR 2026
TABLE 11. Variance analysis across three runs on statement prediction task
Model
Run
Run 2
Run 3
Mean + Std Dev
Qwen2.5-149
44.4%
44.4%
43.3%
44.0%
0.6%
DeepSeek-R Biaeee-RR---Aee 1-Distill-Qwen-14 14B
42.0%
41.8%
43.8%
42.5%
1.0%
Qwen2.5- Twen2.5-Coder-7B
38.4%
38.4%
38.4%
38.3%
0.0%
DeepSSekkRR--Disiil--aaaa----
26.4%
25.7%
27.3%
26.5% 0.8%
A.14 DIFFERENT PROMPTING TECHNIQUES FOR STATEMENT PREDICTION
Table 12 shows the difference between two prompting strategies: using in in-context examples of the
statement type the query versus examples of different statement type.
TABLE 12. Statement Prediction Performance by Type
Model
Same Type Statement
Different Type Statement
Qwen2.5- 11np2255--4BBInsseeee InstruIsttttt--- 1M
0.44
0.42
DeepSeek-RI R1-eck-----isillll 1--iistlll-wen--lll 14B
0.42
0.39
0.38
0.37
0.32
0.29
Phi-4-mini-instruct
0.30
0.25
A.15 FUNCTION SIZE ANALYSIS
Table 13 shows how categorise function difficulties based lines of code.
TABLE 13. Function Size Categories
Category
Length (Lines of Code)
Small
length < 10 < length < 19
Medium
Large
length 2 20
A.16 EXAMPLES OF FINE- CANE-GRAINED INSIGHTS
Codesense's fine-grained task uncover reasoning failures code semantics that not visible to the coarse-grained benchmarks. For example, consider the simple function from our benchmark
def is dis_ascii (s): if isinstance (s, str): for a in s: if ord(c) > 255:
return False return True return _uppootts_unicode (s)
_is is_ascii(( (' 123456789*')
Question:
"How many times will the loop on line iterate?"
Ground Truth: 11 (The length of the input string s, which contains leading space, digits 1-9, and the # character).
However, models such as Owen2. S-Coder-7B incorrectly respond with 10. This error reveals that the model fails at fundamental level: it cannot correctly reason about string iteration, specifically miscounting the characters in simple string literal.
34


Reviews:
[{"review_id": "3dhBqcOHa2", "reviewer": "Reviewer_UCFL", "summary": "This paper proposes CodeSense, a benchmark for code semantic reasoning sourced from real-world code. The authors curate benchmark samples in Python, C, and Java from GitHub repositories and study a variety of research questions  such as block-level semantics, statement-level semantics.", "questions": "- To what extent do you expect training on traces (e.g. SemCoder, Code World Model) to improve the performance of models? Can you do an analysis to see if these models are significantly better than their non-trace-trained counterparts, or a simple experiment to measure this effect?\n- In your opinion, what kinds of insights does your benchmark enable that were not apparent from previous benchmarks?\n- In RQ6, how do you know the results are due to the language difference rather than the difference of the problems in the language?\n- How do the findings in this work relate to previous work? For example, CruxEval-X studies something very close to RQ6, can you compare the findings? I believe some other RQ's have works studying similar questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The benchmark is interesting, addressing the weaknesses of previous benchmarks (CruxEval, REval, CodeMind).\n- The authors study a diverse array of code reasoning and program analysis tasks. RQ3 and RQ5 are particularly interesting and have not been studied before (to my knowledge). Many other RQ's are also studied from a fresh perspective, and there is some analysis for each one (some extended in the Appendix)\n- A wide variety of models are evaluated and studied, and there is a lot of room for improvement among the studied models.", "weaknesses": "- The models studied in the paper are generally weaker than those on the frontier line, even taking into account the lag between the review date and the ICLR submission deadline. The paper would be stronger if it drew insights from failure modes of today's frontier models such as GPT-5, Gemini-2.5-Pro as well as open models like DeepSeek-R1, Qwen3. This would differentiate which of the paper's findings still holds true for the strongest models.\n- The research questions are interesting and open up the potential for in-depth exploration, but analysis is only done at a surface level. For example, the CoT's could be analyzed to understand where models are reasoning incorrectly. This is especially interesting for the strongest models. \n- Another example is that RQ3 could be studied more carefully, analyzing these trends for different programs (similar to the flavor of https://arxiv.org/abs/2402.05980). RQ5 is very interesting and could also benefit from an analysis of how the type and abstraction level of approximations affects accuracy.\n- RQ6 should be studied with paired data, to minimize the effect of confounding variables other than language.\n- No variance numbers are reported for the results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3SfmVoEMXP", "reviewer": "Reviewer_WPpx", "summary": "This paper proposes CodeSense, a benchmark for fine-grained code semantic reasoning constructed from real-world GitHub projects in Python, C, and Java. The benchmark introduces reasoning tasks at statement, code-block, and function levels, targeting semantic properties such as variable values, loop iteration counts, pointer aliasing, and branch conditions. The authors develop an execution tracing framework to automatically generate ground truth from real-world tests, and evaluate 14 state-of-the-art LLMs across six research questions. Results show that models struggle with fine-grained semantic reasoning, particularly on C code and arithmetic operations, with limited improvements from prompting techniques.\n\nThis paper addresses an important problem but lacks critical validation of the benchmark's value relative to existing work. While Table 1 compares features with CruxEval, CruxEval-X, REval, and CodeMind, no experiments demonstrate whether CodeSense provides unique insights or advantages that existing benchmarks cannot capture.", "questions": "Do you have any existing comparison data (even partial) showing how the same models perform on CodeSense vs CruxEval or REval?\n\nCan you provide specific examples of insights that CodeSense reveals but existing benchmarks cannot?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Important problem and motivation: Fine-grained semantic reasoning is crucial for real-world SE tasks like test generation, vulnerability detection, and bug repair. The motivation examples in Figure 1 effectively illustrate this.\n\n2. The paper provides clear presentation and informative figures. \n\n3.Releasing the framework, dataset, and leaderboard supports reproducibility and future work.", "weaknesses": "1. Missing Direct Comparison with Existing Benchmarks\nThe paper claims CodeSense provides advantages over existing benchmarks (Table 1) but provides no experimental validation. Table 1 only compares features (Real-world Projects, Fine-grained Reasoning) without demonstrating whether these features translate to better evaluation quality. The 14 models should be evaluated on both CodeSense and existing benchmarks (CruxEval, REval, CodeMind) to show: (1) whether CodeSense reveals insights that other benchmarks miss, (2) whether fine-grained tasks are more diagnostic than coarse-grained I/O prediction, and (3) whether real-world code introduces challenges absent in synthetic benchmarks. Without this comparison, the benchmark's unique contribution is unproven.\n\n\n2. Unclear Novelty of Research Findings\nThe six RQs (RQ1-RQ6) investigate important questions, but without baseline comparison, their novelty is unclear. For example, RQ4 finds \"chain-of-thought offers limited benefit\" and RQ6 shows \"models perform better on Java/Python than C\". I may have questions like are these findings unique to CodeSense's fine-grained tasks, or would they also be observed on CruxEval? The paper positions findings as new discoveries without demonstrating what existing benchmarks cannot reveal. \n\n\n3.The paper does not adequately justify why fine-grained semantic reasoning (statement-level, loop counts, pointer aliasing) is more valuable than existing coarse-grained tasks. While Section 1 motivates the need for semantic reasoning, no evidence shows that fine-grained evaluation better predicts performance on downstream SE tasks. For instance, does a model's ability to predict loop iteration counts correlate with its ability to generate test inputs or detect vulnerabilities? Without this validation, it's unclear whether the added complexity of fine-grained tasks provides practical benefits beyond existing benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0TRziNab1s", "reviewer": "Reviewer_C7bm", "summary": "Obvious template tampering. Recommended to desk reject.", "questions": "N/A", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "N/A", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BKLvSSmdBn", "reviewer": "Reviewer_cZr3", "summary": "The author propose a benchmark CodeSense that makes available a spectrum of fine-grained code reasoning tasks concerned with the software engineering of real-world code. The results show a clear performance gap for the models to handle finegrained reasoning tasks.", "questions": "* Figure 4: Models are bad at arithmetic so, it seems low accuracy is expected. \n\n* Line 375-376: What if the model is prompted with different type of statement, how would statement prediction perform?\n\n* Could the author include a non-reasoning and reasoning mode for Loop Properties, given the same model (e.g. Qwen3)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The coverage over Python and C are good. \n\n* Sourced from real-world project\n\n* Comprehensive testing over wide range of models.", "weaknesses": "* A bit of overselling: Java examples are only 74, which is negligible compared to the total number.\n\n* Figure 3b: What is “smaller” or “larger” function size? Any quantitative number?\n\n* How does author measure accuracy for API call? For something like `time`, it seems impossible to get it right, e.g. time interval for executing code blocks are simply random.\n\n* Can you give example of “abstract values”? For Figure 7, was the few-shot examples with abstract or concrete values?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)