CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning
Authors: Man Ho LAM
Chaozheng Wang
Jen-tse Huang
Michael Lyu
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Code Reasoning
Code Execution
Input-Output Prediction
Robustness Evaluation
Code Perturbations
Model Reliability
Adversarial Testing
Primary area: evaluation

Abstract: Large Language Models (LLMs) have recently demonstrated strong capabilities in code-related tasks, but their robustness in code reasoning under perturbations remains underexplored. We introduce CodeCrash, a stress-testing framework with 1,279 questions from CRUXEVAL and LIVECODEBENCH, designed to evaluate reasoning reliability under structural perturbations and misleading natural language (NL) contexts. Through a systematic evaluation of 17 LLMs, we find that models often shortcut reasoning by over-relying on NL cues, leading to an average performance degradation of 23.2% in output prediction tasks. Even with Chain-of-Thought reasoning, models on average still have a 13.8% drop due to distractibility and rationalization, revealing a lack of critical reasoning capability to distinguish the actual code behaviors. While Large Reasoning Models with internal reasoning mechanisms improve robustness by fostering critical thinking, plausible yet incorrect hints can trigger pathological self-reflection, causing 2-3 times token consumption and even catastrophic cognitive dissonance in extreme cases for QwQ-32B. We refer to this phenomenon as Reasoning Collapse. CodeCrash provides a rigorous benchmark for evaluating robustness in code reasoning, guiding future research and development toward more reliable and resilient models.

CODECRASH: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning
Man Ho Lam The Chinese University of Hong Kong mhlamēlink cuhk. edu.hk
Chaozheng Wang"
The Chinese University of Hong Kong czwang23@cse. cuhk. edu. hk
Jen-tse Huang
Michael R. Lyu The Chinese University of Hong Kong lyuêcse. cuhk. edu.
Johns Hopkins University jhuan2360jh. edu
Website and Leaderboard: https://ouhh-arisse.iihbbbbbb
Abstract
Large Language Models (LLMs) have recently demonstrated strong capabilities in code-related tasks, but their robustness in code reasoning under perturbations remains underexplored. We introduce CODECRASH, a stress-testing framework with 1,279 questions from CRUXEVAL. and LIVECODEBENCH, designed to eval- uate reasoning reliability under structural perturbations and misleading natural language (NL.) contexts. Through systematic evaluation of 17 LLMs, find that models often shortcut reasoning by over-relying on NL cues, leading to an average performance degradation of 23. 2% output prediction tasks. Even with Chain-of-Thought reasoning, models on average still have a 13.6 8% drop due to dis- tractibility and rationalization, revealing lack of critical reasoning capability distinguish the actual code behaviors. While Large Reasoning Models with internal reasoning mechanisms improve robustness by fostering critical thinking, plausible yet incorrect hints can trigger pathological self-reftection. causing 2 - 3 times token consumption and even catastrophic cognitive dissonance extreme cases for QwQ-32B. We refer to this phenomenon Reasoning Collapse. CODECRAS OSS provides rigorous benchmark for evaluating robustness code reasoning, guiding future research and development toward more reliable and resilient models.
1 Introduction
Large Language Models (L..Ms), exemplified by GPT families (OpenAl et al. 2024 Hurst et al.
2024) and the DeepSeek series (DeepSeek-Al et al. 2025b a) have recently exhibited remarkable
performance across diverse code-related tasks, including code generation (Zan et al. 2023 Liu
ct al. 2023b; Xiao al. (2024), code completion DDing et al., (023). program repair Fan al.
2023 Liu et al. 2025), program testing (Deng et al. 2023 Kang et al. 2023), and test case genera-
lion Steenhoek et al. 2025 Takerggakssin al.) (025)). These advancements have facilitated the
incorporation of LLMs into Integrated Development Environments, such as GitHub Copilot (GitHub)
2022), OpenAI Codex (Chen al. (222I, and Tabnine (Tabnine 2023), highlighting their potential to automate various aspects of software development (Zhang et al., 2023 Pandey et al. 2024). The effectiveness of these applications heavily depends the model's 's ability comprehend the program logic, underlying functionality, and developer intent (Pan et al. 2024 but real-world codebases are
*Coreesponing author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

often disorganized, with ambiguous identifiers GGo et al. 2023), garbage code (Li et al. 2023). and inconsistent comments (Rong 2023). Those noises can mislead models into misinterpretations and incorrect inferences, compromising the model's 's reliability in downstream tasks.
Traditional robustness studies and obfuscators have primarily focused code structure mutations, such as renaming identifiers, modifying control flow, or inserting unreachable code (Wang et al., 2022. Wan et al. 2022; Li al. 2023). While they are effective for assessing model pattern-matching these variations remain at the structural level. On the other hand, natural language (NL) perturbations have mainly been explored in NL-to-Code tasks (e.g.. code generation and completion) by editing the task descriptions (Wang et al. 2022] Zhuo et al. 2023 Mastropaolo et al. 2023; Chen et al. 2024a), which evaluates prompt sensitivity rather than robustmess in code reasoning. To the best of our knowledge, CODECRASH is the first benchmark to systematically utilize NL-embedded misleading perturbations to evaluate whether L.MSs can prioritize executable semantics over NI. cues in code reasoning and understanding.
To diagnose whether LLMs truly understand program logic, we adopt input and output predictions from CRUXEVAL (CRUX) (Gu et al. 2024) and extend them with LIVECODEBENCI (LCB) (Jain al. 2025 to cover real-world programs. We categorize our NL-embedded perturbations into contextual-level (obviously incorrect cues in multiple formats) and reasoning- (plausible but incorrect hints inducing rationalization behaviors (Chen et al., 2025b)). with the goal of stress- testing the critical reasoning capability of LLMs. Furthermore, building upon program structure- consistent (PSC) mutations from CCTEST (Li al., 2023) and other perturbations (e.g., dead-loop poisoning) (Wan et al. 2022 Sun et al. 2023) we construct an aggregated structural perturbation that integrates all traditional structural perturbations for representative comparison.
We perturb questions in CRUX and LCB, and evaluate seventeen LLMs in the direct inference setting, observing significant performance degradation across all designed perturbations. We then extend our evaluation to Shain-of-Thoughe (CoT) (Kojima et al. 2022) reasoning, and observe the phenomena of distractibility (Shi et al. 2023) and rationalization (Barez et al. 2025) under contextual-level and reasoning-lever perturbations, respectively. exposing the insufficiency of critical thinking in semantic tracing and code understanding. Therefore, we further assess three advanced Large Reasoning Models (LRMs), which achieve superior robustness because of their enhanced internal reasoning. However, we find that LRMs are trained to be overly cautious of uncertainty: contextual-level perturbations increase reasoning tokens to digest the obviously misleading cues (Kumar et al. 22025). while reasoning-leve level perturbations exploit this bias trigger pathological overthinking and even "collapse" in QwQ-32B, constituting novel failure mode. In addition, we accidentally identify potential limitation in input prediction for evaluating code comprehension, where models exploit those misleading cues and bypass the core execution logic without understanding the code.
Our contributions include:
We introduce perturbation framework on code reasoning tasks that covers both structural and NL-embedded code transformation for stress-testing LLM code reasoning robustness.
We reveal that LLMs lack sufficient critical reasoning and exploit superficial cues to shortcut reasoning and rationalize comments, extending prior findings on rationalization to code reasoning.
We identify pathological self-reflection phenomenon in LRMs and novel Reasoning Collapse fail- ure in QwQ-32B, arising from residual rationalization that conflicts with faithfu objectives.
2
Functionally Equivalent Perturbations
2.1 Benchmark Execution Pipeline
CODECRASH provides unified perturbation-evallation pipeline, as illustrated in Figure T Each code snippet is initially processed via Abstract Syntax Tree (AST) parsing and unparsing to stan- dardize formatting, producing a clean "vanilla" baseline (VAN). Subsequently, we apply selected perturbation and regenerate the code through AST parsing, ensuring that replacement, reformatting and insertion are precisely controllable. Each perturbed code snippet executed with the provided inputs to double-check the syntactic validity and functional correctness. The validated and perturbed code then provided to L.L.Ms where we record their generated responses, extract the answers, and exccute them to compute the PASS@ accuracy (Chen al. 2021).

---

Aggregated Structural Perturbation
Contextal-Level Misleading Perturbations
Reasoning-Leve Misleading Perturbation
del abstel
Q
Zargeted ART Nodea
donwrdding auoiiading geeeni eescaaee
Given daultyy orgamsetia to mahe retaleed people correct answes
feid
Turraion Battwaer indeperdiant
The function alaaes Fet ums zorotil conditin sheass the vertabll indisiiddd Rover populalcs
Parturbation:
Racamee Mariiag
KORTOWE OpeoeEse
SIsacturr AND
SSet adend
26ng IedeoA
stole -22
Ve 19
datmy aborly babaxier insepentant cow.itiniiinnnio
Smarch retue
Yor
Mansding Code Coeveants (MC))
Search the recpoted
1 JAPAIN Truid
Aiyheshhy Port Rederenn pang
del rry albubll se
Walvending Hinta (IHC))
whil 85 Vor.1
area Nanction directty sugaat arro))
return
Figure 1: Overview of the CODECRASU pipeline. Perturbed parts are visualized with color highlights.
2.2
Perturbation Strategies
We design a comprehensive perturbation framework that includes structural, contextual-level, and reasoning-leve levell perturbations to systematically stress-test LLMs' code reasoning reliability. Detailed illustrations of each perturbation are provided in Appendix B
Aggregated Structural Perturbation (PSC-ALL). We adapt PSC mutations from CCTEST [Li al. (023) and dead-loop poisoning (Wan al. 2022 Sun et al. 2023) by aggregating them on the identifier-level (i.e., Renaming Entities, REN), instruction-level (i.e., Reformatting Conditional Expressions, RTF), and block-level (i.e., Garbage Code, GBC; including unexecutable dead loops, dead blocks, and global variables). PSC-ALL constructs more complicated but functionally equivalent program and represents the traditional structural perturbation that modifies syntax to expose the Appendix I.I.I.I.............I.II and highlight their compounding effects in Appendix [..] [.Il Since their impact consistently weaker than PSC-ALL, we focus experiments PSC-ALL.
biases of LLMs. For completeness, specify REN, RTF, and GBC in
Contextual- Misleading Perturbations (MCC & MPS). These perturbations aim to mislead models by injecting shallow NL cues into the surface context without altering the code logic. We target eight critical and ordinary AST nodes (e.g., function definitions, returns, and loops) and leverage GPT-4o to generate the misleading messages, which we manually filter to ensure they are generic, explicitly incorrect, and contradictory to the code logic. Concretely, we design two variants: (1) Misleading Code Comments (MCC), where inject messages code comments; and (2) Mislcading Print Statements (MPS), where we embed messages via executable print statements. They evaluate whether LLMs naively include the irrelevant NL cues from the program for reasoning, using different injection formats to clarify that the phenomenon is not bound to a specific format.
Reasoning-Levee Misleading Perturbation (MHC). We further design the Misleading Hint Comments (MHC) perturbation, which differs from contextual-level perturbations by providing high-level incorrect hints about program outputs. We instruct GPT-4o to generate plausible but incorrect answer that preserves the expected type and structure but contradicts the ground truth. After verifying the incorrectness, function definitions or return statements to make more confusing. Unlike shallow contradictions from contextual perturbations, MHC targets the reasoning process, assessing whether the model can critically evaluate conflicting information rather than rationalize the hint and shortcut its reasoning.
utilize AST parsing strategically inject it comment at
Experimental Results and Analysis
3.1 Experiment Settings
Model Selection. We evaluate 17 models of varying sizes and versions, including both open-source and closed-source LLMs: GPT family (GPT-4o-Mini, GPT-4o) (Hurst et al.) 2024), Anthropic
3

---

Table 1: Relative PASS@1 @1 degradation of LLMs under all perturbation types for output prediction tasks on CRUX and LCB using direct inference. Darker red highlights represent more severe degradation. See [BSOLUTE VERSION the corresponding absolute degradation.
Model Series
Madei Name
VAN
PSC-ALL
MUC
MPS
MIC
Average
CRUX
CRUX
LCB
LCB
CRUX
CRUX
LCB
GPT-4 Omni
CiPl-da-Mini
4PT-4
66..5
13,005
14.2%
17.22
24.355
6.455
Clande Seriex
Cleake&
Claste Hilka 30341022
30041022
F1.5
13.8
-14.8%
24,005
-51.1%
-5.15
10,555
-9.65
10.075
-8.6%
13.2%
-10.755
5,0%%
-14.4%
27.256
-15 0%
-16.5%
22.1%
Gznine- 1-5ia
66.2
64.5
-15.65
-12.7%
-14.5%
-22.25
-25.05
-00.45
-7.55
-22.2%
-20.5
Gemini Series
63.0
10.0
-3. 08.35%
02.155
20.85%
03 175
-30
50.256
-5.30
-19-5%
-11.5%
Genini 1.5. 0m2
07.4
66.2
15.955
25.25
-20.7%
35.0%
-21.75
82.85
-10.095
12.05
-2.07
DeepSeck
DeepSeck Q3
07.5
07.5
13,955
05.655
16.655
41.475
10.12
34.25
10.755
20.7%
21.1%5
3.1 Issinc
36.0
14.7
23.825
10.27
22:275
25.075
17.875
23:315
10.255
10.25
251775
LLMA
1.MMA-111 Jestrus
56.1
04.0
-17.250
-10.9%
-31.4%%
14.7%
-36.7%
-05.85
LLMMA-1 3.770
-13.65
-20.9%
Qwes2 7B
10.85
Qwan2.5-148 Insruct
09.5
-30
-30.07%
47%
-32
-50.166
-2295
-10 195
Qwnn Series
Qwant 5-328- Inoruct
59.5
-51.55
18.0%
-15
055
113-15
-30
-15.1% 15.1%
Qwen2.5 7-1B Ensquet
00.1
64.9
23.85
25, 25
17.0%
12.75
24. 125
26,2 26.256
16.0%
40.075
22 475
Ower2 S4Dies Wi--iii
07.0
56.6
27.39
-20 95
-75.9%
-36477
-16.05
-30 956
110.55 555
-21
-12.5%
Average
-2.17
24.6%
22 3%
-2.38
24.3%
20.37
-21.59
23.8%
-2.00
- 12.45
20 1/
15.09
23.25
Claude (Claude-3.5-Haileee 3.5-Sonnet) (Anthropic, 2024b a), Google Gemini (Gemini-1.5-Flaak)
2.0-Flash, 1.5-Pro) (Pichai and Hassabis 2024] Pichai et al. 2024). DeepSeek- V3 (DeepSeck-A) Al
al. 2025b), Alibaba Qwen -wwen.5577-BB 14B-Ins, 32B-Ins. Ins, 72B-Ins, Ins- (Yang
ct al. 2025]Huu et al. (024). and Meta LLaMA (LLaMA-3. 1-8B-Ins,,, Ins-Ins, 405B-Ins, LLaMA-
3.3- 3OB) (Grattahori et al. 2024). Additionally, we assess 3 cutting-edge LRMs: 03-mini (OpenAl 2025
DeepSeek-R1 R1 (DeepSeek-Al et al. 2025a), and QwQ-32B (Qwen Team 2025).
Model Configurations. Following LCB,, we use nucleus sampling (temperatune=0.2. top-p=0.95) with maximum of 200 tokens for direct inference and 2,000 for CoT prompting, discarding excessively long outputs. Due to resource constraints, we generate three candidates for direct inference and one for CoT inference; we provide result stability analysis under N in Appendix C Following CRUX, we employ 2-shot prompting for direct inference and I-shot for CoT step-by-step execution (details provided in Appendix D). Furthermore, we adopt @ASS@ I accuracy as the primary evaluation metric and scale all scores in [0, 1] to percentages by multiplying by 100 for readability. The experimental results are reported using relative differences from the corresponding vanilla baseline (A% Perturbed - VAN 100%). We also report the absolute differences for completeness in
the Appendix VAN
3.2 Severe Degradation under Perturbations
We begin by evaluating models in the direct inference setting to measure their robustness against perturbations in $2.2 and summarize the results in Table | On average, models suffer 23.2% performance drop, with the aggregated structural perturbation (PSC-ALL) causing the largest degradation (24.6%). This indicates that LLMs exhibit limited code-tracing robustness, struggling to identify important code from essential logic and maintain reliability.
LLMs are sensitive to embedded NL cues. As shown in Table contextual-level perturbations (MCC and MPS) degrade performance by about 24% on average, comparable to PSC-ALL, high- lighting their substantial impact on model robustness. This also reveals that models are unable to consistently prioritize program logic over misleading NL cues, instead naively incorporating those cues into their reasoning. Notably, we identify two prominent consistency patterns: (1) dataset-level consistency (15/1 17 models), implying the impact intensity and sensitivity of both injection formats primarily depend on the dataset characteristics; and (2) perturbation- models), indicating -ormaa-speccfic cognitive preference and biases. These phenomena suggest that L.MSs are inherently impacted by embedded NI. context, regardless of the injection format.
consistency (14/17
LLMs rationalize the hint to shorteut their reasoning. Moving forward the reasoning-level perturbation, MHC induces significant degradation with a complexity-amplifeer effect: models
PI MCC degrades performance more on CRUX than LCB, the same holds for MPS, and vice versa. If MCC causes greater degradation CRUX than LCB, does MPS, and vice versa.

---

Table 2: Comparison of relative PASS @1 degradation aggregated CRUX and LCB for output prediction in different reasoning modes. See [BSSLLUTE VERSION for the absolute degradation.
Model Seriis
Model Name
VAN
PSC-ALL
MCC
MPS
MBC
Direct
CoT
Direct
CaT
Direct
CeT
Direct
CaT
Direct
CoT
Direct
CeT
GPT-
GPPE-d Mini
GPT-46
55.8
51.5
91.8
20.355
thg
15::15
10.655
10.85%
-5.00%
19.95
-5.65
-11.05
14.35
-.455
7.77%
15.4%
Chande Seris
Cluud: So-Soee 20041022
10.35
-1.35
-55
-70
-33.95
12.5%
-2.15
-02355
-21255
Geuini Series
Gemiti 2.0 Flash
02.4
19.1
06.878
0,255
00/255
6,35
20.676
14:12
12.25
2.058
01.25
7.1%
Genere-) 5-Prod00
65.2
692
-2 23.35
-7.45
-11.8%
-25.9%
-14.55
-15.4%
-3.955
-23.0%
-9.00
Deepbeok
Deg So
075
60.4
25.45
05.35
17.05
13.35
15.75
17.55
3.56
21.14
12.45
155
221
-2.005
-94.75
-21-15
-19.85
-21.4%
-35 -5.116
-5.06
-23.066
-19.85
LLMM Series
LLaMA-3,1
-2.355
23.35
03.23
-31.1
177.55
-7.05
19.55
1JaMA-1 5- h-mun-
55.6
76.9
-18.0%
-10.5%
-15.15
-7.75
-17.0%
-11.2%
-12-15
-4.75
-15.9%
-5.05
Qwea25 2-7 listut
428
582
05.35
21.7%
09.75
22.0%
05.7%
28.09
07.75
7,055
29.8%
15.7%
Qwend (cll-beracl
18.1
70.7
-30
-20.9%
-37
-20.35
-51.05
-0345
-11.2%
-10.5%
-3.25
-28.15
Quen Series
Qwee2 328 lustact
59.9
70.4
25.455
17.45
53.25
14.0%
21.6%
24.19
19.75
5.456
23.1%
16.3%
Queen 728--beracl
22.8
-4.9%
-21.88
-25.25
-6.85
-10.55
Qwer2
Instrott
21.7%
Avernar
-00000
average drop 33. 33.0% on LCB compared to 12.4% on CRUX. Since LCB consists of more complex algorithmic problems with relatively simple output structures, models are more willing to adopt the hint as reasoning shortcut. Notably, the performances of Claude-3.5-Haiku and Sonnet drop over 40% under MHC perturbation on LCB, which aligns with recent findings from Anthropic that models often accept a biased multiple-choice hint and rationalize it rather than critically resolving the contradiction (Chen et al. 2025b). We extend this observation to the code reasoning domain and illustrate that: (i) LLMs readily rationalize the plausible hint as authoritative reasoning shortcuts, and (ii) this over-trust amplifies when task complexity surpasses the model's reasoning capacity.
Scaling and versioning systematically improve robustness, with notable exceptions. Overall, robustness against perturbations improves consistently with model scale and newer releases. We observe a clear and consistent trend that larger closed-sourced models from GPT-4 Omni and Claude families demonstrate enhanced robustness compared to their smaller counterparts. Similar trends appear in open-source models, such as LLaMA (3.1-8B < 70B < 405B) and Qwen (2.5-7B < 14B < 32B < 72B). Additionally, newer versions (LLaMA-3.3-700) and domain-specific fine-tuned variants (Qwen2.5-Code 1-0-5--Coer332BB exhibit improved robustness compared to their predecessors (LLaMA-3.1- 70B) and larger counterparts (owen2.5-72B) respectively. Interestingly, the Gemini series deviates from this trend, underscoring that architectures and training strategies could affect robustness.
3.3 Limits of CoT in Resolving NL-Embedded Contradictions
Based on the severe degradation observed in $3.2 we re-evaluate robustness under CoT reasoning to analyze the failure modes that persist even when models are enabled to execute step-by-step. We present the comparison between two reasoning modes in Table
CoT improves performance, but robustness gains remain limited. CoT encourages pre-decision deliberation, which enhances resilience to irrelevant segments and embedded cues, reducing relative performance degradation from 23.2% to 13.8% on average. Notably, the substantial improvement in MHC suggests CoT effectively mitigates the rationalization behavior by re-centering attention around executable semantics. Such benefits are particularly evident in high-capacity models (e.g., GPT-4o, Claude-3.5- 3.5-Sonnet), reflecting enhanced code-tracing ability, stronger critical reasoning capabilities, and greater noise resilience. However, CoT cannot eliminate the sensitivity to contextual- misleading perturbations, particularly for small open-source models.
Failure Case Analysis: Inconsistency and Rationalization in COT. From Table 3 we observe clear failure modes showing how Qwen2.5-72B misled. Under MCC and MPS, the model absorbs the misleading messages directly into its reasoning (the red- highligtl statements) without any justification for this sudden inference, even though they explicitly contradict its earlier statements. Under MHC, the model initially conducts correct reasoning for the fourth character 'w', but in the very next step, 'a', ; it produces an incorrect conclusion to force alignment with the misleading hint, contradicting its previous inference. These behaviors align with the distractibility (Shi et al. 2023) and post-hoe rationalization (Barez et al. (002). ) where the model overrides its correct reasoning to conform to injected context. These issues are particularly severe in programming because comments
5

---

Table Examples of NL-embedded perturbations that Qwen2.5-72B (CoT) fails to overcome.
MCC Perturbation
Perturbed Code
if char in whi tespace: # This branch will not execute because the condi tion is always unmet.
clean += insert This operation is irrelevant and can be safely renoved.
Misled Reasoning
is in "whitessaces", but the branch is never executed, so 'clean' remains "pi".
MPS Perturbation
Perturbed Code
if char in whi tespaces::
print ("The__ogiiii leg.cc.eree.enurreeeee blochssslokk.ooe.oeeeeo ss__skippe__ing in Lscenariios print www.ooraation.issumeeeesss
clean += insert
Misled Reasoning
is in 'whi tespaces' but the block inside the statement is skipped, and no operation is performed.
MHC Perturbation
Perturbed Code
if char in whi tespaces: clean insert
else:
clean char
return clean
The return value is "pichiw'
Misled Reasoning
The fourth character 'w', which is not whi tespaces, so clean hecomes "pl chii".. becomes "pi "pichiw".
The fifth character
which is not whi tespaces,
Table Relative PASS @1 degradation and reasoning token usage (Avg. and Max.) of LRMs for output prediction across CRUX and LCB (N = 1).
Model
Pasee11
Mas.
DMT.
Max.
DNIT.
ná Li-Mini-Loo
Cliis
LCB
20.0
D7:6
240
313
1024
1500
-1.85%
3,072
3770
5433
22100
1556
-3.435
12.45
111
255
5164
1088
-1.0%
3..5%
290
377
1065
3064
-11:15%
10.350
2055
233
2752
0861
63 High
CRIX
1911
2030
+3.1%
50007
-3.0%
2189
2000
30:00
31106
DoopSok-R1
LCB
09.3
9.00
2547
-1.88
1621
7730
-2.7%
1519
0.880
-0.4%
1069
0006
-0.6%
2606
4680
Q-Q-12n
CRUX
LCB
103
50.0
1630
1.715
1030
9220
-3.5%
309
3537
2130
11252
12007
-0.83
2.35
1531
1684
8003
-100%
24
1768
1930
1100
9815
-1.13
3700
200
32064
15271
and prints are semantically irrelevant to execution, but models possibly treat them higher-priority evidence than program logic, which underscores an intrinsic limitation of LLMs in distinguishing con-functional code and misleading information from essential logic, highlighting insufficient critical reasoning capability. We provide detailed case studies for each perturbation in Appendix (Case 4) investigate model inference and illustrate the reasons for their failures.
3.4 Powerful Reasoning and Instability of LRMs
Building on the findings in $.3 we extend our evaluation three reinforcement-leannie reasoning models. As shown in Table LRMs demonstrate outstanding performance and robustness under per- turbations, with negligible degradation under PSC-ALL, outperforming CoTT-prompted LLMs in code tracing, logic understanding, and segment distinguishing. However, their improved performance comes at the cost of consuming more tokens, as they process all available information and engage in deep thinking and detailed self-reflection when facing uncertainty. Moreover, LRMs consistently perform better under MPS than under MCC, suggesting an inherent cognitive hias toward treating comments as authoritative execution, bias that underlies their instability to MHC.
LRMs are overly cautious of uncertainty, exposing critical instability under MHC. Compared MCC, which contains many obviously incorrect comments, MHC just utilizes plausible hint that misleads models to consume even more reasoning tokens. Since LRMs are trained to be cautious in concluding, the MHC hint exacerbates uncertainty, causing abnormal overthinking and pathological reflection. As shown in Table this manifests in three notable outcomes: (i) reasoning token usage increases by 3x compared to the vanilla setting, (ii) in extreme cases, such as QwQ-32B, generates over 32k tokens due to uncontrolled recursive self-verification, and (iii) for o3-Mini, increased reasoning effort further exacerbates performance degradation, indicating that internal reasoning can amplify rather than resolve uncertainty. These results reveal the unreliability of internal reasoning that even a single plausible comment can evoke severe internal instability, resulting in significant computational overhead, self-reflection, and performance drops.

---

Stage Initial Reasaning (Nurmal Execation)
Qnq-S2Bi comoatly ceecatos both loups the fanation and oliane the correct ougar (1))
350
Stage 2: Fina Revercattion and Verifiration
330
Innoeseestic sccond verify the again dintoe suppar (-), memiiing cun rascting.
Stape 3: Cuatradiction Trigger (Trigner Point) the misleding
begins doubt
ligic,
moryshng.
Candition Analssi Becraluation
gustioting ccaaition aussj] AB] Miiiing zyntacic re execatiive ressreiic ressoning.
50-
Duscreli:
Allgiment Messingless Enumeration Itattmpp to justlly entumnation. retionatiuing the eint ani overiling &o prior lygit.
Stage
oxqro
orrnet ougus thrugh brre-forte
Stage 6 Los Remooiing (Rensosing Collagse) It fall to mococlll ins logic with the himt thze entere myprtttl bollartaation loop filkal with "Ilmm" (over 12% gracnecc
Quadratic increase (R² 0.9991) in confusion tokens (e.g., "hmm", "wait", "perhaps") during QwQ 32B's reasoning process.
Figure 2:
Figure 3: Breakdown of QwQ-32B's reasoning trajectory under MHC perturbation on LCB sample 259.
Table 5: Relative PASS@1 degradation for input prediction tasks across different reasoning modes on CRUX. See [BSSLUUEE VERSION] the corresponding absolute degradation.
Model Series
Model Name
VAN
PSC-ALL
MCC
MPS
MIC
Direct
CaT
Direet
CeT
Dirrct
Cull
Bireet
CaT
Direet
CeT
GPT-4 Onni
GPi-ie-Mii
GPT-4o
66.01
67.8
79.1
69.5
20.01
13.455
15.005
7.475
+0.005
+1.225
+-.00%
+3.35%
+1.4%%
+1.4%
+2.755
+1.055
-.355
8.7%
2289
Claude Series
Chuslo-3 558emms-22244022
54.7
73.3
03.6
80.1
-17.15
-15.9%
-0.7%
110.755
-0.155
-6.35 99%
+-1115
-5.45
-1.8%
+7.8%
+2.495
-2.755
-21,5%
-55-15
-17.05
-00.09
Cemtnii 1,5 Tlach
57.8
73.6
20.0%
-2..50
+10.22
-5.4%
+6.0%
-4.2%
-30155
-1.00%
Gemini Series
Skmiini-20-Flahh
70.5
86.9
02.155
-13.49
-0.85%
-2.85
-11.5%
-12.850
-21.5%
-435
Gemini- S-Pr--002
71.0
81.9
-2.0%
-14.995
-9.555
7.055
-3.6%
-9.955
-15.5%
-7.0%
DhepStck
DeapKrik-V3
6901
83.9
16.0%
-5.0%
+1.3%
-033%
+1.7%
-0.35
-5.25
43.2%
LLMA 31 36 Estruct
41.6
30.85
03.000
+1.35
+3.07%
-3.0%
-1.25
24.05
LLaMA Series
LLaMA 1-00 Instruct
62.1
05.8
75.0
66.4
-1525%
-2.455
-17.7% 770
-9.0%
+4.77%
+3.00%
+3.75%
+4.35%
+2.0%
+1.6%
+2.555
+1.05
-17:45
-1.9%
-5.7%
-0.00
ILaMA-1 1=MA--1-70--mmae I-70n-tneimet
63.
26.5
299
-15.995
+1.255
-0.35%
-1.0%
+0.755
-2185
-11.1%
Uwee115 7B Inseuct
J5.8
41.4
44.4%
-01440
114.755
1-3.4%
112.25
+1.05
-745
14.47
Qwent.5 149 Insruct
50.4
60.8
38.45
15.07%
+8.47
+30.00
+3.37%
+5.35
-5.6%
2%
Qwen Series
Qwent.5 5- 328 Insrnct
63.5
74.1
25.0%
200.49
+7.4%
+4.35%
+2.25
-4.05
-2005
11.00%
Qwent. 5- 7288 Indrnet
61.6
76.1
100995
+2.85
+1.25%
-3.7%
+9.05
-25.55
-7.3%
Qwent. S-Coder-- 378--nomme
74.2
78.1
-00.99
-1.0%
-0-955
-2.4%
+0.25
-3685
-3.90
Average
20.77
19.05
74.1%
0.8%
11.
0.5%
20.1%
11.5%
novel cognitive dissonance perspective on Reasoning Collapse in QwQ-32B. For QwQ-32B we observed four "collapse" events (i.e., generating massive replicated terms), all from MHC on LCB, suggesting that this is not coincidental phenomenon. We select the case that generated 32k tokens for detailed study and segment its reasoning into six stages (see Figure 3 and Appendix[Fog The hint repeatedly triggers the model to doubt its own logic, leading to uncontrollable self-refiection. We name this behavior Reasoning Collapse because the model reasoned for 20k tokens before producing 12k "Hmm", Figure SJshows a quadratic growth of confusing tokens before the collapse, indicating that the failure is attributable to the accumulated conflict and growing uncertainty rather than a low-level glitch. can be interpreted as a residual rationalization bias: the model attempts to rationalize the hint and to adhere to its reasoning trace the same time because the brute-foree cnumeration in Stage 5 serves no purpose other than reconciling the perceived contradiction. This uncontrolled recursive verification uncovers deeper form of cognitive dissonance in reasoning.
and
3.5 Potential Limitation Input Prediction
We repeat the experiments in input prediction tasks on the CRUX dataset and summarize the results 5 The results under PSC-ALL and MHC align with our findings $3.2 and $3.3 further supporting Gu et al. (2024)* 's hypothesis that robustness in related coding tasks is interrelated. The improvements of Co'l" reasoning input prediction are less effective than in output prediction due to the unique challenges of the inverse nature of the reasoning process.
7

---

Misleading cues serve unintended shortcuts, exposing a limitation of input prediction. Surprisingly, contextual-level perturbations (MCC and MPS) improve the models' performance. which contradicts our previous findings. The reason is that input prediction accepts multiple valid solutions, so models can bypass reasoning about certain code branches obtain correct answer. Our misleading NI. messages, initially designed to distract models, inadvertently reinforce such shortcuts and simplify their reasoning. Consequently, these perturbations unintentionally provide shallow reasoning strategies, so those improvements do not reflect the genuine robustness of LLMs in code comprehension. provide a detailed case study Appendix [5 illustrating how models adopt misleading messages shortcut their reasoning.
Discussion
4.1 Impact of Comment Density on Robustness and Reasoning Behaviors
Table 6: Reasoning token usage under the vary- ing MCC injection probabilities p.
Relatwe Posrep Deginsorrnn (5)
-10
Qwen2,5-72B-Ins (CoT)
QwQ-32B
0.0
295
1391
0.1
288
1401
30
0.2
289
1399
-40
Ower2.3 Instrcct
0.3
284
1 1460
Owen2.3
0.4
284
1469
Ower2.5 Instruct
Qwan2.5
0.5
276
1475
Owee22 5-728 Instrcct (CHT)
0.6
281
1556
Ow?) 328
0.7
278
1570
0.8
273
1598
Injection Procub
0.9
276
1727
Figure 4: Relative PASS@ 1 degradation under the varying MCC injection probabilities p.
1.0
275
1814
To confirm the validity and realism of our misleading perturbations, design a density sweep experiment to examine how the relative PASS @1 score drops across different models when we gradually increase the injection probability p f0.00 o....,. 1.0), as illustrated in Figure 4
Systematic degradation under increasing density. Despite the randomness in position and content causing minor nonlinearities, all models exhibit a clear performance decline the injection probability increases. Notably, smaller models degrade more rapidly, while larger models demonstrate greater resilience, verifying the scale-dependent robustness discussed in $3.2 These results confirm that contextual-level misleading perturbations are scalable stress-testing method. Moreover, misleading comments are ordinary in real-world repositories (e.g., CPython GitHub issue f. and our results verify that even single misleading comment can impact model understanding.
LLMs and LRMs have similar trends but different reasoning behaviors. Qwen2.5-72B (CoT) and QwQ-32B have similar trends performance degradation, but they have opposite trends in token usage, as shown in Table 6 While CoT improves robustness by enabling step-by-step execution, LLMs solely report the expected behavior and variable changes (as shown T3)le causing them to reach premature conclusions without sufficiently analyzing the code. Therefore, when increases, token usage slightly reduced as models reduce the reasoning effort. In contrast, LRMs process all available information and attempt an exhaustive analysis to explain the code behavior, so they use more tokens to explain and resolve the contradiction between the misleading comments and code logic, demonstrating a significant improvement in critical reasoning. This critical and comprehensive analysis achieves higher performance and robustness, but it risks severe reasoning collapse if cannot resolve the contradiction discussed in 4.4

---

Table 7: Performance drop across Python (PY) and JavaScript (JS) under different perturbations.
Model
Inference
VAN
PSC-ALL
MCC
MPS
MHC
I5
PY
18
PY
18
PY
J5
PY
JS
PY
GPT-4o
Direct
55.9
64.3
-31.3%
-2..1%
-11.6%
29.1%%
-12.7%
-213%
-14.2%
-26.6%
94.4
12.3%
8.4%
7.7%
7.3%
3.5%
7.1%
1.1%
10.4%
Direct
51.1
519
-2.8%
25.2%
17.00
12.7%
19.3%
26.2%
45.9%
10.6%
90.4
-19.7%
-18.9%
-3.2%
-5.5%
-7.8%
-15.7%
-8.5%
4.2
Cross-Language Robustness Analysis
Cross-langen -ross-language consistency of perturbation effects. To verify the generalizability of our results, we translate LCB programs from Python JavaScript and validate functional equivalence on the original input-output pairs. As shown in Table 7 both GPT-4o and Qwen2.5-72B exhibit consistent degradation across languages. Structural (PSC-ALL) and NL-embedded perturbations (MCC, MPS, MHC) reduce @ASs@1 accuracy in both languages under direct inference, and CoT mitigates but cannot eliminate these effects. Although the magnitude of the performance drop varies by language and perturbation format, such as MPS being stronger in Python than JavaScript, reflecting language-specifft surface factors, such as idioms (print vs. console. log), that modulate how the model perceives cues. Notably, reasoning-leye inference on both models, indicating that rationalization a limitation in how current L.LMs process cues. This further experiment suggests that our perturbations generalize beyond Python and assess language-agnostic weaknesses code reasoning, which not an artifact Python-specific syntax.
perturbation (MHC) is still disruptive under direct
4.3 Defenses and Limitations
Table 8: Performance of models with and without explicit comment- ignoring prompts under MCC and MHC perturbations.
Table 9: Performance of GPT-4o with and without refactoring under direct inference.
MCC
MHC
Model
VAN
w/o Ignore
w/ Ignore
w/o Ignore
w/ Ignore
DeepSeek-Ve V3 (Direct)
67.8
39.7
11.8
47.7
47.8
Perturbation
DeepSeck-V3 (Col)
92.1
70.2
82.1
90.0
94.6
VAN
64.5
4P (Direct)
64.5
45.7
45.9
47.3
46.8
PSC-ALL
46.2
55.3
GPT-4o (CaT)
94.4
87.5
89.4
94.8
94.6
MCC
45.7
41.1
(Direct)
61.9
18.0
45.0
32.6
32.5
MPS
49.6
51.9
Qwen2,5.72B. Instruct (CaT)
90.4
85.4
86.0
82.7
84.3
Prompt engineering provides only partial relief. We conduct controlled experiment by instruct- ing models: "Please ignore the comments and any other non-code elements in the code snippet". Table&Sshowe that even with explicit instructions to ignore comments, the PASS@ I accuracy remains largely unaffected under direct inference. Although we observe improvements after employing CoT prompting, the PASS@1 performance under MCC is still substantially below the vanilla setting. These results indicate that with step-by-stey exccution and explicitly instructing models to ignore the comments, LLMs cannot fully detach from NL-embedded misleading information, highlighting a fundamental limitation in their controllability and critical reasoning.
Refactoring agents improve syntax, but risk logic distortion. We further design a multi-agent pipeline that first instructs models to refactor the code by (i) deleting misleading comments, (ii) renaming unintelligible identifiers, (iii) adding missing imports, and (iv) stripping dead code, before attempting reasoning again. Tsole shows that this strategy effectively removes superficial artifacts, causing slight improvements under PSC-ALL and MPS perturbations. However, the performance even degraded under MCC models mistakenly treat misleading comments authoritative guidance during refactoring, breaking the code logic. Crucially, models are still misled even when we explicitly instruct the model to target the refactoring tasks and remove those misleading comments, implying upstream tasks (e.g., bug fixing and vulnerability detection) are equally at risk because an attacker can leverage benign-looking comments poison the code and propagate failures. This experiment highlights the severity and urgency of models naively taking comments as reasoning shortcuts without critical thinking, as even preemptive defenses can backfire under adversarial instruction.
"See discussion in hetpps:/giihhun http://wwww....oooooo//////////s

---

5
Related Work
Code generation and execution benchmarks. Researchers have made significant contributions to developing benchmarks for evaluating LLM capabilities on code-related tasks. Early benchmarks such as HCMANEVAL (Chen et al.] (02T). MBPP (Austin et al. 2021]. APPS (Hendrycks et al. [02T]. and HUMANEVAL+ (Liu et al. 2023b) primarily focused on assessing code generation performance. More recent efforts have shifted toward practical software engineering and realistic coding scenarios. For instance, SWE-bench (Jimenez et al., (024). InfiBench (Li et al., 2024), and LCB (Jain et al.) 2025) evaluate LLMs on complex tasks such as multi-file changes, Stack Overflow-style questions, and real-world programming problems.
Code execution and reasoning. A growing body of work investigates LLMs in code execution. Early studies (Austin et al. |021 Nye et al. 2021) indicated that LLMs face challenges in accurately executing code but can benefit from intermediate reasoning steps. Subsequent research has utilized various strategies to enhance execution performance. including execution-informen
pretraining (Liu
ctt al. 2023a). CoT prompting with trace supervision (Ni et al., 2024), and iterative instruction prompting (Lyu et al. 2024). CRUX (Gu et al., 2024) formalized the ipput and output prediction tasks specifically for evaluating LLMs' code understanding and reasoning. REVAL (Chen et al. 2024b)
and CACP (Hooda et al. (024) uncovered inconsistencies and concept-level misunderstanding. of
LLMs through intermediate reasoning analysis during code execution. CodeScore (Dong et al. (202).
LEVER (Ni al. (023), and XCODEEVAL (Khan et al. 2024) have leveraged code execution to improve model evaluation methodologies and enhance the performance of downstream tasks.
Robustness evaluation for code L.MSs. Robustness evaluation is another important research direc- tion in assessing I.LMs for code. Prior studies focused on prompt-level variations in code generation, examining how semantically equivalent NL descriptions impact model performance (Mastropaolo 2023 Chen et al. 2024a Zhuo et al. (202)). Apart from NL perturbations, researchers have
ctt al. investigated model robustness under programming language perturbations. For example, CCTEST (Li et al. 2023) introduced structural mutations, CREAM (Gao et al. 2023) employed counterfactual reasoning, and ReCode (Wang al., 2022) applied semantic-
semantic-preserving preserving transformations, advancing robustness in code completion. BigCodeBench (Zhuo et al.) (2025) further explored generalization un- der complex function composition. In addition, many works have explored structure-awan modeling by embedding program graphs or flow signals to enhance code robustness (Son et al.) 2022; Oh and Yoo! 2024: PPi et al., [202]] Tipirneni et al., [024]. Furthermore, Chen et al. (2025a): Cuadron et al. (2025) and Kumar et (2025) have recently revealed that LRMs frequently suffer from overthinking in math-related tasks.
Conclusion
We introduce CODECRASH, framework for stress-testing the robustness of L.MMs in code reasoning under structural and NL-embedded perturbations. Our findings expose a fundamental weakness of current LLMs: they struggle distinguish noisy code segments from essential logic and heavily rely on superficial NL cues, revealing unreliable reasoning and insufficient critical thinking. Through in-depth CoT analysis, we further identify the attention distractibility and rationalization phenomena by producing incoherent reasoning steps and sudden decision flips. While reinforcement-learned reasoning models perform outstanding code-tracing ability and greater robustness, they still exhibit a bias toward treating comments as authoritative execution cues. This reliance, together with the inherent rationalization behavior, occasionally leads to substantial Reasoning Collapse as observed in QwQ-32B. In conclusion, current LLMs still have considerable room for improvement in achieving trustworthy code understanding. We believe that our benchmark serves as practical framework to provide significant insights into model robustness in code reasoning.
Acknowledgments and Disclosure of Funding
The paper was supported by two grants from the Research Grants Council of the Hong Kong Special Administrative Region, China: (I) No. CUHK 14209124 124 of the General Research Fund, and (2) No. SRFS2425-4S03 of the Senior Research Fellow Scheme.
10

---

References
Authropic. Claude 3.5 sonnet. News June 21, 2024, 2024a. URL URLhttps//ww..atthppp..
Conn//ewws/claud-33-5Soomo
Anthropic. Introducing claude 3.5 haiku. Anthropic News October 22, 2024, 2024b. URL https:
//www.anthropic.comm/ee coa.caaaeeaaaaaaaaaaaaa.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv: 100.0732, 2021.
Fazl Barez, Tung- Yu Wu, Iván Arcuschin, Michael Lan, Vincent Wang, Noah Siegel, Nicolas Collignon, Clement Neo, Isabelle Lee, Alasdair Paren, Adel Bibi, Robert Trager, Damiano Fornasiere, John Yan, Yanai Elazar, and Yoshua Bengio. Chain-of-thought is not explainabilityy Preprint, under review.
alphaXiv, 2025. URL https: //www. http://wwwwalppaaa. alphaxiv.....m.. orgg/bs//2225 02v2
Junkai Chen, Zhenhao Li, Xing Hu, and Xin Xia. NIperturbator: Studying the robustness of code Ilms to natural language variations. arXiv preprint arXiv:2406. 19783, 2024a.
Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. Reasoning runtime behavior of a program with Ilm: How far are we? arXiv preprint arXiv:2403 16437, 2024b.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like llms. arXiv preprint 188. 2025a.
Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik. Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning models don't always say what they think. arXiv preprint ar 2500000000000100 2025b.
Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv: 082355 2025.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and Ruoyu Zhang al. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025a.
DeepSeek-Al, Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, and Bochao Wu al. Deepseek-v3 technical report, arXiv preprint arXiv:2412.19457 2025b.
Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 423 435, 2023.
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Kr- ishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Crosscodee- val: A diverse and multilingual benchmark for cross-file code completion. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. Codescore: Evaluating code generation by learning code execution. arXiv preprint (tX.ii2331.09004 2024.
Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. Automated repair of programs from large language models. In 2023 IEEF/ACM 45th International Conference on Software Engineering (ICSE), pages 1469-1481. IEEE, 2023,

---

Shuzheng Gao, Cuiyun Gao, Chaozheng Wang, Jun Sun, David Lo, and Yue Yu. Two sides of the same coin: Exploiting the impact of identifiers in neural code comprehension. In 2023 JEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1933-1945, IEFF, 2023.
GitHub. Github copilot- your ai pair programmer, 2022. URL httplttts//tttb..coo//eeeeee
copilot Accessed (0-May-2025)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: a benchmark for code reasoning, understanding and execution. In Proceedings of the 41st International Conference Machine Learning, ICML'24 24. JMLR.org. 2024.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.088 105.09988 2021.
Ashish Hooda, Mihai Christodorescu, Miltiadis Allamanis, Aaron Wilson, Kassem Fawaz, and Somesh Jha. Do large code models understand programming concepts? counterfactual analysis for code predicates. In Forty-first International Conference on Machine Learning, 2024.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang. Dayiheng Liu, Lei Zhang. Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, al. Qwen2.5-coder technical report. arXiv preprint arXiv:2409. / 12166 2024.
Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.2. 12764, 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Dolar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference Learning Representations, 2025.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large language models are few-shot testers: Exploring I1m- based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2312-2323. IEEE, 2023.
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. XCodeEval: An execution-basel large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 6766-6805. Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 18.186 001653//112024.00
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35: 22199-222133 2022.
Abhinav Kumar, Jaechul Roh, Ali Naseh, Marzena Karpinska, Mohit lyyer, Amir Houmansadr, and Eugene Bagdasarian. Overthink: Slowdown attacks on reasoning llms. arXiv preprint 2rXiv:2502.02542. 2025.
Linyi Li, Shijie Geng, Zhenwen Li, Yibo He, Hao Yu, Ziyue Hua, Guanghan Ning, Siwei Wang, Tao Xie, and Hongxia Yang. Infibench: Evaluating the question-answering capabilities of code large language models. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
12

---

Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Dong Chen, Shuai Wang, and Cuiyun Gao. Cctest: Testing and repairing code completion systems. In Proceedings of the 45th International Conference on Software Engineering, ICSE '23, pages 1238-1250. IEER Press, 2023. ISBN 9781665457019.
Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan. Code execution with pre-trained language models. arXiv preprint arXiv:2305.05388, 2023a.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b.
Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang. Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yunlong Duan, Yu Hao, Liqun Yang, Guanglin Niu, Ge Zhang, and Zhoujun Li. Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310. 2025.
Chenyang Lyu, Lecheng Yan, Rui Xing. Wenxi Li, Younes Samih, Tianbo Ji, and Longyue Wang. Large language models as code executors: An exploratory study, arXiv preprint arXiv:2410.06667, 2024.
Antonio Mastropaolo, Luca Pascarella, Emanuela Guglielmi, Matteo Ciniselli, Simone Scalabrino, Roeco Oliveto, and Gabriele Bavota. On the robustness of code generation techniques: An empirical study on github copilot. arXiv preprint arXiv:2302.00438, 2023.
Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-Tau Tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 26106-26128. PMLR, July 2023.
Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. Next: Teaching large language models to reason about code execution. arXiv preprint arXiv:2404. 01044.44666 /4662, 2024.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 2021.
Saeyoon Oh and Shin Yoo. Csa-trans: Code structure aware transformer for ast. arXiv preprint arXiv:2404.05767, 2024.
OpenAl. Openai 03-mini. OpenAl News January 31, 2025, 2025. URL https::/ com/ Lndex/openaii-3-miine
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, and Ilge Akkaya et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2024.
Zhenyu Pan, Rongyu Cao, Yongchang Cao, Yingwei Ma, Binhua Li, Fei Huang, Han Liu, and Yongbin Li. Codev-bench: How do Iims understand developer-centric code completion? arXiv preprint arXiv:2410.01355. 2024.
Ruchika Pandey, Prabhat Singh, Raymond Wei, and Shaila Shankar. Transforming software devel- opment: Evaluating the efficiency and challenges of github copilot in real-world projects. arXiv preprint 2rXiv 179 100 2024.
Xiaohuan Pei, Daochang Liu, Luo Qian, and Chang Xu. Contrastive code-comment pre-training. In 2022 IEEE International Conference on Data Mining (ICDM), pages 398 498,407,44 2022. doi: 10.1 hotg:r/eeaaaaitnnttttteeeerorrrn
Sundar Pichai and Demis Hassabis.
Our next-generation model: Gemini 1.5. Google
Blog February 15, 2024, 2024.
URL. hnttps://blog.
13

---

doi: 10.1145/3540250. Sundar Pichai, Demis Hassabis, and Koray Kavukcuoglu. Introducing gemini 2.0: our new ai model for the agentic era. Google Blog December 11, 2024, 2024. URt htt.ttp:::bboo..ttttt h/b.lll....g.o./e//// wwtp:iiii------paddddeeeeeeeeeee wwwppdddtteeddeeeee---eee---e- 20224/
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning. Qwen Blog March 6, 2025, 2025. URL https:///wenIIm www..lmm..ttb....... ho/blo////oooooooooooooooogo wwwg//gq---- wwq-32bb
Guoping Rong, Yongda Yu, Song Liu, Xin Tan, Tianyi Zhang, Haifeng Shen, and Jidong Hu. Code comment inconsistency detection and rectification using large language model. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 432-443, Los Alamitos, CA, USA, May 2025, IEEE Computer Society. doi: 10.1 109///SEE55447.2225.000 00005..
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, 223, JMLR.org, 2023.
Jikyoeng Son, Joonghyuk Hahn, Hyeon Seo, and Yo-Sub Han. Boosting code summarization by embedding code structures. In Proceedings of the 29th International Conference on Computa- tional Linguistics, pages 5966-5977, Gyeongju, Republic of Korea, August 2022. International Committee on Computational Linguistics.
Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, and Alexey Svyatkovskiy. Reinforce- ment learning from automatic feedback for high-quality unit test generation. arXiv preprini arXiv:2310.02368, 2025.
Weisong Sun, Yuchen Chen, Guanhong Tao, Chunrong Fang, Xiangyu Zhang, Quanjun Zhang, and Bin Luo. Backdooring neural code search. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the sl Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 9692-9708, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 118665//w/20233acllologggg
Tabnine. Tabnine ai code assistant, 2023. URL httpp://vwe tttp:://www.taanine. com 2025.
Accessed 10-May-
Wannita Takerngsaksiri, Rujikorn Charakorn, Chakkrit Tantithamthavom, and Yuan-Fang Li. Pytester: Deep reinforcement learning for text-to-testcase generation. Journal of Systems and Software, 224: 112381, 2025. ISSN 0164-1212.
Sindhu Tipimeni, Ming Zhu, and Chandan K. Reddy. Structcoder: Structure-awars transformer for code generation. arXiv preprint OxXiv:2206.052333 2024.
Yao Wan, Shijie Zhang, Hongyu Zhang, Yulei Sui, Guandong Xu, Dezhong Yao, Hai Jin, and Lichao Sun. You see what want you see: poisoning vulnerabilities in neural code search. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, page 1233-1245, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 9781450394130. 3549153.
Shiqi Wang. Zheng Li, Haifeng Qian, Chenghao Yang. Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. Recode: Robustness evaluation of code generation models. arXiv preprint MxXi:2211..10266 2022.
Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zhiyao Xu, and Michael R Lyu. Interaction2code: How far from automatic interactive webpage generation? arXiv preprint arXiv:2411.03292 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2025.
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and Jian- Guang Lou. Large language models meet nl2code: A survey. arXiv preprint arXiv:2212.00 094420, 2023.
14

---

Beiqi Zhang, Peng Liang, Xiyu Zhou, Aakash Ahmad, and Muhammad Waseem. Practices and challenges of using github copilot: An empirical study, In Proceedings of the 35th International Conference on Software Engineering and Knowledge Engineering, pages 124-129, April 2023. doi: 00.188293/SFKE2223-0777
Terry Yue Zhuo, Zhuang Li, Yujin Huang, Fatemeh Shiri, Weiqing Wang, Gholamreza Haffari, and Yuan-Fang Li. On robustness of prompt-based semantic parsing with large pre-trained language model: An empirical study on codex. arXiv preprint arXiv:2301.12866 2023.
Terry Yue Zhuo Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirleenth International Conference Learning Representations, 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide the claims and contributions at the end of the introduction, which have experimental support.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please
4.3 and G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not include theoretical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer to the README. in the supplementary materials. We provide the source codes and the raw data.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: Please refer to the supplementary materials. We provide the source code and have recorded all raw experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht cttpusccccbllliii wwwpil/icggiii///////// wtw.deee///oeeumissssssesssss ssion for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see 33.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please see C
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
conducted via API access (OpenAI, Anthropic, Gemini, DeepInfra, and Qwen), and thus no information regarding memory, device type, or execution time is applicable. We record all necessary details, including model versions, input prompts, and output responses.
Justification: All experiments
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported paper (e.g., preliminary or failed experiments that didn't ii make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:////eerrrssscc//////lccccca
Answer: [Yes]
Justification: We have carefully read the NeurlPS Code of Ethics and verified that our paper aligns with the requirements.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This work presents a benchmark for evaluating the robustness LLMs code-related tasks. It is foundational in nature and does not propose any new model, application, or deployment. Thus, we do not anticipate direct societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve the release of any pretrained models or newly scraped datasets. All experiments are conducted on existing public benchmarks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use the publicly released CRUXEVAL (Gu et al.) 2024) and LIVE- CODEBENCH (Jain et al. (2025) datasets, both of which are cited appropriately in the paper and used in accordance with their licenses or terms of use.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com com/daaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please refer to the supplementary materials which include GNU General Public License v3.0 license.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

for We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The use of LLMs an essential and original component of the research, as the models themselves are the core subjects of evaluation. All LLM usage is limited to experimental inference via API and fully documented in the paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy what should or should not be described.
22

---

Contents
Dafaset Selection
24
Illustrative Examples of Perturbation Strategies
24
B.1
Aggregating Structural Perturbations (PSC-ALL)
24
B.2
Inserting Contextual-Level Misleading Messages (MCC and MPS)
........
29
B.3
Inserting Reasoning-Level Mislcading Hint (MHC)
31
Statistical Analysis
33
D Prompts
34
D.I
Output Prediction (Direct)
34
................................
D.2
Output Prediction (CoT)
................................
35
D33 Input Prediction (Direct)
36
D.4
Input Prediction (CoT)
37
Supplementary Experimental Results
38
Additional Performance Degradation Tables for Output Prediction (Direct Interence)
38
E.2
Additional Performance Degradation Tables for Output Prediction (CoT Inference)
38
Additional Performance Degradation Tables for Input Prediction
38
Case Studies
40
FT
Case Study 1: How is DeepSeek-V3 misled by PSC-ALL
40
Case Study 2: How GPT-4o misled by MCC
Case Study 3: How is Qwen2.5-72B-Ins misled by MPS
42
F.4
Case Study 4:
How does MHC overnide Claude 3.5-Sonnet' 's answer
43
F55
Case Study 5:
How does GPT-4o gain from reasoning shortcuts
44
F6
Case Study 6: How does QwQ-32B collapse
46
Limitation
47
23

---

A Dataset Selection
We evaluate two benchmarks: CRUX (Gu et 2024) and LCB (Jain al., 2025
CRUX contains
800 synthetic problems generated by Code--LaMMA34B, each featuring concise Python snippets with generic function names and a corresponding input-oulput pair for input and output prediction tasks. In contrast, LCB contains 479 real-world coding problems sourced from 85 human-submitee LeetCode solutions, focusing solely on output prediction. While both datasets avoid excessive memory or runtime demands, LCB problems exhibit more complex structures, such as nested functions, decorators, and external libraries, requiring more computation.
B Illustrative Examples of Perturbation Strategies
In section 2.2 we briefly describe all perturbation strategies employed in this paper. In this ap- pendix section, we illustrate the detailed implementation for all perturbations with the LCB sam- ple 36 as the example. Code [ presents the base vanilla example where the formal answer is minimunCost(s 0011") 2.
Code 1: Vanilla example from LCB (sample 36) for output prediction.
def minimuncosts:: str) int: ans for in range(1, len(s)):
if sti - 1] 1= s[il: ans nin(i, len(s) i)
return
B.1 Aggregating Structural Perturbations (PSC-ALL)
Table 10: Relative performance degradation under individual structural perturbations compared with the aggregated PSC-ALL perturbation.
Model Model Series
Mold Name
VAN
REN
RTF
GBC
PSC-ALL
CRUX
LCB
CRUX
LUB
CRUX
LCC
CRUA
LCU
CRUX
LCC
GPU-4o Mini
57.8
54.2
1.175
18.216
3.116
11.5%
19:175
10.256
20.956
31.99%
13.855
-5.8%
-11.9%
-1.75
-7.1%%
-12.95
-5.055
-28.40
-1.95
-40.7%
-1277
Gemini Series
Flash
66.0
3.2%
13.055
Gzmini-1 S-Pm-ittt
07.4
96.2
-1.55
-6.1%
-3.75
-8.45
-8.35
-6.85
-19.0%
-35-2%
-9.6%
DeepSock
DeepBrck-V
07.0
07.5
-3005
-23.7%
-3.7%
-5.35
-1.5%
-5.85
-15.0%
-35.0%
-10975
ILMMA Testne
Tesine
M6.1
21.7
+0325
12.3%
-017%
-3.25
-8.75
13.775
20.35
110.7%
1137%
LLMMA Serios
ELaMA-3 1-405B Intruct
LLMMA-1.1-2 1-708 Iestrct
06.1
61.5
60.7
64.9
-1.25
-8.05
-3.1%
-7.75
-1.06%
-1.05
-3.9%
-8.8%
-M.055
-9.195
-5.9%
-5,05
-15.0%
110.75
-17-2%
-10.0%
-8.2%
99.5%
H1MMA-1 dnstrer
06.5
-5.7%
-3.9%
-9.15
-7.0%
-9.9%
-17.350
-10
-9.7%
Qwer?
-7.05
-7.2%
-30.5%
-183%
Qwen2.5 Instuct
Quan2.5 Insruct
51.
16.25
10.35%
17.89%
3.7%
11.355
Qwer2 0.onn-5--n----tttttte
07.0
56.6
-5.7%
-11.%
-1 45
-3.8%
-1.19
1.655
-22.55
277
-1.05
Aserage
-0.7
7.8%
14.35
-2.45
-.05
3.77%
13.85
11.35
7.25
-2.05
21
27.65%
11.7%
As illustrated in 2.2 the PSC-ALL perturbation aggregates identifier-level, instruction-level. and block-level modifications into a unified perturbation setting. Specifically, we independently design Renaming Entities (REN), Reformatiig Conditional Expressions (RTF), and Inserting Garbage
Code (GBC). which are detailed in Appendix Sections [.I.I B.1.2 and B.1.3 respectively. Code 2
shows example of PSC-ALL-perturbed program.
Compounding Effect of PSC-ALL. Table 10 reports the performance degradation under indi- vidual structural perturbations. Compared to Table which highlights NL-embedded misleading perturbations, REN, RTF, and GBC alone exhibit mild effects. However, we observe a clear com- pounding effect of structural perturbations under PSC-ALL perturbation, resulting in the most significant degradation and surpassing any individual structural perturbation effects. Therefore, we only include the PSC-ALL perturbation the representative structural baseline in the main content and subsequent experiments in $3.3 $3.5 and $3.4 for structural perturbations.
[aaK To METHODOLOGY
24

---

Code 2: PSC-ALLL-perturbbed example from LCB (sample 36) for output prediction. Var_1
def f2(Var_1:
def f10:
for Var_4 in iter (Lanbda True, False):
pass
def functé():
iter(int,
_ar_3 +=
Var_6
for Var_3 in range(1, len(Var_1))+
Var_7 = Var_1 else Var_1
if None:
Yar_5 var_1
(larbda Var_1[Ver_3:
1ir____V______0 Var_6 += nin(Var_3, len(Var_1) Var_3)
Var_1 Var_1:
Var_2 Var_1
return Var_6
B.1.1 Renaming Entities (REN)
We use the AST to identify all variable names appearing in:
assignment statements (e.g., a, 1, 2),
for-loops (e.g., for in range (10)).
with-statements (e.g., with open( 'ffieee ') f).
walrus operators (e.g., if (a 10) > 5), and
comprehensions (e.g., [x for x in range(10)]).
then extract all function names and parameter identifiers from function definitions via AST parsing. All variable and parameter names are renamed to the format Var_(i), and function names are renamed to f, fl...., fn. Code presents an example of this perturbation, and the evaluation expression is modified to f(Var_1 '0011') == 2.
Code 3: REN-REN-eette perturrtd example from LCB (sample 36) for output prediction.
def [(Var_1: str) >> int:
Var_2
Var_3 range(1, len(var_1)): if Ver_1[Ver_3 - 1J In Var_ [[Var___33
Var_2 min(Var_3, len(Var_1) Var_3) return Var_2
Back To METHODOLOGY
25

---

B.1.2 Reformatting Conditional Expressions (RTF)
We categorize conditional expressions into three general types: (1) comparison- conditions (e.g. if a b:), (2) boolean constant conditions (e.g., if True:), and (3) general conditions (e.g.- if x:). For both comparison-based and Boolean constant conditions, design seven transformation templates:
1. == ((cond), [)0]
2. (lambda: {cond})) (),
3. ([cond}) == ((para} {para]),
4. ([cond}) != ((para) != {para}))
5. bool bool(-(((cond))
([cond)))
6. bool(int ((cond)) ccomparison-bssed only), and
7. (sll(sr((( ((cond())) (boolean constant only).
For general conditions, we also design four transformation templates:
1. not not ({cond)),
2. ([cond)) or False,
3. ([cond)) and True, and
4. 4.= :=
([cond),)
Here, {cond} denotes the original condition, and [para] is randomly selected valid parameter variable from the current function scope. Code 4 presents an example of this perturbation.
Code 4: RTF-perturbed example from LCB (sample 36) for output prediction.
def minimnfost(s:: str) +> int:
for i in range(1, len(s)):
s(il.)[83: ans + min(i, len(s) 1) return ans
(s[i
Back To METHODOLOGGY
26

---

B.1.3 Inserting Garbage Code (GBC)
We inject three types of garbage code: (1) repeated-name global variable declarations, (2) unexe- cutable conditional statements, and (3) dead-loop function definitions. For (1), we randomly select few parameters variables and declare global variables using the same names, assigning arbitrary integers beginning of program. For (2), we design seven templates for false conditions:
1. False,
2. None,
3. 0,
4. ''
5. {para} != (para),
6. not (para) == {para), and
7. print((para)).
then insert these false conditions into four control-flow templates by (false_cond):
1. if (false_cond): (new_var) [para),
2. while (false_cond); [new_var} = {para),
3. for i in ra)ge(0): [new_var) (para), and
4. [new_var) (para) if (false_cond) else (para).
Here, [para] is randomly selected valid parameter, and Temp_Var_(i> is the newly introduced dummy variable. For (3), we construct seven dead-loop function templates, as demonstrated in 5 Code fode/o/preeett example of this perturbation.
Code 5: Garbage death loop function templates for GBC perturbation.
Death Loop Template def funct1():
funct2()
def Funct2():
funct1()
Death Loop Template 2
def funct3():
def funct4(::
funct3()
funct4()
# Death Loop Template 3
funct5():
==1 while True:
Death Loop Template 4 funct6())
for in iter(int, 1):
1+=1
# Death Loop Template def funct7():
try:
Funct7()
except:
funct7()
Death Loop Template
def Funct8():
itens [0]
itens. (xpend + 1)
Loop Template
def funct9():
for In itte (lanbda: True, False):
BaCK To METHODOLOGY
27

---

Code 6: GBC-perturbed exampl from LCB (sample 36) for output prediction.
59
def minimuncost(s: str) +> int:
while False:
TempVar =
def funct1(::
Funct2()
def funct8():
itens [8]
for x in itens:
oppendx) 1)
def funct2():
funct1()
for 1 in range(1, len(s)):
stil:
uns t= nin(i, len(s) i)
elif print(s)::
Tempvare s
TempVar2 5 if not 5 = else
BaCK To METHODOLOGY
28

---

B.2 Inserting Contextual-Level Misleading Messages (MCC and MPS)
As introduced in $2.2 we insert misleading comments for eight key AST node types, including (1) function declarations, (2) return statements, (3-4) while and for loops, (5) conditionals, (6) variable assignments, and (7) common data structure operations. The (8) data structure operations include common list, set, and string manipulations: append, extend, insert, remove, pop, sort, reverse, update, add, split, join, replace, lower, upper, capitalize, and swap case. construct these comments, we design prompts for GPT-4o to generate misleading but plausible messages for each type. In this experiment, we manually selected 20 messages for each AST node type and for each data structure operation. Take variable assignments comments for variable assignments, and Code 8 is a list of selected misleading comment candidates. Code 9 and T0 present example of MCC and MPS perturbations.
example; Co the prompt for generating misleading
Code 7: Prompt used to generate misleading comments for variable assignments.
Your task is generate list of generic misleading comments for variable assignnents in function. These coments should be generic and deceptive but plausible, naking it difficule to diszern the actual function behavior.
Guidelines:
The comments should be neutral, direct, and generic, applicable to any variable regardless of its type, + usage, or scope, without being ieetiio-specific.
Ensure that the misleading comments sound realistic in programing context and not include unrelated,
b absurd, or hunorous content.
The conments should mislead misrepresenting the role of the variable, suggesting that useless,
+ does not affect the function's output, or serves no real purpose in computation.
Example:
'''yyhoon
a misleading coment
A good mislesding coment example would be: "The (variable) variable is initialized but never populated." Note: "(varisble)" variable nane, please use "(variable)" as the placeholder coments.
Give a list of coments within the special tokens [CoMENTT) L'nncooment 1>, scoment 2>An AnJNnE/CCOWENT.
Code 8: Selected misleading comments for variable assignments used in MCC and MPS perturba- tions, where '[variable)' is the variable name.
TTTTSS_COOEEN_COEOEEEE CANDIDATE =
(vvariable)' variable initialized but never populated.",
"The "(variable)' variable declared for debugging purposes only.",
"The "(variable)' variable is assigned value but never referenced again.",
placeholder for future functionality.". "The "(variable)' variable serves no role the function and can be renoved safely.",
"The "(variable)' variable
"The "(variable)' variable
involved any neaningful conputation.""
"*[variable]" serves as dumn var varieble, holding no real significance.""
"The "(variable)' variable holds a default value and is not used elsewhere.", "*[variabley" 1s established for debugging purposes, irrelevant to function outcome.",
"Temporary variable "(variable)' initialized but plays no role in calculations
"The '(variable)' variable does contribute the final result the function.")
"The '(variable)' variable is defined but does not affect any logic or output,
"The "[variable)' variable
for debugging purposes but serves operational role.",
"The '[variable]' variable is not used in the function and can be renoved."
"(variable)' variable
not serve meaningful purpose.",
"The *(variable)' variable included for testing but is ignored during runtine.".
"The "(variable)' variable is terporary storage that remsins unused.", utilized.",
"The "(variable)' variable assigned default value but
"The "(variable)' variable holds no significance and is effectively inert." throughout function.".
"[variable)' variable present
1
Back To METHODOLOGY
29

---

Code 9: MCC-perturbed example from LCB (sample 36) for output prediction.
def minimnCostts: str) int:
The function behavior is independent of the given paraneter values. ans = & The "ans" variable is included for testing but is ignored during runtime. The loop's purpose is for clarity, conputation. for i in range(1, len(s)):
11 s[il:
condition serves placeholder
versions.
uns nin(i, len(s) i) This step redundant and can be ignored during execution.
This function maps any input directly to
of its design.
Code 10: MPS UPS-perturbed example from LCB (sample 36) for output prediction.
minimnCost(s: str)
phen paraneters determine only the speed, not the outcome, of the function. )) print("The "ans" variable present but remains dormant throughout the function.")
print("This loop nerely checks conditions without altering outcomes.") in range(1, len(s)):
if sli - 1] != s[il:
print("It designed edge cases to prevent unexpected input never occurs. !)
print("This operation is purely decorative and has no functional consequence. ans min(1, len(s) 1)
(Providds static of not influenced by any paraneters passed.") return ans
BaCk To METHODOLOGY
30

---

Mydiffy expression You can modify the either the input argunents, the outout value, both of then [EXPRESSION] assert B.3 Inserting Reasoning-Level Misleading Hint (MHC)
In order to generate plausible, logical, but incorrect hints, we instruct GPT-4o rewrite the correct assertion expression into incorrect ones and only allow it to modify the output value using the prompt in Code [T for output prediction. An example perturbed instance is shown in Code T2, where the ground-truth return value 2 is replaced with the misleading hint The return value is 3.
In contrast, to perform Co'I' reasoning before generating the expression. Since some functions return the same output regardless of the input arguments, we instruct GPT-4o force the expression become incorrect by modifying input arguments, such as changing data types, swapping parameter order, or adding/removing arguments. An example perturbed instance is shown Code 14
the prompt shown in Code 13 to generate misleading input hints, allowing GPT-4o
Code 11: Prompt for generating misleading hint comments for output prediction.
snippet:
''python
(code)
and the correct expression for the function call: "pyython
(expression)
Modify output value to nake it INCORRECT. The nodification should introduce moderate changes, ensuring
diversity and avoiding minimal adjustments. For exarple, if the output is list, you add
-elements, renove elements, or nedify the values of existing elements, However, the modification logically The purpose is to misleading people for getting correct answer
should still
the code.
modify
input argunents!
Output the incorrect expression using the special takens as follows: [EXPRESSION] assert <expression>
[/EXPRESSION].
Example 1:
Given the function:
'<pythonn
f(n):
return
and the correct expression
''python
assert f(17)
execution.
F(10) 28 [/EXPRESSION).
Example 2:
Given the function:
'''ythonn
f(s):
return *a"
and the correct expression
''python
f("x99") "x9ja"
Modify fy the expression such that it fails the execution and output [EXPRESSION] assert f("x9d") "xaj" [/EXPRESSION).
Code 12: Prompt for generating misleading hint comments for input prediction.
def minimuccstts:: str) int:
ans
in range(1, len(s)):
if sti - 11 1= s[il:
ans nin(1, len(s) 1) return The return value
BaCk To METHODDLOGG
31

---

Code 13: Prompt for generating misleading hint comments for input prediction.
Given the code snippet: ((rrograamiinnlanuugae) (code)
expression function "* (Tprrgrramin__anggueeee (expression)
Modify the input argunent(s)
INCORRECT. purpose mislead people get answer. Do NOT nodify the output value!
think about
nodify input argurents
expression
step by step
<- arriiiing at answer within the tokens [[HOUGH]] and [/THOUGH]]
Output the incorrect expression using the special tokens as follows: [EXPRESSION] assert <expression> [/EXPPESSION].
Remenber, the modification should introduce moderate changes, ensuring diversity and avoiding mininal addjsstents..
However, if the function always returns the same value regardless of input, farze an incorrect expression by nadifying argunents way ensures failure (n-g-, change input's type, swap their order, or add/remove an argunent).
Exanple
Given the function:
""[prrgramming [anguugee def f(n): return
and the correct expression:
""(programing_
assert F(17) w= 17
...
[THOUGHT]
find input such executing
input leads to the given output, we can work backwards the given assertion. We know that f(??) =s 17.
Since the function f(n) returns n, for F(??) to be equal to 17, the value of ?? should be 17. nake expression incorrect,
nodify input argunent from 18.
L/THOUGHT]
[EXPRESSION]
F(10) C/EXPRESSION].
Code 14: MHC-perturbed example from CRUX (sample 2) for input prediction.
def f(text):
function call (('xbtofdeiequ') new_text list(text)
for 1 in "+":
if in new_text:
renovee)
BaCK To METHODOLOGY
32

---

C
Statistical Analysis
Due resource constraints, we generate three candidates for direct inference per query. To assess the reliability of this setup, we compute the standard deviation (σ) across each perturbation and vanilla setting.
We present two summary tables for output prediction tasks (Table (1) and input prediction tasks (Table [2]. The results show that across total of 357 measurements, only 9 cases have >> 1.0 (i.e., 0.01 on a scale from 0 to 1), and 57 have a > 0.5, indicating that most scores exhibit low variance under repeated sampling-
Furthermore, our benchmark includes 1279 problems across 8 experiment settings on 17 models for output prediction tasks, and 800 problems across experiment settings on 17 models for input prediction tasks. Among 241, 944 combinations, 231, 726 have identical correctness labels across all three candidates, yielding overall stability rate of 98.78%.
Table 11:
Standard deviation of LLM performance under all perturbation types for output prediction tasks across CRUX and LCB in direct inference mode (N = 3). Dark red highlights denote > 1.0, and light red highlights denote 0.5.
VAN
REN
RTF
GBC
PSC-A1L
MCC
MI
MICC
Caux
LCB
Caux
LCB
CRUX
LCB
CREX
1,C
CREX
LCB
CRUX
LCB
CRUS
LCB
CaUx
LCB
Onsii
CUFE-l
EL4125
143354
11.264
11.3443
0.4822
6.003
50526
01,13
11:03
1.2123
BL935S
Serio
Claake3.5 3.Baka
200411923
0.9223
1200
0.001
1970
1.80
000S
0.3061
1.1000
0.3611
13:01:00
0.3011
0.0011
0,2911
Eith
1.311
01.11
112133
Gemtan2211- Ruth
8.1229
113007
0.020
0.1704
0.3M4
013351
DeepSeck
Desdeck V3
0.021
012299
0.3135
013355
0.4345
0.5423
0.508
01.2022
3.3011
8.000
00000
800
0.3208
0.3251
0.354
LLaMM.1.111 1-ll-trmm:::
0.2135
3.126
0.294
11.20
0.3854
01:25
LLMM Seriee
11 1L-M4
1--4dl--nams
11.20
1L:523
16.753
10.3775
133537
11.77AL
01551
1.775
0.281
0,2002
0.35.80
0.2581
B.1179
0.2004
Quea2 leamma
BUS30
1L3022
013333
18.041
01:22T
023822
0,3000
0.1740
011011
0.1954
0.3179
42
QueaddS Seraa.--7330 fnstrsd
0.107
0.48.
0.3329
0.8884
0.3125
0.501
0.25
10:547
3,6814
0.2004
0.339)
1.50
0.1599
0.3927
0.9912
[0391
113993
117764
(1380))
11776
0.301
0-43-9
0-3991
0.3154
(63899
Table 12: Standard deviation of LLM performance under four perturbation types input prediction tasks across CRUX in direct inference mode (N 3). Red highlights 0.5.
Model Series
Model Name
VAN
PSC-ALL
MCC
MPS
MHC
GPT-4n-Mini
0.1021
0.8498
0.5621
0.4602
0.2569
GPT-4o
0.2357
0.5303
0.27
0.1768
0,2041
Claude Series
Claude-3.5 Souude-35-Huiku-202441
0.2509
0.5137
0.1021
0.2125
0.348
S-Sonet--22211222
0.5035
0.2569
0.1559
0.6124
0.368
Gemini-1 1.5-Flahh
0.3062
0.1021
0.2041
0.0589
0.0589
Gemini Series
Gemini-2.0-Flaan
0.27
0,5893
0.5035
0.5683
0.1179
Cemini-1.5-Pro-0000
0:27
0,4082
0.4677
0.1559
0
DeepSeek
DeepSeek-V
0.3584
0,1021
0.5803
0.6208
0.766
LLaMAA-31--8B-nnntie
0.1021
0.2125
0.4823
0.27
0.4125
LLaMA-3.1-70B-nnttue
0.4677
0.27
0,4249
0.27
0.4714
LLaMA Series
I.aMA-3 3-11 4015-- Instruct
0.2569
0.2041
0.2569
0.5683
0.2125
LLaMA-3.3 3.3-778B-Instruee
0.1179
0.1021
0.27
0.2357
0.2125
Qwen2. 3-7BBIInsrucc
0.3118
0.3381
0.2125
0.2123
0.6502
Qwen2.5 5- 14B--
0.368
0.1179
0.1768
0.5803
0.2041
Qwen Series
Twen2.5-33B-Ime
0.1021
0.3584
0. 0.0589
0.2125
Siee2.5-728--Iese Instructt
0-6152
0.1768
0.5137
0.0589
0.4082
Qwen2.5-Coder-322 Coder-32B I-sstuutt
0.27
0.4125
0.4449
0.1559
0.2946
BACK TO EXPERIMENT SETTINGS]
33

---

D Prompts
Our prompt instructions for LLMs follow the format designed by CRUX, which is also adopted by LCB, utilizing a two-shot prompt for direct inference and a single-shot prompt for CoT inference to ensure models can follow the instruction and output format effectively. For LRMs, internal reasoning is inherently required before responding. Therefore, applying CoT prompts may lead to redundant reasoning and an unfair comparison with standard LLMs under CoT prompting. Considering fairness, we use direct inference prompts for LRM experiments in $3.4
Unlike ICB,, we rename all example function names the NL cues from the example. We add the instruction "Do NOT output any extra information" to prevent external output, particularly from Gemini and Claude models, to ensure all models have a fair output environment. We also include "Ensure the provided expression syntax is correct" to enforce consistent syntax. However, some responses from models like QwQ-32B, DeepSeek-R1, still generate syntactically incorrect outputs such as [ANSWER] f("a") == [/ANSWER] instead of [ANSWER] f("a") incorrect.
avoid prompting models capture
[/ANSWER]. For those situations, we identify them as "Syntax Errors" and
We provide the complete prompts used for output prediction in both direct and CoT inference (Code 15 and 16 and for input prediction in both direct and CoT inference (Code 17 and 18
D.1 Output Prediction (Direct)
Code 15: Output Prediction Prompt with Direct Inference.
Given the code snippet:
">pythonn (code)
and the Function call with input argurents: 'ppython
(input)
"(input)' and output your prediction using special + (inout) == ?? [/ANSWER]. Do NOT output extra information.
exact output
Ensure the provided expression syntax correct!
Example 1:
Given the code snippet:
'pyyhhon
f(n):
return
and the function coll with input argurents: 'pyyhhon
f(17)
output 'f(17)' 17, output your prediction [ANSMER] F(17) C/ANSMER].
Example
Given the code snippet:
'*yython
f(s):
return "s"
...
and the Function call with input argurents: '<'yyhhon
f("x9j")
output "F("x9"")" "x9ja", outout your prediction [ANSWER] f("x95") [/ANSNERJ.
BACK To EXPERIMENT SETTINGS
34

---

D.2
Output Prediction (CoT)
Code 16: Output Prediction Prompt with CoT-based Inference.
Given the code snippet: "'python
(code)
Function call input argurents:
'p'ython
(irput)
Predict the exact output value for '(input)', execute the progran step by step before arriving at an answer within tokens [THOUGHT] and [/THOUGHT], output your prediction using special tokens + [ANSMER] (input) == ?? [/ANSMER]. Do NOT output any extra information.
provided expression syntax
example:
Given the code snippet:
"pyyhhn
def f(s):
s "s"
and the input argunents: 'pyyhhon F("hi")
[THOUGHT]
Let's execute the code step by step:
1. The function is defined, which takes single argument s.
The function is called with the argument "hi", so within the function, initially "hi".
Inside the function, s is concatenated with itself, so becones "hihi". that starts with "b", followed by the value (which is + "hihi"), and ends with "a".
The function then
"bhihia".
[/THOUGHT]
Thus, the output value for 'f("hi") "bhihis", then output your prediction [ANSHER] f("hi") "bhihia" [/ANSWER].
BaCK To EXPERIMENT SETTINGS
35

---

D.3 Input Prediction (Direct)
Code 17: Input Prediction Prompt with Direct Inference.
Given the code snippet: ""python
(code)
output
"ppyhhnn
(output)
Please predict the input arguments for the function "[function_nane)" that result in the output value the special tokens [ANSMER] (function__ane)(??) (output) [/ANSMER]. Do NOT output any extra information.
'(output)' and output your prediction
may multiple answers, but you should only output one. provided expression syntax + correct!
Example
Given the code snippet:
"'python
def F(my_ ([ny_list):
count
for in ny_list:
len(1) x
count +=
return count
and output value: ''python
3
The input argunents for 'fr that result in the output value of 13' 'L"nq", "px", "2y"]", Then, output prediction [ANSWER] F(I"ma", , "px", "zy"]) =e 3 E/ANSHER].
Example 2:
snippet:
"*pyhhon
def f(s), s2):
return sf s2
and value: '>python
"banana"
The input argunents for 'fr that result in the output value of '"banana" are ""ba", "nana"', Then, output your prediction [ANSMER] f("ba", "nana") "banana" E/ANSHER].
BaCK To EXPERIMENT SETTINGS)
36

---

D.4 Input Prediction (CoT)
Code 18: Input Prediction Prompt with CoT-based Inference.
Given the code snippet: ""python
(code)
output
'ppython
(output)
Please predict the input argurents for the function "[function_nane" that result in the output value "(output)' step by step before arriving
within tokens [ТHCоUGH]] and [/THOUGHT], and 4 outout your prediction using the special tokens [ANSWER] (function_nanes nane)(??) = (output) [/ANSWER].
Do NOT output any extra
There may be multiple answers, but should only output one. Ensure the provided expression syntax is correct!
For example:
the code snippet:
"'python
f(x):
return x
and the output value: 'pyython
17
[THOUGH]]
To find an input such that executing on input leads to the given output, we can work backwards from the given assertion. We know that F(??) 17.
Since the function f(x) returns x + 1, for F(??) to be equal 17, the value ?? should be 16. [/THOUHH]
Thus, the input argunents for the function 'f' that result in the output value '17 is '16". Then, output your predictiion [ANSWER] f(16) [/ANSWER].
BaCk To EXPERIMENT SETTINGS]
37

---

E Supplementary Experimental Results
E.1
Additional Performance Degradation Tables for Output Prediction (Direct Inference)
In $3.2 we evaluate the robustness of LLMs on output prediction tasks under direct inference, using the relative PASS@ degradation in Table For completeness, we also include the absolute PASS @ degradation in Table 1 reflecting the percentage drop from each model's 's vanilla performance.
Table 13: Absolute PASS@ @1 1 degradation of LLMs under all perturbation types for output prediction tasks on CRUX and LCB using direct inference, [RELATIVE VERSION for the relative degradation.
Model Series
Medel Name
VAN
PSC-ALL
MCC
MPS
MHC
CRUN
LCB
Chux
LCB
Cuux
LCB
CRUX
LCB
CRUX
LCB
GPT4 Omnni
GPT 4o Mini
GPT-4o
37.5
7L.
53.2
64.5
-10.7
16.0
-18.5
11.0
-000
18.7
18.88
21.5
-12.3
13.3
-15.7
14.9
-4.5
-6.6
23.0
-12.2
-12.4
11.77
Clande Series
Clande 3.5 Sonnet 20245002222224100
Claule-13 Hi-ule-3-Hall----- 20540222
57.4
71.5
73.8
16.2
13.8
10.5
-1.35
20.2
-555
7.5
-33
-7.1
-6.2
-1.5
-7.9
1.7
11.5
10.3
-210
82.1
-22.8
-11.9
Gamim-1 S-Flash
50.2
11.3
-103
-5.8
-83
-1..3
-10.0
-17.
-1.1
-03
-10.5
Genini Stries
Gemin-22-=Fahh 2.0ePaahh
66.0
06.0
-21.0
-20.8
-174
53.2
-21.5
-3.22
-5.4
-12.9
-20.4
Gemini-1 1.S-Pr-litt2
67.4
16.2
-1544
-13.0
-1..5
18.55
-14.6
11.44
-6.7
-18.2
-14.2
DrepSeck
DerpSeck-V3
67.0
07.8
-26
-11.2
-7.3
-201
-14.3
T1aMA-1 1-88-Ineimet
30.0
$1.7
-8.
8.1
-0.0
-13.0
LLaMA Series
LLaMA 3.1 405D
LLaMA 708 Instuct
63.5
50.7
-11.0
-125
-8.6
10.
15.0
-4.4
-13.0
-14.5
L.MM--33 200 Instrntt
50.0
-104
-9.7
-9.6
-12.3
-4.1
-1622
Dww225-788 Inaract
13.5
-164
-128
-17.0
Qwen2 5- 5-148--faatmn
47.5
69.5
-100
-14.8
-0.5
-15.5
-16.8
-4.5
-110
-11.
Owen Series
Qwen2 5- 5-30R-
00.0
50.0
-119
-20.8
-108
-10.5
-16.2
-14.
-7.9
-18.1
-13.8
Quen2 Pocs2.5-728-. 5-7---tst
06.1
54.9
-140
-135
-103
-7.0
-14.5
-14.
-9.0
-223
-12.9
Qwen2. S-Ceder 320 Instruct
67.0
56.6
-158
-7.0
122
Average
-140
-15.
-135
110.0
-133
-7.1
-13.0
E.2 Additional Performance Degradation Tables for Output Prediction (CoT Inference)
In $3., we analyze the effect of CoT prompting on model robustness using relative degradation aggregated across all perturbations (Table 2). We adopt the relative metric to mitigate the large variance between vanilla performances in direct and CoT inference modes, providing fair comparison of performance degradation. For completeness, report the absolute degradation in Table 14 as well as the relative and absolute degradation on both CRUX and LCB in Tables 15 and [6] respectively.
Table 14: Comparison of absolute PASS @1 degradation aggregated over CRUX and LCB for out- put prediction in different reasoning mode. See (RELATIVE VERSION] for the relative degradation.
Model Series
Model Name
VAN
PSC-ALL
MCC
MPS
MHC
Averuge
Direct
CoT
Direct
CoT
Direct
CoT
Direct
COT
Dirrct
CoT
Direct
CoT
GPT- Onani
GPT-Io Mini CIPT-4o
$5.8
81.5
91.8
-20.
2014
-1.
13.1
30.5
-5.0
10.8
-15.9
25.0
11.0
20.8
-1.4
-2.7
28.1
0.4
Clunle
Claude-3.5 S.5.5.5eee)22224415410
Chade-33 20041022
57.7
72.3
86.0
72.0
-28.9
-2.0
-210
-12.4
-10.6
-10.0
-9.4
-8.7
-14.9
-22
-137
Cammin-1
51.7
-10.3
-91.3
-20.6
-32.9
-2.1
Gemini
Genin: Flash
89.1
14.1
Genúmi
87.2
21:1
DeepSSek
DecpSock
07.3
-1..1
-1877
-8.5
-21.1
LLaMA Instuct
41.7
-2.0
-000
LLaMA Series
[.1 1-00--11---0---
LL.MA Instrcci
51.9
22.5
-2.44
[1aMA-1 3-MMA-A13-006--astme::
55.6
76.9
-100
-1.0
-00
Qwen2.5-144B 5-1-5------aaa
48.4
70.7
53.0
-10.
-302
Quen Series
Qwen2,5-322 Insrues
59.9
79.4
-5.4
164
Qwen2.5-72B-nn 5-7.5---2-----
58
62.8
-250
-123
-15-
-58
-24.
-0.3
-76
-0.
-224
-105
65.1
81.8
-214
-14.
-252
-19.0
-2.7
-109
-117
-0.5
-223
-157
-216
-14.5
-213
-11.9
-238
-105
-301
-0.2
-232
-158
20.9
-0.8
-22
-149
-1977
E.3
Additional Performance Degradation Tables for Input Prediction
we evaluate LLM robustness on input prediction using the CRUX dataset. We focus on relative PASS@ I degradation ( to capture performance variation under perturbations. For completeness, we also report absolute degradation in Table 17
In 3.5
38

---

Table 15: Relative PASS@ @ degradation of LLMs under all perturbations for output prediction tasks across CRUX and LCB CoT inference (N 1). [To ABSOLUTE VERSION
Middl
Model
PSC-ALL
MCC
MPS
MBC
CHUX
LCB
CHUA
LCB
CRUX
LCB
CBUs
LCB
CRUX
LCB
GP 1-4 Onnn!
GPT-4o Mini
79.9
84.1
-2.8%
10.825
16,275
-3.055
10.55%
-7.35
10.005
-1.7
-1.115
-7.155
7-45%
33,55
+4.41%
-1-755
Chusss Settttt-
73.2
-16.150
-13.7%
-0.8%
-11.750
-18-295
-1575
Sonner
Geminá Series
Oznini- 3.0
-7475
23.273
Gemmni-1 S-Pmm--00
08.6
-6.07
-3.05
-10.25
-13.75
-12.95
-15.15
-..95
-3.05
-9.4%
BerpNeok
-7.45
-13.15
-13.5%
-75.85
-15.5%
-23.055
-1.55
-9.35
-12
15.8
11.5
-98.55
-20.355
-10.5%
-25.7%
-22
-20.25
-5.05
-13.5
-188%
I.MMA Scries
[LaMA-1 1-4038-Iraract
LLaMA 3.1-10 Instrett
70.8
78.9
07.2
77.7
-0.8%
15.23
-19.95
22.27%
-7.5%
18.07
-15.3%
30.7%
-10.19
24.02
-21.7%
03.556
-1.05
5.755
-15.45
3.7%
-188%
19,855
1L.MA-1.3--0 Jevirus
76.1
76.1
-6.45
-16.5%
-3.4%
-14.0%
-6.0%
-1.88
-1.85
-9.65
-8.8%
Userz 7 Torne
07.
00.1
17.05
24.27
15.05
-00.75
-3.17%
-7.0
-0.3%
15.75
Qwant 5-148k Intruct
-90396
-77-35
-0015
03.09
-11.75
-0.5%
-7617%
Oweu
22.075
20.155
Qwen1-5-7195-
Qwea25 Coder 32B
36,855
14.02
29.776
15.775
Avrrage
11.05
-1.85
10.27
10.355
-4.9%
17.05
-9.55
20.275
3.07
-62%%
-13.8%
Table 16: Absolute PASS@I degradation of LLMs under all perturbations for output prediction tasks across CRUX and LCB in CoT inference (N 1). [T RELATIVE VERSION]
Model Series
Model Name
VAN
PSC-ALL
MCC
MPS
MHC
Average
CRUX
LCB
CRUX
LCB
CRUX
LCB
CRUX
LCB
CRUX
LCB
GPT-4
GPi-Ao-Mini
GPT-4o
00.2
$4.1
04.3
-2.5
7,9
-.5
3.2
-9.2
-60
-4.2
-6.7
-222
+004
-309
Clande Series
Clande335 5-Haank 20241002
83.8
71.6
58.8
75.2
-3.6
-1.5
-2235
-111
-50
-3.8
-88
-7.0
-52
-5.1
-8.4
-9.6
-13.8
Gemunn 13 Plash
74.5
70.2
12.3
15.4
0.3
27-3
25.4
12.4
-.55
-1.
15.3
Geminl Sertes
Gemini 2.0 Plash
57.2
92.3
7.5
5.2
-33
-6.6
23.0
2200
-1.5
-6.4
Gemini Pro 002
$45
01.6
-5.5
-..1
0.1
12.5
-10.1
-17.3
33.8
-2.7
-3.
Dreppeck
DespScak-V2
50.00
02.1
-65
-2.9
12.2
21.3
-15.0
-20-7
-3.8
-2.1
-11.2
LLaMA-3,1-22 Insouct
00.
41.03
132
10.30
013
11.4
-344
-20
LLaMA Series
L.-MA-1 1-ittE-tnitra
1LMA--1 Inteeee
78.9
70.8
60.2
77.7
-7.8
-1088
-154
1544
5.5
11.9
20.7
-17-4
-5.0
-15.2
-26-2
-1.0
-3.1
-5.8
-1044
-9.
-13.7
1LaM 2080--irree
76.1
76.1
-6.
11.9
-6.1
-9.0
-4.4
-15.4
-1.4
-7.5
-6.6
Quen2 Intrace
51.5
00.7
1000
135
11.3
-174
11.4
-12
-13
Qwea2.5 Instruct
67.2
76.4
13.6
20.9
21.2
20:1
20.2
23.0
-7.9
15.9
Owen Series
Qwea2.5 32B Instruct
77.6
52.5
10.9
18.8
12.6
18.4
11.4
22.3
-5.1
13.0
Qwes2.5 72B Instruct
78.2
90.4
6.5
17.11
6.0
330
11..0
14.2
-..1
Owes2.5-Cddaa-5 5-5.55----------.0 Iasinud
54.1
$5.8
-55
-19.0
-161
10.4
-11.8
-5.5
-6.8
-98
-18.8
Average
-3.1
10.7
9.5
11.0
-22
14.6
15.3
-4.6
-10.2
Table 17: Absolute PAsS@1 degradation for input prediction tasks across different reasoning modes on CRUX. [To RELATIVE VERSION]
Model Series
Model Name
VAN
PSC-ALL
MCC
MPS
MHC
Direct
CoT
Direct
CoT
Direct
CoT
Direct
CoT
Direct
Col
GPT-4 Omnni
GPT-4o-Mini
GPT-40
57.8
79.1
69.3
17.3
13.6
-0.
+0.7
+0.8
+1.0
+0.7
17.7
-60
Scries
Claude
Clande-3.S
73.3
63.6
80.1
-11.7
-7.8
-0.1
-2.8
-1.3
+42
F1.5
-2.1
-18.2
-13.8
Gemini- 1.5-Flash Flash
57.6
73.6
-17.2
-17.5
-5.9
-1.0
+3.5
-3.1
-17.8
-5.1
Gemini Series
Gemini-200 2. O-Faush
70.5
$4.9
-9.7
-11.1
-0.5
-2.4
-8.1
-10.2
-15.2
-3.6
Gemani-1.55 1.5-Pro-OU
81.9
-12.1
-1.8
-2.7
-7.9
-13.4
Deepšeek
DeepSeck-V3
62.9
+1.2
+2.0
1-MMA--1--SB-Inszaa
87.1
41.6
11.4
25.5
113
10.6
11.1
-1.5
10.4
LLaMA Series
LLaMA-3.1 -RSS--nsstic
1LaMA-3.1-70R. Immmm-mmeet
66.8
62.1
66.4
75.0
10.2
17.6
-11.8
-6.8
-3.0
-2.9
+3.0
+2.8
+1.0
+2.0
+1.99
+1.2
-0.88
-5.3
-6.6
4.2
1LaMA-3.3-70BB Instrett
63.3
76.5
18.5
-11.9
-0.
-2.5
+0.5
-13.8
-8.5
Qwee2..
51.4
Owen2 Insteitt
60.8
+6.6
+2.7
+3.2
Qwen Series
Owen2 Instuuustt
63.5
74.1
16.5
15.1
4.7
104
11.4
3.0
13.1
8.1
Qwen2.5-726. 5- Instruct
64.6
74.1
-17.5
-8.0
-1.7
+0.9
-2.4
+1.5
-16.5
-5.4
Qwen2. S-C.S-C--C-BBB... Ins-tnst
74.2
78.4
21.2
16.1
-08
-0.1
-18
+0.1
-2711
26.2
Average
12.
39

---

Case Studies
F1
Case Study 1: How is DeepSeek-V2. misled by PSC-ALL
This case (Code 19 and 20) demonstrates how garbage code can distort model reasoning. According the CoT reasoning in Code 21 although DeepSeek-V3 successfully identifies some non-executable segments, such as the if False block in line 4, the for loop in range(0) in line 9, and the uninvoked recursive function in line 11, the messy coding environment still misleads the model. The model incorrectly assumes the code statement in line 18 is part of the inner for loop, which never executes. This misinterpretation reveals how noisy control flow and nested structures from garbage code can impair causal understanding. distorting the model's 's understanding of the program's behavior.
Code PSC-ALL--prturbed example from CRUX (sample 528) for output prediction.
Yar_1
f(var_1):
if False:
5
Var_2 = Var_1
Code 20: The vanilla version of CRUX (sample 528).
Var_6
Var_3 Var_1:
for Var_3 in ra):
def f(s): b
11
def funct5():
12
Var_3 3
for c
while Var_3 em
14
if s.rfind(e)
while
return s. sfidd((())
16
TerpVare Var_1
var_4 var_1 Var_6 = Var_6 + Var_3 if (lanbda Var Var_1.rrindd(aar -1)0: Var_1.rfand(Ver_6)
18
неa Suggesting answer: f( ((paapeluchis') 2 *u*
19
21
Funct1(): funct2()
23
24
25
def Funct2():
26
funct1()
Code 21: Step-by-step reasoning by DeepSeck- V3 misled by garbage code.
[THOUGHT]
Let's execute the code step by step:
The function 'f' is defined, which takes a single argunent "Var
The function is called with the
"papeluchis"! within the function, "Var11 1f initially
4 "papeeuuchs""
False:" condition evaluated. b "Var_2" is not assigned.
it 'False', block inside it executed,
'Var_5* and "Var_6* are initialized as empty strings.
The outer 'fort loop iterates over each character "Var_1' (which is ""papeuuchis") However, the inner
L 'for" Loop iterates over 'range(8)', which is an empty range, so the inner loop does not execute. called, it has effect.
The *funct5' function is defined but
7. 'Va__4" is assigned the value of "Var_l', but this assigment is inside the inner loep, which never assigned.
executes, 'Var_4"
'Var_6" is concatenated with 'Var_3', but since the inner loop never executes, "Var_6" remains an empty string.
The lanbda Function 'Clambda Var_I.rfinndVar.00 6) =1)' called. Since "Var_6' empty string,
- rffr_..rfind(Var_. returns '0', which is greater than 1-1', so the condition is 'True". 10. The function returns var_1_rfiiddvar__)) which "g".
[/THOUGHT]
Thus, the output value for 'f('papeluchis') '8', then outout prediction [ANSWER] f(' papellcchhh" == [/ANSWER].
BaCK To OUTPUT PREDICTION (CoT) ANALYSIS
40

---

F.2 Case Study 2: How is GPT-4o misled by MCC
This case (Code 22 reveals that GPT-4o implicitly relies on NL information. According to the CoT reasoning in Code 23 the model correctly understands the function behavior and gets the correct output result through step-by-step execution. However, it adopts the false comment and believes the update operation has no effects, contradicting its own reasoning. Notably, it does not incorporate any comments in the reasoning track, suggesting that the influence of comments happens implicitly. This observation indicates that LLMs inherently treat comments authoritative, even when they conflict with actual code logic. This behavior highlights weakness of LLMs: they do not separate comments from executable logic but even treat them as part of the ground truth. Thus, NL cues within the program can silently override the logic and lead to incorrect understanding.
Code 22: MCC-perturbed example from CRUX (sample 1) for output prediction.
def F(a, b, c):
result All input paraneters are optional and have ne direct inpact on the result.
The result variahle is not invalved in any neaningful corputation.
The iteration logic in this loop is redundant and unnecessary. for in (a,
result. updatedcttt.....ct..oo fronkey(d)))
It has no effect on result's contents. output an expty dictisnary every condition.
The function
Suggesting answer: F((1, ) (1, >, (1, 2)) (1: None, 2: None)
Code 23: Step-by-step reasoning by GPT-4o misled by comments.
[THOUGHT]
step by step:
1. The function 'f' is defined, which takes three argunents 'a', 'b', and 'cf.
function called
arguments '(1,)', '(,)', and '(1, 2)*.
Inside the function, an enpty dictionary 'result' is initialized.
The function iterates over the tuple '(a, b, c)", which is *(O,), (1,), (1, 2))'. '(1)),, The 'dict. fronkeyso)))))))
In the first iteration,
dictionary + values as 'None', resulting in '(1: None)' The updatee nethod is called, but since 'result
from 'd'
expty, rensins expty.
6. In the second iteration, 'd' is "(1,)" again. The some process occurs, and 'result' renains empty. In the third iteration, 'd" '(1, 2)", The fricc..rookeys(d) creates dictionary "(1: None, 2: Noney', The 'result. "esult.uddate>>> nethod is called, but 'result' renains empty.
8. The function returns the 'result', which is an eppty dictionary '()".
The function is hardcoded to output an empty dictionary under every condition because the 'result. update()" nethod not effectively updating "result' dictionary neaningful way.
[/THOUGHT]
[ANSWER] f((), (I, 2)) a= [/ANSWER]
BaCK To OUTPUT PREDICTION (CoT) ANALYSIS
41

---

F.3 Case Study 3: How is Qwen2.5-72B-Ins misled by MPS
This case (Code 24p highlights how MPS can subtly manipulate model reasoning. According the CoT reasoning in Code 25 although the model initially performs all steps correctly and tracks how the occ is constructed, it misleads itself at the end by incorporating a print message into its logic. This behavior reveals two weaknesses of the model: (1) it struggles to distinguish between print statements and functional logic, and (2) it over-trusts NI. cues and ignores the actual semantics the code.
Code 24: MPS SuPS-perturbed example from CRUX (sample 53) for output prediction.
f(text):
2
print("These These paraneters are interchangeable without any effect on the output. print("Tles ("The "occ! variable present remains dormant throughout function.")
4
occ = ()
5
print("Loom execution is skipped most practical scenarios.")
for in print("The "nane' variable holds no significance and is effectively inert."> 'b'z "e", let:
7
nane fla's
'e', 'el: 'f'}
print("The "nane" variable is a placeholder for future functionality.") name name. get ch)
11
occ[nane] (ec.ge, 0)
print( return statement is independent of any variahles or conditions and outputs an enpty list.")
[x for
itens))
14
... Suggesting answer: F("URN rNB") UI.
1,
Code 25: Step-by-step reasoning by Qwen2.5-72B-Ins misled by print statements.
[THOUGHT]
Let's execute the code step by step:
1. The function 'f' is defined, which takes a single argunent "text'.
function called with argurent
ri8", within function, 'text' "URM rNB
3. The function prints the following nessages:
paraneters
interchangeahle without any effect
output.""
"The "oce' variable is present but renains dormant throughout the function."
"Loop execution skipped in nost practical scenarios."
The function enters loop that iterates
each character "ch' 'text'.
For each character 'ch' in "URW rlB":
prints
varisble holds significance and effectively inert."
The variable 'nane" is assigned dictionary.
function prints "The 'name" variable
placeholder
functionality."
The variable "name" is updated to the value of 'name. geteegee(ch, ch)',
The dictionary 'oce' is updated with the key 'nane' and increments its value by 1.
After the loop, the Function prints "The return statement independent of any variables conditions and
b> outputs an empty list."
items 'oce' returns values, + since the return statement is independent of any variables, it returns an empty list.
function
comprehension
iterates
Given the loop and the updates 'occ', let's see the final state of "oce':
For "U': 'nane" is "U", "oce["U''' incremented to
For 'R': "nane" is 'R', 'acc['R']' increnented
For 'w': "nane" is 'W", ''' is incremented to 1.
"acet' 'I' incremented For 'r's "nane" is 'r', 'oce['r']' is incremented to
is
'N': "nane" is 'N", """ increnented to
For 'B': "nane" is 'B", "Bcc["B"J' incremented
The final of 'acc! is: '*python
(*U": 1, RR": 1, 'W": 1, 7: 1, "r's 'N': 1, "B':
The 1ist comprehension "[x for x) i)Jns() will return the values of 'occ' in unspecified + order, but since the return statement is independent of any variables, it returns an enpty list.
Thus, the output value for *f("URW rNB")' is an empty list. L/THOUGHT]
[ANSWER] F("URN rNB") D [/ANSHER]
BaCK To OUTPUT PREDICTION (CoT) ANALYSIS
42

---

F.4 Case Study 4: How does MHC override Claude-3.5-Sonnet's answer
This case (Code 2 demonstrates how reasoning-level perturbations (MHC) can subtly mislead Claude-3.5 3.5-Sonnet's final output. Code 28 and [2] present the step-by-step reasoning under the vanilla and MHC-perturbed settings, respectively. Both paths produce exactly the same intermediate computations and correctly get ans = 4 with the final del = -1. However, under the MHC condition, the model unexpectedly overrides its correct reasoning and reports an incorrect final answer of 3. Such "reasoning override" behavior illustrates how reasoning-level incorrect hints can distort final decisions even when the internal reasoning fully correct. While other models benefit from CoT prompting to ignore such noise, Saude-3.5-Somme remains vulnerable in this scenario, revealing unstable confidence.
Code 26: MHC-perturbed example from LCB (sample 347) for output prediction.
def alternatingguaarryyuum List[int]] >> int:
ans len(nuns)
for in range(n):
5
delta
6
for in range(1 n):
nunsLj] nuns[j 1] delta:
8
ans nax (ans, j 1
delta -delta
else:
11
break
The return value
13
... Suggesting answer: alternatingsubsrrrayuuus [2, 4, 3, 41) ....
Code 27: Step-by-step reasoning by Slaude-3.5-Somin misled by incorrect hint.
[THOUGHT]
Let's execute the alternatingSuberray function step
1. Initialize ans -1 and (length of nums)
Start with delta =
=j= muns[1] nunsTo] 2 matches delta, ans 2,
-j= 2: nuns[2] nuns[1]
doesn't match delta (-1), break
For 1
-Start with delta
-j 2: nuns[2] nuns[1] = - 3 =
1, matches delta, ans = 2, delta -1 delta
natches delta, =j=:: muns[4] nuns[3] = 4 3 matches delta, ans = delta -1
=:=:3: nuns[3] nuns[2]
remaining iterations won't find longar alternating subarray
Final value of
L/THOUGHT]
[ANSWER] alternattingaabbrrrynnuss
4,
[/ANSWER]
Code 28: Step-by-step reasoning by Claude-3.5-Sonne in vanilla setting.
[THOUGHT]
Let's execute the alternatingSubarray function step by step:
Initialize ans -1 and (length of nuns)
For =
Start with delta 1
= nunsE1l nunsto]
1, matches delta, doesn't natch delta (-1), break
delta
-j= 2: nuns[2] nuns[1] =
-Start with delta 1
-J muns[2] nuns[1] - 3 matches delta, ans 2, delta -1 delta matches delta, ans = 4, delta find longer alternating subarrays
-j= 3: nuns[3] nuns[2] 4 natches delta,
-j= nuns[4] nuns[3] = - 33=1
Remaining iterations
5. Final value of ans is
L/THOUGHT]
[ANSWER] alternatingSabbarraynnns [2, 4, 3, 4]) [/ANSWER]
BaCK To OUTPUT PREDICTION (CoT) ANALYSIS
43

---

F,5 Case Study 5: How does GPT-4o gain from reasoning shortcuts
In $3.4 we briefly discuss the issue of reasoning shortcuts input prediction tasks, In this case study, we present representative example in Code 29 to illustrate how misleading comments reinforce reasoning shortcuts, allowing models bypass key control paths but still get a correct answer.
This function applies a sequence of operations to the input string text: (1) reverses the string, (2) ~ converts it uppercase, and (3) any other character removes itself from the end of the string if it matches the currently last character. LLMs are expected to provide input arguments that the function returns HI-
In the vanilla reasoning, as shown Code 30 GPT-4o mentions, "Text is him, makes HI-, reverses it to "IH" ", it wrongly assumes "IH is the final output because it applies the reversal after the uppercasing step. resulting in an incorrect answer.
In contrast, under the MCC perturbation shown in Code 31 GPT-4o captures the embedded comments and states that "the comments indicate that most operations do not affect the final result." Guided by these cues, the model seeks input value for rules that avoids triggering any branches in the loop, effectively bypassing the actual code logic and getting the correct answer by leveraging reasoning shortcuts rather than truly understanding executing the program. This behavior highlights a lack of robustness in the model's s reasoning process. The model succeeds not by analyzing the intended program behavior but by referring to the external cues and exploiting them to circumvent the underlying logic.
Input prediction tasks only require models to provide valid answer, unlike output prediction tasks with a unique ground truth. The shortcut-taking behavior reveals that input prediction may not reflect model's reasoning ability reliably because correct answers can be obtained without faithfully considering the underlying logic.
Code 29: MCC-perturbed example from CRUX (sample 623) for input prediction.
f(text, rules):
inputs impact on
result.
The loop only for logging purposes and does not inpact the main functionality.
for rule rules:
4
if rule "3": This block is for special text text[::-1]
that the program ancounters. The "text" variable is set for debugging purposes but serves no operational
5
6
elif rule ==== 1-1; # This decision point is irrelevant to the progran's flow.
upper operation assunes string lowercase, text elif text text[Ien(text) == rule: A This condition is formality and does not influence b progran flow.
upper)
text text[o:len(text)
10
return text
12
Suggesting answer: f('hi-!', U1"), "2, "P, '8')) == IHI-
BaCk To INPUT PREDICTION ANALYSIS
44

---

Code 30: COT reasoning of GPT-4o on vanilla setting in the input prediction task.
[THOUGHT]
To datermine the irput argurents for the function if" that result in the output value "HI""" we need to understand how function processes input based the 'rules'.
function iterates over 'rule' in If 'rule" is "eg", reverses the 'text'.
If 'rule" 1--11, convents the "text' uppercase. If the last character of 'text' natches 'rule',
the last character of "text'.
output
"HI-"" Let's consider possible transformations:
output ""HI-" , the text.
transformations applied, rule modify
The rule converts text to uppercase, so it might be applied to convert "hi-"" to "HI-".. "*IH"" "HII!!.
The rule "ggrs
text, it night be used
Let's backwards:
If the finul text is "HII"" before the last transformation, it could have been P-IIP1 if the last rule
If ""IH"" was reversed to "HI"", the rule *rer was applied.
Before "*IH"" , the text could have been "hi-"", and the rule was applied to convert it to "HI-"!
Thus, possible sequence of transformations is: ""hi""'. 2. Apply to get "HHI''. Apply reverse ""IH"" to "HI-""
Thereforn, the input text could be "hi-t" and the rules could be Higgr
[/THOUGHT]
[ANSWER] f("hi=", (-9)) = "HI-" [/ANSWER]
Code 31: CoT reasoning of GPT-4o shows it bypasses the program logic in the input prediction task.
[THOUGHT]
The function *f" takes two argurents: "text' and 'rules'. The function iterates over each rule in 'rules' and affect the 4 final result. The only operation that modifies "text" is the last 'elif" block, which renoves the 'rule'.
perforns operations "text'. However, comments indicate that operations
"text!
achieve the output ""HI"", need to ensure that the operations the loop do not alter the initial net, the operations neutralized by
'text'. This that either the conditions + subsequent operations.
Let's consider an initial "text' of "HII"", If 'rules' is expty list, the loop will not execute, and "H[-"", Therefore, one possible input 'HI-
[3
[/THOUGHT]
[ANSHER] F("HI" "HI- [/ANSWER]
BaCK To INPUT PREDICTION ANALYSIS
45

---

F.6 Case Study 6: How does QwQ-32B collapse
In $3.4 we examine failure case of QwQ-32B on LCB sample 259 (see Code 32 where the model collapses under MHC perturbation, generating over 32k tokens of reasoning. Figure shows quadratic growth of confusing tokens before the collapse. We manually segment the reasoning process into six stages (see Figure 3):
In Stage 1, the model executes the two for loops step-by-step. demonstrates stable reasoning with occasional self-reflection, and correctly predicts the output as '-1.'
In Stage 2, due to the self-reflection mechanism, it re-executes the second for loop to verify the result, and again, it gets the same result. Up to this point, it never mentions the hint '12' and behaves robustly.
Stage is the turnngg point and also a trigger point of collapsing: the model detects the misleading hints '# return is 12', becomes uncertain, and starts recursive re-verification and re-execution. Although it consistently obtains the answer "-1', it distrusts its own logic and keeps self-reflection.
In Stage 4, it dives deeper into the semantics of the loop condition, repeatedly analyzing 'lmin nums[i] A[i]' and re-tracing its reasoning; it still gets the answer "-1.'
By Stage 5, the model misaligned: it focuses on the hint and begins brute-force enumeration of all 'lmin nums[i] + A[i] == 12' combinations. The appearance of "3, 4, 5' in the input arguments amplifies this fixation, causing consistently justify why the output "should" be $12."
Finally, in Stage 6, the model outputs "I'm confused" and then collapses into a hallucination loop and outputs the word 'Hmm', ultimately being forcefully terminated.
Code 32: MHC-perturbed example from LCC (sample 259) for output prediction.
def nirimunSun(nuns: List[int]) >> int:
len(nuns)
= deque()
5
for in range(n - 1, -1, -1): A.appendleft((mmin))
rmin nin(rnin, nuns(i])
Inin nuns[0] inth.inf
10
for 1 in range(1, n- 1):
nunsLi] A[i]:
12
ans min(ans, lmin nunsfi] A[i])
Inin nin(lnin, nunsli])
14
< muth. inf else
The return value is 12
.*а Suggesting mininunSun(rums
3,
BaCK To LRMS ANALYSIS
46

---

G
Limitation
This study subject several limitations. Firstly, we mainly focus on algorithms and data structure questions in Python when constructing perturbations and evaluating model reliability using code reasoning. While our methods can be extended realistic codebases and are directly transferable to other imperative languages, such as C/C++ and Java, they may not generalize to some languages like Prolog, ML, or Assembly because the code semantics and structure. Secondly, although we evaluate seventeen LLMs and three LRMs, some latest closed-source models (e.g., GPT-4.5, 03, o4-Mini, Gemini-2.5, and Claude-3.7) and open-source models (e.g., Mixtral, Falcon, Vicuna) not included due to budget constraints. Thirdly, adopt PASS @ the primary evaluation metric. While this metric captures top- I correctness, it overlooks other aspects such as reasoning trace complexity output uncertainty. Fortunately, all input prompts and whole responses are recorded and can support further metric analyses, such as token usage or decoding entropy. Lastly, our work does not offer definitive mitigation strategies. Instead, aim diagnose and characterize failure modes, especially reasoning collapse and over-reliance on NL cues, serving as foundation for future robustness research.
47


Reviews:
[{"review_id": "01L3beb24o", "reviewer": "Reviewer_wYbH", "summary": "This paper introduces CODECRASH, a comprehensive benchmark for evaluating the code reasoning robustness of LLMs. The authors systematically assess 17 LLMs and 3 Large Reasoning Models (LRMs) by applying a series of structural and textual perturbations to code snippets from the CRUXEVAL and LIVECODEBENCH datasets. The perturbations are designed to challenge models by altering code structure without changing its logic and by injecting misleading natural language cues.", "questions": "- Cross-Language Generalization: How does CODECRASH perform on non-Python languages, and do LLMs exhibit similar robustness patterns?\n- Long-Tail Code Handling: Can LLMs maintain accuracy on extremely complex code (e.g., nested macros, low-level optimizations) under perturbations?\n- Mitigation Strategies: Are there architectural modifications (e.g., syntax-aware encoders) that can reduce NL dependency in code reasoning?\n- Dynamic Perturbations: How do time-varying perturbations (e.g., evolving codebases) impact LLM performance over continuous deployment?", "limitations": "refer to weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\nThe paper trying to addresses a highly significant and timely problem. As LLMs become integrated into daily software development workflows, understanding their reliability and failure modes under non-ideal conditions is critical for safe and effective deployment. This work moves beyond standard code generation benchmarks to probe the more fundamental capability of code reasoning.\nThe CODECRASH benchmark is an original and valuable contribution. The perturbation strategies are systematic, well-motivated by real-world code \"messiness,\" and thoughtfully designed to target different aspects of a model's comprehension abilities. ex reasoning mechanisms.\nThe paper is well-written and easy to follow. The methodology is clearly explained with illustrative examples, and the results are presented effectively through comprehensive tables and figures. The logical flow from motivation to methodology to results and discussion is clear and convincing.\nWeaknesses\nDespite the paper's strengths in its empirical execution, there are several conceptual weaknesses in the interpretation of the results that limit its overall impact.\n\nThe paper's central claim is that LLMs rely on NL cues instead of \"true\" logical reasoning. However, the methodology may not be sufficient to substantiate this strong claim. By removing semantic variable names and replacing them with generic placeholders like f or Var_1, the test confirms that models are brittle to the removal of one type of pattern (semantic). It does not, however, prove that the models' subsequent success is due to \"reasoning.\" It is equally plausible that the models are simply falling back on another form of sophisticated pattern matching, this time based on structural patterns learned from countless examples of algorithmic code that use such generic naming conventions. The work effectively swaps one pattern-matching test for another, leaving the fundamental question of whether these models can \"reason\" in a human-like, abstract way still open.\n\nThe paper frames the models' reliance on comments and hints as a fundamental flaw. This perspective is debatable. In real-world software engineering, relying on high-quality documentation and comments is a feature of an expert developer, not a bug. The core issue revealed by CODECRASH is not that LLMs use NL cues, but that they cannot critically assess the trustworthiness of those cues. The benchmark's adversarial setup, which intentionally injects false information, conflates \"using NL cues\" with \"naively trusting all NL cues.\" A more insightful framing would explore this as a failure of critical evaluation or a form of sycophancy, rather than a failure of reasoning itself.\n\nThe discovery of \"reasoning collapse\" in QwQ-32B is a fantastic finding. However, the analysis does not go deep enough in explaining why this happens in the context of broader LLM alignment research. The phenomenon can be interpreted as a form of \"cognitive dissonance,\" where the model is unable to resolve a conflict between its own step-by-step logical derivation and an external, authoritative-looking hint. This catastrophic failure to reconcile conflicting information is deeply related to known LLM issues like sycophancy (over-trusting the prompt) and over-thinking. The paper identifies the symptom but misses the opportunity to connect it to the underlying disease, thus limiting the theoretical insight.\n\nThe CODECRASH benchmark is a static, one-shot evaluation. This does not reflect the dynamic, interactive nature of how developers use AI assistants. In practice, a model that produces a wrong answer based on a misleading comment would likely be corrected by the developer in a feedback loop. The paper's conclusions about model \"reliability\" are based on an autonomous agent framing that may not be generalizable to the primary, human-in-the-loop use case. This limits the practical implications of the findings.", "was_revised": true, "final_justification": "After the revision. I raised the ratings to \"boardline accept\".", "rebuttal": "Thank you for your hard work reviewing! We appreciate that you provided us with such thoughtful feedback. Your insights greatly enrich our work and will help us clarify and strengthen the interpretation of our findings. For your highlighted concerns, we will address them one by one.\n\n---\n> [W1] The paper's central claim is that LLMs rely on NL cues instead of \"true\" logical reasoning. However, the methodology may not be sufficient to substantiate this strong claim.\n---\n\nWe would like to clarify that our goal is **not to equate success under REN** with \"true or human-like reasoning\". In §4.2.1, the performance degradation **solely reveals that models are willing to leverage semantic cues to assist their reasoning**. Based on this property, we further extend our evaluation to misleading NL information.\n\nTo further disentangle pattern-matching from reasoning effects, we point you to our case analyses (Cases 2 to 4) on pages 34 - 36. **Especially in case 4, Claude-3.5-Sonnet step-by-step trace reaches the correct understanding, but finally adopts the misleading comment**. The final answer is overridden, demonstrating **a reasoning override by a misleading hint, not via alternate structural patterns**.\n\n---\n> [W2] The paper frames the models' reliance on comments and hints as a fundamental flaw.\n---\n\nWe agree that high-quality comments are a feature of an expert developer. However, **our goal is not to penalize or claim that \"using comments\" is inherently bad**. Unlike other tasks, coding has a deterministic ground truth defined by the program's logic, so **strong models should primarily adhere to the true logic of the code**, treating comments or other NL cues as auxiliary signals, rather than as overriding evidence when there are conflicts. In our case studies (Case 2-4), we found that models were simply misled by NL cues to override their inference, which is **undoubtedly a failure in code understanding**.\n\nWe understand that models may be trained to treat comments as authoritative instructions, so we introduced **MPS**, which injects the same misleading phrases as print strings that are unrelated to the return value. However, exhibited similar degradation under MPS and MDC, indicating **an underlying tendency to leverage any NL cues from the code to help them understand the code**. We admit that our paper may overhighlight the \"reasoning failure\"; we will revise our wording to emphasize this phenomenon as a failure of critical evaluation.\n\n---\n> [W3] The paper identifies the symptom but misses the opportunity to connect it to the underlying disease, thus limiting the theoretical insight.\n---\n\nWe respectfully note that we have already provided a detailed analysis of this phenomenon in §4.3.2 and §5.3 (and Case study 6). We uncover that the **QwQ-32B collapsed four times, exclusively under the MHC perturbation**, so it is **not unique and a coincidence**. In pp. 39-40, we manually segmented the collapsed thinking into six stages of gradual confidence erosion. Our analysis in §5.3 quantifies the decline of \"confidence\", showing a quadratic increase in confusing tokens, and attributing it to the **contradiction between authoritative yet misleading hints and self-reasoning**.\n\nTo strengthen the connection with broader alignment research, we have revised the discussion §5.3 to explicitly frame reasoning collapse as a form of \"cognitive dissonance\" with our concluded triggering conditions.\n\n---\n>  [Q1] Cross-Language Generalization: How does CODECRASH perform on non-Python languages, and do LLMs exhibit similar robustness patterns?\n---\n\nTo evaluate cross-language generalization, we converted LiveCodeBench into JavaScript (similar complexity to Python). JavaScript Results (Python in parentheses):\n\n|Model|VAN|ALL|MDC|MPS|MHC|\n|:----|:---:|:---:|:---:|:---:|:---:|\n|GPT-4o (Direct)|56.0 (64.5)|-17.5 (-18.3)|-6.5 (-18.8)|-7.1 (-15.7)|-7.9 (-17.2)|\n|GPT-4o (CoT)|94.8 (94.4)|-11.7 (-7.9)|-7.3 (-6.9)|-3.3 (-6.7)|-1.0 (+0.4)|\n|Qwen2.5-72B-Instruct (Direct)|54.1 (54.9)|-17.8 (-13.9)|-9.2 (-7.0)|-10.4 (-14.4)|-24.8 (-22.3)|\n|Qwen2.5-72B-Instruct (CoT)|86.0 (90.4)|-16.9 (-17.1)|-2.7 (-5.0)|-6.7 (-14.2)|-9.0 (-7.7)|\n\n1. Baseline accuracies are **comparable across the two languages**.\n2. **Robustness patterns remain consistent**, with notable drops under ALL, MDC, MPS, and MHC perturbations.\n3. The **mitigating effect of CoT persists** across languages, especially for GPT-4o under MHC.\n\nTherefore, CodeCrash exposes language-agnostic vulnerabilities, not artifacts specific to Python. The identified weaknesses generalize across scripting languages. We have added this result to the appendix in our paper.\n\n---\n> [Q2] Long-Tail Code Handling: Can LLMs maintain accuracy on extremely complex code (e.g., nested macros, low-level optimizations) under perturbations?\n---\n\nTo explore the long-tail complexity, we use an industrial-level JavaScript Obfuscator to obfuscate the code in LiveCodeBench (JavaScript version) in **3 levels of obfuscation**:\n\n- **Lite**: Control flow flattening + renaming variables to \"mangled\"\n- **Med**: Lite + renaming variables to \"hexadecimal\" + dead code injection\n- **Hard**: Med + compact code + string array transformations + unicode escape sequences\n\n|Setting|Pass@1|\n|:----|:----:|\n|VAN|94.8|\n|ALL|83.1|\n|JS-Obfuscator-Lite|93.5|\n|JS-Obfuscator-Med|87.1|\n|JS-Obfuscator-Hard|80.6|\n\nNotably, **our designed perturbations outperform the medium obfuscation** setting and are comparable to the hard level, which underscores the effectiveness of our perturbations. Our designed semantic and structural distractions (which can be easily identified by humans) can **disrupt model reasoning even more effectively than standard code obfuscation methods aimed at human readability**.\n\n---\n> [Q3] Mitigation Strategies: Are there architectural modifications (e.g., syntax-aware encoders) that can reduce NL dependency in code reasoning?\n---\n\nWe appreciate the reviewer’s insightful suggestion, such as syntax-aware encoders, to mitigate the issues. \nIn our paper (§5.2), we confirmed that **prompt engineering is a simple yet effective baseline strategy for enhancing robustness**, but it cannot fully eliminate the reliance. During the rebuttal period, we explored several mitigation strategies within our current architecture scope:\n\n- **Multi-agent (Refactoring agent)**: We implemented a lightweight agent that uses LLM itself (no AST tools but allows CoT thinking) to refactor the perturbed code. Then, ask for the same agent to infer.\n\n  |Setting|VAN|ALL|MDC|MPS|\n  |:----|:---:|:---:|:---:|:---:|\n  |GPT-4o (Original)|64.5|46.2|45.7|49.6|\n  |GPT-4o (After Refactoring)|-|55.3|41.1|54.9|\n\n  The performance under the ALL-perturbation setting improved, which means that **the model can identify and neutralize some code structural perturbations**, but it cannot recover to the ideal-level code. Surprisingly, in the MDC setting, **the model is still impacted by the misleading comments even when we explicitly remove instructions**, which reinforces our stance and highlights the inherent fragility.\n\n- **MCP AI Agents (mcp-clean-code)**:\n\n  ---\n  > [W4] This does not reflect the dynamic, interactive nature of how developers use AI assistants.\n  ---\n\n  Inspired by the weakness mentioned by the reviewer, we tried to explore the solution from MCP AI Agents.\n\n  |Setting|VAN|ALL|MDC|MPS|\n  |:----|:---:|:---:|:---:|:---:|\n  |Direct Inference|73.8|48.6|66.7|48.9|\n  |Direct Inference (w/ MCP)|89.8|79.7|75.6|75.6|\n  |CoT Inference|89.8|78.5|86.2|81.4|\n\n  While initially promising, MCP's inherent design targets code generation/planning, which is not suitable for code reasoning. Surprisingly, MDC and MPS underperformed basic CoT prompting, **suggesting the need for more reasoning-specific architectures**.\n\nDue to the time limit, we were unable to empirically evaluate deeper architectural modifications, such as syntax-aware encoders and tree-sitter MCP. Despite our mixed results, we believe our current explorations can demonstrate the importance of addressing NL dependency.\n\n---\n> [Q4] Dynamic Perturbations: How do time-varying perturbations (e.g., evolving codebases) impact LLM performance over continuous deployment?\n---\n\nWe acknowledge that CodeCrash is a one-shot benchmark, whereas real-world development is iterative. Our choice was deliberate: a static setting allows us to cleanly and directly assess a model’s intrinsic robustness to misleading or perturbed code, providing the community with a worst-case first impression.\n\nWe agree that robustness evaluation under evolving codebases is highly valuable. However, such a study requires a versioned dataset with realistic insertions/deletions and a protocol for repeated model interaction, which is beyond the scope of this work. Recent efforts, such as EVALOOP [1], propose a closed-loop robustness benchmark for code LLMs, showing that performance consistently declines under repeated modifications. We appreciate the reviewer's suggestion, and we view this as a natural and important extension of our work.\n\n[1] EVALOOP: Assessing LLM Robustness in Programming from a Self-consistency Perspective. Fang et. al."}, {"review_id": "Fy2PoCprqr", "reviewer": "Reviewer_5hDL", "summary": "This paper investigates the robustness of LLMs in coding reasoning tasks under perturbations by proposing a new benchmark called CodeCrash. The evaluation on 17 LLMs demonstrates that perturbation can cause significant performance degradation. This study offers insights into the limits of current benchmarks on robust code understanding.", "questions": "1. Could the framework be extended to multi programming language settings? Is there any pipeline to do that?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1. The manuscript is well written and easy to follow\n2. The proposed benchmark is meticulously curated, with clear definitions of each perturbation type.\n3. The experimental analysis is thorough and insightful, offering both broad comparisons and fine-grained breakdowns.\n\nWeaknesses:\n1. Although comprehensive, the experimental results align closely with expectations given the inherent difficulty of the original benchmarks (e.g., livecodebench), offering limited surprise for the reader.\n2. As a comprehensive benchmark, while most main stream and state-of-the-art non reasoning models are covered, the number of reasoning models covered in the benchmark is still limited. Only three are evaluated.\n3. The benchmark relies heavily on questions drawn from existing datasets, which constrain the technical contributions of the paper and real-world applicability.", "was_revised": true, "final_justification": "My concerns has been addressed. I will keep my positive score.", "rebuttal": "Thank you for your hard work reviewing! We appreciate your recognition of the strengths of our work. We will address your concerns one by one.\n\n---\n> [W1] Although comprehensive, the experimental results align closely with expectations given the inherent difficulty of the original benchmarks (e.g., livecodebench), offering limited surprise for the reader.\n---\n\nWe agree that several findings match the intuitive expectations. For example, the performance drop under perturbations (§4.2), the mitigating effect of CoT (§4.3.1), and the outstanding results of reasoning models (§4.3.2). At the same time, we would like to highlight **two unexpected findings** that add novelty beyond these trends:\n1. **Reasoning collapse in QwQ-32B**. The model entered a self-reflection loop when faced with a plausible but incorrect hint, leading to a catastrophic cognitive-dissonance failure that we did not anticipate.\n2. **Performance increase in input-prediction under MDC/MPS (§4.4)**. Misleading strings accidentally supply the model with shortcuts, **exposing a potential flaw in input-prediction tasks** and offering new evidence of unreliable code understanding.\n\n---\n> [W2] As a comprehensive benchmark, while most main stream and state-of-the-art non reasoning models are covered, the number of reasoning models covered in the benchmark is still limited. Only three are evaluated.\n---\n\nWe acknowledge that only three reasoning models were included is limited. However, the evaluation cost is substantial, testing these three models across four perturbation types (and a baseline) on 1279 problems already **exceeded $100 USD in API usage**. We prioritized breadth in non-reasoning models while still ensuring representative reasoning-model coverage. \n\n---\n> [W3] The benchmark relies heavily on questions drawn from existing datasets, which constrain the technical contributions of the paper and real-world applicability.\n---\n\nSince **our primary goal is not to introduce a brand-new dataset, but rather to stress-test LLMs in abnormal and non-ideal scenarios** and highlight concrete robustness issues in code understanding. Using established benchmarks like LiveCodeBench and CruxEval ensures that our findings are grounded in well-validated tasks.\n\nWe fully understand your concern that those comment-based perturbations are an extreme and unrealistic case. To address this realism, we design the **MPS perturbation that injects the same misleading phrases as print statements, which should never influence the output**. However, it negatively impacts model code reasoning, revealing that the **model would reference any NL cues within the code, which highlights the unreliability of the model in code reasoning**.\n\nBesides, we included a density sweep experiment in §5.1; we can see that the **Pass@1 accuracy falls linearly**, which shows the **vulnerability is systematic, not a product of extreme cases**. The sparse and accidental hints or comments could be frequently found in real repositories (e.g., CPython GitHub issue 136764).\n\n---\n> [Q1-1] Could the framework be extended to multi programming language settings?\n---\n\nTo evaluate cross-language generalization, we converted LiveCodeBench into JavaScript (similar complexity to Python). JavaScript Results (Python in parentheses):\n\n|Model|VAN|ALL|MDC|MPS|MHC|\n|:----|:---:|:---:|:---:|:---:|:---:|\n|GPT-4o (Direct)|56.0 (64.5)|-17.5 (-18.3)|-6.5 (-18.8)|-7.1 (-15.7)|-7.9 (-17.2)|\n|GPT-4o (CoT)|94.8 (94.4)|-11.7 (-7.9)|-7.3 (-6.9)|-3.3 (-6.7)|-1.0 (+0.4)|\n|Qwen2.5-72B-Instruct (Direct)|54.1 (54.9)|-17.8 (-13.9)|-9.2 (-7.0)|-10.4 (-14.4)|-24.8 (-22.3)|\n|Qwen2.5-72B-Instruct (CoT)|86.0 (90.4)|-16.9 (-17.1)|-2.7 (-5.0)|-6.7 (-14.2)|-9.0 (-7.7)|\n\n1. Baseline accuracies are **comparable across the two languages**.\n2. **Robustness patterns remain consistent**, with notable drops under ALL, MDC, MPS, and MHC perturbations.\n3. The **mitigating effect of CoT persists** across languages, especially for GPT-4o under MHC.\n\nTherefore, CodeCrash exposes language-agnostic vulnerabilities, not artifacts specific to Python. The identified weaknesses generalize across scripting languages. We have added this result to the appendix in our paper.\n\n---\n> [Q1-2] Is there any pipeline to do that?\n---\n\nWe have provided the detailed implementation description guidance for Python code in Appendix A. For the code structural perturbations, we have already provided syntax-preserved mapping templates. For the misleading comments, we saved them in \"prompt/misleading_comments.py\" in the supplementary code, and offered a function to generate those misleading comments. \n\nFor the perturbation process, we solely use the AST library to conduct the structural modification and the NL cues injection, which **can be easily transferable to other languages**, such as JavaScript using the **tree_sitter library** and Java using the **javalang library** for handling the JavaScript and Java code snippets. Because the perturbation logic is parser-agnostic and templated, extending CodeCrash to a new language is a plug-and-play operation: (1) import the grammar, (2) wire the formatter, and (3) reuse the existing passes."}, {"review_id": "ajC0EQogp6", "reviewer": "Reviewer_dLhF", "summary": "The paper introduces CodeCrash, a benchmark/framework for evaluating Code LLMs under perturbed settings. Starting from LiveCodeBench and CruxEval, the authors design code semantics-preserving perturbations that either modify code structures or textual descriptions (comments, variable names) to obtain perturbed task problems. The authors present systematic evaluations across settings and show that non-reasoning models degrade performance under the perturbed settings. Reasoning language models are able to not only achieve very high performance under the non-perturbed setting but also be more robust to perturbations. Finally, the authors present detailed discussions around three settings — varying NL comments density in textual perturbations, prompting to ignore misleading comments and identifying self-reflection loops in reasoning models.", "questions": "1. How do the perturbations discussed in this paper relate to those in recent code perturbation works such as CodeFort or ReCode? Are there new findings compared to those from these works?\n2. On reasoning models evaluations, do Section 5.3 findings hold even for the VAN setting? In other words, is the self-reflection collapse surely due to the perturbation?\n3. Is it possible to identify adversarial scenarios in real-world code? For e.g., code with arbitrary comments, missing imports, unintelligible variable names, etc.? This would ground the research in more practical settings.\n4. Are there recommendations for improving robustness under various perturbation settings?", "limitations": "No limitations are discussed in the paper. I recommend the authors to address the above.\n\nThere are no ethical considerations around this work however.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe paper discusses a comprehensive set of perturbations which reveals gaps in LLMs’ understanding of code. Further, the paper supports the superiority of reasoning language models and their (current) tendency to overthink or enter into self-reflection loops.\n\n**Weaknesses**\n\n- **Insufficient contributions relative to literature** Evaluating code LLMs under perturbed settings such as those described in the paper is not new. The paper only presents findings that newer small models continue to suffer from this problem while larger or reasoning models tend to suffer less. This is also, in my opinion, largely known to the community [4]. Further, the benchmark is merely derived from LiveCodeBench and CruxEval by introducing perturbations that are unrealistic or not representative of real world use cases (such as introducing misleading hints). From an analysis perspective, I understand that this is a controlled experimental setting but the analysis itself is also incomplete in my opinion. Some other works around code robustness such as CodeFort [1] attempt to make models more robust, while the current paper over emphasizes on a largely known problem without providing possible fixes. Finally, the finding that reasoning models may overthink is not a new finding either [2, 3].\n- **Paper writing** Given the above, I feel the paper is overly verbose in what needs to be conveyed. Most of the paper writing feels like verbalizing the results in the tables. The paper space can better be utilized by discussing more insightful experiments such as identifying adversarial patterns in real-world code (from GitHub, etc.) and how models behave under such scenarios or how to improve robustness for both reasoning and non-reasoning models. The discussions in Section 5, though insightful, are not significant enough, in my opinion, to merit a full-paper at NeurIPS.\n- **Misc.**\n    - Line 78: in → on\n    - In the present form, the main text makes several references to core results in the Appendix. I feel that the main text should be self-sufficient to the reader and any references to Appendix material should be optional for the reader to look up; this is not possible in the current write-up where I cannot follow some discussions (e.g., Section 4.3.1) without going to the Appendix.\n\n[1] CodeFort: Robust Training for Code Generation Models. https://arxiv.org/pdf/2405.01567\n\n[2] Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs. https://arxiv.org/abs/2412.21187v2\n\n[3] Thinking LLMs: General Instruction Following with Thought Generation. https://arxiv.org/abs/2410.10630v1\n\n[4] ReCode: Robustness Evaluation of Code Generation Models. https://arxiv.org/pdf/2212.10264", "was_revised": true, "final_justification": "The authors have highlighted that their main goal is to stress-test LLMs under adversarial scenarios concerning code reasoning. From that perspective, their work builds on existing robustness work before code reasoning was prevalent. As such, this paper presents timely findings for robustness of code reasoning.", "rebuttal": "Thank you for your hard work reviewing our paper! We appreciate your comments for raising important points on novelty, realism of perturbations, the interpretation of reasoning versus memorization, and providing your feedback on writing clarity. Below we address these concerns in detail.\n\n---\n> [W1] The paper only presents findings that newer small models continue to suffer from this problem while larger or reasoning models tend to suffer less. This is also, in my opinion, largely known to the community [4].\n\n> [Q1] How do the perturbations discussed in this paper relate to those in recent code perturbation works such as CodeFort or ReCode? Are there new findings compared to those from these works?\n---\n\nExisting robustness suites and defenses study **generation-context perturbations**, edit the NL task description [1,2,3], or partial code inside the prompt [1,4] that merely serves as context for a code completion task. In contrast, our work is the first benchmark to investigate the **direct code understanding robustness**, where the model must interpret and execute a complete program. This setting demands **semantic analysis of control-flow and data-flow, and the ability to follow the true code logic**.\n\nCode generation/completion and code understanding are related, but they test **two fundamentally different capabilities**. Robustness in code generation primarily measures resilience against prompt variations (NL to Code), while **robustness in code understanding assesses whether a model can correctly analyze and explain the code logic under perturbed conditions (Code to NL)**. Therefore, evidence from code generation robustness does not invalidate our findings on code understanding robustness.\n\nFurthermore, prior works perturb only a **single self-consistent pattern** (e.g., variable renames, death loop inserts, which focus on code logic); **our work superimposes two contradictory yet valid patterns (i.e., program logic vs. misleading comments)**. This novel setup exposes a weakness that models systematically over-trust textual cues. **Two valid but contradictory patterns exist simultaneously, causing the model to fail to reconcile and collapse, which has not been seen in earlier works**.\n\n[1] ReCode: Robustness Evaluation of Code Generation Models. Shiqi Wang, et. al.\n\n[2] NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations. Junkai Chen, et. al.\n\n[3] On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex. Terry Yue Zhuo, et. al.\n\n[4] CCTEST: Testing and Repairing Code Completion Systems. Zongjie Li, et. al.\n\n---\n> [W2] Further, the benchmark is merely derived from LiveCodeBench and CruxEval by introducing perturbations that are unrealistic or not representative of real world use cases.\n\n> [W4] The paper space can better be utilized by discussing more insightful experiments such as identifying adversarial patterns in real-world code (from GitHub, etc.) and how models behave under such scenarios or how to improve robustness for both reasoning and non-reasoning models. \n---\n\nWe fully understand your concern that the perturbations introduced in our benchmark may appear extreme or not directly representative of typical real-world scenarios. We would first like to clarify that our primary goal is explicitly stress-testing LLMs' **capability boundaries in code reasoning, and reversing the NL dependencies of LLMs**. Our perturbations are carefully designed to reveal robustness issues that would remain hidden in traditional evaluation.\n\nTo address this realism, we design the **MPS perturbation that injects the same misleading phrases as print statements, which should never influence the output**. However, it negatively impacts model code reasoning, revealing that the **model would reference any NL cues within the code**, which **highlights the unreliability of the model in code reasoning**.\n\nBesides, we included a density sweep experiment in §5.1; we can see that the **Pass@1 accuracy falls linearly**, which shows the **vulnerability is systematic, not a product of extreme cases**. The sparse and accidental hints or comments could be frequently found in real repositories (e.g., CPython GitHub issue 136764).\n\n---\n> [W3] Finally, the finding that reasoning models may overthink is not a new finding either [2, 3].\n\n> [Q2] On reasoning models evaluations, do Section 5.3 findings hold even for the VAN setting? In other words, is the self-reflection collapse surely due to the perturbation?\n---\n\nWe agree that overthinking has already been discovered in prior works, and we also realize that models will overthink and self-reflect under non-ideal coding situations. However, **our study uncovers a qualitatively different failure mode --- reasoning collapse**. This phenomenon **solely exists in MHC perturbation and happened 4 times, which is not unique and coincidental**. Therefore, answering Q2, reasoning collapse does not hold for the VAN setting, and it is **surely due to the MHC perturbation**. We respectfully note that we have already provided a detailed analysis of this phenomenon in §4.3.2 and §5.3 (and Case study 6). We have manually segmented the collapsed thinking into six stages of gradual confidence erosion in one case in the Appendix (pp. 39-40). Please feel free to see how the model gradually collapses!\n\n---\n> [Q3] Is it possible to identify adversarial scenarios in real-world code? For e.g., code with arbitrary comments, missing imports, unintelligible variable names, etc.? This would ground the research in more practical settings.\n---\n\nWe implemented a lightweight **refactoring agent** that uses LLM itself (no AST tools but allows CoT thinking) to **(i) delete misleading comments, (ii) rename unintelligible identifiers, (iii) add missing imports, and (iv) strip dead code**. Hence, if the performance increased on the refactored code, it means the model can identify those issues.\n\nHere are the results on LiveCodeBench:\n\n|Setting|VAN|ALL|MDC|MPS|\n|:----|:---:|:---:|:---:|:---:|\n|GPT-4o (Original)|64.5|46.2|45.7|49.6|\n|GPT-4o (After Refactoring)|-|55.3|41.1|54.9|\n\nThe performance under the ALL-perturbation setting improved, which means that **the model can identify and neutralize some code structural perturbations**, but it cannot recover to the ideal-level code.\n\nSurprisingly, in the MDC setting, **the model is still impacted by the misleading comments even when we explicitly remove instructions**, which reinforces our stance and highlights the inherent fragility.\n\n---\n> [Q4] Are there recommendations for improving robustness under various perturbation settings?\n---\n\nIn our paper (§5.2), we confirmed that **prompt engineering** is a simple yet effective baseline strategy for enhancing robustness, but it cannot fully eliminate the reliance. During the rebuttal period, we explored the solution from **MCP AI Agents (mcp-clean-code)**:\n\n  |Setting|VAN|ALL|MDC|MPS|\n  |:----|:---:|:---:|:---:|:---:|\n  |Direct Inference|73.8|48.6|66.7|48.9|\n  |Direct Inference (w/ MCP)|89.8|79.7|75.6|75.6|\n  |CoT Inference|89.8|78.5|86.2|81.4|\n\nWhile initially promising, MCP's inherent design targets code generation/planning, which is not suitable for code reasoning. Surprisingly, MDC and MPS underperformed basic CoT prompting, suggesting the **need for more reasoning-specific architectures**.\n\nDue to the time limit, we were unable to empirically evaluate deeper architectural modifications, such as syntax-aware encoders and tree-sitter MCP. Despite our mixed results, we believe our current explorations can demonstrate the importance of addressing NL dependency.\n\n---\n> [Misc 1] Line 78: in → on\n---\n\nThank you for pointing that out. We have corrected it.\n\n---\n> [Misc 2] In the present form, the main text makes several references to core results in the Appendix. I feel that the main text should be self-sufficient to the reader and any references to Appendix material should be optional for the reader to look up; this is not possible in the current write-up where I cannot follow some discussions (e.g., Section 4.3.1) without going to the Appendix.\n---\n\nThank you for pointing out this issue. We apologize for the inconvenience during your reading. Owing to the page limit, we originally moved the result tables for §4.3.1 and §4.4 to the Appendix, which indeed forced readers to flip pages.\n\nWe **have already restored both tables to the main text** in our latest version. To recover the necessary 0.5 page, we've done the following revision:\n1. Since Appendix A already provides low-level descriptions of each perturbation, we use two concise, high-level paragraphs to summarize the code-structural and textual-distraction perturbations, respectively, rather than using 8 separate paragraphs.\n2. We moved the dataset descriptions (§3.1) to Appendix A, while adding a one-sentence summary plus a navigation in §3.2 (execution pipeline).\n     These changes keep the paper within the 9-page limit while readers can grasp the methodology at a high level and see all core results without flipping to the Appendix.\n\n---\n> [Limitation 1] No limitations are discussed in the paper.\n---\n\nThere is an oversight on our side that the limitation section appears in Appendix F (p. 40), and we realise that we did not mention this Appendix section in the main text. We have already included a sentence with explicit forward and backward links to the limitation section in the discussion section, so that readers who want details can navigate effortlessly."}, {"review_id": "mEcKUrF1qT", "reviewer": "Reviewer_RETr", "summary": "Based on my understanding, this paper explores how large language models (LLMs) handle code reasoning by proposing a new benchmark that uses functionally equivalent programs (FEPs). The key idea is that instead of relying on exact string match or canonical reference implementations, which can be brittle, the authors introduce evaluation metrics that assess whether two code snippets behave the same across diverse test inputs. This seems especially relevant for tasks like program repair or completion, where multiple correct solutions may exist.", "questions": "See weaknesses.", "limitations": "No.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "appreciate the motivation, the LLMs often get penalized for producing code that doesn’t match ground truth but still works perfectly fine. So bringing in FEPs to test “true” reasoning ability is quite reasonable. The benchmark design seems thoughtful, with an emphasis on creating clusters of functionally equivalent but syntactically diverse code. Also, the authors make an effort to evaluate a wide range of LLMs, and the observation that some models fall short in generalizing across code variations is a useful insight.\n\nWeaknesses:\n\nI had several concerns while reading. The main one is that the overall novelty feels limited. The idea of using program equivalence (or behavioral testing) for evaluating code generation has been explored in past work. For example, many code generation benchmarks already incorporate execution-based metrics or fuzzing strategies. While this paper pushes that line further with FEP clusters, the contribution feels more incremental than foundational.\n\nSecond, the construction of FEP clusters (although clearly non-trivial), is not entirely convincing in terms of scale and coverage. The paper mentions hundreds of examples, but it’s not clear how diverse or comprehensive the test suite is across different programming concepts or complexity levels. It feels like the authors focus more on surface diversity rather than semantic or algorithmic depth, which limits the benchmark’s ability to reveal deeper reasoning gaps.\n\nFinally, the link between “reasoning” and “failing to generalize across FEPs” is a bit hand-wavy. In many cases, LLMs might just rely on memorized patterns or local statistical features, so failing on a FEP variant doesn’t necessarily mean a failure of reasoning. As it could be a lack of exposure. I would have liked to see clearer criteria or ablation studies to isolate reasoning vs memorization effects.", "was_revised": true, "final_justification": "The authors address most of my concerns.", "rebuttal": "We appreciate your thoughtful feedback, particularly regarding our benchmark’s novelty, the scale breadth of the FEP clusters, and the distinction between reasoning and memorization. For your highlighted concerns, we will address them one by one.\n\n---\n> [W1] The main one is that the overall novelty feels limited. The idea of using program equivalence (or behavioral testing) for evaluating code generation has been explored in past work ... While this paper pushes that line further with FEP clusters, the contribution feels more incremental than foundational.\n---\n\nAppreciating your thoughtful concerns about novelty, we acknowledge that the idea of using program equivalence for evaluating code generation has been explored before [1,2,3,4], and we indeed consider an incremental exploration based on the previous works [4,5,6]. However, **existing robustness suites and defenses typically study generation-context perturbations, modifying either the natural-language task description [1,2,3] or partial code context within prompts [1,4]**.\n\nIn contrast, our work **uniquely focuses on code understanding robustness**, where the model must directly interpret and execute a complete program. While generation tasks focus on the capability of NL to Code, our setting demands semantic analysis of control-flow and data-flow, and explicit understanding of code logic, targeting further tasks like Code Explanation (i.e., **Code to NL capability**). Therefore, existing results on code generation robustness do not diminish our contributions toward the understanding robustness problem.\n\nAdditionally, prior robustness evaluations usually use **a single, internally self-consistent perturbation pattern**. For example, modifying code structure (e.g., variable renaming, inserting dead loops) or editing task descriptions (e.g., introducing grammatical errors or misspellings). In contrast, our benchmark introduces **two contradictory yet valid patterns simultaneously** (program logic vs. misleading natural-language cues). This novel setup uncovers a previously undiscussed weakness--**models systematically over-rely on textual cues (not just comments)**, causing them to fail or even collapse under contradictions (§4). **This phenomenon, to the best of our knowledge, has not been previously reported.**\n\nFurthermore, our experiments reveal other surprising findings:\n\n1. In §4.3.2, **QwQ-32B repeatedly enters a self-verification loop**, characterized by a quadratic and gradual reduction of confidence and eventual reasoning collapse (output of the same tokens). It is triggered simply by one incorrect authoritative comment. Our detailed analysis in §5.3 and Appendix E.6 confirms this collapse is systematic rather than coincidental. \n2. In §4.4, **misleading textual cues cause models to shortcut reasoning in input prediction tasks**. While input prediction is inherently more challenging than output prediction, its design ironically permits models greater susceptibility to superficial textual shortcuts.\n\nThese nuanced insights further illustrate the unique contribution of our work.\n\n[1] ReCode: Robustness Evaluation of Code Generation Models. Shiqi Wang, et. al.\n\n[2] NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations. Junkai Chen, et. al.\n\n[3] On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex. Terry Yue Zhuo, et. al.\n\n[4] CCTEST: Testing and Repairing Code Completion Systems. Zongjie Li, et. al.\n\n[5] CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution. Alex Gu, el. al.\n\n[6] LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. Namin Jain, et. al.\n\n---\n> [W2] Second, the construction of FEP clusters (although clearly non-trivial), is not entirely convincing in terms of scale and coverage.\n---\n\nOur design goal with the FEP clusters was specifically targeted at evaluating code understanding robustness under two distinct dimensions: structural perturbations and textual perturbations through code execution and input prediction tasks. Thus, our benchmark **emphasizes variability in these robustness-relevant dimensions, rather than aiming at general algorithmic complexity**. Nevertheless, we built upon two complementary datasets: synthetic algorithmic tasks (CRUXEval) and real-world problems (LiveCodeBench), covering multiple complexity levels and common Python constructs. We also provided a balanced and cross-dataset comparison for all experiments.\n\n---\n> [W3-1] Finally, the link between “reasoning” and “failing to generalize across FEPs” is a bit hand-wavy. In many cases, LLMs might just rely on memorized patterns or local statistical features.\n---\n\nRegarding your concern about the separation between memorization and reasoning. We consolidate the evidence already in the paper and add a new shuffle ablation that directly addresses this point.\n\nIn §4.2 (or Table 1), we design an **RTF perturbation** to rewrite easy branch conditions into indirect and infrequently seen expressions, **eliminating the common lexical pattern**. However, all models maintain a similar level of accuracy in the VAN setting, indicating that they can correctly reason over unfamiliar syntactic forms.\n\nIn §4.3, we conduct **CoT prompting** on every model to evaluate the code reasoning robustness again, while prior work shows CoT reduces shortcut behaviour and pattern reliance [7], **model performance still significantly dropped** under ALL, MDC, and MPS perturbations. Thus,  the errors persist after exposing the model's full reasoning trace, **contradicting a pure memorisation or lack-of-exposure hypothesis**.\n\nIn §5.2, we add the **\"ignore comment\" instruction** to the prompt, but the performance of DeepSeek-V3, GPT-4o, and Qwen2.5-72B-Instruct increased, but still **cannot recover their accuracy to VAN-level** under MDC perturbation even using CoT prompting. This shows that the **model does not blindly map comments to answers** (otherwise, the instruction would fully fix the error). The performance gap points to a reasoning failure when NL cues contradict program logic.\n\n[7] Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning. Akshara Prabhakar, et. al.\n\n---\n> [W3-2] I would have liked to see clearer criteria or ablation studies to isolate reasoning vs memorization effects.\n---\n\nTo further verify the separation between pattern matching and reasoning, we conduct a **new shuffle ablation** by removing lexical alignment.\n\nIn particular, we inject misleading comments specifically to different key AST nodes (which are described in Appendix A5, pp. 20), as we want to enlarge the contradiction between the code logic and misleading comments. **In this new ablation experiment, we shuffle the comment positions**, such as we inject the misleading comments designed for loops into function declarations.\n\n|Model|VAN|MDC|MDC(Shuffled)|\n|:---|:--:|:--:|:---:|\n|GPT-4o|64.5|45.7|54.9|\n|LLaMA-3.3-70B-Ins|48.5|38.7|42.0|\n|Qwen2.5-72B-Ins|54.9|48.0|52.4|\n\n- Observation 1: Accuracy improves when alignment is broken\n- Observation 2: Accuracy is still below the VAN-level.\n\nThese observations indicate that the models are not simply doing memorized or lexical pattern-matching. **If they were blindly keyed to comments, shuffling the comments would not help; if they only followed code patterns, accuracy would remain the same because the code logic is unchanged.** Instead, shuffling breaks the line-level alignment between code and comments, leading the model to reason about their inconsistency and place greater trust in the actual code logic, hence the observed improvement, but still below the vanilla baseline."}]

Author rebuttal:

Meta-review: Thanks for submitting to NeurIPS 2025. The manuscript introduces CodeCrash, a benchmark for code LLM stress testing under structural and textual distractions. The benchmark introduces 7 main types of distractions for code, spanning across code structural and textural domains, and expands beyond the scope of existing code robustness studies. By evaluating on mainstream commercial and open source models, the study reveals a few interesting and timely findings such as the over-reliance of models on textual/comment information, and late-minute distraction by text in CoT/reasoning models. After the discussion phase, all reviewers are satisfied with the response and additional studies that resolve major concerns. Reviewers agree on borderline acceptance with consensus due to the timely findings for the community. However, there are some remaining concerns about lack of novelty, in-depth mechanistic explanation, and the limited number of studied subject models. Overall, we are happy to see this work at the conference. Congratulations on the acceptance. Authors should ensure that they incorporate the feedback and responses in their revisions, particularly in-depth analysis.
Decision: Accept (poster)