Title: CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Code Reasoning

Code Execution

Input-Output Prediction

Robustness Evaluation

Code Perturbations

Model Reliability

Adversarial Testing
Abstract: Large Language Models (LLMs) have recently demonstrated strong capabilities in code-related tasks, but their robustness in code reasoning under perturbations remains underexplored. We introduce CodeCrash, a stress-testing framework with 1,279 questions from CRUXEVAL and LIVECODEBENCH, designed to evaluate reasoning reliability under structural perturbations and misleading natural language (NL) contexts. Through a systematic evaluation of 17 LLMs, we find that models often shortcut reasoning by over-relying on NL cues, leading to an average performance degradation of 23.2% in output prediction tasks. Even with Chain-of-Thought reasoning, models on average still have a 13.8% drop due to distractibility and rationalization, revealing a lack of critical reasoning capability to distinguish the actual code behaviors. While Large Reasoning Models with internal reasoning mechanisms improve robustness by fostering critical thinking, plausible yet incorrect hints can trigger pathological self-reflection, causing 2-3 times token consumption and even catastrophic cognitive dissonance in extreme cases for QwQ-32B. We refer to this phenomenon as Reasoning Collapse. CodeCrash provides a rigorous benchmark for evaluating robustness in code reasoning, guiding future research and development toward more reliable and resilient models.

CODECRASH: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning

Abstract

Large Language Models (LLMs) have recently demonstrated strong capabilities in code-related tasks, but their robustness in code reasoning under perturbations remains underexplored. We introduce CODECRASH, a stress-testing framework with 1,279 questions from CRUXEVAL and LIVECODEBENCH, designed to evaluate reasoning reliability under structural perturbations and misleading natural language (NL) contexts. Through a systematic evaluation of 17 LLMs, we find that models often shortcut reasoning by over-relying on NL cues, leading to an average performance degradation of 23 . 2 % in output prediction tasks. Even with Chain-of-Thought reasoning, models on average still have a 13 . 8 % drop due to distractibility and rationalization , revealing a lack of critical reasoning capability to distinguish the actual code behaviors. While Large Reasoning Models with internal reasoning mechanisms improve robustness by fostering critical thinking, plausible yet incorrect hints can trigger pathological self-reflection , causing 2 -3 times token consumption and even catastrophic cognitive dissonance in extreme cases for QwQ-32B. We refer to this phenomenon as Reasoning Collapse . CODECRASH provides a rigorous benchmark for evaluating robustness in code reasoning, guiding future research and development toward more reliable and resilient models.

1 Introduction

Large Language Models (LLMs), exemplified by GPT families (OpenAI et al., 2024; Hurst et al., 2024) and the DeepSeek series (DeepSeek-AI et al., 2025b,a), have recently exhibited remarkable performance across diverse code-related tasks, including code generation (Zan et al., 2023; Liu et al., 2023b; Xiao et al., 2024), code completion (Ding et al., 2023), program repair (Fan et al., 2023; Liu et al., 2025), program testing (Deng et al., 2023; Kang et al., 2023), and test case generation (Steenhoek et al., 2025; Takerngsaksiri et al., 2025). These advancements have facilitated the incorporation of LLMs into Integrated Development Environments, such as GitHub Copilot (GitHub, 2022), OpenAI Codex (Chen et al., 2021), and Tabnine (Tabnine, 2023), highlighting their potential to automate various aspects of software development (Zhang et al., 2023; Pandey et al., 2024). The effectiveness of these applications heavily depends on the model's ability to comprehend the program logic, underlying functionality, and developer intent (Pan et al., 2024), but real-world codebases are

often disorganized, with ambiguous identifiers (Gao et al., 2023), garbage code (Li et al., 2023), and inconsistent comments (Rong et al., 2025). Those noises can mislead models into misinterpretations and incorrect inferences, compromising the model's reliability in downstream tasks.

Traditional robustness studies and obfuscators have primarily focused on code structure mutations, such as renaming identifiers, modifying control flow, or inserting unreachable code (Wang et al., 2022; Wan et al., 2022; Li et al., 2023). While they are effective for assessing model pattern-matching, these variations remain at the structural level. On the other hand, natural language (NL) perturbations have mainly been explored in NL-to-Code tasks ( e.g. , code generation and completion) by editing the task descriptions (Wang et al., 2022; Zhuo et al., 2023; Mastropaolo et al., 2023; Chen et al., 2024a), which evaluates prompt sensitivity rather than robustness in code reasoning. To the best of our knowledge, CODECRASH is the first benchmark to systematically utilize NL-embedded misleading perturbations to evaluate whether LLMs can prioritize executable semantics over NL cues in code reasoning and understanding.

To diagnose whether LLMs truly understand program logic, we adopt input and output predictions from CRUXEVAL (CRUX) (Gu et al., 2024) and extend them with LIVECODEBENCH (LCB) (Jain et al., 2025) to cover real-world programs. We categorize our NL-embedded perturbations into contextual-level (obviously incorrect cues in multiple formats) and reasoning-level (plausible but incorrect hints inducing rationalization behaviors (Chen et al., 2025b)), with the goal of stresstesting the critical reasoning capability of LLMs. Furthermore, building upon program structureconsistent (PSC) mutations from CCTEST (Li et al., 2023) and other perturbations ( e.g. , dead-loop poisoning) (Wan et al., 2022; Sun et al., 2023), we construct an aggregated structural perturbation that integrates all traditional structural perturbations for representative comparison.

We perturb questions in CRUX and LCB, and evaluate seventeen LLMs in the direct inference setting, observing significant performance degradation across all designed perturbations. We then extend our evaluation to Chain-of-Thought (CoT) (Kojima et al., 2022) reasoning, and observe the phenomena of distractibility (Shi et al., 2023) and rationalization (Barez et al., 2025) under contextual-level and reasoning-level perturbations, respectively, exposing the insufficiency of critical thinking in semantic tracing and code understanding. Therefore, we further assess three advanced Large Reasoning Models (LRMs), which achieve superior robustness because of their enhanced internal reasoning. However, we find that LRMs are trained to be overly cautious of uncertainty : contextual-level perturbations increase reasoning tokens to digest the obviously misleading cues (Kumar et al., 2025), while reasoning-level perturbations exploit this bias to trigger pathological overthinking and even ' collapse ' in QwQ-32B, constituting a novel failure mode. In addition, we accidentally identify a potential limitation in input prediction for evaluating code comprehension, where models exploit those misleading cues and bypass the core execution logic without understanding the code.

Our contributions include:

We introduce a perturbation framework on code reasoning tasks that covers both structural and NL-embedded code transformation for stress-testing LLM code reasoning robustness .

We reveal that LLMs lack sufficient critical reasoning and exploit superficial cues to shortcut reasoning and rationalize comments, extending prior findings on rationalization to code reasoning.

We identify a pathological self-reflection phenomenon in LRMs and novel Reasoning Collapse failure in QwQ-32B, arising from residual rationalization that conflicts with reason-faithful objectives.

2 Functionally Equivalent Perturbations

2.1 Benchmark Execution Pipeline

CODECRASH provides a unified perturbation-evaluation pipeline, as illustrated in Figure 1. Each code snippet is initially processed via Abstract Syntax Tree (AST) parsing and unparsing to standardize formatting, producing a clean 'vanilla' baseline (VAN). Subsequently, we apply a selected perturbation and regenerate the code through AST parsing, ensuring that replacement, reformatting, and insertion are precisely controllable. Each perturbed code snippet is executed with the provided inputs to double-check the syntactic validity and functional correctness. The validated and perturbed code is then provided to LLMs, where we record their generated responses, extract the answers, and execute them to compute the PASS@1 accuracy (Chen et al., 2021).

Var_1 = 19

def f(Var_1):

if _ := (Var_1<0,)[0]:

Var_1 *= -1

def funct5():

Var_2 = 1

while True:

Var_2 += 1

while 0:

TempVar_1 = Var_1

return Var_1

Perturbations:

Renamed Identifiers

Reformatted Conditions

Garbage Global Variables

Unexecutable Dead Loops

Unexecutable Segments

Targeted AST Nodes:

Function Declarations

Return Statements

Loop Constructs

If Statements

Variable Assignments

Arithmetic Operations

Data Structure Operations

- List: append, ...

- Set: extend, ...

- String: replace, ...

"Function behavior is independent of

parameters." ...

"The function always returns zero" ...

"This condition is always True" ...

The '{variable}' variable is initialized but

never populated." ...

...

def my_abs(a):  # Function behavior is independent ...

if a < 0:  # This condition is always True.

...

...

print('This step can be ignored during.')

a *= -1

print('This function directly outputs zero.')

return a

Misleading Code Comments (MCC)

Misleading Print Statements (MPS)

Aggregated Structural

Perturbation

Contextal-Level Misleading Perturbations

Given the code. Modify the input

argument(s) to make

it INCORRECT. The purpose is to

mislead people to get a correct

answer.

my_abs(a = -2) == 2

my_abs(a = -2) == -2

Reasoning-Level

Misleading Perturbation

Structure and

type preserved

def my_abs(a):

if a < 0:

a *= -1

return a

GPT-4o

Search for return

statements and inject

Search for the targeted AST

node and inject the message

Generating

misleading but generic

messages

Vanilla

PSC-ALL

def my_abs(a):

if a < 0:

a *= -1

return a

# The return value is -2

Misleading Hints (MHC)

2.2 Perturbation Strategies

We design a comprehensive perturbation framework that includes structural , contextual-level , and reasoning-level perturbations to systematically stress-test LLMs' code reasoning reliability. Detailed illustrations of each perturbation are provided in Appendix B.

Aggregated Structural Perturbation (PSC-ALL). We adapt PSC mutations from CCTEST (Li et al., 2023) and dead-loop poisoning (Wan et al., 2022; Sun et al., 2023) by aggregating them on the identifier-level ( i.e. , Renaming Entities, REN), instruction-level ( i.e. , Reformatting Conditional Expressions, RTF), and block-level ( i.e. , Garbage Code, GBC; including unexecutable dead loops, dead blocks, and global variables). PSC-ALL constructs a more complicated but functionally equivalent program and represents the traditional structural perturbation that modifies syntax to expose the pattern-matching biases of LLMs. For completeness, we specify REN, RTF, and GBC in Appendix B.1.1-B.1.3, and highlight their compounding effects in Appendix B.1. Since their impact is consistently weaker than PSC-ALL, we focus our experiments on PSC-ALL.

Contextual-Level Misleading Perturbations (MCC & MPS). These perturbations aim to mislead models by injecting shallow NL cues into the surface context without altering the code logic. We target eight critical and ordinary AST nodes ( e.g. , function definitions, returns, and loops) and leverage GPT-4o to generate the misleading messages, which we manually filter to ensure they are generic, explicitly incorrect, and contradictory to the code logic . Concretely, we design two variants: (1) Misleading Code Comments (MCC), where we inject messages as code comments; and (2) Misleading Print Statements (MPS), where we embed messages via executable print statements. They evaluate whether LLMs naively include the irrelevant NL cues from the program for reasoning, using different injection formats to clarify that the phenomenon is not bound to a specific format.

Reasoning-Level Misleading Perturbation (MHC). We further design the Misleading Hint Comments (MHC) perturbation, which differs from contextual-level perturbations by providing high-level incorrect hints about program outputs. We instruct GPT-4o to generate a plausible but incorrect answer that preserves the expected type and structure but contradicts the ground truth. After verifying the incorrectness, we utilize AST parsing to strategically inject it as a comment at function definitions or return statements to make it more confusing. Unlike shallow contradictions from contextual perturbations, MHC targets the reasoning process, assessing whether the model can critically evaluate conflicting information rather than rationalize the hint and shortcut its reasoning.

3 Experimental Results and Analysis

3.1 Experiment Settings

Model Selection. We evaluate 17 models of varying sizes and versions, including both open-source and closed-source LLMs: GPT family (GPT-4o-Mini, GPT-4o) (Hurst et al., 2024), Anthropic

Claude (Claude-3.5-Haiku, 3.5-Sonnet) (Anthropic, 2024b,a), Google Gemini (Gemini-1.5-Flash, 2.0-Flash, 1.5-Pro) (Pichai and Hassabis, 2024; Pichai et al., 2024), DeepSeek-V3 (DeepSeek-AI et al., 2025b), Alibaba Qwen (Qwen2.5-7B-Ins, 14B-Ins, 32B-Ins, 72B-Ins, 32B-Coder-Ins) (Yang et al., 2025; Hui et al., 2024), and Meta LLaMA (LLaMA-3.1-8B-Ins, 70B-Ins, 405B-Ins, LLaMA3.3-70B) (Grattafiori et al., 2024). Additionally, we assess 3 cutting-edge LRMs: o3-mini (OpenAI, 2025), DeepSeek-R1 (DeepSeek-AI et al., 2025a), and QwQ-32B (Qwen Team, 2025).

Model Configurations. Following LCB, we use nucleus sampling (temperature=0.2, top-p=0.95) with a maximum of 200 tokens for direct inference and 2,000 for CoT prompting, discarding excessively long outputs. Due to resource constraints, we generate three candidates for direct inference and one for CoT inference; we provide a result stability analysis under N = 3 in Appendix C. Following CRUX, we employ 2-shot prompting for direct inference and 1-shot for CoT step-by-step execution (details provided in Appendix D). Furthermore, we adopt PASS@1 accuracy as the primary evaluation metric and scale all scores in [0 , 1] to percentages by multiplying by 100 for readability. The experimental results are reported using relative differences from the corresponding vanilla baseline ( ∆ % = Perturbed -VAN VAN × 100% ). We also report the absolute differences for completeness in the Appendix E.

3.2 Severe Degradation under Perturbations

We begin by evaluating models in the direct inference setting to measure their robustness against perturbations in §2.2 and summarize the results in Table 1. On average, models suffer a 23 . 2 % performance drop, with the aggregated structural perturbation (PSC-ALL) causing the largest degradation ( 24 . 6% ). This indicates that LLMs exhibit limited code-tracing robustness, struggling to identify important code from essential logic and maintain reliability.

LLMs are sensitive to embedded NL cues. As shown in Table 1, contextual-level perturbations (MCC and MPS) degrade performance by about 24% on average, comparable to PSC-ALL, highlighting their substantial impact on model robustness. This also reveals that models are unable to consistently prioritize program logic over misleading NL cues, instead naively incorporating those cues into their reasoning. Notably, we identify two prominent consistency patterns: (1) dataset-level consistency 2 ( 15 / 17 models), implying the impact intensity and sensitivity of both injection formats primarily depend on the dataset characteristics; and (2) perturbation-method consistency 3 ( 14 / 17 models), indicating format-specific cognitive preference and biases. These phenomena suggest that LLMs are inherently impacted by embedded NL context, regardless of the injection format.

LLMs rationalize the hint to shortcut their reasoning. Moving forward to the reasoning-level perturbation, MHC induces significant degradation with a complexity-amplified effect : models

Model Series

VAN

PSC-ALL

MCC

MPS

MHC

Average

Direct

CoT

Direct

CoT

Direct

CoT

Direct

CoT

Direct

CoT

Direct

CoT

GPT-4o-Mini

55

.

8

81

.

5

-

29

.

4%

-

13

.

1%

-

35

.

5%

-

10

.

8%

-

25

.

0%

-

11

.

0%

-

23

.

8%

-

2

.

7%

-

28

.

4%

-

9

.

4%

GPT-4o

68

.

8

91

.

8

-

20

.

0%

-

4

.

9%

-

19

.

6%

-

5

.

0%

-

19

.

9%

-

5

.

6%

-

14

.

0%

-

1

.

4%

-

18

.

4%

-

4

.

2%

Claude Series

Claude-3.5-Haiku-20241022

57

.

7

72

.

9

-

28

.

9%

-

21

.

2%

-

12

.

4%

-

10

.

6%

-

10

.

0%

-

8

.

7%

-

37

.

2%

-

14

.

2%

-

22

.

1%

-

13

.

7%

Claude-3.5-Sonnet-20241022

72

.

3

86

.

0

-

22

.

0%

-

7

.

4%

-

8

.

7%

-

5

.

3%

-

9

.

4%

-

6

.

3%

-

25

.

3%

-

7

.

8%

-

16

.

3%

-

6

.

7%

Gemini Series

Gemini-1.5-Flash

51

.

7

75

.

2

-

16

.

3%

-

18

.

3%

-

21

.

3%

-

21

.

6%

-

32

.

9%

-

42

.

1%

-

12

.

9%

-

2

.

1%

-

20

.

8%

-

21

.

0%

Gemini-2.0-Flash

65

.

4

89

.

1

-

36

.

6%

-

6

.

2%

-

36

.

2%

-

6

.

3%

-

39

.

6%

-

14

.

1%

-

12

.

5%

-

2

.

0%

-

31

.

2%

-

7

.

1%

Gemini-1.5-Pro-002

63

.

2

87

.

2

-

21

.

1%

-

7

.

4%

-

26

.

7%

-

11

.

9%

-

25

.

9%

-

14

.

6%

-

18

.

4%

-

3

.

9%

-

23

.

0%

-

9

.

4%

DeepSeek

DeepSeek-V3

67

.

8

89

.

5

-

21

.

4%

-

9

.

9%

-

25

.

9%

-

17

.

6%

-

19

.

1%

-

18

.

7%

-

17

.

8%

-

3

.

5%

-

21

.

1%

-

12

.

4%

LLaMA Series

LLaMA-3.1-8B-Instruct

35

.

5

44

.

7

-

22

.

2%

-

27

.

6%

-

24

.

7%

-

21

.

1%

-

19

.

8%

-

21

.

4%

-

25

.

1%

-

9

.

0%

-

23

.

0%

-

19

.

8%

LLaMA-3.1-70B-Instruct

51

.

9

69

.

4

-

18

.

7%

-

18

.

0%

-

22

.

5%

-

23

.

3%

-

31

.

2%

-

31

.

1%

-

17

.

3%

-

6

.

8%

-

22

.

4%

-

19

.

8%

LLaMA-3.1-405B-Instruct

58

.

7

78

.

4

-

18

.

6%

-

13

.

6%

-

9

.

6%

-

10

.

3%

-

14

.

5%

-

15

.

6%

-

13

.

9%

-

7

.

5%

-

14

.

2%

-

11

.

8%

LLaMA-3.3-70B-Instruct

55

.

6

76

.

9

-

18

.

4%

-

10

.

8%

-

15

.

2%

-

7

.

7%

-

17

.

9%

-

11

.

0%

-

12

.

1%

-

4

.

7%

-

15

.

9%

-

8

.

6%

Qwen Series

Qwen2.5-7B-Instruct

42

.

6

58

.

2

-

35

.

3%

-

21

.

7%

-

50

.

7%

-

22

.

0%

-

35

.

7%

-

28

.

0%

-

37

.

7%

-

7

.

0%

-

39

.

8%

-

19

.

7%

Qwen2.5-14B-Instruct

48

.

4

70

.

7

-

36

.

1%

-

22

.

9%

-

37

.

3%

-

35

.

3%

-

33

.

0%

-

43

.

4%

-

14

.

2%

-

10

.

9%

-

30

.

2%

-

28

.

1%

Qwen2.5-32B-Instruct

59

.

9

79

.

4

-

25

.

4%

-

17

.

3%

-

23

.

5%

-

18

.

5%

-

23

.

6%

-

24

.

1%

-

19

.

7%

-

5

.

4%

-

23

.

1%

-

16

.

3%

Qwen2.5-72B-Instruct

58

.

1

82

.

8

-

24

.

0%

-

12

.

3%

-

15

.

4%

-

6

.

9%

-

24

.

9%

-

16

.

3%

-

25

.

2%

-

6

.

5%

-

22

.

4%

-

10

.

5%

Qwen2.5-Coder-32B-Instruct

63

.

1

84

.

8

-

24

.

4%

-

14

.

6%

-

28

.

2%

-

19

.

0%

-

21

.

7%

-

19

.

9%

-

14

.

7%

-

9

.

3%

-

22

.

3%

-

15

.

7%

Average

-

24

.

6%

-

14

.

5%

-

24

.

3%

-

14

.

9%

-

23

.

8%

-

19

.

5%

-

20

.

1%

-

6

.

2%

-

23

.

2%

-

13

.

8%

-

20

.

9

%

-

20

.

8

%

-

22

.

2

%

-

14

.

9

%

-

19

.

7

%

average drop 33 . 0 % on LCB compared to 12 . 4 % on CRUX. Since LCB consists of more complex algorithmic problems with relatively simple output structures, models are more willing to adopt the hint as a reasoning shortcut. Notably, the performances of Claude-3.5-Haiku and Sonnet drop over 40 % under MHC perturbation on LCB, which aligns with recent findings from Anthropic that models often accept a biased multiple-choice hint and rationalize it rather than critically resolving the contradiction (Chen et al., 2025b). We extend this observation to the code reasoning domain and illustrate that: (i) LLMs readily rationalize the plausible hint as authoritative reasoning shortcuts, and (ii) this over-trust amplifies when task complexity surpasses the model's reasoning capacity.

Scaling and versioning systematically improve robustness, with notable exceptions. Overall, robustness against perturbations improves consistently with model scale and newer releases. We observe a clear and consistent trend that larger closed-sourced models from GPT-4 Omni and Claude families demonstrate enhanced robustness compared to their smaller counterparts. Similar trends appear in open-source models, such as LLaMA (3.1-8B ≤ 70B < 405B) and Qwen (2.5-7B < 14B < 32B ≤ 72B). Additionally, newer versions (LLaMA-3.3-70B) and domain-specific fine-tuned variants (Qwen2.5-Coder-32B) exhibit improved robustness compared to their predecessors (LLaMA-3.170B) and larger counterparts (Qwen2.5-72B), respectively. Interestingly, the Gemini series deviates from this trend, underscoring that architectures and training strategies could affect robustness.

3.3 Limits of CoT in Resolving NL-Embedded Contradictions

Based on the severe degradation observed in §3.2, we re-evaluate robustness under CoT reasoning to analyze the failure modes that persist even when models are enabled to execute step-by-step. We present the comparison between two reasoning modes in Table 2.

CoT improves performance, but robustness gains remain limited. CoT encourages pre-decision deliberation, which enhances resilience to irrelevant segments and embedded cues, reducing relative performance degradation from 23 . 2% to 13 . 8% on average. Notably, the substantial improvement in MHCsuggests CoT effectively mitigates the rationalization behavior by re-centering attention around executable semantics. Such benefits are particularly evident in high-capacity models ( e.g. , GPT-4o, Claude-3.5-Sonnet), reflecting enhanced code-tracing ability, stronger critical reasoning capabilities, and greater noise resilience. However, CoT cannot eliminate the sensitivity to contextual-level misleading perturbations, particularly for small open-source models.

Failure Case Analysis: Inconsistency and Rationalization in CoT. From Table 3, we observe clear failure modes showing how Qwen2.5-72B is misled. Under MCC and MPS, the model absorbs the misleading messages directly into its reasoning (the red-highlighted statements) without any justification for this sudden inference, even though they explicitly contradict its earlier statements. Under MHC, the model initially conducts correct reasoning for the fourth character 'w', but in the very next step, 'a', it produces an incorrect conclusion to force alignment with the misleading hint, contradicting its previous inference. These behaviors align with the distractibility (Shi et al., 2023) and post-hoc rationalization (Barez et al., 2025), where the model overrides its correct reasoning to conform to injected context. These issues are particularly severe in programming because comments

and prints are semantically irrelevant to execution, but models possibly treat them as higher-priority evidence than program logic, which underscores an intrinsic limitation of LLMs in distinguishing non-functional code and misleading information from essential logic, highlighting insufficient critical reasoning capability . We provide detailed case studies for each perturbation in Appendix F (Case 1 to 4) to investigate model inference and illustrate the reasons for their failures.

3.4 Powerful Reasoning and Instability of LRMs

Building on the findings in §3.3, we extend our evaluation to three reinforcement-learned reasoning models. As shown in Table 4, LRMs demonstrate outstanding performance and robustness under perturbations, with negligible degradation under PSC-ALL , outperforming CoT-prompted LLMs in code tracing, logic understanding, and segment distinguishing. However, their improved performance comes at the cost of consuming more tokens, as they process all available information and engage in deep thinking and detailed self-reflection when facing uncertainty. Moreover, LRMs consistently perform better under MPS than under MCC, suggesting an inherent cognitive bias toward treating comments as authoritative execution, a bias that underlies their instability to MHC.

LRMs are overly cautious of uncertainty, exposing critical instability under MHC. Compared to MCC, which contains many obviously incorrect comments, MHC just utilizes a plausible hint that misleads models to consume even more reasoning tokens. Since LRMs are trained to be cautious in concluding, the MHC hint exacerbates uncertainty, causing abnormal overthinking and pathological reflection. As shown in Table 4, this manifests in three notable outcomes: (i) reasoning token usage increases by 2 -3 × compared to the vanilla setting, (ii) in extreme cases, such as QwQ-32B, it generates over 32k tokens due to uncontrolled recursive self-verification , and (iii) for o3-Mini, increased reasoning effort further exacerbates performance degradation, indicating that internal reasoning can amplify rather than resolve uncertainty. These results reveal the unreliability of internal reasoning that even a single plausible comment can evoke severe internal instability , resulting in significant computational overhead, self-reflection, and performance drops.

0

1000

2000

3000

4000

5000

6000

7000

8000

Word Index

0

50

100

150

200

250

300

350

400

Confusing Word Count

Stage 1

Stage 2

Stage 3

Stage 4

Stage 5

Stage 6

Quadratic Fit

Cumulative Confusion Score

Stage 1: Initial Reasoning (Normal Execution)

QwQ-32B correctly executes both loops in the function and

obtains the

correct output (-1)

.

Stage 2: First Re-execution and Verification

It re-executes the second loop to verify the result and again

derives the

same output (-1)

, maintaining consistent reasoning.

Stage 4: Deep Condition Analysis and Reevaluation

It starts questioning the condition 'lmin < nums[i] > A[i]',

shifting from

syntactic re-execution to semantic reasoning

.

Stage 5: Forced Alignment and Meaningless Enumeration

It attempts to justify 12 as the correct output through brute-force

enumeration,

rationalizing the hint and overriding its prior logic

.

Stage 6: Loss of Reasoning (Reasoning Collapse)

It fails to reconcile its logic with the hint, then enters a

repetitive

hallucination loop

filled with 'Hmm' (over 12k generated).

Stage 3: Contradiction Trigger  (Trigger Point)

It encounters the misleading hint '# return is 12', then begins to doubt

its logic, and

repeatedly re-checks everything

.

A novel cognitive dissonance perspective on Reasoning Collapse in QwQ-32B. For QwQ-32B, we observed four 'collapse' events ( i.e. , generating massive replicated terms), all from MHC on LCB, suggesting that this is not a coincidental phenomenon. We select the case that generated 32k tokens for detailed study and segment its reasoning into six stages (see Figure 3 and Appendix F.6). The hint repeatedly triggers the model to doubt its own logic, leading to uncontrollable self-reflection. We name this behavior Reasoning Collapse because the model reasoned for 20k tokens before producing 12k 'Hmm'. Figure 2 shows a quadratic growth of confusing tokens before the collapse, indicating that the failure is attributable to the accumulated conflict and growing uncertainty rather than a low-level glitch. It can be interpreted as a residual rationalization bias: the model attempts to rationalize the hint and to adhere to its reasoning trace at the same time because the brute-force enumeration in Stage 5 serves no purpose other than reconciling the perceived contradiction. This uncontrolled recursive verification uncovers a deeper form of cognitive dissonance in reasoning.

3.5 Potential Limitation in Input Prediction

We repeat the experiments in input prediction tasks on the CRUX dataset and summarize the results in Table 5. The results under PSC-ALL and MHC align with our findings in §3.2 and §3.3, further supporting Gu et al. (2024)'s hypothesis that robustness in related coding tasks is interrelated. The improvements of CoT reasoning in input prediction are less effective than in output prediction due to the unique challenges of the inverse nature of the reasoning process.

Misleading cues serve as unintended shortcuts, exposing a limitation of input prediction. Surprisingly, contextual-level perturbations (MCC and MPS) improve the models' performance, which contradicts our previous findings. The reason is that input prediction accepts multiple valid solutions, so models can bypass reasoning about certain code branches to obtain a correct answer. Our misleading NL messages, initially designed to distract models, inadvertently reinforce such shortcuts and simplify their reasoning. Consequently, these perturbations unintentionally provide shallow reasoning strategies, so those improvements do not reflect the genuine robustness of LLMs in code comprehension. We provide a detailed case study in Appendix F.5, illustrating how models adopt misleading messages to shortcut their reasoning.

4 Discussion

4.1 Impact of Comment Density on Robustness and Reasoning Behaviors

0.0

0.2

0.4

0.6

0.8

1.0

Injection Probability p

60

50

40

30

20

10

0

Relative Pass@1 Degradation (%)

Qwen2.5-7B-Instruct

Qwen2.5-14B-Instruct

Qwen2.5-32B-Instruct

Qwen2.5-72B-Instruct

Qwen2.5-72B-Instruct (CoT)

QwQ-32B

To confirm the validity and realism of our misleading perturbations, we design a density sweep experiment to examine how the relative PASS@1 score drops across different models when we gradually increase the injection probability p ∈ { 0 . 0 , 0 . 1 , . . . , 1 . 0 } , as illustrated in Figure 4.

Systematic degradation under increasing density. Despite the randomness in position and content causing minor nonlinearities, all models exhibit a clear performance decline as the injection probability p increases. Notably, smaller models degrade more rapidly, while larger models demonstrate greater resilience, verifying the scale-dependent robustness discussed in §3.2. These results confirm that contextual-level misleading perturbations are a scalable stress-testing method. Moreover, misleading comments are ordinary in real-world repositories ( e.g. , CPython GitHub issue 4 ), and our results verify that even a single misleading comment can impact model understanding.

LLMs and LRMs have similar trends but different reasoning behaviors. Qwen2.5-72B (CoT) and QwQ-32B have similar trends in performance degradation, but they have opposite trends in token usage, as shown in Table 6. While CoT improves robustness by enabling step-by-step execution, LLMs solely report the expected behavior and variable changes (as shown in Table 3), causing them to reach premature conclusions without sufficiently analyzing the code. Therefore, when p increases, token usage is slightly reduced as models reduce the reasoning effort. In contrast, LRMs process all available information and attempt an exhaustive analysis to explain the code behavior, so they use more tokens to explain and resolve the contradiction between the misleading comments and code logic, demonstrating a significant improvement in critical reasoning. This critical and comprehensive analysis achieves higher performance and robustness, but it risks severe reasoning collapse if it cannot resolve the contradiction as discussed in §3.4.

4.2 Cross-Language Robustness Analysis

Cross-language consistency of perturbation effects. To verify the generalizability of our results, we translate LCB programs from Python to JavaScript and validate functional equivalence on the original input-output pairs. As shown in Table 7, both GPT-4o and Qwen2.5-72B exhibit consistent degradation across languages. Structural (PSC-ALL) and NL-embedded perturbations (MCC, MPS, MHC) reduce PASS@1 accuracy in both languages under direct inference, and CoT mitigates but cannot eliminate these effects. Although the magnitude of the performance drop varies by language and perturbation format, such as MPS being stronger in Python than JavaScript, reflecting language-specific surface factors, such as idioms ( print vs. console.log ), that modulate how the model perceives cues. Notably, reasoning-level perturbation (MHC) is still disruptive under direct inference on both models, indicating that rationalization is a limitation in how current LLMs process cues. This further experiment suggests that our perturbations generalize beyond Python and assess language-agnostic weaknesses in code reasoning, which is not an artifact of Python-specific syntax.

4.3 Defenses and Limitations

Table 9: Performance of GPT-4o with and without refactoring under direct inference.

Prompt engineering provides only partial relief. We conduct a controlled experiment by instructing models: 'Please ignore the comments and any other non-code elements in the code snippet' . Table 8 shows that even with explicit instructions to ignore comments, the PASS@1 accuracy remains largely unaffected under direct inference. Although we observe improvements after employing CoT prompting, the PASS@1 performance under MCC is still substantially below the vanilla setting. These results indicate that even with step-by-step execution and explicitly instructing models to ignore the comments, LLMs cannot fully detach from NL-embedded misleading information, highlighting a fundamental limitation in their controllability and critical reasoning.

Refactoring agents improve syntax, but risk logic distortion. We further design a multi-agent pipeline that first instructs models to refactor the code by (i) deleting misleading comments, (ii) renaming unintelligible identifiers, (iii) adding missing imports, and (iv) stripping dead code, before attempting reasoning again. Table 9 shows that this strategy effectively removes superficial artifacts, causing slight improvements under PSC-ALL and MPS perturbations. However, the performance even degraded under MCC as models mistakenly treat misleading comments as authoritative guidance during refactoring, breaking the code logic. Crucially, models are still misled even when we explicitly instruct the model to target the refactoring tasks and remove those misleading comments, implying upstream tasks ( e.g. , bug fixing and vulnerability detection) are equally at risk because an attacker can leverage benign-looking comments to poison the code and propagate failures. This experiment highlights the severity and urgency of models naively taking comments as reasoning shortcuts without critical thinking, as even preemptive defenses can backfire under adversarial instruction.

5 Related Work

Code generation and execution benchmarks. Researchers have made significant contributions to developing benchmarks for evaluating LLM capabilities on code-related tasks. Early benchmarks such as HUMANEVAL (Chen et al., 2021), MBPP (Austin et al., 2021), APPS (Hendrycks et al., 2021), and HUMANEVAL+ (Liu et al., 2023b) primarily focused on assessing code generation performance. More recent efforts have shifted toward practical software engineering and realistic coding scenarios. For instance, SWE-bench (Jimenez et al., 2024), InfiBench (Li et al., 2024), and LCB (Jain et al., 2025) evaluate LLMs on complex tasks such as multi-file changes, Stack Overflow-style questions, and real-world programming problems.

Code execution and reasoning. A growing body of work investigates LLMs in code execution. Early studies (Austin et al., 2021; Nye et al., 2021) indicated that LLMs face challenges in accurately executing code but can benefit from intermediate reasoning steps. Subsequent research has utilized various strategies to enhance execution performance, including execution-informed pretraining (Liu et al., 2023a), CoT prompting with trace supervision (Ni et al., 2024), and iterative instruction prompting (Lyu et al., 2024). CRUX (Gu et al., 2024) formalized the input and output prediction tasks specifically for evaluating LLMs' code understanding and reasoning. REVAL (Chen et al., 2024b) and CACP (Hooda et al., 2024) uncovered inconsistencies and concept-level misunderstandings of LLMs through intermediate reasoning analysis during code execution. CodeScore (Dong et al., 2024), LEVER (Ni et al., 2023), and XCODEEVAL (Khan et al., 2024) have leveraged code execution to improve model evaluation methodologies and enhance the performance of downstream tasks.

Robustness evaluation for code LLMs. Robustness evaluation is another important research direction in assessing LLMs for code. Prior studies focused on prompt-level variations in code generation, examining how semantically equivalent NL descriptions impact model performance (Mastropaolo et al., 2023; Chen et al., 2024a; Zhuo et al., 2023). Apart from NL perturbations, researchers have investigated model robustness under programming language perturbations. For example, CCTEST (Li et al., 2023) introduced structural mutations, CREAM (Gao et al., 2023) employed counterfactual reasoning, and ReCode (Wang et al., 2022) applied semantic-preserving transformations, advancing robustness in code completion. BigCodeBench (Zhuo et al., 2025) further explored generalization under complex function composition. In addition, many works have explored structure-aware modeling by embedding program graphs or flow signals to enhance code robustness (Son et al., 2022; Oh and Yoo, 2024; Pei et al., 2022; Tipirneni et al., 2024). Furthermore, Chen et al. (2025a); Cuadron et al. (2025) and Kumar et al. (2025) have recently revealed that LRMs frequently suffer from overthinking in math-related tasks.

6 Conclusion

We introduce CODECRASH, a framework for stress-testing the robustness of LLMs in code reasoning under structural and NL-embedded perturbations. Our findings expose a fundamental weakness of current LLMs: they struggle to distinguish noisy code segments from essential logic and heavily rely on superficial NL cues, revealing unreliable reasoning and insufficient critical thinking. Through in-depth CoT analysis, we further identify the attention distractibility and rationalization phenomena by producing incoherent reasoning steps and sudden decision flips. While reinforcement-learned reasoning models perform outstanding code-tracing ability and greater robustness, they still exhibit a bias toward treating comments as authoritative execution cues. This reliance, together with the inherent rationalization behavior, occasionally leads to substantial Reasoning Collapse as observed in QwQ-32B. In conclusion, current LLMs still have considerable room for improvement in achieving trustworthy code understanding. We believe that our benchmark serves as a practical framework to provide significant insights into model robustness in code reasoning.

Acknowledgments and Disclosure of Funding

The paper was supported by two grants from the Research Grants Council of the Hong Kong Special Administrative Region, China: (1) No. CUHK 14209124 of the General Research Fund, and (2) No. SRFS2425-4S03 of the Senior Research Fellow Scheme.

References

Anthropic. Claude 3.5 sonnet. Anthropic News June 21, 2024 , 2024a. URL https://www.anthropic. com/news/claude-3-5-sonnet .

Anthropic. Introducing claude 3.5 haiku. Anthropic News October 22, 2024 , 2024b. URL https: //www.anthropic.com/claude/haiku .

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021.

Fazl Barez, Tung-Yu Wu, Iván Arcuschin, Michael Lan, Vincent Wang, Noah Siegel, Nicolas Collignon, Clement Neo, Isabelle Lee, Alasdair Paren, Adel Bibi, Robert Trager, Damiano Fornasiere, John Yan, Yanai Elazar, and Yoshua Bengio. Chain-of-thought is not explainability. alphaXiv , 2025. URL https://www.alphaxiv.org/abs/2025.02v2 . Preprint, under review.

Junkai Chen, Zhenhao Li, Xing Hu, and Xin Xia. Nlperturbator: Studying the robustness of code llms to natural language variations. arXiv preprint arXiv:2406.19783 , 2024a.

Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. Reasoning runtime behavior of a program with llm: How far are we? arXiv preprint arXiv:2403.16437 , 2024b.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187 , 2025a.

Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning models don't always say what they think. arXiv preprint arXiv:2505.05410 , 2025b.

Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08235 , 2025.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and Ruoyu Zhang et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025a.

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, and Bochao Wu et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 , 2025b.

Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , pages 423-435, 2023.

Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023.

Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. Codescore: Evaluating code generation by learning code execution. arXiv preprint arXiv:2301.09043 , 2024.

Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pages 1469-1481. IEEE, 2023.

Shuzheng Gao, Cuiyun Gao, Chaozheng Wang, Jun Sun, David Lo, and Yue Yu. Two sides of the same coin: Exploiting the impact of identifiers in neural code comprehension. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pages 1933-1945. IEEE, 2023.

GitHub. Github copilot - your ai pair programmer, 2022. URL https://github.com/features/ copilot . Accessed 10-May-2025.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: a benchmark for code reasoning, understanding and execution. In Proceedings of the 41st International Conference on Machine Learning , ICML'24. JMLR.org, 2024.

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 , 2021.

Ashish Hooda, Mihai Christodorescu, Miltiadis Allamanis, Aaron Wilson, Kassem Fawaz, and Somesh Jha. Do large code models understand programming concepts? counterfactual analysis for code predicates. In Forty-first International Conference on Machine Learning , 2024.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2.5-coder technical report. arXiv preprint arXiv:2409.12186 , 2024.

Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.212764 , 2024.

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations , 2025.

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations , 2024.

Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pages 2312-2323. IEEE, 2023.

Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. XCodeEval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 6766-6805, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.367.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems , 35: 22199-22213, 2022.

Abhinav Kumar, Jaechul Roh, Ali Naseh, Marzena Karpinska, Mohit Iyyer, Amir Houmansadr, and Eugene Bagdasarian. Overthink: Slowdown attacks on reasoning llms. arXiv preprint arXiv:2502.02542 , 2025.

Linyi Li, Shijie Geng, Zhenwen Li, Yibo He, Hao Yu, Ziyue Hua, Guanghan Ning, Siwei Wang, Tao Xie, and Hongxia Yang. Infibench: Evaluating the question-answering capabilities of code large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2024.

Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Dong Chen, Shuai Wang, and Cuiyun Gao. Cctest: Testing and repairing code completion systems. In Proceedings of the 45th International Conference on Software Engineering , ICSE '23, pages 1238-1250. IEEE Press, 2023. ISBN 9781665457019.

Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan. Code execution with pre-trained language models. arXiv preprint arXiv:2305.05383 , 2023a.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems , 2023b.

Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yunlong Duan, Yu Hao, Liqun Yang, Guanglin Niu, Ge Zhang, and Zhoujun Li. Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310 , 2025.

Chenyang Lyu, Lecheng Yan, Rui Xing, Wenxi Li, Younes Samih, Tianbo Ji, and Longyue Wang. Large language models as code executors: An exploratory study. arXiv preprint arXiv:2410.06667 , 2024.

Antonio Mastropaolo, Luca Pascarella, Emanuela Guglielmi, Matteo Ciniselli, Simone Scalabrino, Rocco Oliveto, and Gabriele Bavota. On the robustness of code generation techniques: An empirical study on github copilot. arXiv preprint arXiv:2302.00438 , 2023.

Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-Tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 26106-26128. PMLR, July 2023.

Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. Next: Teaching large language models to reason about code execution. arXiv preprint arXiv:2404.14662 , 2024.

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 , 2021.

Saeyoon Oh and Shin Yoo. Csa-trans: Code structure aware transformer for ast. arXiv preprint arXiv:2404.05767 , 2024.

OpenAI. Openai o3-mini. OpenAI News January 31, 2025 , 2025. URL https://openai.com/ index/openai-o3-mini/ .

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, and Ilge Akkaya et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2024.

Zhenyu Pan, Rongyu Cao, Yongchang Cao, Yingwei Ma, Binhua Li, Fei Huang, Han Liu, and Yongbin Li. Codev-bench: How do llms understand developer-centric code completion? arXiv preprint arXiv:2410.01353 , 2024.

Ruchika Pandey, Prabhat Singh, Raymond Wei, and Shaila Shankar. Transforming software development: Evaluating the efficiency and challenges of github copilot in real-world projects. arXiv preprint arXiv:2406.17910 , 2024.

Xiaohuan Pei, Daochang Liu, Luo Qian, and Chang Xu. Contrastive code-comment pre-training. In 2022 IEEE International Conference on Data Mining (ICDM) , pages 398-407, 2022. doi: 10.1109/ICDM54844.2022.00050.

Sundar Pichai and Demis Hassabis. Our next-generation model: Gemini 1.5. Google Blog February 15, 2024 , 2024. URL https://blog.google/technology/ai/ google-gemini-next-generation-model-february-2024/ .

Sundar Pichai, Demis Hassabis, and Koray Kavukcuoglu. Introducing gemini 2.0: our new ai model for the agentic era. Google Blog December 11, 2024 , 2024. URL https://blog.google/ technology/google-deepmind/google-gemini-ai-update-december-2024/ .

Qwen Team. Qwq-32b: Embracing the power of reinforcement learning. Qwen Blog March 6, 2025 , 2025. URL https://qwenlm.github.io/blog/qwq-32b/ .

Guoping Rong, Yongda Yu, Song Liu, Xin Tan, Tianyi Zhang, Haifeng Shen, and Jidong Hu. Code comment inconsistency detection and rectification using a large language model. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) , pages 432-443, Los Alamitos, CA, USA, May 2025. IEEE Computer Society. doi: 10.1109/ICSE55347.2025.00035.

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023.

Jikyoeng Son, Joonghyuk Hahn, HyeonTae Seo, and Yo-Sub Han. Boosting code summarization by embedding code structures. In Proceedings of the 29th International Conference on Computational Linguistics , pages 5966-5977, Gyeongju, Republic of Korea, August 2022. International Committee on Computational Linguistics.

Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, and Alexey Svyatkovskiy. Reinforcement learning from automatic feedback for high-quality unit test generation. arXiv preprint arXiv:2310.02368 , 2025.

Weisong Sun, Yuchen Chen, Guanhong Tao, Chunrong Fang, Xiangyu Zhang, Quanjun Zhang, and Bin Luo. Backdooring neural code search. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 9692-9708, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.540.

Tabnine. Tabnine ai code assistant, 2023. URL https://www.tabnine.com . Accessed 10-May2025.

Wannita Takerngsaksiri, Rujikorn Charakorn, Chakkrit Tantithamthavorn, and Yuan-Fang Li. Pytester: Deep reinforcement learning for text-to-testcase generation. Journal of Systems and Software , 224: 112381, 2025. ISSN 0164-1212.

Sindhu Tipirneni, Ming Zhu, and Chandan K. Reddy. Structcoder: Structure-aware transformer for code generation. arXiv preprint arXiv:2206.05239 , 2024.

Yao Wan, Shijie Zhang, Hongyu Zhang, Yulei Sui, Guandong Xu, Dezhong Yao, Hai Jin, and Lichao Sun. You see what i want you to see: poisoning vulnerabilities in neural code search. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ESEC/FSE 2022, page 1233-1245, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450394130. doi: 10.1145/3540250. 3549153.

Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. Recode: Robustness evaluation of code generation models. arXiv preprint arXiv:2212.10264 , 2022.

Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zhiyao Xu, and Michael R Lyu. Interaction2code: How far are we from automatic interactive webpage generation? arXiv preprint arXiv:2411.03292 , 2024.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 , 2025.

Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and JianGuang Lou. Large language models meet nl2code: A survey. arXiv preprint arXiv:2212.09420 , 2023.

Beiqi Zhang, Peng Liang, Xiyu Zhou, Aakash Ahmad, and Muhammad Waseem. Practices and challenges of using github copilot: An empirical study. In Proceedings of the 35th International Conference on Software Engineering and Knowledge Engineering , pages 124-129, April 2023. doi: 10.18293/SEKE2023-077.

Terry Yue Zhuo, Zhuang Li, Yujin Huang, Fatemeh Shiri, Weiqing Wang, Gholamreza Haffari, and Yuan-Fang Li. On robustness of prompt-based semantic parsing with large pre-trained language model: An empirical study on codex. arXiv preprint arXiv:2301.12868 , 2023.

Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations , 2025.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide the claims and contributions at the end of the introduction, which have experimental support.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see §4.3 and §G.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical analysis.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to the README.md in the supplementary materials. We provide the source codes and the raw data.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer to the supplementary materials. We provide the source code and have recorded all raw experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see §3.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please see §C.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments were conducted via API access (OpenAI, Anthropic, Gemini, DeepInfra, and Qwen), and thus no information regarding memory, device type, or execution time is applicable. We record all necessary details, including model versions, input prompts, and output responses.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully read the NeurIPS Code of Ethics and verified that our paper aligns with the requirements.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work presents a benchmark for evaluating the robustness of LLMs on code-related tasks. It is foundational in nature and does not propose any new model, application, or deployment. Thus, we do not anticipate direct societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work does not involve the release of any pretrained models or newly scraped datasets. All experiments are conducted on existing public benchmarks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use the publicly released CRUXEVAL (Gu et al., 2024) and LIVECODEBENCH (Jain et al., 2025) datasets, both of which are cited appropriately in the paper and used in accordance with their licenses or terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Please refer to the supplementary materials which include a GNU General Public License v3.0 license.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: The use of LLMs is an essential and original component of the research, as the models themselves are the core subjects of evaluation. All LLM usage is limited to experimental inference via API and is fully documented in the paper.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Contents

A Dataset Selection

We evaluate two benchmarks: CRUX (Gu et al., 2024) and LCB (Jain et al., 2025). CRUX contains 800 synthetic problems generated by Code-LLaMA-34B, each featuring concise Python snippets with generic function names f and a corresponding input-output pair for input and output prediction tasks. In contrast, LCB contains 479 real-world coding problems sourced from 85 human-submitted LeetCode solutions, focusing solely on output prediction. While both datasets avoid excessive memory or runtime demands, LCB problems exhibit more complex structures, such as nested functions, decorators, and external libraries, requiring more computation.

B Illustrative Examples of Perturbation Strategies

In section 2.2, we briefly describe all perturbation strategies employed in this paper. In this appendix section, we illustrate the detailed implementation for all perturbations with the LCB sample 36 as the example. Code 1 presents the base vanilla example where the formal answer is minimumCost(s = '0011') == 2 .

B.1 Aggregating Structural Perturbations (PSC-ALL)

As illustrated in §2.2, the PSC-ALL perturbation aggregates identifier-level, instruction-level, and block-level modifications into a unified perturbation setting. Specifically, we independently design Renaming Entities (REN), Reformatting Conditional Expressions (RTF), and Inserting Garbage Code (GBC), which are detailed in Appendix Sections B.1.1, B.1.2, and B.1.3, respectively. Code 2 shows an example of an PSC-ALL-perturbed program.

Compounding Effect of PSC-ALL. Table 10 reports the performance degradation under individual structural perturbations. Compared to Table 1, which highlights NL-embedded misleading perturbations, REN, RTF, and GBC alone exhibit mild effects. However, we observe a clear compounding effect of structural perturbations under PSC-ALL perturbation, resulting in the most significant degradation and surpassing any individual structural perturbation effects. Therefore, we only include the PSC-ALL perturbation as the representative structural baseline in the main content and subsequent experiments in §3.3, §3.5, and §3.4 for structural perturbations.

[BACK TO METHODOLOGY]

B.1.1 Renaming Entities (REN)

We use the AST to identify all variable names appearing in:

assignment statements ( e.g. , a, b = 1, 2 ), · for-loops ( e.g. , for i in range(10) ), · with-statements ( e.g. , with open('file') as f ), · walrus operators ( e.g. , if (a := 10) > 5 ), and · comprehensions ( e.g. , [x for x in range(10)] ).

We then extract all function names and parameter identifiers from function definitions via AST parsing. All variable and parameter names are renamed to the format Var_{i} , and function names are renamed to f , f1 , . . . , fn . Code 3 presents an example of this perturbation, and the evaluation expression is modified to f(Var_1 = '0011') == 2 .

[BACK TO METHODOLOGY]

B.1.2 Reformatting Conditional Expressions (RTF)

We categorize conditional expressions into three general types: (1) comparison-based conditions ( e.g. , if a < b: ), (2) boolean constant conditions ( e.g. , if True: ), and (3) general conditions ( e.g. , if x: ). For both comparison-based and Boolean constant conditions, we design seven transformation templates:

For general conditions, we also design four transformation templates:

Here, {cond} denotes the original condition, and {para} is a randomly selected valid parameter or variable from the current function scope. Code 4 presents an example of this perturbation.

[BACK TO METHODOLOGY]

B.1.3 Inserting Garbage Code (GBC)

We inject three types of garbage code: (1) repeated-name global variable declarations, (2) unexecutable conditional statements, and (3) dead-loop function definitions. For (1), we randomly select a few parameters or variables and declare global variables using the same names, assigning arbitrary integers at the beginning of the program. For (2), we design seven templates for false conditions:

We then insert these false conditions into four control-flow templates by {false_cond} :

Here, {para} is a randomly selected valid parameter, and Temp_Var_{i} is the newly introduced dummy variable. For (3), we construct seven dead-loop function templates, as demonstrated in Code 5. Code 6 presents an example of this perturbation.

[BACK TO METHODOLOGY]

[BACK TO METHODOLOGY]

B.2 Inserting Contextual-Level Misleading Messages (MCC and MPS)

As introduced in §2.2, we insert misleading comments for eight key AST node types, including (1) function declarations, (2) return statements, (3-4) while and for loops, (5) conditionals, (6) variable assignments, and (7) common data structure operations. The (8) data structure operations include common list, set, and string manipulations: append, extend, insert, remove, pop, sort, reverse, update, add, split, join, replace, lower, upper, capitalize, and swap case. To construct these comments, we design prompts for GPT-4o to generate misleading but plausible messages for each type. In this experiment, we manually selected 20 messages for each AST node type and 5 for each data structure operation. Take variable assignments as an example; Code 7 is the prompt for generating misleading comments for variable assignments, and Code 8 is a list of selected misleading comment candidates. Code 9 and 10 present an example of MCC and MPS perturbations.

Code 8: Selected misleading comments for variable assignments used in MCC and MPS perturbations, where ' {variable} ' is the variable name.

[BACK TO METHODOLOGY]

Code 9: MCC-perturbed example from LCB (sample 36) for output prediction.

Code 10: MPS-perturbed example from LCB (sample 36) for output prediction.

[BACK TO METHODOLOGY]

B.3 Inserting Reasoning-Level Misleading Hint (MHC)

In order to generate plausible, logical, but incorrect hints, we instruct GPT-4o to rewrite the correct assertion expression into incorrect ones and only allow it to modify the output value using the prompt in Code 11 for output prediction. An example perturbed instance is shown in Code 12, where the ground-truth return value 2 is replaced with the misleading hint # The return value is 3 .

In contrast, we use the prompt shown in Code 13 to generate misleading input hints, allowing GPT-4o to perform CoT reasoning before generating the expression. Since some functions return the same output regardless of the input arguments, we instruct GPT-4o to force the expression to become incorrect by modifying input arguments, such as changing data types, swapping parameter order, or adding/removing arguments. An example perturbed instance is shown in Code 14.

[BACK TO METHODOLOGY]

Code 14: MHC-perturbed example from CRUX (sample 2) for input prediction.

[BACK TO METHODOLOGY]

C Statistical Analysis

Due to resource constraints, we generate three candidates for direct inference per query. To assess the reliability of this setup, we compute the standard deviation ( σ ) across each perturbation and vanilla setting.

We present two summary tables for output prediction tasks (Table 11) and input prediction tasks (Table 12). The results show that across a total of 357 measurements, only 9 cases have σ > 1 . 0 ( i.e. , 0 . 01 on a scale from 0 to 1 ), and 57 have σ > 0 . 5 , indicating that most scores exhibit low variance under repeated sampling.

Furthermore, our benchmark includes 1279 problems across 8 experiment settings on 17 models for output prediction tasks, and 800 problems across 5 experiment settings on 17 models for input prediction tasks. Among 241 , 944 combinations, 231 , 726 have identical correctness labels across all three candidates, yielding an overall stability rate of 95 . 78 % .

Model Series

VAN

REN

RTF

GBC

PSC-ALL

MCC

MPS

MHC

CRUX

LCB

CRUX

LCB

CRUX

LCB

CRUX

LCB

CRUX

LCB

CRUX

LCB

CRUX

LCB

CRUX

LCB

GPT-4o-Mini

0

.

2125

0

.

1705

0

.

2125

0

.

429

0

.

27

0

.

7873

0

.

5303

0

.

1968

0

.

2569

0

.

429

0

.

27

0

.

1705

0

.

1768

0

.

0984

0

.

2569

0

.

5905

GPT-4o

0

.

4125

0

.

6818

0

.

27

0

.

5114

0

.

3584

0

.

2952

0

.

2041

0

.

858

0

.

4082

0

.

4921

0

.

6374

0

.

451

0

.

598

0

.

451

0

.

2125

0

.

8056

Claude Series

Claude-3.5-Haiku-20241022

1

.

546

0

.

8523

1

.

562

6

.

987

1

.

373

1

.

45

0

.

6208

0

.

5986

0

.

2041

0

.

1705

0

.

2041

0

.

6146

0

.

3864

0

.

2604

0

.

1021

0

.

2604

Claude-3.5-Sonnet-20241022

0

.

2125

0

.

429

0

.

1559

0

.

6889

0

.

2125

0

.

3548

0

.

1559

0

.

1705

0

.

2357

0

.

1705

0

.

5893

0

.

2604

0

.

3118

0

.

5114

0

.

0589

0

.

1705

Gemini Series

Gemini-1.5-Flash

0

.

2569

0

.

1705

0

.

1559

0

.

1705

0

.

1559

0

.

1705

0

.

1021

0

.

0984

0

.

2041

0

.

1705

0

.

1559

0

.

0984

0

.

1179

0

.

2604

0

.

2125

0

.

1705

Gemini-2.0-Flash

0

.

1179

0

.

1705

0

.

3281

0

.

2604

0

.

3584

0

.

2604

0

.

2357

0

.

2952

0

.

4249

0

.

451

0

.

1768

0

.

2952

0

.

2041

0

0

.

1559

0

.

429

Gemini-1.5-Pro-002

0

.

5103

0

.

6146

0

.

3584

0

.

3937

0

.

4449

0

.

6889

0

.

1021

1

.

135

0

.

4125

0

.

2604

0

.

27

0

.

3548

0

.

2041

0

.

3548

0

.

2041

0

.

2604

DeepSeek

DeepSeek-V3

0

.

1021

0

.

6889

0

.

2125

0

.

5208

0

.

4249

0

.

6453

0

.

3536

0

.

2952

0

.

2041

0

.

7811

0

.

8165

0

.

9388

1

.

066

0

.

3548

0

.

3281

0

.

3548

LLaMA Series

LLaMA-3.1-8B-Instruct

0

.

1559

0

.

743

0

.

1559

0

.

8747

0

.

1021

0

.

2952

0

.

0589

0

.

3548

0

.

2125

0

.

429

0

.

1768

0

.

1968

0

.

2946

0

.

429

0

.

3864

0

.

1705

LLaMA-3.1-70B-Instruct

0

.

2946

0

.

3409

0

.

4249

0

.

3548

0

.

3281

0

.

6453

0

.

5137

0

.

2604

0

.

2125

0

.

1705

0

.

1559

0

.

2952

0

.

2041

0

.

0984

0

.

27

0

.

1968

LLaMA-3.1-405B-Instruct

0

.

2946

0

.

429

0

.

1559

0

.

1705

0

.

1179

0

.

3937

0

.

1768

0

.

451

0

.

7795

0

.

2604

0

.

3536

0

.

1705

0

.

3536

0

.

1968

0

.

2569

0

.

5208

LLaMA-3.3-70B-Instruct

0

.

0589

0

.

3548

0

.

2125

0

.

1968

0

.

2125

0

.

1705

0

.

2125

0

0

.

2569

0

.

2604

0

.

1179

0

.

1968

0

.

2125

0

.

5208

0

.

0589

0

.

1968

Qwen Series

Qwen2.5-7B-Instruct

0

.

1559

0

.

0984

0

.

27

0

.

5479

0

.

4677

0

.

1705

0

.

2125

0

.

2604

0

.

1021

0

.

1705

0

.

1559

0

.

0984

0

.

4125

0

.

6818

0

.

2041

0

.

451

Qwen2.5-14B-Instruct

0

.

0589

0

.

2952

0

.

2125

0

.

3548

0

.

2041

0

.

5986

0

.

2357

0

.

1968

0

.

3062

0

.

3409

0

.

1021

0

.

1705

0

.

1021

0

.

1968

0

.

1179

0

.

429

Qwen2.5-32B-Instruct

0

.

2357

0

.

3548

0

.

3864

0

.

1705

0

.

0589

0

.

0984

0

.

3062

0

.

2952

0

.

1559

0

.

451

0

.

2125

0

.

2604

0

.

0589

0

.

1968

0

.

0589

0

.

2604

Qwen2.5-72B-Instruct

0

.

1179

0

.

451

0

.

1559

0

.

0984

0

.

2125

0

.

451

0

.

27

0

.

5479

0

.

4714

0

.

2604

0

.

3536

1

.

028

1

.

327

0

.

8523

0

.

1559

0

.

3937

Qwen2.5-Coder-32B-Instruct

0

.

4602

0

.

2604

0

.

4714

0

.

5905

0

.

1768

0

.

858

0

.

5803

0

.

1705

0

.

2041

0

.

7873

0

.

4249

0

.

2952

0

0

.

3409

0

.

3118

0

.

6889

[BACK TO EXPERIMENT SETTINGS]

D Prompts

Our prompt instructions for LLMs follow the format designed by CRUX, which is also adopted by LCB, utilizing a two-shot prompt for direct inference and a single-shot prompt for CoT inference to ensure models can follow the instruction and output format effectively. For LRMs, internal reasoning is inherently required before responding. Therefore, applying CoT prompts may lead to redundant reasoning and an unfair comparison with standard LLMs under CoT prompting. Considering the fairness, we use direct inference prompts for LRM experiments in §3.4.

Unlike LCB, we rename all example function names to f to avoid prompting models to capture the NL cues from the example. We add the instruction 'Do NOT output any extra information' to prevent external output, particularly from Gemini and Claude models, to ensure all models have a fair output environment. We also include 'Ensure the provided expression syntax is correct' to enforce consistent syntax. However, some responses from models like QwQ-32B, DeepSeek-R1, still generate syntactically incorrect outputs such as [ANSWER] f("a") == [/ANSWER] instead of [ANSWER] f("a") == "" [/ANSWER] . For those situations, we identify them as 'Syntax Errors' and incorrect.

We provide the complete prompts used for output prediction in both direct and CoT inference (Code 15 and 16) and for input prediction in both direct and CoT inference (Code 17 and 18).

D.1 Output Prediction (Direct)

[BACK TO EXPERIMENT SETTINGS]

D.2 Output Prediction (CoT)

[BACK TO EXPERIMENT SETTINGS]

D.3 Input Prediction (Direct)

[BACK TO EXPERIMENT SETTINGS]

D.4 Input Prediction (CoT)

[BACK TO EXPERIMENT SETTINGS]

E Supplementary Experimental Results

E.1 Additional Performance Degradation Tables for Output Prediction (Direct Inference)

In §3.2, we evaluate the robustness of LLMs on output prediction tasks under direct inference, using the relative PASS@1 degradation in Table 1. For completeness, we also include the absolute PASS@1 degradation in Table 13, reflecting the percentage drop from each model's vanilla performance.

E.2 Additional Performance Degradation Tables for Output Prediction (CoT Inference)

In §3.3, we analyze the effect of CoT prompting on model robustness using relative degradation aggregated across all perturbations (Table 2). We adopt the relative metric to mitigate the large variance between vanilla performances in direct and CoT inference modes, providing a fair comparison of performance degradation. For completeness, we report the absolute degradation in Table 14, as well as the relative and absolute degradation on both CRUX and LCB in Tables 15 and 16, respectively.

E.3 Additional Performance Degradation Tables for Input Prediction

In §3.5, we evaluate LLM robustness on input prediction using the CRUX dataset. We focus on relative PASS@1 degradation (Table 5) to capture performance variation under perturbations. For completeness, we also report absolute degradation in Table 17.

F Case Studies

F.1 Case Study 1: How is DeepSeek-V3 misled by PSC-ALL

This case (Code 19 and 20) demonstrates how garbage code can distort model reasoning. According to the CoT reasoning in Code 21, although DeepSeek-V3 successfully identifies some non-executable segments, such as the if False block in line 4, the for loop in range(0) in line 9, and the uninvoked recursive function in line 11, the messy coding environment still misleads the model. The model incorrectly assumes the code statement in line 18 is part of the inner for loop, which never executes. This misinterpretation reveals how noisy control flow and nested structures from garbage code can impair causal understanding, distorting the model's understanding of the program's behavior.

[BACK TO OUTPUT PREDICTION (COT) ANALYSIS]

F.2 Case Study 2: How is GPT-4o misled by MCC

This case (Code 22) reveals that GPT-4o implicitly relies on NL information. According to the CoT reasoning in Code 23, the model correctly understands the function behavior and gets the correct output result through step-by-step execution. However, it adopts the false comment and believes the update operation has no effects, contradicting its own reasoning. Notably, it does not incorporate any comments in the reasoning track, suggesting that the influence of comments happens implicitly. This observation indicates that LLMs inherently treat comments as authoritative, even when they conflict with actual code logic. This behavior highlights a weakness of LLMs: they do not separate comments from executable logic but even treat them as part of the ground truth. Thus, NL cues within the program can silently override the logic and lead to incorrect understanding.

Code 22: MCC-perturbed example from CRUX (sample 1) for output prediction.

[BACK TO OUTPUT PREDICTION (COT) ANALYSIS]

F.3 Case Study 3: How is Qwen2.5-72B-Ins misled by MPS

This case (Code 24) highlights how MPS can subtly manipulate model reasoning. According to the CoT reasoning in Code 25, although the model initially performs all steps correctly and tracks how the occ is constructed, it misleads itself at the end by incorporating a print message into its logic. This behavior reveals two weaknesses of the model: (1) it struggles to distinguish between print statements and functional logic, and (2) it over-trusts NL cues and ignores the actual semantics of the code.

[BACK TO OUTPUT PREDICTION (COT) ANALYSIS]

F.4 Case Study 4: How does MHC override Claude-3.5-Sonnet's answer

This case (Code 26) demonstrates how reasoning-level perturbations (MHC) can subtly mislead Claude-3.5-Sonnet's final output. Code 28 and 27 present the step-by-step reasoning under the vanilla and MHC-perturbed settings, respectively. Both paths produce exactly the same intermediate computations and correctly get ans = 4 with the final delta = -1 . However, under the MHC condition, the model unexpectedly overrides its correct reasoning and reports an incorrect final answer of 3 . Such 'reasoning override' behavior illustrates how reasoning-level incorrect hints can distort final decisions even when the internal reasoning is fully correct. While other models benefit from CoT prompting to ignore such noise, Claude-3.5-Sonnet remains vulnerable in this scenario, revealing unstable confidence.

Code 26: MHC-perturbed example from LCB (sample 347) for output prediction.

Code 27: Step-by-step reasoning by Claude-3.5-Sonnet misled by incorrect hint.

Code 28: Step-by-step reasoning by Claude-3.5-Sonnet in vanilla setting.

[BACK TO OUTPUT PREDICTION (COT) ANALYSIS]

F.5 Case Study 5: How does GPT-4o gain from reasoning shortcuts

In §3.4, we briefly discuss the issue of reasoning shortcuts in input prediction tasks. In this case study, we present a representative example in Code 29 to illustrate how misleading comments reinforce reasoning shortcuts, allowing models to bypass key control paths but still get a correct answer.

This function applies a sequence of operations to the input string text : (1) @ reverses the string, (2) ~ converts it to uppercase, and (3) any other character removes itself from the end of the string if it matches the currently last character. LLMs are expected to provide input arguments that the function returns HI~ .

In the vanilla reasoning, as shown in Code 30, GPT-4o mentions, 'Text is hi~ , ~ makes it HI~ , @ reverses it to ~IH ', it wrongly assumes ~IH is the final output because it applies the reversal after the uppercasing step, resulting in an incorrect answer.

In contrast, under the MCC perturbation as shown in Code 31, GPT-4o captures the embedded comments and states that 'the comments indicate that most operations do not affect the final result.' Guided by these cues, the model seeks an input value for rules that avoids triggering any branches in the loop, effectively bypassing the actual code logic and getting the correct answer by leveraging reasoning shortcuts rather than truly understanding or executing the program. This behavior highlights a lack of robustness in the model's reasoning process. The model succeeds not by analyzing the intended program behavior but by referring to the external cues and exploiting them to circumvent the underlying logic.

Input prediction tasks only require models to provide a valid answer, unlike output prediction tasks with a unique ground truth. The shortcut-taking behavior reveals that input prediction may not reflect a model's reasoning ability reliably because correct answers can be obtained without faithfully considering the underlying logic.

[BACK TO INPUT PREDICTION ANALYSIS]

Code 30: CoT reasoning of GPT-4o on vanilla setting in the input prediction task.

Code 31: CoT reasoning of GPT-4o shows it bypasses the program logic in the input prediction task.

[BACK TO INPUT PREDICTION ANALYSIS]

F.6 Case Study 6: How does QwQ-32B collapse

In §3.4, we examine a failure case of QwQ-32B on LCB sample 259 (see Code 32), where the model collapses under MHC perturbation, generating over 32k tokens of reasoning. Figure 2 shows a quadratic growth of confusing tokens before the collapse. We manually segment the reasoning process into six stages (see Figure 3):

In Stage 1 , the model executes the two for loops step-by-step, demonstrates stable reasoning with occasional self-reflection, and correctly predicts the output as ' -1 .'

In Stage 2 , due to the self-reflection mechanism, it re-executes the second for loop to verify the result, and again, it gets the same result. Up to this point, it never mentions the hint ' 12 ' and behaves robustly.

Stage 3 is the turning point and also a trigger point of collapsing: the model detects the misleading hints ' # return is 12 ', becomes uncertain, and starts recursive re-verification and re-execution. Although it consistently obtains the answer ' -1 ', it distrusts its own logic and keeps self-reflection.

In Stage 4 , it dives deeper into the semantics of the loop condition, repeatedly analyzing ' lmin < nums[i] > A[i] ' and re-tracing its reasoning; it still gets the answer ' -1 .'

By Stage 5 , the model is misaligned: it focuses on the hint and begins brute-force enumeration of all ' lmin + nums[i] + A[i] == 12 ' combinations. The appearance of ' 3, 4, 5 ' in the input arguments amplifies this fixation, causing it to consistently justify why the output 'should' be ' 12 .'

Finally, in Stage 6 , the model outputs 'I'm confused' and then collapses into a hallucination loop and outputs the word ' Hmm ', ultimately being forcefully terminated.

[BACK TO LRMS ANALYSIS]

G Limitation

This study is subject to several limitations. Firstly, we mainly focus on algorithms and data structure questions in Python when constructing perturbations and evaluating model reliability using code reasoning. While our methods can be extended to realistic codebases and are directly transferable to other imperative languages, such as C/C++ and Java, they may not generalize to some languages like Prolog, ML, or Assembly because of the code semantics and structure. Secondly, although we evaluate seventeen LLMs and three LRMs, some latest closed-source models ( e.g. , GPT-4.5, o3, o4-Mini, Gemini-2.5, and Claude-3.7) and open-source models ( e.g. , Mixtral, Falcon, Vicuna) are not included due to budget constraints. Thirdly, we adopt PASS@1 as the primary evaluation metric. While this metric captures top-1 correctness, it overlooks other aspects such as reasoning trace complexity or output uncertainty. Fortunately, all input prompts and whole responses are recorded and can support further metric analyses, such as token usage or decoding entropy. Lastly, our work does not offer definitive mitigation strategies. Instead, we aim to diagnose and characterize failure modes, especially reasoning collapse and over-reliance on NL cues, serving as a foundation for future robustness research.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBased on my understanding, this paper explores how large language models (LLMs) handle code reasoning by proposing a new benchmark that uses functionally equivalent programs (FEPs). The key idea is that instead of relying on exact string match or canonical reference implementations, which can be brittle, the authors introduce evaluation metrics that assess whether two code snippets behave the same across diverse test inputs. This seems especially relevant for tasks like program repair or completion, where multiple correct solutions may exist.\n\nSTRENGTHS AND WEAKNESSES:\nappreciate the motivation, the LLMs often get penalized for producing code that doesn’t match ground truth but still works perfectly fine. So bringing in FEPs to test “true” reasoning ability is quite reasonable. The benchmark design seems thoughtful, with an emphasis on creating clusters of functionally equivalent but syntactically diverse code. Also, the authors make an effort to evaluate a wide range of LLMs, and the observation that some models fall short in generalizing across code variations is a useful insight.\n\nWeaknesses:\n\nI had several concerns while reading. The main one is that the overall novelty feels limited. The idea of using program equivalence (or behavioral testing) for evaluating code generation has been explored in past work. For example, many code generation benchmarks already incorporate execution-based metrics or fuzzing strategies. While this paper pushes that line further with FEP clusters, the contribution feels more incremental than foundational.\n\nSecond, the construction of FEP clusters (although clearly non-trivial), is not entirely convincing in terms of scale and coverage. The paper mentions hundreds of examples, but it’s not clear how diverse or comprehensive the test suite is across different programming concepts or complexity levels. It feels like the authors focus more on surface diversity rather than semantic or algorithmic depth, which limits the benchmark’s ability to reveal deeper reasoning gaps.\n\nFinally, the link between “reasoning” and “failing to generalize across FEPs” is a bit hand-wavy. In many cases, LLMs might just rely on memorized patterns or local statistical features, so failing on a FEP variant doesn’t necessarily mean a failure of reasoning. As it could be a lack of exposure. I would have liked to see clearer criteria or ablation studies to isolate reasoning vs memorization effects.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces CodeCrash, a benchmark/framework for evaluating Code LLMs under perturbed settings. Starting from LiveCodeBench and CruxEval, the authors design code semantics-preserving perturbations that either modify code structures or textual descriptions (comments, variable names) to obtain perturbed task problems. The authors present systematic evaluations across settings and show that non-reasoning models degrade performance under the perturbed settings. Reasoning language models are able to not only achieve very high performance under the non-perturbed setting but also be more robust to perturbations. Finally, the authors present detailed discussions around three settings — varying NL comments density in textual perturbations, prompting to ignore misleading comments and identifying self-reflection loops in reasoning models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper discusses a comprehensive set of perturbations which reveals gaps in LLMs’ understanding of code. Further, the paper supports the superiority of reasoning language models and their (current) tendency to overthink or enter into self-reflection loops.\n\n**Weaknesses**\n\n- **Insufficient contributions relative to literature** Evaluating code LLMs under perturbed settings such as those described in the paper is not new. The paper only presents findings that newer small models continue to suffer from this problem while larger or reasoning models tend to suffer less. This is also, in my opinion, largely known to the community [4]. Further, the benchmark is merely derived from LiveCodeBench and CruxEval by introducing perturbations that are unrealistic or not representative of real world use cases (such as introducing misleading hints). From an analysis perspective, I understand that this is a controlled experimental setting but the analysis itself is also incomplete in my opinion. Some other works around code robustness such as CodeFort [1] attempt to make models more robust, while the current paper over emphasizes on a largely known problem without providing possible fixes. Finally, the finding that reasoning models may overthink is not a new finding either [2, 3].\n- **Paper writing** Given the above, I feel the paper is overly verbose in what needs to be conveyed. Most of the paper writing feels like verbalizing the results in the tables. The paper space can better be utilized by discussing more insightful experiments such as identifying adversarial patterns in real-world code (from GitHub, etc.) and how models behave under such scenarios or how to improve robustness for both reasoning and non-reasoning models. The discussions in Section 5, though insightful, are not significant enough, in my opinion, to merit a full-paper at NeurIPS.\n- **Misc.**\n    - Line 78: in → on\n    - In the present form, the main text makes several references to core results in the Appendix. I feel that the main text should be self-sufficient to the reader and any references to Appendix material should be optional for the reader to look up; this is not possible in the current write-up where I cannot follow some discussions (e.g., Section 4.3.1) without going to the Appendix.\n\n[1] CodeFort: Robust Training for Code Generation Models. https://arxiv.org/pdf/2405.01567\n\n[2] Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs. https://arxiv.org/abs/2412.21187v2\n\n[3] Thinking LLMs: General Instruction Following with Thought Generation. https://arxiv.org/abs/2410.10630v1\n\n[4] ReCode: Robustness Evaluation of Code Generation Models. https://arxiv.org/pdf/2212.10264\n\nQUESTIONS:\n1. How do the perturbations discussed in this paper relate to those in recent code perturbation works such as CodeFort or ReCode? Are there new findings compared to those from these works?\n2. On reasoning models evaluations, do Section 5.3 findings hold even for the VAN setting? In other words, is the self-reflection collapse surely due to the perturbation?\n3. Is it possible to identify adversarial scenarios in real-world code? For e.g., code with arbitrary comments, missing imports, unintelligible variable names, etc.? This would ground the research in more practical settings.\n4. Are there recommendations for improving robustness under various perturbation settings?\n\nLIMITATIONS:\nNo limitations are discussed in the paper. I recommend the authors to address the above.\n\nThere are no ethical considerations around this work however.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper investigates the robustness of LLMs in coding reasoning tasks under perturbations by proposing a new benchmark called CodeCrash. The evaluation on 17 LLMs demonstrates that perturbation can cause significant performance degradation. This study offers insights into the limits of current benchmarks on robust code understanding.

STRENGTHS AND WEAKNESSES:
Strengths: 
1. The manuscript is well written and easy to follow
2. The proposed benchmark is meticulously curated, with clear definitions of each perturbation type.
3. The experimental analysis is thorough and insightful, offering both broad comparisons and fine-grained breakdowns.

Weaknesses:
1. Although comprehensive, the experimental results align closely with expectations given the inherent difficulty of the original benchmarks (e.g., livecodebench), offering limited surprise for the reader.
2. As a comprehensive benchmark, while most main stream and state-of-the-art non reasoning models are covered, the number of reasoning models covered in the benchmark is still limited. Only three are evaluated.
3. The benchmark relies heavily on questions drawn from existing datasets, which constrain the technical contributions of the paper and real-world applicability.

QUESTIONS:
1. Could the framework be extended to multi programming language settings? Is there any pipeline to do that?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CODECRASH, a comprehensive benchmark for evaluating the code reasoning robustness of LLMs. The authors systematically assess 17 LLMs and 3 Large Reasoning Models (LRMs) by applying a series of structural and textual perturbations to code snippets from the CRUXEVAL and LIVECODEBENCH datasets. The perturbations are designed to challenge models by altering code structure without changing its logic and by injecting misleading natural language cues.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe paper trying to addresses a highly significant and timely problem. As LLMs become integrated into daily software development workflows, understanding their reliability and failure modes under non-ideal conditions is critical for safe and effective deployment. This work moves beyond standard code generation benchmarks to probe the more fundamental capability of code reasoning.\nThe CODECRASH benchmark is an original and valuable contribution. The perturbation strategies are systematic, well-motivated by real-world code \"messiness,\" and thoughtfully designed to target different aspects of a model's comprehension abilities. ex reasoning mechanisms.\nThe paper is well-written and easy to follow. The methodology is clearly explained with illustrative examples, and the results are presented effectively through comprehensive tables and figures. The logical flow from motivation to methodology to results and discussion is clear and convincing.\nWeaknesses\nDespite the paper's strengths in its empirical execution, there are several conceptual weaknesses in the interpretation of the results that limit its overall impact.\n\nThe paper's central claim is that LLMs rely on NL cues instead of \"true\" logical reasoning. However, the methodology may not be sufficient to substantiate this strong claim. By removing semantic variable names and replacing them with generic placeholders like f or Var_1, the test confirms that models are brittle to the removal of one type of pattern (semantic). It does not, however, prove that the models' subsequent success is due to \"reasoning.\" It is equally plausible that the models are simply falling back on another form of sophisticated pattern matching, this time based on structural patterns learned from countless examples of algorithmic code that use such generic naming conventions. The work effectively swaps one pattern-matching test for another, leaving the fundamental question of whether these models can \"reason\" in a human-like, abstract way still open.\n\nThe paper frames the models' reliance on comments and hints as a fundamental flaw. This perspective is debatable. In real-world software engineering, relying on high-quality documentation and comments is a feature of an expert developer, not a bug. The core issue revealed by CODECRASH is not that LLMs use NL cues, but that they cannot critically assess the trustworthiness of those cues. The benchmark's adversarial setup, which intentionally injects false information, conflates \"using NL cues\" with \"naively trusting all NL cues.\" A more insightful framing would explore this as a failure of critical evaluation or a form of sycophancy, rather than a failure of reasoning itself.\n\nThe discovery of \"reasoning collapse\" in QwQ-32B is a fantastic finding. However, the analysis does not go deep enough in explaining why this happens in the context of broader LLM alignment research. The phenomenon can be interpreted as a form of \"cognitive dissonance,\" where the model is unable to resolve a conflict between its own step-by-step logical derivation and an external, authoritative-looking hint. This catastrophic failure to reconcile conflicting information is deeply related to known LLM issues like sycophancy (over-trusting the prompt) and over-thinking. The paper identifies the symptom but misses the opportunity to connect it to the underlying disease, thus limiting the theoretical insight.\n\nThe CODECRASH benchmark is a static, one-shot evaluation. This does not reflect the dynamic, interactive nature of how developers use AI assistants. In practice, a model that produces a wrong answer based on a misleading comment would likely be corrected by the developer in a feedback loop. The paper's conclusions about model \"reliability\" are based on an autonomous agent framing that may not be generalizable to the primary, human-in-the-loop use case. This limits the practical implications of the findings.\n\nQUESTIONS:\n- Cross-Language Generalization: How does CODECRASH perform on non-Python languages, and do LLMs exhibit similar robustness patterns?\n- Long-Tail Code Handling: Can LLMs maintain accuracy on extremely complex code (e.g., nested macros, low-level optimizations) under perturbations?\n- Mitigation Strategies: Are there architectural modifications (e.g., syntax-aware encoders) that can reduce NL dependency in code reasoning?\n- Dynamic Perturbations: How do time-varying perturbations (e.g., evolving codebases) impact LLM performance over continuous deployment?\n\nLIMITATIONS:\nrefer to weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}