On Code-Induced Reasoning in LLMs
Authors: Abdul Waheed
Zhen Wu
Carolyn Rose
Daphne Ippolito
Venue: ICLR 2026 Poster (2026)
Keywords: code-induced reasoning
systematic perturbations
large language models
data-centric evaluation
Primary area: foundation or frontier models, including LLMs
TL;DR: We investigate which aspects of code benefit LLM reasoning through systematic perturbations and large-scale finetuning, evaluating across natural language, math, and code tasks.

Abstract: Code data has been shown to enhance the reasoning capabilities of large language models (LLMs), but it remains unclear which aspects of code are most responsible. We
investigate this question with a systematic, data-centric framework. We construct parallel instruction datasets across ten programming languages and introduce controlled
perturbations that selectively disrupt structural and semantic properties of code. We
then fine-tune LLMs from five model families and eight scales on each variant and
evaluate their performance on natural language, math, and code tasks. Across 3,331 experiments, our results show that LLMs are more vulnerable to structural perturbations
than semantic ones, particularly on math and code tasks. Appropriate abstractions like
pseudocode and flowcharts can be as effective as code, while encoding the same information with fewer tokens without adhering to original syntax can often retain or even
improve performance. Notably, even corrupted code with misleading signals remains
competitive when surface-level regularities persist. Finally, syntactic styles also shape
task-specific gains, with Python favoring natural language reasoning and lower-level
languages such as Java and Rust favoring math. Through our systematic framework,
we provide a fine-grained analysis of how different aspects of code influence reasoning
and inform the design of training data for enhancing LLM reasoning capabilities.

Published as a conference paper at ICLR 2026
ON CODE-INDUCED REASONING IN LLMs
Abdul Waheed" Zhen Wa" Carolyn Rosé Daphne Ippolito
Carnegie Mellon University {abdulw, zhenwu, cprose, dippolit]8cs. cmu, edu
Q
Code
Data
ABSTRACT
Code data has been shown to enhance the reasoning capabilities of large language mod- els (LLMs), but it remains unclear which aspects of code are most responsible. We investigate this question with a systematic, data-centric framework. We construct par- allel instruction datasets across ten programming languages and introduce controlled perturbations that selectively disrupt structural and semantic properties of code. We then fine-tune LLMs from five model families and cight scales each variant and evaluate their performance on natural language, math, and code tasks. Across 3,331 ex- periments, our results show that LLMs are more vulnerable to structural perturbations than semantic ones, particularly on math and code tasks. Appropriate abstractions like pseudocode and flowcharts can be as effective as code, while encoding the same infor- mation with fewer tokens without adhering to original syntax can often retain or even improve performance. Notably, even corrupted code with misleading signals remains competitive when surface-level regularities persist. Finally, syntactic styles also shape task-specific gains, with Python favoring natural language reasoning and lower-level languages such as Java and Rust favoring math. Through our systematic framework, we provide a fine-gree grained analysis of how different aspects of code influence reasoning and inform the design of training data for enhancing LLM reasoning capabilities.
INTRODUCTION
There has been substantial interest in the last several years in engineering language models that can tackle challenging reasoning tasks (Huang & Chang, 2023 Xu al.) 2025 Wei al. 2022 Kojima et al. 2023). Language reasoning tasks, such as math word problems or logic puzzles, tend to require multi-step. structured "thinking" in order to produce the correct answer. Recent work has found that
training the language model on code, either during pre-training (Fu & Khot 2022 Ma et al. 2023b) or in post-training (Zhang et al.] 2024b) stage, can improve its skill at reasoning tasks, even ones that are unrelated to programming. These prior works have hypothesized that the properties of code data, such as its logical consistency, compositional structure, and reduced ambiguity compared to natural language, provide effective signals that benefit reasoning. Despite the broad effectiveness of code data in training, we still lack a systematic, causal understanding of which specific properties of code drive these reasoning improvements: is it its syntactic regularity, structural abstractions, or linguistic styles?
In this work, systematically investigate which aspects of code provide effective training signals. To this end, we construct parallel instruction datasets in both natural language and code, and fur- ther expand the code dataset into language-speciff variants by generating responses in ten widely used programming languages. This design allows us to examine how structural differences across languages affect downstream reasoning. In addition, we introduce controlled perturbations to the code data to isolate contributing factors: (1) rule-based transformations such as whitespace removal or comment shuffling, and (2) generative transformations where GPT-4o-miní rewrites or reformats the code (e.g., with augmented comments, pseudocode, or flowcharts). We then fine-tune language models on each dataset variant, and evaluate them across natural language and general knowledge, math, as well as code understanding and generation tasks. Our contributions are:
"Equal contribution

---

Published as a conference paper at ICLR 2026
We introduce systematic framework to disentangle what aspects of code data improve rea- soning, combining parallel instruction data construction, controlled perturbations, and large- scale evaluation across five model families and eight scales.
We design a comprehensive and controlled suite of perturbations spanning rule-based edits and generative rewritings.
We provide new insights into the role of code in reasoning to inspire guidance on leveraging its structural and linguistic properties in future training data design.
RELATED WORK
Code data for LLM reasoning Recent work has increasingly demonstrated that incorporating code data can substantially improve the reasoning abilities LLMs. Prior studies show that adding code during pre-training or instruction-tuning consistently improves model performance across rea- soning tasks, domains, model scales and architectures (Ma et al., 2023a Zhang et al. [2024a] Yang et al. 2025b Aryabumi et al. 2024). Several works further explore the synergy between code and reasoning and highlight how code's structured and verifiable properties support logical decomposi- tion and intermediate step generation (Bi et al.; Yang al. (024).. This effect has been observed in multilingual contexts as well, where code-augmented training improves structured reasoning in under-resourced languages (Li et al. 2024). Complementary research focuses on code's impact for alignment and reward modeling, where pre-training with code-preference pairs or code-based intermediate steps can improve model calibration for -intensiee tasks (Yu al., 2024). The closest line of research to our work explores stress stress-testing LLMs with structural and semantic code perturbations (Lam al. 2025), which shows that small corruptions can significantly reduce reasoning performance.
Data impact on LLM performance The performance of LLMs tied to the vast amounts of training data, but the quality. composition, and characteristics of this data greatly shape their abili- ties (Wang et al. 2024 Li et al. 2023 Lee et al. 2022). For example, extensive analyses by] Longpre et al. (0024) have shown that pre-training data curation decisions for dataset age, composition, and content filtering have a systematic impact on downstream performance. These effects persist even after fine-tuning steps. Zhang et al. (2024c) demonstrate that poisoning little as 0.1% (and even 0.001 %) can produce persistent behavioral changes that survive instruction-tuning and alignment. In addition, |Havrilla & Iyer (2024) showed that LLMs sensitive to global, accumulative errors in fiaii--f-thouggt-ttuuttttee
training data, and that it is critical to filter out documents containing large amounts of dynamic, global noise during both pre-training and fine-tuning.
3 METHODOLOGY
We design a controlled experimental framework to understand what aspects of code improve rea- soning in language models. Our methodology consists of three stages: constructing parallel natural language and code instruction datasets ($3.1): applying systematic modifications to code instruction data p: and fine-tuning various language models on each dataset variant and then conducting evaluation
3.3
An overview of this framework is shown in Figure
3.1 INSTRUCTION DATA GENERATION
We construct two parallel instruction datasets: one in natural language and the other in code, each containing 120,000 instruction-ressonss pairs. We collect instructions from publicly available datasets, carefully process and filter them through deduplication and language-agnostic filtering, and augment the code data in a controlled way. This construction enables more controlled comparison of natural- and code-based instruction following under a unified training framework.
Code instructions We aggregate code instructions from Codeforces-CoT (Penedo et al. 2025 Lode-Instruction. 1 122K (TokenBender 2024).. Iyol-Instruct-Code 80k-v1 (nickrosh) 2024). Code-
Instruction (red Ixe 2023). Code-Instrucee Instruct-Sets Untleeniified, 2023), and Lodde-Instrut-AAl -izun-- (rombodawg] 2024). We aim to construct instruction data that is high-quality, diverse, and language-agnostic.
2

---

Published as a conference paper at ICLR 2026
1. Post-training Data Generation
2. Finetune
Natural bmggunge Instructions (1706)
Owend
a
rule bassd perturbations
arillapaae remioval
serialle renemingg
layword replacemant
commannt smapping
Code Instructions (120K)
generative perturbattons
madifialiion
cock cnmancomont
codo cofuccation
synthec GPTRo-min
pesudacode
codde in imaginary lan langlunge
3. Evaluation
odow solut Rlwcheer mai bilroo
Task suite
Jsea Inonterppt
PHP Pytnon
NL & general knckledge
Typedia p4 Cr
Rust
aingle progaamming langunge ablations
code
math
Figure 1: We construct parallel code and natural language instruction datasets, apply targeted mod- ifications (rule-based and generative-base and fine-tune separate LLM on each modified dataset. We then evaluate the resulting models across general natural language, code, and math reasoning tasks.
perturbations, single programming language ablations),
To ensure generality and eliminate redundancy, we first remove all exact-match duplicates across the datasets. We then filter out instructions that are explicitly programming terramminn-llnguageee law.llagnggseeeeeeeeee (e.g,, "Translate this code from Python to java") or whose solutions are inherently tied to particular do- mains, such web development databases (e.g., "webpage", "wehsite", "SQL", "HTMI.").
For each instruction, we prompt GPT-4o-min to generate answers in ten widely used programming languages: Java, JavaScript, PHP, Python, C#, TypeScript, C, C++, Go, and Rust. To create these variants, we design 20 language specification templates that explicitly request a solution in a given programming language (Table 6 For every instruction, we randomly select template, format with one of the target languages, and combine it with the general generation instructions to construct a complete prompt (Figure 55. From these generations. we sample 120K instruction-responee pairs with valid outputs, evenly distributed across all ten languages.
To assess the quality of our synthesized code instruction-resposse pairs, we perform comprehen- sive syntax and compilation check across all ten programming languages. For each instance, we extract the generated code block and apply standard syntax or compilation tools (e.g., ast parse for Python, gcc ffsyntax moox-only only for C, javac c for Java). As shown in Appendix Table 3 the majority of samples compile or execute successfully, with pass rates ranging from 64.08% (Type- Script) to 99. 25%25% (Python) and average pass rate of 82.59% across all languages. These results show that most generated instructions correspond to syntactically valid and exccutable code.
Natural language instructions We sample 120K examples from the OpenHermes 2.5 cor- pus (Teknium) 2023). We exclude instruction-respons pairs associated with categories unrelated to general-purpose instruction following, such as "agent" and "summarization" as well as those la- beled "coding" to ensure the dataset is entirely natural language. To maintain linguistic consistency, we further filter out non- Ennllilish examples. This filtered natural language subset complements our code instruction data, enabling fair comparison between code and natural language instructions.
3.2 SYSTEMATIC PERTURBATION DESIGN
To understand which specific structural and semantic properties are responsible for changes in rea- soning task performances, systematically perturb different aspects of the code dataset. We design the perturbations in two ways: rule-based (deterministic transformations) and generative (model- generated augmentations). Notably, perturbation strategies do not alter the number of examples the dataset. We illustrate examples these perturbations in Table with extended examples and token statisties Appendix A.]
Responses generated with temperature 0.6 and API-default decoding parameters.
3

---

Published as a conference paper at ICLR 2026
Table I: An example of perturbations (Section 3.2
applied to the same original snippet.
Fell Original Suippet
Type
Stralogy
Original Exerpt
Perturbed Excerpt
ded procass,at (inot..arri))))
vowela *acyaclAovaC1"
Whilespaze Removal
xescit. aspendi ("."
chax, chax.lowerd lowez 0)
rcsult appcnd (" mohhar Coner
result
[]
Vanible Renzming
foz chat in inpet.otrings
fox var.4 La vor.d; 1e var.4 noc An var.21
Raloshool
For chac in inpot.st.eing: 12 chat not in vowela: zeault.apseadd"
Keyword
Replacement
1f not
garply net in baz
(Nuncese)
chaz. lowez (2)
Keyword
Replacemest
Eaz chac input stzings
paza ca en entrada
(Non English)
retuen "1, (owncct
Comment
Swapping (Le-
Read input
Malking
Read ingut
cal
Angut rigutusttiing imput it asript
Procees and print the reauit peint (process ane log (inpat, ring))
Comment (Gubll))
Swapiigg
Frocess and print
f/ Queqe for process modes
the recalt
Remooal
input
temoved
Panuloods
for oher
FOR EACH chancter not wore THEN
input_st ring
appon +ineeros
chax not in vowels
Ganendire
Shep by Step
reselt. appendii"
Append before consonaus and convert lnvercase
char. lower
Flowchart
1f char not
(Read char] [Yowe17) <.<looez]
Code Imaginary
result, Cusclt..apper asppend
glot add levefoor)
guage
chaz. chaz.1ouez
Comment Eshaseement
Procesa and print the resule
Remonss lowch and profuct consonanty vith
Comment Obloccatoo
Read input
WARMING: Cade nomoon alivns; T0D0: hondle quanim vosels
3.2.1 RULE-BASED PERTURBATIONS
Rule-based perturbations apply deterministic transformations the code. They are designed to disrupt superficial patterns semantic signals that may influence model predictions without altering the core logic of the code. We describe five such perturbations below:
Whitespace removal All whitespace characters are removed from the code. This tests whether models rely on formatting heuristics, such as indentation or visual grouping of blocks, as implicit structural cues, particularly in languages like Python where whitespace is semantically meaningful.
Variable renaming We replace user-defined variables, function names, and class names with canon- ical placeholders of the form var-i, where E [0, n) and n is the total number of unique identifiers in the code snippet. This removes semantic cues conveyed by meaningful identifier names (e.g., counter, isSorted).
Programming language keyword replacement For each of the ten programming languages in dataset, identify its reserved keywords (e.g., f., return, def in Python) and substitute all occurrences of them using two strategies. The first replaces keywords with nonsense tokens (e.g.. foog quux). which have no semantic meaning in any language. In the second strategy, we use non-English but valid words (c.g., amigo, fleur), which are real words in various languages but semantically unrelated to the programming context. These perturbations aim challenge models' reliance on syntactic and semantic cues from familiar language constructs.
Comment removal We remove all inline and block comments from each code snippet. Code com- ments often provide useful semantic signals for program comprehension (Buse & Weimer 20099 De Souza et al. 2005). This perturbation tests whether models largely leverage such auxiliary natural-language cues.
Comment swapping We introduce local and global swapping that misplace code comments to dis- rupt the semantic alignment between code and documentation. In local swapping, comments within a snippet are randomly reordered, preserving their content but misaligning them with the relevant code segments. In global swapping, we first collect a global pool of comments from the entire dataset. Then, for each comment in a snippet, we replace it with randomly sampled comment from this pool. This results in documentation that is entirely mismatched to the surrounding code.
4

---

Published as a conference paper at ICLR 2026
3.2.2 GENERATIVE PERTURBATIONS
We create generative perturbations by prompting GPT-40-min[] to produce alternative versions of code responses generated according to Section These rewrites preserve the original intent of the code while introducing more diverse variations beyond what rule-based edits can achieve, allowing us to test model sensitivity and robustness to semantically equivalent inputs expressed in different forms. The full set of prompts used is available in Appendix [A.6]
Comment enhancement We prompt GPT-4o-mini to regenerate the code with high-quality docu- mentation and inline comments (Figure 6). The prompt emphasizes two forms of annotation: (1) comprehensive documentation comments for all functions, classes, and key code blocks to describe their purpose, parameters, return values, and assumptions; and (2) informative inline comments that clarify complex or non-obvious logic. These annotations follow the conventions of the target programming language (e.g., Python docstrings, JavaDoe). Unlike the often sparse comments in unperturbed data, the enhanced versions provide consistent, high-quality annotations, which enable us to test the effect of documentation on model performance.
Comment obfuscation Here, we generate deliberately misleading, irrelevant, nonsensical com- ments, while preserving the code's functionality (Figure 7. These include (1) inaccurate, off-topic, absurd documentation (e.g., references to astrology, cooking, or fictitious technologies) and (2) chaotic inline comments that contradict the code's functionality, reference imaginary bugs or fea- tures, and use distracting styles such ALL-CAPS, emojis, and fabricated jargon. This perturbation tests model robustness to extreme noise and deceptive annotations.
Pseudocode We convert code into high-level pseudocode while preserving its logical structure (Fig- ure 8 The model is instructed to replace language-specific syntax with pseudocode constructs (e.g., IF THEN. ENDIF, FOR EACH, etc.), remove low-level implementation details (e.g.. type declarations or library calls), and maintain the original control flow and indentation. This per- turbation evaluates whether models can reason over algorithmic intent without relying on concrete syntax, which offers insight into generalization across abstraction layers in code representation.
Flowchart in Markdown We generate control flow diagram using Mermaid syntax in Markdown for given code snippet (Figure()) The diagram captures all major control structures, such as loops, branches, function calls, and return points, using minimal but descriptive labels. This transformation renders executable code as graphical abstraction, allowing us to understand whether models can reason over symbolic control flow and align with underlying program semantics.
Step-by-step solution We rewrite code as numbered list of natural language steps (Figure 10). Each step preserves the program's 's logic and exccution order but uses declarative, Sanguage-agnoss phrasing (c.g., "Define function named...". "Check if the input is valid"). Unlike pscudocode or flowchart formats, this version entirely removes code or symbolic notation and instead emphasizes procedural understanding in purely narrative form.
Code imaginary language We translate real code into a fictional language that preserves structure and control flow but replaces all syntax and identifiers with invented tokens (Figure [T]. The result is semantically consistent yet entirely ungrounded in real languages. This perturbation allows us to examine whether models rely on surface-form familiarity (e.g., recognizing logical patterns) rather than true program semantics.
To assess the correctness of the perturbed data, we conduct human evaluation with two annota- tors, randomly sampling 30 examples per perturbation type (13 total: 7 rule-based and 6 generative). For the rule-based perturbations and comment cuhancement/bbuusaattion annotators verify that each transformation strictly follows the intended perturbation rule while leaving all unrelated content un- changed. For the generative perturbations (pseudocode, step-by-step instructions, flowchart, imag- inary language), which express the original code in alternative forms, annotators verify that the conveyed semantics remain faithful to the original program. Across all 390 sampled instances, 351 were judged correct (90% overall). Rule-based perturbations achieved 176/210 2s 84% correctness, while generative perturbations achieved 175/180 80 z 97% correctness.
We use temperature of 0.6 and default settings.
5

---

Published as a conference paper at ICLR 2026
3.3 MODEL TRAINING AND EVALUATION
We train a suite of decoder-only LLMs using supervised fine-tuning (SFT) on our instruc- tion-response datasets as detailed in Section [31] along with their perturbed variants described in Section 3.2] To assess the effect of language-specific patterns, additionally fine-tune models on subsets of the code data restricted to a single programming language. This allows us to examine how the syntactical diversity of programming languages influences reasoning performance. Each instruction-response pair is treated as a single input-output sequence, and models are trained to autoregressively predict the response tokens conditioned on the instruction and prior context. All models are fine-tuned from the same pre-trained backbone to ensure comparability across experi- mental conditions. Let = (II, 22...2..... Zmm be the instruction tokens and y = (3/1 y2 ................................ Yn) be the response tokens. The SFT objective is defined as:
CSFT
Σ
log Po(ye xt, Vet)
where P& denotes the model's conditional probability distribution parameterized by 0, and yer rep- resents the prefix of the response up to position
Models
We choose a diverse set of pre- and post-trained language models ranging from 0.6B to 8B parameters. Specifically, we experiment with models from five major families: Qwen3 YYang 2025a), LLaMA-3 (Grattafiori et al. 2024), Gemma3 (Team et al., 2025). OLMo2 (LLMo
et al.
al.
2024p. and SmolLM2 (Allal et 2025). For each model family, we select representative
sizes (e.g., 1IB, ~3-BB, 7 7-8878777 to evaluate performance across different scales.
Training data configurations Our base training set consists of 120K iesttruction-responee pairs spanning both code and natural language formats as detailed in Section 3.1 From this, we construct several configurations: (1) 100% code-only, (2) 100% natural language-only, and (3) mixed data with varying long-uaee ratios. In addition, we train models on each perturbed variant intro- duced in Section 3.2 Finally, we include gaggaannnnn-lnguggg--paee
subsets, training separate models on data from each of the ten languages (~12K examples per language) to assess the effect of language specialization. The implementation details are in Section A.5
Evaluation tasks We evaluate model performance across three categories: natural language and general knowledge, math, and code (
For natural language and general knowledge, we evaluate across commonsense reasoning, science, and textbook-style QA, logical reasoning, and custuuction-oolloinng All tasks are evaluated us- ing accuracy. For math, include both elementary and advanced problem-solving datasets (e.g., GSM8K, HRM8K), as well as arithmetic and math-related subsets of MMLU. Open-ended tasks (GSM8K, HRM8K) use exact match, while arithmetic and MMLU (math) are scored with accuracy.
For code, we evaluate both code understanding and generation. Based on preliminary experiments, adopt the LLLM-as-Judge paradigm (Zheng et al.) 2023a Gu et al.) (025) instead of execution- based evaluation (Huang et al. (0022). Our relatively small, perturbed models often fail to produce fully executable code, making execution-bert based metrics unreliable. More importantly, our goal is to assess code quality and reasoning under perturbations, not just execution success.
Thus, we prompt GPT-4o-mint to first generate an cestance-specifie rubric on 1-10 Likert-type scale given the original instruction, which is expected to capture nuanced quality variation across outputs. The same model is then prompted as a judge to provide brief reasoning step ("thought") and assign score based on that rubric. Examples of the rubric-generation prompt and judging prompt are shown in Appendix A.6 (Figures an] 13). For the main results, we use GPT-4o-mini as the judge due to its strong judging quality and favorable cost-performance tradeoff. To assess the reliability of our LLM-as-judge setup, we additionally conduct an extensive cross-judge analysis using multiple models. The results in Appendix Table demonstrate that our evaluation is stable across judges.
*Due resource constraint, the largest model we could fully finetune is 8B.

---

Published as a conference paper at ICLR 2026
4 RESULTS AND DISCUSSION
RQ1: Does incorporating code in fine-tuning improve task performance? First, we validate prior findings that fine-tuning on code data can enhance reasoning. Following the training setup in Section 3.3 we compare performance across four settings: zero-shot, full code fine-tuning ("code-
ft"), full natural language fine-tuning ("nl-ft"), and mixed data fine-tuning with equal proportions of code and natural language instructions ("mixed-ft") Across model families and scales, code- and mixed-ft generally achieve leading or competitive performance across tasks (Figure 2 and
Figures 14 18) with the trend particularly consistent code generation.
0.75
6.5-
0.563
7.576
0.535
0.311
0.545
0.553
7.003
6.945
00
miiedd A
codd
n
mined
codde R
mited
code
Figure 2: Performance (with stderr bars) of Qwen3-4B-Ban across zero-shot, full code fine-tuning (code-ft), full natural language fine- tuning (nl-ft), and 50-50 code to NL data ratio fine-tuning (mixed ft). Incorporating code improves performance across tasks.
Overall, across the 14 model bases, either code-ft mixed-ft achieves the best performance 64% of natural language tasks, 86% of math and code understanding tasks, and all code generation tasks. Motivated by this, we further examine the effect of varying the proportion of code in mixed fine- tuning (Figure 19). We find that higher fractions of code data generally improve performance across most tasks, with math tasks most sensitive to mixture ratios.
RQ2: How do our systematic perturbations affect performance?
Section Findings
Structural perturbations hurt more than semantic ones, especially for math and code.
Appropriate abstractions, such pseudocode and flowcharts, can substitute for explicit code structure in reasoning.
Models don't need verbose code: reduced-token variants perform well as long as core infor- mation is preserved.
LLMS can reason effectively from corrupted code by exploiting surface-level regularities.
Based on the properties of each perturbation, group them into distinct analysis axes that allow to systemat- ically probe their effects. The grouping details are in Table 5 We present the results for individual perturbations Appendix appendixAA.7.6
Next, we analyze task performance under the perturbations introduced in Section 3.2
Structural vs. Semantics Perturbations. We define structural perturbations as edits that alter the syntactic scaffolding or formatting of code (e.g., whitespace removal, pseudocode, flowcharts), while semantic perturbations modify meaning bearingbbrig tokens such as identifiers, keywords, or com- ments without disrupting the underlying structure. Across model families and scales (Figures 20 24 nearly all perturbations reduce performance compared to the unperturbed code fine-tuned base- line. More importantly, structural perturbations consistently degrade performance more severely than semantic ones, especially for math and code tasks (c.g., Figure Ba). The discrepancy is more evident as models scale up (e.g., Figure 20). This resembles prior work that reasoning structure rather than content is more critical to the learning process (Li al. 2025). We hypothesize that tasks such as math and code rely more heavily on formatting and layout cues to shape reasoning.
Explicitness of Code Structure. Building the importance of structure, examine perturba- tions along a spectrum of how explicitly they preserve code structure: from runnable or code-like forms, through intermediate abstractions such as pseudocode and flowcharts, natural language

---

Published as a conference paper at ICLR 2026
Urtero)
Nyts
Case Unrestardin
Code teneratonn
5.625
twrformncs
8.575
0.545
0.544
0.75
0.729
0.592
8,203
0,550
0.570
0.525
2,500
75
2.475
0.50
78
(a) Structural (e.g. removing whitespace) vs. semantics perturbations (e.g. modifying the comments) of Qwen3-4B-Base. Structural perturbations tend to he more harmful to performance than semantic ones.
Mt. Direnda
9.53
7872
655
60
(b) Performance under levels of explicitness of code structure (less explicit going from runnable code to NL procedure) of Qwen3-8B-BP Certain algorithmic and graphical abstractions benefit reasoning.
101
Cidde Oemendiio
043
0.64
0.423
0.0
0.466
0.355
0.443
0.44
0,62
3.428
5-520
646
5,0
(c) Performance under token count reductions w.r.t unperturbed code (Qwen3-0.6B.
Base). Reductions can perform comparable even better than the baseline.
0622
0.511
0.784
0.689
EXC-
5--
0.30
6.5
75
(d) Performance vs. human interpretability of perturbed code (Qwen3-8B-BB) Base). Low- interpretability with misleading signals can match perform better than other configu- rations.
Figure 3: Aggregated performance (with stderr bars) across RQ2 perturbation settings: structural vs. semantic perturbations, explicitness of code structure, token density reductions, and human in- terpretability.
step-by-step procedures. For code generation, where executable outputs are required, it is natural that perturbations that preserve explicit code structure, whether runnable or not, lead to the best performance. For other tasks, however, certain abstractions such pseudocode flowcharts often match or even surpass unperturbed code, as they highlight algorithmic structure while removing superficial syntax. By contrast, the most implicit form, natural language procedures, provides little advantage and generally performs worst across tasks (e.g. Figure 3b] Figures 29)
Relative Information Density. Because constructed instruction datasets parallel, the amount of information they convey about the code is comparable across perturbations. We define relative information density as (number of tokens in perturbed dataset) (number of tokens in the original
8

---

Published as a conference paper at ICLR 2026
code-ft dataset), which reflects how compactly the same content is represented. Perturbations differ in how they adjust density: highly compact forms strip away most tokens while preserving the algorithmic skeleton (c.g,, flowcharts, pseudocode), moderately reduced forms remove comments or use imaginary languages, and others preserve or even increase density through verbose variable renamings or enriched documentation. We find that strong moderate reductions in density often perform close to, and sometimes better than, the baseline (c.g., Figure 3c] Figures (4) However, this advantage doesn't it extend to code generation, where preserving richer surface detail is important. In addition, smaller models are more sensitive to density differences, whereas larger models remain robust. Overall, this suggests that the benefit of code for reasoning doesn't lie in its verbosity but in the efficiency with which essential information is preserved.
Human Interpretability. We also examine perturbations through the lens of human readability: high-interpretability (enriched explanations and visual scaffolds), medium (local edits leaving most code intact), and low (obscured readability or misleading signals). Interestingly, wwwinteerreetabiltyy variants, despite adding noise distortion, often do not degrade performance too much from the unperturbed baseline, and often match or even surpass medium--nteeppettbiiity ones (e.g. Figure 3dl Figures 35 39). This counterintuitive trend suggests that the models could exploit surface-level regularities and recurring structural cues that persist even in noisy or opaque forms.
RQ3: How does performance vary across programming languages?
Section Findings
Lower-level languages benefit math tasks.
Python aligns best with NL tasks, while Java and Rust often rank among the top for math.
Milccde
201 chet
0.234
4,27'
0.322
0.4
70
Rgg Rolplng
right supto
intemadato
High scrttig
System
High sclkiig
Pyther
Rac-
CSham
-3441
Pytton
Rict
0,462
4.425
0.662
PHP
3672
5442
JwssSrpp:
5,62
CPP
Ga-
Go
1.458
TyooScip:
2.15
3p:52
6.488
Juya
0-624
Tpe
Pytten
8.497
C5harp
jivo
0.527
PHP-
0325
4.322
PyShon
4.544
CShup
4340
Ruc
4,528
2.518
1245
RP
Juza
dP
834
Figure 4: Performance (with stderr bars) of Qwen3-1.78 Top: grouped by abstraction level (low- system, intermediate, high segh-scripting.. Low-system and intermediate languages outperform on math. Botlom: individual programming languages. Python aligns best with NI., Rust leads on math.
The impact of structure in RQ2 motivates the question whether syntactic regularities in program- ming languages also influence model performance. To explore this, we group the ten programming languages into high-scripting (Python, PHI, JavaScript, TypeScript), intermediate (Java, C#), and low-system (C, C++. Rust, Go) according to their abstraction level. Overall, differences across groups are small. On NL and code tasks, the impact of language groups is largely model-dependent. However, math tasks, most high-scripting languages consistently underperform relative in-
9

---

Published as a conference paper at ICLR 2026
termediate and low-system ones (e.g. top Figure Figures 45 48a
We hypothesize that richer structural detail in lower-level languages provides beneficial signals for mathematical reasoning.
For code generation, fine-tuning any single language improves full code fine-tuning, which suggests the benefit of multi-languuae diversity for code generation. At the individual language level (e.g. bottom Figure 4 Figures (46448b) across models, Python often leads on NL tasks, probably due to its surface form being closer to natural language. Aligning with the group-level results, lower-level languages such as Java and Rust often rank among the top for math. For code tasks that span multiple languages, results are more mixed, with no clear leaders, and performance gaps remain relatively small.
zero-shot but lags behind
CONCLUSION
In this work, investigate what aspects of code enhance reasoning in LLMs and which factors matter most. Through 3,331 experiments spanning five model families, eight scales, ten program- ming languages, and suite of systematic perturbations, structural properties of code are critical: disrupting them leads to consistent performance drops, especially on math and code tasks. Second, appropriate abstractions and efficient encodings can be just as effective as raw code. Moreover, models remain surprisingly robust even to corrupted or www-ineepreetability code, exploiting regularities that persist despite surface distortions. Finally, lower-level programming languages provide more benefits for math tasks. Together, we provide more precise account how code supports reasoning and point toward practical design principles for constructing effective training data beyond executable programs.
arrive at four central conclusions. First,
6 LIMITATIONS AND FUTURE WORK
Our study focuses on small- to mid-size base models due to resource constraints. Our perturbations, although diverse, do not exhaustively cover all factors that may influence reasoning, such as code complexity or training data diversity. While we do perform manual inspection of small subset of perturbed examples to verify that transformations behaved as intended, the scale of our dataset precluded comprehensive human evaluation. Finally, although evaluate across a broad suite of reasoning tasks, our benchmarks still capture only part of the reasoning spectrum.
Future work could extend our framework to larger and instruction-tuned models examine whether the trends we observe persist at scale. Exploring additional domains such as of-innof-thhught rea- soning and preference optimization would further clarify the mechanisms by which code shapes reasoning. More broadly, our findings on abstraction level and information density point toward practical guidelines for training data curation that could be operationalized into automated data lection pipelines.
REPRODUCIBILITY STATEMENT
We provide extensive details throughout the paper and supplementary materials. Section 3.1 de- scribes the construction and processing of both the code and natural language datasets. Section A.5 outlines model training and implementation details. Appendix A.6 includes all prompts used for data generation, perturbations, and LLM-as- evaluation.
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlfček, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. Smollm2: When smol goes big- big-data-centrie training of small language model. arXiv preprint 2rXiv:2502.02737, 2025.
Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr F. Locatelli, Marzieh Fadace, A. Ustun, and Sara Hooker. To code, or not to code? exploring impact of code in pre- training. ArXiv, abs/2408.100 10914, 2024. URL https:/ /api. semanticschon .ow.ooorrrroorooooooooooooooo CorpusTD:27190955
10

---

Published as a conference paper at ICLR 2026
AtlasUnified.
Code-instruct-sets:
f Sets, 2023.
Zhen Bi, Ningyu Zhang, Yinuo Jiang, Shumin Deng, Guozhou Zheng, and Huajun Chen. When do program-of-thought works for reasoning? AAAI 2025.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about
physical commonsense natural language, 2019.
116 641
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.
Raymond PL. Buse and Westley Weimer. Learning a metric for code readability. IEEE Transac- fions on software engineering, 36(4):546-558, 2009.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL httttttss//aarxt wwwpxiivvorr///b/. wtt.oo//////////0.0500522 05457
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrro Nakano. Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv
arg/abs/211 14168
Sergio Cozzetti B De Souza, Nicolas Anquetil, and Káthia M De Oliveira. A study of the docu- mentation essential to software maintenance. In Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive information, pp. 68 (8877,22 2005.
Hao Fu, Yao; Peng and Tushar Khot. How does gpt obtain its ability? tracing emergent abilities of language models their sources. Yao Fu's Notion, Dec 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjïe Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on Ilm-as-a-judge, 2025. URL hhttp:://aaat wralvvorrg/aae abs/ 2411. 15594
Alex Havrilla and Maia Iyer. Understanding the effect of noise in Ilm training data with algorithmic chains of thought, 2024. URRphhtss///axiv..oom
/bs.//42.. 04004
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding, 2021. URL https:
03300
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A sur- vey. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.), Findings of the As- sociation for Computational Linguistics: ACL 2023, pp. 1049-1065. Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. https: antho ogy. .wt.orrgg/oo..o 2023. f findiiiiii www.gnssll.....o acl.6.1.11 677
URL
Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan. Haotian Cui, Jeevana Priya Inala, Colin Clement, Nan Duan, and Jianfeng Gao. Execution based evaluation for data science code gener- ation models. arXiv preprint arXiv:2211.099 1.09374, 2022.

---

Published as a conference paper at ICLR 2026
Hyunwoo Ko, Guijin Son, and Dasol Choi. Understand, solve and translate: Bridging the multilin- gual mathematical reasoning gap, 2025. URL http:/ptt////tttta org. wttp//b///a/ ww..aaaa5000 2501. 02448
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners, January 2023. URL http::/axxii.ooe
2205 11916 arXiv:2205. 11916 11916 [cs].
Man Ho Lam, Chaozheng Wang, Jen-Tse Huang, and Michael R Lyu. CodeCrash: Stress testing LLM reasoning under structural and semantic perturbations. arXiv [cs.Al], April 2025.
Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better, 2022. URL hatttps:/aaxxiv.orrt abs/ /aoas/210.... 06499
Bryan Li, Tamer Alkhouli, Daniele Bonadiman, Nikolaos Pappas, and Saab Mansour. Eliciting better multilingual structured reasoning from LLMs through code. In Lun-Wei Ku, Andre Mar- tins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5154 5169, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1/235////222444llllog
URL
2817
Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G Patil, Matei Zaharia, al. Lims can easily learn reason from demon- strations structure, not content, is what matters! arXiv preprint arXiv:2502.07374 2025.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting IIm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308. 12032, 2023.
Jian Liu, Leyang Cui, Hammeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning, 2020. URL https://arxie org/ abs 2010 08124
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pretrainer's ' guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Pupers), pp. 3245-3276, Mexico City, Mexico, June 1/2653////022.naaccll
2024. Association for Computational Linguistics. doi: antheaaatttooloooooooooooo logy- www.g//00000 www.naaaclloooo 1797
179. URL
Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help LLMs reasoning? arXiv [cs.CL], September 2023a.
Yingwei Ma. Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help iims reasoning? arXiv preprint arXiv:2309. 16298, 2023b.
Dung Nguyen Manh, Thang Phan Chau, Nam Le Hai, Thong T Doan, Nam V Nguyen, Quang Pham, and Nghi DQ Bui. Codemmlu: A multi-task benchmark for assessing code understanding capabilities of codellms. arXiv preprint 21t.::241000199//0 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering, 2018. URL bttp:://///axx btg/abs/18880 02789
nickrosh. htgpl/hwwo.oolloa.o//gg///g///en Evol - 1otol=-inntuuct--oe--------
2024.
Team OLMo, Pete Walsh. Luca Soldaini, Dirk Groeneveld, Kyle Lo. Shane Arora, Akshita Bhagia, Yuling Gu Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501. 0066665 2024.
12

---

Published as a conference paper at ICLR 2026
Guilherme Penedo, Anton Lozhkov, Hynek Kydlíččk, Loubna Ben Allal, Edward Beeching, Agustín Piqueres Lajarin, Quentin Gallouédec, Nathan Habib. Lewis Tunstall, and Lean- dro von Werra. Codeforces cots. https://wggnnng.eec../atttt/e//p codeforces cots. 2025.
redixe. code inde_iisttuctions. instrucioons 2023.
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Min- jia Zhang, Dong Li, and Yuxiong He. {Zero-offload): Democratizing (billion-scale) model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pp. 551-564, 2021.
rombodawg. code instruct alpaca vieuna wizardlm 56k backup.
https:/huggingface
co/datasetss backup 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist 1lm assistants. https: 2.5 2023. Accessed via
//huggingface.. ca htt////atseessttttnimm////ppepea Hugging Face Datasets.
TokenBender.
https:/huggingface co/
http:/tttooostnia.aaaaaaaaaaatta 2024.
Jiahao Wang, Bolin Zhang, Qianlong Du, Jiajun Zhang, and Dianhui Chu. A survey on data selection for I1m instruction tuning. arXiv preprint arXiv:2442.051123 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. offin-of-thoughhe prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-24831 2022.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang. Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li. Jie Feng, Chen Gao, and Yong Li. Towards large reasoning models: A survey of reinforced reasoning with large language models, 2025. URI.
https://erxit www.xxiv..gavaa.. 2201 09688
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.003388 2025a.
Dayu Yang, Tianyang Liu, Daoan Zhang, Antoine Simoulin, Xiaoyi Liu, Yuwei Cao, Zhaopu Teng, Xin Qian, Grey Yang, Jiebo Luo, and Julian McAuley. Code to think, think to code: A survey on code-enhanced reasoning and reasoning-driven code intelligence LLMs. arXiv [cs. (Cs.CL), February 2025b.
Ke Yang. Jiateng Liu, John Wu, Chaoqi Yang, YiR Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Yiquan Wang, Heng Ji, and Chengxiang Zhai. If LLM is the wizard, then code is the wand: A survey how code empowers large language models to serve intelligent agents. arXiv [cs.CL), January 2024.
Huimu Yu, Xing Wu, Haotian Xu, Debing Zhang, and Songlin Hu. CodePMP: Scalable preference model pretraining for large language model reasoning. arXiv [cs.Al], October 2024.
Xinlu Zhang, Zhiyu Zoey Chen, Xi Ye, Xianjun Yang. Lichang Chen, William Yang Wang, and Linda Ruth Petzold. Unveiling the impact of coding data instruction fine-tuning on large language models reasoning. arXiv [cs.Al], May 2024a.
Xinlu Zhang, Zhiyu Zoey Chen, Xi Ye, Xianjun Yang. Lichang Chen, William Yang Wang, and Linda Ruth Petzold. Unveiling the impact of coding data instruction fine-tuning on large language models reasoning, 2024b. URL htt.llttss////t
org/ abs 2405. 20535
13

---

Published as a conference paper at ICLR 2026
Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramer, and Daphne Ippolito. Persistent pre-training poisoning of Mms., 2024c. URL
https://arxiz: org/ abs 2410.13722
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang. Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023a. URLhttps: http:///axvvvvvvoe org/ abs/230 05685
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5673-5684, 2023b.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguis- tics. UtLp:ttp://axxiv.c
alss 22.p/2403..0 13372
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follooiing evaluation for large language models, 2023. URL https:
//arxiv warxxw..rrg/bbs//31117777
A APPENDIX
A.1
EXTENDED DETAILS OF PERTURBATION DATA
For each perturbation, we report the original and transformed code along with token statistics in Table 2
Table 2: Perturbation type along with the corresponding original and transformed excerpts and token statistics, illustrating how different perturbations alter code structure and information density.
Pertarbation
Original Excerpt
Perturhed Excerpt
Total Tokens
Arg Tokens per Instruction
whitespace removaa
Foe chass in Imput aat slngg
Zorchar wni.apttttttinn
78.353,430
634.61
corisblle renaming
for ohar in inpatustiringg
for weruf in varil:
87,619, 500
730 16
keyward replaced with non sense
12 not voNelas
qeeply not in vowels:
87,123,587
726.03
keyword replaced with Eaglish
11 not vaselas
glow not. in vowelas
88,07%.906
733.59
comment removal
input
80,238,050
668 65
comment swapping
Input
Process print the
85,324,436
711.04
reault
global comment swapping
Prococs and peint the result
Queue for procescing nodes
85,129,862
Blocchrt (Markdow))
chaz not in vowels:
IRaad chasl + (Voual3) [over] 84,250,378
67,553,411
[pprend
step by step explanation
result. srscie.sppendi".""
apperd ..* betore
702.09
chaz lowex for char in inpatustring:
conacnant a FOR EACH sharacter
pandoode
73,722,933
614,36
not vovel THES gloff add '
imaginary language
seault wwppendappwedd..... (o)er())
81,011,032
675.09
Procesa the resuit
vevels comsonentt
119,3986611
comment
Read input
WARNING: Coso may sumnon cliena
111,771,640
A.2 VERIFICATION OF QUALITY OF SYNTHETIC CODE DATA
As shown in Table most samples pass syntax and compilation checks across languages, with an average pass rate of 82.59%. This indicates that our code data is largely syntactically valid.
A.3 EVALUATION SUITE DETAILS
We provide details about evaluation data in Table 4
14

---

Published as a conference paper at ICLR 2026
Table Syntax and compilation check results across all ten programming languages. The majority of samples successfully compiled or executed, with a mean pass rate of 82.59%
Language
Total
% Passed
11,998
81.49
PHP
12,009
94.81
JavaScript
11,996
91.57
Python
11,993
99.25
C++
11,997
83.20
TypeScript
12,001
64.08
Rust
11,995
66.71
C#
11,996
81.06
Go
12,012
77.77
Java
12,003
88.94
Table 4: Evaluation suite spanning natural language and general knowledge, math, and code tasks.
Task Type
Topic
Benchmarks
Metric
Commonsense
PIQA (Bisk al. 2019
ARC-Easy (Clark et al.
2018
Natural Language
ARC-Challenge (Clark al.
2018
Accuracy
& General Knowledge
Science / Textbook
OpenBookQA (Mhhaylov al
2018
MMLU (non (non-math) (Hendryck
2021
Logic-Heavy
LogiQA (Liu et al.
2020
Instruction Following
FFaal Zhou et al
2023
Prompt-level Ac- curacy
GSM8K (Cobhe al. 2021
Exact Match
HRM8K Ko
Math
Arithmetic (Brown et al. 2020
Accuracy
MMLU (math) [Hendrycks
2021
Code Understanddng
CodeMMLU Manh al.
2024
Accuracy
Code
Code Generation
HumanFvalX (hhegg et al. 2023b
I.IM-as-Judge
A.4 CATEGORIZATION OF PERTURBATIONS FOR RQ2 ANALYSIS
We group all perturbations across four axes: structural vs. semantic changes, explicitness of code structure (ECS), relative information density (RID), and human interpretability (HI), summarized in Table 5
A.5 IMPLEMENTATION DETAILS
We train all models under identical hyperparameter settings to ensure fair comparison across model sizes and data configurations. All experiments are conducted using full fine-tuning in BF16 precision with a maximum sequence length of 2048 tokens. We run all experiments on 4 A100 80G node. Models are trained for epochs with cumulative batch size of 64 for most experiments, except for language-specific settings, where the batch size is reduced to 32. The learning rate is fixed at 1e-5 and follows a cosine decay schedule with a warmup ratio of 0.1. For memory-efficient parallelism and distributed training. using the LLaMA-Factory (Zheng et al.) 2024). All other parameters and configurations follow the default setting unless otherwise specified.
DeepSpeed ZeRO Stage 3 (Ren et al. 2021). All models trained
A.6 PROMPTS
Standard generation prompt We provide the standard prompt to generate code for given in- struction in specific language in Figure where the instruction can be instantiated using one of the templates in Table
15

---

Published as a conference paper at ICLR 2026
Table 5: Categorization of perturbations across four analysis axes: structural vs. semantic (S/S) perturbations, explicitness of code structure (ECS), relative information density (RID), and human interpretability (HI).
Perturbation
S/S Perturbations
ECS
RID
HI
Whitespace removal
Broken syntax
Moderate Moderate-reducce reduced
Medium
Pscudocode
Algorithmic
Strong-reduud croong-reduced
High
Imaginary
Structural
Broken syntax
moderate-reducces
Low
Step-by-step
NL procedure
Moderate recrueed-reuedd
High
Flowchart
Graphical
Strong-read reduced
High
Comment removal
Runnable
Moderate-reduced
Medium
Variable renaming
Runnable
Increased
Medium
Keyword repl. (nonsense)
Broken syntax
Increased
Low
Keyword repl. (non-Fing.)
Broken syntax
Increased
Low
Semantic
Comment swap (global)
Runnable
Near-baseline
Low
Comment swap (local)
Runnable
Near-baseline
Low
Comment enhancement
Runnable
Increased
High
Comment obfuscation
Runnable
Increased
Low
Table Language specification templates with placeholders that can instantiated with different programming languages.
Generate the code in (language).
Provide code in [language].
Write the code (language). Draft the code in (language).
Build the code using (language].
Create the code using (language). )
Produce code snippet in (language).
Develop the code using {language].
Generate solution in {language].
Create script in (languago)-
Implement the code in (languago).
Design the code in (language).
Construct code using (language)-
Format code (language), Write a function in (language).
Write program (language) Deliver the code in {language).
Prepare code snippet in (language).
Comment enhancement prompt The prompt to enhance the quality and readability of given code snippet by adding detailed documentation is shown in Figure
Comment obfuscation prompt given instruction is presented in Figure
The prompt used to generate obfuscated versions of code from
Pseudo generation prompt instruction in Figure
We illustrate the prompt designed to produce pseudocode for given
Flowchart generation prompt instruction is provided Figure
The prompt for generating flowchart-style representation of an
Step-by-step implementation guide generation prompt The prompt used to create sequential step-by-step implementation guide for an instruction is shown in Figure TO
Imaginary language code generation We paragraph the prompt for generating code in imagi- nary programming language in Figure II
LLM-as-Judhe Evaluation We the prompt shown in Figure 12 to generate instance-specific rubrics for LLM-as-judge evaluation on the code generation task. The prompt to evaluate model response is shown in the Figure 13
16

---

Published as a conference paper at ICLR 2026
Code Instruction Data Generation Prompt
tacked with generating code based on specifled programming language and Instruction. goal generate
the syntax and semautics of the specifled language. If the instruction Incalid (c-g., contradicts the language's rules or references functions or constructs from different language), you must strietly respond with "iwvalid."
code should itself. Imaalid Instraction: - the inctraction references construcss, functions, or systax not supported by the speciliod languege respard with "invaldd". Do not attenge to conrect the invalid instraction just respand with "invalid" Do not provide reason or explanation for why the instraction is invalad.
systactically semuntically follow standard conventions and best practices for given language.
Guidelines: Valid Code: The gesemied code
accoeding the specified langaage
provide any explanation valid code only output
Examples:
Examplc
Instraction:
a function convert list to set"
Language: Python
Response:
list__ te_.tt__eeeee set Cangut 1ib)) iipur
Esample
cluss method prints 'Hello" using comooo.log0
Language: Python
Response: im-lid
Example
Instraction:
including hidden ones, the current directory."
Langage: Shell
Response: Is
Esample
Instruction: "Deline function using 'def' that longth string"
Language: JavaScript
Respunse: invlid
If the instruction valid, output the code directly tho explanation).. If the instruction insalid, respond with "invalid" (no explatation).
Input:
Instruction: finstraction)
Language: (language)
Output:
( (sesponse)
Figure 5: Code instruction data generation prompt. The task is to generate valid code or respond with "invalid" for unsupported instructions.
Comment Enhancement Prompt
You tasked with eahancing the response the given code instrnction by adding meaningfal comments and decumentation. The goal improve code's readability maintainability. clasity any pengramming language, without altering ociginal Your modilications must include:
1. Decumentation Comments: Add elear, technically accarste, and concise dncamentation for every function, mathed, clus, and major eode bookk. and usage), remurn values, and oosumptions notes relevant to conrect usage. Use the progumming langage (e... Python doestrings, IwaDoe 2. Comments: Insert informative and contestaally helpfal inline comments near complex, usiatuitive, important openaions. Foens explaining logic, control flow, edge case handling, design decisions, or dependencies. Avoid redundiat, obvious, or overly literal comments (egg, avoid "I = 0.7 set to D").
Deserite the
all parmeters (with conect
standard docomentation
oppropriate
Dosygen C/C++).
Guidelines:
Do not change the logic, structure, or behavior of origini code not introduce new functionality, abstrations, or fumnatting changes. Keep comments srictly technical, relevant, and meefil-areed verbosily or informal rone Do include any mota-commetss or explanntory notes about what was changed (e.g.. no "This version ad& comments" or similar). Apply these medifications only the code portion of the original response. Leave any parts completely unmodified.. Ensure This Iingugg-ognooticc and applies sny of following: Python, lava, C++, IaesSeript TypuScript, Go, Rost, CR, Ruby, Swin, Korlin, Scala, and Shell.
convenions of language use.
Inpul: Instructio:: (instraction) Original Response: (mespone)
Outgot: ((updared response) }
Figure 6: Comment enhancement prompt. The task is to improve code clarity through meaningful comments while preserving original functionality.
A.7
EXTENDED RESULTS
A.7.1
TASK PERFORMANCE SHOWCASING CODE DATA IMPACT IN FINETUNING (RQ1)
Qwen3 model family results See task performance of zero-shot, full code finetuned, full natural language finetuned, and code-NL. mixed finetuned models in Figure
17

---

Published as a conference paper at ICLR 2026
Comment Obfuscation Prompt
You are tacked modifylng the response to given code instruction a way significantly degrades Its quality and clarity. Your goal is make the code as coufusinng unhelpful, and indleading as possible through the use deliberately bad, distracting, and nousensical comments- Withoot altering the actual lopic or functicnallty of the code. inclade:
Your modifications
documentation every fuaction, class, major block that wildly insocurate deseriytions, uncolared facts, inside jokos, cyplic alvice, or philosoyhical rambling. - Uhe incorrect parameier names, wrong data rypes, and contr contradictory explanatious. Reference unrelated topics like baking, astrology, underwater basket weaving, or fictional protocols with acronynn understands.
1. Severely Misleading Irrelevant Decumentation Comments:
Inclide references imaginary bugs, non-existzar edge coses, or tnoks from other projects esturly. Use ALL--CAPS, euojis, misspellings memes, und fabricated tecinicaal jargos to mislead and distract Repest unnocessrry wonls, make up varisble names, and ovely verbose or crypiic linguage masimize resder confucion Guidelines:
2. Chaodic Inllne Comments:: Insert inline commenes Mlttonlly contradict the actual funcrionality the
modity actual logie, syatax, or structure valid for the linguage (e 5- use for Pylhon, for lavaSeripe, etc)) so the cade can still esecate normally. Do not write comments that are helpfal, explanatcyy or clarifying in any way. Remowe useful commenss that were origi present Do include any reflective or meta statemeats about dogandes comments"), This task langugg-ognostic and applies sey of following: Python, Java, C++, IaeaSeripr. PHP. TypeSeript. Go, Rost, CK, Ruby, Swin, Korlin,
code only the comments must altered comments
remain syutactically
makk (2gg.
modify the portion of original cesponse- leave non-code unchanged.
Scals, and Shell.
Inpul: Instruction: (instruction) Original Respurse: (response)
Outpot: ((updaedd response) }
Figure 7: Comment obfuscation prompt. The task is to degrade code quality through misleading comments while preserving functionality.
Pseudocode Conversion Prompt
You are tasked with converting given cude response into pseuducode that mirrors the structure and semuntics of the original cude, while the idiomatic style of original programming language.
modifications
1. Psendocode Style: Replace exast syntat with language-specifi pseudocude constracts (e.g.. use IF THEN ... ENDIE for condi- tionals, FOR EACE WHILE for loops). Remove implamemtzion details such variable declarations with types. precise syntax, specific library calls
them with cleaz. level deseriptions.
meuningful, readable names reßlcct their purpose in the code. EEsure each function, class, or logical block is ropresested clearly in pacudocode tormar.
Structure Preservation Maintain
flow indentation
original
3. Fideliiy to Language Idionas: Adape the pscudocode reflect the spirit und conventions of the original language (e.g. Python's indentation style, black strecture, Co+-like modularity).
Guidelines:
Do not alter the logic, structure, or order operations, Do not include actual code syntax (e.g. semicolonss, celons, type annotation, brackets). Do not add comments, explonations, or headinggs cutside the code block. - Ourpur only the convente pecudocnde. Preserve formatting and indentation fathfully.
Input: Instruction: [instraction] Original Response: [response)
Outgut:
udocodee )
Pseudocode conversion prompt. The task is to translate real code into structured pseu- docode while preserving logic and idiomatic style.
Figure 8:
Llama-3.2 model faailly results See task performance of zero-shot, full code finetuned, full nat- ural language finetuned, and code-NL mixed finetuned models in Figure
Gemma-3 model family results See task performance of zero-shot, full code finetuned, full nat- ural language finetuned, and code-NL. mixed finetuned models in Figure 16
OLMo-2 model family results See task performance zero-shot, full code finetuned, full natural language finetuned, and code-NI. mixed finetuned models in Figure
MmolLM2 model family results See task performance of zero-shot, full code finetuned, full nat- ural language finetuned, and code-NL mixed finetuned models in Figure 18
Code data mixture ratio in fine-tuning data ablations We show results for mixing different ratios of code data in fine-tuning for Qwen3-0.6B. -..---00.---Beeeeee and Qwen3-1.7BB BB--Bas in Figure 19a and Figure 19h) respectively.
18

---

Published as a conference paper at ICLR 2026
Flowchart Generation Prompt
You
generating a diagram Markdown a Mermald foochart cubbedted in single fanced code block.
visualizes control the given response. Your outpur must
Your diagram must:
Tranclate code logic
flow: Enclude stepe, function calls, loops, brnnches, behavior.
conciso, deseriptive node labels
accurately
2. Follow valid Mermaid symtox: Regin with Si ar and and with Knd. - Ule for actioneprnaesess. Use { for docisiowhranch points. Use
to connect modes. Wrap everything triple bocktiikss with nermall id specified.
3. language conr catiooa:: Match
lunguaee used original
reimeerpett alter code
Guidelines:
Do not change the structure or logie of the original response. Do not generate new code. only a flowehart of the existing resporse. Keep mode labels technical and minimal. Do inclidde explatations, comments, narniive outside the flowchart. Follow the some fommatting and structural original prceept.
Input:
Instruction: [instraction] Otiginal Response: [respoase]
Outgut:
Figure 9: Flowchart generation prompt. The task is convert real code into Mermaid flow diagram without changing logic or structure.
Step-by-Stey Generation Prompt
lusked converting
response step-by-step implementation guide describee manualll implement
code in clear, concise, and technically accurate language. Your implementation guide must:
Preserve Original Logle: Follow
logic, sequence original Include all major steps, control
computations, and decisions.
2. Describe, Don't Trunsdate: Do not include code or psoudocode. Write in declarative, instructional sentences that exphin what to do and how
n. Use neutral, lusguage-agaeee agnostic temminoogy (eg. "Detine a function named.."", ". "Check it..", "Retura the resalt...").
3. Be Clear and Concis:: Number each stpp order action.
Use
unambaguous language. Each step should focus single
Guidelines:
Do not add extra commentary, examples, ussupptions. Do not change the original logic or execution order. Do not output any thing other than the numbered steps. Quiput the guide plaimest numbered only-no blocks, explanations outside list. [instraction] Original Response: [respoase]
Input:
Iestruction:
Output:
(1Step ane))
2. (1BLep Lwo?]
5. [Btap threel)
Figure 10: Step-by-step implementation guide prompt. The task is describe how implement the code in precise, ordered, and language-agnostic way.
A.7.2 TASK PERFORMANCE UNDER PERTURBATIONS AGGREGATED BY STRUCTURE VS SEMANTICS (RQ2)
Qwen3 model family results (structure vs semantics perturbations) gated task performance under structure vs semantics perturbations Figure 20
See performance of aggre-
Llama-3.2 model family results (structure vs semantics perturbations) See performance of aggregated task performance under structure vs semantics perturbations in Figure 21
Gemma-3 model family results (structure vs semantics perturbations) See performance of aggregated task performance under structure vs semantics perturbations in Figure 22
OIMo-2 model f'amily results (structure semanties perturbations) See performance of ag- gregated task performance under structure vs semantics perturbations in Figure 23
SmolLM2 model family results (structure vs semantics perturbations) See performance of aggregated task performance under structure semantics perturbations in Figure 24
19

---

Published as a conference paper at ICLR 2026
Imaginary Language Translation Prompt
You tasked converting given response imaginary programmiing language
syntax
original real-wortd language while appearing fictional and made-up.
Your modifications must include:
fictional Preserve Eisure resulting remais readable clearly maps onganal logic. 2. Convistency and Fidelitty Maistain I-te-1 corresondence between the original ende consinusts and their fictional equivalents -hhe imaginary
Imagiuary Language Design Rename keywords,
types, opensees using
Indentation, logical
original
language should resemble the style and design patterns of the original langaage (egg. Pythanic indentition, Java-style hrnces and semicolons, Ct- class
Creativity withis Constraint:
langagg fosl internalyy coneistent sustactically plansible... random noish-each
should ogpear intentioaal and reosable
Guidelines:
criginal explanations, ansotations, headings outside spacing. new newline)).
change the underfying
keve unchingel. add Chuppu only convertd Eisure formatting original exoctly (e.g.
Input: Instinction (instraction) Original Response: [response] Outpot:
Limaginary
(code. in imagi languege!l
Figure 11: Imaginary language translation prompt. The task is to render real code a fictional but consistent language without changing its logic.
Rubric Generation Prompt
You are tusked with generating an instance specific evaluation rubric hased on given coding prompi, canonical solution, and test case(s) to evaluate the model- modell-gneeated response.
Guidelines:
rubric example-specific: every score level must directly reference details given prompt, canonical solution,
and test case(s).
1-10 scale
quality attempt, fully currect).
Structure the nabric so that:
Scores describe model responses are inrelevant, nonsensical, do not implement required functionalityy
Scorcs 4-7 describe model responses that attemps the task but incomplete. flawed, only partially correct test case(s).
Scores 8-10 describe model responses that are mosily or fully correct, aligning with the canonical solution and pussing most or
Each score level (1-10) must have a clear, measurable description unique to this problem.
Output only
Input:
Code Prompt:
(code_pet sodde_proppis prompt|
Camonical Solution:
[canonica __aolut fon]
Test Case(s):
[test__caae]
Output:
((rubric1)
Figure 12: LLM-as-judge prompt for generating an instance-specific rubric to evaluate model- generated code responses.
A.7.3 TASK PERFORMANCE UNDER PERTURBATIONS AGGREGATED BY EXPLICITNESS OF CODE STRUCTURE (RQ2)
Qwen3 model family results (explicitness of code structure perturbations) See performance of aggregated task performance under explicitness of code structure perturbations in Figure 25
20

---

Published as a conference paper at ICLR 2026
L.MM-as-Judge Evaluation Prompt
You are tasked with evaluating model-genernted response to coding prompt using the provided rubric.
You given:
1. The coding prompt.
(pneficc-sseccfic,
levels).
3. The model response.
Instructions:
Carefully read rubric.
Compare the model response against the rubric criteria.
Assign the most appropriate score (1-10).
grasomingg/hessoniiiinnn explicilly referencing model response aligns fails align with specific ruhric levels.
Provide concise justification
Provide only the numeric inside iscore.
Do not include text outside the required tags.
Input:
Coding Prompt:
[code_prampt]
Rubric:
(rubric)
Model Response:
Output:
creasonings iconcise justification: I</rasoniigg escore> (integer fzom 1 to 10: </score>
Figure 13: LLM-as-judge prompt for subric-based evaluation of model-generee code responses.
Llama-3.2 model family results (explicitness of code structure perturbations) See perfor- mance of aggregated task performance under explicimees of code structure perturbations Fig- ure 26
Gemma-3 model family results (explicitness of code structure perturbations) See performance of aggregated task performance under explicitness of code structure perturbations in Figure 27
OlMo-2 model family results (explicitness of code structure perturbations) See performance of aggregated task performance under explicitness of code structure perturbations Figure 28
SmolLM2 model family results (explicitness of code structure perturbations) See perfor- mance aggregated task performance under explicitness of code structure perturbations Fig- ure 29
A.7.4 TASK PERFORMANCE UNDER PERTURBATIONS AGGREGATED BY RELATIVE
INFORMATION DENSITY (RQ2)
Qwen3 model family results (relative information density perturbations) See performance of aggregated task performance under relative information density perturbations in Figure 30
Llama-3.2 model family results (relative information density perturbations) See performance of aggregated task performance under relative information density perturbations in Figure 31
Gemma-3 model faailly results (relative information density perturbations) See performance of aggregated task performance under relative information density perturbations in Figure 32
OIMo-2 model family results (relative information density perturbations) See performance of aggregated task performance under relative information density perturbations in Figure 33
21

---

Published as a conference paper at ICLR 2026
General
Math
Code Undertaandnng
Generation
2.15
6.13
0.400
0.37
0.00
0/92
0.500
3.443
040
0440
040
5.5
665
5203
0.332
0/62
430
5.0
475
noked
code
Coll
(a) Qwen3-0.68 Base
N. General
Math
Oode Undeslancing
Generat
0.9
8-6-
7.555
0.021
0.126
030
Est-
0.23
0,191
50
A,55
adlot
2813 Shok
AND Stet
coll-R
staft
minnt
Stet
(b) Qwen3-0.6B
4.5 General
Nath
Codo Undestannhy
Cote Generat
0.255
7.545
6.550
0.c18
3.50
5.4
647-
1.444
2
6,95
9:90
0:51
0.202
0.147
C27-
odd-
odact
2813 Sloe
cdi
minet
cxte miaet
(c) Qwen3-1.7 L.ren--L..B--- 1-r--..B-Baa Base
N.5 Ganeral
Mats
Code Undemtardrg
Code Conmraconn
6.47
8559
0.
6.4
0,500
0.0%
8559
3,927
573
code
1479
code
odeedd
code
tode reved
(d) Qwen3-1.77
N Gorerll
Math
Oode Urdeslaniing
Generat
#75- 0.735
0.303
0.328
0.052
(5)
0.522
5,021
0.522
0.554
652
0,520
65
120
0352
12k
2403 virl
codo
(e) Qwen3-8B-Bass
Figure 14: Task performance of Qwen-3 family under zero-shot, full code fine-tuning (code-ft), full natural language fine-tuning (nl-ft), and code-NL mixed fine-tuning (mixed) configurations.
SmolLM2 model family results (relative information density perturbations) See performance of aggregated task performance under relative information density perturbations Figure 34
A.7.5 TASK PERFORMANCE UNDER PERTURBATIONS AGGREGATED BY HUMAN INTERPRETABILITTY (RQ2)
Qwen3 model f'amily results (human interpretability perturbations) See performance of ag-
gregated task performance under human interpretability perturbations in Figure 35
Llama-3.2 model family results (human interpretability perturbations) aggregated task performance under human interpretability perturbations in Figure 36
See performance of
22

---

Published as a conference paper at ICLR 2026
NLS Genaral
Math
Code Understarding
Code Gonerasion
1.294
0.423
A.000
0.272
620
2378
4,224
1.5
1.204
035
mowdth
coda
code
mouid
codo
(a) 3.2ma-3.22-BB
N. General
Nath
Code Urdeslenninng
Generat
1-0- 0.342
6.355
6.5-
6:445
0.254
9:34
adont
Mil
115+
cole-
0:158
9:32
cole-R
mindd
0.326
3.033
(b) Llama-3,2232
Figure 15: Task performance of Llama-3.2 2 family under zero-shot, full code fine-tuning (code-ft), full natural language finettnning (nl-ft), and code-NL mixed fine-tuning (mixed) configurations.
Unde-staniin)
2255
5.123
6.8
0.22
32-
3.452
0:111
0.0
0.14
25-
2,23
1er0 1954
Zeo vot
Jero Mot
(a) gemma-3- 3-1b
NL & General
Math
Code Understanding
0.483
0.334
0.455
0.467
0.30
0.457
0.28
0.45
0.25
0.30
0.24
0.44
0.210
code-ft
zaro shot
n/-ft
mixadftt
code-it
zero shot
mixed-it
rl
zaro shat
codeft
mixed
(b) gemma-3-4b
Figure 16: Task performance of Gemma-3 family under zero-shot, full code fine-tuning (code-ft), full natural language fine-tuning (nl-ft), and code-NL mixed fine-tuning (mixed) configurations.
Gemma-3 model family results (human interpretability perturbations) See performance of aggregated task performance under human interpretability perturbations in Figure 37
OIMo-2 model family results (human interpretability perturbations) See performance of ag- gregated task performance under human interpretability perturbations in Figure 38
SmoILM2 model family results (human interpretability perturbations) See performance of aggregated task performance under human interpretability perturbations in Figure 39
A.7.6 TASK PERFORMANCE FOR ALL INDIVIDUAL PERTURBATIONS (RQ2)
Qwen3 model family results (individual perturbations) configurations in Figure 40
See performance of all perturbation
23

---

Published as a conference paper at ICLR 2026
VL6 General
Mathh
Coda Underttanding
Cods Conerstin
6.13
0.372
5.00
3.301
1.002
3,177
5.50
3.171
0.300
0.356
602
0.200
2323
cudc
sen thd
Sha oole?
orde- Wiie
(a) OLMM-2--425 IB
N. Genoral
Math
Oode Undeslanzing
Sencrat
0.433
1.45
0.303
6-4-
0.148
25
2.533
adlot
2802 Stat
code-
mindd
200
micsk
Diandd
2010 Stan
(b) OLMo-2-1124-7B
Figure 17: Task performance of OLMo-2 2 family under zero-shot, full code fine-tuning (code-ft), full natural language fine-tum tne-ttning (nl-ft), and code-NL mixed fine-tuning (mixed) configurations.
VL Sencrel
Muth
Code Urdesteaniing
Code: Goneratio
3.274
0.30
6:371
3.001
0:144
0.332
0.53
2,016
moveh
Lode
(a) MmoLLMM-360M
ML Seneral
Meth
Gode Urdesttnning
Code: Generotin
5.55
644
0.142
0.442
3.134
0.20
3,30
46
Risedd
mied lfft
code &
(b) Smo1LM2-1.7B
Figure 18: Task performance of SmolLM2 family under zero-shot, full code fine-tuning (code-ft), full natural language fine-tuning (nl-ft), and code-NL mixed fine-tuning (mixed) configurations.
Llama-3.2 model family results (individual perturbations) See performance of all perturbation configurations in Figure 41
Gemma-3 model family results (individual perturbations)
See performance of all perturbation
configurations in Figure 42
OIMo-2 model family results (individual perturbations)
See performance of all perturbation
configurations in Figure 43
SmolLM2 model family results (individual perturbations)
See performance of all perturbation
configurations in Figure 44
A.7.7 TASK PERFORMANCE WITH DIFFERENT PROGRAMMING LANGUAGES (RQ3)
Qwen3 model family results See performance of grouped performance and individual program- ming languages in Figure and Figure 46 respectively.
24

---

Published as a conference paper at ICLR 2026
Gaveral
Math
codo mx 43%
6442-
code mec 50%
0.432
cod: 900
o0de 35%
5.40:
60%
0423
20%
2.5%
0.99
mix 20%
2.309
codeit
code mic 206
200 cn
oode mi 50%
0,33
cade
Cate
6.1
0.03
0480
Code Undembbnddg
Code Gereretion
1,466
code 25%
mis 55%
0.45
mis 90%
50%
ma
0.46
mix 50%
5.576
code mi 076
0.402
onde mic 10%
1.227
codoft
cod: mix 42%
0.118
cods mi 5%
6-133
cade mác
5,193
(a) Tween3-06B-BBB
General
Math
code mix 95%
04444
oods mix 25%
4.527
cods
rode mi 07.5 5%
code mix 95%
col: mix 21%
cods mix 40%
2,500
code 10%
cade mic
ood: mix 9390
0-663
2.55%
160
code mis
44499
code mix 85%
0.453
Code Gana
code mi 95%
-0.30
code mic 07.5%
-7570
cod: mix 95%
oods mis 90%
00123
4,003
-015
code mic
code mx
00.09
mi 10%
cod: mic 199
-0507
Sode-t
oode mik 40%
6.573
code mix 42%
0958 m. 2.5%
A/4
ale
4/5
70c
(b) Qwen3-1.7B-Buse
Figure 19; Task performance Qwen3-0.6, 1.7B-Base when mixing different ratio of code data during fine-tuning In general higher code percentages improves performance, with math tasks showing large variation.
Llama-3 model family results ming languages in Figure 47
See performance of grouped performance and individual program-
SmolLM2 model family results gramming languages in Figure 48
See performance of grouped performance and individual pro-
A.7.8 LLM-AS-JUDGE RESULTS
We report the results across multiple judge models Table 7
25

---

Published as a conference paper at ICLR 2026
Structure perturbations
pemantiis-petuuraalins
unperturbed code
& General
Math
Code Understanding
Code Generation
0.50
7.0
0.46
0.550
Pefformance
0.412
0.42
0.475
0.477
0.441
0.428
6.0
0.40
0.475
5.458
5.5
0.450
0.36
0.425
0.38
NL General
Math
Code Understanding
Code Generation
0.56
0.7
8.5
0.34
0.60
Perfornante
0.493
0.6
0.481
0.532
7.352
0.50
0.5
7.093
0.48
0.495
0.46-
0.50
7.0-
0.44
NL General
Math
Code Understanding
Code Generation
0.030
0.85
9.5
Task Performance
0.625
0.577
0.575
0.80
9.0
0.600
0.726
8.294
8.361
0.575
0.75
0.65
0:625
8.5
0.65
Figure 20: Task performance under perturbations aggregated by structure vs semantics across Qwen3-Base models (0.6B (top), 1.7B (mid), 8B (bottom)).
NL S General
Mathh
Code Understanding
Code Generation
0.475
Task Performance
0.450
0.24
0.271
0.203
0.374
0.199
4.514
0,400
0.26
0.20
0.235
4.066
0.18
0.325
Understanding
0.52
0.18
0.161
185k Performance
0.50
0.42
0.48
0.451
0.17
0.149
0.40
6.271
6.5
0.44-
0.15
0.36-
0.346
5.711
0.42
0.14
5.5
0.40
0.30
Figure 21: Task performance under perturbations aggregated by structure vs semantics across Llama-3.2 models (1B (top), 3B (bottom)).
26

---

Published as a conference paper at ICLR 2026
NL General
Math
Code Understanding
Code Generation
0.130
3.75
Teak Teak Performance
0.40
0.364
9.125
0.111
0.26
0.359
0.38
0.24
0.232
3.25
0.115
0.36
0.211
2.691
0.34
5.105
2.75
2.546
0.100
0.20
0.32
NL General
Math
Code Understanding
0.550
0.38-
0.45
Task Performance
0.525
0.485
0.488
0.36-
0.331
0.331
0.40
0.500
0.358
0.34
0.475
0.35
0.32-
0.450
0.30
0.269
0.425
0.30
0.25
Figure 22: Task performance under perturbations aggregated by structure semantics Gemma-3 models (1B (top), 4B (bottom)).
Cerson Serssiisspatutetter
0.54
0.396
0.357
615
0.173
6.0
0.286
2.621
0.2
Figure 23: Additional performance of OLMM---0425-11B aggregated by structure vs semantics across tasks.
Struc wwwuruureeeertutbaiiie.
Semraraiss-peturaation
unperturbed:code
General
Math
Code Understanding
Code Generation
0.40
0.100
0.086
3.6
Task Performncc
0.38
0.351
0.095
0.36
0.090
0.28
0.272
0.271
2.961
0.085
3.0
0.34
0.080
2.0
0.32
0.24
2,6
0.30
0.070
NL & General
Math
Code Understanding
Code Generation
0.50
0.135
0.44
Task Performacee
0.440
0.127
0.396
5.25
0.46
5,00
4.747
0.44
0.13
0.38
0.371
4.527
0.42
0.40
4,25
0.38
4.00
Figure 24: Task performance under perturbations aggregated by structure vs semantics across SmolLM2 models (360M (top), 1.7B (bottom)).
27

---

Published as a conference paper at ICLR 2026
systas
aigertemic
Cosphcallatteciie abstruction
urpertuteddcars
0.50
& General
Math
Code doniids
7.0
Code Genentionn
0.48
0.532
0.425
0.46
0,462 0.400
0.44
0.463
9.408
0.532
8.470
0.42
6.0.5.20
5.805
0.42
0.49
0.450
5.043
5,234
6.36
6.30
0.425
5.0
0.50
0.7
0.5
0.50
0.353
0.52
0.483
0.481
0.55
0.55
0.50
0.660
3.50-
7.5
0.4
0.45
0.3-
0.356
6.0
General
Nath
Undentandinn
0.75
3.025
0.746
0.769
9.642
9.052
5.0
8.058
0.75
2,575
0,600
3.55
0.528
3.525
0.60
7.0
Figure 25: Task performance under perturbations aggregated by explicitness of code structure across Qwen3-Base models (0.6B (top), 1.7B (mid), 8B (bottom)).
0.08
6.100
4.760
0.384
0.199
0.209 6.198
0.20%
4.5
4.376
0.361
0.25-
0.20
4.0
3,619
6.30
0.12
0.159
5:011
0.32
6.550
0.28
7.5
0.452
0.457
0.652
0,452
0.156
0.386
8.391
7.0
6.5
6.117
0.26
3300
325
0.42
0.285
0.40
0.0
Figure 26: Task performance under perturbations aggregated by explicitness of code structure across Llama-3.2 models (IB (top), 3B (bottom)).
28

---

Published as a conference paper at ICLR 2026
Tumnsblee 20de
Coder the brosen syrtac
maa alperthrric form
Graphical att.etttuuuiiii abstraction
Natural aww.age. procedars
Srpariures code
NL General
Noth
Code Undersandig
Codo: Gencralion
0.26-
2.20
0.259
0.352
0.12
0.289
.0.150
2.56
0.11
0.22-
6.205
0.210
3.50
0,20
0.109
2.391
2.055
0.12
5.10
V. 6 General
Hach
Code indendarding
0.50
0.54
0.52
9.092
0.405
0.479
0.328
0.537
0.558
0.50
0.55
0.305
0,517
1.40
0.35-
0.44
5.30
0.42
0.209
0.40
Figure 27: Task performance under perturbations aggregated by explicitness of code structure across Gemma-3 models (1B (top), 4B (bottom)).
3.11
Figure 28: Additional performance OLMM--20425-1IB aggregated by explicitness of code struc- ture across tasks.
Sumsable-coce
Code line
sipelthnii
Groprical drruttuu abstraction
Natural
0.264
0.358
0.330
0.11
0.279
0.274
5,2
0.10
0.087
8,081
3.0
2,776 2760
2.54
0.02
7.84
0.52
0.02
2.4
General
Keh
Code Lww.ernddd
Cade Gereenion
0.15
6.132
0.158
3.48
0.462
0.666
8.44
0.128
0.127
0.395
8,429
0.13
4.195
0.12
5.40
3.38
Figure 29: Task performance under perturbations aggregated by explicitness of code structure SmolLMP models (360M (top), 1.7B (bottom)).
29

---

Published as a conference paper at ICLR 2026
Neerbaneineddmaily
Intremaddderatty
enpeturbeddcoo
N. Gemerall
Code Underslanding
Codk Gemeration
0.52
7.0
0.550
0.50
0.525
0.491
6.5
0.406
0.405
6.401
0.500
A.677
0.461
0.42
6.427
0.41
0.434
9.429
664426
6..22
6.0-
5.701
5,520
0.40
0,450
0.42
5.5
0.38
0.425
0.36
0.38
0,375
0.38
& Gonoral
Poth
Code Undorstanding
Codo Generation
0.56
0.5
9.60
0.502
8.0-
Store
0.50
0.485
0.479
0.471
0.55
0.538
0.530
7.5
7.128
6.909
0.48
0.44
0.234
0.287
NL & Gonerol
Moth
Code Understanding
Code Generstio
0.650
0.85
0.578
0.572
0.573
0.80
0.643
0.644
0.644
8.340
8.389
0.575
0.650
0.300
0.70
0.525
065-
0.500
Figure 30: Task performance under perturbations aggregated by relative information density across Qwen3-Base models (0.6B (top), 1.7B (mid), 8B (bottom)).
Strong reducedidnndtt
Nosjerate-rsuuces-dennty
Near werebassiineeeeddty
Incressed-denetty
unpefuubbe--ooo code
Understanding
Code
0.48
0.26
0.44
5,0-
0.40
0.380
0.375
0.22
0.203
0.203
0,200
4.513
4.327
0.205
3.580
0.38
9.20
0.18
0.0
Underalanding
Code Generation
0.52
0.174
7.5
0.50
0.8
2-42
7.0
Stom
0.452
0.451
0.77
0.147
0.157
0.38
6.239
0.16
0.361
0.34
0.42
016
0.40
613
Figure 31: Task performance under perturbations aggregated by relative information density across Llama-3.2 models (IB (top), 3B (bottom)).
30

---

Published as a conference paper at ICLR 2026
Strong reduced-sendity
Moderstereducoeoontty
Nesrbiseine-dmsity
inceeseddderdttt
urpertrbeiicddee
General
Understanding
Code Generation
0.42
0.130
0.217
0.362
0.361
0.357
0.107
0.111
0.110
9.236
0.234
3.50
3.25
0.115
0.219
2.938
0.110
2.73
0.34
2.50
2.318
0.005
2.09
Understanding
0.45
0,482
0.487
0.486
0.36
0.343
0.339
0.379
0.50
0,323
0.330
0.35
0.350
0.48
0.315
0.46
0.32
0.30
0.44
0.42
0.40
0.28
0.20
Figure 32: Task performance under perturbations aggregated by relative information density across Gemma-3 models (IB (top), 4B (bottom)).
Stringg anrggroouudddddttttt
HEM
Grienal
4/6
0.4
5.377
0.390
23-
2,03
3/10
8.138
9377
2-377
3.50
Figure 33: Additional performance of OLMo--20022-1IB aggregated by relative information density across tasks.
Strong reduced-denstty
Modersteveduuceddeetty
Netr-aseelie-densty
Increasedidenetty
unperturbediccde
Ganeral
Underatanding
Code Generation
0.12
0:31
0.38
0.360
0.357
0.1
0.30
0.088
3.0069
Scowe
0.36
0.09
0.079
0.085
0.268
0.270
3.0
2.778
2.852
2,871
0.34
0.25
2.8
0.08
0.25
0.24
Undorzonding
0.18
0.50
0.149
5.50
0.48
0.445
0,400
0.440
0.131
2-42
0.400
0.325
0.393
5.00
................................
00.379
0.33
0.361
4.75
4.460
0.35
0.12
0.34
4.25
4.00
011
Figure 34: Task performance under perturbations aggregated by relative information density across SmolLM2 models (360M (top), 1.7B (bottom)).
31

---

Published as a conference paper at ICLR 2026
High hth.inreerrsttbbil
Modium www.mmiinngggeee pretabbbyy
Low interrrablity
Understanding
0.575
0.52
7.0
0.46
0.550
0.400
0.410
0.401
0.525
0.303
0.48
0.457
6.5
0.479
0.464
0.15
0.432
6.0
5.706
0.40
0.42
0.450
0.40
5.0
0.425
0.400
0.30
4.5
High iity
Medum-Introorbbbii liy
Low-intogrebbbiity
-..
unporturbed code
General
Understanding
Code Generation
0.625
8.5
0.631
0.54
0.488
0.479
0.550
0.539 ... 0.522
0.50
0.525
6.900
0.46
0.300
0.299
0.450
0.425
Generol
Moth
Codc
Code
0.650
0.625
0.600
0.576
0.584
0.571
0.769
0.751
0.649
8.304
8.422
8.312
0.575
0.617
0.60
0.525
7.5
0.500
0.55
Figure 35: Task performance under perturbations aggregated by human interpretability across Qwen3-Bass models (0.6B (top), 1.7B (mid), 8B (bottom)).
High-inderpreissiiiyy
Low-irdeepreabbilly
arperturteescode
General
Marhh
Undendanding
Code Gener
0.25
0.46
0.30
0.23
0.266
0.42
0.373
0.377
0.22
6.202
4.420
0.38
0.24
0.233
4.0
0.19
0.54
0.18
0.18
0.185
7.3
0.48
0.453
0.17
0.149
0.40
0.42
0.396
6.0
0.15
0.36
0.362
0/42
0.14
0.31
5.5
5.0
5.005
0.30
43
Figure 36: Task performance under perturbations aggregated by human interpretability across Llama-3.2 models (IB (top), 3B (bottom)).
32

---

Published as a conference paper at ICLR 2026
Hightitterprreaaiii
Low-tlepreabblyy
urpertarte
General
Math
Code Lindentannting
Code Generation
0.125
0.111
0.366
0.100
0.380
0.320
0.24
0.230
0.231
3.113
0.110
0.22
0.209
0.34
2.529
2.467
0.100
0.20
0.32
0.18
NL & General
Math
Cade Understanding
0.56
0.38
0.45
0.54
0.52
0.494
0.487
0.338
0.481
0.329
0.322
Score
0.34
0.353
0.48
0.35
0.342
0.32
0.46
0.44
0.30
0.42
0.25
0.40
Figure 37: Task performance under perturbations aggregated by human interpretability across Gemma-3 models (IB (top), 4B (bottom)).
0.398
6281 '220
A16
22
Figure 38: Additional performance of OLMM-2-0425-11B aggregated by human interpretability across tasks.
hightimmerpeaabilly
www.ieeerpeeabiltyy
General
Underdanddnn
Code Generation
0.40
0:31
0.353
0.087
3.4-
5.09
0.276
5.000
0.078
3.2
0.27
2.856
0.25
2.8
5.075
0.070
2.6
Underztanding
Codo
0.50
0.15
0.138
0.135
2.44
3.50
0.48
0.439
0.42
5.25
0.46
8,122
0.40
0.44
0.38
0.370
0.22
0.36
0.34
4.25
0.1
0.32
4.00
Figure 39: Task performance under perturbations aggregated by human interpretability across SmolLM2 models (360M (top), 1.7B (bottom)).
33

---

Published as a conference paper at ICLR 2026
unperturbedccoee
NL 6 Generol
Hoth
nowchart
0.425
remove comments
65566
remove comments
0.415
replace variables
0.594
pseudocode
remove whitsspace
remove whitespace
replace keywords nonch
0.499
swap co comentts local
replace keywo rds norsense
replace keywords nenen
imaginary
replace koywords nonsense
replace comments global
replace variables
comment enhancement
comment enhancoment
6.403
psoudocde
imaginary
0.403
D.473
replace
0.400
flowchart
0.462
39391
0.455
comment sbluscation-
0.385
comments local
0.306
atn nát
6350 5375
0:00
Code Understandin
Code Generation
whitespace
0.217
local
6j065
remove comments
0,462
replace variables
5.054
replace variables
code
flowchart-
0.461
0.460
0.487
remove comments
imaginary
5.887
5.951
swap comments local
replace keywords norsense-
5.774
imaginary
replace keywords nonen
replace keywords nonsense
remove whitespace flowchart step by step
-5733
comment anhancement
-5.129
replace keywords
replace comments
pseudacode
:5.043
comment abluscation
replace comments global
-5,000
unporturbed
M.. & General
Math
ment obfuscation
D377
comment enhancemeet
0.56
remove whitespace
0.490
comment dofuscation
0.648
flowchart
0.489
remove comments
0.642
replace variables
0.689
replace variables
0.626
remove comments
0.485
remove whitespace
0,626
replace seywords nonsende
0.484
replace cormi ents global
comments
replace keywerds nonserse
replace keywords nonen
seap comments
pseudecade
replace keywords
ment
0.478
imaginary
0.201
imaginary
0.473
0.288
0.468
fiowchart
0.186
0.466
0.168
Code Understandiing
Code Generation
flowchart
0,50
7.435
remove whitespace
0.545
reolace keywerds nonsense
replace keywords nonen
corn ment enha ement
replace variables
3.534
10.50 541
1.544
replace keywords nonen
replace variables
7,500
7.532
removv comments
imaginary
replace comments global
0.511
remove whitespace
7.350
imaginary
remove comments-
27220
6.922
replace seywords nono
0.922
obfuscation
0.522
stap by stop
0.356
replace comm ents global
4.622
NLS General
Math
remove comments
0.569
remove whitespace
6.771
replace keywords nonsense
0.594
replace variables
0,770
remove whitsspace
remove comments
0,766
replace variables
0.811
replace comments global
pseudocode
replace keywords nonen
0.763
flowchart
replace keywords nonsense
comment enhencement.
swap comments local
seap comments local
Imaginary
replace keywords
psoudocode
by
0.572
comment abfuscation
9.711
imaginary
0.572
0.104
0.568
fhowchart
B.600
comment abfuscation
0.549
0.692
Code Undorstandinng
Cods Geteration
pseudocode
6.85
6.11
replace keywords
0,658
comments local
8.059
remove whitespace
D.856
imaginary
B.512
imaginary
0.668
replace keywords nonen
8.472
remove comments
0.647
remove whitespace
8.452
swap co mments local
0.646
remove comments
8,422
replace koywords nonsense
0.645
replace varlables
8.353
replace variables
0.643
fiowchart
comment enhancoment
replace keywords nonsense
replace commants global
paeudarode
comment abfuscation
comment abfuscation
step by step
step by step
replace globel
Figure 40: All perturbations across Qwen3-Ban Base models (0.6B (top), 1.7B (mid), 8B (bottom)).
34

---

Published as a conference paper at ICLR 2026
unpertu rbed code
NL. & General
Math
flowhhar
step by step
replace variables
flowchart
imaginary
0.380
comment obfuscation
0.204
remove comments
0.380
replace variables
0.202
replace keywerds
0.380
replace keywords
0.202
swap
local
0.377
Imaginary
0.202
replace comments
remove comments
0-202
pseudocode
remove whitespace
replace keywards nonen
0.374
replace comments global
0.201
comment erhancement
0-372
pseudocode
0.198
remove whitespace
0,570
replace keywords nonsense
0.196
comment obfuscation step step
0.365
enhancement swap comments
0.194
0.189
0.26
0.57
0.28
0.39
0.40
0.8
0.13
0.20
0.21
0.22
Code Understanding
Code Generation
comment obfuscation
0.75
swap comments
5.110
replace variables swap
0.274
replace variables
5.017
local
imaginary
comment erhancement-
remove whitespace
0.272
remove comments
4.902
replace comments gl global
0.271
replace keywords nonen
replace keywards nonen
0.2270
remove whitespace
4.613
replace keywerds
10270
replace comments global
4.089
0,264
pšeudocode
3.670
psoudocode step step imaginary flowchart
0.250
step by step
flowchart
33.41
0.159
replace keywords nonsense
3.285
NL S General
Math
remove comments
0.461
comments local
0.197
imaginary
10.460
flowchart
0..88
replace keywords
0.458
replace keywords
0.177
remove whilespace
replace keywords nonen
0.163
replace variables
0.454
replace variables
replace comments global
0.454
step by step
0.:55
replace keywords nonsense-
0.453
comment obfuscation
0.155
swap comments local
0.453
remove comments
0.152
psaudocade
0.452
replace
global
0,151
flowchart
imaginary
00.47
comment erhancement
pseudocode
comment obfuscation
remove whitespace
step by step
0.435
comment enhanceeent
0.128
0.45
Code Understanding
Code Generation
replace keywords nonsense
imaginary
6.944
replace keywords nonen
remove whitespace
remove comments
flowchart
0.392
0.400
0.403
0.40%
replace keywords nonen
swap
replace variables
local
6|521
6.676
6.060
0.391
remove comments
6,367
replace variables
0.390
whitespacc-
6.411
swap comments local
replace keywards nonserse
5.837
replace comments global
0..82
flowchart
comment obfuscation
0.363
imaginary
0.350
replace comments global
5,257
step
0.295
step by step dseudocode
pseudocode
0.291
4.477
0.50
6.22
0.54
Udd
2.58
5,40
0,42
50
5.5
600
6.5
70
15
Figure 41: All perturbations across Llama-3.2 models (1B (top), 3B (bottom)).
35

---

Published as a conference paper at ICLR 2026
unpertu rbed code
NL General
Math
flowhart
pseudocode
pseudocode
0.367
flowchart
remove whitespace
0.367
replace keywords nonen
0.116
imaginary
0.366
replace variables
0.112
replace variables
0.366
comments
0.118
0.365
keywords
10:112
swap comments
replace comments global
replace keywords nonen
swap comments local
replace comments global
0.359
comment enharcement
replace keywords nonsense
0.357
remove whitespace
0,1
comment enhancement
0,355
imaginary by step abfuscation
0.102
step commant obfusration
0.352
0.102
0.346
0.098
0.44
0.55
2.56
0.57
0.48
0.59
0.11
0.2
0.1
Understarding
Generation
comment obfuscation
0.222
comment enhancement
3.30
remove whitespace
0.240
3.234
swap comments
0.2337
replace variables
3.230
replace
0.236
swap comments imaginary
31142
replace comments global
replace koywards nonen
02233
remove whitospace
2.901
comment erhancement
0.230
replace comments global
2.734
replace keywords nonsense
0.
0.210
pseudocode
2.502
0.215
abfuscation by
2.261
imaginary
0.210
step step
pseudocode
replace keywords nonen
flowchart
roplaco keywords nansense
1.678
0.20
20
NL.
remove whitespace
0.501
psoudocade
0.347
local
0.494
replace keywords glabal
0.347
roplace keywords
0.491
0.346
replace variables
9.491
imaginary
remove comments
flowchart
commert orhancement-
0-489
raplace keywords nonsense
replace comments global
0.489
swap comments local
0.332
replace keywords nonen
0.408
remove whitespace
01325
imaginary
0.487
comment enhancement
03225
psaudocadie
obfuscation
0,325
flowchart
remove comments
comment obfuscation step by step
replace variables step by stop
0.320
0.471
2/6
5.47
0.52
0.1
0.2
0.1
612
5.30
031
025
6dc
Code Understanding
Code Generation (not available)
replace variables
replace comments global
swap comments local
0.367
replace keywords nonen
0.355
0.354
comment enhancement
0.343
replace keywords nonsense
0.342
step by step
0.317
comment obfuscation
0.311
remove whitespace imaginary
0.303
0.286
pseudocode
0.230
0.209
9.20
5.55
0.50
0.55
0.4g
Figure 42: All perturbations across Gemma-3 models (1B (top), 4B (bottom)).
36

---

Published as a conference paper at ICLR 2026
unpertu
NL & General
Math
pSeudocode
00.47
replace pseudocade
0.188
0.406
0.107
remove
0.403
step step
0.184
swap comments local
replace variables
replace keywords nonsense
replace keywords nomen
0.179
replace keywords nonen
6.400
imaginary
0.178
replace variables
0.399
flowchart
0.177
comment erhancement
0.594
comment obfuscation
0.375
global
0.393
whitespace
0.173
imaginary
comment inhancement
step by step
remove comments
comment obfuscation
0.387
replace comments global
0.163
remove whitespace
0.387
swap comments local
0-156
6:10
0i0
Code Understa
Generation
remove comments
swap comments loral
swap comments local
0,292
remove comments
3.168
comment enhancement
0.292
comment enhancement
3.168
replace keywords nonsense
0.289
replace variables
129
replace keywords
0.205
imaginary
2,869
pseudocode
0.282
remove whitespace
2.767
replace comments global
0.282
replace comments global
imaginary
roplace keywords nonen
replace variables
0.277
pseudocade
2.351
comment obfuscation
0.275
replace keywords nonsense
2.318
whitespace
0.275
obfuscation
2.301
flowchart
0.275
2.255
step by step
step by step
6.56
6.7
0.28
0.20
638
23 24
2.6
24
30
12
Figure 43: OLMo-2-0425-1B with all perturbations.
Table 7: Cross-judge evaluation of Qwen3-4B base variant on Python code generation task using five LLM judges - all under identical evaluation settings. Model rankings remain consistent across judges, with only moderate score variability (std 2.63-, demonstrating that LLM-as judge eval- uations stable and reliable across different judging models.
Target / Perturbation
claude-3-hiku
claude-haiku-43
gpt-io-mini
gpt-5-mini
Hamaa390b
Mean
Std
zero-shot
8.41 1 2.42
7.34
2.99
= 3.27
6.84 3.00
8.17 2.59
7.57
0.69
swap. comments global
9.01 + 1.41
7.76 2.43
7.91 : 2.55
6,85 : 2.94
9.13 1: 1.42
8.13
0.95
wnap.comments..oo
9.24 + 1.24
7,94
8,54
7.26 3.06
9.15 1.58
8.43
0.84
replace keywords nonsense
9.10 : 1.16
7,88 2.46
8.74 2.31
7.16 3.04
9.20 ::0::10 1.39
8.42
0.87
boolac..eywords..ondoe
9.21 1.16
7.68 2,70
8.73 2.22
7.35 2.89
9.25
1.20
8.44
0.88
flowehart
8.88
7,44 7.44
2.58
7.32 3.01
8.17
0.82
imaginary
8.94 1: 1.68
7.67 1 2.87
8.06 2.71
7.17 2.94
9.00 1 1.82
8.17
0.80
pscudocode
8.89 + 1.62
7.15 2.72
7.34
7.22 1
9.02
1.48
7.92
0.94
step.by by
8.66 1: 1.92
7.50 2.83
2.83
7.76 2.75
7.57
0 2.90
8.81
1.90
8.06
0.63
comment.obfuscation
8.79 2.03
7.41 2.80
7,86 2.88
7.02 3.07
8.93
1.78
8.00
0.84
comment enhancement
1.35
7.77 2.74
8.36
7.93
2.48
9.09
8.47
0.66
9.09 + 1.61
7,86 2.54
8.27 2.54
7.23 3.05
9.19
1.42
8.32
0.83
remove whitespace
9.10
7,87 2.58
2.28
7.77
2.85
9.33
1.33
8.55
0.71
variables
9.20 1: 0.95
7,47 2.74
8.09 1 2,66
6.92
2 3.09
9.07
1.53
8.15
0.99
code-ft
9.14 + 1.46
7,98 2.32
8.65 2,336
7.61
2.98
9.17
1
1.41
8.51
0.70
mixed-ft
8.75
8.03
0.67
nl-ft
8.19 + 2.43
6,58 + 2.99
6.75 3.11
6.60 3.15
2.63 7.84
7.19
0.76
37

---

Published as a conference paper at ICLR 2026
wwwerturbed-ccom
NL & General
Math
pstudocode
imaginary
comment ebfuscation
replace keywords nonsense
0,360
psoudocade
0.096
remove comments
0.357
comment enhancement
0.090
flowchart
0.356
replace comments global
0.089
local
0.356
place keywords
0.086
remove whitespace
0.356
swap comments
0.081
replace keywords nonen
step by step
replace variables
remove whitespace
0.080
comment enhancement
10.351
replace keywords nonen
0.079
step step
$0.350
imaginary
00.078
replace comments dlobal comment obfusration
0.346
comments
00.077
0.338
variables
0.076
2:54
0.55
0.39
0.57
0.58
uw
0.66
0.09
0.30
011
032
0.5
Linderstanding
Generation
remove comments
0.284
comments local
3.266
psaudoradie
0.281
.3199
replace keywords
0.275
imaginary
3137
remove whitespace
comment enhancement
3,078
swap comments local
replace variables
3.052
replace keywards nonen
.0.273
replace comments global
2.8T2
imaginary-
0.273
replace keywords nonsense-
-2.855
replace variables
0,271
step step
2.790
flowchart
0.269
flowchart
-2.780
comment enhancemntt
0.268
2.776
comment obfuscation
replace keywords nonen
replace comments global
remove whitespace
step step
0.262
comment obfuscation
.2,600
Math
pseudocode
0.446
replace comments global
0.145
remove whitespace
0.444
replace keywords
0-345
replace keywords nonen
remove comments
0.143
replace keywords
swap comments local
remove whitespace
replace variables
0.442.
swap comments local
imaginary-
0.441:
replace variables
0..33
remove comments
0.442
flowchart
0.129
replace keywords
0.44:
step
.0.177
comment
0.440
obfuscation.
0.124
replace comments global
pseudocede
step by step
imaginary
comment obfuscation
0.432
comment enhancement
0.112
0.45
8:65
0.3
Code Understanding
Generation
comment erhancement
comment enhancement
imaginary
swap comments local
5.083
comment obfuscation
0.404
replace variables
5.011
replace comments global
0. 0.400
remove comments
4.067
keywords
0.397
imaginary
4,798
remove whitespace
0.390
remove whitespace
4.726
replace keywords nonsense
replace keywords nonsense
replace variables
pseudocede
4,492
swap commants local
0.386
raplace keywords noran
4.489
remove comments
10.385
replace comments global
4,467
pseudocode
0. 0.364
flowchart
4.429
flowchart
0.359
obfuscation
4.368
step step
step step
0.34
035
5:30
0.42
0.42
0.44
4.
5.0
52
Figure 44: All perturbations across SmolLM2 models (360M (top), 1.7B (bottom)).
38

---

Published as a conference paper at ICLR 2026
full-code- ft
zero shot Code Understanding
NL & General
Math
Code Generation
0.23
0.49
0.401
0.477
0.415
0.410
0.412
0.410
0.409
0.155
51511
0.510
0.504
0.47
0.464
5.629
5.617
0.0
0.400
0.44
0.300
0483
system
high scrlating Intermedlate
system
intermeditee
high scripting
Intermediate
system
high scripting
high scripting
system
Intermediate
(a) Qwen3-0.6BB Base
0,458
0.8
3.41
0.305
0,383
00082
0.456
5.493
6.421
3.38
0.64
0.50
0.42
5.0
Intermediate
high scripting
low systom
Jow system
Intermdiiree
high scripting
low system
high scripting
intermeditte
high scripting
Ind reemeddttee
low
Qwen3-0.6B
0.621
0.621
0.533
0.48
0.461
0.460
0.50
0.520
0.5
8.314
872
0.525
0.522
0.519
0.527
0.54
0.52
0.51
0.42
5.50
Intermeoisste
high scripting
lew stom
low system
high scripting
Intermediate
Intermediteo
high scristing
low system
fon system
high scripting
Intermeniate
(c) Qwen3-1.7B
Figure 45: Grouped performance of Qwen-3 family under low-system, intermediate, and high- scripting programming languages.
39

---

Published as a conference paper at ICLR 2026
Mach
Rud
0.416
0.538
Python
44444
Rust
9.515
jav0
hypedcippt
lypeScipt
0.512
Go
9.19
JavaScrpp
0.531
CStarp
0A10
Chhrp
0.40
Pythen
0.506
SvaScnpt
PHP
0,406
PH
0.488
141
142
8.1
455
TypeSctpt
java
0.485
PHP
-5742
0.400
5.742
Rus
5,477
5.651
5.093
Python
JaveScript
Sotttt
9.464
CSham
5.563
-0.4652
Python
9.483
0.42
6.45
9.44
046
EAT
945
5:00
825
5.50
535
6.06
6.8
(a) Qwen3-0.6B-B
fabboade-ft
zero-shot
NL General
Matth
Python
Python
0.991
0
CSharp
Java
Rust
0,400
Java
0.305
CPP
0,400
0.304
TypeStript
TypeScript
0.388
Go
6-456
Go
CSharp
JavaScritt
0.319
PHP
0.454
CPP
JavaScrpt
sia
Code Understandinng
Code: Generation
JavaScript
Rust
0.478
Python
6546
TypeStript
Go
0.470
6.487
CPP
$ .466
JavaScrppt
6,472
0/463
Python
0/483
CPP
6.421
SypeSrript
PHP
0.451
CSharp
24299
CSharp
0,450
0.4
(b) Qwen3-0.6B
fo
Python
0.4G5
Rust
0.625
CSharp
Python
Rust
0.462
0.25
Java
0.462
PHР
0.622
JavaScrfpt
JavaScript
5.461
CSharp
3672
0-459
Go
0.17
Go
0.458
TypeScript
0.636
TypeStript
Java
04
8.4G
6:4
0:64
6.46
646
647
0.48
646
6.30
0.22
0.5
336
656
6.00
0.2
0.64
Undorstanalin
Code Gonoratnttnn
TypeScript
0.529
Python
8.437
CSharp
Go-
0.527
8,401
0.525
8.392
Python
CSharp
8.340
Go
0.522
Rust
8.328
JavaScript
TypaSoript
Rust
0.519
4.244
51.
Jave
JavaScript
8.168
053
655
033
15c
1.0
75
80
05
90
(c) Qwen3-1.7B
Figure 46: All programming language specific performance of Qwen-3 family.
40

---

Published as a conference paper at ICLR 2026
full-code-ft
General
Math
Code Understanding
Cade Generation
6,281-
0.292
0.42
0.220
0.200
4.0
0.215
3.200
0.274
277
0.379
3,205
0.205
0.204
0,222
4,171
low ster
high scripting
Intermddiate
intermediats
How system
high scripting
high scripling
Intermediate
low System
Internediale
low syctsm
scripting
(a) Grouped results (low-system, intermediate, high-scripting)
full-code
shot
NL & General
Math
Go
0.385
Go
0.210
PHP
Rust
CSharp
0.200
TypeScript
0.379
0,205
0.378
TypeScript
0.205
CSharp
0.370
Python
0,204
Python
0.378
Java
0.202
Java
0.376
JavaScript
0.201
CPP
CPP
JavaScript
5.50
5.50
Code Understanding
Code Generation
TypeScript
0.295
Java
4.429
CSharp
0.286
Go
4.359
Python
0.276
Python
Rust
CPP
2774
0.270
JavaScript
0.66
CSharp
4.116
Go
0.267
TypeSorpt
4.078
Java
0.261
JavaScript- 4.049
CPP
0.260
Rust 3.922
233
(b) Per-language results
Figure 47: Performance for 3.2ma-3.22-B.. (a) Programming language groups, (b) individual lan-
guages.
41

---

Published as a conference paper at ICLR 2026
full-code-It
General
Math
Code Understanding
Generation
0.420
0.035
.003
0.4475
1147
0.397
0.415
0.394
0.4425
0.440
5.135
0.135
0.133
0.355
0.320
4.4
2.500
0.375
4.2
hight scripting low System
interediate
intermeiiate
low system
high scripting
scripting
low system
Intermediatee
low system
Intermediats
high scripting
(a) Grouped results (low-system, intermediate, high-scriptig)
full coll-codeft
shot
NL General
Math
TypeScript
0.446
Java
Python
CSharp
0.138
CPP
0.441
Python
0.137
0.441
JavaScript
0.136
JavaScript
0.440
CPP
0.132
Rust
0.439
Go
0,)82
c
CSharp
0.130
Java
PHP
PHP
TypeStript
ci1
ole
Code Understanding
Code Generation
TypeScript
0.415
JavaScript
4.094
Go
0.405
Java
4889
JavaScript
0.403
Go
871
Rust
0,400
Python
Python
0.398
0.997
4.060
03386
4.813
0.379
CSharp
4.688
PHP
0.371
Rust
4.613
CSharp
0.370
TypeScript
4,556
235
(b) Per-language results
Figure 48: Performance for SmoIL.M-1,.7B. (a) Programming language groups, (b) individual lan-
guages.
42


Reviews:
[{"review_id": "1Ndx1z62iC", "reviewer": "Reviewer_zrYv", "summary": "In this work, the authors set out to validate the claim that coding data enhances reasoning abilities of LLMs, while also deepening the analysis by **ablating how different aspects of coding data correlate with downstream evaluations across three categories of tasks**: Natural language and general knowledge, math, and code generation and understanding.\n\nIn particular, the authors collate prompts from various open-source code instruction-tuning benchmarks, and ask GPT-4o to generate completions in 10 popular programming languages, resulting in a final dataset of 120k samples (12k per each language). They then provide 11 perturbed versions of the original dataset - 5 variants the result of programmatic perturbations (e.g. removing whitespace, renaming variables, removing comments etc.), and 6 variants of LLM rewrites (e.g. comment enhancement, conversion to pseudocode, or to natural language). \n\nThose datasets - along with a 120k general instruction following examples from OpenHermes 2.5 that constitute a non-code baseline - are then grouped into logical subcategories, and used to finetune models of various model families (e.g. Qwen3, Llama3, Gemma3), ranging from sizes of 0.6B to 8B.\n\nThe main findings:\n- Incorporating (unperturbed) **code data generally improves performance** across all tasks, especially math and coding (Figure 1).\n- **The perturbed datasets reduce model performance**, as compared to the unperturbed dataset.\n- **Semantic perturbations reduce performance less than structural** (Figure 3).\n- Perturbations which **highlight algorithmic structure** while removing superficial syntax, like pseudocode or flowcharts, often **match or surpass unperturbed code baseline** (Figure 4).\n- Models remain **robust to corrupted or low-interpretability code** (Figure 6).", "questions": "See \"Weaknesses\" section", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Investigating the of usefulness of different aspects of code data is a **timely and relevant, with clear practical utility.**\n\n- The work is **original in both framing and experimental design.** Disentangling different aspects of code data (structural vs semantic, human interpretable vs not) in the context of the data’s generalization to reasoning is a novel advance over prior, more-coarse grained (“code helps reasoning”) empirical investigations.\n\n- The **experiments are extensive**, comprising 3,331 fine-tuning runs across ten programming languages, eleven perturbation types, five model families, and multiple model scales ranging from 0.6B to 8B parameters. The appendix presents these results exhaustively, providing detailed plots and breakdowns that reinforce the empirical trends highlighted in the main text.\n\n- The paper **provides comprehensive replication details** in its appendix, including exact hyperparameters, dataset processing steps, and all prompt templates used for generation and evaluation.", "weaknesses": "- The **presentation of the perturbed datasets lacks detail** - the paper would be enhanced by a) providing overall statistics like the number of tokens, and b) providing qualitative examples of each dataset.\n\n- All **experiments are restricted to small, up to 8B parameter models**, leaving doubt whether the observations would generalize to frontier-scale models. If possible, the authors could corroborate their main findings by a few experiments on larger models. Barring that, scaling laws could be established across the 1B -> 8B scale.\n\n- Both the baseline and generative-perturbation code data are synthetically generated with GPT-4o (except for the prompts). However, **the paper provides no assessment of the quality and correctness of this generated data**. This could be mitigated by including a small-scale human evaluation—for example, manually annotating around 30 samples from each perturbation set. Alternatively, if resources permit, the authors could regenerate the data using a different frontier model and re-run key analyses to verify that the observed trends are consistent across data sources.\n\n- The paper presents numerous ablations, but results are shown separately for each model–size combination, with only a single representative plot per research question in the main text. This makes it difficult to see aggregate patterns, as readers must cross-reference many individual plots in the appendix. **Aggregating results across models or scales would make the reported trends clearer and more convincing.**\n\n- **Some claims in the main text are not fully substantiated by the experimental results.** For instance, when discussing performance across programming languages, the authors state that “on math tasks, high-scripting languages consistently underperform relative to intermediate and low-system ones,” referencing Figure 7 as evidence. However, in Figure 7 (top), high-scripting languages perform on par with intermediate ones and even surpass low-system languages on math tasks. In Figure 7 (bottom), the top five individual languages include three high-level ones, with all five well within each other’s error bars. Overall, the data shown (at least in the main body of the text) does not support the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aQMYd8Ugjl", "reviewer": "Reviewer_i4ZX", "summary": "- The paper studies why code improves reasoning in LLMs through a large-scale, systematic framework.\n\n- Constructs parallel instruction datasets in 10 programming languages and applies controlled perturbations (rule-based and generative) that selectively distort syntax, semantics, or structure.\n\n- Finetunes 5 model families across 8 scales (3,331 experiments) and evaluates effects on natural language, math, and code reasoning tasks.\n\n- Finds that structural features of code drive reasoning improvements more than semantics; pseudocode and flowcharts can substitute for code; lower-level languages aid math reasoning.", "questions": "- How robust are results to using a different evaluator model (e.g., Claude, Gemini) for judging code quality?\n\n- How would findings change if applied during pretraining rather than instruction finetuning?\n\n- Could smaller-scale runs (e.g., 20k data) reproduce the same trends?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clever perturbation design isolating structure vs. semantics; highlights causal signals in code-induced reasoning.\n\n- Impressive experimental scale and rigor - 3k+ controlled finetunes across models, languages, and perturbation types.", "weaknesses": "- Heavy reliance on GPT-4o-mini for generating and evaluating code; possible bias leakage or contamination\n\n- Finetuning limited to ≤8B models; it's unclear if findings hold for frontier scales where reasoning mechanisms differ.\n\n- Focus on finetuning setups - the model performance could depend on the pretrain LLMs.\n\n- It would be great to have some ablations on dataset size or diversity. This would help disentangle whether effects persist with smaller or noisier samples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "60iGWTeJlZ", "reviewer": "Reviewer_345W", "summary": "The paper probes which properties of code drive LLM reasoning via 3,331 finetuning experiments across five model families, eight scales, ten languages, and systematic perturbations. It finds structural features are pivotal, with their disruption reliably hurting math and code performance. Abstractions and efficient encodings can rival raw code, and even corrupted code stays competitive when surface regularities persist. Lower level languages yield larger math gains, suggesting practical principles for training data design beyond executable programs.", "questions": "Please refer to weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper demonstrates, from multiple complementary perspectives, that code data confers genuine benefits to LLM reasoning, and the evidence supporting the central claim that structural properties are decisive is clear, consistent, and persuasive.\n2. The experimental suite is broad in scope and methodologically solid, spanning diverse model families, sizes, languages, and perturbations, and it cleanly validates how targeted changes to structure or semantics translate into measurable differences in downstream performance.\n3. Several results crystallize actionable insights for the community, pointing toward data design practices that emphasize structural signals and efficient abstractions, and suggesting concrete priorities for future training and evaluation.", "weaknesses": "1. It would be valuable to extend the study to larger model sizes to observe how the perturbations interact with scaling.\n2. It is unclear whether the conclusions still hold in a reasoning model setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lNYuMW6PqH", "reviewer": "Reviewer_DDwQ", "summary": "This paper presents a systematic investigation into how different properties of code data influence the reasoning capabilities of large language models (LLMs) during fine-tuning. The authors construct parallel instruction datasets in natural language and across ten programming languages, then apply controlled perturbations targeting structural (e.g., whitespace removal, pseudocode conversion) and semantic (e.g., variable renaming, comment modifications) properties of code. Through lots of fine-tuning experiments spanning varied model families at different scales, they evaluate performance on natural language, math, and code tasks. Key findings reveal that structural perturbations degrade performance more severely than semantic ones, particularly on math and code tasks, while appropriate abstractions like pseudocode and flowcharts can match or exceed the effectiveness of original source code.", "questions": "- Can the authors provide analysis or at least some conceptual arguments to disentangle pretraining effects from intrinsic code properties? Or the authors would discuss the findings specifically for post-training rather than general LLM training (including all training phases)?\n\n- What steps were taken to ensure code correctness in the training data to avoid misleading signals, and can you discuss the error rates and potential impacts in your training data?\n\n- Why does pure code fine-tuning mostly outperform mixed training in RQ1?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The paper presents an interesting study to understand the impact of code data for LLM reasoning\n+ The experiments are conducted in a comprehensive way that include multiple different models with varied settings", "weaknesses": "**Flawed Experimental Design - Post-Training Phase Limits Causal Claims About Code Impact on Reasoning**\n\nA fundamental limitation of this work is that all experiments are conducted during the supervised fine-tuning phase on base models already extensively pretrained on massive corpora containing both natural language and code. This design weaknes the paper's central claims about which code properties enhance reasoning, as the observed performance differences may primarily reflect how well different formats activate existing pretrained knowledge rather than demonstrating the intrinsic learning value of code properties. For instance, when fine-tuning on perturbed code (e.g., whitespace removed, variables renamed), poor performance could simply indicate distributional mismatch with pretraining data rather than proving that structural properties are fundamentally important for reasoning. The surprising finding that \"corrupted code with misleading signals remains competitive\" may merely show that fine-tuning cannot override deeply ingrained pretraining patterns rather than demonstrating genuine robustness to surface regularities. This limitation calls into question whether the paper's guidance for \"design of training data\" will generalize broadly.\n\n**Correctness in the code data is not guaranteed, and the structural perturbations are overly simplified**\n\nAnother weakness of this work is the lack of validation for code correctness in the training data and the oversimplified nature of perturbations, which significantly limits the validity and depth of the findings. The authors generate all code responses using GPT-4o-mini, a relatively weak model, and seems apply no execution-based validation to ensure correctness (or I missed this part?). Without test cases to reject incorrect solutions during data generation, the 120K training examples may contain substantial amounts of buggy or non-functional code, fundamentally compromising the study's premise. If models learn from a majority of flawed examples, observed performance patterns cannot reliably indicate which code properties enhance reasoning but rather which properties are less affected by learning from errors. \n\nMoreover, the absence of test cases severely constrains the types of perturbations possible, forcing reliance on superficial transformations like whitespace removal and variable renaming that preserve surface syntax but cannot guarantee semantic equivalence. The study cannot perform meaningful semantically-equivalent transformations such as algorithmic refactoring, loop-to-recursion conversion, or function extraction that would provide richer insights into whether models learn genuine computational logic versus syntactic templates.\n\n**Inadequate Analysis of Why Pure Code Fine-tuning Outperforms Mixed Training in RQ1**\n\nThe paper lacks an in-depth analysis of why pure code fine-tuning (code-ft) consistently outperforms or matches mixed training (mixed-ft) across most tasks in RQ1, despite mixed training intuitively seeming more aligned with evaluation on diverse task types. Several plausible hypotheses remain unexplored, including task distribution mismatch, data quality difference among data resources, or the observations are mostly for post-training? Without ablation studies and analysis to distinguish between these possibilities, readers cannot assess whether the findings reflect fundamental properties of code for reasoning or other factors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)