ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding
Authors: Indraneil Paul
Haoyi Yang
Goran Glavaš
Kristian Kersting
Iryna Gurevych
Venue: ICLR 2025 Poster (2025)
Keywords: code generation
code obfuscation
language modelling
code lm
Primary area: foundation or frontier models, including LLMs
TL;DR: Training Code LMs on a translation objective of source code to obfuscated code improves their syntactic and semantic understanding of code along with their competency in library-oriented code generation.

Abstract: Language models (LMs) have become a staple of the code-writing toolbox. Their pre-training recipe has, however, remained stagnant over recent years, barring the occasional changes in data sourcing and filtering strategies. In particular, research exploring modifications to Code-LMs' pre-training objectives, geared towards improving data efficiency and better disentangling between syntax and semantics, has been noticeably sparse, especially compared with corresponding efforts in natural language LMs. In this work, we examine grounding on obfuscated code as a means of helping Code-LMs look beyond the surface-form syntax and enhance their pre-training sample efficiency. To this end, we compile ObscuraX, a dataset of approximately 55M source and obfuscated code pairs in seven languages. Subsequently, we pre-train ObscuraCoder models, ranging in size from 255M to 2.8B parameters, on a 272B-token corpus that includes ObscuraX and demonstrate that our obfuscation-based pre-training recipe leads to consistent improvements in Code-LMs' abilities compared to both vanilla autoregressive pre-training as well as existing de-obfuscation (DOBF) objectives. ObscuraCoder demonstrates sizeable gains across multiple tests of syntactic and semantic code understanding, along with improved capabilities in multilingual code completion, multilingual code commit summarization, and multi-purpose library-oriented code generation.

Published as a conference paper at ICLR 2025
POWERING EFFICIENT CODE LM TRAINING VIA OBFUSCATION GROUNDING
Indraneil Paul Haoyi Yang
Goran Glavaš Kristian Kersting Iryna Gurevych AIML Lab, TU Darmstadt
@ UKP Lab, TU Darmstadt
E CAIDAS, JMU Würzburg
ABSTRACT
Language models (LMs) have become a staple of the code-writing toolbox. Their pre-training recipe has, however, remained stagnant over recent years, barring the occasional changes in data sourcing and filtering strategies. In particular, research exploring modifications to Code-LMs' pre-training objectives, geared towards improving data efficiency and better disentangling between syntax and semantics, has been noticeably sparse, especially compared with corresponding efforts in natural language LMs. In this work, we examine grounding on obfuscated code as means of helping Code-LMs look beyond the surface-form syntax and enhance their pre-training sample efficiency. To this end, we compile ObscuraX, dataset of approximately 55M source and obfuscated code pairs in seven languages. Subse- quently, we pre-train ObscuraCoder models, ranging in size from 255M to 2.8B parameters, 272B-token corpus that includes Obscural aX and demonstrate that our obfuscation-bass pre-training recipe leads to consistent improvements Code-LMs' abilities compared to both vanilla autoregressive pre-training as well as existing de-obfuscation (DOBF) objectives. ObscuraCodes sizeable gains across multiple tests of syntactic and semantic code understanding, along with improved capabilities in multilingual code completion, multilingual code commit summarization, and multi- purpose library-oriented code generation.
demonstrates
INTRODUCTION
In recent years, language models for code generation (Code-LMs) have become near- sear-indispenssalee accessory a developer's ' toolbox. Their enhancement of productivity has proliferated into most of the software development lifecycle, including automating unit test generation, code infilling, predicting build errors, and code refactoring, inter alia, propelling their broad adoption in production (Dunay
al. 2024 Frömmgen al. 2024; Murali al. 2024). Code-LMs owe their competence gains in these areas to an ever-increasing parameter count (Luu et al. 2024) coupled with ever-larger scale of their pre-training: code corpora not only increasing in size but also in quality due to progressively better data sourcing and filtering (Yang et al.) (202)). While improvements to the hardware (Prabhakar jet al. 2024) and software (Shah et al. 2024) stack are likely to continue and in turn facilitate training of even larger models, LM training scaling-laws (Hoffmann et al., 2022) predict diminishing gains from the relatively slow growth of publicly available code corpora (Lozhkov et al., 2024). This warrants departure from the dominant trend of autoregressively training larger Code-LMe more code and towards alternative pre-training objectives that more efficiently exploit the inherent structure and regularity (Hindle et al., 2016) of source code.
Disentangling code syntax and semantics. All modern high-level programming source code inherently contains two information channels: syntactic and semantic (Knuth) 1984 Casalnuovo al. 2020). While the former refers conventional syntactic and naming choices developers make to facilitate interpretation and maintainability of the code (Casalnuovo et al.) 2209)), the latter pertains to the algorithmic intent behind the code. These two channels latently constrain each other and cannot be (fully) separated in form, only in understanding. To become a mainstay of software development, Code-LMs must excel in both syntactic and semantic code understanding.
The mastery of this dual-channel nature of programming code has so far, however, eluded Code- LMs. Existing studies show that Code-LMs' grasp of syntax is much better and obtained much
Equal contribution
Corresponding author: indr Indraneii il www.pp@@g-ddaassttttt....

---

Published as a conference paper at ICLR 2025
quicker during pre-training than their understanding of code semanties (aaik al. 2022 with their representations being sensitive to semantic- semantic-preserving surface- form changes (Rabin et al. [022]. J. The standard LM objective latches to common syntactic sub-spans, failing to capture the higher-level constructs in code (Ma et al. 2022) or common modes of usage (Fried et al. 2023), shown to be efficiently captured by specialized ized architectures (Shi et al. 2022). This is especially true for verbose languages, where separators account for significant portion of all tokens (Rahman et al. 2019).
Consequenly, disentangling Code-LMs' understanding of code syntax and semantics can unlock improvements on downstream tasks. Evidence for this has been demonstrated at inferenc--time Code-LMs completion performance drastically improves when provided with fine- ane-grained syntactic intent, described in natural language (Mueller et al., 2024 [Sun et al. 2024 Li et al.. 2023a Jain iet al. 2024 Wang et al. 2024a). A similar effect occurs when Code-LMs are prompted to generate code in multiple phases with the aid of code sketches that break down complex compositional
syntactic structures (Guo et al., 20226 Zan et al.] (022). Attempts at similarly untwining the two channels at pre-training time using contrastive learning (Wang et al.) 2022) or translation (Chakraborty et al. 2022) between semantically equivalent code fragments have been hamstrung by the limited attainability and diversity of such data (Bui et al. (02T). Other attempts have resorted to architectural cannot be utilized when text and code are intertwined (Kim et al. 202I), which is the most common mode of operation for Code-LMs. Thus, there is need for objective that fosters both semantic and syntactic code understanding while being easy to incorporate into modern autoregressive pre- pre-training of Code-LMs.
choices that are hard to scale (Zhang et al. 202
Surmounting the code data-wall. The choice of model and corpus size in language model pre- training has traditionally been guided by scaling laws (Kaplan et al. 2020; Hoffmann et al.) 2022) which provide a recipe on how to best trade the two off under a fixed computational budget. However, the widespread adoption of LMs has warranted revision of scaling laws to account for inference costs and latency and favours smaller models trained on more data (Sardana et al. 2024). With quality filtering in place, transformer- based models have demonstrated the ability to keep learning on orders of magnitude more data than what was considered optimal according to scaling laws (Gadre et al.) 2024).
This finding has subsequently been validated for Code-LMs in both language modelling (Rozière)
et al. 2023) and downstream task performance (Liu et al. 2024 Li et al. 2023b Allal et al. 2023).
Consequently, scaling up high-quality corpora for LM pre-training has hit the data-wall
the
exhaustion of openly accessible, unique and high-quality data (Bi et al., 2024). Recent LM re- leases (Dubey et al. 2024) may have already hit the limit of clean and openly crawled data (Penedo al., 2024) 15 trillion tokens). Code-LM developers face an even more alarming data-constrained reality, with the most extensive corpus of freely licensed source code containing under a trillion tokens by most measures (Lozhkov et al. 2024). Attempts circumvent this problem include repeating
data (Muennighoff et al. 2023). synthesizing data (Gunasekar et al., 2023] Wei et al.) (0224) and obtaining code-adjacent data from natural language corpora using
verdicts (Liu
et al. 2024). Further attempts have sought to obtain more data through the interplay between mener--genrratd code and programming language toolchains (Ding et al. 2024 Zheng et al., 2024).
Nonetheless, the best means to source the next trillion tokens of code data remains an open question.
The case for (de-sobfuscation as an objective. We posit that using obfuscated code in Code-LM pre- training enables: 1) Crafting objectives that better disentangle syntactic and semantic understanding of code; 2) A way out of the code data bottleneck. Obfuscation refers to syntactically mangling source code files in a manner that preserves their semantic correctness, primarily as a measure to prevent adversaries from reverse-engineering its function (Schrittwieser et al. 2016). This is mainly achieved by substituting variable, function, class and namespace identifiers with uninformative, generic names that make it harder for readers to discern semantic intent (Lawric et al. 2006). We provide complete examples of obfuscated code in Appendix B.3
Specifically, we introduce a translation objective for pairs of the original source code and correspond- ing obfuscated code for part of the pre-training corpus, including training the Code-LM explicitly on the structure of the obfuscated code itself, in contrast to prior work that leverages de-obfuscation objectives (Lachaux et al. 2021). Stripping away these natural language components in code (e.g.. variable and class names) that reveal semantic intent (Petrescu et al., 2023) forces the Code-LM to reason about the algorithmic meaning of the code from the syntactic structure alone. without allowing it to exploit semantic shortcuts. Simultaneously, the translation objective grounds the Code-IMMs
i iuaational- www.wwwwwaeeee.eeo..e ai attp:/ooooooot--toaaa/////------
wall
2

---

Published as a conference paper at ICLR 2025
understanding of the obfuscated code in the original source code, ensuring no modifications to the Code-LM are needed for inference. Additionally, obfuscation can be performed stochastically, thus providing easy way to scale up code pre-training corpora with diverse semantic-preserving preserving transformations in way existing rule-based techniques cannot match.
Contributions and research questions. this work, we: 1) create ObscuraX, a source-to- obfuscated-code translation pairs dataset containing approximately 55M pairs in seven programming languages; 2) conduct a systematic investigation of semantically grounding Code-LMs in obfuscated code, demonstrating sizeable and consistent empirical gains across broad range of tasks and programming languages; 3) train ObscuraCoder, a suite of base Code-LMs pre-trained on data mix containing Obscurak ax, ranging from in size 255M 2.8B parameters.
Our study of obfuscation-basse semantic grounding for training Code-LMs is structured as follows:
RQ1: Does training on obfuscated code help improve the performance of Code-LMs on tasks
requiring (a) syntactic and (b) semantic code understanding?
RQ2: Can supplementing identifier obfuscation with import obfuscation improve Code-LMs on library-oriented code generation?
RQ3: Are there any (undesirable) side-effects of obfuscation training, i.c. can it lead to performance regression tasks on which causal LM excel, e.g., code-completion?
RQ4: How does the relative performance of obfuscatioo- training differ and scale with increasing model size?
RELATED WORK
We briefly outline four relevant lines of existing work: syntactically- and semantically aaware code representation learning; 2) code translation as a pre-training objective; 3) code representation learning based on programming language toolchains; and, most related, 4) code (de- dle-)oofuscction.
Syntactically- and semantically-awarr code representation learning. The conventional autore- gressive objective used to train Code-LMs mimics their natural language counterparts, offering the convenience of reusing the same data preparation and training pipelines. This, however, fails to capture all the nuances of programming language syntax (Velasco et al., 2024), with resulting Code-LMs displaying uncertainty over certain syntactic structures such as exception handling and type inference (Palacio et al. (2023). Prior bids to address this issue at training time have resorted to ensuring syntactically faithful generations from Code-IMs using external signals via reinforce- ment learning (Parsert & Polgreen 2024) or by training dedicated syntax verification models (Yang) jet al., 2023b), inter alia. Other approaches include (i) targeting input embeddings by deploying differentiable adversarial perturbations for improved robustness (Li et al. 2022a). (ii) incorporating
auxiliary syntax tree position (Guo et al.)
2022c) and node-type information (Park et al. 2023
Takemgsakksri al. [0244] (Saber & Fard
2023) and (iii) test-time interventions such syntactic 2023a Shen et al., 2024).
grammar constrained decoding (Yang et al.
Similarly, while causal LM-ing imparts some understanding of code semantics (Ahmed al. 2023b), code representations learnt this way often fail on tests of semantic equivalence for complex code frag- ments (Troshin & Chirkova 2022). Existing work has thus sought to explicitly encode the semantics of the source code by leveraging data flow graphs (Chac et al.) 2017), control flow graphs (David) et al. 2020), program dependence graphs (Bichsel et al. (016)) and compiler intermediate representa- tions (Ben-Nun et al. 2018). Attempts along these lines usually employ task-specific graph-based (Du) & Yu 2023 Liu et al.] 2023c; Pei et al. 2024 Guo et al. (221)) tree-based (Jiang et al. (002bb), 2023c) or hybrid (Shi et al. 2023] Wang 2019) architectures and objectives. Other work seeks to improve semantic understanding by training Code-LMs on runtime execution traces (Wang & Sul 2019: Wang et al.) 2023a| Liu et al.) (023a). In contrast, we show that teaching a model the code structure via obfuscated code while simultaneoush training it to deobfuscate that code improves Code-I.Ms* syntactic and semantic code understanding abilities.
lensor-based (Yang
Translation pre-training objective. Most mainstream Code-LMs (Lozhkov et al. 2024 Liu et al., 2024 Mishra et al.) 2024 Zhao et al. 2024) -by virtue of being trained on GitHub code are least implicitly grounded in comment text- to-code translation objectives. However, they have
PIn this work, library-oriented refers to non-ohject-oriented Code Function Call APIs, the Gori LLa haxoooeoototonnnoooooooooooooo taxonomy. Typical examples external packages like Numpy, NetworkX, Seaborn, etc.
OpenFunctions
3

---

Published as a conference paper at ICLR 2025
mixed record on code-to-code translation (Pan al. 2024).
Attempts to improve code-to-code
translation have trained Code-LMs on parallel data, usually sourced from programming contests (Zhu
et al. (022bba) a). Subsequent work tried to generalize parallel data collection implicitly via, e.g., back-
translation (Ahmad et al., 2023; Chen & Lampouras 2023 Rozière et al. (020)) with applications in
code repair (Drain et al. 2021 Silva al. (023) and code explanation (Mahbub et al. (023).
Translation objectives have also been employed in training to improve Code-LMs' application-spee sppecifi representations. These include semantic grounding in (i) synthetic code outlines for multi-step generation (Shi al.) (024), (ii) syntax tree leaf nodes for retrieval (Phan & Jannesari (023). (iii) compiled binary code for vulnerability detection (Ahmad & Luo 2024). (iv) pseudo-code for code comprehension (Oda al.) (2115) and (v) compiler intermediate representations for improved multilingual performance (Szafraniec el al. 2023). We extend this body of work by demonstrating the benefits of obfuscation-based translation.
Programming language toolchain grounded representation learning. There is an abundance of work that grounds code in artefacts that originate from various stages of compilation. Amongst
the frontend artefacts, attempts to leverage syntax are the most common, most often encoding the
(linearization of) syntactic trees with recurrent (Jiang et al. 2022a), convolutional (Mou et al. (016).
graph-based (Zhang al., 2022) and transformer-based models (Guo al. 2022a). Other approaches
(i) use syntactic trees as a search prior for graph-based decoding (Brockschmidt et al. 2019). (ii) use
syntactic tree leaf element boundaries to inform masking (Wang et al. al.] (02 and tokenization (Gong
et al., (024), (iii) predict (heuristically selected) paths from the tree as an auxiliary pre-training objective (Tipirneni et al. 2024) and, (iv) leverage syntactic trees for data augmentation: heuristic generation of semantic- preeearriiiieeeviigggg.. transformations, leveraged for contrastive learning (Jain et al., 2021 Wang et al.] 2021aa] |Bahrami et al. 202T). Further compiler frontend artefacts such as data flow graphs [Brauckmann et al.) (2200) and control flow graphs (Nair et al. 2020) have also been leveraged to ground program understanding. Other work (Shojace et al.] [023]] Le et al.] (2022) uses comparisons of data flow and control flow graphs between the generated and reference code to shape a reward function for reinforcement caariingguiiee generation.
Compiler intermediate representations have also been leveraged for grounding, as (i) a source of meaning-preservingg preserving transformations (Li et al. 2022c) and as (ii) means to improve multilingual
performance of Code-LMs (Paul et al. 2024). Further from the developer, compiler backend artefacts such diagnostics (Ahmed et al. 2023a) and textual feedback (Ren et al. 2024 Liu et al. 20236) have been utilized to reduce functionally erroneous Code-LMs outputs. In this work, we leverage obfuscated code, typically produced by the compilers' backend, for improved grounding and dual- channel (i.e., syntax vs. semantics) disentanglement. However, aiming for customizability and broad applicability across programming languages, we write our own custom obfuscator in this work.
Code (de-)obfuscation. Code obfuscation is the crmnantic-preserving modification of source code to hide its execution intent. Neural approaches mainly focus on control flow (Ma et al. 2014). data (Yala) al. 2022) and identifier (Zhou et al.) 2022 Datta 202T) obfuscation but have limited adoption due to their tendency to introduce performance regressions and execution faults (Skolka et al. 2019))
In contrast, due to its undcrspecified and formally intractable nature Takaag et al.) 1996). code
de-obfuscation is natural task for neural methods. In particular, probabilistic (Raychev et al. 2019
Alon et al. 2018), recurrent (Bavishi et al. 2018 Lacomis et al. (219) and transformer-bast based (David al. 2020 models have all been employed for code de-obfuscation. More recently, DOBF (Lachaux
ct al. 2021) pioneered de-obfuscation as a pre-training objective for encoder-decoder Code-LMs.
rendering more effective than traditional sqquence-to-sequene denoising objectives (Raffel et al.
2020) on many-shot translation and retrieval tasks. DOBF, however, only trains the Code-LM on the identifier map and does not backpropagate gradients w.r.t. the source or the obfuscated code; instead, it relies on initialization with model trained on other objectives to maximize its code understanding abilities. contrast, our RoscuraCoder. models are trained from scratch on a bidirectional translation objective while mixing in language modelling on the unpaired source and obfuscated code samples. To the best of our knowledge, we are the first to show that (de-jobfuscation translation is viable pre-training objective for improving modern multilingual decoder-only Code- LMs. Explicitly modelling the structure of the original and obfuscated code, AbscuraCodes yields significant improvements in semantic robustness and pero-shot code completion compared to both vanilla autoregressive LMS and (a decoder-only variant of) DOBF. Moreover, we are the first to show that trading off semantic correctness (for fraction of the obfuscated samples) by mangling import statements in addition to identifiers leads to large gains in Abrrar-oriented code generation.
4

---

Published as a conference paper at ICLR 2025
3 OBSCURAX: SOURCE TO OBFUSCATED CODE TRANSLATION PAIRS
Source Code
Obfuscated Code
Eduntifier Nap
1.14 1e amaall1
If
Osfascnor
and lear
obfusoate Source Code
MA MURMUU WA.
Code: 7iies
Rre-Traiin
Code EX
TS
Caupal LM
DOBF
cbscuza Coder
cacca bubiide eust.... (D00)
easidat budale mark... [R28]
[808]
didthiat IMC (802)
poopp saleta porn
(B00)
(N0])OA TIM D CNt seeader badk porr (ROA)
Legend
Source Code Tokens
Identiter Mapping Tokens
Obluscated Code Takens
Mesked Tokens (No Dackpropi
Figure An overview of the ObscuraX data sourcing (Step and 2) and the ObscuraCoder pre-training objective, contrasted against that of causal LM and DOBF (Step3). Notably, Obscu- raCoder does not mask the obfuscated tokens and additionally trains on samples comprising only source or obfuscated data.
Seeking to test the hypotheses we posit in Section 1 we construct multilingual source-code-to- obfuscated-codd parallel dataset. We initiate this effort by acquiring source code files containing fewer than 2000 lines of code from the Stack corpus (Kocetkov et al., (023) in seven languages C, C++, Go, Java, Python, Rust and TypeScript. We ensure to retain all code sourced from research repositories [and and accepted programming contest solutions (Puri et al. 2021). owing their tendency to be self-contained. The resultant corpus is MinHash (Broder 1997) de-duplicated with the surviving source files' syntax trees extracted using Tree-sitter"" Well-known for its fault-tolerant approach to parsing. Tree-sitter affords us a reliable way to handle open-domain source code of vague provenance whose executability and correctness not assured.
The syntax trees obtained contain the span indices of the code's constituent syntactic elements. However, the targeted obfuscation of specific categories of syntactic structures requires reliably mapping this category information to the sub-trees of the syntax tree. Hence, write customized obfuscator that leverages Tree-sitter's own tree query language We build on top of queries employed by popular IDEs to account for the inherent complexity in accurately tracking syntactic structure compositions and their scopes for the multi-paradigm programming languages in ObscuraX.
The consequent mapping between spans in the source files and syntactic element category is em- ployed to create mapping from the original variable, function and class names to their obfuscated counterparts. This is stochastic and is influenced by the obfuscation proportion hyper-parameter Potf- which informs the proportion of the original elements uniformly varied up to 0.9, with higher values avoided to provide the model with some information learn the deobfuscation in a principled manner. We limit obfuscation to 150 members of specific syntactic category to reasonably restrict the samples' de-obfuscation difficulty level. The processed
wish to obfuscate. In practice, Poby is
git tree-sitter/trre-sittr
git tree-siteerrgrammrs/rrssssttee
gtt www.intreesstterr//iiitteeeeeeee
5

---

Published as a conference paper at ICLR 2025
counterparts are formatted as VAR_ [n], FUNC_(n) and CLASS_ [n] respectively, where ranges from to 149. Finally, for cases where the same name is shared by different syntactic structures or by syntactic structures of the same variety but different scope (e.g. global and local variables), preserve their shared semantic grounding using a catch-all category formatted as ID_(n}.
The described process is fully prarecttess-presevvvie allowing the curation of learning objectives that better disentangle the syntactic and semantic channels. However, to improve library-oiiented code generation, we sacrifice this property for =25% of the samples and obfuscate the imports as IMPORT_ [n]] This, in turn, facilitates the Code-LM's understanding of API usage during de- obfuscation, which can be vital for its ability on infrequently invoked libraries or rare use-cases (Rabin) al. 2023). Our resultant dataset, which we dub ObscuraX, comprises 5:5MM samples and the largest multilingual collection of source code to obfuscated code translation pairs yet. Figure I details high-level view of how ObscuraX is a critical part of the Obs ObscuraCoder
aCoder pre-training pipeline and also lists some examples. Refer to Appendix B.3 for more detailed samples in all languages.
EXPERIMENTAL SETUP
Pre-training data recipe. We compile two comparable pre-training corpora to facilitate a fair comparison between ObscuraCoder and standard autoregressive pre-training. We begin by ob- taining 105B tokens of high-quality filtered and de-duplicated text data, with the sourcing biased towards informative and instructional content from sources such as books, wikis, news articles and research papers following existing literature (Penedo et al., 2024). This is further augmented with 15B tokens of codc-adjacent text and code-text data such as library documentation, coding tutorials, GitHub commits and issues. We dub this 120B-token corpus filtered-code--enn
We then collect source code data from the Stack V1 (Kocetkov et al., 2023) corpus. Specifically, we select only the splits of languages from Obscur ax, i.e., C. C++, Go, Java, Python, Rust and TypeScript. We also select the Markdown split because of its importance for Code-LMs' eestrcctionfoolowiii abilities (Luo et al.] (024)). We subject this code corpus to further quality filtering and de-duplication, obtaining the final 76B-token corpus term filtereed-surrc--code
We organize the causal LM pre-training corpus into two phases: (1) 90B tokens sampled from fi -ooeee-code-eee text, and (2) two copies of siltteed-sourre code (152B tokens) plus the remaining 30B tokens of fi lilteeeddeee ceoe--ooeeeeeee randomly shuffled; for total training data of 227B tokens. By contrast, the Obscur
pre-training corpus reuses the first phase of causal LM training but compiles the second phase random shuffle of (1) 64B tokens from fi teltered- source code, (2) 30B tokens of obfuscated code from Obscur ax, (3) 58B tokens of translation pairs from Obscurax and (4) the (same) remaining 30B tokens of fi filtered codd-code-teed totalling 22B tokens. The translation pairs from Obscural aX are separated by two sentinel tokens which we add to the Code-LM's vocabulary- <src
also
and <obf_ to -u__ for the respective translation directions (see Figure T}. which are each sampled for 50% of the instances.
Both the corpora are designed to follow existing Code-LM best practices of initially priming the model on text-only data before training on a code-text mixture dominated by source code data (Guo et al.)
2024 Rozière et al. 2023 Hernandez et al.) 2021: Tao et al. 2024). Similarly, when shuffling the various splits during corpora creation, we ensure no source code fragment is included more than three times, thus staying within the optimal learning regime (Muennighoff et al.) (023). Finally, we ensure that our corpora are n n-gram decontaminated w.r.t. downstream tasks on which we evaluate (Chen
et al. 2021) Appendix B.2 further details our data sourcing and filtering pipeline.
Pre-training details. For direct comparability between vanilla Code-LM pre-training and Obscu- raCoder, we train both using the Llama architecture (Rozière al.) 2023) in four model sizes: 255M, 491M, 1.2B and 2.8B parameters. While most frontier models are pre-trained on corpora roughly an order of magnitude larger than ours, we argue that the scale of our pre-training runs suffice to demonstrate the merits of obfuscation-be based semantic grounding, as they are nearly an order of magnitude greater than what would be deemed optimal by the scaling-lawe (Hoffmann et al.) 2022).
All training runs are performed using an open-source implementation of the Megatron-LM (Shoeybil et al. 2019) kernels and leverage Deepspeed Stage-2 (Rajbhandari et al. 2020) sharding on BF16 precision. We use the Adam optimizer (Kingma & Ba] [2015), with learning rate of 5e-4 and a cosine annealed schedule that terminates at 5% of the peak learning rate. The models are trained
git EleutherAA/gptt-ne

---

Published as a conference paper at ICLR 2025
using F (Dao 2024 with sequence length of 2048 tokens and batch size of 256 for 520K steps. We use custom byte-level BPE tokenizer (Wang et al. (020) with a vocabulary size of 49152 tokens in total that we train 5B token corpus comprising of code and code-adjjccnt text data. The tokenizer is augmented with special tokens pertaining to the outputs of our obfuscator ID_(n}, CLASS_ {n}, FUNC_ {n}, [n} and [MPORT_[n]
n ranging from 0 to 149.
Inference details. For inference, resort to a Paged Attention (Kwon et al. 2023) enabled fork of an open-source evaluation harness We conduct our evaluation using model checkpoints loaded in half-precision (for efficiency) with nucleus sampling (p=0.9). All inference runs are conducted Nvidia A 100 80GB GPUs with 95% of the GPU VRAM explicitly reserved for vLLMs GPU pages. We further aside 64GB of RAM CPU swap, allowing for offloading pages the CPU during bursts of long sequences. We additionally limit the continuous batching parameter to 32.
5 EXPERIMENTAL RESEARCH QUESTIONS & RESULTS
Our evaluation comprises a mix of five zero-shot and fine-tuning tasks, selected to provide answers to the three research questions from Secciinn I For each task, we compare the performance of ObscuraCoder against an equally-sizen autoregressive LMM. We further contextualize the sample- efficiency benefits our obfuscation objective by including comparisons to seven frontier Code-L.Ms from the Deepseek-Coder Codder (Guo et al.
20024), CodeGemma (Zhao al., 2024). Phi (Gunasekar et al.,
2024 Li et al. 2023b) families that are under 3B parameters in 2023) and StarCoder (Lozhkov et al.
size and pre-trained on corpora between 5x to 22x larger than the one used train ObscuraCoder.
aCoder.
For fine-tuning tasks, all models are trained for three epochs using cosine scheduler with a peak learning rate of 5e-5 using LoRA (Xu et al.] (2024) modules coupled with trainable embeddings. We follow prior work (Poth al. (023) and train LoRA modules with rank 64 for classification tasks and 256 for open-ended generation. Unless otherwise specified, we use greedy decoding at inference.
Pre-Training
Defect
ReCode passel
BigCodeBench
Token Count
Acc.
F1-Score
Fornat
Syntax
Function
Passe1
Passgie
Passg25
CausalLM 255N
2726
62.39
61.11
11.37
8.98
3.13
2.12
3.87
1.32
ObscuraCoder 255H
2726
62-60
+0.41
62.14
+1.03
14.17
+2.80
11.80
+2.62
+1.82
4,95
40.52
2.64
+0.58
4.45
+0.65
4.97
Causalik 4918
2728
63.36
62.88
15.04
11.38
5.22
2.83
6.29
8.91
ObscuraCoder 491N
2726
64.27
+0.91
63.72
+0.04
23.68
+8.84
20.04
+8.74
+2.78
8.81
10.19
3.02
10.00
7.89
11.97
13.06
Causalik 1.28 2728
63.49
63.59
27.65 65
21.81
11.38
7,98
14.18
23.
ObscuraCoder 1.28 2728
64.49
11.00
64.76
+1.17
36.37
+8.73
27.86
#5.85
66.65
+5.36
11.69
9.67
17-69
13.71
28-84
14.27
Causalik 2. 2. 2728
64.73
64.36
34.78
26.
14.72
16.75
22.67
31.95
Obscuratoder 2.98 2728
65.58
+0.85
65.42
+1.06
25.29
+16.59
35.97
+9.77
29.11
+5.38
14.
+3.96
31.94
+9.67
18.78
+16.83
StarCoderBase 18 1.87
64.96
64.68
28.97
25.92
13.84
5.83
13.58
22.
DeegSeekCoder 1.36 2.0T
05.21
64.59
48.01
41.42
25.27
19.57
34.18
17.04
StarCodergase 38 1.97
65.62
64.92
37.21
38.25
17.49
8.26
26.79
37.74
StarCoder2 38 3.37
53.62
36.11
Stabl ecade 5.6T
58.08
CodeGenna 3.5T
61.93
16.31
33.12
Phi-2 1.4T
64.56
63.97
59.99
53.71
32.71
21.83
38.54
53.78
Table I: Results table for RQ1 and
RQ2 comparing syntactic understanding for code defect de-
tection on CodeXGLUE (Lu et al.] 2021), semantic understanding for robust code completion on
ReCode (Wang et al. 2023b) and library-oriented code generation on BigCodeBench (Zhuo et al.
2024). For detailed sp split-level breakdowns in ReCode results refer to Tables to
in Appendix C
RQ1: Obfuscation translation -> better syntactic and semantic understanding?
We first test if our Iituucattion-bseee training leads to a better understanding of complex semantic and syntactic structures, which are crucial for solving complex tasks like detecting vulnerable code.
CodeXGL.E Code defect detection. Detecting code defects requires integrating information from both channels due to the inherent dual-channel nature of software bugs. While some bugs, such as insecure argument usage, demand grasp of syntactic structures (Yamaguchi et al.] 2014; Ray et al.) 2016) and are easier to detect when code syntax is simplified (Thummalapenta & 2011), others, like memory leaks and null pointer references, require an understanding of control and data flow semantics (Zhou et al. 2019 Wang et al. 2016). We benchmark ObscuraCode and baseline CodeLMs on a binary defect detection classification task from CodeXGLUE (Lu et al., 202T).

---

Published as a conference paper at ICLR 2025
ReCode. We next employ five differently seeded ReCode (Wang al. 2023b) transformations of HumanEval (Chen et al. [202]. These examine a Code-LM's (zero-shot) robustness to semantically- preserving syntactic perturbations based its greedy decoding completions code generation prompts mangled using an extensive battery of Format, Function, and Syntax transforms, thus explicitly testing the model's understanding of the interplay between code syntax and semantics.
Results. Table Jummmaiizes the results of the above two tasks. We observe that obfuscation-basee objectives of bbbscuracon cCoder consistently bring significant performance gains on top of causal language modelling for all model sizes and both tasks. The gains are particularly large the ReCode task, keeping with prior work, which reports that translation between semantically equivalent programs benefits understanding of code intent (Guizzo al. 2024). The Obscur aCoder models frequently match or surpass the performance of the next-size CausalLMs, and AbscuraCoder 2. 8B beats several frontier models on semantic code understanding.
RQ2: Does code obfuscation training lead to better library-oriented code generation?
BigCodeBench. We investigate the effects of our de-obfuscation objective taken to its logical extreme, including the choice to obfuscate imports and package names, on Code-LMs' abilities in library-oriented code generation. Such obfuscation, we hypothesized, should have forced Code-LMs understand what library APIs do and when to invoke them. Prior work (Zhang al. 2021b) points to the demanding nature of package de-obfuscation. Because of this, we evaluate our Code-LMMs zero-shot on the BigCodeBench (Zhuo et al. 2024) benchmark in the completion setting, thus testing models' competence in API usage. We sample 50 generations per problem and report the pass@k performance for k E {1, 10, 25). The pass@l performance emulates usage scenarios where correctness is paramount: we thus decode with temperature sampling with a low temperature of 1. In contrast, pass@10 O and pass@25 estimates correspond to scenarios where creativity and diversity of generations are more important: we thus use higher sampling temperature of 0.8.
The results in the BigCodeBench column of Table 1 demonstrate substantial gains in library- oriented code generation performance facilitated by obfuscation-grounded grounded pre- pre-training that includes obfuscation of imports and package names. We believe that our design choice to represent obfuscated packages, which are often multi-token strings, with a single (special) token is one key driver of these gains, in line with suggestions from prior work (Hadi et al. (022). Particularly encouraging is the observation that Obscur acoderr' library-oriented generation gains (over the vanilla autoregressive LM-ing) widen with increasing model size. Obscur aCoder represents significant advancement in addressing API misuse, a major obstacle to wider Code-LM adoption (Wang et al., 2024b). Promisingly, ObscuraCoder 2.8B is competitive with the best frontier open-source models in this challenging setting that most closely mimics real-world usage.
Results.
RQ3: Are there negative side-effects of obfuscation -aase pre-training?
We next test whether improvements in dual-channel understanding and library-oriented code gen- eration come at a cost, i.e., performance deterioration, for zero-shot multilingual code completion, which is still the most common application of Code-LMs.
Multipl-E. We first carry out multlingual evaluation of passčk performance for k E {1, 10, 100} in five languages (C++, Java, Python, Rust and TypeScript) using the Multipl-E benchmark (Cassano) et al. 2023). We sample 200 generations per problem. Like in BigCodeBench evaluation, we set the sampling temperature to 0.1 for pass@1 and to 0.8 for pass@10 and pass@100.
ComnitChronicle. As a further measure of multilingual code competence, we evaluate Code-LMs on CommiiChronicle (Eliseeva et al. 2023), a fine-tuning code-change summarization benchmark in seven languages (C, C++, Go, Java, Python, Rust and TypeSSrript. Summarizing code changes is a good proxy for multilingual code understanding due to its dual nature w.r.t. code completion (Weil cct al. 2019) and the tendency of the capabilities in each tos--text-to-coee and code-to-text- to reinforce the other. We construct language-specific splits by filtering the original dataset and partitioning 75%, 15% and 10% of the data into train, validation and test splits, respectively. In fine-tuning, we ensure that autoregressive losses are backpropagated only for the target summaries.
Results. Table 2 points to the strong performance of Obscur on both multilingual bench- marks. Not only do our (de-)obfuscction pre-training objectives not hurt multilingual generation
As early adopters, ran vo. 1. of the benchmark, where 13 network usage problems were found to contain unreliable tests and were discarded. We report all results the remaining 1127 problems.
8

---

Published as a conference paper at ICLR 2025
Nodel
Pre-Training
Hultipl-E
Comnit Chronicle
Token Count
Passe1
Pass@10
Passg100
ROUGE-1
ROUGE-2
ROUGE-L
Ca 255H
272B
4.29
7.13
14.36
32.76
10.
31.14
Obscur eCoder 255H
2728
+1.65
5.83
+2.53
9.66
18.20
13.84
33.40
18.65
+0.48
18.64
31-78
10.56
CausalLM 491H
2728
5.84
10.64
20.69
34.08
11.02
32.48
ObscuraCoder 491M
272B
+2.92
8.76
14.33
+3.69
25.86
+5.27
34.85
18.78
+0.42
11.44
33.83
10.65
CausalLM 1.26 2728
12.07
19.50
31.10
35.
11.96
33.79
ObscuraCoder 1.20 2728
18.34
16.27
24.34
+9.83
47.79
13..69
37.00
+1.25
13.20
+1.24
35.34
+1.65
CausalLM 2.88 2728
16.78
28.83
47.53
36.
12.89
85.
Obstur sCoder 2.88
2728
23.55
16.85
+10.59
39. 41
61.93
+14.39
38.07
+1.25
14.15
+1.26
36.68
+1.63
StarCodorBase 18 1.0T
13.22
22.53
37.78
37.85
13.19
35.38
DeepSeekCoder 1.36 2.0T
26.61
47.46
71.70
35.86
12.95
34.42
StarCoderBase 38
1,01
19.21
33.48
57.60
38.65
14.44
36.88
StarCoder; 36 3,37
27.38
49.63
72.86
38.75
14.58
36.96
StableCede 38
26.73
38.98
CodeGemaa 2B 3.5T
22.71
62.14
36.
35.28
Phi-2 2
22.27
39.96
57.30
35.24
12.16
33.61
Table 2: Results table for RQ3 comparing multilingual code completion averages on Multipl-E Cas
sano et al., 2023) (C++, Java, Python, Rust, TypeScript) and multilingual commit summarization
averages on CommitChronicle (Fliseeva et al. (023) (C, C++, Go. Java, Python, Rust, TypeScript). For detailed language-wise breakdowns in Multipl-E results refer to Tables 10 to 12 and Com-
mitChronicle results refer to Tables 13 to I5 in Appendix C
performance compared causal LM-ing, but they seem to improve it. Morcover, similar to the results on BigCodeBench, the gains from obfuscation pre-training over autoregressive LM-ing generally increase with the model size. Prior work (Bavarian et al. 2022 Wang al., 2021b) refers to incorpo- ration of additional objectives during pre-training as being "for free" when these objectives do not incur any deterioration in autoregressive performance. Owing its pre-training objective, Obscu- r raCoder goes beyond this and actually displays strong evidence of positive transfer from improved syntactic, semantic and API understanding to multilingual code completion and summarization.
RQ4: Do the benefits of obfuscation obfuscation-groundingg scale?
155 Code Defect Detection
Accerace
ReCode HumanEval
Fermat
15
Big Bldeeoooon ench Comh-ooo
Pasege
15
MuttiplE HumanEval
Paaeg1
18
CommitChronidle
ROUSG1
Syste
ename 2
1.0
12
12
10
12
0.5
2339
2539
Figure 2: Downstream performance gains of ObscuraCoder viz-a-viz Causal Code-LMs.
Due to computational constraints, we only train models up to 2.8B parameters in size. We attempt to extrapolate the effects of obfuscation pre- pre-training to parameter sizes resembling those of the most widely adopted Code-LMs [0 by investigating the scaling trends of downstream task performance.
Figure 2 /summarizes the performance of ObscuraCoder across all five downstream tasks function of model size. Expectedly, scaling model size brings more modest gains in fine-tuning setups (CodeXGLUE Defect Detection and Comni(Chronicle) than in zero-shot evaluations:
observe saturation in fine-tuning performance also for the larger frontier model (see Table and Table 2). Scaling trends are very prominent in zero-shot tasks, especially library- -orente (einChd)Bench) and multilingual (Multipl-E) generation: this is encouraging, as these tasks correspond to Code-LMs' most prevalent modes of use. Finally, we observe the strongest scaling trends in settings that reward diverse code generation: high pass rate estimates on high-temperature generations. This is promising for senerate-then-reraae settings, where being able choose from deep bank of valid solutions drives further improvements to end-user utility (Li et al. 2022b Chen et al. 2023) and opens room for optimizing non-functional axes (Waghjale et al. 2024).
Generally, our results strongly agree with prior work that establishes the benefits of information- preserving transforms in LM pre-training Yuuan & Liu 2022 Maini et al., 2024). Specifically, we translation objectives.
establish that Code-LMs can substantially benefit from jete--ooouccttonbaaeee
At the time of writing, this ranges from approx. 3x 150x larger than the largest ObscuraCoder model.

---

Published as a conference paper at ICLR 2025
6 ABLATIONS
How Does ObscuraCoder Compare Against DOBF? We attempt explicit head-to-head comparison between the de-obfuscation objective pioneered by DOBF (Lachaux et al. 202T) and our obfuscation-groundee translation objective. However, the originally released DOBF model an encoder-decoder model with 126M parameters, making it half the size of our smallest model. Hence, to facilitate fair comparison, we train a decoder-only adaptation of DOBF on a modified version of the Obscur cur aCoder corpus of 2272 tokens outlined in Secttion 4 Specifically, the 30B tokens of obfuscated code and the 58B tokens of translation pairs are replaced by 88B tokens of source code to identifier map translation pairs sourced from ObscuraX (see Figure 1 for the DOBF objective).
We train size-matched versions of DOBF on the above-detailed corpus. Staying faithful to the original de-obfuscation objective, we mask the obfuscated code from the loss computation and only back- propagate gradients for the identifier maps targets - 12 Results in Table 3hww that ObscuraCodes has an advantage over DOBF across the board, with particularly prominent gaps on on zero-shot tasks. Promisingly, the performance gap appears to widen with increasing model size.
G Model
Defect
ReCode
BigCodeBench
HultipI-E
F-1 Score
Fornat
Syntax
Function
Pass@1
Pass@25
Pass@1
Pass@100
ROUGE-2
obscuratader 255M
62.14
14.17
11.58
4.95
2.64
4.97
5.93
18.20
18. 64
62.22
14.80
11-67
4.97
2.68
4.70
4.62
16.85
18.36
DOBF 255M
10.08
-B.17
-0.13
10.02
10.04
-B.18
-1.31
-1.35
-0.28
CaUSa 255M
62.83
-8.11
11.85
-3.12
-3.53
8.27
-2.14
2.81
-0.78
1.86
-0.76
4.21
-1.93
4.88
15.
-3.18
18.88
-0.56
ObscuraCoder 491M
63.72
23.08
20.84
8.01
3.02
11.
5.76
25.86
11.44
DOBF 491M
63.65
-8.07
15.93
-7.95
12-78
-7.33
-1.10
6.91
-0.13
2.89
18.96
-1.91
6.75
28.32
-5.54
11.15
-B.25
CousalUN CP 491M
62.27
15.59
11.91
-8.13
6.188
2.46
8.
6.89
22.04
18.92
-0.52
ObscuraCoder 1.28
64.76
36.37
27.
16.
9.67
28.84
16.34
47.
13.20
DOBF 1.28
64.67
-0.09
30.86
-6.31
23.65
-4.21
13.87
-3.59
-0.53
9.14
25.95
-2.09
15.25
48.
-7.49
12.52
-9.68
CausaTLN-CP 1.28
63.46
26.58
28.71
11.38
6.78
19.84
13.34
37.
12.16
ObscuraCoder 2.88
35.97
20.11
14.77
23.55
61.93
DOBF 2.88
65.37
37.92
29.18
16.69
13.18
43.65
28.32
51.27
13.39
CausalLN-CP 2.88
63.94
35.65
26.86
15.85
18.68
33.74
17.92
47.
13.24
Table 3: Results table for the comparative ablations of Obwcobbaoodooooo compared to comparatively pre-trained DOBF model and a comparatively pre-trained CausalLM model, continually pre-traines on 8B tokens (CausalLM-CP) using obfuscation-grouse grounded translation.
Finally, we test the effectiveness of our obfuscation translation training when it is post-hoc applied on an already pre-trained causal Code-I.M. To this end, we subsample 8B tokens of translation pairs from ObscuraX and continue pre-training our causal LMs on obfuscation translation. The resulting models, dubbed Causall M-CP, are compared against AbscuraCodes in Table 3 Unfortunately, we find that post-hoe obfuscation training does not confer performance gains to causally pre-trained LMs. This points to the importance of early obfuscation Defuucation-grounding grounding of code representations.
Does Post-Hoc Obfuscation Training Suffice?
7 CONCLUSION
This work investigates the effects of bofuscattin--grondde Code-LMs' representations across multi- ple programming languages. We create ObscuraX, an =119-ttoeen code parallel dataset containing =55M training instances across seven languages. We then pre-train AbscuraCodes a suite of models ranging from 255M to 2.8B parameters in size, on a R=22BB token corpus that samples from ObscuraX and demonstrate that obfuscation grounding leads to substantial gains in syntactic and semantic understanding of code. We further uncover broader benefits of adding obfuscation translation to CodeLMs' pre-training mix: improved library-oriente code gen- eration, multilingual code completion and summarization. Our results render obfuscation based code translation viable objective for bypassing the code data bottleneck. We hope our promising results catalyze the incorporation of obfuscated code into the pre- pre-training recipes of frontier Code-LMs.
"While this leads to fewer gradient propagatossss for the DOBF model, maintain that this is an inherent disadvantage that training with this objective in the current data-constrained regime incurs.
1he mainline GPTNeoX library does not support custom loss masking. For DOBF training, we employ hitherto un-merged pull request containing the feature: 1 EleutherAA/ggot-nee//plllll
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The work benefited from the Hessian Ministry of Higher Education and the Research and the Arts (HMWK) cluster project "The Third Wave of AI" as well as from the HMWK and BMBF joint support of the National Research Center for Applied Cybersecurity Center ATHENE. Additionally, the work acknowledges the support of Huawei Technologies (Ireland) Co., Ltd. Finally, the pre- training jobs were supported by compute grant the "FortyTwo" AI SuperPOD as part of the Hessian.Al Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation), grant no. S-DIW04/001 3/003) and the hessian.AISC Service Center (funded by the Federal Ministry of Education and Research, BMBF, grant No 011S22091).
REFERENCES
Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. Towards a cleaner document-oriented multilingual crawled corpus. In LREC, pp. 4344-4355 European Language Resources Association, 2022.
Iftakhar Ahmad and Lannan Luo. Unsupervised binary code translation with application to code similarity detection and vulnerability discovery. CoRR, abs/2404. 19025, 2024. doi: 10.48550/ ARXIV.2404.19025. URL httpss://doi.rrg.
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Summarize and generate to back-translate: Unsupervised translation of programming languages. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2- 6, 2023, pp. 1520-1534. Association for Computational Linguistics, 2023. doi: 10. 10.18653/V1/2022 1/2033/ EACL-MAIN,112 URL httth::p:/// /oi or 10. 18653 / 26231 www.-aacaaannn 112
Toufique Ahmed, Noah Rose Ledesma, and Premkumar T. Devanbu. Synshine: Improved fixing of syntax errors. IEEE Trans. Software Eng., 49(4):2169-2181, 2023a. doi: 10.1 3212635. URL htttt::////o/ oog.rrrggg//ooo 10. 2t1.//32....222.222655555
Toufique Ahmed, Dian Yu, Chengxuan Huang, Cathy Wang, Prem Devanbu, and Kenji Sagae. Towards understanding what code language models learned. CORR, abs/2306. 1943, 2023b. doi: (044550/AXXI.2236.1115
URL http::://oo....r////..4//433 www.:o........///r......
2306
11943
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Muñoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy-Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García del Río, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, and Leandro von Werra. Santacoder: don't reach for the stars! CoRR, abs/2301 033988, 2023. doi: 10. 00448500/RRXI.2300 .03988. URL. http::tps:////e //doi wtt.:oorrr..11188881 4855000.455554111 2301 03988
Miltiadis Allamanis. The adverse effects of code duplication machine learning models of code. In Onward!, pp. 143-153. ACM, 2019.
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. A general path-based representation for predicting program properties. In Jeffrey S. Foster and Dan Grossman (eds.), Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, Philadelphia, PA, USA, June 18-22, 22, 2018, pp. 404 419. ACM, 2018. doi: 10. 10.1145/3192366 3192412. URL https:://// /ooii orgo 10. 2114533192366 3192412
Mehdi Bahrami, N. C. Shrikanth, Yuji Mizobuchi, Lei Liu, Masahiro Fukuyori, Wei-Peng Chen, and Kazuki Munakata. Augmentedcode: Examining the effects of natural language resources in code retrieval models. CoRR, abs/211100885 12, 2021. URL httph:tt////axttt wwwp/biooorg/bb..... 2110 08512
11

---

Published as a conference paper at ICLR 2025
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. CoRR. abs/2207.14255, 2022. doi: 10.48500ARRXIV2220.1122 48550/arXiv. 2207. 14255
URL https: oorgo.oooooo00
Rohan Bavishi, Michael Pradel, and Koushik Sen. Contex(2name: A deep learning-basee approach to infer natural variable names from usage contexts, CoRR, abs/ his//880 80980519395 2018. URL. http:
//arxiv.org ats// 1et.s199990009
Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. Neural code comprehen- sion: A learnable representation of code semantics. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 31: Annual Conference Neural Informa- tion Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. neurips. hccppappr//008//heee
3589-3601, 2018. URL htt.a//eeee000000000oo///0000--0
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang. Fangyun Lin, Alex X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nic, Tian Pei. Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren. Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun. Minghui Tang, Bingxuan Wang. Peiyi Wang, Shiyu Wang, Yaohui Wang. Yongji Wang. Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong. Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang. Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseek LLM: scaling open-source language models with longtermism. CoRR, abs/2401 022554, 2024. doi: 10.48550/ARXIV2240.0000 02295444......... URL https://doi.org wtw.o///.....////////// art55///arxv.v.o 240.0/255400009444444
Benjamin Bichsel, Veselin Raychev, Petar Tsankov, and Martin T. Vechev. Statistical deobfuscation of android applications. In Edgar R. Weippl, Stefan Katzenbeisser, Christopher Knuegel, Andrew C. Myers, and Shai Halevi (eds.), Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. Vienna, Austria, October 24-28, 2016, pp. 343-355 ACM, 2016. doi: 10.1145/297674. 29784422. URL httpps:///ooooo or / 10. 1145/9976749.2978222 101
Alexander Brauckmann, Andrés Goens, Sebastian Ertel, and Jerónimo Castrillón. Compiler- Compiler-based based graph representations for deep learning models of code. In Louis-Noël Pouchet and Alexandra
Jimborean (eds.), CC '20: 29th International Conference on Compiler Construction, San Diego CA, USA, February 22-23, 2020, pp. 201-211. ACM, 2020, doi: 10.1145/337755533377994 URL https://doi org/ 16. 3114413337755337777777
Mare Brockschmidt, Miltiadis Allamanis, Alexander L. Gaunt, and Oleksandr Polozov. Gen- erative code modeling with graphs. In 7th International Conference on Learning Represen- tations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL het. id=Bke4KsA5FX
Andrei Z. Broder. On the resemblance and containment of documents. In Bruno Carpentieri, Alfredo De Santis, Ugo Vaccaro, and James A. Storer (eds.), Compression and Complexity of SEQUENCES 1997, Positano, Amalfitan Coast, Salerno, Italy, June 11-13, 1997, Proceedings. pp. 21-29. IEEE, 1997. doi: 10. W9W.:EQUEN.199..666698 0997.666990
JtL/https:/ htattp:://doi.orgg///
B109/SEQUE
Nghi D. Q. Bui, Yijun Yu, and Lingxiao Jiang. Self-supervised contrastive learning for code retrieval and summarization via semantic-preserving transformations. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR '21: The 44th International ACM SIGIR Conference Virtual Event, Canada, July 11-15, 2021, pp. 51 511- ACM, 2021. doi: 10.1145/3404835. 3462840. URL https:: .ariiiorrgoo org 10 011.111111115444441 3462840
Research and Development in Information Retrieval,
12

---

Published as a conference paper at ICLR 2025
Casey Casalnuovo, Kevin Lee, Hulin Wang. Prem Devanbu, and Emily Morgan. Do people prefer "nat- ural" code? CoRR, abs/1910.03704, 2019. URL
a9w/ 03 704
Casey Casalnuovo, Earl T. Barr. Santanu Kumar Dash, Prem Devanbu, and Emily Morgan. theory of dual channel constraints. In Gregg Rothermel and Doo-Hwan Bae (eds.), ICSE-NIER 2020: 42nd International Conference on Software Engineering, New Ideas and Emerging Results, Seoul, South Korea, 27 June 19 July, 2020. pp. 25-28. ACM, 2020. doi: 10.1145/377886633381222 URL https: / /tti//doioooo org 10 /00.//33771661 3381720
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-HH Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: A scalable and polyglot approach to bench- marking neural code generation. JEEE Trans. Software Eng., 49(7): 3691, 2023. doi: / TSE. 2623 3267446
20.1109/TSEE2223.3333 3266444.... URL pnttps://dos org g 10. 1169
Kwonsoo Chae, Hakjoo Oh, Kihong Heo, and Hongseok Yang. Automatically generating features for learning program analysis heuristics for c-like languages. Proc. ACM Program. Lang.. ((0PSL))
101:1-101:25, 2017. doi: 10.1145/3133925. URL hRtps::s: org. 10.
Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar T. Devanbu, and Baishakhi Ray. Natgen: generative pre-training by "naturalizing" source code. In Abhik Roychoudhury, Cristian Cadar, and Miryung Kim (eds.), Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, pp. 18-30. ACM, 2022. doi: 10.1145/ 3540250.3549162 URL httpt: org 10. 1145 3546250 3549162
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin. Jian Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. net/ www/forrrtee forum? d==trw68Cmu99
URL https::t////m http:/t:://pppeeeeeeeeeeee http:///ppeeeveeeeeeeeeee
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra. Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CORR, 0e..221770033744 2021. URL htthh::/t///rrt or g/ abs/ / 0tt.//217.033777 03374
Pinzhen Chen and Gerasimos Lampouras. Exploring data augmentation for code generation tasks. In Andreas Vlachos and Isabelle Augenstein (eds.), Findings the Association for Computational Linguistics: EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pp. 1497-1505. Association for Computational Linguisties, 2023. doi: 10. FINDIINS //doi org. 10. 1......166333112222200 findings-eac 114
URL https :
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:/ httpp:://ppeerevieee nww./ooumwffttteeeooooooo
mZn2Xyh9Ec
Siddhartha Datta. Deepobfuscode: Source code obfuscation through sequence-to-sequeecee networks. In Kohei Arai (ed.), Intelligent Computing Proceedings of the 2021 Computing Conference, Volume 2, SAI 2021, Virtual Event, 15-16 July, 2021. volume 284 of Lecture Notes in Networks and Systems, pp. 637-647. Springer, 2021. doi: 10.1007/978- 3-778----000---- 80126-7V 11166-7445 45. URL https
7/doi . org 10. 1007 1907 3- 030 80126-7_45
13

---

Published as a conference paper at ICLR 2025
Yaniv David, Uri Alon, and Eran Yahav. Neural reverse engineering of stripped binaries using augmented control flow graphs. Proc. ACM Program. Lang. (0:00PSLA::2:1122222222 2020. doi: 10.1145/3428293. httpss/: www.oorgg... 10. 1145 3428293
Yangruibo Ding, Benjamin Steenhoek, Kexin Pei, Gail E. Kaiser, Wei Le, and Baishakhi Ray. TRACED: execution-awer aware pre-training for source code. In Proceedings of the 46th IEEF/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pp. 36:1 1-6:: 12. ACM, 2024. doi: 10.1145/3597500336600 3608 140. URL https: //dp::://do.... org/ 10. 1145 3608140
Dawn Drain, Colin B. Clement, Guillermo Serrato, and Neel Sundaresan. Deepdebug; Fixing python bugs using stack traces, backtranslation, and code skeletons. CoRR, abs/2105.09352, 2021. URL
https://aad /tt////rxvvvv// or hbwgg//b////...9//333
Yali Du and Zhongxing Yu. Pre-training code representation with semantic flow graph for effective bug localization. In Satish Chandra, Kelly Blincoe, and Paolo Tonella (eds.), Proceedings the 31 ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/ESE 2023, San Francisco, CA, USA, December 3-9, 2023, pp. URL
579-591, ACM, 2023. doi: 8611643.3616333
oor.doo..r..o o00. 1145/
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, and et al. The llama herd of models. CoRR, abs/2407.2178.. 2024. doi: 10.45500ARXXV.2407..12 htttps://oii.om org/ 48550 arwiv. 0..400227776
URL
Omer Dunay, Daniel Cheng, Adam Tait, Parth Thakkar, Peter C. Rigby, Andy Chiu, Imad Ahmad, Arun Ganesan, Chandra Shekhar Maddila, Vijayaraghavan Murali, Ali Tayyebi, and Nachiap- pan Nagappan. Multi-line ai-assisted code authoring. In Marcelo d' Amorim (ed.), Companion Proceedings of the 32nd ACM International Conference on the Foundations of Sofwwree Engi- neering, FSE 2024, Porto de Galinhas, Brazil, July 15-19, 2024, pp. 150-160. ACM, 2024. doi: 3663836
01014453366322.366333888 URL htLhtttps:// //toi. org/ 10 1.0...14555
Aleksandra Eliseeva, Yaroslav Sokolov, Egor Bogomolov, Yaroslav Golubev, Danny Dig, and Timofey Bryksin. From commit message generation to history-aware commit message completion. In 38th JEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 1 11-15, 2023, pp. 723-735. IEEE, 2023. doi: LLL.9//AE56299.023...... 10. 1109 ASE56229 2023. 00078
URL
https://doi1 orgo /
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. In ICLR. OpenReview.nee 2023.
Alexander Frömmgen, Jacob Austin, Peter Choy, Nimesh Ghelani, Lera Kharatyan, Gabriela Surita, Elena Khrapko, Pascal Lamblin, Pierre-Antoine Manzagol, Marcus Revaj, Maxim Tabachnyk, Daniel Tarlow, Kevin Villela, Daniel Zheng, Satish Chandra, and Petros Maniatis. Resolving code review comments with machine learning. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, ICSE- SEIP 2024, Lisbon, Portugal, April 14-20, 2024, pp. 204 215, ACM, 2024. doi: 10.1145/663947733639766 URL https 7/doi . .rg 11tp://111111///377777777777177
14

---

Published as a conference paper at ICLR 2025
Samir Yitzhak Gadre, Georgios Smymis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on downstream tasks. CoRR, abx/2403 2024. doi: 10.4 48550//AXXI.24403088228 08540
URL httpp:t//tii.oooo htt.::///0.....o0g/0/04444555//
Linyuan Gong, Mostafa Elhoushi, and Alvin Cheung. AST-T5: structure-aware pretraining for code generation and understanding. In ICML. OpenReview.net, 2024.
Giovani Guizzo, Jie M. Zhang, Federica Sarro, Christoph Treude. and Mark Harman. Mutation analysis for evaluating code translation. Empir. Softw. Eng. 29(1); 2024. doi: 10.1007/ S10664-023- 10385-W. hRtps:tss://doo org/ 10. 1007 011164-06644222 002 10385-w
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah. Harkirat Singh Behl, Xin Wang. Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. CoRR, abs/2306. 1644, 2023. doi: 10. 48550/ARXIV.2306. 1644. URL https: //doi. org 10rg//0... 48550 arxiv 2306. 11644
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng. Colin B. Clement, Dawn Drain, Neel Sundarcsan, Jian Yin, Daxin Jiang, and Ming Zhou. Graphcodebert: Pre-training code representations with data flow. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https::/ openreview. net/ htt/fformm??deee (ofoormm21ddjooA4e4433333
Daya Guo, Shuai Lu, Nan Duan. Yanlin Wang, Ming Zhou, and Jian Yin. Unixcoder: Unified cross-modal pre-training for code representation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational
Linguistics ( (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 7212-7225 URL
Association for Computational Linguistics. 2022a. doi: https://dof..rg///I htt.://///g..111////////33 aww. 499
Daya Guo, Alexey Svyatkovskiy, Jian Yin, Nan Duan, Marc Brockschmidr, and Miltiadis Allamanis. Learning to complete code with sketches. In The Tenth International Conference on Learning Representaations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022b. URL /ttps://opeerevii eew. net/ wot///ormmmmii wtt.rummmmddg790ss5555555
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseck-coder: When the large language model meets programming the rise of code intelligence. CoRR, abs/2401. 2024. doi: 00448550/AXXXV2241..111 14196
URL htwpt:tpt/////m //dpi: org/ 10. 48550/ arxiv 2401
Juncai Guo, Jin Liu, Yao Wan, Li Li, and Pingyi Zhou. Modeling hierarchical syntax structure with triplet position for source code summarization. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 486-500, Association for Computational Linguistics, 2022c. doi: 11111665//1/2222A11-...0 ac Lacg-long.33 .7
URL
vittps://det or / 10.
Mohammad Abdul Hadi, Imam Nur Bani Yusuf, Ferdian Thung, Kien Gia Luong, Lingxiao Jiang, Fatemeh H. Fard, and David Lo. On the effectiveness of pretrained models for API learning. In Ayushi Rastogi, Rosalia Tufano, Gabriele Bavota, Venera Arnaoudova, and Sonia Haidue (eds.), Proceedings of the 30th IEEF/ACM International Conference on Program Comprehension, ICPC 2022, Virtual Event, May 17, 2022, pp. 209-3200-220000 ACM, 2022. doi: 10. 00.1455/5246103557886.. 3527886
URL https::////..... / /tt. org 10.
Kenneth Heafield. Kenlm: Faster and smaller language model queries. In Chris Callison-Burch.
Philipp Koehn, Christof Monz, and Omar Zaidan (eds.), Proceedings of the Sixth Workshop
15

---

Published as a conference paper at ICLR 2025
Statistical Machine Translation, ENNLP 2011. Edinburgh, Scotland, UK. July 30- 31. 2011, pp. 187-197. Association for Computational Linguisties, 2011. URL https:/ // ac Lanathho Logy. or..org//////.. //111-1123////1111111
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish, Scaling laws for transfer. CoRR, abs/2102.01294) 2021. URI. /
01293
Abram Hindle, Earl T. Barr, Mark Gabel, Zhendong Su, and Premkumar T. Devanbu. On the naturalness of software. Communn ACM, 52(5):12--331, 2016. doi: 10.1145/2902362. URL
https://doi. wrg..o.g........ 1145/29623622
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training caannpute-optimaa large language models. CoRR, abw.22233.1555.. 2022. doi: 10.48550/ ARXIV.2203.15566 URL httpp:://too....o ooo.ooggrroo..... 4tte///1..111/////
48550 www.aaawwii.......... 2
Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E. Gonzalez, Koushik Sen, and lon Stoica. Lim-assisted code cleaning for training accurate code generators. In The Twelfth International Conference on Learning Reprexentations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenRe- net/ mtt:forrmmmmmmmmFFFF==
view.net, 2024. URLhhttss://ppenreee
Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph Gonzalez, and Ion Stoica. Contrastive code representation learning.
FranciFr Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 5954-5971. Association for Computational Linguistics, 2021. doi:
10.18653/V1/20211 11066633///221.MMMMPPPPPMM WWW.EEENNNLLNNL.E 482. URL https: //doi or g 10. 18653/v1/22211 emnlp-mainn 482
Hui Jiang, Linfeng Song, Yubin Ge, Fandong Meng, Junfeng Yao, and Jinsong Su. An AST structure enhanced decoder for code generation. IEEE ACM Trans. Audio Speech Lang. Process. 30: 468 466.476, 2022a. doi: 10. 119/TTSLL..021.3111330
URL URL/https://doi.orr 1109/
TASLP 2021. 138717
Yuan Jiang, Xiaohong Su, Christoph Treude, and Tiantian Wang. Hierarchical semantic-aware neural code representation. J. Syst. Softw.. 191:111355, 2022b. doi: 10.10166.JS5..022.1133555 URL
https::/doi..rg org 10. 00.0101166 1016 jta.///...ss..../. 2022. 111355
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford. Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020. URL hhtpps:t//// /aaaia.xvvrrr/. www.rrg//////aaeee /0oss/200... 08361
Scohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees to transformers. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pp. 150- 162. IEEE, 2021. doi: 10.1 RB.1009ICCS449922220000000
URL wttpthtt:/// /tttt:::/doi..oo /ttp://ooooooooooooooooooo 10. 1169 / ICSE4390EE.400... 2 00026
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua
Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. httpt
//arxiv. or abs/ 1412.411110 6980
Donald E. Knuth. Literate programming. Comput. J., 27(2):97111, 1984, doi: 10.1093/COMINL/ 1093/ (093/commml/27.2.97
27.2.97. URL http:https///o.o.o
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: 3 TE of permissively licensed source code. Trans. Mach. Learn. Res., 2023, 2023. URI http:///peeeeeeeeeeeeefrrrrrrtre
16

---

Published as a conference paper at ICLR 2025
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace (eds.), Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pp- 61 1-626. ACM, 2023. doi: 10.1145) 3600006.3613165 httppts:://dii. 1t.1.////0.111//30000060 3613165
16. 110..1145/3800 /600000666006...
Marie-Anne Lachaux. Baptiste Rozière, Marc Szafraniec, and Guillaume Lample. DOBF: A deobfuscation pre-training objective for programming languages. In Mare' Aurelio Ran- zato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 14967-14979, 2021. URL /tthttps::/prroeeeinnn neuris neurips wwt.neur....ccppprrrrre.... hater//222///ahase
Jeremy Lacomis, Pengcheng Yin, Edward J. Schwartz, Miltiadis Allamanis, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. DIRE: A neural approach to decompiled identifier naming. In 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA. USA, November 11-15, 2019, pp. 628-639. IEEE, 2019. doi: (0.1109/ASE:2019.0000 990000900444 URL
https: https::/doi.orr/10.. 1109 ASE. 2019. 00064
Dawn J. Lawrie, Christopher Morrell, Henry Feild, and David W. Binkley. What's in a name? A study of identifiers. In 14th International Conference on Program Comprehension (ICPC 2006). 14-16 June 2006, Athens, Greece, pp. 3-12. IEEE Computer Society, 2006. doi: 10.1109/1CPC.2006.511
URL https:t///// //tp://doiiooo orr org g / 10. 1109 / IIPC. 2006. 51
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9. 2022, 2022. http: ni nip. cc hwcpc//aper__aee fi filerfffleeeeeeeee paper fapeeppppppeellppppppperrrrrr 2622 /hahh/ htmll
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In ACL (1), pp. 8424 8445. Association for Computational Linguistics, 2022.
Jia Li, Ge Li, Yongming Li, and Zhi Jin. Structured chain-of thouugh prompting for code generation. ACM Transactions on Software Engineering and Methodology, 2023a. URL TtRLhttps:// /api Semanticscholas org/ Corpus 258615421
Raymond Li, Loubna Ben Allal, Yangtian Zi Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy Jason T. Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas. Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lec, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavin Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! Trans, Mach. Learn. Res., 2023, 2023b. URL 1d=KoFOg41haE
met/foruu
Yiyang Li, Hongqiu Wu, and Hai Zhao. Semantic-preserve adversarial code comprehension. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus,
17

---

Published as a conference paper at ICLR 2025
Francis Bond, and Seung-Hoon Na (eds.), Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Cyeongju, Republic of Korea, October 12-17, 2022, pp. 3017-3028. International Committee on Computational Linguistics, 2022a. URL https::/aclanthoot wwwpo/oogg.gggg22..o.o/// col.gg- 267
Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman. Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling. Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang. Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu. and Oriol Vinyals. Competition-leeel code generation with alphacode. CoRR, wttp///22..081111 2022b, doi: 10.48550/ARXI.2203.0777 URL https:://oii..oo. org 10 . 48550/ attp550//aaivv.. 210.3
Zongjie Li, Pingchuan Ma. Huaijin Wang, Shuai Wang, Qiyi Tang. Sen Nie, and Shi Wu. Unleashing the power of compiler intermediate representation to enhance neural program embeddings. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, ACM, 2022c. doi: 10.1145/35100033351002 URL https://don org::://doooor////////// .og. 10. 1145 3510003. 3510217
PA. USA, May 25-27, 2022, pp.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, Hao Zhang, Hanwei Xu, Hao Yang. Haowei Zhang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong. Kaige Gao. Kang Guan, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao. Liyue Zhang. Meng Li, Miaojun Wang, Mingehuan Zhang. Minghua Zhang, Minghui Tang. Mingming Li, Ning Tian, panpan Huang, Peiyi Wang. Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. Chen, R. I.. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma. Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng. Tao Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang. X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, and Xiaowen Sun. Deepseek- v2: A strong, economical, and efficient mixture-of- experts language model. CORR, abs/2405 04444,, 2024. doi: 1048550/ARXIV.2455..00 044344 URL https://doi.org 10 48550 arxiv 2405 04434
Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan. Code execution with pre-trained language models. In Anna Rogers, Jordan I.. Boyd-Graber, and Naoaki Okazaki (eds.). Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 4984-495 4999, Association for Computational Linguistics, 2023a. doi: 10. 101118/5/1/2222 FINDINGS- //doi org 10. 11w53/v1/1/// / find acl
Utt/tt :
Jiate Liu, Yigin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. RLTF: reinforcement learning from unit test feedback. Trans. Mach. Learn. Res., 2023, 2023b. URL https://openrev/
met/forumm (ol.roorruuuhhho jd=
Shangqing Liu, Xiaofei Xie, Jing Kai Siow, Lei Ma. Guozhu Meng, and Yang Liu. Graphsearchnet: Enhancing gnns via capturing global dependencies for semantic code search. IEEE Trans. Software Fng., 29(4) 2855, 2023c. doi: 10.1109/TSE.2022.33333 10. 1109 1101//TE...... TSE. 2622. 3233901
URL httphttss///do.rgg
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- -aiiieri Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Ni Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He. Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, and et al. Starcoder 2 and the stack v2: The next generation. CoRR, Ab1/2440.199133, 2024. doi: 10.48550/ARXIV.2400.2 19173, URL hitts:://do .rg 10tp:/.0..855500 www.:/55//axivv..o http:/aawxvv...2...22 19173
18

---

Published as a conference paper at ICLR 2025
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL /201/
neur ottpp//ddeeeeeeereerreoocp//p//p www.seeeee--bbtttt-t---e-------e ttah::ccaa550ffaa533008555466665
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna. Austria, May 7-11, 2024. OpenReview.net, 2024. forum?1d=UnnWSIIgke.
net/
Haoyu Ma, Xinjie Ma, Weijie Liu, Zhipeng Huang. Debin Gao, and Chunfu Jia. Control flow obfuscation using neural network to fight concolic testing. In Jing Tian, Jiwu Jing, and Mud- hakar Srivatsa (eds.), International Conference on Security and Privacy in Communication Net- works- 10th International ICST Conference, SecureComm 2014, Beijing, China, September 24-26, 2014, Revised Selected Papers, Part I, volume 152 of Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecomnunications Engineering, pp. 287-304. Springer, 2014. doi:
0010//9788-3319-23888 _1_ URL htLLhtts::///// org 10. 1007/ 378-3-319-23829-8
Wei Ma, Mengjie Zhao, Xiaofei Xie, Qiang Hu, Shangqing Liu, Jiexin Zhang, Wenhan Wang, and Yang Liu. Unveiling code pre-trained models: Investigating syntax and semantics capacities. ACM Transactions on Software Engineering and Methodology, 2022. URL https://api
semanticscholars org/ CorppusD oot./aarrr.rr/ooouuuII.. 258556996
Parvez Mahbub, Ohiduzzaman Shuvo, and Mohammad Masudur Rahman. Explaining software bugs leveraging code structures in neural machine translation. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 640-652. IEEE, 2023. doi: 10. 1111009CE/881199000000 2.0446669.202.00066. 1109 ICSEIC6155511 2023 00063
10 /ttthttts://doo
Pratyush Maini, Skyler Seto, Richard He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACI. 2024, Bangkok, Thailand, August I 1-16, 2024, pp. 14044-14072. Association for Computational I. Linguistics, 2024. doi: 10.18653/V WTA.1///222AAL-LLNGG.CO
URL https:://ooooooo or org. 10 18653 /v122222/
acl- - long. 757
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzich Fadace, and Sara Hooker. When less is more: Investigating data pruning for pretraining lims at scale. CoRR, abs/2309.00564, 2023.
Mayank Mishra, Matt Stallone, Gaoyuan Zhang. Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, Manish Sethi, Xuan- Hong Dang. Pengyuan Li. Kun-Lung Wu, Syed Zawad, Andrew Coleman, Matthew White, Mark Lewis, Raju Pavuluri, Yan Koyfman, Boris Lublinsky, Maximilien de Bayser, Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Yi Zhou, Chris Johnson, Aanchal Goyal, Hima Patel, S. Yousaf Shah, Petros Zerfos, Heiko Ludwig, Asim Munawar, Maxwell Crouse, Pavan Kapanipathi, Shweta Salaria, Bob Calio, Sophia Wen, Seetharami Seelam, Brian Belgodere, Carlos A. Fonseca, Amith Singhee, Nirmit Desai, David D. Cox, Ruchir Puri, and Rameswar Panda. Granite code models: A family of open foundation models for code intelligence. CoRR, abs/ 044244, 2024. doi: 10. 48550/ARXIV.2405.000 04324. URL https: /oi. org/ 48556 ar 2465. 04324
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree structures for programming language processing. In Dale Schuurmans and Michael P. Wellman (eds.), Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016,
19

---

Published as a conference paper at ICLR 2025
Phoenix, Arizona, USA. pp. 1287-1293. AAAI Press, 2016. doi: 141100//AAAALV3011...00
URL Inttps:///oil . .rg 16. 1609/ / aaai. /3011.10139
Aaron Mueller, Albert Webson, Jackson Petty, and Tal Linzen. In-context learning generalizes, but not always robustly: The case of syntax. In Kevin Duh, Helena Gómez-Adomo, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 4761-4779. Association for Computational Linguistics, 2024. doi: 18653/v1/2024 naacl- Lon.eooooaaaaaanoooooo. 267
URL httptttt///.oooooo /rgp////oooor//////
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo. Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz. Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URLhttp://paperrsttt nips cc ctcpccpppppaeeeeeceeecccp files/ paper f /023// / htmll
Vijayaraghavan Murali, Chandra Shekhar Maddila, Imad Ahmad, Michael Bolin, Daniel Cheng, Negar Ghorbani, Renuka Fernandez, Nachiappan Nagappan, and Peter C. Rigby. Ai-assisted code authoring scale: Fine-tuning, deploying, and mixed methods evaluation. Proc. ACM Softw. Eng., (F$E):1006-1000 2024. doi: 10.1 14/364446437 URL httpst:s:: /ttp///oi.ooo org 10/0 1145/ 3643774
Shounak Naik, Rajaswa Patil, Swati Agarwal, and Veeky Baths. Probing semantic grounding in language models of code with representational similarity analysis. In Weitong Chen, Lina Yao, Taotao Cai. Shirui Pan, Tao Shen, and Xue Li (eds.), Advanced Data Mining and Ap- plications 18th International Conference, ADMA 2022, Brisbane, QLD, Australia, Novem- ber 28-30, 2022, Proceedings, Part II. volume 13726 of Lecture Notes Computer Sci- ence, pp. 395 406. Springer, 2022. doi: 10. 1011007/978-3303-22211777 //doi . or g 10. 1007 978
29. URL https
12737-2211782.. 8_29
Aravind Ashok Nair, Avijit Roy, and Karl Meinke. funegnn: graph neural network approach to program similarity. In Maria Teresa Baldassarre, Filippo Lanubile, Marcos Kalinowski, and Federica Sarro (eds.), ESEM '20: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, Bari, Italy, October 5-7, 2020, pp. 10:1-10:11 11, ACM, 2020. doi: 10.1145/388249444410077 Uttsttpps: //t.//oooooo/oo//oooo org 10. 1145/ 1ee./11111/////// 3382494.2... 3410675
Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. Learning to generate pseudo-code from source code using statistical machine translation (T). In Myra B. Cohen, Lars Grunske, and Michael Whalen (eds.), 30th JEEE/ACM International Conference on Automated Software Engineering, ASE 2015, Lincoln, NE, USA, November 9-13, 2015, pp. 574-584. IEEE Computer Society, 2015. doi: 10. 1..1109/EE..015.356 BRL/https: /ooii oogg
100 10. 1109 ASE. 2015.36 36
David N. Palacio, Alejandro Velasco, Daniel Rodríguez-Cárdenns, Kevin Moran, and Denys Poshyvanyk. Evaluating and explaining large language models for code using syntactic struc- tures. CoRR, 0tlss2308.03877. 2023. doi: 10.45550/ARXI..2300.0333
URL h
//doi. org 48550//4455/arra Xrviv 2308. 03873
Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. Lost in translation: A study of bugs introduced by large language models while translating code. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pp. 82: 82: ACM, 2024. doi: (0.1455355975033639222 URL 3639226
https: http:://doooorrroo org. 10 1145 359
Youngmi Park, Ahjeong Park, and Chulyun Kim. trsiffraanform::: Transformer- code comment generation with aligned lexical and syntactic information. IEEE Access, 1 11:39037-39047, 2023, doi: 10. /CO///CCESSS.2233266622 URLhhttp:://do1 org/ 10. 1109 AACp:/ACCESS..22 2023
3268638
20

---

Published as a conference paper at ICLR 2025
Julian Parsert and Elizabeth Polgreen. Reinforcement learning and data Kata-generation for syntax- -uiddd synthesis. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Eits Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty- Conference on Innovaiie Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational A Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 0010060-16677 AAAI Press, 2024, doi: 10.1609/AAAI 13811. 28933. URL WRLLhttss///dioooo 1669 1609/
aaai w3w.9 28938
Indraneil Paul, Goran Glavas, and Iryna Gurevych. Ircoder: Intermediate representations make language models robust multilingual code generators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11- 16, 2024, pp. 15023-15041. Association for Computational Linguisties, 2024. URL https; org/ 2024.022222 ac acl-long. 802
Kexin Pei, Weichen Li, Qirui Jin, Shuyang Liu, Scott Geng, Lorenzo Cavallaro, Junfeng Yang, and Suman Jana. Exploiting code symmetries for learning program semantics. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
/tttps::/openrreveeww poppenreviee.nee/oorumm www./eeett/tnieeoeeoeeeeo (ottf/orrrm??ddloggLLtt
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobcidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data only. In NeurIPS, 2023.
Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. CoRR, abs/2406. 17557, 2024. doi: 10. 10.48550//AXXI.24066.11 17557. URL https://doi.org/
48550 arxiv. 2406. 17557
Constantin Cezar Petrescu, Sam Smith, Rafail Giavrimis, and Santanu Kumar Dash. Do names echo semantics? A large-scale study of identifiers used in c++ named casts, J. Syst. Softw., 202:111693, 2023. doi: 10.10166J.JS..222..1 11693. URL WRL/https://doot .og. /10 1016/j . jss. 111693
Hung Phan and Ali Jannesari. Evaluating and optimizing the effectiveness of neural machine translation in supporting code retrieval models: study on the CAT benchmark. In Ingo Frommholz, Frank Hopfgartner, Mark Lee, Michael Oakes, Mounia Lalmas, Min Zhang, and Rodrygo L. T. Santos (eds.), Proceedings of the 32nd ACM International Conference on In- formation and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023, pp. 2055-2066 ACM, 2023, doi: 10.1145/3583770036146666 URL https //doi .o.orr//// 10. 1145 3583780 3614869
Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulic, Sebastian Ruder, Iryna Gurevych, and Jonas Pfeiffer. Adapters: A unified library for parameter--fffciiett and modular transfer learning. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 System Demonstrations, Singapore, December 6-10, 2023, pp. 149-160. Associ- ation for Computational Linguistics, 2023. doi: 10. 10.18653/V1/200 B112223.ENNL---EE /ttps:://oi..oo or 100 11w5116661v
URL
www.p-deno....ooo 13
Raghu Prabhakar, Ram Sivaramakrishnan. Darshan Gandhi, Yun Du, Mingran Wang, Xiangyu Song, Kejie Zhang, Tianren Gao, Angela Wang. Karen Li, Yongning Sheng, Joshua Brot, Denis Sokolov, Apury Vivek, Calvin Leung. Arjun Sabnis, Jiayu Bai, Tuowen Zhao, Mark Gottscho, David Jackson, Mark Luttrell, Manish K. Shah, Edison Chen, Kaizhao Liang, Swayambhoo Jain, Urmish Thakker, Dawei Huang, Sumti Jairath, Kevin J. Brown, and Kunle Olukotun. Sambanova SN40L: scaling the AI memory wall with dataflow and composition of experts. CoRR, abs/2405.0751 075518, 2024. doi: 10.48550/ARXIV24000. 2405. 07518
URL httpthtts///t http:/ddoo.ooggogg
10 0001//104880000 48550/
Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R. Choudhury. Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. Codenet: A
21

---

Published as a conference paper at ICLR 2025
large-scale Al for code dataset for learning diversity of coding tasks. In Joaquin Vanschoren and Sai- Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NewrIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL wwwcheakks httpss///ftttsstteeeeeemmaaaspse /ppppp/reoeeeeensssssseissiooopp
nwwrips.rccccccc wwwpcc/pppppp/eeeeeeeeeeeeeeeeee pappr/ / hast://5frceeo77677aaaaeeefsfsss wwtrrr-bbbbtttttateeeeeeeaaaaaee round2. round..hmmm
Md. Rafiqul Islam Rabin, Nghi D. Q. Bui, Ke Wang, Yijun Yu, Lingxiao Jiang, and Mohammad Amin Alipour. On the generalizability of neural program models with respect semantic-preservinn program transformations. Inf. Softw. Techmol.. 135: 106552, 2021. doi: INFSSF2/.. 106552. WRLptsts:://oi.
or org 1 http:/..nnffssee wnf.sffff....o 2021. 106552
Md. Rafiqul Islam Rabin, Aftab Hussain, Mohammad Amin Alipour, and Vincent J. Hellendoorn. Memorization and generalization neural code intelligence models. Inf. Softw. Technol. 153: 107066, 2023. doi: (0.1016/.NNFSOF22222 107066. UtL/https: //tttp:://do..oo org / 10. 1016/] infsof. 2022 107966
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:5 2020. URL WRRpsttss://mrr...
papers/v21/20- 074. html
Musfiqur Rahman, Dharani Palani, and Peter C. Rigby. Natural software revisited. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (eds.), Proceedings of the s1 International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pp. 37-48. IEEE / ACM, 2019, doi: 10. 10.1109/CCSE.22 2019.000000000 URI. (ttlshttps: //doi org/ 10. 1109 ICSE 2019. 00022
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. Christine Cuicchi, Irene Qualters, and William Kramer (eds.). Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, pp. 20. IEEE/ACM, 2020. doi: N0.11099$C41445.222000000 //doi org 10. 1119//C414005 2620. 00024
URL https
Baishakhi Ray, Vincent J. Hellendoom, Saheel Godhane, Zhaopeng Tu, Alberto Bacchelli, and Premkumar T. Devanbu. On the "naturalness" of buggy code. In Laura K. Dillon, Willem Visser, and Laurie A. Williams (eds.), Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016, pp. 428-439, ACM, 2016. doi: 10.1 (0.1145//88471.2222 2884488 URL :/ttpp:://do... /oo.i or org/ 10. 1 1145 20817818888888 2884848
Veselin Raychev, Martin T. Vechev, and Andreas Krause. Predicting program properties from 'big code'. Commun. ACM, 62(3):99-107, 2019. doi: 10. 10.1144/3306204, URL https://dooi.rgg
10. 11.4/33336660
Houxing Ren, Mingjie Zhan, Zhongyuan Wu, Aojun Zhou, Junting Pan, and Hongsheng Li. Re- flectioncoder: Learning from reflection sequence for enhanced one-off code generation. CoRR. 010024055170557 2024. doi: 10. 10448550AARXI.24455.000
URL httpp:://doi.orgg:
48550/arXiv. 2405 17057
Baptiste Rozière, Maric-Ama Lachaux. Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, neurips. cttpspcrcppppprrrr
December 6-12, 2020, virtual, 2020. URL http:httts://poceeettte http:////222..111cc133eeeeeeeee.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usumier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code. CoRR, 1bb/2238..12900..... 2023. doi: 10. 48550/ARXIV 12550. 12950. httphtt org/ 10 48550 arxiv. 2wt.ii..338... 2308 12950
22

---

Published as a conference paper at ICLR 2025
Iman Saberi and Fatemeh H. Fard. Model Model-agnostic syntactical information for pre-trained pro- gramming language models. In 20th IEEE/ACM International Conference on Mining Software Repositories, MSR 2023, Melbourne, Australia, May 15-16, 2023, pp- 183-193, IEEE, 2023. doi: 10. (0099M/SR59973..222...00 00036
URL. wttpt:tts///.o....t.
/ottps://doi... /ttp:/ooi.oo orgg 10. 1109
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
URL https: http:ss///peeeeeeee..eeeeeeeeeee hatpeeerreiiee.ne/ffrrmmmm
Sebastian Schrittwieser, Stefan Katzenbeisser, Johannes Kinder, Georg Merzdovnik, and Edgar R. Weippl. Protecting software through obfuscation: Can it keep pace with progress in code analysis? ACM Comput. Surv., (9(1::::144: 1::37, 2016. doi: 10.1 14/1245288 httttss///dooi org/
10. 1145/2886012
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. CoRR, abx/2407 086088 2024. doi: 10448550/ARXXV.240..0666 48550/arXiv.. 2407 08608
URL httpptt://ooi.rrr////
Yiheng Shen, Xiaolin Ju, Xiang Chen, and Guang Yang. Bash comment generation via data augmentation and semantic-aware codebert. Autom, Softw. Eng., 31)3)30, 2024. doi: 10. 10.1007/ S10515-024-00431-2 URL
http::::/doooorr///// 1007 s10515 024-00431-2
Chaochen Shi, Borui Cai, Yao Zhao, Longxiang Gao, Keshav Sood, and Yong Xiang. Coss: Leverag- ing statement semantics for code summarization. IEEE Trans. Software Eng.. 496))3472-3486 2023. doi: 10.11099/SEE22233325660 3256362
URL https: or 10. /9019TTT...223
Kensen Shi, Deniz Altinbüken, Saswat Anand, Mihai Christodorescu, Katja Grünwedel, Alexa Koenings, Sai Naidu, Anurag Pathak, Mare Rasi, Fredde Ribeiro, Brandon Ruffin, Siddhant Sanyam, Maxim Tabachnyk, Sara Toth, Roy Tu, Tobias Welp, Pengcheng Yin, Manzil Zaheer, Satish Chandra, and Charles Sutton. Natural language outlines for code: Literate programming in the LLM era. CoRR, abs/2408 04820. 2024. doi: 00.48550/ARXV.240000 048200. URL https 7/doi , or ooo.orgg/////oo 10. 48550 arxiv 2408 04820
Yucen Shi, Ying Yin, Zhengkui Wang, David Lo, Tao Zhang, Xin Xia. Yuhai Zhao, and Bowen Xu. How to better utilize code graphs in semantic code search? In Abhik Roychoudhury, Cristian Cadar, and Miryung Kim (eds.), Proceedings the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, pp. 722-733. ACM, 2022. doi: 10.1 1455/ 3540250.3549087 URL /nttpss://oii: org 16.
3549087
Mohammad Shocybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Ing Training multi-billion parameter language models using model parallelism. CoRR, (ww11999000553.. 2019. URL BRL/http:: /ttttp:://axiv...t/t org awwoorg//////aaaeeaeaaaaeaaeeeee 1909, 08053
Parshin Shojace, Aneesh Jain, Sindhu Tipimeni,, and Chandan K. Reddy. Exccution- code generation using deep reinforcement learning. Trans. Mach. Learn. Res., 2023, 2023. URL
https://openrewe
net/ www.wourrrrrrme www.:n//orrummm//0XxeeeeeeCe dd=
André Silva, João Ferreira, He Ye, and Martin Monperrus. MUFIN: improving neural repair
models with back-translation. CoRR, abs/2330.00331. 2023. doi: 1044850AARXIV.300..000
URL https::///oiooooo/ http://ooooorr/1///// /t5p:///00o1110011144555///////
48550 / 2304. 02301
Philippe Skolka, Cristian- Staicu, and Michael Pradel. Anything hide? studying minified and obfuscated code in the web. In Ling Liu, Ryen W. White, Amin Mantrach, Fabrizio Silvestri, Julian J. McAuley, Ricardo Baeza- Yates, and Leila Zia (eds.). The World Wide Web Conference, WWW 2019, San Francisco, CA. USA, May 13-17, 2019, pp. 1735-1746. 1746. ACM, 2019. doi: 10.1 /4.114433088558831333222 URL URI.httss: or 10 11w.1111111113338188888888 3313752
23

---

Published as a conference paper at ICLR 2025
Tao Sun, Linzheng Chai, Jian Yang. Yuwei Yin, Hongcheng Guo, Jiaheng Liu, Bing Wang, Liqun Yang. and Zhoujun Li. Unicoder: Scaling code large language model via universal code. In Lun-Wa Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), 2024, Bangkok, Thailand, August 11-16, 2024, pp. 1812 1824. Association for Computational Linguistics, 2024. URL https: http::://acaanthououo logy. grg/ 2024. ac
long. 100
Marc Szafraniec, Baptiste Rozière, Hugh Leather, Patrick Labatut, François Charton, and Gabriel Synnaeve. Code translation with compiler representations. The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
net www.:neetforr???0 ft Xomuunde=xXUe
Armstrong A. Takang, Penny A. Grubh, and Robert D. Macredie. The effects of comments and identifier names on program comprchensibility: an experimental investigatiion, Program, Lang., (3):143-167, 1996. html
dcs kc http://e.............o/////////. abs
Wannita Takerngsaksiri, Chakkrit Tantithamthavom, and Mhan-Faagg& Li. Syntax-aware th-the-fly code completion. Inf. Softw. Technol., 165:107336, 2024. doi: 0..01660IFFSOOF.023..73666 URL http:: //cpi. org 10. 1016 http:/jiiinffoooo infsof. wtt.ssff..22....0 107336
Tianhua Tao, Junbo Li, Bowen Tan, Hongyi Wang, William Marshall, Bhargav M Kanakiya, Joel Hestness, Natalia Vassilieva, Zhiqiang Shen, Eric Xing, and Zhengzhong Liu. Crystal: Illumi- nating LLM abilities on language and code. In First Conference on Language Modeling, 2024.
URL https: //tp://////eeevveee ew net/ forum? dd==nkk
Suresh Thummalapenta and Tao Xie. Alattin: mining alternative patterns for defect detection. Autom. Softw: Eng., 1803-4):93-323, 2011. doi: (011007/110515-01-0000-- https:
//doi.org/ 10. 010.000rr//....0011111111111---1
Sindhu Tipirneni, Ming Zhu, and Chandan K. Reddy. Structcoder: Structure-aware transformer for
code generation. ACM Trans. Knowl. Discov. Data, 18(3):70:1-70:20, 2024. doi: 10.1 10.1145/3636400 URL https::///oo org/
oro/.o....... 10. 1145/3636430
Sergey Troshin and Nadezhda Chirkova. Probing pretrained models of source codes. In Jasmijn Bastings, Yonatan Belinkov, Yanai Flazar, Dieuwke Hupkes, Naomi Saphra, and Sarah Wiegreffe (eds.), Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, BlackboxNLPGG EMNLP 2022, Abu Dhabi, United Arab Emirates (Hybrid), December 2022, pp. 371-383. Association for Computational Linguistics, 2022. doi: 10. BBBEV///2222BBBACKBBO 1.31. URL https::/toi.i.... org org. 10. 18653/v1/2002 b SLackboxnlp-1.33
Alejandro Velasco, David N. Palacio, Daniel syntactic capabilities are statistically learned by masked language models for code? In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, NIER @ ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pp. 72-76. ACM, 2024. 10. 1141/311439999 363947634... 003.3376.6397676..33333 3639768
Cárdenas, and Denys Poshyvanyk. Which
doi: 10. 121114//66947663369777
URL hnttps:///di.oo.t
Siddhant Waghjale, Vishruth Veerendranath, Zora Zhiruo Wang, and Daniel Fried. ECCO: can we improve model-generater code efficiency without sacrificing functional correctness? CoRR, abs/2407. 000407..40444 14044, 2024. doi: 10448550/RXIV..4.... 14044. URL httts://///::tttrrr 48550/arXiv. 2407 14044
Changhan Wang, Kyunghyun Cho, and Jiatao Gu. Neural machine translation with byte-level subwords. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020. The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 12, 2020, pp. 1154-9160.. AAAI Press, 2020. doi: 10.1609/AAAI. V34105 6451. URL hwwp::ttp////e / org 10. 1609 aaai v34105.c.. 645111
Deze Wang, Zhouyang Jia, Shanshan Li, Yue Yu, Yun Xiong, Wei Dong, and Xiangke Liao. Bridging pre-trained models and downstream tasks for source code understanding. In 44th IEFF/ACM 44th
24

---

Published as a conference paper at ICLR 2025
International Conference Software Engineering, ICSE 2022, Pittsburgh, PA. USA. May 25-27, 2022. pp. 287-298 298. ACM, 2022. doi: 10.114575510003331100022 URL https: /ttp:///oooorggoooo 10 19.1145/3510000 .50.:/3335500033000060 3510062
Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song. Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. Planning in natural language improves 1lm search for code generation. CoRR, abs/2409. 037333, 2024a. URL http:: //api semanticschol lar org/ Co:pp/oI::
Huaijin Wang. Pingchuan Ma, Shuai Wang, Qiyi Tang, Sen Nie, and Shi Wu. sem2vec: Semantics- aware assembly tracelet embedding, ACM Trans. Softw. Eng. Methodol. 32(4):90:1-90:34 2023a. doi: 10.1145/3569933. URL ItRl:ttps:/ //tp::://doii. htt.ooo.orrglg 10. 11.15//35911
Ke Wang. Learning scalable and precise representationn program semantics. CoRR, abs/1905.05251, 2019.
Ke Wang and Zhendong Su. Learning blended, precise semantic program embeddings. CoRR, abs/1907.02116,, 2019. URL httt:://aarxii.orr///////1000...
Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang. Zijian Wang. Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. Recode: Robustness evaluation of code generation models. In Anna Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9- 14, 2023, pp. 13818- 11818813322 13843. Association for Computational Linguisties, 2023b. doi: 2023. acl longall-.oo.oooooooeoooooooll 773.
Rogers, Jordan L.
773. URL http:: 100.///o00..00111118833..63 18653/v..
Song Wang, Taiyue Liu, and Lin Tan. Automatically learning semantic features for defect prediction.
In Laura K. Dillon, Willem Visser, and Laurie A. Williams (eds.). Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016, pp. 297-308. ACM, 2016. doi: 10.1145/2884781.288888 URL hhttps://det or oog. 10
1145 2145/2884788 00044788.2884804. 2884804
Xin Wang, Yasheng Wang, Fei Mi, Pingyi Zhou, Yao Wan, Xiao Liu, Li, Hao Wu, Jin Liu, and Xin Jiang. Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation. CoRR, a08/2108..n 10.56550.85 2021a. URL /aattss//api.... semanticcmmmtmcccecoee lar org orgg 237450712
CorpusID Dar.ooooou...g//
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier-aware unified pre- trained encoder-decoder models for code understanding and generation. In
Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tan Yih (eds.). Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 8696-8708. Association for Computational Wttp:///2222..EN....M.MMM..
Linguistics, 2021b. doi: 10. 10.
URL https: //doi. org/
emn Lp-main. 685
Zhijie Wang, Zijie Zhou, Da Song, Yuheng Huang, Shengmai Chen, Lei Ma, and Tianyi Zhang. Where do large language models fail when generating code? CoRR, abs/2406.08731, 087331, 2024b. doi: 10. (0.4555//ARXII.4466....
10. URL https:t//oo.o..... .tt.://0...//////iii... 2406
08731
Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. Code generation as dual task of code summarization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' d'Ache-BBuc, Emily B. Fox, and Roman Gamett (eds.), Advances in Neural In- formation Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 6559 6569, 2019. URL http:: htt3://s...oo.uuuunptp////ooeee/ seaeeddinggneuuipss.. wwtpc/ppppe/e////aaeeeeeee ee22abseraffs4444offffefffff---e
25

---

Published as a conference paper at ICLR 2025
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Em- powering code generation with oss-instruct. In Forty-firs Interaational Conference on Ma- chine Learning. ICML 2024. Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL
net/ forum? dd=xxeooBidXx3x htttps://openree oe
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models. In The Twelfth International Conference on Leurning Representations, ICLR 2024, Vienna, net forum?
Austria, May 7-11, 2024. OpenReview.net, 2024. BtLlhttps///opeereeeee idd-wFoJccpp8
Adam Yala, Victor Quach, Homa Esfahanizadeh, Rafael G. D'Oliveira, Ken R. Duffy, Muriel Médard, Tommi S. Jaakkola, and Regina Barzilay. Syfer: Neural obfuscation for private data release. CoRR, 0tns/2001.12406. 2022. WRRLhttps:
http://bb//22......// 12406
Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. Modeling and discovering vul- nerabilities with code property graphs. In 2014 IEEE Symposium on Security and Privacy, SP 2014, Berkeley, CA, USA, May 18-21, 2014, pp. 590-604 IEEE Computer Society, 2014. doi: 10.1109/SP.201.44. URL https:::: /wt./oiioooo org 10. 1101/ SP 2014. .44
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Kegin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge. Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yun- fei Chu, Yuqiong Liu, Zeyu Cui. Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. CoRR, abs/2407.10677 2024. doi: (0.4850/ARXXV.240..066)) URL https://do1 .rg. 10. 48550 ar 2407 10671
Guang Yang, Yu Zhou, Xiang Chen, Xiangyu Zhang, Yiran Xu, Tingting Han, and Taolue Chen. A syntax-guided multi-task learning approach for turducke--style code generation. Empir. Softw. Eng., 28(6): 141, 2023a. doi: 10. 10.1007/S1166 10664-023-10372-11 URL 1007 0107/411644--11 023 10372-1
Guang Yang, Yu Zhou, Xiangyu Zhang. Xiang Chen, Tingting Han, and Taolue Chen. Assessing and improving syntactic adversarial robustness of pre-trained models for code translation. CoRR, abs/23 18587, 2023b. doi: 10.48550/ARXIV.330000 18587. URL /ictps://doi.ors 48550/arXiv. 18587
Jia Yang, Cai Fu, Fengyang Deng, Ming Wen, Xiaowei Guo, and Chuanhao Wan. Toward interpretable graph tensor convolution neural network for code semantics embedding. ACM Trans. Softw. Eng. Methodol, 32:5): 2023c. doi: 10. 145/3582574 URL http:httss//do..rrgggg 0144/3552274
Weizhe Yuan and Pengfei Liu. restructured pre-training. CoRR, abs/2206.11147, 2022. doi: 10. 48550/ARXIV.2206 11147. URL htthht:t: / 48550/8500/00 http://awwvv222..111.11 2206.
Daoguang Zan, Bei Chen, Dejian Yang. Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang. Weizhu Chen, and Jian-Guang Lou. CERT: continual pre- pre-traine training on sketches for library-oriented code generation. In Luc De Raedt (ed.), Proceedings of the Thisty-First International Joint Conference Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pp. 2369 2375. ijcai.org, 2022. doi: 10.24963/UCAA.2222222 URL / doi org/ 10. 24963 ijcai 2022/329
Jingfeng Zhang, Haiwen Hong, Yin Zhang, Yao Wan, Ye Liu, and Yulei Sui. Disentangled code representation learning for multiple programming languages. In Chengqing Zong, Fei Xia. Wenjie Li. and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/JCNNP 2021 of Findings of ACL, pp. 4454 4466. Association for Computational Linguistics, 2021a. doi: or org/ 10. 18653/
10.18633/112221.IINDDD MADDDNNG--CLL911.. URL htt/https:// /
Findings-acll.91
26

---

Published as a conference paper at ICLR 2025
Kechi Zhang, Wenhan Wang, Huangzhao Zhang, Ge Li, and Zhi Jin. Learning to represent pro- grams with heterogeneous graphs. In Ayushi Rastogi, Rosalia Tufano, Gabriele Bavota, Ven- Arnaudooa,, and Sonia Haidue (eds.), Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, ICPC 2022, Virtual Event, May 16-17, 2022 pp. 378- URL
389. ACM, 2022. doi: 10.1 10.1445355241103557990. 88524610.5222 3527905
Xiaolu Zhang. Frank Breitinger, Engelbert Luechinger. and Stephen O' O'Shaughnessy. Android application forensics: A survey of obfuscation, obfuscation detection and deobfuscation techniques and their impact on investigations. Digit. Investig.. 39:301285, 2021b. doi: 10. 1..101/J.FSIII.2000 301285. URL
www.doo.orrooooooooooooooooooo.. 10. 1016/]. fsidi. 2021 301285
Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, Christopher A. Choquette-Chode Jingyue Shen, Joe Kelley, Kshitij Bansal, Luke Vilnis, Mateo Wirth, Paul Michel, Peter Choy, Pratik Joshi, Ravin Kumar, Sarmad Hashmi, Shubham Agrawal, Zhitao Gong, Jane Fine, Tris Warkentin, Ale Jakse Hartman, Bin Ni, Kathy Korevec, Kelly Schaefer, and Scott Huffman. Codegemma: Open code models based on gemma. CoRR, abs/2406. I 1409, 2024. doi: 4..8500/ARXV.2246.1111 1409. URL https:: :/ orgg A0w44855550XX 2406 11469
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 12834 12859, Association for Computational Linguistics, 2024. URL https:// ac aclantho www.thoooog..org22222222 findings-n-iii acl.
Tong Zhou, Shaolei Ren, and Xiaolin Xu. Obfunas: A neural architecture search-based DNN obfus- cation approach. In Tulika Mitra, Evangeline Y. Young. and Jinjun Xiong (eds.), Proceedings of the 41st IEEE/ACM International Conference on Computer-Aides Design, ICCAD 2022, San Diego California, USA, 30 October 2022 3 November 2022, pp. 81: 11::9. ACM, 2022. doi: 10. 11114/5508352.5549.22
URL https: /tt.//oi.ggg///// 10. 1145 10..0..1111111810 35083520835552 3549429
Yaqin Zhou, Shangqing Liu, Jing Kai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnera- bility identification by learning comprehensive program semantics via graph neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 10197-10207, 2019, URL httptttpss////ceeenngn.....e neuripss.....i.................. /C/ 2000r///04//hs111122644444661113
Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipimeni,, and Chandan K. Reddy. Xlcost: A benchmark dataset for cross-lingual code intelligence. CoRR, (18474 2022a. doi: 10.45550/RRXI12206008841
URL https: //ooi org 10. 48550 arxiv
2206.088 08474
Ming Zhu, Karthik Suresh, and Chandan K. Reddy. Multilingual code snippets training for program translation. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Sympo- sium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 URL
March 1, 2022, pp. 1 11783 11790. AAAI Press, 2022b. doi: 10.1109//AAA..3361..2.. / orr.oo.orrg// 1669 aaai a ww6.10.6100.... 21434
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng. Jiawei Liu, Qian Liu, Zijian Wang, David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Hanm de Vries, and Leandro von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. CoRR, abs/2406. 15877, 2024. doi: (0.48550/ARXIV2444 15877. URL
27

---

Published as a conference paper at ICLR 2025
A MODEL ARCHITECTURE & TRAINING DETAILS
Attribute
ObscuraCoder
Training Attributes
Scheduler Type
Cosine
Scheduler Marnup Prop.
0.85
Optimizer Type
Adanti
LB
5e-4
Terminal
2.5e-5
Bota
(0.9, 0.95)
1e-8
Gradfent Clipping
1.0
Decay
8.1
Deepspeed varfant
Stage-2
Model Batatype
bfloat15
Softmax Datatype
float32
FP32 AllRaduce
True
Pipeline Parallel
False
Activation Checkpointing
Global Batch Size
256
Training
528880
Contiguous Gradients
True
Round Robin Gradients
True
Scattered Reduce
Partitionad Allgother
True
Architecture Attributes
Type
Norm Epsilon
1e-6
Enbed. Type
Retary
Pos. Inbed. Period
1080080
Pas. Enbed. Prop-
1.9
Sequence Length
2048
MLP Type
Llana
Activation Function
SiLU
Weight Tying
False
Attention Vartant
Flash Attontion-2
Tokenfzer Varfant
8PE
Tokemizer Vocab, Size
49152
OK Norm
False
Layer
QKV Heads Count
8
12
16
24
Hidden Size
1536
2048
Internediate Size
2816
4096
5632
8192
Non Enbed. Parans.
264N
415M
1128M
2E43N
Table 4: Training and architectural attributes of the ObscuraCoder suite of models.
B DATASET DETAILS
B.1 OBSCURAX COLLECTION
Language
Sample Count
Taken Count
Original
Obfuscated
Total
5, 5,986,186 186
163,
10,962, 297, 717
22, 58,,
C+-
5,374,188
9,479,872,459
8,961,976,799
18,432, 849,
5,188,074
5,063,713,140
4,903,380,
9,967, 368
Java
18,529,273
491,
17,411,757, 778
35,476, 249,413
Pytton
13,680,833
11, 18,188,172,888
18,718, 436, 155
981, 686, 199
Rust
1,263,892
1,533,690, 227
1,485,340,
3,019,
Typescript
5,403,241
8,855,820 820,996
3,726,795,,25
7,581,886,,66 266
Total:
55,339,607
30,799,153,,35
58,169, 984,
118,968,
Table 5: Language-wise sample and token count breakdown of the Obscural ax dataset.
B.1.1 DATASET LIMITATIONS & FAILED COLLECTION ATTEMPTS
Limitations. ObscuraX is collected with the stated objective of allowing Code-LMs better disentangle code syntax and semantics using obfuscation as a semantic- semantic-preserving augmentation. However, there exist some corner cases beyond the import obfuscation alluded to in Section where the correctness of the original code is not preserved.
One such scenario wildcard or global module-Ievel imports, where our obfuscator treats the imported members differently from user-defined ones. This affect correctness in cases such
28

---

Published as a conference paper at ICLR 2025
* imports in Python when header-only libraries used in languages like C++. In practice, find the occurrence these to be limited and hence do not tailor our obfuscator for them. Another scenario is the potential obfuscation of standard macros in languages like Rust and C. These pre-defined by the language standard and encapsulate platform-level functionality.. However, their use is exceedingly rare, and avoid any special handling of these structures.
Failed collection attempts. Before building custom obfuscator, attempted to re-purpose existing open-source semantic highlighting toolchains for obfuscation. We detail two failed efforts below:
git g Srlub/semantic/ The semantic highlighting package made public by GitHub was
natural first attempt. However, the framework seeks to map elements in all programming languages closed set of Haskell constructs, which can be limiting. Furthermore, it does not expose Tree-sitter-like query layer for targeted extraction of information.
We attempted to instrument the semantic highlighting mechanism of popular IDEs for obfuscation, by directly extracting all spans relating to specific identifiers. However, in practice, the language servers of many languages are immature and slow. We further hamstrung by the fact that language servers work reliably only the repository level, which was at odds with the file-level sourcing of Obscurak aX.
git
B.2 PRE-TRAINING CORPUS COLLECTION
Filtered-code-tenn collection. The fi Filtered-ccom codde-ooee-eeee nextt collection consists of text- only split consisting of roughly 105B tokens and a code-adjacent split comprised of further 15B tokens. The text-only split's sourcing is biased towards technical, academic and educational content. We list the constituent HuggingFace sources of the text-only split along with the at-source filtering procedures (if any) below:
allenai/c4 We subset for the realnews l Like split.
Skylfion00//penweebtext
allenai/peS20 We use regex-based removal of citation text and bibliography information.
SchPh//errtoocs-rraall-oooeeee-
wentingghao/math--eettookk
We further bias for educational content by selecting samples with an educational score greater than 3.5 and a repeat count greater than 2.
sirtraii-ai/fineeeb--dddd----
The code-adjacent split focuses on organic and synthetic data sources where text and code appear interspersed, with the text fulfilling descriptive pedagogical role. The following lists the constituent HuggingFace sources of this split along with the at-source filtering procedures (if any):
vikp:/ertbook.uuulltyyyrrrmmmm
We select QA pairs where the answer has been accepted by the community and the answerer has a reputation of 23. We also use QA pairs from only the following sub-domains: stackoverflow, serverfault. askubuntu, superuser, and stackexchange.
siccde/stackk-exccaag--rrree
vilp/code.wittheeppaaattio
nampdn-ai/ddevoocs.io
hti-phii/progrrmmin___oooooo
SiveTTaarrnsstarrodee---ouumnnnn
bigcole/codinggtuttriiass
29

---

Published as a conference paper at ICLR 2025
b Digcode/comitppacki
b gggode/the--taaeeee githhu
The splits are merged, shuffled and subsequently subjected the following steps of filtering:
We follow recent work (Marion et al. 2023) and prune our corpus using perplexity filter to improve pre-training efficiency. We leverage KenLM (Heaheld 201T) model trained on the EN OSCAR (Abadji et al. (022) corpus and discard documents with a length-normalize perplexity of more than 325 and less than 7.
We follow established practice (Penedo al. 2023 and discard samples with any <=4-gram whose repeats constitute greater than 15% of the respective n-gram count.
We remove samples whose English language score is lower than 99%. We employ the Lingua language detector to make this determination.
Finally, we perform aggressive MinHash (Broder 1997) de-duplication using shingle size and a similarity threshold of 0.5. This allows us to efficiently approximate frontier model performance with constrained compute (Lee et al. 2022).
fi -ittered-source-code collection. The silteeee--suuccecoomm of pruning an already filtered version [4] of the Stack (Kocetkov et al. 2023). We subset for the seven language splits, which we evaluate on C, C++, Go, Java, Python, Rust, and TypeScript - and supplement it with the Markdown split. On the resultant data, further apply the following filters:
collection result
files forked more than 25 times, we retain them the average line length is than 140, maximum line length is less than 500, and the alphanumeric fraction is more than 25%.
For files forked between 10 and 25 times, we retain them if the average line length is less than 120, the maximum line length less than 200, and the alphanumeric fraction is more than 35%.
For files forked less than 10 times, we retain them the average line length is less than 100, the maximum line length is less than 200, and the alphanumeric fraction is more than 40%.
We only retain samples from conventionally used file extensions and drop samples from valid but uncommon extensions.
Seeking to avoid the deleterious effects of near-duplicate data on Code-LMs (Allamanis 2019), we subject the resultant data to an aggressive MinHash (Broder 1997) de-duplication, using a shingle size of 20 and similarity threshold of 0.75,
Language
Chosen Extensions
Sample Count
Token Count
[.c,
8,119,577
18,381, 796,
9 C++
(.cpp, .ctt, .cc, .cox, .htt, .hh, hpp, .hxo
265
18,912, 828 828
G0
(.go)
4,507,348
9,967, 9,967,123,38 123,
(-java)
19,324,891
15,2499,36,,21
3 Harkdown
(.nd, markkoww)
12,623,415
5,614 00214,520,44 528,687
Python
(-py]
12,384,123
12,888,526,,13
Rust
(.rs)
1,324,569
1,932,835,77,
Typescript
(.ts, ttx
8,207,133
4,437,582,
12,351,222
76,363,877,412
Table 6: Language-wise chosen extensions along with sample and token count breakdown of the ittered-sources collection used to construct ObscuraCoder. DOBF, and Causal LM pre-training corpora.
B.3 OBFUSCATED CODE EXAMPLES
For the reader's reference, we provide complete ObscuraX samples all supported languages along with the accompanying metadata.
At pemiistahl/Linguappy
bigcode//te-stackeee
30

---

Published as a conference paper at ICLR 2025
Original Python Code
Obfuscated Python Code
future.. pries, diviatin, atselute,
34O0T,5 24v0r,1, 2HPORT,2 DET0112
Sopersttat "statto Sass parse check apprr()
VARUR "VTATOO Baae parto Chack" FUM,AO
.fd copaat, jetejos.patt (f
MALLI FIMETTOON TRRORT RUSE, (W0L,1)
srrctinsseecffee path
VALL Sere
panter_path joth.jotrt..d,, "osamen")
COMmOL DRAONT LA AIR, L/VAL "cammon))
.ptt depprettngg serstooto
.ptt, pepeeneeeedder COL.NESTTOREE
verafor
preffie
lase (areffu(e] 1.15 Lantoreffig alce
base
(VARL
(rafer,
'pondaa""
verston, www.tfic..pttt........ co.path.jetni.com drawe
7A4.3
D.P.RT COMELL (ML.5)
238_0b,, ioport
VAI,10
wrew ba0n dhame
"eadt"
VAI,21 buse VARLD try:
Sapprrtttt.. L_, (reeme, Frast tace feportttar, tevelruse_A (opar) try?
RIWALA, MRR.ARA WALDEYMA
except
glanale()
(AKCANEEEAA_AA) RUNG,
Metadata
Obfuscation proportion: 85.6%
THeMe i "pelerr Furceties, "ERPORTLT "steietan", 11 "Mboolurs, Sepertt, "EMPORT,TT "aart", "WALees Lisportiion, Hort, TRUSELatE featien TOWLLTS Lcety 1rusc,ate "ponst, TUNCLEt "atrmane", "WE,2": "..ttle".., "AARL3T "serator peseetttccppti pubss, "NARLAT) Teagen', 1942,511 fetnort, 1WAL.27 tveranon_ "WALT' "peatter, "NsELOre "Luremee"" "TLM, 3td Traplret, YEuRC Tuent, "WALD" "orome", TaeE 201: Tase ataaeatoo rww.etou.mperrr. "WA0,33" "ename", "N4ELE2' wet, rrssc.on pr "uutteeetuu rucaaa: "toaet, "AARaaTr "eiterane", (AARERRT, "arcfitat, raascaat "ectuta", "WALLAR' laadt", a01s Sparsely Faasc Tcheck".
2511 reremitstty pa1y "tevelty 1WA1 Deporttrrent, T948 201y rer, inai..or globeta", "TIC,T") "gecertent
Figure 3: An example of original and obfuscated Python code in Obscural aX along with accompany- ing metadata stating the obfuscation proportion and the identifier map.
Original Java Code
Obfuscated Java Code
aackage con 451 Goog whop:
perlage TRPORT wwys Mibee letele
fepoit androfdion auectee
support aepp-pyctieesseetiiity..
Ieport andrond perost, J ana typCompstcccterttyy
teport aneroid content (HPORT 22
toport ardrotd.uiidgs. TeutVies
auttic class stoudetart eccends kestanppsicttvvity
30verride
protected sold weleaaeeeeedle coveltestaerrrran) operr www.aaaattanaaeeeeaaeee betttettttt)
SetContersthen Lepectt www..tttataaaaaa
sentonsatrieet Laysrt
Textrice resute (Tes mn) de textriedt)
Tentitinn resute (Teststea) rinderedogdd bauch s TextViin recalttons (learotie) Mirovieeiyy::0e te- tewtefee) 20p0R1_2 yak, getiatenc(1:
Tactriez reautttata (Tactriee) findriaddyddi, taworeedds Intert in getintersis;
resuts, Rasc BlW.L www.grrrrrrnggooo www.gerrggittttrtate... Cngistractrame
recal mota Fokr wpww gerstnt trrttetogtttratt wightrtttdaaan daraa)))
Metadata
Obfuscation proportion. 12.0%
13P01T.91: Tcwe',
12PPORT_A "assport', "CNFORT_21: Intwnbt 1ARR_01: 10m), 'TUNC "aesTeck".
Figure 4: An example original and obfuscated Java code in ObscuraX along with accompanying metadata stating the obfuscation proportion and the identifier map.
31

---

Published as a conference paper at ICLR 2025
Original C++ Code
Obfuscated C++ Code
siselade
risctade condis.to
Rrnciuee catring.to
conrtng.1t
nomesoace
sat nua:
snt deca;
snt (000])
trea[as66]:
chir [x4):
char str(ac][AR)
int twairgten (W4,2)
recurn S6C-101
neturn 1APL2AC 3AR
vord addcirt 1AL.,, the VARLE:
ant suntint 17
WARL?T
fio anamis
anand; wivtt (NNELENNR
ints risterretii
"aturn aras
Metadata
Obfuscation proportion. 29.9%
Sandin.a", NNRUATS Taly NAALETS fet, PAA
Figure 5: An example of original and obfuscated C++ code in Obscuras aX along with accompanying metadata stating the obfuscation proportion and the identifier map.
32

---

Published as a conference paper at ICLR 2025
Original C Code
Obfuscated C Code
vord PORE
weile redanst.cmm cen phase apt_ (9781)
RMK_ACHALL) (RMN,A))
KTT (tteeateroa)) stee) state, looptine ledr)
VAR,4
if ceahot_dos_phoae 1o [I apt_ddo, wtw..ttnnwttssFP__nrr.. recurn;
If (HARLT
t ej. (ee)
for curnRoLt VAT,I b VAT, 10; VML
VAR, FLARLS +: VAL ANN.7/NERSS
sccor porst]1]
date, persC[-]
SAILE(MARSS) 22
Adettre Moros, 12s/port, stn, pin tieer, chemets
soefine TUNC SIWALL, MALLOO, LI, VAL,22 W8L (occhen,
SXSMM33
Sowrr, pinll (ARDOR))
rater wattrr_cataaaoo carcee()
SPDD pappo 143:
eloe (ssttorport
elae cOPrORepent
nettr_.dttapprrae porttt])
(0f20weport EPSOC)
wotor.date, portep)- OUL, PIN
Bundet NOTOT
NOTDR_PORS
(arker) de 5
runder ACTOR_PIN
WAR (4k))4i))
CLASS_E
scher_packenta] is
SWL 1411)
(chet_pccett2)
SWA [427]
pehet_pccktt3)
for Cire WAR
WALLS WALK WAL VARLS
perkbuffers)
VAR_EE(WRLO) (OR.GDARLL3)
jurt[juuffr(]] dat
ister_dex_svsee DO4S FORTT
T2Ng vccaL 2:
T2Ng vootz 20101 TON,
T2N voots 20107 72M, if GOERLA
WAL,23 VAR 2S1OTT.OALTTHE
1AL,23 HARR_2 20/OT_TULTTOE
daher_dex_ssrtkk::
1F (444
else (WAL27
"URC 5211
sahes_dee_parstu7
TUNC_603
(VAR
Metadata
Obfuscation proportion: 82.1%
fotatet, wocctt choepriant
TeaK "nater
sARLors fporet, 19A8,201s tarnt, (148L117: "ptruat,, "248L1215 "eteer' "WAR,32 3231 "timer, charnsl, "CLASS_0": o "HOTORUPESS" "WALL4' "aahtt "9A8,22"
191, 1ak 201; toores butter", "NALLAT: "portt, butter" 1000,200 "porto buffert, "D48_201; 1051C1, COTTOOOTOOOUTTT COUNT", "WARLaZ"s ITDRL", "VAR,227: ASR", "9A8,22 rosier rier, 2oe 1ocest, 1Wi.,241 241 ro0ne", 1990,251 recest, 19ks 2011 1os1et, ruK,at: dehet,dei_serre, 2s11 tasest "S900__ FFMcLstt "saber.daaggorsss
(TNNC "cotot,dex, sorsA), 1948,271 1051on cPT0.""
Figure 6: An example of original and obfuscated C code in ObscuraX along with accompanying metadata stating the obfuscation proportion and the identifier map.
33

---

Published as a conference paper at ICLR 2025
Original Rust Code
Obfuscated Rust Code
Generate eecrypted beacer
115 neserate an excrypted beader
ressurce encryatec sainng Iybrid crypto schane.
155 for retserce encrypted uctin an hyarte cryats schowe.
177
177 A rarece symeetrie key to pererated for the spec ife syenetric scheee,
encrytted
psltey
2/2 thas pre panded ayenaor feallly ancrypted nstacata
277 extrapted using the publis key the ase scheoe and polticy atte iuutes
arersps.lyertt headeer,,,, policy:: Broticy.
pollcy: APotiay,
publtc_.ey: Fuc atcrtbutes: 6(Att-10ute).
wrr.Aata: Syrtin Sprtontersaiito
3 -> www.:eeeeoyyyppeedeee.ee Sontattive
Formatiirrs
wtore
AbeSchere 3td inerkerrer Synt soareeeerr (Send, Syweet ictrypio.
Atstcheme
abds Sece:merker::Sen.
SymmetttoCrytts
lec waginn Engines cr
1st engire segtre:ssweeneee))
encryttcc.d
(eyeeeyy aoy,ie_koy,
Stdkey
1st (st, syt4,, ancrypted_as)
regine gww.rateeeeeeeoo eeseratee.yaaerrr.... Jeylont fry, puid fr. Jkey, attritures, S114ey ::10000))
syserbric cjkey Sockeys bytee (yl_
conwert bytss
haader, bytea 132_ www.eeeeeerrpeeed..oooo a401. oe.
header Jyyee. www.eeeddeeccoopood.....
Lwn.rreeeeeeeee..eee (ants)
tex (netaa)))) nata_cata
Tot axece Aiunncee cnmet (ane)))))
Nexce
5: S::tenne: readdeett Categetnee(33
beader, wyy.ee......... eetee/enterreeeeeee... bytees111
recder bytes exttoelinnnee FUNC./0333
Let. excryptedd metedete Screncrgpttaaymee wwytrttttttttttttcccccccmc Jaey Secta ty
Enerypoed metadata
tet end woccyytedd.coo awtadate Scowncrysh May.atttrittt to, Jey, Ansto. Anete.FUWC..D DU3,
Precypttd netwiata
Sronce, Nome)3:
Sayeer_yttt.... axtenddas aeddeenrrggoedeeersttoe bytes
beader,
syenetr ic. key,
encryster headere haader_byten
Metadata
Obfuscation proportion: 2.6%
toc.bytes",
Tencrypted, www.ppteddhhheeeeeele byytes
Figure 7: An example of original and obfuscated Rust code ObscuraX along with accompanying metadata stating the obfuscation proportion and the identifier map.
34

---

Published as a conference paper at ICLR 2025
Original Typescript Code
Obfuscated Typescript Code
detaltt center antl perfod: posit "BU.."
defeoit (dete: arytly fet positton "BLLL"
211
71 consota datal
// console (dat)))a)
I/ pet date free the and of the datases bacovers (poatitive negetivs conec datasse data, stece abe (perfed))
Xf dets than the and of the dotaset beckward (ocatctive to regacive natber))
const <'Ecaeotath abolperlad))
coleutate
everage
steple woving everage
(previ namber,
redbee((prev
to balov
the price belon censs lastsataPuint date stoeee-))) 27 conole legiten, (esebrabbioat)
cont lestbotahhiee tite-st/ce
// castale bogerey boomarateeen)
seras: aterstiww, clase LeatSetuPoint
const [ starttiiee cloow tescdutaaotee
(closs eno)
(AAL)) WAL
poststinn
postetoon
sea,
Metadata
Obfuscation proportion: 25.
(ARLO!:
1 rotose",
Yourn".
Figure 8: An example of original and obfuscated TypeScript code in Obscur aX along with accom- panying metadata stating the obfuscation proportion and the identifier map.
35

---

Published as a conference paper at ICLR 2025
Original Go Code
Obfuscated Go Code
17 fees pats the reptied telp packet
Recx restied foap sacket
fane (1 strare) secvild see intt cecertesp orear) I
func CHALLE <Tracs) RAM_MOONRL 116) (LA65,M)
war
(lyyt, 222) Monl!
PEBC LET TE ASL
VARLA FANC
reas 2ONPRRR
NED teol
KARS teol
loval
MOst bool
for
from syacesl, Decrtrant 1,16,
day tiie, www.raaaaaraaaiett..oooooo waatt)
If aer space1l.cAGNS CAGACS La stew Stoce/tal continue
spscatt EAGADE vMR_2.5face((0)
return res,,
reture reep, SAAA
B[on]
Len.VAAL O. FENC 5E3 on net. E7r4tan
relo conpliteeeeeeseee
reap-tup INviloeryasktheell WARLL
Sertrgl) Strpent
Strilge) St.rrfigg
<loe
reeg oor are wep Ceryorall Sackodieteers wWd [-])
VCD reao top Invt as id
ENGIOPTyystcheel WARL
VAR,
resp.1p- Sttrirrr)) Sorpsttriegt)
sng respuceg
Hast resp-fp.det Strlite WA.....
(VEL,6 wwr.e.iooo 65 abeil wost II wID du, VOL 4TO0L 1f VAR 3.5toettta)
fnc. Crnontt (farrr/ reaponan")
reep,
response")
trink
returs resp, mL
Metadata
Obfuscation proportion: 42.6%
Figure 9: An example of original and obfuscated Go code in Obscural aX along with accompanying metadata stating the obfuscation proportion and the identifier map.
36

---

Published as a conference paper at ICLR 2025
C DETAILED RESULTS
For completeness, detail the split and language-wise performance of the models on all tasks (where applicable) discussed Section 5
@ Model
Doc to Comments
Newline After Code
B Newline After Doc
& Newline Randon
Vine Split
-Tab Indent
CausalLN 255M
9.75
12.19
18.39
18.99
11.51
12.
ObscuraCoder 255M
11.58
+1.83
15.91
+3,72
14.98
+3.99
12.91
+1.92
14.97
13,46
14.67
+1.90
CausalLN 1914
12.19
15.86
16.16
13.41
14.02
18.29
ObscuraCoder 491M
23.34
26.21
26.82
21.95
22.56
24.39
CausalLN 1.28
29.59
ObscuraCoder 1.28
33.56
+9.03
CausatLN 2.60
36.
35.11
36.93
34.81
35.67
37.67
obscuratoder 2.88
38.93
46.
18.36
14.74
18.96
44.31
StarCoderBase 18 75.
31.67
27.43
28.65
28.65
Deep Seek 1.38
41.36
52.94
16.88
50.68
44.91
51.47
StarCoderBase 38
35.36
37.35
29.63
35.67
37.87
38.17
Startoder2 3B
49.12
60.31
49.
51.94
54.58
57.21
StableCade 38
44.89
59.79
55.48
48.79
58.28
58.28
CodeGenna 2B
9.56
11.71
12.97
10.53
7.92
Ph1-2
59.34
Table 7: ReCode Format pass@l comparison between ObscuraCoder and comparable Causal LM models, along with frontier models for context.
A Dead Code Insert
* For Mhile Transform
= :0perand Swap
m Var Renaming
os Var Renaming Naive
xt Var Renaring
di Model
CB
RN
CL 25SM
2.63
10.43
12.04
12.43
6,11
9,45
ObscuraCoder
11.03
15.05
+2.64
15.67
+3.44
12.26
6.37
+2.51
CausaliN 491N
4.87
13.74
16.46
13.74
7.92
11.86
ObscuraCader 491N
+3.66
6.53
+12.63
26.37
25.14
+8.68
26.99
+13.25
14.21
+6.29
18.97
+7.91
26.96
23.21
ObscuraCoder 1.28
10.82
34.77
37-91
34.56
19.96
20.13
Causa 2.86
10.22
34.13
36.15
35,97
14.98
25.76
obscuraCader 2.88
17.83
+6.81
+16.75
+11.41
+9.01
+8.27
+12.37
StarCoderBerr Base 18 6.53
32.19
29.12
32.
28.77
24.78
DeepSeekCoder
42.67
StarCoda rease
38.61
Startoder2 14.
59.34
58.85
StableCode
52-77
56.81
49.64
49.86
30.97
CodeGenna 2B 6.
12.12
11.67
12.49
9.54
6.11
Ph1-2 18.15
64.19
63.56
62.26
59.77
$4.35
Table 8: ReCode Syntax pass@1 comparison between ObscuraCoder and comparable Causal LM models, along with frontier models for context.
Butter Fingers
IInflectional Variation
Synonyn
Characters
Character Case
CeuselLN 255M
1.26
1.97
2.03
2.78
2.89
1.82
ObscuraCoder 255M
+1.48
5.74
+2.29
4.26
+2.75
4.78
+8:26
3.14
+2.74
5.63
41.31
6.13
CouselLR 4914
4.67
5.97
ObscuraCoder
11.58
6.71
6.11
CausalLM 1.28
15.49
11.55
18.16
8.74
11.69
18.16
ObscuraCeder 1.28
20.08
+4.59
16.21
+4.66
17.17
+7.01
13.44
+4.70
17.17
+5.48
15.89
+5.73
CeusslLR 2,88
16.12
14.29
15.96
6.53
15.98
15.44
ObscuraCoder 2.88
25.32
19.26
+4.97
17.12
16.68
+8.15
21.48
+3,42
20.86
+5.42
StarCoderBase
15.07
13,85
13.09
13.41
14.19
13.41
DeepSeckCeder 1,38
27.43
24.82
26.56
19.45
28.28
25.10
Sta StarCoderBase 38
20.34
17.24
16.89
15.01
16.56
18.91
StarCader2 36
36.65
26.57
23.61
18.
25.38
23.17
StableCode 26.97
23.48
23.11
19.35
26.68
23.
Codešenma Phi-2
5.67
41.36
29.14
30.22
37.44
Table 9: ReCode Function pass@1 comparison between ObscuraCoder and comparable Causal LM models, along with frontier models for context.
37

---

Published as a conference paper at ICLR 2025
Python
TypeScript
CausalLN 255M
3.51
1.29
4.32
2.78
6.33
ObscuraCoder 255M
6.94
+1.98
6.27
5.47
3,46
6.92
CausalLN 491M
ObscuraCoder 491M
CausalLN 1.2M
12.97
13.01
13.89
9,71
10.92
ObscuraCoder 1.28
19.83
+6.16
16.26
+5.22
18. 73
+5.98
15.69
15.95
18.92
18.10
CausalLM 1.2M
17.
ObscuraCoder 2.88
25.88
23.56
26.31
16.33
25.
StarCaderBase
12.85
DeepSeekCoder 1.33
29.98
29.11
28.87
13.59
27.52
StarCoderBase
38
20.46
18.64
28.19
16.83
19.92
Startoder2
38
26.78
27.89
26.49
25.56
30.
StablaCade
38
28.84
25.31
28 84
23.83
27.44
CodeGerna
28
14.65
20.86
Table 10: Multipl-E pass@l comparison between ObscuraCoder and comparable Causal LM models, along with frontier models for context.
Model
C+*
Java
é Python
Rust
TypeScript
CausalLN 25SM
9.45
ObscuraCoder 255M
+2.41
CausalLM 4918
10.41
10.36
13.81
7.59
11.02
ObscuraCoder 4918
14.29
+3.88
13.24
+2.86
17.31
+3.50
12:35
9.94
16.88
+5.86
CausálLı 1:56
19.53
22..11
19.89
ObscuraCoder 1.28
28.41
+11.31
30.84
33. 54
22.87
+6.26
31.83
CausalLM 2.88
Obscur sCoder
+12.10
+10.70
StartaderBase
18
DeepSeekCader
StarCoderBase
33.99
StarCadar2
47.78
StableCade
38
58.51
46.89
54.64
48.17
47.12
CodeGerna
28 45.31
40.87
32.63
16
25.
Phi-2 17.17
37.13
71.78
11.76
31.38
Table 11: Multipl-E pass@10 comparison between ObscuraCoder and comparable Causal LM models, along with frontier models for context.
@ Nodel
S++
Java
Python
Rust
I TypaScript
CausalLM 255M
15.97
15.22
17.88
8.93
14.62
ObscuraCoder 255M
16.93
18.96
16.38
13.16
25.58
15.50
11.43
12.50
18.70
14.08
CausalLM
21.21
21.56
25.13
11.96
23.59
ObscuraCoder 491M
28.53
27.88
+6.32
38.04
+4.91
18.91
+6.95
23.96
+8.37
CausalLM 1.26
34.84
37.69
36.35
28.84
Obscur aCoder
113.54
51.91
414.22
+13.52
f11.31
48.64
$15.87
Caus@lLM 2.86
47.71
49.44
52
41.42
48.96
CascuraCoder 2.88
+17.16
84.97
74
+14.30
67.41
+17.29
53.45
+12.03
58.17
+11.15
StartoderBase 16 39.11
38.
48.75
30.64
46.
DespšeekCoder
1.38
71.48
57.42
77.79
64.35
17.16
CadeCoerBass
36
55.65
56,41
61.27
55.91
58.
Startader2
67.67
73.12
StableCade
58.85
54.63
Phi
62.46
57.40
83.95
26.91
55.76
Table 12: Multipl-E pass@100 comparison between ObscuraCoder and comparable Causal LM models, along with frontier models for context.
38

---

Published as a conference paper at ICLR 2025
a.c
C++
180
Java
Python
TypeScript
CausaLLM 255N
32.07
32.45
33.13
33.79
33.09
32.15
32.58
ObscuraCoder 255N
32.68
+6.62
33.26
34.04
34.42
33.58
32.59
33.24
CausaliM
33.37
34.87
34.16
ObscuraCoder
34.07
+0.95
+0.65
CausalLM 1.28
35.28
34.97
39.
36.61
35.32
31.59
35.59
ObscuraCader 1.28
36.23
+0.55
36.
+1.70
38.97
+1.
37.69
+1.08
36.11
+8.79
36.24
+1.65
37.69
+1.50
CausaliM 36.08
36.41
38.
ObscuraCoder 2.88
37.58
07.04
+1.43
39.
30.97
37.73
36.28
38.52
StarCoderBase
36.11
37.82
38.75
36.88
DeepSeekCoder 1.28
35.81
35.79
35.64
36.68
35.99
35.08
36.81
Basr
38
36.69
38.49
48.03
39.98
38.14
37.59
38.52
StarCoder2
39
38.09
38.78
48.48
39.14
38.39
37.45
39.89
Stabletode
38
36.75
38.11
39.34
38.26
37.34
36.88
37.88
Codetenna
36.62
Table 13: CommieChronicle ROUGE-1 comparison between ObscuraCoder and comparable Causal LM models, along with frontier models for context.
in Model
9.
Cte
-60
Java
Python
TypeScript
Rust
CausalLM
10.42
18.51
ObscuraCodar
.0.57
40.53
10.45
40.35
CausaiLM 191N
16.82
10.64
11.38
12.09
11.20
16.01
11.01
ObscuraCoder 491N
10.40
11.22
10.92
18.28
11.83
+0.45
12.51
10.42
11.56
18.38
18.59
+0.58
11.45
10.44
Causaliı 1:58
11..6
11.57
13.12
12.81
11.75
ObscuraCoder 1.28
13.16
12.95
+1.35
13.98
14.11
13.96
+1.07
12.04
13.85
CausaLLM 13.04
ObscuraCoder
StarCoderBase
DeepSeekCoder
BSarCoderBase
14.84
StarCoder2
14.75
13.37
Stabletode
28 15.46
13.87
15.05
14.65
13.87
12.89
14.63
Codefinnaa 28
15.08
11.84
13.47
14.86
12.77
11.86
13.95
Phi-2 12.18
11.77
12.19
13:13
12.38
11.19
12.29
Table 14: CormicChronicle ROUGE-2 comparison between ObscuraCoder and comparable Causal LM models, along with frontier models for context.
d Model
9.
C++
-00 Go
Java
Python
B TypeScript
Rust
CausalLM 255N
29.85
39
31.79
32.11
31.53
38.77
31.14
ObscuraCoder 255N
36.47
+0.62
31.52
10.53
37.07
10.28
37.74
10.63
32.11
10.48
31.23
10:46
31.79
10.65
Causatin 491N
31.21
31.99
33.39
33.22
32.75
32.62
ObscuraCoder 491N
31.97
32.76
34.01
33.88
+0.66
33.23
22.34
33.02
CausaliM 1.28
33.05
33.19
34.49
ObscuraCoder
#1.65
11.56
+1.39
11.60
+1.16
CausalLM 2.68
34.46
34.78
36.21
35.64
34,96
34.36
35.09
Obscuracoder 2.38
36.19
-1.73
36.38
+1.60
74
+1.53
37.19
+1.55
36.82
+1.84
35.21
+0.85
36.67
+1.58
StarCoderBase 18
34.44
34.94
36.58
35.91
35.48
34.57
19
DeepSeekCoder 1.30
33.94
34.42
34.24
35.49
34,61
33.62
34.52
S Star CoderrBaBae 38
36,51
36.
37.63
37.32
76.
36.12
37.11
StarCoder2 36.12
Stab Lecode
37.
CodeGenna 35.87
35.43
Phi 33.01
32.98
34.97
34.17
34.23
32.56
33.36
Table 15: CommitChronicle ROUGE-L comparison between ObscuraCoder and comparable Causal LM models, along with fronticr models for context.
39


Reviews:
[{"review_id": "ZF25b1hdML", "reviewer": "Reviewer_v87f", "summary": "This paper introduces a pre-training strategy that relies on code obfuscation to help models disentangle the semantics and syntax of code during pre-training. Obfuscation is applied to function names, variable names, import statements as well as package names. Some of the details were unclear (especially on the masking applied) but it appears that, given a source code snippet the pretraining data may see the source code as-is (as in regular CausalLM), the source code with function name obfuscation or pre-training with bi-directional obfuscation translation data. In contrast, DOBF a baseline method of obfuscation uses a obfuscated code along with identify maps or regular source-code. Data used for pre-training includes mixed-medium data (code + text) (90B tokens) as well as source-code only data (152B tokens) in the first phrase along with 30B of randomly-shuffled (code+text). In the second phrase, they use 30B tokens of obfuscated code, 58B tokens of obfuscation translation (bi-directional),  and a randomly shuffle of code+text for 30 B tokens. Models are primed with text only before source code is introducted in pre-training. Evaluations presented are presented at different model scales (255M - 2.8B) all with a fixed data budget of 272B tokens. In contrast to using the regular CausalLM training, they report a gain across all metrics on tasks of defect detection, code completion and library oriented code generation. In contrast to DOBF (which appears to be better than vanilla CausalLM), the method still does better. \n\nOverall a good, well documented paper.", "questions": "1. See Weakness. Would it be possible to study this at the smallest scale if there's training budget (and adequate time) during the rebuttal?\n2. The motivation for obfuscating library and names and package names was a bit counter-intuitive to me. Wouldn't we want the model to *learn* different package names and what they do? I note the results presented in 391-409 but I am not sure I'm convinced that specifically this type of obfuscation could explain the gain? To make this claim, wouldn't you need to pretrain without this type of obfuscation present?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting use of code-obfuscation\n- Well constructed experiments to help answer the empirical questions posed\n- Dataset shall be released after paper publication", "weaknesses": "- While the goal of the work was to demonstrate the benefit of code obfuscation (motivated in trying to separate semantics and syntax to improve code-understanding), it would have been helpful to have at least one baseline from the related work in Lines 147-157. At the moment its unclear how one should act on the findings in the paper or how they compare with other approaches (that share the same motivation but are not obfuscation based).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EilV6sO17a", "reviewer": "Reviewer_Wout", "summary": "This paper proposes a pre-training objective for code language models (Code-LMs) that leverage obfuscated code. The authors claim that \"grounding\" Code-LMs on obfuscated code helps to disentangle syntax and semantics, which is essential for improving their performance on downstream tasks such as code defect detection, library-oriented code generation, and multilingual code completion.\n\nHowever, there is a large line of missing literature, mostly on natural language processing and machine learning, that discusses training neural networks to disentangle syntax and semantics in natural language, and some of them even deal with code. In my opinion, the authors should have discussed these papers in more detail and compared their work with them.\n\nDisclosure: my related expertise is more on the machine learning and language modeling side, as well as the related (computational) linguistics perspective. I am not very familiar with the large body of software engineering literature this paper has cited, nor can I evaluate whether the paper has made a significant contribution to the area of SE. However, since ICLR is more considered a machine learning conference, my evaluation is provided from the perspective of machine learning.", "questions": "1. It is not very clear to me what the authors mean by \"grounding,\" which has already been an overloaded term in the machine learning community (cf. Chai et al., 2018). Can the authors provide a more detailed explanation of what they mean by \"grounding\" in the context? Is this a software engineering term?\n\n[Chai et al., 2018] https://www.ijcai.org/Proceedings/2018/0001.pdf\n\n2. L397: have you tried evaluating Pass@1 with greedy decoding (i.e., temperature = 0)? Since you are anyway submitting one prediction for testing, I would expect greedy decoding generating a program with higher probability (assigned by the model).\n\n3. Table 1: the metrics of ReCode are never introduced in the paper. Please add a brief explanation of what the metrics are.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea is well presented, and the paper is generally easy to follow. All research questions are stated and, to some extent, well answered. A comprehensive ablation study is conducted.\n- A promising code model will be released.", "weaknesses": "- The authors' background may be more on the software engineering side---most of the cited work is on software engineering, and there is a significant line of literature missing in machine learning and natural language processing.\n\n    In terms of disentangling syntax and semantics, the following two papers discuss training neural networks to disentangle syntax and semantics in natural language:\n    - https://aclanthology.org/N19-1254/\n    - https://aclanthology.org/P19-1602/\n\n    In terms of semantic-aware language/code generation with language models, the following papers are very relevant:\n    - (natural language) https://aclanthology.org/2021.emnlp-main.564/\n    - (code) https://aclanthology.org/2022.emnlp-main.231/\n    I especially wonder how the performance of the proposed method compares to other models with the decoding strategy proposed in this paper.\n\n    In training code models, the following ICLR 2023 paper is worth comparing to:\n    - https://openreview.net/forum?id=hQwb-lbM6EL\n\n    Related to library-oriented code generation, the following ICLR paper is worth referencing:\n    - https://openreview.net/forum?id=ZTCxT2t2Ru\n\n- I'm not sure if this comes from the cross-domain difference: to me, with backgrounds in machine learning and linguistics, both syntax and semantics of the original and obfuscated code shown in Figure 1 are all the same. The difference comes from the lexical-level symbols (i.e., variable and function names); therefore, the approach does not really disentangle syntax and semantics but rather disentangling form and meaning. I believe there would be a large amount of ICLR audience who would think the same.\n\n- While the main performance in Table 1 is promising, it is not sufficient to demonstrate the effectiveness of the proposed method. On BigCodeBench, the model performs much worse than most existing models (except StarCode) with a comparable number of parameters Pass@1. Given that DOBF (Lachaux et al., 2021) has already been introduced, I would expect the performance of the proposed model to be better in order to demonstrate enough contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JwZHosyywi", "reviewer": "Reviewer_ZqQD", "summary": "This paper introduces ObscuraCoder, an approach to enhance the pre-training of Code-LMs by focusing on the disentanglement of syntax and semantics through the use of obfuscated code. Traditionally, Code-LMs rely on autoregressive pre-training which often struggles to handle semantic-preserving syntactic perturbations, thereby reaching a plateau in terms of data efficiency and model robustness. To address these challenges, the authors propose grounding the pre-training on a dataset called ObscuraX, which comprises approximately 55 million pairs of source and obfuscated code across seven programming languages. This method contrasts with previous efforts such as DOBF, which primarily aimed to revert obfuscated code to its original form. ObscuraCoder, ranging from 255M to 2.8B parameters, is pre-trained on a substantial 272B token corpus, including ObscuraX. It achieves significant improvements in syntactic and semantic understanding of code, surpassing both conventional autoregressive pre-training and DOBF models. Notably, ObscuraCoder facilitates enhancements in multilingual code completion, commit summarization, and versatile library-focused code generation, without necessitating modifications to the Code-LM during inference.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The evaluation of this paper is exceptionally solid:\n    1. ObscuraCoder is benchmarked against strong well-established models such as DeepSeekCoder and StarCoder. Those models are very strong at 1B scale, ensuring a competitive comparison.\n    2. The model is trained across four different sizes, showing its scalability.\n    3. Controlled experiments provide a fair comparison by training ObscuraCoder and baseline models (CausalLM and DOBF) under the same conditions across all scales, attributing its good performance to the proposed methods rather than discrepancies in training setups.\n    4. A comprehensive selection of benchmarks is used for evaluation, covering a broad spectrum of tasks across large, multilingual datasets.\n\n2. ObscuraCoder consistently surpasses baseline models in a variety of tasks and scales, with performance gains increasing with model size. This showcases both the effectiveness of the deobfuscation pretraining and its scalability.", "weaknesses": "1. The methodology of ObscuraCoder closely resembles the DOBF, which slightly diminishes the novelty of the paper. However, the scope and depth of the experiments and evaluations presented significantly exceed those found in prior DOBF studies.\n\n2. The quality of the pretraining data is not sufficiently controlled in the experiments:\n    - The ObscuraX dataset has been curated using Tree-Sitter to drop non-parsable code, and contain more self-contained code. So it contains higher-quality, cleaner code compared to other corpora.\n    - Only ObscuraCoder is trained with the higher-quality ObscuraX, whereas the CausalLM and DOBF models are not, raising concerns that performance gains might be attributed to dataset quality rather than the training objective.\n    - To verify the effectiveness of the deobfuscation objective, the authors could:\n        1. Develop a non-obfuscated version of ObscuraX.\n        2. Pretrain a CausalLM using this dataset (mixed with other datasets used by ObscuraCoder, except obfuscated ObscuraX)\n        3. Compare its performance against ObscuraCoder to conclusively attribute any improvements to the deobfuscation objective.\n\n3. The paper lacks comprehensive discussion on related works that are highly relevant:\n    - While CodeT5 (https://arxiv.org/abs/2109.00859) is mentioned in this paper, CodeT5 actually uses an \"identifier-aware\" pretraining objective related to code obfuscation. Those pertinent details are not discussed in this paper.\n    - AST-T5 (https://arxiv.org/abs/2401.03003), another model trained with syntax-aware objectives and known for its code translation capabilities, should also be included in the related work section.\n\n4. The writing style of the paper, particularly in the abstract and introduction, could be simplified. The use of overly complex vocabulary may hinder understanding of the content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N6p0Ag9Svr", "reviewer": "Reviewer_P8nX", "summary": "This paper introduces ObscuraX and ObscuraCoder, respectively a novel dataset and model for the code LM domain, based on the premise that current approaches to training code language models do not sufficiently take into account the inherent differences in syntactic and semantic structure between programming and natural languages.\n\nStarting from the observation that current data approaches tend to focus on scaling, which may in the near future lead to a data bottleneck, but not on disentangling/making available for learning the syntactic and semantic aspects of modern high-level programming languages, the authors propose a code obfuscation training paradigm, and set out to answer a number of interesting research questions:\n1) Does their paradigm lead to improvements on tasks requiring code syntax/semantic understanding?\n2) Does obfuscating imports improve library use?\n3) Is there any performance regression associated with obfuscation training?\n4) How does model performance scale under the proposed training paradigm?\n\nTo this end, they lay out in detail how their ObscuraX dataset was created (including code obfuscation, part of which includes obfuscating library import and use), and how their ObscuraCoder models are subsequently trained.\\\nObscuraX consists of a large collection of code data scraped from the web; contrary to previous work which uses some code obfuscation as a translation target, the authors introduce a mix of 4 pretraining \"objectives\": (1) translating original code to obfuscated code; (2) translating obfuscated code to original code; (3) CLM training on original code; (4) CLM training on obfuscated code.\n\nExtensive evaluations against the same model architecture (Llama) on data of comparable size and from the same sources trained as \"classic\" causal LM (i.e., original code only) indicates that the proposed mixed objective pretraining provides meaningful signals related to syntactic and semantic understanding to the model, improving performance on all evaluated tasks.\\\nIn some conditions, the proposed ObscuraCoder models can even compete with or outperform current frontier models, which are often much larger or trained on at least an order of magnitude more data.\\\nComparison to a DOBF-inspired decoder-only model also indicates perfomance gains over using code-to-obfuscation translation as the only added pretraining objective.", "questions": "I really like this paper, and how well-structured and principled it approaches its objective and research questions.\n\nFor questions, see my answer to \"Weaknesses\": Am I right that the shown ablation on DOBF only partially covers RQ2? I don't think it was intended as part of addressing RQ2, since it comes in a section after the main experiments. Did you (or could you?) train models that use ObscureX data that has not gone through library import obfuscation? That would show better if that particular obfuscation method yields additional performance gains past \"standard\" code obfuscation.\n\nOtherwise, there are just some typos/minor things I noticed during reading:\n- Figure 1: The \"FUNC_0\" on the translation side (purple) of DOBF should be \"def\"?\n- Line 185: \"Other work work\"\n- Line 279: \"confiscated code\" -> obfuscated?\n- Lines 316-317: \"[...] as it is nearly an order of magnitude greater that what would be optimal according to the scaling-law.\" I do not understand this sentence, could you rephrase this?\n- Line 518: \"effectiveness of out obfuscation\" -> our\n- Line 522-523: \"Unfortunately, we find that post-hoc obfuscation training does bring performance gains to causally pretrained LMs.\" -> pretty sure this should be \"does *not* bring performance gains\"?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written and presented. It was very easy to follow, and the authors very clearly laid out the objectives and research questions they want to address.\n\nThe paper does an excellent job explaining the data collection and model training methodology, and the provided experiments are meaningful in addressing the posed research questions.\n\nThe performance of models trained on the proposed novel obfuscation-grounded dataset indicate that the authors are onto something, and that the proposed training paradigm is meaningful and effective.", "weaknesses": "There are some strong claims made about the benefits of the proposed dataset, without testing these by ablating the corpus.\\\nIn particular, RQ2 asks whether obfuscating library imports improved model performance, and is answered in the positive in Section 5; however, this assertion seems to be based on training on data using the full ObscureX obfuscation method, which includes library obfuscation (25%) as well as \"standard\" code obfuscation.\n\nI am not sure this is a valid conclusion, as these results may as well stem from training on the obfuscated data in general, and not the library obfuscation in particular. It would have been more convincing to use an ablation here and train a model on a version of ObscureX that does not contain library obfuscation at all. This model could then be compared to the model presented here (trained on the full ObscureX data paradigm), to see if there is an improvement that can be specifically attributed to the introduction of library obfuscation.\n\nTo some extend this is indicated by the DOBF-based decoder model in Section 6; however, that model is trained differently, with backpropagation on the identifier maps only.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)