DOBF: A Deobfuscation Pre-Training Objective for Programming Languages

1 Introduction . Model pre-training with self-supervised methods such as BERT Devlin et al . [ 2018 ] , RoBERTa Liu et al . [ 2019 ] , XLM Lample and Conneau [ 2019 ] or XLNet Yang et al . [ 2019 ] , has become ubiquitous in Natural Language Processing ( NLP ) , and led to significant improvements in many tasks . These approaches are based on the Masked Language Modeling ( MLM ) objective , which consists in randomly masking words from an input text , and training a model to recover the original input . In the original approach proposed by Devlin et al . [ 2018 ] , a fraction of selected masked words is replaced by masked tokens , another is replaced by random words , and another remains unchanged . Since then , a myriad of studies have proposed to modify the MLM objective , either by masking contiguous spans of text Song et al . [ 2019 ] , Joshi et al . [ 2020 ] , masking named entities and phrases Sun et al . [ 2019 ] , sampling masked words according to their frequencies Lample and Conneau [ 2019 ] , replacing words with plausible alternatives Clark et al . [ 2020 ] , etc . Overall , most of these pre-training objectives boil down to denoising auto-encoding tasks with different methods to add noise to the input , using arbitrary noise functions . In our case , we are interested in pre-training deep learning models for programming languages . As in natural language , pre-training was shown to be effective for source code Feng et al . [ 2020 ] , Roziere et al . [ 2020 ] . However , these studies both rely on the original MLM objective proposed by Devlin et al . [ 2018 ] , which was initially designed for natural languages and does not leverage the particular structure of source code . We argue that this objective ⇤Equal contribution . The order was determined randomly . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is actually suboptimal in the context of programming languages , and propose a new objective based on deobfuscation of identifier names in source code . Code obfuscation consists in modifying source code in order to make it harder for humans to understand , or smaller while keeping its behaviour unchanged . In some ancient interpreted languages , name minimization could also reduce the memory usage of the program . Today , it is used to protect intellectual property by preventing people from understanding and modifying the code , to prevent malware detection , and to compress programs ( e.g . Javascript code ) to reduce network payload sizes . Moreover , C compilers discard variable names , and current rule-based and neural-based decompilers generate obfuscated C code with uninformative variable names Fu et al . [ 2019 ] . Obfuscators typically apply several transformations to the code . While some operations can be reversed ( e.g . dead code injection ) , the obfuscation of identifier names—renaming every variable , method and class with uninformative names—is irreversible and has a substantial impact on code comprehension Gellenbeck and Cook [ 1991 ] , Takang et al . [ 1996 ] , Lawrie et al . [ 2006 ] . By analyzing the overall structure of an obfuscated file , an experienced programmer can always , with time , understand the meaning of the obfuscated code . For instance , in the obfuscated example in Figure 1 , one can recognize the function and guess that it implements a breadth-first search algorithm . We also expect neural networks , that excel in pattern recognition , to perform well on this task . We propose to pre-train a model to revert the obfuscation function , by training a sequence-to-sequence ( seq2seq ) model to convert obfuscated functions , where names of functions and variables have been replaced by uninformative names , back to their original forms . Suggesting proper variable and function names is a difficult task that requires to understand what the program does . In the context of source code , it is a more sensible , but also a more difficult task than MLM . Indeed , we observe ( c.f . Figure 1 ) that predicting the content of randomly masked tokens is usually quite simple , as it often boils down to making syntax related predictions ( e.g . predicting that was has been masked out is a parenthesis , a semi-column , etc. ) . These simple predictions actually provide little training signal to the model . In practice , MLM also masks out variable names , but if a given variable appears multiple times in a function , it will be easy for the model to simply copy its name from one of the other occurrences . Our model does not have this issue , as all occurrences of masked variables are replaced by the same VAR_i special tokens . In this paper , we make the following contributions : • We present DOBF , a new pre-training objective based on deobfuscation , and show its effectiveness on multiple programming languages . • We show that DOBF significantly outperform MLM ( e.g . BERT ) on multiple tasks such as code search , code summarization or unsupervised code translation . We show that pretraining methods based on DOBF outperform all existing pre-training methods on all the considered tasks . • We show that , by design , models pre-trained with DOBF have interesting applications and can be used to understand functions with uninformative identifier names . Besides , the model is able to successfully deobfuscate fully obfuscated source files . Our method improves other machine learning methods for programming languages . Automatic deobfuscation and identifier name proposal can also make code more accessible , and facilitate innovation and malware detection . Conversely , automatic deobfuscation could facilitate theft of proprietary code , therefore hindering the distribution of software and reducing investments in innovative softwares . Socially undesirable uses of our work ( e.g . intellectual property theft ) are targetable legally , while desirable ones ( e.g . malware detection , IDE tools ) can be seen as primarily technical problems . Therefore , we believe that the impact of our work will be mostly positive . 2 Related work . Masked Language Modeling pre-training . Large pre-trained transformers such as BERT Devlin et al . [ 2018 ] or RoBERTa Liu et al . [ 2019 ] led to significant improvements in the majority of natural language processing tasks . The quality of pre-training mainly comes from the MLM objective ( i.e . the cloze task ) , that allows the model to make predictions by leveraging left and right contexts , unlike causal language modeling ( CLM ) where the model predictions are only conditioned on previous words . In MLM , the model takes as input a sentence and uniformly selects 15 % of its tokens . Of the selected tokens , 80 % are replaced by a special symbol [ MASK ] , 10 % are left unchanged , and the remaining 10 % are replaced by random tokens from the vocabulary . The MLM objective consists in recovering the initial sentence given the corrupted one . Lample and Conneau [ 2019 ] noticed that the masked words are often easy to predict , and proposed to sample the 15 % masked words according to their frequencies instead of uniformly . This way , rare words are sampled more often , making the pre-training task more difficult for the model , which results in a better learning signal and faster training . Sun et al . [ 2019 ] also noticed that recovering the tokens masked by MLM is too simple in some contexts ( e.g . predicting the two tokens “ Harry Potter ” is much harder than predicting only “ Harry ” if you know the next word is “ Potter ” ) . To address this issue , they proposed to mask phrases and named entities instead of individual tokens . Joshi et al . [ 2020 ] and Song et al . [ 2019 ] made a similar observation and proposed to mask random spans of text . They showed that this simple modification improves the performance on many downstream NLP tasks . Alternative objectives . Other pre-training objectives have been proposed in addition to MLM . For instance , Devlin et al . [ 2018 ] also uses the next sentence prediction ( NSP ) objective , a binary classification task that consists in predicting whether two input sentences follow each other in the original corpus . The NSP objective was originally designed to improve the performance on downstream NLP tasks , but recent studies Lample and Conneau [ 2019 ] , Liu et al . [ 2019 ] showed that training MLM on a stream of sentences to leverage longer context , and removing the NSP objective improves the quality of pre-training . To improve the sample-efficiency of MLM ( where only 15 % of tokens are predicted ) , Electra Clark et al . [ 2020 ] proposed to replace ( and not mask ) some tokens with plausible alternatives , and to train a network to detect the tokens that have been replaced . They showed that this new Replaced Token Detection ( RTD ) objective matches the performance of RoBERTa while using four times less computational resources . Dong et al . [ 2019 ] proposed a model that combines multiple pre-training tasks , including bidirectional , but also left-to-right and right-to-left language modeling objectives . Lewis et al . [ 2019 ] also proposed different pre-training objectives , to detect whether input sentences have been permuted , tokens have been deleted or inserted , etc . Code Generation Pre-training . Recent studies showed that pre-training methods developed for natural language processing are also effective for programming languages . For instance , Feng et al . [ 2020 ] proposed CodeBERT , a RoBERTa-based model trained on source code using the MLM and RTD objectives . With GraphCodeBERT Guo et al . [ 2020 ] , the MLM objective is complemented by an edge-prediction objective , in which the model predicts edges in the data flow graph to make the model understand the structure of the code . In Jain et al . [ 2020 ] , a model is trained on javascript code using a contrastive loss ensuring that the representations are robust to some semantic-preserving transformations . They showed that their model performs well on downstream code generation tasks and outperforms previous pre-training approaches . Kanade et al . [ 2020 ] applied MLM and the next sentence prediction objectives to pre-train models on Python code . More recently , Roziere et al . [ 2020 ] applied the unsupervised machine translation principles of Lample et al . [ 2018a , b ] to monolingual source code from GitHub . They showed that the resulting model , TransCoder , was able to translate source code between Python , Java , and C++ , in a fully unsupervised way . In this paper , we propose to use a code-specific objective to better pre-train models designed to be fine-tuned on code generation tasks : code deobfuscation . Machine learning is frequently used on tasks involving programming languages , including code completion Li et al . [ 2018 ] , Liu et al . [ 2020 ] , Kim et al . [ 2020 ] , Svyatkovskoy et al . [ 2020 ] , bug detection and code repair Allamanis et al . [ 2018 ] , Wang et al . [ 2017 ] , Chen et al . [ 2019 ] , Murali et al . [ 2020 ] , Tufano et al . [ 2019 ] , Tarlow et al . [ 2020 ] , code summarization Alon et al . [ 2019a ] , Hu et al . [ 2018 ] , Xie et al . [ 2021 ] , clone detection Wei and Li [ 2017 ] , Ain et al . [ 2019 ] , Wang et al . [ 2020 ] , code search Gu et al . [ 2018 ] , Cambronero et al . [ 2019 ] and code translation Chen et al . [ 2018 ] , Roziere et al . [ 2020 ] . Most of these tasks can benefit from pre-trained models that capture the semantics of the code . Code deobfuscation . Empirical studies show that naming conventions and the use of informative identifier names make code more understandable , easier to maintain and lead to fewer bugs Takang et al . [ 1996 ] , Liblit et al . [ 2006 ] , Butler et al . [ 2009 ] . It motivated other works studying deobfuscation of identifier names and identifier name proposal using n-grams Allamanis et al . [ 2014 , 2015 ] , probabilistic models Raychev et al . [ 2015 ] , Bichsel et al . [ 2016 ] , Vasilescu et al . [ 2017 ] , Alon et al . [ 2018 ] , and recurrent neural networks Bavishi et al . [ 2018 ] , Lacomis et al . [ 2019 ] . Alon et al . [ 2018 ] extract features from Abstract Syntax Tree ( AST ) paths and train a Conditional Random Field to predict variable and method names , and infer types for several languages . DIRE Lacomis et al . [ 2019 ] uses a commercial decompiler to obtain C code with uninformative identifier names from binaries . They also use AST features , which go through a Graph Neural Network trained jointly with a LSTM model on the sequence of C tokens to retrieve relevant identifier names . More recently , David et al . [ 2020 ] used a transformer together with augmented representations obtained from static analysis to infer procedure names in stripped binary files . These models are already used to understand obfuscated and compiled source code . However , none of these studies investigated the use of deobfuscation for model pre-training .

Summary: This work follows up with earlier efforts to pre-train BERT-like architectures on code, like CuBERT and CodeBERT. Instead of using a language-agnostic masked language model pre-training objective, however, it uses a de-obfuscation objective: masking all occurrences of an identifier (variable, function, class), ask the model to predict it. This code-specific pre-training objective is combined with other recent options (e.g., a denoising autoencoder) as well as traditional mask-language modelling, to produce improved performance on a number of code tasks such as summarization, clone detection, and transpilation.