DOBF: A Deobfuscation Pre-Training Objective for Programming Languages

1 Introduction . Model pre-training with self-supervised methods such as BERT Devlin et al . [ 2018 ] , RoBERTa Liu et al . [ 2019 ] , XLM Lample and Conneau [ 2019 ] or XLNet Yang et al . [ 2019 ] , has become ubiquitous in Natural Language Processing ( NLP ) , and led to significant improvements in many tasks . These approaches are based on the Masked Language Modeling ( MLM ) objective , which consists in randomly masking words from an input text , and training a model to recover the original input . In the original approach proposed by Devlin et al . [ 2018 ] , a fraction of selected masked words is replaced by masked tokens , another is replaced by random words , and another remains unchanged . Since then , a myriad of studies have proposed to modify the MLM objective , either by masking contiguous spans of text Song et al . [ 2019 ] , Joshi et al . [ 2020 ] , masking named entities and phrases Sun et al . [ 2019 ] , sampling masked words according to their frequencies Lample and Conneau [ 2019 ] , replacing words with plausible alternatives Clark et al . [ 2020 ] , etc . Overall , most of these pre-training objectives boil down to denoising auto-encoding tasks with different methods to add noise to the input , using arbitrary noise functions . In our case , we are interested in pre-training deep learning models for programming languages . As in natural language , pre-training was shown to be effective for source code Feng et al . [ 2020 ] , Roziere et al . [ 2020 ] . However , these studies both rely on the original MLM objective proposed by Devlin et al . [ 2018 ] , which was initially designed for natural languages and does not leverage the particular structure of source code . We argue that this objective ⇤Equal contribution . The order was determined randomly . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is actually suboptimal in the context of programming languages , and propose a new objective based on deobfuscation of identifier names in source code . Code obfuscation consists in modifying source code in order to make it harder for humans to understand , or smaller while keeping its behaviour unchanged . In some ancient interpreted languages , name minimization could also reduce the memory usage of the program . Today , it is used to protect intellectual property by preventing people from understanding and modifying the code , to prevent malware detection , and to compress programs ( e.g . Javascript code ) to reduce network payload sizes . Moreover , C compilers discard variable names , and current rule-based and neural-based decompilers generate obfuscated C code with uninformative variable names Fu et al . [ 2019 ] . Obfuscators typically apply several transformations to the code . While some operations can be reversed ( e.g . dead code injection ) , the obfuscation of identifier names—renaming every variable , method and class with uninformative names—is irreversible and has a substantial impact on code comprehension Gellenbeck and Cook [ 1991 ] , Takang et al . [ 1996 ] , Lawrie et al . [ 2006 ] . By analyzing the overall structure of an obfuscated file , an experienced programmer can always , with time , understand the meaning of the obfuscated code . For instance , in the obfuscated example in Figure 1 , one can recognize the function and guess that it implements a breadth-first search algorithm . We also expect neural networks , that excel in pattern recognition , to perform well on this task . We propose to pre-train a model to revert the obfuscation function , by training a sequence-to-sequence ( seq2seq ) model to convert obfuscated functions , where names of functions and variables have been replaced by uninformative names , back to their original forms . Suggesting proper variable and function names is a difficult task that requires to understand what the program does . In the context of source code , it is a more sensible , but also a more difficult task than MLM . Indeed , we observe ( c.f . Figure 1 ) that predicting the content of randomly masked tokens is usually quite simple , as it often boils down to making syntax related predictions ( e.g . predicting that was has been masked out is a parenthesis , a semi-column , etc. ) . These simple predictions actually provide little training signal to the model . In practice , MLM also masks out variable names , but if a given variable appears multiple times in a function , it will be easy for the model to simply copy its name from one of the other occurrences . Our model does not have this issue , as all occurrences of masked variables are replaced by the same VAR_i special tokens . In this paper , we make the following contributions : • We present DOBF , a new pre-training objective based on deobfuscation , and show its effectiveness on multiple programming languages . • We show that DOBF significantly outperform MLM ( e.g . BERT ) on multiple tasks such as code search , code summarization or unsupervised code translation . We show that pretraining methods based on DOBF outperform all existing pre-training methods on all the considered tasks . • We show that , by design , models pre-trained with DOBF have interesting applications and can be used to understand functions with uninformative identifier names . Besides , the model is able to successfully deobfuscate fully obfuscated source files . Our method improves other machine learning methods for programming languages . Automatic deobfuscation and identifier name proposal can also make code more accessible , and facilitate innovation and malware detection . Conversely , automatic deobfuscation could facilitate theft of proprietary code , therefore hindering the distribution of software and reducing investments in innovative softwares . Socially undesirable uses of our work ( e.g . intellectual property theft ) are targetable legally , while desirable ones ( e.g . malware detection , IDE tools ) can be seen as primarily technical problems . Therefore , we believe that the impact of our work will be mostly positive . 2 Related work . Masked Language Modeling pre-training . Large pre-trained transformers such as BERT Devlin et al . [ 2018 ] or RoBERTa Liu et al . [ 2019 ] led to significant improvements in the majority of natural language processing tasks . The quality of pre-training mainly comes from the MLM objective ( i.e . the cloze task ) , that allows the model to make predictions by leveraging left and right contexts , unlike causal language modeling ( CLM ) where the model predictions are only conditioned on previous words . In MLM , the model takes as input a sentence and uniformly selects 15 % of its tokens . Of the selected tokens , 80 % are replaced by a special symbol [ MASK ] , 10 % are left unchanged , and the remaining 10 % are replaced by random tokens from the vocabulary . The MLM objective consists in recovering the initial sentence given the corrupted one . Lample and Conneau [ 2019 ] noticed that the masked words are often easy to predict , and proposed to sample the 15 % masked words according to their frequencies instead of uniformly . This way , rare words are sampled more often , making the pre-training task more difficult for the model , which results in a better learning signal and faster training . Sun et al . [ 2019 ] also noticed that recovering the tokens masked by MLM is too simple in some contexts ( e.g . predicting the two tokens “ Harry Potter ” is much harder than predicting only “ Harry ” if you know the next word is “ Potter ” ) . To address this issue , they proposed to mask phrases and named entities instead of individual tokens . Joshi et al . [ 2020 ] and Song et al . [ 2019 ] made a similar observation and proposed to mask random spans of text . They showed that this simple modification improves the performance on many downstream NLP tasks . Alternative objectives . Other pre-training objectives have been proposed in addition to MLM . For instance , Devlin et al . [ 2018 ] also uses the next sentence prediction ( NSP ) objective , a binary classification task that consists in predicting whether two input sentences follow each other in the original corpus . The NSP objective was originally designed to improve the performance on downstream NLP tasks , but recent studies Lample and Conneau [ 2019 ] , Liu et al . [ 2019 ] showed that training MLM on a stream of sentences to leverage longer context , and removing the NSP objective improves the quality of pre-training . To improve the sample-efficiency of MLM ( where only 15 % of tokens are predicted ) , Electra Clark et al . [ 2020 ] proposed to replace ( and not mask ) some tokens with plausible alternatives , and to train a network to detect the tokens that have been replaced . They showed that this new Replaced Token Detection ( RTD ) objective matches the performance of RoBERTa while using four times less computational resources . Dong et al . [ 2019 ] proposed a model that combines multiple pre-training tasks , including bidirectional , but also left-to-right and right-to-left language modeling objectives . Lewis et al . [ 2019 ] also proposed different pre-training objectives , to detect whether input sentences have been permuted , tokens have been deleted or inserted , etc . Code Generation Pre-training . Recent studies showed that pre-training methods developed for natural language processing are also effective for programming languages . For instance , Feng et al . [ 2020 ] proposed CodeBERT , a RoBERTa-based model trained on source code using the MLM and RTD objectives . With GraphCodeBERT Guo et al . [ 2020 ] , the MLM objective is complemented by an edge-prediction objective , in which the model predicts edges in the data flow graph to make the model understand the structure of the code . In Jain et al . [ 2020 ] , a model is trained on javascript code using a contrastive loss ensuring that the representations are robust to some semantic-preserving transformations . They showed that their model performs well on downstream code generation tasks and outperforms previous pre-training approaches . Kanade et al . [ 2020 ] applied MLM and the next sentence prediction objectives to pre-train models on Python code . More recently , Roziere et al . [ 2020 ] applied the unsupervised machine translation principles of Lample et al . [ 2018a , b ] to monolingual source code from GitHub . They showed that the resulting model , TransCoder , was able to translate source code between Python , Java , and C++ , in a fully unsupervised way . In this paper , we propose to use a code-specific objective to better pre-train models designed to be fine-tuned on code generation tasks : code deobfuscation . Machine learning is frequently used on tasks involving programming languages , including code completion Li et al . [ 2018 ] , Liu et al . [ 2020 ] , Kim et al . [ 2020 ] , Svyatkovskoy et al . [ 2020 ] , bug detection and code repair Allamanis et al . [ 2018 ] , Wang et al . [ 2017 ] , Chen et al . [ 2019 ] , Murali et al . [ 2020 ] , Tufano et al . [ 2019 ] , Tarlow et al . [ 2020 ] , code summarization Alon et al . [ 2019a ] , Hu et al . [ 2018 ] , Xie et al . [ 2021 ] , clone detection Wei and Li [ 2017 ] , Ain et al . [ 2019 ] , Wang et al . [ 2020 ] , code search Gu et al . [ 2018 ] , Cambronero et al . [ 2019 ] and code translation Chen et al . [ 2018 ] , Roziere et al . [ 2020 ] . Most of these tasks can benefit from pre-trained models that capture the semantics of the code . Code deobfuscation . Empirical studies show that naming conventions and the use of informative identifier names make code more understandable , easier to maintain and lead to fewer bugs Takang et al . [ 1996 ] , Liblit et al . [ 2006 ] , Butler et al . [ 2009 ] . It motivated other works studying deobfuscation of identifier names and identifier name proposal using n-grams Allamanis et al . [ 2014 , 2015 ] , probabilistic models Raychev et al . [ 2015 ] , Bichsel et al . [ 2016 ] , Vasilescu et al . [ 2017 ] , Alon et al . [ 2018 ] , and recurrent neural networks Bavishi et al . [ 2018 ] , Lacomis et al . [ 2019 ] . Alon et al . [ 2018 ] extract features from Abstract Syntax Tree ( AST ) paths and train a Conditional Random Field to predict variable and method names , and infer types for several languages . DIRE Lacomis et al . [ 2019 ] uses a commercial decompiler to obtain C code with uninformative identifier names from binaries . They also use AST features , which go through a Graph Neural Network trained jointly with a LSTM model on the sequence of C tokens to retrieve relevant identifier names . More recently , David et al . [ 2020 ] used a transformer together with augmented representations obtained from static analysis to infer procedure names in stripped binary files . These models are already used to understand obfuscated and compiled source code . However , none of these studies investigated the use of deobfuscation for model pre-training .

Summary: This paper produces a new pretraining objective for models that provide program embeddings via the use of a name deobfuscation objective. This objective has advantages over the standard MLM objective as it excludes obvious masking possibilities and also does not allow the model to copy the names of one token from part of the document to another part.  The authors construct this objective and demonstrate that their model can learn this task fairly effectively. Interestingly, initializing their objective with a model pretrained on MLM seems to aid the performance of their model. They also demonstrate improvement over some baselines on some standard program processing objectives, and improvement over all baselines using a combination of their objective and a separate program recovery objective.