DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
Authors: Marie-anne Lachaux
Baptiste Roziere
Marc Szafraniec
Guillaume Lample
Venue: NeurIPS 2021 Poster (2021)
Keywords: code
obfuscation
programming languages
pre-training
deobfuscation
ML for code
ML for Programming Languages

Abstract: Recent advances in self-supervised learning have dramatically improved the state of the art on a wide variety of tasks. However, research in language model pre-training has mostly focused on natural languages, and it is unclear whether models like BERT and its variants provide the best pre-training when applied to other modalities, such as source code. 
In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original version of obfuscated source code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks, providing relative improvements of up to 12.2% in unsupervised code translation, and 5.3% in natural language code search. Incidentally, we found that our pre-trained model is able to deobfuscate fully obfuscated source files, and to suggest descriptive variable names.

DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
Marie-Anne Lachaux* Facebook AI Research
Baptiste Roziere" Facebook AI Research
Mare Szafraniec Facebook AI Research szaframiec fb.com
malachaux @ fb.com
Paris- Paris-Dauphine University
broz@fb.com
Guillaume Lample
Facebook AI Research
glample fb.com
Abstract
Recent advances in self-supervised learning have dramatically improved the state of the art on a wide variety of tasks. However, research in language model pre- training has mostly focused on natural languages, and it unclear whether models like BERT and its variants provide the best pre-training when applied to other modalities, such as source code. this paper, we introduce new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains model recover the original version of obfuscated source code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks, providing relative improvements up to 12.2% in unsupervised code translation, and 5.3% in natural language code search. Incidentally, we found that our pre-trained model able to deobfuscate fully obfuscated source files, and to suggest descriptive variable names.
1 Introduction
Model pre-training with self-supervised methods such BERT Devlin et al. 2018 RoBERTa Luu
et al. 2019), XLM Lample and Conneau [209 or XLNNe Yang et al. 2019). has become ubiquitous in Natural Language Processing (NLP), and led to significant improvements in many tasks. These approaches are based on the Masked Language Modeling (MLM) objective, which consists in randomly masking words from input text, and training model to recover the original input. In the original approach proposed by Devlin et al. (2018), fraction of selected masked words is replaced by masked tokens, another is replaced by random words, and another remains unchanged. Since then, a myriad of studies have proposed to modify the MLM objective, either by masking contiguous spans of text [Song et al. 2019 Joshi et al. [2020]. masking named entities and phrases Sun et al. 2019]. sampling masked words according to their frequencies Lample and Conneau 2019) replacing words with plausible alternatives Clark et al. (2020). etc. Overall, most of these pre-training objectives boil down to denoising auto-encoding tasks with different methods to add noise to the input, using arbitrary noise functions. our case, we are interested in pre- pre-training deep learning models for programming languages. As in natural language, pre-training was shown to be effective for source code Feng et al. 22200), Roziere et al. [2020] However, these studies both rely on the original MLM objective proposed by Devlin et al. 2018 which was initially designed for natural languages and does not leverage the particular structure of source code. We argue that this objective
"Equal contribution. The order was determined randomly.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

is actually suboptimal in the context of programming languages, and propose a new objective based on deobfuscation of identifier names in source code.
Code obfuscation consists in modifying source code in order to make it harder for humans to understand, or smaller while keeping its behaviour unchanged. In some ancient interpreted languages, name minimization could also reduce the memory usage of the program. Today, it is used to protect intellectual property by preventing people from understanding and modifying the code, to prevent malware detection, and to compress programs (e.g. Javascript code) to reduce network payload sizes. Moreover, compilers discard variable names, and current rule-based and neural-base decompilers generate obfuscated C code with uninformative variable names Fu et al. (2019). Obfuscators typically apply several transfornations to the code. While some operations can be reversed (e.g. dead code injection). the obfuscation of identifier names renaming every variable, method and class with uninformative names- is irreversible and has a substantial impact on code comprehension Gellenbeck and Cook 1991], Takang et al. 1996]. Lawrie et al. [2006]
By analyzing the overall structure of an obfuscated file, an experienced programmer can always, with time, understand the meaning of the obfuscated code. For instance, in the obfuscated example in Figure D one can recognize the function and guess that implements breadth-fistt search algorithm. We also expect neural networks, that excel in pattern recognition, to perform well on this task. We propose to pre-train model to revert the obfuscation function, by training (seq2seq) model to convert obfuscated functions, where names of functions and variables have been replaced by uninformative names. back their original forms. Suggesting proper variable and function names is a difficult task that requires to understand what the program does. In the context of source code, it is a more sensible, but also a more difficult task than ML.. Indeed, we observe (c.f. Figure TT that predicting the content of randomly masked tokens is usually quite simple, as it often boils down to making syntax related predictions (e.g. predicting that was has been masked out is a parenthesis, a semi-column, etc.). These simple predictions actually provide little training signal to the model. In practice, MLM also masks out variable names, but if given variable appears multiple times in function, it will be easy for the model to simply copy its name from one of the other occurrences. Our model does not have this issue, as all occurrences of masked variables are replaced by the same VAR_i i special tokens.
sequeence-o-sequeeee
In this paper, we make the following contributions:
We present DOBF, new pre-training objective based on deobfuscation, and show its effectiveness on multiple programming languages.
We show that DOBF significantly outperform MLM (e.g. BERT) multiple tasks such as code search, code summarization or unsupervised code translation. We show that pre- training methods based on DOBF outperform all existing pre-training methods on all the considered tasks.
We show that, by design. models pre-trained with DOBF have interesting applications and can be used to understand functions with uninformative identifier names, Besides, the model is able to successfully deobfuscate fully obfuscated source files.
Our method improves other machine learning methods for programming languages, Automatic deob- fuscation and identifier name proposal can also make code more accessible, and facilitate innovation and malware detection. Conversely, automatic deobfuscation could facilitate theft of proprictary code, therefore hindering the distribution of software and reducing investments in innovative softwares. Socially undesirable uses of our work (e.g. intellectual property theft) are targetable legally, while desirable ones (e.g. malware detection, IDE tools) can be seen as primarily technical problems. Therefore, we believe that the impact of our work will be mostly positive.
2 Related work
Masked Language Modeling pre-training. Large pre-trained transformers such as BERT Devlin et al. 2018] or RoBERTa Liu et al. (2099) led to significant improvements in the majority of natural language processing tasks. The quality of pre-training mainly comes from the MLM objective (i.e. the cloze task), that allows the model to make predictions by leveraging left and right contexts, unlike causal language modeling (CLM) where the model predictions are only conditioned on previous words. In MLM, the model takes input sentence and uniformly selects 15% of its tokens. Of the

---

MUM
MLM output
b (MASK] root::
visited [MASK]
[MASK] [root]
queue
[MASK] outue:
while
MLM
[M6SK] qucue. pople)
Input codle
neighbor [neak] [node]:
graph
def bosrg,,,,, rone]:
[MASK] ne neightor not in vis teds
if
visitedd. he reigoorrrii (MASK]
queue [root]
queue
while queue:
(MASK] visited
return
poplo)
neighaor in praph [
DOBF Input
DOBF output
if neighbor not in visited:
def FQ1A0, Vi
visited append neionbor))
quaue, append (neighbo))))))
[V1]
V0
graph
return ited
DOBF
While va:
V3.pople)
V1
virited
root
for pe]ss]:
queue
11 ve not VB:
V4
neiigbbr
V2-appendr
VS
node
(V.s
return V2
Illustration the MLM and DOBF objectives. Given an input function, the masked language modeling (MLM) task randomly samples tokens to mask out. With source code, a large fraction of these tokens are related to the language syntax (e.g. commas, parentheses, etc.) that are trivial for the model to predict, and provide poor training signal. Instead, we propose obfuscate the code by masking the name of functions and variables, and to train the model to recover the original function by deobfuscating the code (DOBF). When a variable is masked out, we mask all occurrences of this variable with the same mask symbol (e.g. all occurrences of "visited" are replaced by "VO") to prevent the model from copying names. The DOBF objective is more difficult and provides better learning signal.
Figure 1:
selected tokens, 80% are replaced by a special symbol [MASK], 10% are left unchanged, and the remaining 10% are replaced by random tokens from the vocabulary. The MLM objective consists in recovering the initial sentence given the corrupted one. Lample and Conneau 2019 noticed that the masked words are often easy to predict, and proposed sample the 15% masked words according to their frequencies instead of uniformly. This way, rare words are sampled more often, making the pre-training task more difficult for the model, which results in a better learning signal and faster training. Sun et al. [2019] also noticed that recovering the tokens masked by MLM is too simple in some contexts (e.g. predicting the two tokens "Harry Potter" is much harder than predicting only "Harry" if you know the next word is "Potter"). To address this issue, they proposed to mask phrases and named entities instead of individual tokens. Poshi et al.) (2020) and Soog et al.. [2019] made a similar observation and proposed to mask random spans of text. They showed that this simple modification improves the performance on many downstream NLP tasks.
Alternative objectives. Other pre-training objectives have been proposed in addition to MLM. For instance, Devlin et al. 2018) also uses the next sentence prediction (NSP) objective, a binary classification task that consists in predicting whether two input sentences follow each other in the original corpus. The NSP objective was originally designed to improve the performance on downstream NLP tasks, but recent Studies[ampp and Conneau [019], Liu al. (219) showed that training MLM on stream of sentences to leverage longer context, and removing the NSP objective improves the quality of pre-training. To improve the cample-efficienc, of MLM (where only 15% of tokens are predicted), Electra Clark et al. (020 proposed to replace (and not mask) some tokens with plausible alternatives, and to train a network detect the tokens that have been replaced. They showed that this new Replaced Token Detection (RTD) objective matches the performance of RoBERTa while using four times less computational resources. Dong et al. 2019 proposed a model that combines multiple pre-training tasks, including bidirectional, but also left-to-right and right-to-left language modeling objectives. Lewis et al. (2019) also proposed different pre-training objectives, to detect whether input sentences have been permuted, tokens have been deleted or inserted, etc.
Code Generation Pre-training. Recent studies showed that pre-training methods developed for natural language processing are also effective for programming languages. For instance, Feng et al. [2200] proposed CodeBERT, a RoBERTa-based model trained on source code using the MLM and RTD objectives. With GraphCodeBERT Guo et al. 2020). the ML.M objective complemented by an edge-prediction objective, in which the model predicts edges in the data flow graph to make the
3

---

model understand the structure of the code. In Jain et al. 2020), model is trained on javascript code using contrastive loss ensuring that the representations are robust to some cemantic-preservvee
transformations. They showed that their model performs well on downstream code generation tasks and outperforms previous pre-training approaches. Kanade et al. [020] applied MLM and the next sentence prediction objectives to pre-train models on Python code. More recently, Roziere et al. 2020] applied the unsupervised machine translation principles of Lample et al. 22118a bJ to monolingual source code from GitHub. They showed that the resulting model, TransCoder, was able to translate source code between Python, Java, and C++, in fully unsupervised way. In this paper, we propose use code-specific objective to better pre-train models designed to be fine-tuned on code generation tasks: code deobfuscation. Machine learning is frequently used on tasks involving programming languages, including code completion Li et al. [2018]. [iu et al. [2020). 4 Kim et al.
[2020] Svyatkovskoy et al. [2020], bug detection and code repair Allamanis et al. 20188, Wang
et at. 2017]. Chen et al. [ Murali et al. 2020 Tufano al. (2019), Tarlow al. [2020] code
summarization Alon et al. [201 9a], Hu et al. 2018 Xie et al. [2021], J, clone detection Wei and Li
2017]. Ain al. 2019). Wang al. [2020]. , code search Gu et al. (018]] Cambronero et al. 2019
and code translation Chen et al. 22018). Roziere et al. [2020]. Most of these tasks can benefit from
pre-trained models that capture the semantics of the code.
Code deobfuscation. Empirical studies show that naming conventions and the use of informative identifier names make code more understandable, easier to maintain and lead to fewer bugs Takang
et al. [1996] Liblit et al. [2006] Butler et al. 22099. It motivated other works studying deobfuscation
of identifier names and identifier name proposal using n-grams Allamanis et al. [2014] [013],
probabilistic models Raychev et al. [2015] Bichsel et al. [2016] Vasilescu et al. [2017 7J. Alon et al.
(0188) and recurrent neural networks Bavishi et al. [2018]. Lacomis et al. 2019). Alon et al.
2018 extract features from Abstract Syntax Tree (AST) paths and train a Conditional Random
Field to predict variable and method names, and infer types for several languages. DIRE Lacomis
al. 2019 uses a commercial decompiler obtain C code with uninformative identifier names from binaries. They also use AST features, which go through a Graph Neural Network trained jointly with a LSTM model on the sequence of C tokens to retrieve relevant identifier names. More recently, David et al. 2020 used transformer together with augmented representations obtained from static analysis to infer procedure names in stripped binary files. These models are already used to understand obfuscated and compiled source code. However, none of these studies investigated the use of deobfuscation for model pre-training.
3 Model
3.1 ML. and denoising for Programming Languages
countless number of pre- tre-training objectives have been introduced in the literature Devlin al. [2018). Clark et al. [2020). Lewis et al. [2019] Liu et al. (2019), Dong et al. (019). Most of them rely on hyper-parameters and seemingly arbitrary decisions (Should we mask individual tokens or spans? Which fraction of them? What do we do with masked out tokens? etc.). These choices are typically based on intuition and validated empirically on natural language processing tasks. However, source code is much more structured than natural language, which makes predicting masked tokens much easier for programming languages.
The first row in Figure [ shows an example of input/ / output for the MLM objective. We can see that the majority of tokens are composed of Python keywords or symbols related to syntax: [ while if return. These symbols are easy to recover, and a model will quickly learn to predict them with perfect accuracy. This effect accentuated by the verbosity of the language. For instance, we would see significantly more of these tokens in Java. Retrieving the obfuscated graph token also relatively simple: the model only needs to retrieve the most relevant variable the scope. More generally, retrieving an identifier name is often easy when given its full context, including its definition and usages. The denoising-autee auto-encoding (DAE) objective Vincent et al. [2008]. which trains an decoder model to retrieve masked token and recover randomly modified input sentences, is quite similar to MLM and the model can also retrieve identifier names easily by finding their definition or usages. Overall, we suspect that the MLL objective is too simple in programming languages and we introduce a new objective, DOBF, which encourages the model to learn a deeper understanding of code semanties.

---

3.2 Deobfuscation Objective
Instead of MLM, we propose a new pre- vre-tranning objective, DOBF, that leverages the particular structure of programming languages. We obfuscate code snippets by replacing class, function and variable names with special tokens, and train a model recover the original names. When an identifier is selected, all of its instances in the code are replaced by the same special token. This differs from MLM where the name of a variable can appear multiple times while being masked single time. For instance, in Figure | DOBF will replace the two occurrences of node by the same symbol V5, while MLM will only mask one of these occurrences. As result, the fraction of meaningful tokens masked by the objective is language independent: for more verbose languages (e.g. Java), the less informative syntax-related tokens will not be masked out by the DOBF objective.
Each identifier is replaced with probability Pobs E [0, 1]. We ensure that the original input is modified: if no identifier is replaced, we draw random one to obfuscate. When Pobs 0, we always obfuscate exactly one random identifier in the input. When Pobs = 1, we obfuscate all the identifiers defined in the file. We ensure that the obfuscated code has the same behavior as the original. The second row in Figure T shows an example of obfuscated code with Poef = 1, where we obfuscate a function bfs which implements a breadth-fing search. The function append is not obfuscated as is a standard Python function not defined in the file. The model is given the obfuscated code as input and has to restore the original name of each special token CLASS_ i, FUNC_ and VAR_i. i. In other words, the model needs to output dictionary mapping special tokens to their initial values.
Finding informative names for obfuscated identifiers requires the model learn deep understanding of code semantics, which is desirable for a pre-training task. MLM will mask only some of the occurrences of the identifiers and leave the other ones unchanged so that the model can simply copy identifier names. In Figure I1 with MLM masking, the model can simply notice that a variable named queue called on the fourth line. Since the variable not defined, the model can easily guess that queue has to be defined on the third line, and infer the value of the corresponding [MASK] token. With the deobfuscation objective, the model needs to analyze code patterns and understand the semantics of the variable to infer that, since its elements are popped with pop (0), the variable v3 implements a queue. If its elements were popped with () our model would name stack instead of queue (c.f. Figure 7 in the appendix).
3.3 Implementation
Overall, the deobfuscation objective operates like a supervised machine translation objective, where a seq2seq model is trained to map an obfuscated code into dictionary represented as a sequence of to- kens. inference time, the model able to suggest meaningful class, function and variable names for piece of code with an arbitrary number of obfuscated identifiers. Obfuscated classes, functions, and variables, are replaced with associated special tokens: CLASS ... CLASS N, FUNC_O FUNC_N and VAR_O. ... VAR_N. We serialize the output dictionary as a sequence of tokens where the entries are separated by a delimiter symbol 1.
Experiments
We train DOBF with the deobfuscation objective. First, evaluate our model on two straightforward deobfuscation applications. Then, we show its performance on multiple downstream tasks.
4.1 Deobfuscation
We evaluate model on two applications of the deobfuscation task: when Potf = 0 (the model has to retrieve a single identifier name), and Pobs = 1 (the model has to retrieve all the identifier names).
Deobfuscating single identifier When Pobs O, only identifier is obfuscated. In that case, the model has to propose a relevant name for that identifier using the rest of the non-obfuscated file context. It can be used as tool that suggests relevant variable names. Integrated development environments (e.g. PyCharm, VSCode) already perform this task, often using handcrafted rules.
in the obfuscated example given in Figure 1 the model is trained to generate: FUNC_O bfs VAR_0 graph VAR_1 root VAR_2 visited VAR_3 queue VAR_4 neighbor VAR_5 node.
5

---

Deobfuscating all identifiers Obfuscators are commonly used to make code smaller and more efficient or to protect it by making more difficult understand and reuse. They typically apply several transformations, one of them being to replace every identifier name with short and uninfor- mative names (e.g. a, b, c). our work, such a transformation corresponds to obfuscating file with Pobf = 1. To measure our model's ability to revert the obfuscation operation, we evaluate its accuracy when obfuscating all identifier names, Another application would be to help understand source code written with uninformative variable names.
Evaluation metric We evaluate the ability of our model to retrieve identifier names from the original non-obfuscated code. We report the accuracy, which is the percentage of recovered tokens that exactly match the ground truth. Following previous works vilamanis al. 2015 2016], Alon et al. 2018 2019b), we also report the subtoken score, more flexible metric which computes the precision, recall, and F1 scores for retrieving the original aase-insensitive subtokens. Each token is broken into subtokens using uppercase letters for camlCase and underscores for snake_case. For instance, decoderAttention would be considered to be perfect match for decoder_attentior attentioo or attentionDecoder. attention would have perfect precision but recall of 0.5, so a F1 score of 66.7. crossAttentionDecoo to F1 score of 80.0. We compute the overall subtoken precision, recall and F1 scores averaged over each file in our validation and test datasets.
would have a perfect recall but a precision of 5. corresponding
4.2 Fine-tuming on downstream tasks
In order to evaluate DOBF as a pre-training model, we fine-tune DOBF on TransCoder and on three tasks from CodeXGLUE Lu all 202T]. benchmark for programming languages. The data, code and models from CodeXGLUE and TransCoder are available respectively under the MIT and the Creative Commons license. We only consider the Java and Python tasks with encoder in the model architecture for which the training, validation, and test sets are publicly available.
CodeXGLUE Clone Detection This task is binary classification problem where the model has to predict whether two code snippets are semantically equivalent. It is evaluated using the macro Fl score. The model is composed of a single encoder and classification layer. An input consists in two snippets of code, which are concatenated before being fed to the model. This task available in Java.
CodEX Code Summarization Given code snippet, the model is trained to generate the corresponding documentation natural language. The architecture a sequence-to-sequene transformer model evaluated using BLEU score Papineni et al. 2002). The dataset includes both Java and Python source code.
CoUEXGLUE NL Code Search Given code search query in natural language the model has to retrieve the most semantically related code within a collection of code snippets. This is a ranking problem evaluated using the Mcan Reciprocal Rank (MRR) metric. The model is composed of two encoders. The natural language query and the code are encoded separately, and we compute the dot product between the first hidden states of the encoders' last layers. This task is available in Python.
TransCoder
TransCoder Roziere al. 2020 is unsupervised machine translation model which translates functions and methods between C++, Java, and Python. A single seq2seq model is trained for all languages. In the original work, TransCoder is pre-trained with MLM, and trained with denoising auto-encoding and back-translation. TransCoder is evaluated using the Computational Accuracy metric, which computes the percentage of correct solutions according to series of unit tests. We only consider a single model output (CA @ I), with beam sizes of and 10.
4.3 Experimental details
Model Architecture We consider seq2seq model with attention, composed of an encoder and
decoder using a transformer architecture Vaswani et al. (2017). We train models with the same
architecture and tokenizer as CodeBERT Feng et al. [2020] and GraphCodeBERT (uuo et al. 2020
in order to provide fair comparisons: 12 layers, 12 attention heads and a hidden dimension of 768. We also train model with the same parameters as TransCoder (see Figure 4 in the Appendix).
Training dataset As in Roziere et al. 2200). we use the GitHub public dataset available on Google BigQuery and select all Python and Java files within the projects with licenses authorizing use for

---

def FUNC_ OVARR O, VAR_1):
def (gs start);
VAR_2 [VAR_[]
visited [start]
VAR_3 [VAR_1]
quaue [start]
shile VAR_3:
while queue:
VAR_4 VARL 3
mode queue.poptee po)(0)
VAR_5 [AR__[[AR_A] 4]
for neighbor in graph [node]:
if (VAR_5 not in 2)R
if (neighbor not in vieited): visited add (dd(nigghbor)
V. add(VAR
VAR_3. AVAR_)
queue. (naighbor)
roturn 2
return Visited
Figure 2: Full deobfuscation of a breadth-firt-searde function by DOBF. The code on top has been fully obfuscated. The code on the bottom was recovered using DOBF by replacing the function name and every variable name using the generated dictionary. DOBF is able to suggest relevant function and variable names. It makes the code much more readable and casier to understand.
research purposes. Following Lopes al. 2017] and Allamanis 2019, we remove duplicate files. We also ensure that each fork belongs to the same split as its source repository. We obfuscate each file and create the corresponding dictionary of masked identifier names, resulting in a parallel (obfuscated file - dictionary) dataset of 19 GB for Python and 26 GB for Java. We show some statistics about this dataset in Table 3 in the appendix. For comparison purposes, we apply either the BPE codes used by Roziere et al. 2020)] or by Feng et al. 2020]. In practice we train only on files containing less than than 90% and 80% of the Java and Python files respectively.
2000 tokens, which corresponds
Training details We train DOBF to translate obfuscated files into lists of identifier names. During DOBF training. we alternate between batches of Java and Python composed of 3000 tokens per GPU, We optimize DOBF with the Adam optimizer Kingma and Ba 2014] and an inverse square-root learning rate scheduler Vaswani al. 2017]. We implement our models in PyTorch Paszke et al. [219]] and train them on 32 VI0O GPUs for eight days. We use float16 operations to speed up training and to reduce the memory usage of our models. We try different initialization schemes: training from scratch and with a Python-Jave MLM model following Roziere et al. [2020]. J. We train DOBF with three different obfuscation probability parameters: Pobf {0, 0.5, 1). For each Pobf value, train models with multiple initial learning rates ranging from 10 to 3.10 and select the best one using the average subtoken Fl score computed on the validation dataset.
Fine-tuning details Depending on the fine-tuning tasks, we consider different model architectures: seq2seq models with encoder and decoder, architectures with two encoders or single encoder. In all cases, we initialize the encoders of these models with the encoder of DOBF and fine-tune all parameters. For fair comparison, we rerun all baselines, and train models with the same architectures, number of GPUs, batch sizes and optimizers. For CodeXGLUE,, we noticed that the tasks are quite sensitive to the learning rate parameter used during fine-tuning. We perform grid search on five learning rate parameters ranging from 5.10-6 to 10 and select the best parameter validation dataset. For TransCoder, we use a learning rate of 10 as in Roziere et al. 2020 and we train the models for 2 day on 32 Tesla V100 GPUs.
the
Results
5.1 Deobfuscation
In Table I. we evaluate the ability of our model to recover identifier names, either when only one identifier obfuscated (pobs = 0) or when all identifiers are obfuscated (pors = 1), for models trained with Potf E {0, 0.5, 1). Even when evaluating with Po6f = O, training with Poaf = 0 is less efficient than Pobff = 0.5 since the model is only trained to generate a single variable for each input sequence. Training with Pobf = 0.5 is more difficult task that requires the model to learn and understand more about code semantics. Forcing the model to understand the structure of the code may be useful even when testing with Pot O, as some identifier names cannot be guessed only from the names of other identifiers. When DOBF has to recover a fully obfuscated function, it obtains the best accuracy when trained with Pobf 1. It manages to recover 45.6% of the initial identifier names. We also observe that, for every configuration, initializing DOBF with MLM improves the performance.
Figure 2 shows example of a fully obfuscated function recovered by our model. DOBF successfully manages to understand the purpose the function and to predict appropriate variable names. Figure shows examples of function name proposal by DOBF for functions implementing matrix operations in
7

---

Input Code
Function Name Proposals
def FUNC_O (m1, m2): shape abape
matrix_sd
25.9%
ml shape rango(m)] for range(a))
matris Add
22.5%
matrixadd
18.8%
for 1 in (n):
matrix sum matrix, audittion
16.7%
for range(n):
res[il mi[:][]] m2[4]
return res
def FURC_O (matrix):
transpose rotate minte matrix
36.7%
natrix. shape
29.5%
for 1 in rango(n):
17.1%
ran))
symmetric
matrix []],, matrix(;] [atrix(j][t), matrix[s][]]
rotate_matiin hac___marri__________ bow
7.7%
def PUNC_O (m1, m2): n1,
shape
n2, m2 ahape sesert n2 ml
matrix product
28.8%
[[0 for in (m2)) for in range (ai)) matmal mat matprod
17.0%
for i in rango(nt):
for ranga(n2):
matrixProduct
14.4%
ren [5]
sun (in [k] m2 [j]
range (n2)1)
return rea
Figure 3: Additional examples function name proposals for matrix operations Python. DOBF able to find the right name for each matrix operation, showing that it learned to attend to the most important parts of the code. Even when the functions are similar, DOBF successfully and confidently (c.f. scores) understands the semantics of the function and its purpose.
Table 1: Results on partial and full deobfuscation. Token accuracy and subtoken F1 score of DOBF evaluated with Posy (i.e. name proposal. where single token is obfuscated) and Pats (i.e. full deobfuscation. where all tokens are obfuscated). We consider models trained with different obfuscation probabilities Pokf DOBFu.s performs well for both tasks, and even performs better than DOBF, for Identifier Name Proposal. DOBFu and DOBF; perform poorly when evaluated on other Potf parameters. Pre-training DOBF with MLM further improves the performance.
Eval Pabr
Eval Pabr
Acc
F1
Acc
F1
DOBF,
56.3
68.0
0.4
0.9
DOBFo.5
61.1
71.2
41.8
54.8
DOBF;
18.1
27.0
45.6
58.1
DOBFo.6 init MLM
67.6
76.3
45.7
58.0
DOBF, init MLM
20.0
28.3
49.7
61.1
Python. We observe that DOBF manages to identify the key tokens and properly infer the purpose of similar but very different functions. Figures and in the appendix show additional examples of function name proposals by DOBF in Java and Python. Figure 7 in the appendix shows additional examples where we show that DOBF also leverages non-obfuscated identifier names to understand the appendix show examples of deobfuscation
the meaning of input functions. Figures and of fully obfuscated Python code snippets using DOBF. It is able to understand the semantics and purposes of a variety of obfuscated classes and functions, including LSTM cell.
5.2 Downstream tasks
Our results on downstream task using the same architecture as CodeBERT and GraphCodeBERT are shown in Table and discussed below. Our results using the architecture of TransCoder are shown on Table in the Appendix. For fine-tuning, we considered models pre-trained with Pobs 0.5

---

Table 2: Results on downstream tasks for different pre-training configurations. Models pre-trained with DOBF initialized with MLM significantly outperform both CodeBERT and models trained with MLM only. DOBF+DAE outperforms other models on every task but clone detection, on which CodeBERT scores much higher than our MLM. It outperforms GraphCodeBERT by 0,02 MRR (+5.3%) on natural language code search (NLCS)), and by 4.6% in Java Python computational accuracy with beam size 10 (+12.2% correct translations). The tasks where ML.M provides large improvements over the transformer baseline (first row, pre-training) also the tasks where DOBF provides the largest gains (clone detection, NI. code search, unsupervised translation). The DAE baseline (initialized with MLM) already provides substantial improvements over MLM on most tasks and yields the best results for Python to Java translation while its results are poor for Java to Python.
Clone Det (FI score)
Code Sum Java (BLEU)
Code Sum Python (BLEU)
NLCS (MRR)
Python JJav (CA@
Java Java-Pyyhoon
(CA
k=l
k=10
k=l
k=10
Transformer
88. 14
16.58
16.43
0.025
24.0
28.4
29.0
29.7
MLM
91.89
18.59
17.95
0.308
44.8
45.4
34.5
35.6
DAE
96.30
19.19
18.28
0.380
48.3
49.2
32.1
32.8
CodeBERT
96.50
18.25
18.22
0.315
40.8
45.6
36.5
36.7
GraphCodeBEKT
96.38
18.78
18.51
0,377
44.3
44.1
35.6
37.8
DOBF init scratch
96.52
18.19
17.51
0.272
43.9
44.1
35.2
34.7
DOBF
95.87
19.05
18.24
0.383
43.5
44.1
38.7
40.0
DOBF+DAE
95.82
19.36
18.58
0.397
46.6
47.3
40.6
42.4
and Pobs 1. Since they gave very similar results on downstream tasks, we only use models pre-trained with Poaf 0.5 in the rest of the paper. We initialize DOBF with MLM as it leads to better performance on our deobfuscation metrics. We still consider DOBF initialized randomly as baseline in Table 2 We also consider version where DOBF is trained together with a denoising auto-encoding (DAE) objective Vincent et al. 2008). which was shown to be effective at learning code representations in Roziere al. 2020). With DAE, the model is trained to recover the original version of a sequence which has been corrupted (by removing and shuffling tokens). As basclines, consider a randomly initialized model and model pre-trained with MLM only, and model pre-trained with denoising and initialized with MLM. For CodeXGLUE tasks, we also consider CodeBERT baseline. We compare results for DOBF trained from scratch and DOBF initialized with MLM, and report results in Table 2 The randomly initialized model is useful to measure the importance of pre-training on a given task. Pre-training particularly important for the NLCS task: without pre-training, the model achieves a performance of 0.025 MMR while goes up to 0.308 with MLM pre-training. The main differences between our MLM baseline and Code are that I) CodeBERT was trained on a different dataset which contains functions with their documentation, 2) uses an additional RTD objective, and 3) is initialized from a RoBERTa model. Although code summarization and NL code search involve natural language and may benefit from CodeBERT's dataset that contains code documentation, we obtained very similar results on this task using simpler dataset. However, our MLM baseline did not match their performance on clone detection. We also tried initialize our MLM model with RoBERTa, but did not observe any substantial impact on the performance on downstream tasks.
The models based DOBF obtain stateoff-the-ar results all downstream tasks, outperforming GraphCodeBERRT, CodeBERT and ML.. The deobfuscation objective is already effective as a pre-training task. Even when initialized randomly. it leads results comparable MLM on most tasks and is much more effective on clone detection. The DOBF+DAE model outperforms MLM on all downstream tasks, the major improvement being for NL code search, which is also the task that benefited the most from MLM pretraining For unsupervised translation, DOBF+DAE increases the computational accuracy by 1.9% when translating from Python to Java, and by 6.8% when translating from Java to Python with beam size 10. Also, DOBF beats CodeBERT by a wide margin on NL code search and code summarization, showing that programming language data aligned with natural language is not necessary to train effective model on those tasks. DOBF initialized with MLM and combined with DAE yields higher scores than both DOBF alone and MIM, on most tasks. It shows that objectives such MLM and DAE that provide unstructured noise are complementary to DOBF.

---

6 Conclusion
In this paper, we introduce a new deobfuscation objective and show that can used for three purposes: recover fully obfuscated code, suggest relevant identifier names, and pre-train transformer models for programming language related tasks. Although it does not require any parallel corpora of source code aligned natural language, methods based on DOBF outperform BRR,, CodeBERT and MLM pre-training on multiple downstream tasks, including clone detection, code summarization, natural language code search, and unsupervised code translation. These results show that DOBF leverages the particular structure of source code to add noise to the input sequence in a particularly effective way. Other noise functions or surrogate objectives adapted to source code may improve the performance further. For instance, by training model to find the type of given variables, the signature of a method, or repair piece of code which has been corrupted.
Since models pretrained on source code benefit from structured noise, it would be interesting to see whether these findings can be applied to natural languages well. Although ambiguous, natural languages also have an underlying structure. Leveraging the constituency or dependency parse trees of sentences (as opposed abstract syntax trees in programning languages) may help designing better pre-training objectives for natural languages.
Funding Disclosure
The authors of this paper are employed by Facebook France and this work was done using hardware and software provided by Facebook.
References
Qurat UI Ain, Wasi Haider Butt, Muhammad Waseem Anwar, Farooque Azam, and Bilal Maqbool. A systematic review on code clone detection. IEEE Access, 7:86121-866 1-114444444 2019.
Miltiadis Allamanis. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, pages 143-153, 2019.
Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. Learning natural coding con- ventions. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 281 281-293, 2014.
Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. Suggesting accurate method and class names. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pages 38-49, 2015.
Miltiadis Allamanis, Hao Peng, and Charles Sutton. A convolutional attention network for extreme summarization source code. In International conference on machine learning, pages 2091-2100, 2016.
Miltiadis Allamanis, Mare Brockschmidt, and M. Khademi. Learning to represent programs with graphs. ArXiv, abs/1711.00740 2018.
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. A general path- representation for predicting program properties. ACM SIGPLAN Notices, 53(4):404 2018
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured representations of code. ICLR, 2019a.
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning distributed represen- tations of code. Proceedings the ACM on Programming Languages, 3(POPL): 2019b.
Rohan Bavishi, Michael Pradel, and Koushik Sen. Context2name: A deep learning- based approach to infer natural variable names from usage contexts. arXiv preprint arXiv: (afe:::89900000 809,05593,, 2018.
10

---

Benjamin Bichsel, Veselin Raychev, Petar Tsankov, and Martin Vechev. Statistical deobfuscation of android applications. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 343-355, 2016.
Simon Butler, Michel Wermelinger, Yijun Yu, and Helen Sharp. Relating identifier naming flaws and code quality: An empirical study. In 2009 16th Working Conference on Reverse Engineering. pages 31-35. IEFF, 2009.
Jose Cambronero, Hongyu Li, Seohyun Kim, Koushik Sen, and Satish Chandra. When deep learning met code search. Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 964-974, 2019.
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation. In Advances in neural information processing systems, pages 2547-2557, 2018.
Zimin Chen, Steve James Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. Sequencer: Sequence-tor -orennccet--squeneee learning for end-to-end program repair. IEEE Transactions on Software Engineering, 2019.
Kevin Clark, Minh-Thang Luong, Quoc Le, and Christopher D Manning. Electra: Pre-training text encoders discriminators rather than generators. arXiv preprint 10ii:2003.10055, 2020.
Yaniv David, Uri Alon, and Eran Yahav. Neural reverse engineering of stripped binaries using aug- mented control flow graphs. Proceedings of the ACM on Programming Languages, 4(0PPSLA): 1-28, 2020.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-tinin for natural language understanding and generation. arXiv preprint arXiv: 190.01995 2019.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155: 2020.
Cheng Fu, Huili Chen, Haolan Liu, Xinyun Chen, Yuandong Tian, Farinaz Koushanfar, and Jishen Zhao. Coda: An end-to-end neural program decompiler. In Advances in Neural Information Processing Systems, pages 3703-3714, 2019.
Edward M Gellenbeck and Curtis R Cook. An investigation of procedure and variable names as beacons during program comprehension. In Empirical studies of programmers: Fourth workshop. pages 65 81. Ablex Publishing, Norwood, NJ, 1991.
Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. Deep code search. In 2018 IEEE/ACM 40th International Conference on Software Engincering (ICSE), pages 933-944. IEEE, 2018.
Daya Guo. Shuo Ren, Shuai Lu, Zhangyin Feng. Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009. 003666 2020.
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep code comment generation. In Proceedings of the 26th Conference on Program Comprehension, pages 200-210, 2018
Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph E Gonzalez, and Ion Stoica. Contrastive code representation learning. arXiv preprint arXiv:2007.04997 2020.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Oner Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020.

---

Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning, pages 5110-5121. PMLR, 2020.
Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees to transformers. arXiv preprint MXiv:2003.13844 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Jeremy Lacomis, Pengcheng Yin, Edward Schwartz, Miltiadis Allamanis, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. Dire: A neural approach to decompiled identifier naming. In 2019 34th IEEE/ACM International Conference Automated Software Engineering (ASE), pages
628-639 IEEE, 2019.
Guillaume Lample and Alexis Conneau. Cross- Cross-lingual language model pretraining. arXiv preprint arXiv:1901.0722 007291, 2019.
Guillaume Lample, Alexis Conneau, Ludovie Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. ICLR, 201 20.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Phrase- based & neural unsupervised machine translation. In EMNLP, 2018b.
Dawn Lawrie, Christopher Morrell, Henry Feild, and David Binkley. What's in name? study of identifiers. In 14th IEEE International Conference on Program Comprehension (ICPC'06), pages 3-12. IEEE, 2006.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeee pre-training for natural language generation, translation, and comprehension. arXiv preprint 2019.
Jian Li, Yue Wang, Michael R Lyu, and Irwin King. Code completion with neural attention and pointer networks. JJCAI, 2018.
Ben Liblit, Andrew Begel, and Eve Sweetser. Cognitive perspectives on the role of naming in computer programs. In PPIG, page 11, 2006.
Fang Liu, Ge Li, Bolin Wei, Xin Xia, Zhiyi Fu, and Zhi Jin. self-attentional neural architecture for code completion with multi-task learning. In Proceedings of the 28th International Conference Program Comprehension, pages 37-47, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 16692 1692, 2019.
Cristina V Lopes, Petr Maj, Pedro Martins, Vaibhay Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek. Déjävu: a map of code duplicates on github. Proceedings of the ACM on Programming Languages, 1(00PSLA):1-2) 2017.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint
2021.
Vijayaraghavan Murali, Lee Gross, Rebecca Qian, and Satish Chandra. Industry-scale ir-based bug localization: A perspective from facebook. arXiv preprint arXiv:2010.09977,, 2020.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics, 2002.
12

---

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pages 8026-8037, 2019.
Veselin Raychev, Martin Vechev, and Andreas Krause. Predicting program properties from" big code". ACM SIGPLAN Notices, 50(1):111-1244 2015.
Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems, 33, 2020.
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mass: Masked sequence to sequence pre-training for language generation. International Conference on Machine Learning, pages 2019.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng. Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint EXXiv1190400922 2019.
Alexey Svyatkovskoy, Sebastian Lee, Anna Hadjitofi, Maik Riechert, Juliana Franco, and Miltiadis Allamanis. Fast and memory-efficient neural code completion. arXiv preprint arXiv:2004. 13655,, 2020.
Armstrong A Takang, Penny A Grubb, and Robert D Macredie. The effects of comments and identifier names on program comprehensibility: an experimental investigation. J. Prog. Lang., (3):143-167, 1996.
Daniel Tarlow, Subhodeep Moitra, Andrew Rice, Zimin Chen, Pierre-Antoine Manzagol, Charles Sutton, and Edward Aftandilian. Learning to fix build errors with graph2diff neural net- works. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, pages 19-20, 2020.
Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(4):1-29, 2019.
Bogdan Vasilescu, Casey Casalnuovo, and Premkumar Devanbu. Recovering clear, natural identifiers from obfuscated js names. Proceedings of the 2017 1 11th Joint Meeting on Foundations of Software Engineering, pages 683-693, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 1998-6008, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103, 2008
Ke Wang, Rishabh Singh, and Zhendong Su. Dynamic neural program embedding for program repair. arXiv preprint arx:: 171.1171..07... 2017.
Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. Detecting code clones with graph neural network and flow-augmented abstract syntax tree. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 261-271. IEEE, 2020.
Huihui Wei and Ming Li. Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code. In IJCAI, pages 3034-3040, 2017.
Rui Xie, Wei Ye, Jinan Sun, and Shikun Zhang. Exploiting method names to improve code summa- rization: deliberation multi-task learning approach. arXiv preprint arXiv:2103.11448, 2021.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le.
Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, pages 5753-5763, 2019.
13


Reviews:
[{"review_id": "aXgKwjtdhj2", "reviewer": "Reviewer_K81R", "summary": "This paper proposes a pre-training method for programming languages. The main idea is pre-training a seq2seq model to convert obfuscated functions back to their original forms. The method demonstrates performance improvement on multiple downstream tasks, e.g., code translation and code search.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed pre-training objective, named DOBF,  is based on the classical masked language model (MLM). The main differences with MLM lie in that (1) DOBF only masks variable, method, and class identifiers rather than syntax-related tokens (e.g., parenthesis and semi-column), and  (2) all occurrences of a masked variable will be replaced by the same special token. \n\nMy main concern is that the motivation behind DOBF is not well justified. The authors describe the limitations of MLM for programming languages in Lines 50-56, where some critical claims are somewhat inaccurate. \n\n* They claim that syntax-related predictions \"provide little training signal to the model,\" but I guess these predictions do help to capture **structure** semantics. That may also be the main reason for the poor performance of DOBF without MLM initialized. Besides, the authors state that \"DOBF leverages the structural aspect of programming languages\" in the abstract, but I can not see how DOBF leverages the **structural aspect**. Anyway, variables are more about ambiguous information instead of structure information.   \n\n* DOBF masks all occurrences of a variable with the same special token since the authors believe \"it will be easy for the model to simply copy its name from one of the other occurrences.\" From my point of view, learning to copy which identifier to the masked position is a non-trivial task, which will help capture contextual information. However, masking with the same token will tell the model that the two variable is the same. I believe both masking strategies have their merits and they are somewhat complementary, but there is no in-depth evaluation to compare them. I see the overall performance improvement, but I am wondering whether this strategy works.\n\nI am very interested in the experimental results of deobfuscation. According to a previous code summarization work [1], it is non-trivial to generate meaningful code summaries when removing all variable identifiers in a function.   Though generating code summaries is different from recovering masked variables,  it is surprising that DOBF_1 recovers 45.6% of the initial identiﬁer names, and the recovered code in Figure 2 looks perfect. How do you split the training set and test set for this evaluation?  Maybe there is something that I misunderstand, but I guess the surprising performance may attribute to the memorization of duplicated code rather than the generalization abilities.\n\nRegarding the experiments of the downstream tasks, the overall performance improvements over strong baselines (e.g., GraphCodeBERT) look like mainly coming from DAE, which is not the technical contribution of this paper. Without DAE, DOBF demonstrates no obvious superiority. \n\nBesides, there has been a pre-training method to predict a function name [2]. The idea is not quite new given the current technical trend, though it is a more general method targeting more downstream tasks of programming languages. [2] should be an important missing reference.\n\nOverall, this paper is an incremental study, and the evaluation is weak, especially for a high-profile conference. Investigating further and more fine-grained how and why DOBF improve over MLM could improve this work to a more solid one.\n\n-----After the authors' rebuttal-----\n\nThe response generally makes sense. This work's technical novelty and overall quality look more fit for a short paper. I will keep my score.\n\n\n[1] A neural model for generating natural language summaries of program subroutines, ICSE 2019\n\n[2] Exploiting Method Names to Improve Code Summarization: A Deliberation Multi-Task Learning Approach, ICPC 2021\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations and potential negative societal impact (e.g., theft of proprietary code) of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "WIIuoU2UR4T", "reviewer": "Reviewer_udeg", "summary": "This work follows up with earlier efforts to pre-train BERT-like architectures on code, like CuBERT and CodeBERT. Instead of using a language-agnostic masked language model pre-training objective, however, it uses a de-obfuscation objective: masking all occurrences of an identifier (variable, function, class), ask the model to predict it. This code-specific pre-training objective is combined with other recent options (e.g., a denoising autoencoder) as well as traditional mask-language modelling, to produce improved performance on a number of code tasks such as summarization, clone detection, and transpilation.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Overall Evaluation\n\nOriginality: This is a relatively incremental idea that seems sensible and appears (at first glance) to improve results.\n\nQuality: There are some concerns on investigating the source of performance improvements and identifying if they're due to the new contribution (DOBF), more samples trained on, or ancillary additions (e.g., DAE). A bit of experimental refinement should resolve the issues. Some concerns also with information leak across pre-training and fine-tuning datasets, that could be clarified during rebuttal.\n\nClarity: The submission is clearly written and related work is well presented and contrasted.\n\nSignificance: Although the new nugget here is relatively small, assuming the evaluation is strengthened, it could be useful in lots of future work in the space of ML for code. \n\n\n\n# Main Comments\n\nThis work suggests a sensible improvement over the prior art on unsupervised pre-training on code. You make a good case with traditional MLM has a mix of very easy and not-so-easy instances in code contexts. Of course, the same argument applies to natural language as well (eg., predicting articles, pronouns, expressions is much easier than the typical cloze predictions), so this is a sensible thing to try, albeit not particular surprising.\n\nThis work involved a lot of engineering (getting the DOBF implementation to play with the CodeXGLUE tasks and TransCoder), and this is quite appreciated. But the novel contribution here doesn't contain much in terms of modelling. All are known building blocks put together to implement and evaluate a simple idea.\n\nI had some concerns about the evaluation that might be possible to address during rebuttals.\n\n1. MLM still helps DOBF, so the \"easy\" cases still seem to provide value. Given that MLM still seems to improve with even more training, what would happen if you produced twice as many MLM examples (say using additional random seeds or going beyond the 15% MLM tokens or 20 MLM tokens cap from traditional BERT) instead of MLM + DOBF? Is it truly the case that using more MLM examples and instances would never catch up?\n\n2. DAE seems to be an orthogonal addition to the picture (and not a contribution here, other than just trying it out on code), that could compose with MLM just as well as it does with DOBF. Would MLM+DAE provide a similar boost to MLM? More?\n\n2. It's not clear what the sample cost of each row on Table 2 is (this is related to the above). Is DOBF + DAE pre-training on more samples than DOBF alone or MLM alone? Fixing the number of samples across all combinations seems like a more rational way to compare the approaches on their merits, rather than making an uncontrolled comparison.\n\n1. The differences about some of the numbers in Table 2 are small. Confidence intervals are very important to draw sound conclusions here. Please add confidence intervals.\n\n1. I'm not entirely clear on Table 1's metrics. It seems your metrics are per deobfuscation instance, treating samples with say 10 de-obfuscations the same as samples with 1. Is that true? Are you measuring any performance with respect to the per-sample F1 score of token de-obfuscations? This is a weak concern, since this is a pre-training task, and ultimately it's the fine-tuning tasks we care about. But for a de-obfuscation task as a first-class outcome, we'd want to know how well a single sample is deobfuscated, not just the deobfuscated variables in the test dataset. Please clarify your metrics.\n\n1. In your experimental details, you mention that you deduplicate your pre-training dataset, but you also mention \"duplicate files\". Does that mean you're only removing identical files? Or do you use the set/multi-set similarity scores of Allamanis to remove even near-duplicate files? Please specify what you do exactly.\n\n1. Also, what about your fine-tuning datasets? Do you ensure that none of your fine-tuning samples (in either fine-tuning split) appear exactly or approximately in your pre-training dataset? In other words, do you do what Kanade et al. did for cubert to remove all fine-tuning files from the pre-training files? Otherwise, you may easily end up with severe label leaking problems.\n\n1. You mention in the fine-tuning details that you train models with the same architectures, batch sizes, optimizers, etc. I'd like to understand if that means you performed hyperparameter searches that include all the same sizes (but still chose the best for each)? Or did you just choose the same hyperparameters and just search across learning rates? The issue here is that fairness doesn't mean that all baselines have the exact same architecture; it is, instead, that all baselines are *given the chance* to be best with the same architecture (the same number of trainable parameters, hidden sizes, etc) even if for the particular dataset, the best choice might not be the same choice for each. \n\n\n\n# Smaller Comments\n\n* Line 49--53: This claim seems false, given lines 304 and 305, where you point out that you use MLM to initialize DOBF and found that initializing DOBF from scratch was inferior. It seems you need both for good performance.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No concerns.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Njrsu6sbL3g", "reviewer": "Reviewer_unSs", "summary": "This paper produces a new pretraining objective for models that provide program embeddings via the use of a name deobfuscation objective. This objective has advantages over the standard MLM objective as it excludes obvious masking possibilities and also does not allow the model to copy the names of one token from part of the document to another part.\n\nThe authors construct this objective and demonstrate that their model can learn this task fairly effectively. Interestingly, initializing their objective with a model pretrained on MLM seems to aid the performance of their model. They also demonstrate improvement over some baselines on some standard program processing objectives, and improvement over all baselines using a combination of their objective and a separate program recovery objective.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: this paper’s deobfuscation objective seems interesting and novel.\n\nQuality: The paper’s analysis of this new task and the techniques used to solve it directly seem sound, as does the inclusion of MLM training and how it improves the results.\n\nOne problematic element of this paper is the inclusion of DAE as an additional objective on top of DOBF. As it is an element added in to the work on top of MLM, it should probably also be included as a baseline as it is clearly providing a great degree of support to the work. In table 2, if DAE were to be removed, the DOBF models would not provide the best result in two of the six settings (Code Sum Python and Python → Java). Thus I think it is important to include a DAE baseline so we can be sure it is DOBF and not DAE providing the benefits.\n\nClarity: The paper is generally clearly written and goes over the topics in sufficient detail to be understood. \n\nOne minor criticism is that the word “deobfuscation” as used throughout this paper is substantially narrower than what people might picture when they hear the word being used. An explicit sentence about this in the introduction perhaps in the paragraph on obfuscation would help a reader understand what is going on in the paper\n\nMinor clarity complaints:\n280: if possible please move the table closer\n\n\n========================\n\nEdit (following reply to this review): the DAE-onnly baseline was helpful in improving the robustness of the paper. My score has been adjusted correspondingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I agree with the authors that research on deobfuscation does more good than harm, but disagree with their reasoning about this work in particular. In general, the kinds of obfuscation performed by socially desirable/legal and socially undesirable/criminal actors are fairly similar and thus helping solve some but not all obfuscation tasks does not harm one group more than the other. I think in general the reason this kind of work is socially desirable is because the issue of reverse engineering of software is targetable legally whereas deobfuscation for virus detection, etc., is a primarily technical problem.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.25"}, "extra_text": {}}, {"review_id": "JaE75bPGP8r", "reviewer": "Reviewer_NjSg", "summary": "This paper proposes a self-supervised pre-training task: code deobfuscation (recovering erased identifier names). This pre-training task only requires unlabeled natural programs as training data. The authors claim that models pre-trained with code deobfuscation significantly outperform existing approaches on many downstream program understanding tasks, including clone detection, code summarization, NL code search, and unsupervised program translation.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a novel pre-training task for natural programs (programs written by humans in real-world projects): code deobfuscation. This idea is not only theoretically reasonable but also empirically effective on both deobfuscation and multiple downstream tasks.\n\nThe proposed method is evaluated on multiple well-established public benchmarks, including three subtasks in CodeXGLUE and TransCoder. It is easy to reproduce the results in this paper with the provided source code. Also, the author reruns other baselines to keep the model size similar, making comparisons fair and convincing. Although there are some inconsistent/confusing numbers in the experiment results (see below), the evaluation is sound in general.\n\nThis paper includes enough implementation details of the proposed method. It also includes experiments on several variations (different p_obf, with or without MLM) to show how they make every design choice.\n\nHowever, the usage of DOBF with MLM and DAE puts the effectiveness of DOBF in question.\n\n- MLM: In Table 1, we can see that the best models on two deobfuscation tasks are all initialized with masked language modeling (MLM). In Table 2, \"DOBF init MLM\" outperforms \"DOBF init scratch\" on all tasks except Clone Detection, and \"DOBF init scratch\" is even worse than MLM/CodeBERT on multiple tasks. Therefore, it seems that the combo of MLM and DOBF is what really works, so the author should not overclaim that \"DOBF is better than MLM\".\n\n- DAE: The authors do not mention DAE at all in the \"Model\" section, but in Table 2, we find that in all tasks that requires generation (i.e. all tasks except Clone Det), additional DAE training is required to get SOTA performance consistently. However, no enough explanation related to DAE is found in this paper. To make the experiment section more solid, I think \"MLM + DAE\" and \"DAE only\" should also be evaluated and reported.\n\n**Inconsistent numbers:** Table 2 reports the results of unsupervised Python/Java translation, where the experimental setting should be the same as the cited TransCoder paper (https://arxiv.org/pdf/2006.03511.pdf). The \"MLM\" row in Table 2 should correspond to the model in the TransCoder paper. However, the numbers reported here are very different from those reported in the TransCoder paper (the \"TransCoder Beam 10 - Top 1\" row in Table 2). Moreover, in the TransCoder paper, Java -> Python systems usually have higher computational accuracies; but in this paper, Python -> Java systems has better CA@1. I suggest that the author should explain these differences in detail during the author feedback session.\n\nTo summarize, this paper proposed a novel and effective method, has a high-quality experiment section, but still needs to add more explanation in the camera-ready version.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the key idea presented in the paper of using a new pre-training objective DOBF to recover obfuscated variable names from programs, and its usefulness for several downstream tasks. Adding a new DAE-only baseline for comparison was also greatly appreciated. While there were still some concerns regarding fairness of comparisons across models, overall the recommendation is for acceptance. Hopefully the authors can incorporate the detailed feedback from reviews and add additional experiments in the final version.
Decision: Accept (Poster)