Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a pre-training method for programming languages. The main idea is pre-training a seq2seq model to convert obfuscated functions back to their original forms. The method demonstrates performance improvement on multiple downstream tasks, e.g., code translation and code search.
main_review: The proposed pre-training objective, named DOBF,  is based on the classical masked language model (MLM). The main differences with MLM lie in that (1) DOBF only masks variable, method, and class identifiers rather than syntax-related tokens (e.g., parenthesis and semi-column), and  (2) all occurrences of a masked variable will be replaced by the same special token. 

My main concern is that the motivation behind DOBF is not well justified. The authors describe the limitations of MLM for programming languages in Lines 50-56, where some critical claims are somewhat inaccurate. 

* They claim that syntax-related predictions "provide little training signal to the model," but I guess these predictions do help to capture **structure** semantics. That may also be the main reason for the poor performance of DOBF without MLM initialized. Besides, the authors state that "DOBF leverages the structural aspect of programming languages" in the abstract, but I can not see how DOBF leverages the **structural aspect**. Anyway, variables are more about ambiguous information instead of structure information.   

* DOBF masks all occurrences of a variable with the same special token since the authors believe "it will be easy for the model to simply copy its name from one of the other occurrences." From my point of view, learning to copy which identifier to the masked position is a non-trivial task, which will help capture contextual information. However, masking with the same token will tell the model that the two variable is the same. I believe both masking strategies have their merits and they are somewhat complementary, but there is no in-depth evaluation to compare them. I see the overall performance improvement, but I am wondering whether this strategy works.

I am very interested in the experimental results of deobfuscation. According to a previous code summarization work [1], it is non-trivial to generate meaningful code summaries when removing all variable identifiers in a function.   Though generating code summaries is different from recovering masked variables,  it is surprising that DOBF_1 recovers 45.6% of the initial identiﬁer names, and the recovered code in Figure 2 looks perfect. How do you split the training set and test set for this evaluation?  Maybe there is something that I misunderstand, but I guess the surprising performance may attribute to the memorization of duplicated code rather than the generalization abilities.

Regarding the experiments of the downstream tasks, the overall performance improvements over strong baselines (e.g., GraphCodeBERT) look like mainly coming from DAE, which is not the technical contribution of this paper. Without DAE, DOBF demonstrates no obvious superiority. 

Besides, there has been a pre-training method to predict a function name [2]. The idea is not quite new given the current technical trend, though it is a more general method targeting more downstream tasks of programming languages. [2] should be an important missing reference.

Overall, this paper is an incremental study, and the evaluation is weak, especially for a high-profile conference. Investigating further and more fine-grained how and why DOBF improve over MLM could improve this work to a more solid one.

-----After the authors' rebuttal-----

The response generally makes sense. This work's technical novelty and overall quality look more fit for a short paper. I will keep my score.


[1] A neural model for generating natural language summaries of program subroutines, ICSE 2019

[2] Exploiting Method Names to Improve Code Summarization: A Deliberation Multi-Task Learning Approach, ICPC 2021

limitations_and_societal_impact: The authors have addressed the limitations and potential negative societal impact (e.g., theft of proprietary code) of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper produces a new pretraining objective for models that provide program embeddings via the use of a name deobfuscation objective. This objective has advantages over the standard MLM objective as it excludes obvious masking possibilities and also does not allow the model to copy the names of one token from part of the document to another part.

The authors construct this objective and demonstrate that their model can learn this task fairly effectively. Interestingly, initializing their objective with a model pretrained on MLM seems to aid the performance of their model. They also demonstrate improvement over some baselines on some standard program processing objectives, and improvement over all baselines using a combination of their objective and a separate program recovery objective.

main_review: Originality: this paper’s deobfuscation objective seems interesting and novel.

Quality: The paper’s analysis of this new task and the techniques used to solve it directly seem sound, as does the inclusion of MLM training and how it improves the results.

One problematic element of this paper is the inclusion of DAE as an additional objective on top of DOBF. As it is an element added in to the work on top of MLM, it should probably also be included as a baseline as it is clearly providing a great degree of support to the work. In table 2, if DAE were to be removed, the DOBF models would not provide the best result in two of the six settings (Code Sum Python and Python → Java). Thus I think it is important to include a DAE baseline so we can be sure it is DOBF and not DAE providing the benefits.

Clarity: The paper is generally clearly written and goes over the topics in sufficient detail to be understood. 

One minor criticism is that the word “deobfuscation” as used throughout this paper is substantially narrower than what people might picture when they hear the word being used. An explicit sentence about this in the introduction perhaps in the paragraph on obfuscation would help a reader understand what is going on in the paper

Minor clarity complaints:
280: if possible please move the table closer


========================

Edit (following reply to this review): the DAE-onnly baseline was helpful in improving the robustness of the paper. My score has been adjusted correspondingly.
limitations_and_societal_impact: I agree with the authors that research on deobfuscation does more good than harm, but disagree with their reasoning about this work in particular. In general, the kinds of obfuscation performed by socially desirable/legal and socially undesirable/criminal actors are fairly similar and thus helping solve some but not all obfuscation tasks does not harm one group more than the other. I think in general the reason this kind of work is socially desirable is because the issue of reverse engineering of software is targetable legally whereas deobfuscation for virus detection, etc., is a primarily technical problem.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work follows up with earlier efforts to pre-train BERT-like architectures on code, like CuBERT and CodeBERT. Instead of using a language-agnostic masked language model pre-training objective, however, it uses a de-obfuscation objective: masking all occurrences of an identifier (variable, function, class), ask the model to predict it. This code-specific pre-training objective is combined with other recent options (e.g., a denoising autoencoder) as well as traditional mask-language modelling, to produce improved performance on a number of code tasks such as summarization, clone detection, and transpilation.
main_review: # Overall Evaluation

Originality: This is a relatively incremental idea that seems sensible and appears (at first glance) to improve results.

Quality: There are some concerns on investigating the source of performance improvements and identifying if they're due to the new contribution (DOBF), more samples trained on, or ancillary additions (e.g., DAE). A bit of experimental refinement should resolve the issues. Some concerns also with information leak across pre-training and fine-tuning datasets, that could be clarified during rebuttal.

Clarity: The submission is clearly written and related work is well presented and contrasted.

Significance: Although the new nugget here is relatively small, assuming the evaluation is strengthened, it could be useful in lots of future work in the space of ML for code. 



# Main Comments

This work suggests a sensible improvement over the prior art on unsupervised pre-training on code. You make a good case with traditional MLM has a mix of very easy and not-so-easy instances in code contexts. Of course, the same argument applies to natural language as well (eg., predicting articles, pronouns, expressions is much easier than the typical cloze predictions), so this is a sensible thing to try, albeit not particular surprising.

This work involved a lot of engineering (getting the DOBF implementation to play with the CodeXGLUE tasks and TransCoder), and this is quite appreciated. But the novel contribution here doesn't contain much in terms of modelling. All are known building blocks put together to implement and evaluate a simple idea.

I had some concerns about the evaluation that might be possible to address during rebuttals.

1. MLM still helps DOBF, so the "easy" cases still seem to provide value. Given that MLM still seems to improve with even more training, what would happen if you produced twice as many MLM examples (say using additional random seeds or going beyond the 15% MLM tokens or 20 MLM tokens cap from traditional BERT) instead of MLM + DOBF? Is it truly the case that using more MLM examples and instances would never catch up?

2. DAE seems to be an orthogonal addition to the picture (and not a contribution here, other than just trying it out on code), that could compose with MLM just as well as it does with DOBF. Would MLM+DAE provide a similar boost to MLM? More?

2. It's not clear what the sample cost of each row on Table 2 is (this is related to the above). Is DOBF + DAE pre-training on more samples than DOBF alone or MLM alone? Fixing the number of samples across all combinations seems like a more rational way to compare the approaches on their merits, rather than making an uncontrolled comparison.

1. The differences about some of the numbers in Table 2 are small. Confidence intervals are very important to draw sound conclusions here. Please add confidence intervals.

1. I'm not entirely clear on Table 1's metrics. It seems your metrics are per deobfuscation instance, treating samples with say 10 de-obfuscations the same as samples with 1. Is that true? Are you measuring any performance with respect to the per-sample F1 score of token de-obfuscations? This is a weak concern, since this is a pre-training task, and ultimately it's the fine-tuning tasks we care about. But for a de-obfuscation task as a first-class outcome, we'd want to know how well a single sample is deobfuscated, not just the deobfuscated variables in the test dataset. Please clarify your metrics.

1. In your experimental details, you mention that you deduplicate your pre-training dataset, but you also mention "duplicate files". Does that mean you're only removing identical files? Or do you use the set/multi-set similarity scores of Allamanis to remove even near-duplicate files? Please specify what you do exactly.

1. Also, what about your fine-tuning datasets? Do you ensure that none of your fine-tuning samples (in either fine-tuning split) appear exactly or approximately in your pre-training dataset? In other words, do you do what Kanade et al. did for cubert to remove all fine-tuning files from the pre-training files? Otherwise, you may easily end up with severe label leaking problems.

1. You mention in the fine-tuning details that you train models with the same architectures, batch sizes, optimizers, etc. I'd like to understand if that means you performed hyperparameter searches that include all the same sizes (but still chose the best for each)? Or did you just choose the same hyperparameters and just search across learning rates? The issue here is that fairness doesn't mean that all baselines have the exact same architecture; it is, instead, that all baselines are *given the chance* to be best with the same architecture (the same number of trainable parameters, hidden sizes, etc) even if for the particular dataset, the best choice might not be the same choice for each. 



# Smaller Comments

* Line 49--53: This claim seems false, given lines 304 and 305, where you point out that you use MLM to initialize DOBF and found that initializing DOBF from scratch was inferior. It seems you need both for good performance.


limitations_and_societal_impact: No concerns.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a self-supervised pre-training task: code deobfuscation (recovering erased identifier names). This pre-training task only requires unlabeled natural programs as training data. The authors claim that models pre-trained with code deobfuscation significantly outperform existing approaches on many downstream program understanding tasks, including clone detection, code summarization, NL code search, and unsupervised program translation.
main_review: This paper proposes a novel pre-training task for natural programs (programs written by humans in real-world projects): code deobfuscation. This idea is not only theoretically reasonable but also empirically effective on both deobfuscation and multiple downstream tasks.

The proposed method is evaluated on multiple well-established public benchmarks, including three subtasks in CodeXGLUE and TransCoder. It is easy to reproduce the results in this paper with the provided source code. Also, the author reruns other baselines to keep the model size similar, making comparisons fair and convincing. Although there are some inconsistent/confusing numbers in the experiment results (see below), the evaluation is sound in general.

This paper includes enough implementation details of the proposed method. It also includes experiments on several variations (different p_obf, with or without MLM) to show how they make every design choice.

However, the usage of DOBF with MLM and DAE puts the effectiveness of DOBF in question.

- MLM: In Table 1, we can see that the best models on two deobfuscation tasks are all initialized with masked language modeling (MLM). In Table 2, "DOBF init MLM" outperforms "DOBF init scratch" on all tasks except Clone Detection, and "DOBF init scratch" is even worse than MLM/CodeBERT on multiple tasks. Therefore, it seems that the combo of MLM and DOBF is what really works, so the author should not overclaim that "DOBF is better than MLM".

- DAE: The authors do not mention DAE at all in the "Model" section, but in Table 2, we find that in all tasks that requires generation (i.e. all tasks except Clone Det), additional DAE training is required to get SOTA performance consistently. However, no enough explanation related to DAE is found in this paper. To make the experiment section more solid, I think "MLM + DAE" and "DAE only" should also be evaluated and reported.

**Inconsistent numbers:** Table 2 reports the results of unsupervised Python/Java translation, where the experimental setting should be the same as the cited TransCoder paper (https://arxiv.org/pdf/2006.03511.pdf). The "MLM" row in Table 2 should correspond to the model in the TransCoder paper. However, the numbers reported here are very different from those reported in the TransCoder paper (the "TransCoder Beam 10 - Top 1" row in Table 2). Moreover, in the TransCoder paper, Java -> Python systems usually have higher computational accuracies; but in this paper, Python -> Java systems has better CA@1. I suggest that the author should explain these differences in detail during the author feedback session.

To summarize, this paper proposed a novel and effective method, has a high-quality experiment section, but still needs to add more explanation in the camera-ready version.
limitations_and_societal_impact: N/A
review_title: null

Meta-review: The reviewers appreciated the key idea presented in the paper of using a new pre-training objective DOBF to recover obfuscated variable names from programs, and its usefulness for several downstream tasks. Adding a new DAE-only baseline for comparison was also greatly appreciated. While there were still some concerns regarding fairness of comparisons across models, overall the recommendation is for acceptance. Hopefully the authors can incorporate the detailed feedback from reviews and add additional experiments in the final version.
Decision: Accept (Poster)