Contrastive Code Representation Learning
Authors: Paras Jain
Ajay Jain
Tianjun Zhang
Pieter Abbeel
Joseph E. Gonzalez
Ion Stoica
Venue:  (2021)
Keywords: programming languages
representation learning
contrastive learning
unsupervised learning
self-supervised learning
transfer learning
nlp
pretraining
type inference
summarization

Abstract: Machine-aided programming tools such as automated type predictors and autocomplete are increasingly learning-based. However, current approaches predominantly rely on supervised learning with task-specific datasets. We propose Contrastive Code Representation Learning (ContraCode), a self-supervised algorithm for learning task-agnostic semantic representations of programs via contrastive learning. Our approach uses no human-provided labels, only the raw text of programs. ContraCode optimizes for a representation that is invariant to semantic-preserving code transformations. We develop an automated source-to-source compiler that generates textually divergent variants of source programs. We then train a neural network to identify variants of anchor programs within a large batch of non-equivalent negatives. To solve this task, the network must extract features representing the functionality, not form, of the program. In experiments, we pre-train ContraCode with 1.8M unannotated JavaScript methods mined from GitHub, then transfer to downstream tasks by fine-tuning. Pre-training with ContraCode consistently improves the F1 score of code summarization baselines and top-1 accuracy of type inference baselines by 2% to 13%. ContraCode achieves 9% higher top-1 accuracy than the current state-of-the-art static type analyzer for TypeScript. Finally, representations learned through a hybrid contrastive and reconstruction objective transfer in zero-shot to code clone detection with +10% AUROC over a static text similarity measure and +5% over reconstruction alone.

Under review as a conference paper at ICLR 2021
CONTRASTIVE CODE REPRESENTATION LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Machine-aided programming tools such as automated type predictors and autocom- plete are increasingly learning-based, However, current approaches predominantly rely on supervised learning with task-specific datasets. We propose Contrastive Code Representation Learning (ContraCode), self-supervised algorithm for learn- ing task-agnostic semantic representations of programs via contrastive learning, Our approach uses no human-provided labels, only the raw text of programs. ContraCode optimizes for representation that is invariant to semaatic-preservnng code transformations. We develop an automated source-tor souree-t--souree compiler that generates textually divergent variants of source programs. We then train neural net- work to identify variants of anchor programs within large batch of non-equivalent negatives. To solve this task, the network must extract features representing the functionality, not form, of the program. In experiments, we pre-train ContraCode with 1.8M unannotated JavaScript methods mined from GitHub, then transfer to downstream tasks by fine-tuning. Pre-training with ContraCode consistently improves the Fl score of code summarization baselines and top-1 accuracy of type inference baselines by 2% to 13%. ContraCode achieves 9% higher top- accu- racy than the current state-of-the -frte-----e--rt--ee static type analyzer for TypeScript. Finally, representations learned through hybrid contrastive and reconstruction objective transfer in zero-shot to code clone detection with +10% AUROC over a static text similarity measure and +5% over reconstruction alone.
INTRODUCTION
Programmers increasingly rely on machine-aided programming tools to aid software develop- ment (Kim et al. 2012). However, the wide diversity of programs encountered in practice limits the generalization of hand-written rules. Catching semantic bugs such as naming errors requires deeper language understanding, motivating learning-based programming tools. Recent work uses machine learning for bug detection (Pradel & Sen 2018) and optimization (Mendis et al., (019). Consider predicting the type of the variable declaration "var median an ;" Static analysis fails as the type is underspecified, but the variable name indicates the statement is a float.
Programming language datasets suffer from scarce annotations due the time and expertise required to label. State-of-thc-aa approaches generally rely on either (1) synthetic supervised datasets or (2) self-supervised pre-training. Synthetic getto-geerrted labels have been used for method naming (Alon)
ct al. 2019a b) and bug detection (Ferenc et al. 2018 Benton et al. 2019 Pradel & Sen 2018). However, synthetic code datasets suffer from duplication issues (Allamanis] 2019) and biases (Shin
et al. 2019) which degrade generalization. Moreover, auto-generated data does not cover the diverse program behaviors encountered in the wild.
In contrast, self-supervised learning can leverage large open-source repositories such as GitHub with limited or no annotations. Inspired by the success of pre-training in natural language processing, recent work uses self-supervision to learn code representations. Authors have explored context-based token embeddings (Ben-Nun et al. 2018)) and masked language modeling, where tokens are corrupted and reconstructed (Feng al.) 2020 Kanade et al. 2020) However, reconstruction focuses on superficial language reasoning and does not explicitly address the underlying program functionality. The resulting models attend to program implementation specifics such as variable names.
We hypothesize that programs with the same functionality should have the same underlying repre- sentation for downstream code understanding tasks, principle illustrated in Fig. T While it is time

---

Under review as a conference paper at ICLR 2021
Given program,
Maximize similarity with equivalent programs (1 (
Function (ler) For (1 Bj len, irt)
function (str, 1ms) ( return scr. str.slice(o, len);
function
f(on)) f(or2);
)
Minimize similarity with functionally different programs
function (orr] for (1 arr) (
Figure Programs with the same functionality should have the same underlying representation. ContraCode learns such representations with contrastive learning: the network is trained to find equivalent programs among many distractors, encoding semantics into the representation.
intensive identify equivalent programs in large corpus, it is cheap to leverage static compiler transformations to automatically generate many equivalent versions of particular source program.
In this work, we develop ContraCode, a self-supervised representation learning algorithm that uses source-to-source compiler transformation techniques (e.g.. dead code elimination, obfuscation and constant folding) to generate syntactically diverse but functionally equivalent programs. ContraCode uses these equivalent programs to construct a challenging discriminative pretext task that requires the model to identify equivalent programs out of large dataset distractors. In doing so, it has to embed the functionality, not the form, of the code. In essence, the domain knowledge from our code transformations induces the knowledge of the structure programs onto learned representations. The contributions of our work include:
1. the novel use of compiler-inspired transformations as data augmentations for code,
2. the concept of program representation learning based on functional equivalence, and
3. a detailed analysis of architectures, code transforms and pre-train strategies, where ContraCode improves static type inference top-1 accuracy by 9%, learned inference by 2% 13%, summarization F1 score by up to 8% and clone detection AUROC by 5% 10%.
RELATED WORK
Self-supervised learning (SSL.) a general representation learning strategy where some dimensions or attributes of a datapoint are predicted from the remaining parts. These methods are unsupervised in the sense that they do not rely on labels, but SSL tasks often adapt losses and architectures designed for supervised learning. Self-supervised pre-training has yielded large improvements in both NLP (Howard & Ruder 2018: Devlin et al., 2018 Radford et al.) 2018 2019) and computer vision (Mahajan et al. 20188 by improving generalization (Erhan et al. 22100 Hao al., 2019). Weak visual features, such as orientation (Gidaris et al. 2018), color (Zhang et al.) 2016), and context (Pathak al. 2016), meaningful signals for representations (Mahajan al. 2018
Contrastive learning unifies many past SSL approaches that compare pairs or collections of similar and dissimilar items (Hadsell al., 2006). Rather than training the network to predict labels or reconstruct data, contrastive methods minimize the distance between the representations of similar examples (positives) while maximizing the distance between dissimilar examples (negatives). Examples include Siamese networks (Bromley et al.) 1994) and triplet losses (Schroff et al. 2015). Contrastive predictive coding (Oord al. [0188] Hénaff et al. 2019) learns to encode chunks of sequential data to predict of future chunks with the InfoNCE loss, variational lower bound on mutual information between views the data (Tian al. 2019: Wu et. al., 22200) inspired by noise- constrastive estimation (Gutmann & Hyvärinen] 2010). In instance discrimination tasks (Wu et al.) 2018), views and not pieces of an entire image are compared. SimCLR (Chen et al.] (2020a) and Momentum Contrast (He et al., 2019 Chen et al. 2020bb recently made progress by using many negatives for dense loss signal. Beyond images, InfoNCE has been applied to NLP (Chuang et al.) 2020 Giorgi et al. 2020), but may require supervision (Fang & Xie (020)..
Code representation learning There has been substantial work on architectures and tasks for machine learning code (Allamanis al. 2018). We adopt the summarization task of Alon et al.)
2

---

Under review as a conference paper at ICLR 2021
Function x(reaLine) { const section -
text: cate
function x(t) const {
k:
for ( 1 novLine; +- section. text - 'L
data': data
}
for (s <; n.text linest]] "Mn"s
Tunct X(C)ooost
if (section)
ne[ text 1 'data' ddata)y 1)n, text [iines]]
parsingCtx. sections. www.ssonss..scccaccttnnnn
parsingCtx, wwectttt.eeetioosss........ pesthoos..ssshhh)):
inilpprrsingta pectiion..puhieee push(()3
Renamed variables, explicit object style, explicit concatenation, inline conditional
Mangled source
Original JavaScript method
compressed whitespace
Figure 2: A JavaScript method from the unlabeled training set with two automatically generated semantically-equivalenn programs. The original method is from the StackEdit Markdown editor.
(2019)), and the variable type inference task of Deep'Typer (Hellendoomn et al. 2018). Other authors
have explored summarization (Movshovizz-Attiis & Cohen 2013] Allamanis et al. |0166] I et al.
2016) and type inference (Pradel al., [2119] Pandi et al., 2020 Wei et al. [2020] Allamanis et al.
2020 Bielik & Vechev| 2020) with different languages and datasets. The tree or graph structure of
code can exploited to encode invariances in the representation. Inst2vee (Ben-Nun et al.) 2018) locally embeds individual statements in LLVM IR by processing contextual flow graph with context prediction objective (Mikolov et al.] 2013). Tree-Based CNN embeds the Abstract Syntax Tree (AST) nodes of high-level source code. Code2seq (Alon et al. 2019a) embeds AST paths with an attention-based encoder and LSTM decoder for supervised sequence-to-see -equeencee tasks. Kanade ct al. (2020); Feng al. (2020) pre-train the Transformer (Vaswani et al. 2017) on code using the masked language modeling objective (Devlim et al. 2018). an instance of the cloze task (Taylor) 1953)
where the model reconstructs corrupted tokens. Recurrent networks have also been pre-trained
code (Hussain et al.] 2020) as language models (Peters et al.) 2018 Karampatsis & Sutton 2020).
Wang & Christodorescu (2019); Wang & Su (2019) assess the stability of program analyzers under semi-automated program transformations. Concurrent work by Rabin & Alipour (2020) found that code2vec and code2seq often change their classifications when statements are permuted, variables are renamed, or
preserving transformations are applied.
3 METHOD: CONTRASTIVE CODE REPRESENTATION LEARNING
Understanding program functionality and global structure is important for difficult tasks like summa- rizing code in natural language. For these problems. learned code representations should be similar for functionally equivalent programs and dissimilar for non-equivalent programs (Figure [ The principle of contrastive learning offers simple objective for learning such representations if data can be organized into pairs of positives and negatives. We use each pair to shape representation space, drawing positives together and pushing negatives apart. However, a major question remains: given an unlabeled corpus of programs, how do identify generate similar programs? We address this question in Sec. 3.1 then introduce our learning framework in Sec. 3.2
3.1 COMPILATION AS DATA AUGMENTATION
Modern programming languages afford great flexibility to software developers, allowing them to implement the same desired functionality in different ways. Crowdsourced datasets mined from developers, such as GitHub repositories, have many near-duplicates in terms of textual similar- ity (Allamanis) 2019), and are bound to contain even more functional equivalences for common tasks. Satistiability solvers can identify these equivalent programs (Joshi et al. 2002 Bansal &
Aiken 2006), but functional equivalence is also undecidable in general (Rice) 1953). Also, formal
documentation of semantics is required. Programs can instead be compared approximately using test-cases (Massalin) 1987), but this is costly and requires executing untrusted code.
Instead of searching for equivalences. we propose correct by construction data augmentation. Our insight is apply source-to-sonne compiler transformations to unlabeled code to generate many variants with the same functionality. For example, dead-code elimination (DCE) is a common compiler optimization that removes operations that leave the output of function unchanged. While
3

---

Under review as a conference paper at ICLR 2021
Code compression
Identifier modification
Reformatting (R)
Variable renaming (VR)
Frequency (x1000)
Beautification (B)
Identifier mangling (IM)
150
Compression (C)
Regularization
Dead-code elimination (DCE)
Dead-code insertion (DCI)
100-
Type upconversion (T)
Suhword regularization (SW)
Constant folding (CF)
X
Line subsampling (LS)
50
/ semantics- semnantis-preservinng transformation
X lossy transformation
10
20
Unique transformed program variants
Table I:
We augment programs with 11 automated source-to- source compiler transformations. 10 of the 11 transformations are and do not modify operational semantics. More details are in Section A.3
Figure 3: Histogram of the num- ber of transformed variants per method during pre-trat pre-training.
DCE preserves program functionality, Wang & Christodorescu (2019) find that up 12.7% of the predictions of current algorithm classification models change after DCE- supessuee datasets were not enough to acquire the domain knowledge that DCE does not matter.
particular source code sequence, e.g. "W*x b" is parsed unambiguously into a tree-structured representation "++ (* W x) b)". This is then transformed by automated traversal algorithms. A rich body of prior programming language work explores parsing then tranforming Abstract Syntax Trees optimize program prior machine code generation. If source code is output rather than machine code, this is called cource-to-soure transformation. Source-to-souree source transformations are common for optimization and obfuscation purposes in dynamic languages like JavaScript. If each transformation preserves code functionality, then any composition also preserves code functionality.
We leverage the Babel and Terser compiler infrastructure tools for JavaScript (McKenzie et al.) 2020) Santos et al. (2200) to parse code into an Abstract Syntax Tree (AST) and then perform correctness- preserving transformations on method bodies. Table [ and Appendix A.33 list all transformations, but we broadly group program transformations into three categories. Code compression changes the syntactic structure of code and performs correct-by-cee cynnttt----oontuuction transformations such as pre- computing constant expressions at compile time. Identifier modification substitutes method and variable names with random tokens, thereby masking part of the semantic information in programs. Finally, transformations for Regularization improve model generalization by reducing the number of trivial positive pairs with high text overlap; this group potentially modifies program semantics through the line subsampling pass.
3.2 CONTRASTIVE PRE-TRAINING
Representations of semantically equivalent programs (positives) should have representations that each are closer to each other than semantically dissimilar programs (negatives). Contrastive learning natural framework to induce invariances into a model by attracting positives while repelling negatives. To adapt recent contrastive learning objectives for images code representation learning, we leverage the augmentations discussed in Section 3.1
We extend the Momentum Contrast method (He al. 2019) that was designed for image represen- lation learning. Our training procedure is depicted in Figure 4 Each transformation is a function T: P P,,, where the space of programs of programs in source form. At the beginning of an iteration, a batch of programs is sampled from large database. Each program in the batch is transformed twice using two different, random subsets of transformations to derive textually different query programs and key programs according to Algorithm!] T] Unlike computer vision data augmentations such as random cropping that are stochastic, compiler-based transformations are deterministic.
composed of both the of valid ASTs and the set
To produce a diverse set of transformed programs, we randomly apply a subset of available compiler passes in a pre-specified order, applying transform with probability p:. Intermediate programs are converted between AST and source form as needed. As all augmentations are precomputed, deduplicate programs variants before are-training. Figure 3 measures this diversity. 89% of the JavaScript functions our dataset have more than one alternative after applying 20 random sequences of transformations. The remaining programs without syntactically distinct alternatives
4

---

Under review as a conference paper at ICLR 2021
Oo
fg
Sample takenzze positivee
Faretie Go
Embed
Maximize
Sg
Minimize
{T]
or
Enqueue future regative
fg
O G...)
Rectice O C...
A Unlabeled
B ContraCode compiler
C Augmented variants
D
E Finetune for tasks
Contrastive pre-training
programs
Figure 4: ContraCode pre-trains a neural program encoder fy and transfers it to downstream tasks. A-B. Unlabeled programs transformed C. into augmented variants. D. We pre-train fy by maximizing similarity embeddings of positive program varrs-variants of the same program-and minimizing similarity with queue of cached negatives. E. fg is fine-tuned smaller labeled datasets.
include one-line functions that are obfuscated. We apply subword regularization (Kudo 2018) as final transformation derive different tokenizations every batch, pairs will still differ. All transformations are fast; our compiler transforms 300 functions per second on a single CPU core.
To reduce memory consumption during pre-training, enqueue past batches to cache activations for negative samples. These cached samples are valid negatives if the queue is smaller than the dataset size. Following He et al. (2019), the query encoder fg trained via gradient descent while the key encoder f& is trained slowly via an exponential moving average (EMA) of the query encoder parameters. The EMA update stabilizes the pre-computed key embeddings across training iterations. Since keys are only embedded once per epoch, we use very large set of negatives, over 100K, with minimal additional computational cost and no explicit hard negative mining.
ContraCode supports different encoder architectures. We evaluate contrastive pre-training of Trans- former (Vaswani et al. 2017) and BiLSTMM (Schuster & Paliwal 1997 Huang et al. 2015) architec- tures, with specific details in Section 4
Pre-training objective The contrastive objective maximizes the similarity of positives without collapsing onto a single representation. Like He et al. (2019), we use InfoNCE (Oord et al. (0118). a tractable objective that frames contrastive learning as a classification task: can the positives be identified among batch of sampled negatives? InfoNCE computes the probability of classifying the positive (transformed program) by taking the softmax of representation similarities across batch of negatives. Equation ( shows the InfoNCE loss for instance discrimination from [He et al. (2019), function whose value is low when is similar to the positive key embedding k and dissimilar to negative key embeddings k= t is temperature hyperparameter proposed by Wu et al. (0018)
exp(q k+ k+/t)
Cg k- log
(1)
exple A+ k+/t) + Ex- explg k
The query representation fy (x²) ) is computed by the encoder network fg. and x a query program. Likewise, & = (x*) using the EMA key encoder fx- Views x", x depend on the specific domain and pretext task. In our case, the views tokenized representations of the augmented programs, and the summation Σx in the normalizing denominator is taken over the queue of pre-computed negatives as well as other non-matching keys in the batch.
Transfer learning After pre-training converges, the encoder fa is transferred to downstream tasks. As the output space of the task can differ from the encoder, we add task-specific MLP or Transformer decoder after fq. then train the resulting network end-to-end on task data.
4 EXPERIMENTS
We evaluate whether self-supervised pre-training with ContraCode improves JavaScript and Type- Script code analysis. We benchmark on (1) extreme code summarization (Allamanis et al.) 2016)
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Stochastic augmentation of programs with two possible encodings (AST or source).
1: Input: Program source æ; transformation functions Th, transform probabilities pl....Pk
2: V <- {x), set of augmented program variants
3: for SAMPLE +- N........ do
fx
5: for transform + I... do
6: Sample 1/+ ~ Bernouli(pp.) if Vt- then if (FQURESAAS(((()) and -ISAST(a')) then x' PARSETOAST(Z') else if (EQUIRESAST((.))) and Is) then æ' <- LOWEKTOSOURCCE()) x' <- (2") end if end for
7:
10:
11:
12:
13: ISAST(2²) then r'< LOWERTOSOURCCE(2)
14: V+ U-VUfe {
15:
end for
16:
return V
and (2) TypeScript type inference (Hellendoorn et al., 2018). ContraCode improves accuracy on both tasks. As baseline self-supervised approach, pre-train RoBERTa model with the masked language modeling (MLM) loss on our augmented dataset, then fine-tune on each downstream task. Contrastive pre-traine trainininng with compiler-based augmentations outperforms baseline supervised learning methods as well as MLM self-supervision. To probe the semantic content of representa- tions learned with MLM, ContraCode, and hybrid model combining both objectives, we evaluate zero-shot performance of code clone detection (Kamiya et al.. 2002), a binary classification task that reveals that contrastive and hybrid representations are highly predictive of program functionality in-the-wild. Further, we find it is better to augment the large set of unlabeled programs during pre- training rather than augmenting smaller supervised datasets. As ContraCode makes no modifications to model architecture, we find that contrastive pre-training can be applied to diverse baselines while improving accuracy across the board.
We pre-train over a large corpus of methods extracted from popular GitHub repositories. The CodeSearchNeet dataset collected by Husain et al. (2019) contains 1,843,099 JavaScript programs. Only 81,487 methods have both a documentation string and method name. The asymmetry between labeled and unlabeled programs stems from JavaScript coding practices where anonymous functions widespread. The pre-training dataset described in Section 3. is the result of augmenting CodeSearchNet's 's 1.8m programs.
4.1 IMPACT CONTRACODE PRE-TRAINING TYPE INFERENCE
JavaScript is dynamically typed language, where variable types determined at runtime based on the values they represent. However, annotating code with types helps tools flag possible bugs before runtime by statically detecting incompatible types. These annotations also help programmers document and understand code. However, maintaining type annotations is tedious. Type inference tools automatically predict variable types from context.
To learn to infer types, we use the same annotated dataset of TypeScript programs from Deep- Typer (Hellendoorn al. 2018), without GitHub repos that were made private deleted since publication. The training set consists of 15,570 TypeScript files from 187 projects with 6,902,642 total tokens. Validation and test sets are from held-out repositories. For additional supervision during training. additional types are inferred by static analysis to augment user-defined types as targets. All type annotations are removed from the input to the model. We evaluate 2-layer Bidirectional L.STM, used by DeepTypen and 6-layer Transformer, modified from RoBERTa to have comparable parameter count. A 2-layer MLP head predicts types from the model's embedding of each token. We perform early stopping based validation set top-1 accuracy.
Benefiting from pre-training challenging because it requires knowledge transfer across dialects. Our models pre-trained on JavaScript, not TypeScript. TypeScript supports superset of the JavaScript grammar, adding type annotations and syntactic sugar that must be learned during fine- tuning. Further, the pre-training dataset consists of methods, while the DeepTyper dataset includes

---

Under review as a conference paper at ICLR 2021
Table 2: Type inference accuracy on TypeScript programs in the Hellendoom al. (2018) dataset. ContraCode (BiLSTM) outperforms baseline top-1 accuracies by 2.28% to 13.16%. As ContraCode does not modify model architecture, contrastive pre-training can be combined with each baseline. Compared with TypeScript's built-in type inference, ContraCode improves top-1 accuracy by 8.9%.
Acc@1 1
Acc@5
Baseline
Method
(all types)
(all types)
TypeScript CheckJS (Bierman et al. 2014)
45. 28.94%
Static analysis
Name only (Hellendoom et al. 2018
70.07%
Transformer (supervised)
45.66%
80.08%
Transformer
with ContraCode pre- pre-training
46.86%
81.85% %
Transformer (RoBERTa MLM pre-training) with ContraCode pre-training
40.85%
75.76%
RoBERTa
47.16% %
81.44 %
Deep'Typer (supervised)
51.73%
82.71
DeepTyper (BiLSTM)
with RoBERTa MLM pre-training (10K steps)
50.24%
82.85%
with ContraCode pre-training
52.65%
84.60%
with ContraCode pre-training (w/ subword reg. ft.)
54.01%
85.55%
entire modules. Table summarizes results. Contrastive pre-training outperforms all baseline learned methods, showing meaningful transfer. Our pest-performin model (bottom row) achieves +8.3% higher top-1 accuracy than supervised Transformer model trained from scratch, + +13.2% higher than a pre-trained RoBERTa model and +2.3% higher than DeepTyper.
ContraCode can also be applied in a drop-in fashion to each of the baselines without modifying model architecture. Simply pre-training each baseline with our contrastive objective and data augmentations yields absolute accuracy improvements of + +1.2%, +6.3%, +2.3% top-1 and +1.8%, +5.7%, +2.8% top-5 over the Transformer, RoBERTa, and DeepTyper, respectively. The RoBERTa baseline may perform poorly since its masked language modeling (MLM) objective focuses token reconstruction that is overly sensitive to local syntactic structure. To combine the approaches, we minimized our loss in addition to MLM as hybrid local-global objective during pre-training.
Learning outperforms static analysis by a large margin. Overall, ContraCode achieves +8.9% higher top-1 accuracy than the best static type inference system, the built-in TypeScript CheckIS system, showing the promise of learned code analysis. Surfacing multiple candidate types can be useful to users. While CheckJS only produces single prediction which often incorrect, one of the top-5 predictions of ContraCode is correct for 85.55% of labeled tokens.
4.2
IMPACT OF CONTRACODE PRE- TRAINING ON EXTREME CODE SUMMARIZATION
The extreme code summarization task asks model to predict the of a method given its body (Allamanis et al., 2016). Tokenized method names often contain a short summary of func- tionality, such reversestring documented code. We create a JavaScript summarization dataset using the 81,487 labeled methods in the CoreSearchNet dataset. The method name is masked in the declaration of the function and then predicted by a sequence-to-sequemee model with an autoregressive decoder trained to maximize log likelihood of the ground-truth name, a form of abstractive summarization. All models overfit, so we use early stopping according to validation loss. As proposed by Allamanis et al. (2016). we evaluate model predictions by precision, recall and Fl scores over the set of method name tokens.
Summarization models could explain obfuscated or poorly
Table 3 shows code summarization results in four settings: (1) supervised training using baseline trce-structured architectures that analyze the AST (code2vec, code2seq), (2) pre-training on all 1.84M programs using masked language modeling followed by fine-tuning on the labeled programs (RoBERTa), (3) supervised training from scratch with a Transformer architecture and (4) contrastive pre-training with all 1.84M programs followed by fine-tuning with augmentations (ContraCode).
Contrastive pre-training with fine-tuning outperforms the prior code2seq model, a competitive supervised baseline, by 8.2% in test precision, 7.3% in recall, and 7.9% in Fl score. The tree-based
7

---

Under review as a conference paper at ICLR 2021
Table 3: Results for different settings the code summarization task: supervised training with 81k functions, masked language model pre-training and contrastive pre-training with fine-tuning.
Method
Precision
Recall
code2vec Alon al. 2019b
10.78%
8.24%
9.34%
code2seq
Alon ct al.
2019a
12.17%
7.65%
9.39%
RoBERTa MLM (Lil ef al.
2019
15.13%
11.47%
12.45%
Transformer Vasswan et al.
2017)
18.11%
15.78%
16.86%
Transformer + ContraCode + augmentation
20.34%
14.96%
17.24%
code2seq architecture is a way to encode code-specific invariances into the model, while contrastive pre-training induces domain invariances through data augmentation; reduced inductive biases in the Transformer model architecture leads to better performance. ContraCode outperforms self-supervised pre-training with RoBERTa by 4.8% Fl. ContraCode also achieves higher performance than the Transformer learned from scratch with the same network architecture. While this improvement is relatively smaller, code summarization is a difficult task. Naming conventions aren't consistent between programmers, and the metric measures exact token matches.
4.3 PROBING REPRESENTATION, OF FUNCTIONALITY: ZERO-SHOT CODE CLONE DETECTION
ContraCode learns to match variants of programs with similar functionality. While these transformations pro- duce highly diverse token sequences (Section (A.4). they are artificial and do not change the underlying algorithm. Human programmers can solve a problem with many data structures, algorithms and program- ming models. Are pre-trained representations consis- tent across programs written by different people? We benchmark on the code clone detection task, binary classification task to distinguish pairs of programs solv- ing the same problem from pairs solving different ones. This is useful for deduplicating and refactoring code, checking approximate code correctness.
Table 4: Code clone detection results with cosine similarity probe. Contrastive and hy- brid representations are predictive of func- tionality, with +6.2%. +10% AUROC over textual similarity (edit distance).
Representation
AUROC
AP
Edit distance heuristic
69.55
73.75
Transformer w/o pre-train train
74.28
76.40
MLM pre- tre-train
74.41
75.96
ContraCode pre-train
75.76
78.16
ContraCode MLM
79.55
81.74
Datasets exist like BigCloneBench (Svajlenko et al., 2014), but the best of our knowledge, there is no benchmark for the JavaScript programming language. We collected 274 in-the-wild JavaScript programs correctly solving 33 problems from the HackerRank interview preparation website. There are 2065 pairs solving the same problem and 70k pairs solving different problems, which we randomly subsample to 2065 to balance the classes. Since we probe zero-shot performance, there is training set. Traditional code analysis methods for clone detection measure textual similarity. As a baseline heuristic classifier, we threshold the dissimilarity score (Eq. ( a scaled edit distance between two normalized and tokenized programs (to exclude formatting changes). For continuous representations, we threshold cosine similarity u" u²'v/ //|/||]|/
Table shows results according to the area under the ROC curve (AUROC) and average precision (AP, area under precision-recall.. Continuous representation improves clone detection over the heuristic. However, self-supervision through masked language modeling for nearly 100 epochs of pre-training does not help, indicating that MLM is a poor fit for representing functionality. Contrastive pre-training achieves +6.21% % higher AUROC than the baseline. A hybrid objective combining both the contrastive loss and MLM has the best performance with +10% AUROC (+5.14% over MLM alone).
4.4 UNDERSTANDING THE IMPORTANCE OF DATA AUGMENTATION
We first analyze the effect of our proposed augmentations on supervised learning without a pre-training phase. We then study the importance of individual augmentations during pre-training.
Supervised learning with data augmentation As baseline, we re-train models from scratch with compiler transforms during supervised learning rather than pre-training. Data augmentation
8

---

Under review as a conference paper at ICLR 2021
Table 6: Ablating compiler transformations used during contrastive pre-training. The Dyep BiLSTM is pre-trained with constrastive learning for 20K steps, then fine-tuned for type inference. Augmentations are only used during pre-training. Each transformation contributes to accuracy.
Augmentations used during pre- pre-training
Acc@
Acc@5
All augmentations (Table
52.65% %
84.60%
without identifier modilication (-VR, -IM)
51.94%
84.43%
without line subsampling (-LS) without code compression (-T,C,DCE,CF)
51.05%
81.63%
50.69%
81.95%
artificially expands labeled training sets. For sequence-to- sequence summarization, we apply variety of augmenta- tions; these all preserve the method name label. For type inference, labels are aligned to input tokens, so they must be realigned after transformation. We apply all token-level transformations that track label locations.
Table 5: Compiler data augmentations degrade performance when training su- pervised models from scratch.
Code summarization
F1
Transformer (Table w/ WR aug.
16.86
T shows results. Compiler-based data augmentations degrade supervised models, perhaps by creating training distribution not reflective of evaluation programs. How- ever, shown in |.. 4.3 augmenting during ContraCode pre-training yields a more robust model. Our contrastive learning framework also allows learning bers of unlabeled programs that supervised learning alone cannot leverage. The ablation indicates that augmentations do not suffice, and contrastive learning is important.
15.65
Type Inference
Acc@
Transformer (Table w/ SW reg. w/ LS,SW aug.
45.66
43.96
large num-
44.14
DeepTyper (Table 2
51.73
w/ SW reg.
49.93
w/ LS,S aug.
50.93
A Ablating data pre-training augmentations Some data augmentations may be more valuable than others for learn- ing representation via instance discrimination. Empirically, pre-training converges faster with smaller set of augmentations at the same hatch size since the positives are syntactically more similar, but this hurts downstream performance. ()shoos that type inference accuracy degrades when different groups of augmentations are removed. Semanticc- require code analysis are the most important, improving top-1 accuracy by 1.95% when included. Line subsampling serves as regularizer, but changes program semantics. LS is relatively less important, but does help accuracy. Identifier modification passes preserve semantics, but remove potentially useful naming information. Removing these hurts accuracy the least.
w/ stronger LSSSW aug.
50.33
code compression passes that
Additional results We perform additional ablations in Section A.1 by transferring different parts of the network to downstream tasks, computing the contrastive objective with representations taken from different encoder layers, varying architecture, and tuning the pre-training procedure. These experiments suggest that as many parameters as possible should be transferred the downstream task. Details of the pre-training strategy are also important. Computing the contrastive objective using "global" representation q summarizing the whole input sequence x outperforms more a "local" representation based on aggregating token representations. Further, a large batch size is helpful to stabilize pre-traine pre-training. Section A.2 includes qualitative results.
CONCLUSIONS
Large-scale unannotated repositories of code like GitHub are powerful resource for learning machine-aided programming tools. However, most current approaches to code representation learning do leverage unannotated data. We propose ContraCode, a contrastive self-supervised algorithm that learns representations that are invariant to code transformations. Our method optimizes for this invariance via novel compiler-basee data augmentations for code. ContraCode significantly improves the accuracy of extreme code summarization baselines (+2.3% to +13.2%), TypeScript type inference models (up to +7.9% FI) and code clone detection (+5 to +10% AUROC). ContraCode outperforms self-supervised RoBERTa pre-training. Moreover, contrastive pre-training outperforms supervised training with our augmentations. As ContraCode makes no modifications to model architecture and simply adds training phase, it consistently improves accuracies when applied to diverse baselines.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Miltiadis Allamanis. The adverse effects code duplication machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Onward! 2019, pp. 143-153, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450369954, doi: 10.1145/3359591. 3359735.
Miltiadis Allamanis, Hao Peng, and Charles Sutton. A convolutional attention network for extreme summarization of source code. In International Conference on Machine Learning (ICML), 2016.
Miltiadis Allamanis, Earl Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), 51(4):81, 2018.
Miltiadis Allamanis, Earl T. Barr, Soline Ducousso, and Zheng Gao. Typilus: Neural type hints. In Programming Language Design and Implementation (PLDI), 2020.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured representations of code. In International Conference on Learning Representations, 2019a.
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: tations of code. Proceedings of the ACM on Programming Languages, 2019h.
Learning distributed represen-
Sorav Bansal and Alex Aiken. Automatic generation of peephole superoptimizers. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XII, pp. 394-403, New York, NY, USA, 2006. Association for
Computing Machinery. ISBN 1595934510. doi: 10. 145/1 168857. 169906.
Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. Neural code comprehension: A learnable representation of code semantics. In NeurlPS, 2018.
Samuel Benton, Ali Ghanbari, and Lingming Zhang. Defexts: A curated dataset of reproducible real-world bugs for modern jvm languages. 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 47-50. IEEE, 2019.
Pavol Bielik and Martin Vechev. Adversarial robustness for code. CoRR, 2020.
Gavin Bierman, Martín Abadi, and Mads Torgersen. Understanding typescript. In European Conference on Object-Oriented Programming (ECOOP), 2014.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. In Advances in neural information processing systems, pp. 737-744, 1994.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Macnine Learning, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. arXiv preprint 2rXiv:2003.04227 2020b.
Ching-Ye Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. Debi- ased contrastive learning, 2020.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2018.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Fierre-Antoing Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, ((Feb))625-666 2010.
Hongchao Fang and Pengtao Xie. CERT: Contrastive self-supervised learning for language under- standing. arXiv preprint arXiv:2005. /2766, May 2020.
10

---

Under review as a conference paper at ICLR 2021
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. CodeBERT: A pre-trained model for programming and natural languages. arXiv preprint arXi::2002.081555, 2020.
Rudolf Ferenc, Zoltán Tóth, Gergely Ladányi, István Siket, and Tibor Gyimóthy. A public unified bug dataset for Java. In Proceedings of the 14th International Conference on Predictive Models and Data Analytics in Software Engineering, pp. 12-21, 2018.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint
2018.
John M. Giorgi, Osvald Nitski, Gary D. Bader, and Bo Wang. DeCLUTR: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006. 2066069900 03659, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 06), volume 2, pp. 1735-1742. IEEE, 2006.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of bert. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4134 4143, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722 2019.
Vincent J Hellendoom, Christian Bird, Earl T Barr, and Miltiadis Allamanis. Deep learning type inference. Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium the Foundations of Software Engineering, pp. 152-162, 2018.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning, 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 928-339,220 2018.
Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional Istm-crf models for sequence tagging. arXiv preprint 1rXiv:150..0/99) 2015.
Hamel Husain, Ho-Hsiang Wu. Tiferet Gazit, Miltiadis Allamanis. and Mare Brockschmidt. CodeSearch challenge: Evaluating the state of semantic code search. arXiv preprint arXiv: /000:/999.0000 09436, 2019.
Yasir Hussain, Zhiqiu Huang, Yu Zhou, and Senzhang Wang. Deep transfer learning for source code modeling. International Journal of Software Engineering and Knowledge Engineering, 30(05): 649-668, May 2020. ISSN 1793-6403. doi: 10.1 1111422/022111144005002222
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoye. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2073-2083, 2016.
Rajeev Joshi, Greg Nelson, and Keith Randall. Denali: A goal-directed superoptimizer. In Proceed- ings of the ACM SIGPLAN 2002 Conference on Programming Language Design and Implementa- tion, PLDI '02, pp. 304-314, New York, NY, USA, 2002. Association for Computing Machinery. ISBN 1581 134630. doi: 10. 20.144/5125295113555

---

Under review as a conference paper at ICLR 2021
T. Kamiya, S. Kusumoto, and K. Inoue. Cefinder: multilinguistic token-based code clone detection system for large scale source code. IEEE Transactions on Software Engineering, 28(7))654 670, 2002. doi: 10.1099/SE.220211114888
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Pre-trained contextual embedding of source code. ArXiv, abs/2001, 000//2001.00599 2020.
Rafael-Michael Karampatsis and Charles Sutton. SCELMo: Source code embeddings from language models. arXiv preprint 2.22:2000..32214 2020.
Miryung Kim, Thomas Zimmermann, and Nachiappan Nagappan. A field study of refactoring challenges and benefits. In Proceedings of the ACM SIGSOFT 20th International Symposium the Foundations of Software Engineering, pp. 1-11, 2012.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple sub- word candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 66-75, 2018.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9NNov):559-22655, 2008
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision, pp. 181-196, 2018.
Henry Massalin. Superoptimizer: A look at the smallest program. In Proceedings of the Second International Conference on Architectual Support for Systems, ASPLOS II. pp.
for Programming Languages and Operating
Washington, DC, USA, 1987. IEEE Computer Society Press. doi: 10. 10.11440 141114666666666994
ISBN 081 0818608056.
Sebastian McKenzie et al. Babel: compiler for writing next generation javascript. htttps///githrn thub com/babel/babeil 2020.
Charith Mendis, Alex Renda, Saman Amarasinghe, and Michael Carbin. Ithemal: Accurate, portable and fast basic block throughput estimation using deep neural networks. In International Conference Machine Learning, pp. 4505 4515, 2019.
Tomas Mikolov, Ila Sutskever, Kai Chen, Greg S Corrado. and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111-3119, 2013.
Dana Movshovitz-Attia and William Cohen, Natural language models for predicting programming comments. In Proceedings of the 51 Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pp. 35-40, 2013.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03744 2018
Irene Vlassi Pandi, Earl T. Barr, Andrew D. Gordon, and Charles Sutton. Opttyper: Probabilistic type inference by optimising logical and natural constraints, 2020.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018
Michael Pradel and Koushik Sen. Deepbugs: A learning approach to name-based based bug detection. Proceedings of the ACM Programming Languages, 15 2018.
12

---

Under review as a conference paper at ICLR 2021
Michael Pradel, Georgios Gousios, Jason Liu, and Satish Chandra. Typewriter: Neural type prediction with search-based validation. arXiv preprint drXiv:1912.03368, 2019.
Md. Rafiqul Islam Rabin and Mohammad Amin Alipour. Evaluation of generalizability of neural program analyzers under semaatic-preserving transformations, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. ArXiv, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. ArXiv, 2019.
Henry Gordon Rice. Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical Society, 74(2):358-366 1953.
Fábio Santos et al. Terser: Javascript parser, mangler and compressor toolkit for es6+. https:
//github. wtt./ooooeeeeeeeee com/terrser/terser/ 2020.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Mike Schuster and Kuldip Paliwal. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45:2673 - 2681, 12 1997. doi: 10.1 199/77..650099
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv: 1508000999 2015.
Richard Shin, Neel Kant, Kavi Gupta, Christopher Bender, Brandon Trabucco, Rishabh Singh, and Dawn Song. Synthetic datasets for neural program synthesis. arXiv preprint 17X:::1912.12345. 2019.
Jeffrey Svajlenko, Judith F. Islam, Iman Keivanloo, Chanchal K. Roy, and Mohammad Mamun Mia. Towards big data curated benchmark of inter-project code clones. In Proceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution, ICSME '1,, pp. 476-480, USA. 2014. IEEE Computer Society. ISBN 9781479961467. doi: 10. 10.1109/CSME22044.77 URL
https://doi 000 wttg//0..1009/CC50 1109/ICSME 2014.77
Wilson L Taylo. "Cloze procedure": A new tool for measuring readability. Journalism Quarterly, 30(4):415-433, 1953.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. CoRR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-9999999 6008, 2017.
Ke Wang and Mihai Christodorescu. COSET: A benchmark for evaluating neural program embed- dings. arXiv preprint 1rXiv:1905.11444 2019.
Ke Wang and Zhendong Su. Learning blended, precise semantic program embeddings. ArXiv, 2019.
Jiayi Wei, Maruth Goyal, Greg Durrett, and Isil Dillig. Lambdanet: Probabilistic type inference using graph neural networks. In International Conference on Learning Representations, 2020.
Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual informa- tion in contrastive learning for visual representations, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3732-3742, 2018.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Conference on Computer Vision, pp. 649-666. Springer, 2016.
13

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 ADDITIONAL RESULTS AND ABLATIONS
Code clone detection ROC and PR curves 5 true postive rate vs false positive rate and precision recall for different zero-shot classifiers the code clone detection downstream tasks. These classifiers threshold a similarity score given by token-level edit distance for the heuristic approach or cosine similarity for the neural network representations. The hybrid self-supervised model combining ContraCode's contrastive objective and masked language modeling achieves better tradcoffss than the other approaches.
D-92 2H04
derasce MILEEEE
3.6
fanspastiaras
Figure 5: Receiver Operating Characteristic (ROC, left) and Rrecision-Recall (PR, right) curves for zero-shot classifiers on the code clone detection task. Equal F1 score curves are shown on right.
Which part of the model should be transferred? SimCLR CChnn et al. 2020a) proposed using small MLP head reduce the dimensionality of the representation used in the InfoNCE loss during pre-training, and did not transfer the MLP to the downstream image-classificatio task. In contrast, we find it beneficial transfer part of the contrastive MLP head to type inference. showing 2% improvement in top-5 accuracy over transferring the encoder only (Table We believe the improvement stems from fine-tuning both the encoder and MLP which allows feature adaptation, while SimCLR trained a linear model on top of frozen features. We only transferred the MLP when contrasting the mean token embeddings during pre-training, not the terminal hidden states, as the dimensionality of the MLP head differs. These representations are compared next.
Table 7: If local representations are learned, transferring part of the Contrastive MLP head improves type inference. The encoder is a 2-layer BiLSTM (d=512), with a 2-layer MLP head for both pre-training purposes and type inference. The mean hidden state representation is optimized for 10K iterations for the purposes of this ablation.
Transferred from pre- pre-training
Acc@1 1
Acc@5
Transfer BiLSTM
49.32
80.03%
Transfer BiLSTM, 1 layer of MLP 49.15%
82.58 %
Should we pre-train global local representations? We compare pre-training DeepTypprr with two variants of ContraCode. We either use the mean of token hidden states across the program (averaging local features), or the terminal hidden states as input the MLP used to extract the contrastive representation y (z) (global features). Token-level features might capture more syntactic details, but averaging pooling ignores order. Table S shows the accuracy of a BiL.STM pre-trained with each strategy. Using the global features for pre-training yiclds significantly improved performance, +2.38% acc@) after 10K iterations of pre-training (not converged for the purposes of ablation). The global pre-training strategy achieves the best results in Table 2
Do pre-trdi encoders help more with shallow decoders? For the seuuencc-to-sequeeee code summarization task, ContraCode only pre- pre-trains the encoder of the Transformer. In Table []] ablate the depth of the decoder to understand how much shallow decoders benefit from contrastive pre-training of the encoder. Similar experiments were performed in vision context by Erhan et al.
14

---

Under review as a conference paper at ICLR 2021
Table 8:
Contrasting global, sequence-level representations outperforms contrasting local representa- tions. We compare using the terminal (global) hidden states of the Deep'lyper BiLSTM and the mean pooled token-level (local) hidden states.
Representation
Optimization
Acc@) I
Acc@5
InfoNCE with terminal hidden state, 20K steps (Table InfoNCE with terminal hidden state, 10K steps
52.65 %
84.60 %
Global
83.03 51.70%
Local
InfoNCE with mean token rep., 10K steps
49.32%
80.03%
Table 9: Training time and decoder depth ablation the method name prediction task. Longer pre-training significantly improves downstream performance when a shallow, layer decoder is used.
Pre-training
Supervision (81k programs)
Decoder
Precision
Recall
F1
(1.8M programs)
Transformer, 1 layer
MoCo, 10k steps
Original set
11.91%
5.96%
7.49%
Transformer, 1 layer
MoCo, 45k steps
Original set
17.71 %
12.57%
13.79%
Transformer, 4 layers
MoCo, 45k steps
Original set
18.21 %
13.21% %
14.56%
(010)), where different numbers of layers of a classifier are pretrained. After 45k pre-training steps, the layer decoder achieves 0.50% higher precision, 0. 0.64% higher recall and 0.77% higher F1 score than the 1-layer model, additional decoder depth is helpful for the downstream task. The 1-layer decoder model also benefits significantly from longer pre-training, with a 6.3% increase in F1 from 10k to 45k iterations. This large of an improvement indicates that ContraCode could be more helpful for pre-training when the number of randomly initialized parameters at the start of fine-tuning is small. For larger decoders, more parameters must be optimized pre-training is diminished.
finttuninn and the value of
Contrastive representation learning strategies In Figure 6 we compare two strategies of refreshing the MoCo queue of key embed- dings (the dictionary of negative program representations assumed to be non-equivalent to the batch of positives). In the first strategy, we add 8 items out of the batch to the queue (1 x), while in the second use a larger queue (65k versus we add 96 items (12x). In addition,
Top accuracy
100
80
125k keys) and a slightly larger batch size (64 versus 96). We observe that for the baseline queue fill rate, the accuracy decreases for the first 8125 iterations as the queue fills. This decrease in accuracy is expected the task becomes more difficult due the increasing number of neg- atives during queue warmup. However, it is surprising that accuracy grows so slowly once the queue is filled. We suspect this is because the key encoder changes significantly over thousands of iterations: with momentum term m 0.999, the original key encoder parameters are decayed by a factor of 2.9 x 10-4 by the moving average. If the queue rapidly refreshed, queue embeddings are predicted by recent key encoders, not old parameters. This also indicates that large diversity of negative, non-equivalent programs are helpful for rapid convergence of ContraCode pre-training.
1x queue fill rate queue rate
20
0
Figure 6: Pre-training quickly converges if nega- tive programs in the queue are frequently changed.
A.2 QUALITATIVE RESULTS
t-SNE visualization of representations We qualitatively inspect the structure of the learned rep- resentation space by visualizing self-supervised representations variants of 28 programs using t-SNE (Maaten & Hinton! 2008) in Figure 7 Representations of transformed variants of the same program are plotted with the same color. ContraCode (BiLSTM) clusters variants closely together. Indeed, contrastive learning learns representations that are invariant to wide class of automated compiler- compiler-based transformations. In comparison, the representations learned by masked language mod- eling (RoBERTa) show more overlap between different programs, and variants do not cleanly cluster.
15

---

Under review as a conference paper at ICLR 2021
RoBERTA embeddings
ContraCoce embeddings
RoBERTA Contra embeddings
CINGL
TSNES
TSKEI
TONEI
Figure 7: t-SNE (Maaten Hinton 2008) plot of program representations learned with masked language modeling (RoBERTa), contrastive learning (ContraCode), and hybrid loss (RoBERTa + ContraCode). Transformed variants of the same program share the same color, though colors may be similar across different programs.
With a hybrid loss combining masked language modeling and contrastive learning, representations of variants of the same program again cluster.
Code summaries Figure 8 shows a qualitative example of predictions for the code summarization task. The JavaScript method not seen during training. A Transformer pretrained with ContraCode predicts the correct method name as the most likely decoding through beam search. The next four predictions are reasonable, capturing that the method processes an image. The 2nd and 3rd most likely decodings, getImageIten and createImage, use get and create as synonyms for load, though the final two unlikely decodings include terms not mentioned the method body.
function x(url, callback, error) { var img new Image(); img. url; if if(ing. complete){ return cingbackiimg);
Ground truth: loadImage Prediction: loadImage
img. onload function(){ img. onload null;
Other predictions:
getImageItem
cal calkback callback(img) (img);
2. createlmage
};
img. onerror function(e){ img. onerror = null; error(e);
3. LoadImageeForBreakpoint
4. getImageSreCSS
}:
}
Figure 8: A JavaScript program from the CodeSearchNet dataset not seen during training and the predicted method names from Transformer pre-trained with ContraCode. ContraCode predicts the correct method name as its most likely decoding.
Type inferences We can also visualize outputs of the type inference model. Figure shows two TypeScript programs from the held-out test set. User-provided type annotations are removed from the programs, and the model is provided with tokenized form without access to dependencies. We visualize predictions from variant of DeepTyper pretrained with ContraCode, the perf-perfonning model in Table 8 In the first program, our model consistently predicts the correct return and parameter type. While a tool based on static analysis could infer the void return types, the type of the message argument is ambiguous without access to the imported write method signature. Still, the model correctly predicts with high confidence that the variable message a string. In the second program, ContraCode correctly predicts 4 of 8 types including the ViewContainerRef and ChangeDetectorRef types, each imported from the AngularJS library. As this sample held-out from the training set, these predictions show generalization from other repositories using AngularIS.
16

---

Under review as a conference paper at ICLR 2021
Import: {
write,
categories, messagetype 's";
export const animttionsTreceCateppr "s")
export renderer/rracedateegoyy "s")
export const vievih:1Cataagory 's";
export costertracetategory 's";
export const cettslesserreeeeggaaaaaaaa
"s";
export const Hist/LenfraancCaaaaprry = "s";
export function animationsiog message: string 100.08 )1 (nessage, stinattoonTasceetteeyyy/
export function renderering [mag): vold 53.73
write rendererftreeenneeorry))
export function vendereršrror message: string 99.51 ): void 99.71
write(nessage, zenderertraeCateppryy nessagetype.crz00])
export function viewttillog (msg): vold 100.0V
write(meg, viewttilCateporyl
export routerlog ( message: string 99.91 ): 100.01
writee rontarfraceCategoryy).
export function toutsseesssttratespilog [ mensage: string 99.85 ): void 99.981 onese(ees,,ge, poutseueskctteetractteeee
export function styleError neasage: string 99.973 void 100.01
(nitsee,,, categories.Style. nesssgetype.erroo
export function listViewiog message: string
vold
write LartViinfrroetttegrry)
export function listViewError message: string 99.931 ): ...
Import {
ComponentReč, Componentracctory, VieuContainerRet, Component,
Type Conponant.PactoryRasooooo ChangeDetectorRef
} from "s"; import write
from "8"; export: const CATRGORY "s";
Logl message: string write (messsge, CATESORY);
Component((
selector: "s",
tamplate: "template"
}) export class DetachedIoader
constructorprrivate resolveri VienContainerRef
ComponentFactoorrkesooleee private changeDetector: ChangeDotectorRet 100.0s
private containerReti viewContainerket 100.0:
private losdIniocation
componentyppccany::: TemplataRef 99.63 (GP: Type)) <ComponentBef<aanpp: Pronise 100.03 03 factory const componentRef cels..cootiiieekee creatsccomooeett/ factory, bbts..ontainecret.. length, cotaanoettieereet. par.rratttt.prannnjjjootrto)
log("s " return Promise. peooiieeereeooleeeeepeeeeeeee
public detectChanges(( thise.chaneeeeeettr.. hot.eeeeeeettemmeiteccccecikk))
loadComponent
componentType<smys remplateRef 99.9t (GT) Type)) <ComponentRefcanyyh Promise 100.0t log "**");
return htt..aacctttccettiocccooooooroot
Figure 9: Our model, a variant of DeepTyperr pretrained with ContraCode, generates type annotations for two programs in the held-out set. The model consistently predicts the correct return type of functions, and even predicts project-specific types imported at the top of the file. The model corresponds to the top row of Table though is not our best performing model.
17

---

Under review as a conference paper at ICLR 2021
A.3 PROGRAM TRANSFORMATION DETAILS
We use the Babel compiler infrastructure (McKenzie et al. 2020) and the terser JavaScript library for AST-based program transformations. We perform variable renaming and dead code insertion (variable declaration insertion) using custom Babel transforms, subword regularization with sentencepiece Python tokenization library, line subsampling using JavaScript string manipulation primatives and other transformations with terser. Terser has two high-level transformation modes, mangling and compression, each with finer grained controls such as formatting, comment and log removal, and dead code elimination. We show an example merge sort with example equivalent variants in Figure [T]
Reformatting, beautification, compression (R, B, C): Personal coding conventions do not affect the semanties of code; auto-formatting normalizes according to style convention.
Dead-code elimination (DCE): In this pass, all unused code with no side effects are removed. Various statements can be inlined or removed as stale or unneeded functionality.
Type upconversion (T): In JavaScript, some types are polymorphic & can be converted between each other. As an example, booleans can be represented as true or as 1.
Constant folding (CF): During constant folding, all expressions that can be pre-computed at compi- lation time can be inlined. For example, the expression (2 3) 4 replaced with 20.
Variable renaming, identifier mangling (VR, IM): Arguments can be renamed with random word sequences and identifiers can be replaced with short tokens to make the model robust to naming choices. Program behavior is preserved despite obfuscation.
Dead-code insertion (DCT): Commonly used no-ops such as comments and logging are inserted.
Subword regularization (SW): From Kudo (2018). text is tokenized in several different ways, with a single word (_function) or subtokens (_func tion).
Line subsampling (LS): We randomly sample (p = 0.9) lines from method body. While not semantics-prest prreeriig........nngg line subsampling serves as a regularizer.
While compilers are generally deterministic, we require a variety of alternatives to each program for contrastive representation learning. Algorithm[[sampp samples N augmented variants of a source program using set of deterministic compiler transformations т;. Stochasticity is introduced by randomly toggling each transformation according to Bernoulli samples with probabilities pt. When adding a program to the set of variants V, uniqueness is determined by string comparison.
A.4 HOW SIMILARR ARE TRANSFOR MED PROGRAMS?
To understand the diversity created by program transformations, we compute the Levenshtein mini- mum edit distance between positive pairs in the precomputed pre-training dataset, i.e. transformed variants of the same source method. For comparison, we also compute the edit distance between negative pairs, i.e. transformed variants of different programs. The edit distance D D(sg .xk computes the minimum number of token insertions, deletions or substitutions needed to transform the tokenized query progrm Ty into the key program Tg.. To normalize by sequence length let
D(Eg 2xx)
dissimilarity (xg mk)
max(| [zgl |æx))
Dissimilarity ranges from 0% for programs with the same token sequence such as positives before applying our transformations, to 100% for programs without any shared tokens. Note that whitespace transformations do not affect the metric because the tokenizer collapses repeated whitespace. For the positives, we estimate dissimilarity by sampling dataset (1.6M source programs with at least one pair). We sample the same number of negative pairs,
pair per source program in the CodeSearchNet
Figure 10 shows histogram of token dissimilarity. Positive pairs have 65% mean dissimilarity, while negatives have 86% mean dissimilarity. Negatives are more dissimilar on average as source sequences could have different lengths, idioms and functionality. Still, the transformations generated quite different positive sequences, with less than half of their tokens shared. The 25th, median and 75th percentile dissimilarity is 59%, 66% and 73% for positives, and 82%, 87% and 90% for negatives.
18

---

Under review as a conference paper at ICLR 2021
Positives
0.06
Negatives
Fraction of pairs
0.05
0.04
0.01
0.00
60
80
100
Token difference (%)
Figure 10: Histogram of pairwise token dissimilarity for contrastive positives (transformed variants of the same method) and negatives (transformed variants of different methods). Code transformations produce positives with dissimilar token sequences.
A.5 EXPERIMENTAL. SETUP
Architectures The Transformer has 6 encoder layers (23M parameters) in all experiments, and 4 decoder layers for method name prediction in T leverage the default positional embedding function (sin, cos) as used in the original Transformer architecture. The network originally proposed in DeepTyper (Hellendoorn et al. 2018 had 1 11M parameters with 300 dimensional hidden state. We increase the hidden state size to 512 to increase model capacity, prediction has 17.5M parameters. During fine-tuning, across all experiments, we optimize parameters using Adam with linear learning rate warmup and decay. For the Transformer, the learning rate is linearly increased for 5,000 steps from 0) to maximum of 10-1 For the bidirectional LSTM, the learning rate is increased for between 2,500 and 10,000 steps to maximum of 10-3, Type inference hyperparameters are selected by validation top-1 accuracy.
BiLSTM for type
ContraCode pretraining The InfoNCE objective (1) is minimized with temperature 0.07 following He et al.) (2019). Also following He et al. (2019), D, the key encoder's parameters are computed with the momentum update equation 0₂ +- mb, + (1 m)fy, equivalent an EMA of the query encoder parameters 0g. To pretrain a Transformer using the ContraCode objective, we first embed each token the program using the Transformer. However, InfoNCE objective is defined in terms of single embedding for the full program. The ContraCode Transformer is pretrained with a batch size of 96. Our model averages the 512-dimensional token embeddings across the sequence, then applies two-layer MLP with 512 hidden units and ReLU activation to extract 128-dimensional program embedding for the loss.
The DeepTyper bidirectional LSTM architecture offers two choices for extracting global program representation. We aggregate a 4024-dimensional global representation of the program by concate- nating its four terminal hidden states (from sequence processing directions and two stacked LSTM layers), then apply the same MLP architecture as before to extract a 128-dimensional program representation. Alternatively, we can average the hidden state concatenated from each direction across the tokens in the sequence before applying the MLP head. We refer to the hidden-state configuration as a global representation and the sequence averaging configuration
local representation in We pre-train the BiLSTM with large batch size of 512 and apply weight decay.
Table g
Type prediction Following DeepTyper (Hellendoorn et al. 2018), our regenerated dataset for type prediction has 187 training projects with 15,570 TypeScript files, totaling 6,902.642 tokens. We tune hyperparameters on a validation set of 23 distinct projects with 1,803 files and 490,335 tokens, and evaluate on held-out test set of 24 projects with 2,206 files and 958,821. The training set is smaller than originally used in DeepTTyper as several projects were made private or deleted from GitHub before May 2020 when downloaded the data, but we used the same commit hashes for available projects so our splits are a subset of the original. We have released the data with our open-source code facilitate further work
stable benchmark more repositories are deleted over time. We perform early stopping to select the number of training epochs. We train each model for 100 epochs and select the checkpoint with the minimum accuracy @ metric (all types, including
19

---

Under review as a conference paper at ICLR 2021
// Split the array into halves and merge them recursively function mergeSart (arr)
if (arr. length // return once we hit an array with single item return arr
1) {
const middle Math. get the middle item of the array rounded down const left arr. slrcslic,(0, middle) // items on the left side const right arr. slice(middle)) items the right side return merge(
(rrgeoor((eef))
mergeSort (right)
Original merge sort program
function mergeSart(e) { if (e.length === 1) {
} return e;
const Math. floor (e. 2); const e. sl t);
const slice
return merge (m)) mergeSort(n));
After variable renaming, comment removal, reformatting (mangling)
function mergeSort(e)
if (1 == length) return const Math. floor length 2), r slice t), e.slice(t); return merge (mergeSort(r), mergeSort(n));
After combining variable declarations, inlining conditional (mangling and compression)
Figure 11: Given a JavaScript code snippet implementing the merge sort algorithm, we apply transformations to produce functionally-equiialeee yet textually distinct code sequences. Compression passes eliminates unnecessary characters such as redundant variable decla- rations and brackets, while mangling passes can change variable names.
any) on the validation set. Except for the model learned from scratch, the Transformer architectures pre-trained for 240K steps. Models with the DeepTyper architecture converge faster the pre-training tasks and are pre-trained for 20k iterations (unless otherwise noted).
Extreme code summarization by method name prediction We train method prediction models using the labeled subset of CodeScarchNet. Neither method names nor docstrings are provided as input to the model: the docstring deleted, and the method name replaced with the token 'x'. Thus, the task is to predict the method name using the method body and comments alone. To decode method names from all models except the code2vec and code2seq baselines which implement their own decoding procedures, we use beam search with beam of size 5 and maximum target sequence length of 20 subword tokens. We detail the cumulative distribution program lengths Figure [2 The ContraCode summarization Transformer only needed be pre-trained for 20k iterations, with substantially faster convergence than RoBERTa (240k iterations). During fine-tuning we apply the L.S.S.VR,DCI VR, augmentations to ContraCode.
20

---

Under review as a conference paper at ICLR 2021
0.75
0.50
0 0.50
0.25
0.25
5000
10000
15000
20
40
60
Method body length
length
(a) Character length per code sample
(b) Character length per method name
Figure 12: CodeSearchNet code summarization dataset statistics: (a) The majority of code sequences under 2000 characters, but there is long tail of programs that span up to 15000 characters long, (b) JavaSeript method names are relatively short compared to languages like C2 and Java.
A.6 BASELINES
Baselines for code summarization and type prediction trained their models inconsistent set of programming languages and datasets. In order to normalize the effect datasets, we selected several diverse state-of-the-art baselines and reimplemented them the JavaScript dataset.
AST-based models The authors of code2vec (Alon et al. 2019b) and code2seq (Alon et al. 2019a), AST-based code understanding models, made both data and code available, but train their model on the Java programming language. In order to extend the results in their paper to JavaScript for comparison with our approach, we generated an AST path dataset for the CodeSearchNet dataset. The sensitivity of path-mining embeddings to different datasets is documented in prior work, so published F1 scores are not directly comparable; F1 scores for code2vec (Alon et al. 2019b) vary between 19 (Alon et al. 2019a) and 43 (Alon et al.) 2019b) depending on the dataset used. Therefore, we use the same dataset generation code as the authors for fair comparison. We first parse the source functions using the Babel compiler infrastructure. Using the original code on these ASTs, up 300 token-to-token (leaf-to-leaf) paths are extracted from each function's AST as a precomputed dataset. Then, generate token and AST node vocabulary using the same author-prov provided code, and train the models for 20 epochs, using early stopping for code2seq. We observed that code2vec overfits not beneficial.
after 20 epochs, and longer training
DeepTTyper (Hellendoorn et al., 2018) DeepTyper uses a two layer GRU with a projection over possible classes, with embedding size of 300 and hidden dimension of 650. However, we found improved performance by replacing the GRU with a bidirectional LSTM (BiLSTM). We normalize the LSTM parameter count to match our model, and therefore
hidden dimension size of 512. We also use subword tokenization rather than space delimited tokens according to Kudo (2018). as subword tokenization is a key part of state-of-the-at models for NLP (Sennrich et al. 2015).
RoBERTa We pre-trained an encoder using RoBERTa's masked language modeling loss on our augmented version of CodeSearchNet, the same data used pretrain ContraCode. This model is then fine-tuned on downstream datasets. Unlike the original BERT paper which cuBERT (Kanade al. 2020)) based on, hyperparameters from RoBERTa have been found to produce better results during pre-training. RoBERTa pre-trains using masked language modeling (MLM) objective, where 15% of tokens in sentence are masked or replaced and are reconstructed by the model. We did not use the BERT Next Sentence Prediction (NSP) loss which RoBERTa finds to be unnecessary. We normalize baseline parameter count by reducing the number of Transformer layers from 24 to 6 for a total of 23M parameters.
21


Reviews:
[{"review_id": "wqwdSyuW6L", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper proposes Contrastive Code Representation Learning (ContraCode), a\nself-supervised algorithm to learn task-agnostic semantic representations of\nprograms via contrastive learning. The guiding principle for contrastive\nlearning is that programs with the same functionality should have the same\nrepresentation. The authors develop an automated source-to-source compiler to\ngenerate different (most of the time equivalent) variants of the same program.\nThe neural network, which is basically identical to the Momentum Contrast\narchitecture, is trained using the programs generated by this compiler.\n\n## Pros\n\n- A very nice and intuitive application of Momentum Contrast to code\n  representation learning.\n- The use of source-to-source compilers for contrastive learning.\n- The results are consistently better than previous state-of-the-art on two\n  different downstream tasks.\n- The authors have done extensive ablation studies which provide further\n  insight.\n\n## Concerns\n\n- I believe that the most emphasized result -- 40% higher top-5 accuracy than\n  the current state-of-the-art static type analyzer for TypeScript -- is\n  misleading. As far as I know, TypeScript's built-in type inference returns\n  only a single suggestion. This is reinforced by the fact that Acc@1 and Acc@5\n  is exactly the same for CheckJS in Table 2. So comparing Acc@5 between the two\n  algorithms is not fair and it is pointless.\n- In Section 3.2, the extent to which He et al. (Momentum Contrast) is followed\n  is not clear enough. Part of the method that's described in \"Pre-training\n  objective\" strictly follows/summarizes Momentum Contrast but that's not\n  apparent from the paper.\n- In 4.1., I find \"cross-language knowledge transfer\" a bit of an overstatement,\n  as TypeScript is a superset of JavaScript.\n- In 4.2., the difference in F1 score between Transformer and\n  Transformer+ContraCode+augmentation is very small. I would like the authors to\n  discuss this.\n- In 4.2., I find the statement \"showing that code token reconstruction is not\n  an effective pre-training strategy\" too general.\n\n## Questions\n\nI have questions about the transformations done with the source-to-source\ncompiler, which could also be made clearer in the paper.\n\nFigure 3 shows the uniqe transformed program variants after applying 20\nsequences of random transformations.\nHow many transformations were done in a sequence?\n\nIn 3.2, the authors write that each program is transformed twice (similarly to\nMomentum Contrast). However, in Section 4, Pre-training dataset, they write that\nthe augmented dataset is pre-computed by sampling up to 20 unique transformed\nvariants per program.\n\nDoes this mean that the two transformed programs are sampled from these 20?\nHow many transformations were done in a sequence? Is that the same as for Figure 3?\n\n## Reasons for Ranking\n\nMy main concern is the +40% Acc@5 claim compared to CheckJS, which I find\nmisleading. However, I think that this is an interesting paper and a valuable\ncontribution. If my concerns are addressed I'm willing to improve my score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ReLyrLXY4aU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to combine contrastive learning with code representation. The different transformations of code snippets are inspried by static complier.\n\nMy mainly concern is about the novelty. I agree with the claim about programs with the same functionality should have the same underlying representation. However, it's unclear to me that why using it as contrastive loss is a better choice than MLM loss in code understanding downstream tasks, especially for type inference. Any theoretical or intuitive explaination is good.\n\nIt seems that the performance gain about the proposed method is overcliamed, especially for the 40\\% top-5 accuracy gain of TypeScript which is a deterministic method. The actual gain compared to SOTA learning-based method is less than 3\\%. Also, the experimental results are unconvincing to me, for example, pre-training with MLM loss (then finetune on the downstream task? The corresponding descriptions are not clear) get poor accuracy on type inference task. The authors intuitively explain it as that MLM loss is not  suitable for this kind of task. However, it performs better when combining with contrastive loss than only using contrastive pre-training.  Whether on earth MLM loss is good for this task? \n\nSome necessary baseline methods are missing. For example, [1] and [2] for code summerazation task . And some important downstream tasks are also missing to demonstrate the ability of proposed method in code understanding, e.g., code clone detection (which I personally think that is more suitable for contrastive code representation).\n\n[1] A Transformer-based Approach for Source Code Summarization, ACL 2020\n[2] https://github.com/microsoft/CodeXGLUE/tree/main/Code-Text/code-to-text\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5w4Riya4Skg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the self-supervised code functional representation learning and proposes a method called ContraCode. ContraCode utilizes some code functionality invariant transformations to generate positive pairs from the same code and negative pairs from different codes. After that, these codes pairs will be used to do the contrastive pre-training. Experiment results based on two tasks are reported.\n\nPros:\n-\tThe task of  code functional representation learning is important and valuable.\n-\tThe transformations proposed in this paper may produce some vaviance to the code while maintaining the same functionality.\n\nCons:\n-\tThe superiority of the proposed method is unclear. Many self-supervised code representation learning methods are mentioned in the introduction, such as [Ben-Nun et al., 2018; Feng et al., 2020; Kanade et al., 2020]. However, this paper fail to discuss of the differences (especially the advantages) between ContraCode and other self-supervised methods empirically.\n-\tSince no addtional supervision is evolved, unsupervised feature leanring models are good competing baselines.. The authors are  strongly recommended to compare the performance of ContraCode with other unsupervised methods under the same training dataset (both augmented).\n-\tThe key question is the whether the self-supervision generated by such transformation really makes any difference. Some transformation only change the formatting, which usually resulting the same feature representation because the formatting information is usually not considered in most of the feature learning methods for code. It appears that by applying the set of transformation, the code would not differ from its previous appearance much. Consequently, the feature representations generated by some unsupervised method from the original code and its transformed counterpart could be very similar to each other EVEN IF no self-supervision is enforced, which means self-supervision is not necessary.  Please clarify this be providing empirical evidences such as the portion of the changed lines or tokens from the original code, the similarity between the original code and its transformed counterpart over any two different pieces of code based on the features learned in some unsupervised way (with the same scale of training data).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a nice paper using contrastive learning for code representation. The idea is to generate variations on unlabeled source code (using domain knowledge) by creating equivalent version of code. Improvements over baselines on two multiple tasks are shown. While some of the reviewers liked the (and R4 should have responded), none of the reviewers found the paper exciting enough to strongly recommend its acceptance. 
Decision: Reject