Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper studies the self-supervised code functional representation learning and proposes a method called ContraCode. ContraCode utilizes some code functionality invariant transformations to generate positive pairs from the same code and negative pairs from different codes. After that, these codes pairs will be used to do the contrastive pre-training. Experiment results based on two tasks are reported.

Pros:
-	The task of  code functional representation learning is important and valuable.
-	The transformations proposed in this paper may produce some vaviance to the code while maintaining the same functionality.

Cons:
-	The superiority of the proposed method is unclear. Many self-supervised code representation learning methods are mentioned in the introduction, such as [Ben-Nun et al., 2018; Feng et al., 2020; Kanade et al., 2020]. However, this paper fail to discuss of the differences (especially the advantages) between ContraCode and other self-supervised methods empirically.
-	Since no addtional supervision is evolved, unsupervised feature leanring models are good competing baselines.. The authors are  strongly recommended to compare the performance of ContraCode with other unsupervised methods under the same training dataset (both augmented).
-	The key question is the whether the self-supervision generated by such transformation really makes any difference. Some transformation only change the formatting, which usually resulting the same feature representation because the formatting information is usually not considered in most of the feature learning methods for code. It appears that by applying the set of transformation, the code would not differ from its previous appearance much. Consequently, the feature representations generated by some unsupervised method from the original code and its transformed counterpart could be very similar to each other EVEN IF no self-supervision is enforced, which means self-supervision is not necessary.  Please clarify this be providing empirical evidences such as the portion of the changed lines or tokens from the original code, the similarity between the original code and its transformed counterpart over any two different pieces of code based on the features learned in some unsupervised way (with the same scale of training data).

review_title: Review for "Contrastive Code Representation Learning"

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work proposes to combine contrastive learning with code representation. The different transformations of code snippets are inspried by static complier.

My mainly concern is about the novelty. I agree with the claim about programs with the same functionality should have the same underlying representation. However, it's unclear to me that why using it as contrastive loss is a better choice than MLM loss in code understanding downstream tasks, especially for type inference. Any theoretical or intuitive explaination is good.

It seems that the performance gain about the proposed method is overcliamed, especially for the 40\% top-5 accuracy gain of TypeScript which is a deterministic method. The actual gain compared to SOTA learning-based method is less than 3\%. Also, the experimental results are unconvincing to me, for example, pre-training with MLM loss (then finetune on the downstream task? The corresponding descriptions are not clear) get poor accuracy on type inference task. The authors intuitively explain it as that MLM loss is not  suitable for this kind of task. However, it performs better when combining with contrastive loss than only using contrastive pre-training.  Whether on earth MLM loss is good for this task? 

Some necessary baseline methods are missing. For example, [1] and [2] for code summerazation task . And some important downstream tasks are also missing to demonstrate the ability of proposed method in code understanding, e.g., code clone detection (which I personally think that is more suitable for contrastive code representation).

[1] A Transformer-based Approach for Source Code Summarization, ACL 2020
[2] https://github.com/microsoft/CodeXGLUE/tree/main/Code-Text/code-to-text

review_title: Contrastive Learning on Source Code Representation

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary

The paper proposes Contrastive Code Representation Learning (ContraCode), a
self-supervised algorithm to learn task-agnostic semantic representations of
programs via contrastive learning. The guiding principle for contrastive
learning is that programs with the same functionality should have the same
representation. The authors develop an automated source-to-source compiler to
generate different (most of the time equivalent) variants of the same program.
The neural network, which is basically identical to the Momentum Contrast
architecture, is trained using the programs generated by this compiler.

## Pros

- A very nice and intuitive application of Momentum Contrast to code
  representation learning.
- The use of source-to-source compilers for contrastive learning.
- The results are consistently better than previous state-of-the-art on two
  different downstream tasks.
- The authors have done extensive ablation studies which provide further
  insight.

## Concerns

- I believe that the most emphasized result -- 40% higher top-5 accuracy than
  the current state-of-the-art static type analyzer for TypeScript -- is
  misleading. As far as I know, TypeScript's built-in type inference returns
  only a single suggestion. This is reinforced by the fact that Acc@1 and Acc@5
  is exactly the same for CheckJS in Table 2. So comparing Acc@5 between the two
  algorithms is not fair and it is pointless.
- In Section 3.2, the extent to which He et al. (Momentum Contrast) is followed
  is not clear enough. Part of the method that's described in "Pre-training
  objective" strictly follows/summarizes Momentum Contrast but that's not
  apparent from the paper.
- In 4.1., I find "cross-language knowledge transfer" a bit of an overstatement,
  as TypeScript is a superset of JavaScript.
- In 4.2., the difference in F1 score between Transformer and
  Transformer+ContraCode+augmentation is very small. I would like the authors to
  discuss this.
- In 4.2., I find the statement "showing that code token reconstruction is not
  an effective pre-training strategy" too general.

## Questions

I have questions about the transformations done with the source-to-source
compiler, which could also be made clearer in the paper.

Figure 3 shows the uniqe transformed program variants after applying 20
sequences of random transformations.
How many transformations were done in a sequence?

In 3.2, the authors write that each program is transformed twice (similarly to
Momentum Contrast). However, in Section 4, Pre-training dataset, they write that
the augmented dataset is pre-computed by sampling up to 20 unique transformed
variants per program.

Does this mean that the two transformed programs are sampled from these 20?
How many transformations were done in a sequence? Is that the same as for Figure 3?

## Reasons for Ranking

My main concern is the +40% Acc@5 claim compared to CheckJS, which I find
misleading. However, I think that this is an interesting paper and a valuable
contribution. If my concerns are addressed I'm willing to improve my score.

review_title: Interesting paper with some debatable claims

Meta-review: This is a nice paper using contrastive learning for code representation. The idea is to generate variations on unlabeled source code (using domain knowledge) by creating equivalent version of code. Improvements over baselines on two multiple tasks are shown. While some of the reviewers liked the (and R4 should have responded), none of the reviewers found the paper exciting enough to strongly recommend its acceptance. 
Decision: Reject