CCT: Cross-consistency training for Clone Detection and Code Search Tasks
Authors: Nikita Sorokin
Dmitry Abulkhanov
Valentin Malykh
Venue: Submitted to ICLR 2023 (2023)
Keywords: pre-training
language model

Abstract: Clone detection is a well known task, which could be formulated on any programming language. Although to the best of our knowledge there is no cross-lingual clone detection task formulated. In this work we formulate such a task alongside with a specific training procedure CCT for a deep leaning language model. This procedure allows CCT-trained model to outperform the existing approaches on POJ-104 benchmark with result of 95.67\% MAP and on newly created cross-lingual clone detection benchmark XCD. Moreover, CCT model shows new state of the art results in code search task AdvTest 47.15\% MRR.

Under review as a conference paper at ICLR 2023
CCT: CROSS-CONSISTENCE TRAINING FOR CLONE DE- TECTION AND CODE SEARCH TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
Clone detection is well known task, which could be formulated any program- ming language. Although to the best of our knowledge there is no cross-lingual clone detection task formulated. In this work we formulate such a task alongside with a specific training procedure CCT for a deep leaning language model. This procedure allows CCT-trained model outperform the existing approaches on POJ- 104 benchmark with result of 95.67% % MAP and on newly created cross-lingual clone detection benchmark XCD. Moreover, CCT model shows new state of the art results in code search task AdvTest 47.15% MRR.
INTRODUCTION
In software development practice in it is sometimes important identify the code with the same effective output. This could be useful e.g. for unification and control of side effects. To meet this need there was clone detection task formulated (Mou et al., 2016). In the mentioned work the task is formulated for C/C++ programming code. Although for compiled languages it seems to be the most profitable detect similar behaviour of the code, instead of compiling and running, still could be useful for many other languages, including interpreted ones. The next step which could be done, one could detect the same output for the programming code in different languages'. We formulate cross-lingual clone detection task and establish some baselines on it.
There are various approaches to solve clone detection task problem starting from algorithmic based methods (?) and continuing with modern machine learning ones (). Most machine learning approaches are based on embedding representation of the code snippet. This approach allows find duplicate code snippets by similarity between their embedding representation. The performance of such systems depends on the quality of obtained embeddings. We present novel technique of training (CCT) for language models which allows them to embed the code snippets in effective way. We demonstrate this on previously formulated clone detection task POJ-104 (Mou et al., 2016) and on newly formulated cross-lingus clone detection task XCD.
Interestingly, we also found out that this CCT technique allows a model to produce representations useful for code search task also. Code search itself is task where code snippet should be mapped to some text description, as formulated (Lu et al., 2021a).
The contributions of our work is as follows: (i) present pre-training method CCT allowing a model to align code snippets in different languages; (ii) we present novel cross-lingual clone detection task XCD; (iii) we present results for language model trained with CCT on clone detection tasks 104 and XCD; (iv) we present the results of CCT-model for code search AdvTest task. 2
DATASETS
In our work we use two types of the datasets, one is for clone detection, the other is for code search.
As example of the programmes with the effective output one could refer to http:// he1 oworl doo1 lection in de/ website, which contains "Hello, world!" snippets in 603 programming languages.
We are going relcase CCT code after the review process is over.

---

Under review as a conference paper at ICLR 2023
Cross-Lingual Alignment
WEAK
STRONG
A query
relevant
cpp
python
context
non-relevant
java
Figure I: Here is difference between strong and weak cross-lingual alignment. In strongly aligned embedding space, the most semantically similar items are always the closest, regardless language. weakly aligned multilingual embedding space just enables zero-shot transfer between languages.
2.1
CODE SEARCH
For code search we use CodeSearchNet dataset first described in (Husain al., 2019a). The original version of this dataset consists of natural languages queries paired with most relevant code snippets written six programming languages. Each snippet is code function collected from GitHub open source code. (Lu et al., 2021a) modified CodeSearchNet and created Adv dataset.
CodeSearchNet AdvTest is Python language only dataset constructed from the CodeSearchNet corpus. Each example includes function paired with a document. The authors AdvTest followed the original work (Husain et al., 2019a) in taking the first paragraph of the documentation as the query for the corresponding function. To improve the quality of the dataset, they filter it by removing the following examples:
Examples whose code could not be parsed into abstract syntax tree.
Examples whose document is shorter than 3 or larger than 256 tokens.
Examples whose document contains special tokens such "http://".
Examples whose document is empty or not written in English.
The filtered dataset contains 251,820 9,604 19,210 examples for raaining/alidation/eee respec- tively. The idea of this dataset is based on the following observation. After normalizing function or variable names with special tokens, the authors observe that the Mean Reciprocal Rank (MRR) scores of RoBERTa (?) and CodeBERT (?) for the code search task on original CodesearchNet (Husain al., 2019a) dataset drop from 0.809 to 0.419 and from 0.869 0.507, respectively, in Python programming language. To better test the understanding and generalization abilities of the model, they normalize function and variable names in testing and development sets like func for the function name and arg; ;; for the i-th variable name. The task aims to search source codes from candidates for a natural language query. In contrast to the testing phase of previous works (Husain et al., 201 9a; the entire testing set for each query. which
?) that only involved 1,000 candidates, the authors makes CodeSearchNet Adv dataset more difficult. The training set for this task comes from the filtered Net dataset (Husain al., 2019a).
2

---

Under review as a conference paper at ICLR 2023
As metric for AdvTest MRR is used. It is defined
MRR 20
rank;
where is set queries and rank is position of ground truth answer document among the ranked candidates.
2.2 CLONE DETECTION
The task aims retrieve semantically similar codes given code the query and use POJ-104 dataset to perform it. POJ-104 dataset described in (Mou et al., 2016) comes from a pedagogical programming open judge (OJ) system that automatically judges the validity of submitted source code for specific problems by running the code. We use the POJ-104 dataset, which consists of 104 problems and includes 500 student-written C/C++ programs for each problem. The task of POJ-104 aims to retrieve other programs that solve the same problem given a program. The problems are grouped in three sets (64/16/24) for training, validation, and testing respectively.
The metric for POJ-104 dataset is Mean Average Precision (MAP). We start with defining average precision (AP).
100
AP
Σ RR, R2-1)
where R and P; are the precision and recall at the threshold, i.e. they are computed taking into account only top i items from the candidate list. MAP is the mean of AP over all the queries. It important to mention that for 104-104 the maximal possible is 499, since there only 500 candidates in total.
2.3 CROSS-LINGUAL CLONE DETECTION
In previous works, cross-lingual and multilingual abilities CodeLM were not sufficiently investi- gated. To fill this gap, we introducing (XCD). Dataset constructed in three different setups: in similar to BUCC dataset (Xu et al., 2018) retrieval way, in similar code clone detection way POJ-104 (Mou al., 2016) and in hybrid way. Moreover, we investigated zero-shot transfer from python to Java, Ruby, PHP, Go and JavaScript on NodeSearchNet dataset for previously introduced CodeLM models and our CCT LM.
Cross-lingual Clone detection/Code retrieval dataset
As a data source CodeForces submissions dump was leveraged. We filtered 100 problems, which used during pre-training. For similar to BUCC approach, randomly choose accepted solutions per problem on 9 different program language. Positive examples are accepted solutions for same problems in different language. In total, for each language in the corpus we mined 50000 randomly chosen submissions, and 10000 of them have positive pair. The dataset was split dev and test 20/80 respectively.
For code clone retrieval approach we follow (Hu et al., 2020; Tien & the models are evaluated on the dev and test split of the corpora and the threshold of the similarity score cutting off translations from non-translations is optimized for each language pair. The similarity scores are calculated based on dot products of CLS vectors representations, also in pre-training. Our task is interpreted binary classification, thus classic Fl is used.
2022), where
For Clone detection approach, we follow POJ-104 dataset design. For 100 problems were mined, 100 accepted solutions per problem different program languages. The task aims to retrieve 100 snippets per language solving same problem from the test set in zero-shot setup.
Finally, for hybrid approach, we leverage data from first dataset and retrieval method from the second. As a metric we chose MRR@20. is the same metric as in AdvTest.
2.4 ADDITIONAL MARKUP
In addition to the solution status ("Accepted" or not), we also mined the statuses of errors in the solutions, since the platforms used for problem solving are often provide them. Finally, we mined more than 97 million code snippets in more than 10 programming languages.
3

---

Under review as a conference paper at ICLR 2023
2.4.1 VERDICTS
CodeForces platform can return 15 types of verdicts for submitted solution. We split verdicts on groups: Defect (code marked as contained defect), Skip (code which cannot be judged), Accepted (code without any detected defect), Wrong solution (code failing some tests respectively). Below we describe the most common of verdicts:
Memory limit exceeded The program tries to consume more memory than is indicated in the problem statement. (Wrong solution)
Time limit exceeded The program hadn't terminated in time indicated in the problem statement. (Wrong solution)
Runtime error The program terminated with non-zero return code (possible reasons: array out of bound error, division by zero, stack overflow, incorrect pointers usage, etc.) (Defect)
Wrong answer Wrong answer. (Wrong solution)
Idleness limit exceeded The program did not use the CPU time for considerable time. (Defect)
Denial of judgement, Judgement failed The solution was impossible to run, perhaps, due judging error. The most probable cause is an error in the program (for example, using extra large arrays). (Defect)
Rejected on The program does not pass tests and we do not know why is happened. (Skip)
Accepted Solution passed tests. (Accepted)
METHOD
In this section, we will describe our pre-training approach (CCT). The goal of our method is to robustly learn the embedding space of code snippets and create strong alignment between code solving the same problems across different programming languages. To achieve this, introduce new contrastive learning objective: for a randomly masked code snippet (extracted from XCD in our experiments), train the CLS vector such that the CLS embedding will be closer to the snippet in another language or user but solving the same problem or the same snippet but masked differently than random similar but different (hard negative) snippet. More formally, for given random problem from CodeForces we have n relevant snippet D = [d1, d2,,.,, da] in different languages from L [4, l2,., [n] and j E I...n with p(j) D loss:
We optimize noise contrastive estimation
sim((d;,d))
Lcontr
llog (dim(d dx) + csim(d,dd))
So in general, code solving the same problem should have similar representations and the ones with different problems should have different representations.
Also in final loss function we specified another two: error detection loss Lbug and standard language modelling loss Lim. Error detection loss is native for code data and can help model to deeply understand code syntax and recognise errors. Task is formulated such as binary classification of code which can raise error during code execution. The language modelling loss is standard masked token prediction loss, which is widely used for language models, including the models trained on programming code.
Our final loss function composes all the listed ones: L -cootr + Liuu + Lim-
3.1 HARD NEGATIVE MINING
Previous works in contrastive learning (Qu et al., 2021; Izacard & Grave, 2020) show the importance of hard negatives examples. All of them are using iterative training to get the ones. However, our data already contain some strong negative examples, namely, solutions solving the same problem from same users but failing the tests. Thus, our hard negative examples are mined as failed solution from the same user exists. If there is no such solution, then a failed solution from a random user used. If there is no such solutions, then
simply random submission.
4

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
In this section, we will describe details about parallel data pre-training and pipeline for fine-tuning for cross-lingual open domain question answering and cross-lingual sentence retrieval tasks.
4.1 PRE-TRAINING
Generally, we initialize our model with pre-trained GraphCodeBERT base (Guo et al., 2020). We call these models CC-LMaase We Adam optimizer with learning rate 1c-4, weight decay of 0.01, and linear learning rate decay. We also use gradient cashing for pre-training, similar to (Gao & Callan, 2021). The accumulated batch size equal to 500. We train models on 4 NVIDIA Tesla V100 GPUs.
4.2
RESULTS
Clone Detection MAP
Code Search MRR
Metric
RoBERTa-base (Liu et al., 2019)
76.67
18.33
CodeBERT (Feng et al., 2020b)
82.67
27.19
SynCoBERT (Wang et al., 2021a)
88.24
38.1
CodeRoBERTa
42.35
GraphCodeBERT (Guo et al., 2020)
85.16
CasCode (Gotmare al., 2021a)
43.98
(Villmow et al., 2022)
91.34
CCT-LMaase
96.73
47.18
Table 1: Results on code clone detection on POJ-104 and Code Search on AdvTest
In Tab. we present the results of our CCT-LM model (a model pre-trained with proposed CCT method) in comparison to the existing approaches. As one can see our CCT-LM outperforms all the previous models by a large margin.
F1
python
java
csharp
ruby
haskell
php
ocaml
perl
Avg
graphcodebert-barg
0.02
0.05
0.04
0.00
0.02
0.01
0.01
0.02
grapheodebert-bas--PO
0.04
0.00
0.01
0.06
0.07
0.08
0.06
0.06
0.06
0.05
CCT-LMM,ase
22.24
18.39
17.33
23.33
10.46
17.64
21.43
17.01
16.40
18.24
Table 2: Performance cross-lingual code retrieval dataset.
4.2. CROSS-LINGUAL RESULTS
Results for cross-lingual code retrieval task are presented in Tab. As it can be seen, task the more complex than similar task for Natural language, as reference interesting, that we cannot track knowledge transferring from POJ-104 dataset as the metrics are low. However, CCT approach shows better result due more obvious cross-lingual pre-training.
(Sorokin al., 2022). It is
MAP@ 100
python
java
csharp
ruby
js
haskell
php
ocaml
perl
Avg
graphcodebert- CCTLM
7.21
9.25
1.33
4.28
1.59
5.78
2.90
10.37
87.42
55.99
65.35
72.12
74.32
81.05
83.21
71.53
71.89
73.65
Table 3: Performance on clone detection cross-lingual dataset.
For clone detection setup the results presented in Tab. 3. Similarly POJ-104 leverage MAP@ 100 since for every example we have 100 positive duplicates. As one can see, our method is strongly outperforms the baseline.
5

---

Under review as a conference paper at ICLR 2023
MRR @2
python
java
esharp
ruby
js
haskell
php
ocaml
perl
Avg
BM25
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
grapheodebert-bas base
2.08
5,42
0.22
2.59
0,80
1,99
2.90
1,40
5.23
2.51
graphcodehert-base- -astertt-aaee------- 104
27.10
20.04
19.44
30.98
28.37
19.70
32.89
30.08
39.98
27.62
CCT4M
74.97
62.08
58.77
80.60
74.56
62.27
81.21
72.64
79.16
71.80
Table 4: Performance on hybrid cross-lingual dataset.
Results for hybrid seen more interpreted 4. Ability of Code language models transfer knowledge between programming language more obvious, since metries improvement noticeably greater between fully unsupervised method and pre-training BM25 (a strong baseline for natural language information retrieval tasks) does not work for cross- lingual code retrieval task. That shows ability of CodeLM models to cross-lingual semantic search.
monolingual clone detection. Moreover,
5 ANALYSIS
Fig. 2a shows the abstract representation of the idea that semantically aligned embedding space is language-agnostic. On Fig. 2b there are the actual embeddings for samples in six languages before and after CCT training. As can see, CCT representations of code snippets not aligned by the same language. Thus we can conclude that CCT training significantly improves the semantic closeness for different language code snippets. The code snippets here solutions for sampled 12 tasks from CodeForces on different programming languages.
5.1 ABLATION STUDY
Clone Detection MAP
Code Search MRR
Defect-detection Accuracy
Metric
GraphCodeBERT
85.16
45.80
62.51
GraphCodeBERT Lcontr GraphCodeBERT + Lcontr + LLM
95.92
29.93
61.05
95.67
47.18
63.68
GraphCodeBERT
96.03
45.22
64.91
GraphCodeBERT + +C+++++ C+ s.l.
96,46
47.33
GraphCodeBERT C s.l. v.f.
96.73
47.57
65.58
Table 5: Results on code clone detection on POJ-104 and Code Search on Adv Test; s.l. stands for size limitation, v.f. verdict verification.
In this section, will discuss the effectiveness of the proposed approach. Tab. displays the result of retrieval on clone detection, Code search, and Defect detection tasks. We decided to add Defect detection task for our analysis, since part of loss easily aligned with this task. As dataset for it we used the one presented in (Zhou et al., 2019). This dataset contains 21,854/2,,32/2,7322 for train/dev/test split. As metric it uses straightforward Accuracy.
We compare here the GraphCodeBERT base model with different pre-training loss, verdict filtration, this size limitation. How can be seen, changes gives not uniform metries gain. Pure contrastive loss significantly reduce quality on all tasks but significantly improve MAP on Clone Detection task. Adding masked language modelling loss fill in these metrics gap, probably due the representations from the general pre-training explained by similarity of CF data and functions from Code search data. Loug improve quality for significantly for Defect detection task and little bit for the clone detection Limitation of size significantly reduce number code snippets and problems respectively. However, it improves metrics for all tasks. Same result for verdict filtration.
saved better. The biggest increase for Code Search task, can be
Generally, the increase from the change is uneven and it is possible to achieve a better result in other configurations.

---

Under review as a conference paper at ICLR 2023
CodeBERT
Our Approach
problem
D18
G656
G784 D926
D795 C953
A795
K774
J795
E795
F953
F926
(a) Projected embeddings of sampled 12 coding problems.
CodeBERT
Our Approach
language
java
cpp
python
csharp
ruby
(b) Same embeddings differentiated by programming language.
Figure Analysis of same-language bias the cross-lingual embedding space.
F1
java
ruby
php
go
js
Avg
CodeBertsasee
46.37
50.65
37.83
50.65
50.48
47.19
GraphCode
47.33
59.95
37.47
60.28
52.04
51.41
CCT-LMaase
48.71
62.25
42.78
61.44
51.06
53.24
Table 6: Zero-shot retrieval on CodeSearchNet dataset.
interesting to analyse the zero-shot results presented at Tab. 6. As one can see, the existing approaches showing pretty good results, though they have not been pre-trained retrieval
7

---

Under review as a conference paper at ICLR 2023
task. It is additional evidence of the pre-trained language models' power. Although, our method is leveraging its cross-lingual ability and improves over the baselines in zero-shot setup. However, for JavaSeript (js) language pre-training method CCT shows lower performance than the baseline, we leave this fact for further investigation.
6 RELATED WORK
Our methods are inspired by natural language processing field, thus present some works from this field in addition to the programming language processing ones.
Datasets To the best of our knowledge, the first attempt to make a code search dataset from natural language question-answe pairs was described in (Li et al., 2019). The resulting dataset NeuralCodeSearch contains about 4 million code snippets scraped from GitHub and more importantly an evaluation part that curated from dump of StackOverflow questions. This part contains only 287 question-snipped pairs. The next approach is described in work (Husain et al., 2019b). This dataset NodeSearchNet is also built from GitHuub dump, but in this case, the authors split the method bodies into the code itself and a description. This dataset contains million code snippet & description pairs in 6 programming languages including Python. This dataset is partially used in the next work (Hasan et al., 2021). In this paper, the authors take several datasets CodeSearchNoe and others and created a bigger one. From CodeSearchNet they used the Java part and Python part which is translated automatically into Java. The resulting dataset contains 4 million code snippet-descriptio pairs. Interestingly. the biggest dataset was built in the earlier work (Gu et al., 2018). This dataset, CODEan -rrin, contains 18 million method definition & one sentence description pairs and consists of code written in Java. More recent work CodeXGLUE presented (Lu et al., 2021b). is a machine learning benchmark collection of datasets for code understanding and generation tasks, which includes a modification of mentioned CodeSearchNet. CodeXGLUE provides a benchmark for different code-to-code, code-to-text, text-to-code tasks, and code search one of them. This benchmark includes code 10 programming languages. Another multi-task dataset was presented in (Puri et al., 2021). It has 14 million code snippets in 5 programming languages.
For clone detection there are two main works POJ-104 (Mou et al., 2016) and BigCloneBench (Wang al., 2020). The first one represents comparatively small corpus of C++ problem solution from a student judging system. While the other is bigger, and includes a lot of mined data in several langauges.
Code Search Approaches There is line of early work code search (Bacchelli et al., 2010; Brandt et al., 2010; Campbell & Treude, 2017; Chan et al., 2012). These works mostly relied on classic information retrieval, which turned out be still strong baseline in our experiments. (Brandt et al., 2010; Barzilay et al., 2013; Ponzanelli et al., 2014) were relied on the existing industry scale web search engines.
In (Gu et al., 2018) modern dense vector representations were used for information retrieval. The authors use two recurrent neural networks to represent the code and textual descriptions. In the work (Feng et al., 2020a) the authors presented a language model-based approach to produce these representations. The authors of paper (Gotmare et al., 2021b) use three Transformer-based models, as the encoders and 1 as classifier, for hierarchical representation of code and text, although they propose to share the parameters in the encoders, it lowers the final quality of their model. Our model in contrast uses single encoder for embedding the queries and documents and skips the classifier part.
There is also a line of recent work, which address different code search aspects, not directly the code search itself in the formulation CodeSearchNet (Chen & Abedjan, 2021; Hammad et al., 2022; Gu et al., 2022; Di Grazia, 2022; Luong et al., 2022; Zhang et al., 2022).
Code LM After the success of BERT-like models for natural language, the attention of the commu- nity was pointed to programming languages. Thus recently there were presented several pre- pre-trained programming language models, namely CodeBERT (Feng et al., 2020a) which is bimodal pre- trained model for programming language and natural language, based on RoBERTa (?) transformer architecture, trained masked language modelling and replaced token detection objectives; Graph-
8

---

Under review as a conference paper at ICLR 2023
CodeBERT (Guo al., 2021) model uses data flow in the pre-training stage to solve MLM, edge prediction and node alignment tasks; SynCoBERT (Wang et al., 2021b) model uses multi-modal contrastive learning achieve better code representations. The model is pre-trained on identifier prediction and abstract syntax tree edges prediction tasks.
Natural Language Systems Recent research was focused on creating non-Enge Non-English question answering datasets and applying cross-lingual transfer learning techniques, from English to other languages. Until recently. the availability of appropriate train and test datasets has been a key factor in the development of the field: however, in recent years, many works have focused the collection of loosely aligned data obtained through automatic translation or by parsing similar multilingual sources. (Lee & Lee, 2019) have shown transfer learning applicability for cross-lingual QA with training on English data and evaluation on Chinese data. (Artetxe et al., 2020) studied wross-lingual transferability of monolingual representations of a transformer-bast based masked language model. (M ha'mami al., 2021) examined a cross -ooss-liggual optimization based meta-learning approach (meta- training from the source language to the target language(s) meta-adaptation on the same target language(s) for more Linguag-speccife adaptation), to learn to adapt to new languages for question answering. (Gao & Callan, 2021) proposed unsupervised pre-training for dense passage retrieval, although the authors concentrated on retrieval itself, ignoring cross-lingual nature of the data.
In most previous approaches the authors use extractive models to generate the actual answer. This could be explained by the mental inertia from SQuAD-like datasets. By SQuAD-like we mean dataset where labelled data includes an explicitly stated question, a passage, containing an answer, and span markup for the answer. Such markup presented for the question answering task called SQuAD in (Rajpurkar et al., 2016). Recently several works on cross-lingual generation of answers from raw texts has been presented. Kumar al. (2019); Chi al. (2019) studied lons-lin question generation. Riabi et al. (2020) also suggested a method to produce synthetic questions in cross-lingual way, using Multilingual MiniLM. Shakeri al. (2020) proposed method generate multilingual question and answer pairs by a generative model (namely, a fine-tuned multilingual T5 model), it is based on automatically translated samples from English the target domain.
Generative question answering was mostly considered in previous work for long answers datasets. However, FiD model (Izacard & Grave, 2021) archives competitive results on SQuAD-like datasets, where an answer is supposed to be short text span. For open domain question answering, one of the first approaches named RAG used generative models was presented in (Lewis al., 2021). key idea of this RAG model is to process several (top k) passages from the retriever in the encoder simultaneously. The produced dense representations of the passages are used in the decoder for the answer generation, this process is called fusion. Processing the passages independently in the encoder allows a model to scale to many contexts, as it only runs self-attention over one context at a time.
For question answering over knowledge graph, (Zhou et al., 2021) studied unsupervised bilingual lexicon induction for zero-shot cross-lingual transfer for multilingual question answering, in order to map training questions in the source language into those in the target language as augmented training data, which is important for zero-resource languages.
7 CONCLUSION
The understanding of semantic similarity is an important ability to solve many different tasks. We present new method to improve this ability and demonstrate its improvement on tasks of Clone Detection and Code Search. We also formulate a novel task of Cross-lingual Clone Detection, thus present XCD dataset and describe three different setups on it, namely clone detection, clone retrieval, and a hybrid one. Our model had outperformed strong baselines in all presented tasks, proving that our pre- pre-training method gives mode power for semantic similarity understanding to particular language model.
We hope that our method will be helpful in other programming language processing tasks, which we have not covered in our work. So we leave them as a future work. But in addition, we think that method could be useful for other researchers in this and other fields.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the lonssllnggul transferabilityy of monolin- gual representations. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 4623-4637, Online, July 2020. Association for Computational Linguistics.
Alberto Bacchelli, Michele Lanza, and Romain Robbes. Linking e-mails and source code artifacts. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volaee 1. pp. 375-384, 2010.
Ohad Barzilay, Christoph Treude, and Alexey Zagalsky. Facilitating crowd sourced software engi- neering via stack overflow. In Finding Source Code on the Web for Remix and Reuse, pp. 289-308. Springer, 2013.
Joel Brandt, Mira Dontcheva, Marcos Weskamp, and Scott R Klemmer. Example-centric program- ming: integrating web search into the development environment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 513-522, 2010.
Brock Angus Campbell and Christoph Treude. NIp2code: Code snippet content assist via natural language tasks. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 628-632 632. IEEE, 2017.
Wing-Kwan Kwan Chan, Hong Cheng, and David Lo. Searching connected api subgraph via text phrases. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, pp. 1-11, 2012.
Binger Chen and Ziawasch Abedjan. Interactive cross-language code retrieval with auto-encoders. In 2021 36th JEEF/ACM International Conference on Automated Softwre Engineering (ASE), pp. 167-178, 2021.
doi: 10.1 10.110///A5511522220..997777
Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xian-Ling Mao, and Heyan Huang. Cross-lingual natural language generation via pre-training, 2019.
Luca Di Grazia. Efficiently and precisely searching for code changes with diffsearch. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) pp. 313-315, 2022. doi: 10. 10.1109/ICSE 000p0//SSE--COmaaao5552222222222
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages, 2020a. URL https: / /aarivivvvv./////////aaa///aaavv org/ abs 2002 08155.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjum Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1536- 1547, Online, November 2020b. Association for Computational Linguistics. doi: 10.18653/ ht2p://.innn2222seee.... findings-emnlp.1399 URL findings emn 11.199.
org/2020.
Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense /2008. 05540.
passage retrieval. 2021. URL httph:ttp/// httpp::://axxi..rrr
Akhilesh Deepak Gotmare, Junnan Li, Shafiq Joty, and Steven C. H. Hoi. Cascaded fast and slow models for efficient semantic code search, 2021a. URL httptt/////// /ttp:s/////vvv./// org/abs /2100.
07811.
Akhilesh Deepak Gotmare, Junnan Li, Shafiq Joty, and Steven CH Hoi. Cascaded fast and slow models for efficient semantic code search. arXiv preprint arXiv:21 2021b.
Lian Gu, Zihui Wang, Jiaxin Liu, Yating Zhang, Dong Yang. and Wei Dong. Extension-comprest learning: A deep learning code search method that simulates reading habits. In 2022 26th International Conference on Engineering of Complex Computer Systems (ICECCS), pp. 195-200 2022. doi: 10. 2.11099CCCC0544200022220000
10

---

Under review as a conference paper at ICLR 2023
Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. Deep code search. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). pp. 933-944. IEEE, 2018
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. Graphcodebert: Pre-training code representations with data flow. 2020. doi: 1044550/ARXXV.200000022 URL https: // ///aaxxv.rrg/ab////22..
08366.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. GraphCodeBERT: Pre-training Code Representations with Data Flow. Technical Report arXiv:2009.08 arXiv, September 2021. URL /estt://arxiie org/ abs abs./00/.. 2009. 08366. arXiv:2009.08366 [cs] type: article.
Muhammad Hammad, Önder Babur, Hamid Abdul Basit, and Mark Van Den Brand. Clone-seeker: Effective code clone search using annotations. IEEE Access, 10:11696-1171,, 2022. doi: 10.1109/ ACCESS.2022.0 145686.
Masum Hasan, Tanveer Muttaqueen, Abdullah Al Ishtiaq, Kazi Sajeed Mehrab, Md Mahim Anjum Haque, Tahmid Hasan, Wasi Ahmad, Anindya lqbal, and Rifat Shahriyar. Codese: A large descrieennn parallel dataset. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 20.22100 210-218, 2021.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross- Pross-lingual generalization. arXiv, 2020. doi: 10.48550/ARXIV.20003. 11000. URL https: //tr///aavvvv//////////e///a.ai/ htta//axii....gaaaa// /2003. 11080.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search. arXiv preprint
2019a.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Code- SearchNet Challenge: Evaluating the State of Semantic Code Search. September 2019b. doi: 10.48550/arXiv. 1909 1999.099366 URL https: /ttas://arxeee arxiv. org/ abs 1909. 09436v3
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering. 2020.
Gautier Izacard and Édouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874-880, 2021.
Vishwajeet Kumar, Nitish Joshi, Arijit Mukherjee, Ganesh Ramakrishnan, and Preethi Jyothi. Cross- lingual training for automatic question generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4863-4872. Association for Computational Linguistics, July 2019.
Chia- Mhia-Hsuan Lee and Hung- Yi Lee. Cross-lingual transfer learning for question answering. arXiv preprint arXiv:1900 190.00044. 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intenseee
nlp tasks. 2021.
Hongyu Li, Seohyun Kim, and Satish Chandra. Neural code search evaluation dataset. arXiv preprint /rXiv:/908.098000 2019.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL https: //arxiv wwtp:///////////o//oo 11 692.

---

Under review as a conference paper at ICLR 2023
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021a.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021b.
Kien Luong, Ferdian Thung, and David Lo. Arsearch: Searching for api related resources from stack overflow and github. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE- CCSE-Companion), pp. 11-15, 2022. doi: 10.1109/
ICSE ICSE-Companinn Companion5522912222..777
Meryem M'hamdi, Doo Soon Kim, Franck Dernoncourt, Trung Bui, Xiang Ren, and Jonathan May. X- METRA-ADA: Cross- Progs-lingual meta- transfer learning adaptation natural language understanding and question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3617-3632. Online, 2021. Association for Computational Linguistics.
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree structures for programming language processing. 2016.
Luca Ponzanelli, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, and Michele Lanza. Mining stackoverflow turn the ide into self-confident programming prompter. In Proceedings of the 1 Ith working conference on mining software repositories, pp. 102-111, 2014.
Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang. Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. Codenet: A large-scale ai for code dataset for learning diversity of coding tasks. In Thirty-fisth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. ArXiv, abs/2010.0819) 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016.
Arij Riabi, Thomas Scialom, Rachel Keraron, Benoît Sagot, Djamé Seddah, and Jacopo Staiano. Synthetic data augmentation for zero-shot cross-lingual question answering. arXiv preprint arXiv:2010. /2643, 2020.
Siamak Shakeri, Noah Constant, Mihir Sanjay Kale, and Linting Xue. Multilingual synthetic question and answer generation for cross-lingual reading comprehension. arXiv preprint arXiv:2010.12008:1 2020.
Nikita Sorokin, Dmitry Abulkhanov, Irina Piontkovskaya, and Valentin Malykh. Ask me anything in your native language. Proceedings of NAACI. 2022, 2022.
Chih-chan Tien and Shane
Bilingual alignment transfers to multilingual alignment for unsupervised parallel text mining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8696-8706, Dublin, Ireland, May URL
2022. Association for Computational Linguisties. doi: 10. 12118665//002222acllol https: httt::://acclatthooooge org/2022........ooo www.::222.cccco.. acl-longe 595.
Johannes Villmow, Viola Campos, Adrian Ulges, and UIrich Schwanecke. Addressing leakage in self-supervised contextualized code retrieval, 2022. URL https: /ttttps///axiv....t wwtp://wwwwr/r.///////oooo.o.s 2 11594.
Wenhan Wang Ge Li, Bo Ma, Xin Xia, and Zhi Jin. Detecting code clones with graph neural network and flom-aumeettdd abstract syntax tree. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 261-271. IEFE, 2020.
12

---

Under review as a conference paper at ICLR 2023
Xin Wang, Yasheng Wang, Fei Mi. Pingyi Zhou, Yao Wan, Xiao Liu, Li Li, Hao Wu, Jin Liu, and Xin Jiang. Syncobert: Syntax-guided multi molti- contrastive pre-training for code representation, 2021a. URL htwpt:h/ttaaaat.m /ariiv../rriiv.e/aa org/ abs /210. 04556.
Xin Wang, Yasheng Wang, Fei Mi, Pingyi Zhou, Yao Wan, Xiao Liu, Li Li, Hao Wu, Jin Liu, and Xin Jiang. SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation, September 2021b. URL http: httt:://arxiyy wwwpaawwwww..../////////.// 04556. Number: arXiv:2108.04556 arXiv:2108.0445 0045666004566 [cs].
Ruochen Xu, Yiming Yang, Naoki Otani, and Yuexin Wu. Unsupervised cross-lingual transfer of word embedding spaces. pp. 2465-2474, October-November 2018, doi: 10.18653/v1/D) 1-66..
URL httph:://///// /aa//aaaaaa http://aaaaanttoooeo/
wrg/ooog///..o..o..o.... 1268.
Jiyang Zhang, Yu Liu, Milos Gligoric, Owolabi Legunsen, and August Shi. Comparing and combining analysis-based and learning-base regression test selection. In 2022 IEEE/ACM International Conference on Automation of Software Test (AST), pp. 17-28, 2022.
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnera- bility identification by learning comprehensive program semantics via graph neural networks. In Advances in Neural Information Processing Systems, pp. 10197-10207, 2019.
Yucheng Zhou, Xiubo Geng, Tao Shen, Wenqiang Zhang, and Daxin Jiang. Improving zero-shot cross- lingual transfer for multilingual question answering
knowledge graph. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5822-5834, Online, 2021. Association for Computational Linguistics.
13


Reviews:
[{"review_id": "tEZtu7I0tEK", "reviewer": "Reviewer_2qh1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach which incorporates contrastive learning into the training procedure to address the novel task of cross-lingual clone detection. For evaluation, authors construct a benchmark (XCD) by mining CodeForces. They show that their approach outperforms baselines on existing benchmarks for clone detection as well as on their new benchmark. On a sample of examples, they show that their model learns a latent embedding space in which cross-lingual examples implementing the same functionality are close together, in comparison to baselines which group examples of the same underlying programming language together.\n\nContributions:\n- Formulation of the task of cross-lingual clone detection\n- Training procedure which combines contrastive learning, standard LM loss, and error detection\n- XCD benchmark for code clone detection which spans 9 different languages\n", "summary_of_the_review": "Overall, there are a lot of missing details with respect to the techniques as well as the experimental settings, making it difficult to evaluate some aspects of this paper. While I appreciate the idea of explicitly studying cross-lingual clone detection, the general intuition has been previously studied before and the technique of using contrastive learning for clone detection has also been studied before, so I feel that the novelty of the work is somewhat limited.", "strength_and_weaknesses": "Strengths:\n- The idea of cross-lingual clone detection is nice, and it could be useful for acquiring parallel data for studying tasks like code translation.\n- The analysis that shows that the model learns to embed similar programs of different programming languages near one another while other baselines tend to group by programming language is nice, though it is not clear whether the sample was random and the sample size is quite small.\n- The ablation study in 5.1 is nice, as it breaks down the improvements achieved through the new data as well as the various loss components. This table shows that there is a clear advantage in using contrastive loss and combining it with the standard LM loss. Nonetheless, there are a few things that are clear about Table 5 (described below).\n\nWeaknesses:\n- There are many things that are unclear in the paper which make it difficult to understand some aspects of the technique as well as the experimental setup. Namely, the approach taken to compile XCD is not clear. It is written that there are three different setups: retrieval way, code clone detection way, and hybrid. There are not enough details given to understand these setups. Next, the size of the training set is not mentioned. Additionally, the zero-shot setting is not adequately described. The error detection loss is described as a binary classification task, but there are 8 classes described in 2.4.1, so it is not clear what the connection is. Table 5 suggests that there is some “verdict verification” component in the model, but this was never described in the paper. Also, in Table 5, it is not clear what “size limitation” is in reference to.\n- The baselines used in this work are not at all described. Furthermore, it seems like authors could have considered stronger, more recent multilingual models of code like CodeGen.\n- It seems that the authors could have also tested on some of the benchmarked code translation datasets, since they do provide pairs for cross-lingual clone detection.\n- An ablation without the hard negative mining (either with a less stringent criteria for mining hard negatives or random sampling) would have been nice to further dive into the contrastive learning aspect of this work.\n- Some analysis for in-domain vs out-of-domain cross-lingual clone detection would have made the paper stronger. For example, if you train on only languages A, B, and C, how well does the model generalize to detecting functionally equivalent programs in languages D and E.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VeuGR74Ldb", "reviewer": "Reviewer_Y4fF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets cross-lingual clone detection. It presents a contrastive learning-based technique to learn the embedding space of code snippets and to create alignments between code solving the same problems across different programming languages. The authors also conduct experiments on code search and code clone detection. The experimental results show the effectiveness of the proposed model.", "summary_of_the_review": "The experimental settings are weak: important baselines [1,2] are missing and the comparisons are unfair. Specifically, previous work [3] has found that the larger batch size can boost the performance of contrastive learning-based models. The proposed methods are contrastive learning-based and batch size is set to 500 in this paper.  However, the baselines are all smaller than 500. It is better to study the effect/contribution of large batch sizes. \n\n[1] Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J. (2022). UniXcoder: Unified Cross-Modal Pre-training for Code Representation. ACL 2022.\n\n[2] Zhang, J., Wang, X., et al., A novel neural source code representation based on abstract syntax tree. In IEEE/ACM 41st International Conference on Software Engineering (ICSE 2019), May 2019.\n\n[3] Shi, E., Gub, W., Wang, Y., et al., (2022). Enhancing Semantic Code Search with Multimodal Contrastive Learning and Soft Data Augmentation. arXiv preprint arXiv:2204.03293.\n\nThe analysis of the experimental results is not sufficient. This paper does not provide a convincing argument why the proposed methods is effective. In which cases can it perform better than current baselines and why? Specifically, in the section EXPERIMENTS, the authors only present the experimental results on the table and conclude that the proposed methods outperform all the baselines. However, first, it is better to explain what the baselines are. For example, what are the meaning of “graphcodebert-base” and “graphcodebert-base-POJ-104” in Table2? What is the difference between the “GraphCodeBERT” in table1 and “graphcodebert-base” in table 2. Second, the paper needs to provide some convincing arguments to justify why the proposed approach is effective.\n\nThe presentation of the paper is poor. Important details and references are missing or hard to understand. The figures/tables miss some explanations. In addition, there are some grammar errors in the paper: \n\n• In software development practice in it is sometimes important to identify the code with the same effective output. (Should delete “in”).\n\n• Ability of Code language models to transfer knowledge between programming language more obvious… (Case error, also the sentence is not clear).\n\nMissing references:\n\n• There are various approaches to solve clone detection task problem starting from algorithmic based methods (?) and continuing with modern machine learning ones (). \n\n• The authors observe that the Mean Reciprocal Rank (MRR) scores of RoBERTa (?) and CodeBERT (?) for the code search task...\n\nSome references are duplicated in the REFERENCE section.\n", "strength_and_weaknesses": "Strengths:\n\n(+) Code search and clone detection are important and useful for software development and maintenance.\n\n(+) The authors discuss data quality issues and evaluate the approaches on a good quality dataset.\n\nWeaknesses:\n\n(-) The significance of the work is weak. \n\n(-) Experiments are weak.\n\n(-) The analysis of the experimental results is not sufficient. \n\n(-) The presentation of the paper is poor. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MCOYGPP4XL", "reviewer": "Reviewer_Q8fk", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work constructs a multi-lingual dataset of solutions to programming problems in order to train a cross-lingual code clone detection model. This model is powered primarily by a contrastive loss that helps it learn lagnuage-agnostic (and instead problem-specific) representations of programs. The resulting model is evaluated on code clone, search & clone retrieval tasks where it generally achieves stronger performance than baselines.", "summary_of_the_review": "The contributions in this work are limited and primarily geared towards a different audience. The writing needs to be substantially improved. Many experimental details are unclear or problematic. As a whole, this work is not fit for publication in its current form.", "strength_and_weaknesses": "The work aims to show that cross-lingual training using a contrastive loss improves performance on several tasks that evaluate the semantics representing quality of code embeddings, and succeeds at doing so. However, the only novel aspect of this work is the multi-lingual dataset, which is a software engineering contribution (and also quite limited, considering just 100 distinct programming problems). The use of a contrastive loss is otherwise quite common place, even within \"AI for Code\" papers.\n\nThere are several other significant issues. For one, the heavy, almost exclusive use of GraphCodeBERT as a baseline is problematic because this model was not trained to perform well on this task. Its performance on cross-lingual code retrieval is effectively 0, which clearly highlights that stronger baselines should have been used. Many methods have been proposed for the tasks studied in this work, most of which could have been retrained or extended with the newly constructed cross-lingual dataset in order to provide much stronger baselines. On P8, the work mentions even mentions several candidate baseline methods for the code search task and states that these \"[...] turned out to be still a strong baseline in our experiment\", yet I could not find any of these in the results. This greatly undercuts the perceived usefulness of this technique.\n\nFurthermore, the work suffers from major quality and clarity issues (see below). As a consequence, many experimental conditions are under- or unexplained. To name a few examples (there are too many to enumerate): Tab. 5 discusses a \"size limitation\" constraint that was never introduced in the methodology. Section 2.1 mentions MRR@20 but never indicates what the 20 refers to. The description for Eq. 1 (Sec. 3) mentions both languages and users, but neither appears in the equation. The fact that the model aims to discriminate between users also seems quite surprising given the stated goal of the work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "KsEmegHsU8E", "reviewer": "Reviewer_Fdtp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies some tasks related to programming language (e.g. clone detection, code search). The paper makes two contributions in the domain: firstly, through online OJ code mining, the paper proposes a new benchmark XCD that evaluates model over multiple program languages. Secondly, the paper proposes a new pretraining approach called CCT which combines masked language modelling with contrastive loss as well as code specific bug loss. Empirical results show that the proposed model achieves good performance compared to previous approaches in various tasks including code retrieval and code cloning as well as in various settings including single programming language settings, multi-language settings and zero shot setting.", "summary_of_the_review": "The paper studies some tasks related to programming language (e.g. clone detection, code search). The paper makes two contributions in the domain: firstly, through online OJ code mining, the paper proposes a new benchmark XCD that evaluates model over multiple program languages. Secondly, the paper proposes a new pretraining approach called CCT which combines masked language modelling with contrastive loss as well as code specific bug loss. The paper has done experiments showing convincing results.\n\nOn the other side, the paper misses a strong motivation explaining why this line of work can be impactful and interesting. Although with clear contributions, the discussion of the paper (e.g. natural language system related works) is confusing; there is some serious formatting issues as well in the paper.", "strength_and_weaknesses": "Strength:\n- The paper extends the search/detection tasks on programming languages, notably from single language setting to a multi-lingual setting; proposing the XCD dataset in the paper.\n- For tasks related to programming language search/retrieval, the paper propose a new pretraining objective. It shows that the new pretrained model outperforms previous models in general with associated ablation studies.\n\nWeakness:\n- I miss the motivation to study the multi programming language settings. I encourage the authors to think through the impact that such settings might bring from research to application perspectives and explain to readers why studying these are important.\n- The presentation of the paper is not satisfactory: 1) in related works, the authors specify natural language systems in detail, I can guess why multilinguality and the development there connects to the work in this paper; but they are not explicit and confuse readers like me. 2) 3.1 mentions hard negative mining which is interesting, however, I fail to see its impact in the result section. 3) CCT is mentioned in the abstract without readers knowing what it means 4) in several places, the reference shows a ? mark simply. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject