SlowGuess/NAACL_2022_OCR huggingface dataset:
Code Similarity. This task is always categorized into two groups: code-to-code retrieval and code clone detection. We conduct experiments on the Python800 dataset (Puri et al., 2021), which is composed of 800 problems with each problem having 300 unique Python solution files. We remove those files not in UTF-8 encoding formats and randomly select 100 solutions for each problem. In code-to-code retrieval, the filtered dataset is split to 720/40/40 problems for training, validation, and testing. Given a program, this task aims to retrieve other programs that solve the same problem; we evaluate using Mean Average Precision (MAP). Regarding the task of code clone detection, we treat it as binary classification and evaluate it using the Accuracy score, following (Puri et al., 2021).
To train these two tasks, we set the learning rate as  $2e - 5$ , the batch size as 32, the epoch number as 20. In code-to-code retrieval, we set the maximum length of both query and code sequence as 256. In code clone detection, we set the maximum concatenation sequence length of the two code snippets to 512. We save the optimal checkpoint on the validation set, and test it on the testing set.
Models	AdvTest	CoNaLa	CoSQA	Average
RoBERTa	18.3	30.7	57.6	35.5
CodeBERT	27.2	38.9	64.2	43.4
GraphCodeBERT	35.2	47.3	68.2	50.2
PLBART	34.3	45.5	65.3	48.4
CodeT5	36.5	47.7	67.7	50.6
SynCoBERT	38.1	48.4	69.6	52.0
CODE-MVP	40.4	50.6	72.1	54.4
Code Defect Detection. This task aims to identify whether a given piece of code snippet is vulnerable or not, which is usually treated as a binary classification task. We evaluate all models on the GREAT dataset (Hellendoorn et al., 2020), which is originally built from the ETH Py150 dataset (Raychev et al., 2016). We evaluate the performance of code defect detection using the Accuracy score. We randomly select 100K samples for training, 5K samples for validation and 5K samples for testing, respectively. We set the learning rate as  $5e - 5$ , the batch size as 32, the maximum fine-tuning epoch as 50, the maximum length of both query and code sequence as 256. We save the optimal checkpoint on the validation set, and test it on the testing set.
We compare CODE-MVP with various state-of-the-art models. RoBERTa (Liu et al., 2019) is a robustly optimized BERT (Devlin et al., 2019), which is originally pre-trained on a large-scale natural-language corpus. We fine-tune it on source code datasets of downstream tasks. CodeBERT (Feng et al., 2020) is pre-trained on NL-PL pairs using both masked language modeling (Devlin et al., 2019) and replaced token detection (Clark et al., 2020) objectives. GraphCodeBERT (Guo et al., 2021) is a pre-trained language model of source code which incorporates the data flow information of source code. PLBART (Ahmad et al., 2021) is based on the BART (Lewis et al., 2020) architecture and pre-trained on Python and Java functions using denoising autoencoding. CodeT5 (Wang et al., 2021b) is based on the T5 (Raffel et al., 2020) architecture and employs denoising sequence-to-sequence pre-training on seven programming languages. SynCoBERT (Wang et al., 2021) incorporates AST by edge prediction and uses contrastive learning to maximize the mutual information among programs, documents, and ASTs.
Table 3: Results on the natural language code retrieval task evaluating with MRR, using the AdvTest, CoNaLa, and CoSQA datasets.
Models	MAP@R	Accuracy
RoBERTa	82.9	94.4
CodeBERT	86.1	95.2
GraphCodeBERT	88.8	95.9
PLBART	86.7	95.5
CodeT5	88.1	95.7
SynCoBERT	89.2	96.1
CODE-MVP	91.5	97.4
Table 4: Results on the code-to-code retrieval and code clone detection tasks evaluating with MAP and Accuracy score, using the Python800 dataset.
5.1 Performance on Downstream Tasks (RQ1)
Natural Language Code Retrieval. Table 3 shows the results of natural language code retrieval on three datasets. We can observe that CODE-MVP outperforms all baseline models on all datasets. Specifically, it outperforms CodeT5 by 3.8 points on average. Compared to the previous state-of-the-art SynCoBERT, CODE-MVP also performs better with an average improvement of 2.4 points. This significant performance improvement indicates that the code representation learned by CODE-MVP preserves more code semantics. We attribute this improvement to our introduced multi-view contrastive pre-training strategy.
Code Similarity. Table 4 presents the results for code similarity calculation, including code-to-code retrieval and code clone detection. We can see that CODE-MVP significantly outperforms all baseline models on these two tasks. In the task of code-to-code retrieval, CODE-MVP outperforms CodeT5 and SynCoBERT by 3.4 points and 2.3 points, respectively. In the task of code clone detection, CODE-MVP achieves 1.5 and 1.3 points higher compared to GraphCodeBERT and SynCoBERT, respectively. These results show that CODE-MVP can better identify those programs with the same semantics and distinguish those programs with different semantics.
Code Defect Detection. Table 5 shows the experimental results of code defect detection. CODE-MVP consistently outperforms all models. Specifically, it outperforms GraphCodeBERT and SynCoBERT by 1.8 and 1.1 points, respectively. These results indicate that CODE-MVP can effectively preserve the semantics of programs, which is beneficial for code defect detection.
Models	Accuracy
RoBERTa	81.9
CodeBERT	85.5
GraphCodeBERT	87.5
PLBART	86.8
CodeT5	87.4
SynCoBERT	88.2
CODE-MVP	89.3
Table 5: Results on the code defect detection task evaluating with Accuracy score, using the GREAT dataset.
Models	AdvTest	CoNaLa	CoSQA	Average
CODE-MVP	40.4	50.6	72.1	54.4
w/o MVCL	36.2	47.7	69.2	51.0
w/o FGTI	38.0	48.9	70.8	52.6
w/o AST	39.1	48.5	71.3	53.0
w/o PT	38.2	48.6	70.8	52.5
w/o CFG	37.8	47.9	70.5	52.1
Table 6: Ablation study on the task of natural language code retrieval, evaluated using MRR.
5.2 Ablation Study (RQ2)
We empirically study several simplified variants of CODE-MVP to understand the contributions of each component, including the Multi-View Contrastive Learning (MVCL), Fine-Grained Type Inference (FGTI), Abstract Syntax Tree (AST), Program Transformation (PT), and Control Flow Graph (CFG). Taking the natural language code retrieval task as an example, Table 6 shows the experimental results of each variant on that task. The setting of w/o (MVCL, FGTI) indicates that these pre-training objectives are removed from CODE-MVP respectively. The setting of w/o (AST, PT, CFG) indicates that different views of programs are removed from CODE-MVP respectively. From Table 6, several meaningful observations can be drawn. (1) Both MVCL and FGTI effectively increase the performance, which confirms that the two proposed pre-training objectives can indeed improve the ability of the model for program comprehension. (2) Exploiting different views of programs can bring performance improvements to the model as arbitrarily discarding any view of programs degrades the performance. Additionally, the introduction of CFG brings more performance improvements, indicating the importance of execution information for program understanding.
Pre-Trained Models for Source Code. Benefiting from the strong power of pre-trained models
in natural language processing (Liu et al., 2019; Devlin et al., 2019; Wang et al., 2021a, 2020a,b), several recent works attempt to use the pre-training techniques on programs (Svyatkovskiy et al., 2020). Kanade et al. (2020) proposed CuBERT which follows the architecture of BERT (Devlin et al., 2019), and is pre-trained with a masked language modeling objective on a large-scale Python corpus. Feng et al. (2020) proposed CodeBERT, which is pretrained on NL-PL pairs in six programming languages, introducing the replaced token detection objective (Clark et al., 2020). Furthermore, Guo et al. (2021) proposed GraphCodeBERT, which incorporates the data flow of programs into the model pre-training process. Wang et al. (2021) proposed SynCoBERT, which incorporates ASTs via edge prediction to enhance the structural information of programs. They also used contrastive learning to maximize the mutual information among programs, documents, and ASTs. Lu et al. (2021) proposed CodeGPT for code completion, which is pre-trained using a unidirectional language modeling objective. Ahmad et al. (2021) proposed PLBART based on BART (Lewis et al., 2020), which is pre-trained on a large-scale corpus of Java and Python programs paired with their corresponding comments via denoising autoencoding. Wang et al. (2021b) proposed CodeT5 following the architecture of T5 (Raffel et al., 2020). It employs denoising sequence-to-sequence pre-training on seven programming languages. Recently, Wan et al. (2022b) conducted a thorough structural analysis aiming to provide an interpretation of pre-trained language models for source code (e.g., CodeBERT and GraphCodeBERT).
Program Analysis for Code Intelligence. In addition to the lexical information of programs, many recent works attempt to leverage program analysis techniques to capture the structural and syntactic representations of programs (Cummins et al., 2020). Kim et al. (2021) designed several strategies to feed the ASTs of programs into Transformer (Vaswani et al., 2017). Li et al. (2019) proposed a graph matching network, which utilizes the CFG of the program to deal with the challenge of binary function similarity search. Ling et al. (2021) proposed a deep graph matching and searching model based on graph neural networks (Kipf and Welling, 2017; Wang et al., 2021b,a; Yu et al., 2022; Zhao et al., 2022) for code retrieval. They represented both natural language queries and code snippets based on
the unified graph-structured data. Iyer et al. (2020) presented the program-derived semantic graph to capture the semantics of programs at multiple levels of abstraction. Ben-Nun et al. (2018) presented inst2vec, which locally embeds individual statement in LLVM intermediate representations by processing a contextual flow graph with a context prediction objective (Mikolov et al., 2013).
Contrastive Learning on Programs. Recently, several attempts have been made to leverage contrastive learning for better code semantics. ContraCode (Jain et al., 2021) and Corder (Bui et al., 2021b) first utilized semantic-preserving program transformations such as identifier renaming, dead code insertion, to build positive instances. Then a contrastive learning objective is designed to maximize the mutual information among the positive and negative instances. Ding et al. (2021) presented a self-supervised pre-training technique called BOOST based on contrastive learning. They inject real-world bugs to build hard negative pairs. In CODE-MVP, we construct the positive pairs throughout the compilation process of programs, including lexical analysis, syntax analysis, semantic analysis, and static analysis. It is the first pretrained model that integrates multi-views of programs for program comprehension.
In this paper, we have proposed CODE-MVP, a novel approach to represent the source code with multi-view contrastive pre-training learning. We extract multiple code views with compiler tools and learn the complement among them under a contrastive learning framework. We also propose a fine-grained type inference task in the pre-training process. Comprehensive experiments on three downstream tasks over five datasets verify the effectiveness of CODE-MVP when compared with several state-of-the-art baselines.
We would like to thank Gerasimos Lampouras and Ignacio Iacobacci from Huawei London Research Institute for their constructive comments on this paper. Jin Liu is supported by National Natural Science Foundation of China under Grant No. 61972290. Yao Wan is partially supported by National Natural Science Foundation of China under Grant No. 62102157. Hao Wu is supported by National Natural Science Foundation of
China under Grant No. 61962061, and partially supported by Yunnan Provincial Foundation for Leaders of Disciplines in Science and Technology (202005AC160005).
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2655-2668. Association for Computational Linguistics.
Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, and Charles Sutton. 2018. A survey of machine learning for big code and naturalness. ACM Comput. Surv., 51(4):81:1-81:37.
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019. code2vec: learning distributed representations of code. Proc. ACM Program. Lang., 3(POPL):40:1-40:29.
Jong-hoon (David) An, Avik Chaudhuri, Jeffrey S. Foster, and Michael Hicks. 2011. Dynamic inference of static types for ruby. In Proceedings of the 38th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2011, Austin, TX, USA, January 26-28, 2011, pages 459-472. ACM.
Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018. Neural code comprehension: A learnable representation of code semantics. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada, pages 3589-3601.
Nghi D. Q. Bui, Yijun Yu, and Lingxiao Jiang. 2021a. Infercode: Self-supervised learning of code representations by predicting subtrees. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pages 1186-1197. IEEE.
Nghi D. Q. Bui, Yijun Yu, and Lingxiao Jiang. 2021b. Self-supervised contrastive learning for code retrieval and summarization via semantic-preserving transformations. In SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 511-521. ACM.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pretraining text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Chris Cummins, Zacharias V. Fisches, Tal Ben-Nun, Torsten Hoefler, and Hugh Leather. 2020. Programl: Graph-based deep learning for program optimization and analysis. CoRR, abs/2003.10536.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Yangruibo Ding, Luca Buratti, Saurabh Pajar, Alessandro Morari, Baishakhi Ray, and Saikat Chakraborty. 2021. Contrastive learning for source code with structural and functional properties. CoRR, abs/2110.03868.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1536-1547. Association for Computational Linguistics.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. Graphcodebert: Pre-training code representations with data flow. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. 2020. Global relational models of source code. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. 2021. Cosqa: 20, 000+ web queries for code search and question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5690-5700. Association for Computational Linguistics.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Code-searchnet challenge: Evaluating the state of semantic code search. CoRR, abs/1909.09436.
Roshni G. Iyer, Yizhou Sun, Wei Wang, and Justin Gottschlich. 2020. Software language comprehension using a program-derived semantic graph. CoRR, abs/2004.00768.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph Gonzalez, and Ion Stoica. 2021. Contrastive code representation learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 5954-5971. Association for Computational Linguistics.
Faizan Javed, Barrett R. Bryant, Matej Crepinsek, Marjan Mernik, and Alan P. Sprague. 2004. Context-free grammar induction using genetic programming. In Proceedings of the 42nd Annual Southeast Regional Conference, 2004, Huntsville, Alabama, USA, April 2-3, 2004, pages 404-405. ACM.
Xue Jiang, Zhuoran Zheng, Chen Lyu, Liang Li, and Lei Lyu. 2021. Treebert: A tree-based pre-trained model for programming language. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, UAI 2021, Virtual Event, 27-30 July 2021, volume 161 of Proceedings of Machine Learning Research, pages 54-63. AUAI Press.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5110-5121. PMLR.
Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. 2021. Code prediction by feeding trees to transformers. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pages 150-162. IEEE.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880. Association for Computational Linguistics.
Li Li, Jiawei Wang, and Haowei Quan. 2022. Scalpel: The python static analysis framework. CoRR, abs/2202.11840.
Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. 2019. Graph matching networks for learning the similarity of graph structured objects. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 3835-3845. PMLR.
Xiang Ling, Lingfei Wu, Saizhuo Wang, Gaoning Pan, Tengfei Ma, Fangli Xu, Alex X. Liu, Chunming Wu, and Shouling Ji. 2021. Deep graph matching and searching for semantic code retrieval. ACM Trans. Knowl. Discov. Data, 15(5):88:1-88:21.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. CoRR, abs/2102.04664.
Charith Mendis, Alex Renda, Saman P. Amarasinghe, and Michael Carbin. 2019. Ithemal: Accurate, portable and fast basic block throughput estimation using deep neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 4505-4515. PMLR.
Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 3111-3119.
Safa Omri and Carsten Sinz. 2020. Deep learning for software defect prediction: A survey. In ICSE '20: