GLUECode: A Benchmark for Source Code Machine Learning Models

1 INTRODUCTION . In recent years , there has been considerable interest in researching machine learning models on source code artifacts . Machine learning models have been used to address a variety of software engineering tasks , as the inherent rich structure of code has allowed machine learning researchers to explore new models and ideas . However , research has focused on single-purpose application models , targeting a single task each time while using varying source code representations and datasets . This impedes progress towards general-purpose machine learning models of code that can learn and reason across many tasks . In this work , we present GLUECode ( Global and Local Understanding Evaluation of Code ) , with the goal of measuring progress in source code modelling across a range of tasks that account for the diverse characteristics of software and require diverse reasoning capabilities over several thousands of software projects . As GLUE ( Wang et al. , 2018 ) and SuperGLUE ( Wang et al. , 2019 ) does for natural language , GLUECode highlights important aspects of reasoning about code : ( 1 ) since code in software is composed of multiple interacting entities , it includes tasks that leverage both local ( single method ) and global ( multiple inter-related methods , information beyond the local method ) reasoning to varying degrees . This is in contrast to most tasks and models that have been introduced so far that focus on local reasoning ; ( 2 ) since source code mixes structured and unstructured information , GLUECode tasks leverage both kinds of information , and ( 3 ) since the space of modelling choices is large , we provide several source code representations ranging from raw text to abstract syntax trees ( AST ) and graph representations , lowering the barrier to entry and ease of experimentation . The design space for source code models is extremely large and spans a wide range of source code representations . These range from the simplest ( software metrics and n-grams ) , to very complex that fully take advantage of the structure and semantics of source code ( such as graph-based representations ) . Even seemingly simple choices , such as how to preprocess identifiers , can be handled in many different ways and have disproportionate impact ( Karampatsis et al. , 2020 ) . GLUECode aims to provide a unified benchmark to explore this design space . We provide performance results on a set of baselines , ranging from simple neural architectures such as LSTMs and CNNs , to variants of pre-trained transformers . These models leverage purely local reasoning and limited amounts of structural information . We show that existing models perform well in a few tasks but fail to yield good results in others : In contrast to NLP , where ( pre-trained ) transformers outperform other models , we find that no single model of code consistently outperforms the others in all tasks . Finally , while models can be evaluated on any single task in the benchmark in isolation ( as the field is presently doing ) , a long-term goal of GLUECode is the creation of unified multi-task source code models that perform well across multiple tasks . A source code model that is jointly trained and can perform well on all the task in the benchmark would be a significant step towards more versatile models , that can , beyond the tasks they were trained , also adapt to downstream tasks , especially when there is not enough data . Given the performance of our baselines in the single-task scenario , defining a model that performs well across the board is thus very much an open problem . 2 THE GLUECODE BENCHMARK . Benchmarks are a common practice in machine learning and NLP , prominently featuring GLUE and SuperGLUE ( Wang et al. , 2018 ; 2019 ) among others . In the domain of machine learning on source code , several benchmarks have been proposed . However , in contrast to GLUECode , they consider relatively local contexts and do not incentivize non-local reasoning : Idbench looks at identifiers , ( Wainakh et al. , 2019 ) , BigCloneBench ( Svajlenko & Roy , 2015 ) and OJClone ( Mou et al. , 2016 ) at clone detection , and CodeSearchNet at a function-level text-to-code search ( Husain et al. , 2020 ) . Finally , COSET concerns classifying small programs by their functionality in 38 classes ( Wang & Christodorescu , 2019 ) , and CoNaLa is a line-level text-to-code generation benchmark ( Yin et al. , 2018 ) . In this section , we provide an overview of GLUECode . We first describe the software-specific characteristics that impact the choice of tasks , before detailing the dataset and the tasks involved . Details about other related benchmarks can be found in the Appendix D . 2.1 LOCAL VERSUS GLOBAL CONTEXT . Most existing machine learning models of source code work at the level of a single function or method . We call these local models , as they reason over the local context of a single software entity . This is in contrast to global models that reason over multiple software entities and scales . Global models are highly desirable since software systems are composed of multiple entities such as modules and functions , that communicate with each other . This composition of communicating entities dictates the behavior of a software system . For instance , a function may have a radically different behavior , depending on its arguments . Indeed , small local changes can manifest in large changes in behaviour in distant program locations . Only global models will be able to detect that . To push forward the state of the art , it is thus critical to focus on global models . Fully global models are currently out of reach but GLUECode incentivizes building models that feature some form of global reasoning , in addition to local reasoning . Existing work uses simplified projections of global representations : the GNN works of Allamanis et al . ( 2017 ; 2020 ) look solely at file-level tokens , syntax , data and control flow information . CocoGum ( Wang et al. , 2020 ) uses class context represented as abstracted UML diagrams . LambdaNet extracts type dependencies in JavaScript into a single graph ( Wei et al. , 2020 ) for a few mid-sized projects ( 500-10k lines of code ) , ignoring syntactic information , code comments , etc . Finally , Func2Vec ( DeFreez et al. , 2018 ) computes function embeddings over an interprocedural call graph , ignoring local syntax , function arguments , etc . An extended related work discussion can be found in Appendix D. Instead to reason over global contexts two limitations need to be overcome : First , time-consuming interprocedural static analyses need to be performed at scale . These require compiling projects and resolving all its dependencies . In GLUECode , we take a step towards this direction , by using the largest publicly available corpus of compilable Java code ( Sec . 2.3 ) . ( 2 ) Existing methods do not operate well on large and sparse inputs and thus representations are tailored to use only the necessary information . In GLUECode , we provide access to a variety of representations and propose a set of tasks that can not focus solely on local or global information ( Sec 2.2 ) . 2.2 FLEXIBILITY IN REPRESENTATIONS OF CODE . Representations of source code in machine learning are a central topic of research . Source code has a known rich structure , as it can be unambiguously parsed , while valuable information is present in identifiers , literals , and comments , which are unstructured . As a result , there has been sustained work in exploring architectures and representations that leverage the different structural aspects of software , ranging from treating software as a textual artifact , to tree and graph-based representations . These representations come with distinct trade-offs . Sequence-level models treating source code as text are simpler and easy to scale to large amounts of data , at the expense of obscuring the information obtained from distinct structural inter-relations in code . LSTM ( Zaremba & Sutskever , 2014 ) , CNN ( Allamanis et al. , 2016 ) and transformer ( Husain et al. , 2020 ; Kanade et al. , 2020 ; Feng et al. , 2020 ) based models for source code have been explored . Meanwhile , more structured models commonly learn from less data thanks to the provided structure , but are harder to scale as they require extensive pre-processing . Such models use a program ’ s abstract syntax tree ( AST ) in Tree-LSTMs ( Wei & Li , 2017 ) , tree-based CNNs ( Mou et al. , 2014 ) , or use linearized forms fed to sequence models ( LeClair et al. , 2019 ; Kim et al. , 2020 ) , or linearized as bags of AST paths ( Alon et al. , 2018c ; a ) . Graph representations have been used in conjunctions with GNNs ( Allamanis et al. , 2017 ; Brockschmidt et al. , 2018 ; Wei et al. , 2020 ) and have been recently combined with RNNs and ( relational ) transformers ( Hellendoorn et al. , 2019b ) . Yet , most of these works are evaluated on a single task , yielding limited insights on the tradeoffs of various representations and models . GLUECode ’ s goal is to ease experimentation across representation and modelling choices on a variety of local and global tasks . To achieve this , we provide several pre-processed representations at the level of source code files : raw text , tokenized code , abstract syntax trees , graph representations ( as in Allamanis et al . ( 2017 ) ) , and bags of AST paths as in Alon et al . ( 2018c ; a ) . For global context we provide project-level call graphs . Across all representations , source code entities ( methods and classes ) are identified via a Universally Unique Identifier ( UUID ) , and can be linked together . Appendix A provides details and examples . Modelling decisions have significant impact on the performance of models and many different representations are possible , especially when considering models that perform global reasoning . GLUECode tasks are defined as a mapping from the UUID of the entity of interest to its label . Researchers can build their own input representations based on how they want to solve GLUECode . This allows researchers to combine these preprocessed representations as they see fit . GLUECode provides an API to efficiently access these representations to define the model . We show examples of the representations in Appendix A . 2.3 DATA . Performing pre-processing at scale is very challenging and time consuming . To extract the representations and some of the labels for the tasks , we use a variety of tools . Some of these tools perform extensive static analyses , and for this reason they require code that is compilable . Automatically compiling large amounts of arbitrary code is surprisingly difficult , as some systems may have convoluted build processes , or depend on a large number of libraries that may need to be present at compile time . We restrict our scope to Java since it is a popular language , with a lot of mature projects , and extensive tool support . To ease this task , our starting point is the 50K-C dataset ( Martins et al. , 2018 ) , which is a set of 50,000 Java projects extracted from GitHub , that are compilable . Of the 50,000 projects in 50K-C , many are too small to represent realistic software projects , such as projects authored by students . This is why we restrict our scope to projects that have 50 or more Java files . This leaves us with 6,925 projects , of which we were able to compile ∼5,300 . These projects have a combined total of 371,492 class files , and 2,361,111 method declarations . Once the projects are compiled , we run additional tools to extract all the representations , and extract some of the labels that the tasks need . Note that the entire process took several months , which we thus spare other researchers—simply trying to compile ∼7,000 projects is a weeks-long endeavour . We provide additional data processing details in Appendix A .

Summary: The objective of this paper is to present a benchmark of code understanding tasks in the spirit of GLUE benchmarks in NLP. Towards this, it designs 5 Java language tasks: NPath complexity, operator prediction, method naming, completion of method calls, and null dereference prediction. An evaluation on some common neural architectures is performed.