Evaluating Semantic Representations of Source Code

1 INTRODUCTION . Reasoning about source code based on learned representations has various applications , such as predicting method names ( Allamanis et al. , 2015 ) , detecting bugs ( Pradel & Sen , 2018 ) and vulnerabilities ( Harer et al. , 2018 ) , predicting types ( Malik et al. , 2019 ) , detecting similar code ( White et al. , 2016 ; Xu et al. , 2017 ) , inferring specifications ( DeFreez et al. , 2018 ) , code de-obfuscation ( Raychev et al. , 2015 ; Alon et al. , 2018a ) , and program repair ( Devlin et al. , 2017 ) . Many of these techniques are based on embeddings of source code , which map a given piece of code into a continuous vector representation that encodes some aspect of the semantics of the code . A core component of most code embeddings are semantic representations of identifier names , i.e. , the names of variables , functions , classes , fields , etc . in source code . Similar to words in natural languages , identifiers are the basic building block of source code . Identifiers not only account for the majority of the vocabulary of source code , but they also convey important information about the ( intended ) meaning of code . To reason about identifiers and their meaning , code analysis techniques build on learned embeddings of identifiers , either by adapting embeddings that were originally proposed for natural languages ( Mikolov et al. , 2013a ; b ) or with embeddings specifically designed for source code ( Alon et al. , 2018a ) . Given the importance of identifier embeddings , a crucial challenge is measuring how effective an embedding represents the semantic relationships between identifiers . For word embeddings in natural language , the community has addressed this question through a series of gold standards ( Finkelstein et al. , 2002 ; Bruni et al. , 2014a ; Rubenstein & Goodenough , 1965 ; Miller & Charles , 1991 ; Hill et al. , 2015 ; Gerz et al. , 2016 ) . These gold standards define how similar two words are based on ratings by human judges , enabling an evaluation that measures how well an embedding reflects the human ratings . Unfortunately , simply reusing existing gold standards to identifiers in source code would be misleading . One reason is that the vocabularies of natural languages and source code overlap only partially , because source code contains various terms and abbreviations not found in natural language texts . Moreover , source code has a constantly growing vocabulary , as developers tend to invent new identifiers , e.g. , for newly emerging application domains Babii et al . ( 2019 ) . Finally , even words present in both natural languages and source code may differ in their meaning due to computer science-specific meanings of some words , e.g. , “ float ” or “ string ” . This paper addresses the problem of measuring and comparing the effectiveness of embeddings of identifiers . We present IdBench , a benchmark for evaluating techniques that represent semantic similarities of identifiers . The basis of the benchmark is a dataset of developer opinions about the similarity of pairs of identifiers . We gather this dataset through two surveys that show realworld identifiers and code snippets to hundreds of developers , asking them to rate their similarity . Taking the developer opinions as a gold standard , IdBench allows for evaluating embeddings in a systematic way by measuring to what extent an embedding agrees with ratings given by developers . Moreover , inspecting pairs of identifiers for which an embedding strongly agrees or disagrees with the benchmark helps understand the strengths and weaknesses of current embeddings . Overall , we gather thousands of ratings from 500 developers . Cleaning and compiling this raw dataset into a benchmark yields several hundreds of pairs of identifiers with gold standard similarities , including identifiers from a wide range of application domains . We apply our approach to a corpus of JavaScript code , because several recent pieces of work on identifier names and code embeddings focus on this language ( Pradel & Sen , 2018 ; Alon et al. , 2018b ; a ; Malik et al. , 2019 ) . Applying our methodology to another language is straightforward . Based on the newly created benchmark , we evaluate and compare state-of-the-art embeddings of identifiers . We find that different embedding techniques differ heavily in terms of their ability to accurately represent identifier relatedness and similarity . The best available technique , the CBOW variant of FastText , accurately represents relatedness , but none of the available techniques accurately represents identifier similarities . One reason is that some embeddings are confused about identifiers with opposite meaning , e.g. , rows and cols , and about identifiers that belong to the same application domain but are not similar . Another reason is that some embeddings miss synonyms , e.g. , file and record . We also find that simple string distance functions , which measure the similarity of identifiers without any learning , are surprisingly effective , and even outperform some learned embeddings for the similarity task . In summary , this paper makes the following contributions . ( 1 ) Methodology : To the best of our knowledge , we are the first to systematically evaluate embeddings of identifiers . Our methodology is based on surveying developers and summarizing their opinions into gold standard similarities of pairs of identifiers . ( 2 ) Reusable benchmark : We make available a benchmark of hundreds of pairs of identifiers , providing a way to systematically evaluate existing and future embeddings.1 ( 3 ) Comparison of state-of-the-art embeddings : We evaluate seven existing embeddings and string similarity functions , and discuss their strengths and weaknesses . 2 BENCHMARK DESIGN . We design IdBench so that it distinguishes two kinds of semantic relationships between identifiers ( Miller & Charles , 1991 ; Hill et al. , 2015 ; Faruqui et al. , 2016 ) . On the one hand , relatedness refers to the degree of association between two identifiers and covers various possible relations between them . For example , top and bottom are related because they are opposites , click and dblclick are related because they belong to the same general concept , and getBorderWidth and getPadding are related because they belong to the same application domain . On the other hand , similarity refers to the degree to which two identifiers have the same meaning , in the sense that one could substitute the other ( Miller & Charles , 1991 ) without changing the overall meaning . For example , length and size , as well as username and userid , are similar to each other . Similarity is a stronger semantic relationship than relatedness , because the former implies the latter , but not vice-versa . For example , the identifiers start and end are related , as they are opposites , 1https : //my.pcloud.com/publink/show ? code=kZR2LzkZAViNngFuJ3ykFqNzbQF1TJRedJs7 Identifiers : radians , angle 1 ) How related are the identifiers ? . 2 ) Could one substitute the other ? . Which identifier fits best into the blanks ? but they are not similar , because replacing one with the other would change the meaning of the program . IdBench includes three benchmark tasks : A relatedness task and two task to measure how well an embedding reflects the similarity of identifiers : a similarity task and a contextual similarity task . The following describes how we gather developer opinions that provide data for these tasks . 2.1 DEVELOPER SURVEYS . Direct Survey of Developer Opinions This survey shows two identifiers to a developer and then directly asks how related and how similar the identifiers are . Figure 1a shows an example question from the survey . The developer is shown pairs of identifiers and is then asked to rate on a five-point Likert scale how related and how similar these identifiers are to each other . In total , each developer is shown 18 pairs of identifiers , which we randomly sample from a larger pool of pairs . Before showing the questions , we provide a brief description of what the developers are supposed to do , including an explanation of the terms “ related ” and “ substitutable ” . The ratings gathered in the direct survey are the basis for the relatedness task and the similarity task of IdBench . Indirect Survey of Developer Opinions This survey asks developers to pick an identifier that best fits a given code context , which indirectly asks about the similarity of identifiers The motivation is that identifier names alone may not provide enough information to fully judge how similar they are ( Miller & Charles , 1991 ) . For example , without any context , identifiers idx and hl may cause confusion for developers who are trying to judge their similarity . The survey addresses this challenge by showing the code context in which an identifier occurs , and by asking the developers to decide which of two given identifiers best fits this context . If , for a specific pair of identifiers , the developers choose both identifiers equally often , then the identifiers are likely to be similar to each other , since one can substitute the other . Figure 1b shows a question asked during the indirect survey . As shown in the example , for code contexts where the identifier occurs multiple times , we show multiple blanks that all refer to the same identifier . In total , we show 15 such questions to each participant of the survey . The ratings gathered in the indirect survey are the basis for the contextual similarity tasks of IdBench . Selection of Identifiers and Code Examples We select identifiers and code contexts from a corpus of 50,000 JavaScript files ( Raychev et al. , 2016 ) . We select 300 pairs , made out of 488 identifiers , through a combination of automated and manual selection , aimed at a diverse set that covers different degrees of similarity and relatedness . The selection is guided by similarities between identifiers as judged by word embeddings Bruni et al . ( 2014b ) . The first step is to extract from the code corpus all identifier names that appear more than 100 times , which results in about 17,000 identifiers , including method names , variable names , property names , other types of identifiers . Next , we compute word embeddings of the extracted identifiers , compute the cosine distance of all pairs of identifiers , and then sort the pairs according to their cosine distances . We use two word embeddings Mikolov et al . ( 2013a ) ; Alon et al . ( 2018a ) , giving two sorted lists of pairs , and sample 150 pairs from each list as follows : ( i ) Pick 75 of the first 1,000 pairs , i.e. , the most similar pairs according to the respective embedding , where 38 pairs are randomly sampled and 37 pairs are manually selected . We manually select some pairs because otherwise , we had observed a lack of pairs of synonymous identifiers . ( ii ) Randomly sample 50 from pairs 1,001 to 10,000 , i.e. , pairs that still have a relatively high similarity . ( iii ) Randomly sample 25 pairs from the remaining pairs , i.e. , pairs that are mostly dissimilar to each other . To gather the code contexts for the indirect survey , we search the code corpus for occurrences of the selected identifiers . As the size of the context , we choose five lines , which provides sufficient context to choose the best identifier without overwhelming developers with large amounts of code . For each identifier , we randomly select five different contexts . When showing a specific pair of identifiers to a developer , we randomly select one of the gathered contexts for one of the two identifiers . Participants We pay developers via Amazon Mechanical Turk to perform the surveys . Participants take around 15 minutes to complete both surveys . In total , 500 developers participate in the survey , which yields at least 10 ratings for each pair of identifiers .

Summary: This paper proposed a benchmark dataset for evaluating different embedding methods for identifiers (like variables, functions) in programs. The groudtruth evaluation (similar/related) are labeled via Amazon MTurk. Experiments are carried out on several word embedding methods, and it seems these methods didn’t get good enough correlation with human scores.