A New Dataset for Source Code Comment Coherence
Authors: Anna Corazza, Valerio Maggio, Giuseppe Scanniello (Università di Napoli Federico II, Fondazione Bruno Kessler, Università della Basilicata, 2016)
Topic: Lexical and Semantic Resources and Analysis

Abstract
Source code comments provide useful insights on a codebase and on the intent behind design decisions and goals. Often, the information provided in the comment of a method and in its corresponding implementation may be not coherent with each other (i.e., the comment does not properly describe the implementation). Several could be the motivations for this issue (e.g., comment and source code do not evolve coherently). In this paper, we present the results of a manual assessment on the coherence between comments and implementations of $3,636$ methods, gathered from $4$ Java open-source software. The results of this assessment has been collected in a dataset that we made publicly available on the web. We also sketch here the protocol to create this dataset.

Introduction
Natural language is used in different ways in the development process of a software system and therefore techniques of natural language processing (NLP) and information retrieval (IR) are more and more frequently integrated into software development and maintenance tools. Many automated or semi-automated techniques have been proposed to aid developers in the comprehension and in the evolution of existing systems, e.g., AUTHOR. Natural language information is provided in source code comments and in the name of identifiers. In the former case, standard natural language is usually adopted, although quite technical. Comments are written in English, even when developers have different mother-tongues. On the other hand, identifiers are typically constructed by composing multiple terms and abbreviations. Therefore, more sophisticated techniques are necessary to extract the lexical information contained in each identifier AUTHOR. Most of these techniques assume that the same words are used whenever referring to a particular concept AUTHOR. In many cases, this represents an oversimplification: methods are often modified without updating the corresponding comments AUTHOR. In these cases, comments might convey information unrelated or inconsistent with the corresponding implementation. Nevertheless, comments are extremely important because they are expected to convey the main intent behind design decisions, along with some implementation details (e.g., types of parameters and of returned values). Therefore, more sophisticated models are necessary to determine if there is coherence between the lexicon provided in comments and in its corresponding source code. Hence, there exists coherence between a lead comment of a method and its source code (also simply coherence, from here on) if that comment describes the intent of the method and its actual implementation. In this work, we focus on the lead comment of methods. This kind of comments precedes the definition of a given method and is supposed to provide its documentation and details about the implementation. We discuss here a dataset we made publicly available on the web. It contains annotations about the coherence of 3,636 methods collected from 4 implementations of 3 open source projects written in Java. The defined protocol used for its creation is also sketched, to give researchers the opportunity to possibly extend it. Further details on this protocol can be found in AUTHOR. For the assessment of the quality of the annotation with special focus on computational linguistics applications, a few indexes have been considered AUTHOR, among which the kappa index AUTHOR is the most widely adopted because of its favorable characteristics. The inter-annotator agreement has therefore been assessed by this parameter. We expect that making freely available this dataset could give impulse to the research for approaches to assess the coherence between the implementation of a method and its lead comment (simply coherence, from here on). In fact, although no approach has been yet proposed in this regard, they could be of great help for software maintenance and evolution activities. The paper is structured as follows. In Section approach, we discuss the methodology used to create our dataset. A description of the main characteristics of the dataset is given in Section~sec:dataset, while in Section~sec:results the annotation is assessed. Some final considerations conclude the paper.

Conclusion
In this paper, we have presented the early steps of our research on the coherence between the lead comment of methods and their implementations. In particular, we have provided a description of the problem settings, along with the experimental protocol defined to create our dataset. We made it publicly available on the web. We also sketched the results of quantitative analysis conducted on a codebase of 3,636 methods, gathered from 4 different open-source systems written in Java. There could be many possible future directions for our research. For example, it would be interesting to conduct an empirical study to investigate the effect of maintenance operations on the coherence for multiple versions of the same system. As a step forward in this future research direction, we already included in the dataset two versions of the JFreeChart system. Our results and those by Fluri et al. AUTHOR represent a viable starting point. Finally, we would like to exploit the collected data as an evaluation set to assess the performance of approaches able to discern method coherence.