SlowGuess/ACL_2020_OCR huggingface dataset:
Sheena Panthaplackel<sup>1</sup>, Pengyu Nie<sup>2</sup>, Milos Gligoric<sup>2</sup>, Junyi Jessy Li<sup>3</sup>, Raymond J. Mooney<sup>1</sup>
$^{2}$ Department of Electrical and Computer Engineering
$^{3}$ Department of Linguistics
spantha@cs.utexas.edu, pynie@utexas.edu, gligoric@utexas.edu,
jessy@austin.utexas.edu, mooney@cs.utexas.edu
We formulate the novel task of automatically updating an existing natural language comment based on changes in the body of code it accompanies. We propose an approach that learns to correlate changes across two distinct language representations, to generate a sequence of edits that are applied to the existing comment to reflect the source code modifications. We train and evaluate our model using a dataset that we collected from commit histories of open-source software projects, with each example consisting of a concurrent update to a method and its corresponding comment. We compare our approach against multiple baselines using both automatic metrics and human evaluation. Results reflect the challenge of this task and that our model outperforms baselines with respect to making edits.
Software developers include natural language comments alongside source code as a way to document various aspects of the code such as functionality, use cases, pre-conditions, and post-conditions. With the growing popularity of open-source software that is widely used and jointly developed, the need for efficient communication among developers about code details has increased. Consequently, comments have assumed a vital role in the development cycle. With developers regularly refactoring and iteratively incorporating new functionality, source code is constantly evolving; however, the accompanying comments are not always updated to reflect the code changes (Tan et al., 2007; Ratol and Robillard, 2017). Inconsistency between code and comments can not only lead time-wasting confusion in tight project schedules (Hu et al., 2018) but can also result in bugs (Tan et al., 2007). To address this problem, we propose an approach that can automatically suggest comment updates when the associated methods are changed.
Figure 1: Changes in the getRotX method and its corresponding @return comment between two subsequent commits of the rajawali-rajawali project, available on GitHub.
Prior work explored rule-based approaches for detecting inconsistencies for a limited set of cases; however, they do not present ways to automatically fix these inconsistencies (Tan et al., 2007; Ratol and Robillard, 2017). Recent work in automatic comment generation aims to generate a comment given a code representation (Liang and Zhu, 2018; Hu et al., 2018; Fernandes et al., 2019); although these techniques could be used to produce a completely new comment that corresponds to the most recent version of the code, this could potentially discard salient content from the existing comment that should be retained. To the best of our knowledge, we are the first to formulate the task of automatically updating an existing comment when the corresponding body of code is modified.
This task is intended to align with how developers edit a comment when they introduce changes in the corresponding method. Rather than deleting it and starting from scratch, they would likely only modify the specific parts relevant to the code updates. For example, Figure 1 shows the getRotX method being modified to have the return value parsed into degrees. Within the same commit, the corresponding comment is revised to indicate this, without imposing changes on parts of the comment that pertain to other aspects of the return value. We replicate this process through a novel approach which is designed to correlate edits across two distinct language representations: source code and natural language comments. Namely, our model
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1853-1868 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
is trained to generate a sequence of edit actions, which are to be applied to the existing comment, by conditioning on learned representations of the code edits and existing comment. We additionally incorporate linguistic and lexical features to guide the model in determining where edits should be made in the existing comment. Furthermore, we develop an output reranking scheme that aims to produce edited comments that are fluent, preserve content that should not be changed, and maintain stylistic properties of the existing comment.
We train and evaluate our system on a corpus constructed from open-source Java projects on GitHub, by mining their commit histories and extracting examples from consecutive commits in which there was a change to both the code within a method as well as the corresponding Javadoc comment, specifically, the @return Javadoc tag. These comments, which have been previously studied for learning associations between comment and code entities (Panthaplackel et al., 2020), follow a well-defined structure and describe characteristics of the output of a method. For this reason, as an initial step, we focus on @return comments in this work. Our evaluation consists of several automatic metrics that are used to evaluate language generation tasks as well as tasks that relate to editing natural language text. We also conduct human evaluation, and assess whether human judgments correlate with the automatic metrics.
The main contributions of this work include (1) the task of automatically updating an existing comment based on source code changes and (2) a novel approach for learning to relate edits between source code and natural language that outperforms multiple baselines on several automatic metrics and human evaluation. Our implementation and data are publicly available. $^{1}$
Given a method, its corresponding comment, and an updated version of the method, the task is to update the comment so that it is consistent with the code in the new method. For the example in Figure 1, we want to generate "@return double the roll euler angle in degrees." based on the changes between the two versions of the method and the existing comment "@return double the roll euler angle." Concretely, given  $(\mathbf{M}_{\mathrm{old}},\mathbf{C}_{\mathrm{old}})$  and  $\mathbf{M}_{\mathrm{new}}$
Figure 2: High-level overview of our system.
where  $\mathbf{M}_{\mathrm{old}}$  and  $\mathbf{M}_{\mathrm{new}}$  denote the old and new versions of the method, and  $\mathbf{C}_{\mathrm{old}}$  signifies the previous version of the comment, the task is to produce  $\mathbf{C}_{\mathrm{new}}$ , the updated version of the comment.
3 Edit Model Overview
We design a system that examines source code changes and how they relate to the existing comment in order to produce an updated comment that reflects the code modifications. Since  $C_{old}$  and  $C_{new}$  are closely related, training a model to directly generate  $C_{new}$  risks having it learn to just copy  $C_{old}$ . To explicitly inform the model of edits, we define the target output as a sequence of edit actions,  $C_{edit}$ , to indicate how the existing comment should be revised (e.g., for  $C_{old} = ABC$ ,  $C_{edit} = <Delete>A <DeleteEnd>$  implies that A should be deleted to produce  $C_{new} = BC$ ). Furthermore, in order to better correlate these edits with changes in the code, we unify  $M_{old}$  and  $M_{new}$  into a single diff sequence that explicitly identifies code edits,  $M_{edit}$ . We discuss in more detail how  $M_{edit}$  and the training  $C_{edit}$  are constructed in §4.
Figure 2 shows a high-level overview of our system. We design an encoder-decoder architecture consisting of three components: a two-layer, bidirectional GRU (Cho et al., 2014) that encodes the code changes  $(\mathbf{M}_{\mathrm{edit}})$ , another two-layer, bidirectional GRU that encodes the existing comment  $(C_{\mathrm{old}})$ , and a GRU that is trained to decode a sequence of edit actions  $(C_{\mathrm{edit}})$ . We concatenate the
2We refrain from using the self-attention model (Vaswani et al., 2017) because prior work (Fernandes et al., 2019) suggests that it yields lower performance for comment generation.
$^{1}$ https://github.com/panthap2/LearningToUpdateNLComments
final states of the two encoders to form a vector that summarizes the content in  $\mathbf{M}_{\mathrm{edit}}$  and  $\mathbf{C}_{\mathrm{old}}$ , and use this vector as the initial state of the decoder. The decoder essentially has three subtasks: (1) identify edit locations in  $\mathbf{C}_{\mathrm{old}}$ ; (2) determine parts of  $\mathbf{M}_{\mathrm{edit}}$  that pertain to making these edits; and (3) apply updates in the given locations based on the relevant code changes. We rely on an attention mechanism (Luong et al., 2015) over the hidden states of the two encoders to accomplish the first two goals. At every decoding step, rather than aligning the current decoder state with all the encoder hidden states jointly, we align it with the hidden states of the two encoders separately. We concatenate the two resulting context vectors to form a unified context vector that is used in the final step of computing attention, ensuring that we incorporate pertinent content from both input sequences. Consequently, the resulting attention vector carries information relating to the current decoder state as well as knowledge aggregated from relevant portions of  $\mathbf{C}_{\mathrm{old}}$  and  $\mathbf{M}_{\mathrm{edit}}$ .
Using this information, the decoder performs the third subtask, which requires reasoning across language representations. Specifically, it must determine how the source code changes that are relevant to the current decoding step should manifest as natural language updates to the relevant portions of  $C_{\mathrm{old}}$ . At each step, it decides whether it should begin a new edit action by generating an edit start keyword, continue the present action by generating a comment token, or terminate the present action by generating an end-edit keyword. Because actions relating to deletions will include tokens in  $C_{\mathrm{old}}$ , and actions relating to insertions are likely to include tokens in  $M_{\mathrm{edit}}$ , we equip the decoder with a pointer network (Vinyals et al., 2015) to accommodate copying tokens from  $C_{\mathrm{old}}$  and  $M_{\mathrm{edit}}$ . The decoder generates a sequence of edit actions, which will have to be parsed into a comment (\$4.4).
4 Representing Edits
Here we define the edit lexicon that is used to construct the input code edit sequence,  $\mathbf{M}_{\mathrm{edit}}$ , and the target comment edit sequence,  $\mathbf{C}_{\mathrm{edit}}$ .
4.1 Edit Lexicon
We use difflib<sup>3</sup> to extract code edits and target comment edits. Both the input code edit sequence and the target comment edit sequence consist of a se
ries of edit actions; each edit action is structured as <Action> [span of tokens] <ActionEnd>.4
We define four types of edit actions: Insert, Delete, Replace, and Keep. Because the Replace action must simultaneously incorporate distinct content from two versions (i.e., tokens in the old version that will be replaced, and tokens in the new version that will take their place), it follows a slightly different structure:
$$
\begin{array}{l} \text {<   R e p l a c e O l d >} \quad [ \text {s p a n o f o l d t o k e n s} ] \\ \text {<   R e p l a c e N e w >} \quad [ \text {s p a n o f n e w t o k e n s} ] \\ \text {<   R e p l a c e E n d >} \end{array}
$$
4.2 Code Edits
We extract the edits between  $\mathbf{M}_{\mathrm{old}}$  and  $\mathbf{M}_{\mathrm{new}}$  using the edit lexicon to construct  $\mathbf{M}_{\mathrm{edit}}$ , the code edit sequence used as input in one of the encoders. Figure 2 (top right) shows the  $\mathbf{M}_{\mathrm{edit}}$  corresponding to code changes in Figure 1.
In contrast to line-level code diffs that are commonly used for commit message generation (Loyola et al., 2017; Jiang et al., 2017; Xu et al., 2019), this representation allows us to explicitly capture more fine-grained edits. While we could exploit the abstract syntax tree (AST) structure of source code and represent the changes between the ASTs corresponding to the two versions of code, prior work suggests that such techniques do not always lead to improved performance (Yin et al., 2019). We leave it to future work to investigate how the AST structure can be leveraged for this task.
4.3 Comment Edits
We identify the changes between  $C_{\mathrm{old}}$  and  $C_{\mathrm{new}}$  to construct  $C_{\mathrm{edit}}$ , the target comment edit sequence. During inference, the output comment is produced by parsing the predicted edit sequence (§4.4). We introduce a slightly modified set of specifications that disregards the Keep type when constructing the sequence of edit actions, referred to as the condensed edit sequence.
The intuition for disregarding Keep and the span of tokens to which it applies is that we can simply copy the content that is retained between  $C_{\text{old}}$  and  $C_{\text{new}}$ , instead of generating it anew. By doing post-hoc copying, we simplify learning for the model since it has to only learn what to change rather than also having to learn what to keep.
We design a method to deterministically place edits in their correct positions in the absence of
4 Preliminary experiments showed that this performed better than structuring edits at the token-level as in other tasks (Shin et al., 2018; Li et al., 2018; Dong et al., 2019; Awasthi et al., 2019).
3https://docs.python.org/3/library/ difflib.html
Keep spans. For the example in Figure 1, the raw sequence <Insert>in degrees<InsertEnd> does not encode information as to where "in degrees" should be inserted. To address this, we bind an insert sequence with the minimum number of words (aka "anchors") such that the place of insertion can be uniquely identified. This results in the structure that is shown for Cedit in Figure 2. Here "angle" serves as the anchor point, identifying the insert location. Following the structure of Replace, this sequence indicates that "angle" should be replaced with "angle in degrees," effectively inserting "in degrees" and keeping "angle" from Cold, which appears immediately before the insert location. See Appendix A for details on this procedure.
4.4 Parsing Edit Sequences
Since the decoder is trained to predict a sequence of edit actions, we must align it with  $C_{\text{old}}$  and copy unchanged tokens in order to produce the edited comment. We denote the predicted edit sequence as  $C'_{\text{edit}}$  and the corresponding parsed output as  $C'_{\text{new}}$ . This procedure entails simultaneously following pointers, left-to-right, on  $C_{\text{old}}$  and  $C'_{\text{edit}}$ , which we refer to as  $P_{\text{old}}$  and  $P_{\text{edit}}$  respectively.  $P_{\text{old}}$  is advanced, copying the current token into  $C'_{\text{new}}$  at each point, until an edit location is reached. The edit action corresponding to the current position of  $P_{\text{edit}}$  is then applied, and the tokens from its relevant span are copied into  $C'_{\text{new}}$  if applicable. Finally,  $P_{\text{edit}}$  is advanced to the next action, and  $P_{\text{old}}$  is also advanced to the appropriate position in cases involving deletions and replacements. This process repeats until both pointers reach the end of their respective sequences.
5 Features
We extract linguistic and lexical features for tokens in  $\mathbf{M}_{\mathrm{edit}}$  and  $\mathbf{C}_{\mathrm{edit}}$ , many of which were shown to improve learning associations between @return comment and source code entities in our prior work (Panthaplackel et al., 2020). We incorporate these features into the network as one-hot vectors that are concatenated to  $\mathbf{M}_{\mathrm{edit}}$  and  $\mathbf{C}_{\mathrm{edit}}$  embeddings and then passed through a linear layer. These vectors are provided as inputs to the two encoders. All sequences are subtokenized, e.g., camelCase → camel, case.
Features specific to  $\mathbf{M}_{\mathrm{edit}}$ : We aim to take advantage of common patterns among different types of code tokens by incorporating features that identify certain categories: edit keywords, Java keywords,
and operators. If a token is not an edit keyword, we have indicator features for whether it is part of a Insert, Delete, ReplaceNew, ReplaceOld, or Keep span. We believe this will be particularly helpful for longer spans since edit keywords only appear at either the beginning or end of a span. Finally, we include a feature to indicate whether the token matches a token in  $C_{\text{old}}$ . This is intended to help the model identify locations in  $M_{\text{edit}}$  that may be relevant to editing  $C_{\text{old}}$ .
Features specific to  $\mathbf{C}_{\mathrm{old}}$ : We include whether a token matches a code token that is inserted, deleted, or replaced in  $\mathbf{M}_{\mathrm{edit}}$ . These help align parts of  $\mathbf{C}_{\mathrm{old}}$  with code edits, assisting the model in determining where edits should be made. In order to exploit common patterns for different types of tokens, we incorporate features that identify whether the token appears more than once in  $\mathbf{C}_{\mathrm{old}}$  or is a stop word, and its part-of-speech.
Shared features: We include whether the token is a substring that was originally part of a larger token and its index if so (e.g., split from camelCase, camel and case are subtokens with indices 0 and 1 respectively). These features aim to encode important relationships between adjacent tokens that are lost once the body of code and comment are transformed into a single, substringed sequences. Additionally, because we focus on @return comments, we introduce features intended to guide the model in identifying relevant tokens in Medit and Cold. Namely, we include whether a given token matches a token in a return statement that is unique to Mold, unique to Mnew, or present in both. Similarly, we indicate whether the token matches a token in the substringed return type that is unique to Mold, unique to Mnew, or present in both.
6Reranking
Reranking allows the incorporation of additional priors that are difficult to back-propagate, by re-scoring candidate sequences during beam search (Neubig et al., 2015; Ko et al., 2019; Kriz et al., 2019). We incorporate two heuristics to rescore the candidates: 1) generation likelihood and 2) similarity to  $\mathbf{C}_{\mathrm{old}}$ . These heuristics are computed after parsing the candidate edit sequences (\$4.4).
Generation likelihood. Since the edit model is trained on edit actions only, it does not globally score the resulting comment in terms of aspects such as fluency and overall suitability for the updated method. To this end, we make use of a pretrained comment generation model (§8.2) that is
Train	Valid	Test
Examples	5,791	712	736
Projects	526	274	281
Edit Actions	8,350	1,038	1,046
Sim (Mold, Mnew)	0.773	0.778	0.759
Sim (Cold, Cnew)	0.623	0.645	0.635
Code	Unique	7,271	2,473	2,690
Mean	86.4	87.4	97.4
Median	46	49	50
Comm.	Unique	4,823	1,695	1,737
Mean	10.8	11.2	11.1
Median	8	9	9
Table 1: Number of examples, projects, and edit actions; average similarity between  ${\mathrm{M}}_{\text{old }}$  and  ${\mathrm{M}}_{\text{new }}$  as the ratio of overlap; average similarity between  ${\mathrm{C}}_{\text{old }}$  and  ${\mathrm{C}}_{\text{new }}$  as the ratio of overlap; number of unique code tokens and mean and median number of tokens in a method; and number of unique comment tokens and mean and median number of tokens in a comment.
trained on a substantial amount of data for generating  $C_{\mathrm{new}}$  given only  $M_{\mathrm{new}}$ . We compute the length-normalized probability of this model generating the parsed candidate comment,  $C_{\mathrm{new}}'$ , (i.e.,  $P(C_{\mathrm{new}}'|M_{\mathrm{new}})^{1/N}$  where  $N$  is the number of tokens in  $C_{\mathrm{new}}'$ ). This model gives preference to comments that are more likely for  $M_{\mathrm{new}}$  and are more consistent with the general style of comments.
Similarity to  $\mathbf{C}_{\mathrm{old}}$ . So far, our model is mainly trained to produce accurate edits; however, we also follow intuitions that edits should be minimal (as an analogy, the use of Levenshtein distance in spelling correction). To give preference to predictions that accurately update the comment with minimal modifications, we use similarity to  $\mathbf{C}_{\mathrm{old}}$  as a heuristic for reranking. We measure similarity between the parsed candidate prediction and  $\mathbf{C}_{\mathrm{old}}$  using ME-TEOR (Banerjee and Lavie, 2005).
Reranking score. The reranking score for each candidate is a linear combination of the original beam score, the generation likelihood, and the similarity to  $C_{\mathrm{old}}$  with coefficients 0.5, 0.3, and 0.2 respectively (tuned on validation data).
7 Data
We extracted examples from popular, open-source Java projects using GitHub's commit history. We extract pairs of the form (method, comment) for the same method across two consecutive commits where there is a simultaneous change to both the code and comment. This creates somewhat noisy data for the task of comment update; Appendix B describes filtering techniques to reduce this noise.
We first tokenize  $\mathbf{M}_{\mathrm{old}}$  and  $\mathbf{M}_{\mathrm{new}}$  using the javalang library. We substringize based on camelCase and snake(case, as in previous work (Allamanis et al., 2016; Alon et al., 2019; Fernandes et al., 2019). We then form  $\mathbf{M}_{\mathrm{edit}}$  from the substringized forms of  $\mathbf{M}_{\mathrm{old}}$  and  $\mathbf{M}_{\mathrm{new}}$ . We concatenate  $\mathbf{C}_{\mathrm{old}}$  and  $\mathbf{C}_{\mathrm{new}}$  by splitting by space and punctuation. We remove HTML tags and the "@return" that precedes all comments, and also substringize tokens since code tokens may appear in comments as well. The gold edit action sequence,  $\mathbf{C}_{\mathrm{edit}}$ , is computed from these processed forms of  $\mathbf{C}_{\mathrm{old}}$  and  $\mathbf{C}_{\mathrm{new}}$ .
To avoid having examples that closely resemble one another in training and test, the projects in the training, test, and validation sets are disjoint, similar to Movshovitz-Attias and Cohen (2013). Table 1 gives dataset statistics. Of the 7,239 examples in our final dataset, 833 of them were extracted from the diffs used in Panthaplackel et al. (2020). Including code and comment tokens that appear at least twice in the training data as well as the predefined edit keywords, the code and comment vocabulary sizes are 5,945 and 3,642 respectively.
8 Experimental Method
We evaluate our approach against multiple rule-based baselines and comment generation models.
8.1 Baselines
Copy: Since much of the content of  $C_{\mathrm{old}}$  is typically retained in the update, we include a baseline that merely copies  $C_{\mathrm{old}}$  as the prediction for  $C_{\mathrm{new}}$ .
Return type substitution: The return type of a method often appears in its @return comment. If the return type of  $\mathbf{M}_{\mathrm{old}}$  appears in  $\mathbf{C}_{\mathrm{old}}$  and the return type is updated in the code, we substitute the new return type while copying all other parts of  $\mathbf{C}_{\mathrm{old}}$ . Otherwise,  $\mathbf{C}_{\mathrm{old}}$  is copied as the prediction.
Return type substitution w/ null handling: As an addition to the previous method, we also check whether the token null is added to either a return statement or if statement in the code. If so, we copy  $C_{\text{old}}$  and append the string or null if null, otherwise, we simply copy  $C_{\text{old}}$ . This baseline addresses a pattern we observed in the data in which ways to handle null input or cases that could result in null output were added.
<sup>5</sup>We attempted to integrate this model into the training procedure of the edit model through joint training; however, this deteriorated performance.
$^{6}$ https://pypi.org/project/javalang/
8.2 Generation Model