SlowGuess/ACL_2020_OCR huggingface dataset:
One of our main hypotheses is that modeling edit sequences is better suited for this task than generating comments from scratch. However, a counter argument could be that a comment generation model could be trained from substantially more data, since it is much easier to obtain parallel data in the form (method, comment), without the constraints of simultaneous code/comment edits. Hence the power of large-scale training could out-weigh edit modeling. To this end, we compare with a generation model trained on 103,473 method/@return comment pairs collected from GitHub.
We use the same underlying neural architecture as our edit model to make sure that the difference in results comes from the amount of training data and from using edit of representations only: a two-layer, bi-directional GRU that encodes the sequence of tokens in the method, and an attention-based GRU decoder with a copy mechanism that decodes a sequence of comment tokens. We expect the incorporation of more complicated architectures, e.g., tree-based (Alon et al., 2019) and graph-based (Fernandes et al., 2019) encoders which exploit AST structure, can be applied to both an edit model and a generation model, which we leave for future work.
Evaluation is based on the 736  $(\mathbf{M}_{\mathrm{new}},\mathbf{C}_{\mathrm{new}})$  pairs in the test set described in  $\S 7$  .We ensure that the projects from which training examples are extracted are disjoint from those in the test set.
8.3 Reranked Generation Model
In order to allow the generation model to exploit the old comment, this system uses similarity to  $C_{\mathrm{old}}$  (cf. §6) as a heuristic for reranking the top candidates from the previous model. The reranking score is a linear combination of the original beam score and the METEOR score between the candidate prediction and  $C_{\mathrm{old}}$ , both with coefficient 0.5 (tuned on validation data).
8.4 Model Training
Model parameters are identical across the edit model and generation model, tuned on validation data. Encoders have hidden dimension 64, the decoder has hidden dimension 128, and the dimension for code and comment embeddings is 64. The embeddings used in the edit model are initialized using the pre-trained embedding vectors from the generation model. We use a dropout rate of 0.6, a batch size of 100, an initial learning rate of 0.001,
and Adam optimizer. Models are trained to minimize negative log likelihood, and we terminate training if the validation loss does not decrease for ten consecutive epochs. During inference, we use beam search with beam width  $= 20$ .
9 Evaluation
9.1 Automatic Evaluation
Metrics: We compute exact match, i.e., the percentage of examples for which the model prediction is identical to the reference comment  $C_{\mathrm{new}}$ . This is often used to evaluate tasks involving source code edits (Shin et al., 2018; Yin et al., 2019). We also report two prevailing language generation metrics: METEOR (Banerjee and Lavie, 2005), and average sentence-level BLEU-4 (Papineni et al., 2002) that is previously used in code-language tasks (Iyer et al., 2016; Loyola et al., 2017).
Previous work suggests that BLEU-4 fails to accurately capture performance for tasks related to edits, such as text simplification (Xu et al., 2016), grammatical error correction (Napoles et al., 2015), and style transfer (Sudhakar et al., 2019), since a system that merely copies the input text often achieves a high score. Therefore, we also include two text-editing metrics to measure how well our system learns to edit: SARI (Xu et al., 2016), originally proposed to evaluate text simplification, is essentially the average of N-gram F1 scores corresponding to add, delete, and keep edit operations; GLEU (Napoles et al., 2015), used in grammatical error correction and style transfer, takes into account the source sentence and deviates from BLEU by giving more importance to n-grams that have been correctly changed.
Results: We report automatic metrics averaged across three random initializations for all learned models, and use bootstrap tests (Berg-Kirkpatrick et al., 2012) for statistical significance. Table 2 presents the results. While reranking using  $C_{old}$  appears to help the generation model, it still substantially underperforms all other models, across all metrics. Although this model is trained on considerably more data, it does not have access to  $C_{old}$  during training and uses fewer inputs and consequently has less context than the edit model. Reranking slightly deteriorates the edit model's
7Although the original formulation only used precision for the delete operation, more recent work computes F1 for this as well (Dong et al., 2019; Alva-Manchego et al., 2019).
Model	xMatch (%)	METEOR	BLEU-4	SARI	GLEU
Baselines	Copy	0.000	34.611	46.218	19.282	35.400
Return type subt.	13.723$	43.106¶	50.796||	31.723	42.507*
Return type subst. + null	13.723$	43.359	51.160†	32.109	42.627*
Models	Generation	1.132	11.875	10.515	21.164	17.350
Edit	17.663	42.222¶	48.217	46.376	45.060
Reranked models	Generation	2.083	18.170	18.891	25.641	22.685
Edit	18.433	44.698	50.717||†	45.486	46.118
performance with respect to SARI; however, it provides statistically significant improvements on most other metrics.
Although two of the baselines achieve slightly higher BLEU-4 scores than our best model, these differences are not statistically significant, and our model is better at editing comments, as shown by the results on exact match, SARI, and GLEU. In particular, our edit models beat all other models with wide, statistically significant, margins on SARI, which explicitly measures performance on edit operations. Furthermore, merely copying  $C_{\mathrm{old}}$ , yields a relatively high BLEU-4 score of 46.218. The return type substitution and return type substitution w/ null handling baselines produce predictions that are identical to  $C_{\mathrm{old}}$  for  $74.73\%$  and  $65.76\%$  of the test examples, respectively, while it is only  $9.33\%$  for the reranked edit model. In other words, the baselines attain high scores on automatic metrics and even beat our model on BLEU-4, without actually performing edits on the majority of examples. This further underlines the shortcomings of some of these metrics and the importance of conducting human evaluation for this task.
9.2 Human Evaluation
Automatic metrics often fail to incorporate semantic meaning and sentence structure in evaluation as well as accurately capture performance when there is only one gold-standard reference; indeed, these metrics do not align with human judgment in other generation tasks like grammatical error correction (Napoles et al., 2015) and dialogue generation (Liu et al., 2016). Since automatic metrics have not yet been explored in the context of the new task we are proposing, we find it necessary to conduct human evaluation and study whether these metrics are consistent with human judgment.
User study design: Our study aims to reflect how a comment update system would be used in practice, such as in an Integrated Development En
Table 2: Exact match, METEOR, BLEU-4,SARI,and GLEU scores. Scores for which the difference in performance is not statistically significant  $\left( {\mathrm{p} < {0.05}}\right)$  are indicated with matching symbols.
Baseline	Generation	Edit	None
18.4%	12.4%	30.2%	55.0%
Table 3: Percentage of annotations for which users selected comment suggestions produced by each model. All differences are statistically significant  $(p < 0.05)$ .
vironment (IDE). When developers change code, they would be shown suggestions for updating the existing comment. If they think the comment needs to be updated to reflect the code changes, they could select the one that is most suitable for the new version of the code or edit the existing comment themselves if none of the options are appropriate.
We simulated this setting by asking a user to select the most appropriate updated comment from a list of suggestions, given  $C_{old}$  as well as the diff between  $M_{old}$  and  $M_{new}$  displayed using GitHub's diff interface. The user can select multiple options if they are equally good or a separate None option if no update is needed or all suggestions are poor.
The list of suggestions consists of up to three comments, predicted by the strongest benchmarks and our model: (1) return type substitution w/ null handling, (2) reranked generation model, and (3) reranked edit model, arranged in randomized order. We collapse identical predictions into a single suggestion and reward all associated models if the user selects that comment. Additionally, we remove any prediction that is identical to  $C_{\mathrm{old}}$  to avoid confusion as the user should never select such a suggestion. We excluded 6 examples from the test set for which all three models predicted  $C_{\mathrm{old}}$  for the updated comment.
Nine students (8 graduate/1 undergraduate) and one full-time developer at a large software company, all with  $2+$  years of Java experience, participated in our study. To measure inter-annotator agreement, we ensured that every example was evaluated by two users. We conducted a total of 500 evaluations, across 250 distinct test examples.
Results: Table 3 presents the percentage of annotations (out of 500) for which users selected
Figure 3: Changes in the getComplex method and its corresponding @return comment between two subsequent commits of the eclipse-january project, available on GitHub.
comment suggestions that were produced by each model. Using Krippendorff's  $\alpha$  (Krippendorff, 2011) with MASI distance (Passonneau, 2006) (which accommodates our multi-label setting), inter-annotator agreement is 0.64, indicating satisfactory agreement. The reranked edit model beats the strongest baseline and reranked generation by wide statistically-significant margins. From rationales provided by two annotators, we observe that some options were not selected because they removed relevant information from the existing comment, and not surprisingly, these options often corresponded to the comment generation model.
Users selected none of the suggested comments  $55\%$  of the time, indicating there are many cases for which either the existing comment did not need updating, or comments produced by all models were poor. Based on our inspection of a sample these, we observe that in a large portion of these cases, the comment did not warrant an update. This is consistent with prior work in sentence simplification which shows that, very often, there are sentences that do not need to be simplified (Li and Nenkova, 2015). Despite our efforts to minimize such cases in our dataset through rule-based filtering techniques, we found that many remain. This suggests that it would be beneficial to train a classifier that first determines whether a comment needs to be updated before proposing a revision. Furthermore, the cases for which the existing comment does need to be updated but none of the models produce reasonable predictions illustrate the scope for improvement for our proposed task.
10 Error Analysis
We find that our model performs poorly in cases requiring external knowledge and more context than that provided by the given method. For instance, correctly updating the comment shown in Figure 3 requires knowing that get returns the item in the first position if no argument is provided. Our model does not have access to this information, and it
fails to generate a reasonable update: "@return complex in given position." On the other hand, the reranked generation model produces "@return the complex value" which is arguably reasonable for the given context. This suggests that incorporating more code context could be beneficial for both models. Furthermore, we find that our model tends to make more mistakes when it must reason about a large amount of code change between  $\mathbf{M}_{\mathrm{old}}$  and  $\mathbf{M}_{\mathrm{new}}$ , and we found that in many such cases, the output of the reranked generation model was better. This suggests that when there are substantial code changes,  $\mathbf{M}_{\mathrm{new}}$  effectively becomes a new method, and generating a comment from scratch may be more appropriate. Ensembling generation with our system through a regression model that predicts the extent of editing that is needed may lead to a more generalizable approach that can accommodate such cases. Sample outputs are given in Appendix C.
11 Ablations
We empirically study the effect of training the network to encode explicit code edits and decode explicit comment edits. As discussed in Section 3, the edit model consists of two encoders, one that encodes  $\mathbf{C}_{\mathrm{old}}$  and another that encodes the code representation,  $\mathbf{M}_{\mathrm{edit}}$ . We conduct experiments in which the code representation instead consists of either (1)  $\mathbf{M}_{\mathrm{new}}$  or (2) both  $\mathbf{M}_{\mathrm{old}}$  and  $\mathbf{M}_{\mathrm{new}}$  (encoded separately and hidden states concatenated). Additionally, rather than having the decoder generate comment edits in the form  $\mathbf{C}_{\mathrm{edit}}$ , we introduce experiments in which it directly generates  $\mathbf{C}_{\mathrm{new}}$ , with no intermediate edit sequence. For this, we use only the underlying architecture of the edit model (without features or reranking). The performance for various combinations of input code and target comment representations are shown in Table 4.
By comparing performance across combinations consisting of the same input code representation and varying target comment representations, the importance of training the decoder to generate a sequence of edit actions rather than the full updated comment is very evident. Furthermore, comparing across varying code representations under the  $C_{\text{edit}}$  target comment representation, it is clear that explicitly encoding the code changes, as  $M_{\text{edit}}$ , leads to significant improvements across most metrics.
We further ablate the features introduced in §5. As shown in Table 5, these features improve performance by wide margins, across all metrics.
Inputs	Output	xM (%)	METEOR	BLEU-4	SARI	GLEU
Cold, Mnew	Cnew	5.707‡	29.259†	33.534§	28.024	30.000*
Cedit	4.755‡*	33.796	43.315	35.516	37.970||
Cold, Mold, Mnew	Cnew	3.714*	18.729	20.060	23.914	21.956
Cedit	5.163‡¶	34.895	44.006*	33.479	37.618||
Cold, Medit	Cnew	6.114¶	29.968†	34.164§	28.980	30.491*
Cedit	8.922	36.229	44.283*	40.538	39.879
Table 4: Exact match, METEOR, BLEU-4,SARI,and GLEU for various combinations of code input and target comment output configurations. Features and reranking are disabled for all models. Scores for which the difference in performance is not statistically significant  $\left( {\mathrm{p} < {0.05}}\right)$  are indicated with matching symbols.
Model	xM (%)	METEOR	BLEU-4	SARI	GLEU
Models	Edit	17.663	42.222	48.217	46.376	45.060
- feats.	8.922†	36.229	44.283	40.538	39.879*
Reranked models	Edit	18.433	44.698	50.717	45.486	46.118
- feats.	8.877†	38.446	46.665	36.924	40.317*
Table 5: Exact match, METEOR, BLEU-4, SARI, and GLEU scores of ablated models. Scores for which the difference in performance is not statistically significant (p < 0.05) are indicated with matching symbols.
12 Related Work
Learning from source code changes: Lee et al. (2019) use rule-based techniques to automatically detect and revise outdated API names in code documentation; however, their approach cannot be extended to full natural language comments that are the focus of this work. Zhai et al. (2020) propose a technique for updating incomplete and buggy comments by propagating comments from different code elements (e.g., variables, methods, classes) based on program analysis and several heuristics. Rather than simply copying a related comment, we aim to revise an outdated comment by reasoning about code changes. Yin et al. (2019) present an approach for learning structural and semantic properties of source code edits so that they can be generalized to new code inputs. Similar to their work, we learn vector representations from source code changes; however, unlike their setting, we apply these representations to natural language. Prior work in automatic commit message generation aims to learn from code changes in order to generate a natural language summary of these changes (Loyola et al., 2017; Jiang et al., 2017; Xu et al., 2019). Instead of generating natural language content from scratch as done in their work, we focus on applying edits to existing natural language text. We also show that generating a comment from scratch does not perform as well as our proposed edit model for the comment update setting.
Editing natural language text: Approaches for editing natural language text have been studied extensively through tasks such as sentence simplification (Dong et al., 2019), style transfer (Li et al., 2018), grammatical error correction (Awasthi et al.,
2019), and language modeling (Guu et al., 2018). The focus of this prior work is to revise sentences to conform to stylistic and grammatical conventions, and it does not generally consider broader contextual constraints. On the contrary, our goal is not to make cosmetic revisions to a given span of text, but rather amend its semantic meaning to be in sync with the content of a separate body of information on which it is dependent. More recently, Shah et al. (2020) proposed an approach for rewriting an outdated sentence based on a sentence stating a new factual claim, which is more closely aligned with our task. However, in our case, the separate body of information is not natural language and is generally much longer than a single sentence.
13 Conclusion
We have addressed the novel task of automatically updating an existing programming comment based on changes to the related code. We designed a new approach for this task which aims to correlate cross-modal edits in order to generate a sequence of edit actions specifying how the comment should be updated. We find that our model outperforms multiple rule-based baselines and comment generation models, with respect to several automatic metrics and human evaluation.
We thank reviewers for their feedback on this work and the participants of our user study for their time. This work was partially supported by a Google Faculty Research Award and the US National Science Foundation under Grant Nos. CCF-1652517 and IIS-1850153.
Miltiadis Allamanis. 2019. The adverse effects of code duplication in machine learning models of code. In SPLASH, Onward!, pages 143-153.
Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for extreme summarization of source code. In International Conference on Machine Learning, pages 2091-2100.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019. code2seq: Generating sequences from structured representations of code. In International Conference on Learning Representations.
Fernando Alva-Manchego, Louis Martin, Carolina Scarton, and Lucia Specia. 2019. EASSE: Easier automatic sentence simplification evaluation. In Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing: System Demonstrations, pages 49-54.
Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019. Parallel iterative edit models for local sequence transduction. In Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing, pages 4251-4261.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for MT evaluation with improved correlation with human judgments. In Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72.
Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in NLP. In Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995-1005.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing, pages 1724-1734.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural programmer-Interpreter model for sentence simplification through explicit editing. In Annual Meeting of the Association for Computational Linguistics, pages 3393-3402.
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Structured neural summarization. In International Conference on Learning Representations.
Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450.
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment generation. In International Conference on Program Comprehension, pages 200-210.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Annual Meeting of the Association for Computational Linguistics, pages 2073-2083.
Siyuan Jiang, Ameer Armaly, and Collin McMillan. 2017. Automatically generating commit messages from diffs using neural machine translation. In International Conference on Automated Software Engineering, pages 135-146.
Wei-Jen Ko, Greg Durrett, and Junyi Jessy Li. 2019. Linguistically-informed specificity and semantic plausibility for dialogue generation. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3456-3466.
Klaus Krippendorff. 2011. Computing Krippendorff's alpha reliability. Technical report, University of Pennsylvania.
Reno Kriz, João Sedoc, Marianna Apidianaki, Carolina Zheng, Gaurav Kumar, Eleni Miltsakaki, and Chris Callison-Burch. 2019. Complexity-weighted loss and diverse reranking for sentence simplification. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3137-3147.
Seonah Lee, Rongxin Wu, S.C. Cheung, and Sungwon Kang. 2019. Automatic detection and update suggestion for outdated API names in documentation. Transactions on Software Engineering.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1865-1874.
Junyi Jessy Li and Ani Nenkova. 2015. Fast and accurate prediction of sentence specificity. In AAAI Conference on Artificial Intelligence, pages 2281-2287.
Yuding Liang and Kenny Q. Zhu. 2018. Automatic generation of text descriptive comments for code blocks. In AAAI Conference on Artificial Intelligence, pages 5229-5236.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Conference on