SlowGuess/ACL_2020_OCR huggingface dataset:
Empirical Methods in Natural Language Processing, pages 2122-2132.
Pablo Loyola, Edison Marrese-Taylor, and Yutaka Matsuo. 2017. A neural architecture for generating natural language descriptions from source code changes. In Annual Meeting of the Association for Computational Linguistics, pages 287-292.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Conference on Empirical Methods in Natural Language Processing, pages 1412-1421.
Dana Movshovitz-Attias and William W. Cohen. 2013. Natural language models for predicting programming comments. In Annual Meeting of the Association for Computational Linguistics, pages 35-40.
Courtney Naples, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, pages 588-593.
Graham Neubig, Makoto Morishita, and Satoshi Nakamura. 2015. Neural reranking improves subjective quality of machine translation: NAIST at WAT2015. In Workshop on Asian Translation, pages 35-41.
Sheena Panthaplackel, Milos Gligoric, Raymond J. Mooney, and Junyi Jessy Li. 2020. Associating natural language comment and source code entities. In AAAI Conference on Artificial Intelligence.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics, pages 311-318.
Rebecca Passonneau. 2006. Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In International Conference on Language Resources and Evaluation.
Inderjot Kaur Ratol and Martin P. Robillard. 2017. Detecting fragile comments. International Conference on Automated Software Engineering, pages 112-122.
Darsh J. Shah, Tal Schuster, and Regina Barzilay. 2020. Automatic fact-guided sentence modification. In AAAI Conference on Artificial Intelligence.
Richard Shin, Illia Polosukhin, and Dawn Song. 2018. Towards specification-directed program repair. In International Conference on Learning Representations Workshop.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. 2019. "Transforming" delete, retrieve, generate approach for controlled text style transfer. In Conference on Empirical Methods in Natural Language Processing and the International Joint
Conference on Natural Language Processing, pages 3267-3277.
Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. /*iComment: Bugs or bad comments?*/. In Symposium on Operating Systems Principles, pages 145-158.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692-2700.
Shengbin Xu, Yuan Yao, Feng Xu, Tianxiao Gu, Hanghang Tong, and Jian Lu. 2019. Commit message generation for source code changes. In International Joint Conference on Artificial Intelligence, pages 3975-3981.
Wei Xu, Courtney Naples, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401-415.
Ziyu Yao, Daniel S. Weld, Wei-Peng Chen, and Huan Sun. 2018. StaQC: A systematically mined question-code dataset from Stack Overflow. In International Conference on World Wide Web, pages 1693-1703.
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning to mine aligned code and natural language pairs from Stack Overflow. In International Conference on Mining Software Repositories, pages 476-486.
Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Marc Brockschmidt, and Alexander L. Gaunt. 2019. Learning to represent edits. In International Conference on Learning Representations.
Juan Zhai, Xiangzhe Xu, Yu Shi, Guanhong Tao, Minxue Pan, Shiqing Ma, Lei Xu, Weifeng Zhang, Lin Tan, and Xiangyu Zhang. 2020. CPC: Automatically classifying and propagating natural language comments via program analysis. In International Conference on Software Engineering.
Train	Valid	Test
Total actions	8,350	1,038	1,046
Avg. # actions per example	1.44	1.46	1.42
Replace	51.9%	49.7%	50.1%
ReplaceKeepBefore	2.9%	2.6%	3.5%
ReplaceKeepAfter	0.7%	0.3%	0.4%
InsertKeepBefore	21.5%	24.1%	23.2%
InsertKeepAfter	4.2%	4.0%	3.3%
Delete	17.4%	18.0%	17.8%
DeleteKeepBefore	1.3%	0.7%	1.1%
DeleteKeepAfter	0.2%	0.5%	0.6%
Table 6: Total number of edit actions; average number of edit actions per example; percentage of total actions that is accounted by each edit action type.
A Modified Comment Edit Lexicon
We first transform insertions and ambiguous deletions into a structure that resembles Replace, characterized by InsertOld/InsertNew and DeleteOld/DeleteNew spans respectively. Next, we require the span of tokens attached to ReplaceOld, InsertOld, and DeleteOld to be unique across  $C_\mathrm{old}$  so that we can uniquely identify the edit location. We enforce this by iteratively searching through unchanged tokens before and after the span, incorporating additional tokens into the span, until the span becomes unique. These added tokens are then included in both components of the action. For instance, if the last A is to be replaced with C in ABA, the ReplaceOld span would be BA and the ReplaceNew span would be BC. We also augment the edit types to differentiate between the various scenarios that may arise from this search procedure.
Replace actions for which this procedure is performed deviate from the typical nature of Replace in which there is no overlap between the spans attached to ReplaceOld and ReplaceNew. This is because the tokens that are added to make the ReplaceOld span unique will appear in both spans. These tokens, which are effectively kept between  $C_{\text{old}}$  and  $C_{\text{new}}$ , could appear before or after the edit location. We differentiate between these scenarios by augmenting the edit lexicon with new edit types. In addition to Replace, we have ReplaceKeepBefore and ReplaceKeepAfter to signify that the action entails retaining some content before or after, respectively. We include the same for the other types as well with InsertKeepBefore, InsertKeepAfter, DeleteKeepBefore, DeleteKeepAfter. Table 6 shows statistics on how often each of these edit actions are used. We present more details about the actions in the sections that follow.
A.1 Replacements
Replace This action is defined as shown below:
<ReplaceOld>[old span] <ReplaceNew>[new span] <ReplaceEnd>
It prescribes that the tokens attached to ReplaceOld are deleted and the tokens attached to ReplaceNew are inserted in their place. There is almost never overlap between the span of tokens attached to ReplaceOld and ReplaceNew. Example: if B is to be replaced with C in  $C_{old} = AB$  to produce  $C_{new} = AC$ , the corresponding  $C_{edit}$  is:
<ReplaceOld>B <ReplaceNew>C <ReplaceEnd>
Note that the span attached to ReplaceOld must be unique across  $C_{old}$  for this edit type to be used.
ReplaceKeepBefore This action is defined as shown below:
<ReplaceOldKeepBefore>[old span] <ReplaceNewKeepBefore>[new span] <ReplaceEnd>
Replace is transformed into this structure if the span attached to ReplaceOld is not unique. For example, suppose the first B is to be replaced with D in  $C_{old} = ABCB$  to produce  $C_{new} = ADCB$ . If  $C_{edit}$  consists of a ReplaceOld span carrying just B, it is not obvious whether the first or last B should be replaced. To address this, we introduce a new edit type, ReplaceKeepBefore, which forms a unique span by searching before the edit location. It prescribes that the tokens attached to ReplaceOldKeepBefore are deleted and the tokens attached to ReplaceNewKeepBefore are inserted in their place. Unlike Replace, there will be some overlap at the beginning of the spans attached to ReplaceOldKeepBefore and ReplaceNewKeepBefore. To represent edits  $C_{old} = ABCB$  to produce  $C_{new} = ADCB$ ,  $C_{edit}$  is:
<ReplaceOldKeepBefore> AB <ReplaceNewKeepBefore> AD <ReplaceEnd>
The span attached to ReplaceOldKeepBefore is unique, making it clear that the first B is to be replaced with D. It also indicates that we are effectively keeping A, before the edit location.
ReplaceKeepAfter This action is defined as shown below:
Replace is transformed into this structure if the span attached to ReplaceOld is not unique and ReplaceKeepBefore cannot be used because we are unable to find a unique sequence of unchanged tokens before the edit location. For example, suppose the first B is to be replaced with D in  $C_{old} = ABCAB$  to produce  $C_{new} = ADCAB$ . Searching before the edit location, we find only AB, which is not unique across  $C_{old}$ , and so it would still not be clear which B is to be edited. To address this, we introduce a new edit type, ReplaceKeepAfter, which forms a unique span by searching after the edit location. It prescribes that the tokens attached to ReplaceOldKeepAfter are deleted and the tokens attached to ReplaceNewKeepAfter are inserted in their place. Unlike Replace and ReplaceKeepBefore, there will be some overlap at the end of the spans attached to ReplaceOldKeepAfter and ReplaceNewKeepAfter. Therefore, to represent editing  $C_{old} = ABCAB$  to produce  $C_{new} = ADCAB$ ,  $C_{edit}$  is:
The span attached to ReplaceOldKeepAfter is unique, making it clear that the first B is to be replaced with D. It also indicates that we are effectively keeping C, which appears after the edit location.
A.2 Insertions
We disregard basic Insert actions since it is always ambiguous where an insertion should occur without an anchor point. Following what is done for ambiguous Replace actions, we introduce InsertKeepBefore and InsertKeepAfter.
InsertKeepBefore This action is defined as shown below:
In this representation, the span of tokens attached to InsertOldKeepBefore must be unique and serve as the anchor point for where the new tokens should be inserted. We do this by searching before the
edit location. The structure is identical to that of ReplaceKeepBefore in that the tokens attached to InsertOldKeepBefore are replaced with the tokens in InsertNewKeepBefore and that there is some overlap at the beginning of the two spans. As an example, suppose C is to be inserted at the end of  $\mathrm{C_{old} = AB}$  to form  $\mathrm{C_{new} = ABC}$ . Then the corresponding  $\mathrm{C_{edit}}$  is as follows:
This states that we are effectively inserting C and keeping B, which appears before the edit location.
InsertKeepAfter This action is defined as shown below:
We rely on this when we are unable to use InsertKeepBefore because we cannot find a unique span of tokens to identify the anchor point, by searching before the edit location. For instance, suppose C is to be inserted at the beginning of  $C_{old} = AB$  to form  $C_{new} = CAB$ . There are no tokens that appear before the insert point, so we instead choose to search after. The structure is identical to that of ReplaceKeepAfter in that the tokens attached to InsertOldKeepAfter are replaced with the tokens in InsertNewKeepAfter and that there is some overlap at the end of the two spans. The corresponding  $C_{edit}$  from our example is as follows:
This states that we are effectively inserting C and keeping A, which appears after the edit location.
A.3 Deletions
Delete This action is defined as shown below:
It prescribes that the tokens that appear in the Delete span are removed from  $C_{old}$ . Example: if B is to be deleted from  $C_{old} = AB$  to produce  $C_{new} = A$ , the corresponding  $C_{edit}$  is:
Note that the Delete span must be unique across  $C_{old}$  for this edit type to be used.
DeleteKeepBefore This action is defined as shown below:
<DeleteOldKeepBefore>[old span] <DeleteNewKeepBefore>[new span] <DeleteEnd>
Delete is transformed into this structure if the Delete span is not unique. For example, suppose the first B is to be deleted from  $C_{old} = ABCB$  to produce  $C_{new} = ACB$ . From just  $C_{edit} = <Delete>B <DeleteEnd>$ , it is unclear which B is to be deleted. To address this, we introduce a new edit type, DeleteKeepBefore, which forms a unique span by searching before the edit location. The structure is identical to that of ReplaceKeepBefore in that the tokens attached to DeleteOldKeepBefore are replaced with the tokens in DeleteNewKeepBefore and that there is some overlap at the beginning of the two spans. For the example under consideration, the corresponding  $C_{edit}$  is given below:
<DeleteOldKeepBefore> AB <DeleteNewKeepBefore> A <DeleteEnd>
The span attached to DeleteOldKeepBefore is unique, making it clear that the first B is to be deleted. It also indicates that we are effectively keeping A, which appears before the edit location.
DeleteKeepAfter This action is defined as shown below:
<DeleteOldKeepAfter>[old span] <DeleteNewKeepAfter>[new span] <DeleteEnd>
Delete is transformed into this structure if the Delete span is not unique and DeleteKeepBefore cannot be used because we are unable to find a unique sequence of unchanged tokens before the edit location. For example, suppose the first B is to be deleted from  $C_{old} = ABCAB$  to produce  $C_{new} = ACAB$ . Searching before the edit location, we find only AB, which is not unique across  $C_{old}$ , and so it would still not be clear which B is to be deleted. To address this, we introduce a new edit type, DeleteKeepAfter, which forms a unique span by searching after the edit location. The structure is identical to that of ReplaceKeepAfter in that the tokens attached to DeleteOldKeepAfter are replaced with the tokens in DeleteNewKeepAfter and that there is some overlap at the end of the two spans. For the example under consideration,  $C_{edit}$  is as follows:
<DeleteOldKeepAfter> BC <DeleteNewKeepAfter> C <DeleteEnd>
The span attached to DeleteOldKeepAfter is unique, making it clear that the first B is to be deleted. It also indicates that we are effectively keeping C, which appears after the edit location.
B Data Filtering
As done in Panthaplackel et al. (2020), we apply heuristics to reduce the number of cases in which the code and comment changes are unrelated. First, because we focus on @return comments that pertain to the return values of a given method, we discard any example in which the code changes does not entail either a change to the return type or at least one return statement. Then, to identify the correct mapping of two versions of a method among other changes in a commit, we focus on the code changes that preserve the method names. It may happen sometimes that developers change the method name as well as code and comment in one commit, but we leave it as future work to improve this filtering heuristic. Next, we attempt to remove examples in which the comment change appears to be purely stylistic (e.g. spelling corrections, reformatting, and rephrasing). Furthermore, prior work (Allamanis, 2019) has shown that duplication can adversely affect evaluation of machine learning models for code and language tasks. For this reason, we remove duplicates from our corpus.
Despite having mined commit histories for thousands of projects, upon filtering, we are left with a total of 7,239 examples belonging to 1,081 different projects. This demonstrates the challenge of collecting large datasets with relatively low levels of noise in this domain. Although online code resources like GitHub and StackOverflow host large quantities of data that can be exploited for transduction tasks between source code and natural language, prior work has shown that much of this data is unusable without cleaning (Yin et al., 2018).
Some have used rule-based techniques to do data cleaning (Allamanis et al., 2016; Hu et al., 2018; Fernandes et al., 2019), and others train classifiers on hand-labeled examples that can be applied to a much larger pool of examples in order to differentiate between clean and noisy examples (Iyer et al., 2016; Yao et al., 2018; Yin et al., 2018). Most of these approaches focus on code summarization or comment generation which only require single code-NL pairs for training and evaluation
as the task entails generating a natural language summary of a given code snippet. On the contrary, our proposed task requires two code-NL pairs that are assumed to hold specific parallel relationships with one another. Namely, the relationship between  $\mathbf{C}_{\mathrm{new}}$  and  $\mathbf{M}_{\mathrm{new}}$  is expected to be similar to that of  $\mathbf{C}_{\mathrm{old}}$  and  $\mathbf{M}_{\mathrm{old}}$ . The relationship between  $\mathbf{C}_{\mathrm{new}}$  and  $\mathbf{C}_{\mathrm{old}}$  is expected to correlate with the relationship between  $\mathbf{M}_{\mathrm{new}}$  and  $\mathbf{M}_{\mathrm{old}}$ . Not only does having four moving parts in one example magnify noise, but the need to hold these relationships makes data cleaning particularly difficult. We leave building classifiers for aiding this process as future work.
C Sample Output
In Table 7, we show predictions for various examples in the test set.