Venue: ACL 2017

# A Neural Architecture for Generating Natural Language Descriptions

from Source Code Changes

 Pablo Loyola, Edison Marrese-Taylor and Yutaka Matsuo

Graduate School of Engineering

The University of Tokyo

Tokyo, Japan {pablo,emarrese,matsuo}@weblab.t.u-tokyo.ac.jp

###### Abstract

We propose a model to automatically describe changes introduced in the source code of a program using natural language. Our method receives as input a set of code commits, which contains both the modifications and message introduced by an user. These two modalities are used to train an encoder-decoder architecture. We evaluated our approach on twelve real world open source projects from four different programming languages. Quantitative and qualitative results showed that the proposed approach can generate feasible and semantically sound descriptions not only in standard in-project settings, but also in a cross-project setting.

## 1 Introduction

Source code, while conceived as a set of structured and sequential instructions, inherently reflects human intent: it encodes the way we command a machine to perform a task. In that sense, it is expected that it follows to some extent the same distributional regularities that a proper natural language manifests Hindle et al. (2012). Moreover, the unambiguous nature of source code, comprised in plain and human-readable format, allows an indirect way of communication between developers, a phenomenon boosted in recent years given the current software development paradigm, where billions of lines code are written in a distributed and asynchronous way Gousios et al. (2014).

The scale and complexity of software systems these days has naturally led to explore automated ways to support developers' code comprehension Letovsky (1987) from a linguistic perspective. One of these attempts is automatic summarization, which aims to generate a compact representation of the source code in a portion of natural language Haiduc et al. (2010).

While existing code summarization methods are able to provide relevant insights about the purpose and functional features of the code, their scope is inherently static. In contrast, software development can be seen as a sequence of incremental changes, intended to either generate a new functionality or to repair an existing one. Source code changes are critical for understanding program evolution, which motivated us to explore if it is possible to extend the notion of summarization to encode code changes into natural language representations, i.e., develop a model able to _explain_ a source code level modification. With this, we envision a tool for developers that is able to _i)_ ease the comprehension of the dynamics of the system, which could be useful for debugging and repairing purposes and _ii)_ automate the documentation of source code changes.

To this end, we rely on the concept of code commit, the standard contribution procedure implemented in modern subversion systems Gousios et al. (2014), which provides both the actual change and a short explanatory paragraph. Our model consists of an encoder-decoder architecture which is trained on a set of triples conformed by the version of a system before and after the change, along with the comment. Given the high heterogeneity of the modalities involved, we rely on an attention mechanism to efficiently learn the parts of the sequences that are more expressive and have more explanatory power.

We performed an empirical study on twelve real world software systems, from which we obtained the commit activity to evaluate our model. Our experiments explored in-project and cross-project scenarios, and our results showed that the proposed model is able to generate semantically sound descriptions.



Related Work

The use natural language processing to support software engineering tasks has increased consistently over the years, mainly in terms of source code search, traceability and program feature location [16; 2].

The emergence of unifying paradigms that explicitly relate programming and natural languages in distributional terms [30] and the availability of large corpus mainly from open source software opened the door for the use of language modeling for several tasks [45]. Examples of this are approaches for learning program representations [9], bug localization [8], API suggestion [7] and code completion [45].

Source code summarization has received special attention, ranging from the use of information retrieval techniques to the addition of physiological features such as eye tracking [12]. In recent years several representation learning approaches have been proposed, such as [1], where the authors employ a convolutional architecture embedded inside an attention mechanism to learn an efficient mapping between source code tokens and natural language keywords.

More recently, [13] proposed a encoder-decoder model that learns to summarize from Stackoverflow data, which contains snippet of code along with descriptions. Both approaches share the use of attention mechanisms [1] to overcome the natural disparity between the modalities when finding relevant token alignments. Although we also use an attention mechanism, we differ from them in the sense we are targeting the changes in the code rather than the description of a file.

In terms of specifically working on code change summarization, Cortes-Coy et al. (2014); Linares-Vasquez et al. (2015) propose a method based on a set of rules that considers the type and impact of the changes, and [10] combines summarization with symbolic execution. To the best of our knowledge, our approach represents the first attempt to generate natural language descriptions from code changes without the use of hand-crafted features, a desirable setting given the heterogeneity of the data involved.

## 3 Proposed Model

Our model assumes the existence of \(T\) versions of a given project \(\{v_{1},\ldots,v_{T}\}\). Given a pair of consecutive versions \((v_{t-1},v_{t})\), we define the tuple \((C_{t},N_{t})\), where \(C_{t}=\Delta^{t}_{t-1}(v)\) represents a code snippet associated to changes over \(v\) in time \(t\) and \(N_{t}\) represents its corresponding natural language (NL) description. Let \(\mathbb{C}\) be the set of all source code snippets and \(\mathbb{N}\) be the set of all descriptions in NL. We consider a training corpus with \(T\) code snippets and summary pairs \((C_{t},N_{t})\), \(1\leq t\leq T,\)\(C_{t}\in\mathbb{C}\) , \(N_{t}\in\mathbb{N}\). Then, for a given code snippet \(C_{k}\in\mathbb{C}\), the goal of our model is to produce the most likely NL description \(N^{\star}\).

Concretely, similarly to [13], we use an attention-augmented encoder-decoder architecture. The encoder can be seen as a lookup layer, which simply reads through the source input sequence and returns the embedded tokens. The decoder is a RNN that reads this representation and generates NL words one at a time based on its current hidden state and guided by a global attention model [14]. We model the probability of a description as a product of the conditional next-word probabilities. More formally, for each NL token \(n_{i}\in N_{t}\) we define,

\[h_{i}=f(n_{i-1}E,h_{i-1}) \tag{1}\] \[p(n_{i}|n_{1},...,n_{i-1})\propto W\tanh(W_{1}h_{i}+W_{2}a_{i}) \tag{2}\]

where \(E\) is the embedding matrix for NL tokens, \(\propto\) denotes a softmax operation, \(h_{i}\) represents the hidden state and \(a_{i}\) is the contribution from the attention model on the source code. \(W\), \(W_{1}\) and \(W_{2}\) are trainable combination matrices. The decoder repeats the recurrence until a fixed number of words or a special _END_ token is generated. The attention contribution \(a_{i}\) is defined as \(a_{i}=\sum_{j=1}^{k}\alpha_{i,j}\cdot c_{j}F\), where \(c_{j}\in C_{t}\) is a source code token, \(F\) is the source code token embedding matrix and \(\alpha_{i,j}\) is:

\[\alpha_{i,j}=\frac{\exp{(h_{i}^{\top}c_{j}F)}}{\sum_{c_{j}\in C_{t}}\exp{(h_{ i}^{\top}c_{j}F)}} \tag{3}\]

We use a dropout-regularized LSTM cell for the decoder [10] and also add dropout at the NL embeddings and at the output softmax layer, to prevent over-fitting. We added special _START_ and _END_ tokens to our training sequences and replaced all tokens and output words occurring less than 2 and 3 times, respectively, 

with a special _UNK_ token. We set the maximum code and NL length to be 100 tokens. For decoding, we approximate \(N^{*}\) by performing a beam search on the space of all possible summaries using the model output, with a beam size of 10 and a maximum summary length of 20 words.

To evaluate the quality of our generated descriptions we use both METEOR Lavie and Agarwal (2007) and sentence level BLEU-4 Papineni et al. (2002). Since the training objective does not directly optimize for these scores, we compute METEOR on our validation set after every epoch and save the intermediate model that gives the maximum score as the final model. For evaluation on our test set we used the BLEU-4 score.

## 4 Empirical Study

**Data and pre-processing:** We captured historical data from twelve open source projects hosted on Github based on their popularity and maturity, selecting 3 projects for each of the following languages: _python_, _java_, _javascript_ and _c++_. For each project, we downloaded diff files and metadata of the full commit history. Diff files encode per-line differences between two files or sets of files in a standard format, allowing us to recover source code changes in each commit at the line level. On the other hand, medatada allows us to recover information such as the author and message of each commit.

The extracted commit messages were processed using the Penn Treebank tokenizer Marcus et al. (1993), which nicely deals with punctuation and other text marks. To obtain a source code representation of each commit, we parsed the diff files and used a lexer Brandl (2016) to tokenize their contents in a per-line fashion allowing us to maximize the amount of source code recovered from the diff files. Data and source code available1.

Footnote 1: [http://github.com/epochx/commitgen](http://github.com/epochx/commitgen)

**Experimental Setup:** Given the flat structure of the diff file, source code in contiguous lines might not necessarily correspond to originally neighboring code lines. Moreover, they might come from different files in the project. To deal with this issue, we first worked only with those commits that modify a single file in the project; we call this the _atomicity_ assumption. By using only _atomic_ commits we reduced our training data by an average of roughly 50%, but in exchange we made sure all the extracted code lines came from the same file. At the same time, we expect to maximize the likelihood of observing a direct relation between the commit message and the lines altered.

We then relaxed our _atomicity_ assumption and experimented with the _full_ commit history. Given our maximum sequence length constrain of 100 tokens, we only observed an average of 1,97% extra data on each project. Since source code lines may come from different files, we added a delimiting token _NEW_FILE_ when corresponding.

We were also interested in studying the performance of the model in a cross-project setting. Given the additional challenges that this involves, we designed a more controlled experiment. Starting from the _atomic_ dataset, we selected commits that only add or only remove code lines, conforming a derived dataset that we call _uni-action_. We chose the _python_ language to maximize the available data. See Table 1.

**Results and Discussion:** We begin by training our model on the _atomic_ dataset. As baseline we used MOSES Koehn et al. (2007) which although is designed as a phrase-based machine translation system, was previously used by Iyer et al. (2016) to generate text from source code. Concretely, we treated the tokenized code snippet as the source language and the NL description as the target. We trained a 3-gram language model using KenLM Heafield et al. (2013) and used mGiza to obtain alignments. For validation, we use minimum error rate training Bertoldi et al. (2009); Och (2003) in our validation set.

As Table 3 shows, our model trained on _atomic_ data outperforms the baseline in all but one project with an average gain of 5 BLEU points. In particular, we observe bigger gains for java projects such as _CoreNLP_ and _guava_. We hypothesize this is because program differences in Java tend to be longer than the rest. While this impacts on training time, at the same time it allows the model to

\begin{table}
\begin{tabular}{c c c c c} \hline
**Language** & **Project** & **Full** & **Atomic** & **Added** & **Renn.** \\ \hline  & Theano & 24,200 & 65.40\% & 11.43\% & 2.33\% \\ python & keras & 28,855 & 66.02\% & 11.07\% & 3.01\% \\  & youtube-dl & 13,968 & 74.49\% & 11.52\% & 2.59\% \\ \hline  & node & 15,811 & 53.71\% & 11.87\% & 3.21\% \\ javascript & angular & 6,204 & 32.90\% & 5.59\% & 1.72\% \\  & react & 7,806 & 53.29\% & 12.67\% & 2.72\% \\ \multirow{3}{*}{c++} & opency & 20,480 & 50.68\% & 8.83\% & 1.66\% \\  & CNTK & 10,792 & 38.36\% & 6.00\% & 2.23\% \\  & biconn & 12,596 & 48.11\% & 9.84\% & 2.56\% \\ \cline{2-5}  & CoreNLP & 9,149 & 42.77\% & 7.84\% & 1.98\% \\ java & elasticsearch & 25,764 & 43.77\% & 9.02\% & 2.61\% \\  & gawa & 3,821 & 38.63\% & 8.90\% & 2.64\% \\ \hline  & Average & 12,787 & 50.58\% & 9.55\% & 2.48\% \\ \hline \end{tabular}
\end{table}
Table 1: Summary of our collected data.

 

work with a larger vocabulary space. On the other hand, our model performs similarly to MOSES for the _node_ and slightly worse for the _youtube-dl_. A detailed inspection of the NL messages for _node_ showed that many of them exhibit a fixed pattern in their structure. We believe this rigidity restrains the generation capabilities of the decoder, making it more prone to memorization.

Table 2 shows examples of generated descriptions for real changes and their references. Results suggest that our model is able to generate semantically sound descriptions for the changes. We can also visualize the summarizing power of the model, as seen in the _Theano_ and _bitcoin_ examples. We observe a tendency to choose more general terms over too specific ones meanwhile also avoiding irrelevant words such as numbers or names. Results also suggest the emergence of rephrasing capabilities, specifically in the second example from _Theano_. Finally, our generated descriptions are, in most cases, semantically well correlated to the reference descriptions. We also report not so successful results, such as case of _youtube-dl_, where we can see signs of memorization on the generated descriptions.

Regarding the cross-project setting experiments on _python_, we obtained BLEU scores of 14.6 and 18.9 for only-adding and only-removing instances in the _uni-action_ dataset, respectively. We also obtained validation accuracies up to 43.94%, suggesting feasibility in this more challenging scenario. Moreover, as the generated descriptions from the _keras_ project in Table 2 show, the model is still able to generate semantically sound descriptions.

Despite the small data increase, we also trained our model on _full_ datasets as a way to confirm the generative power of our model. In particular, we wanted to test the model is able leverage on _atomic_ data to also capture and compress multifile changes. As shown in Table 3, results in terms of BLEU and validation accuracy manifest reasonable consistency, despite the higher disparity between source code and natural language on this dataset, which means the model was able to learn representations with more compressive power.

Soft alignments derived from Figure 1, which shows examples of attention heatmaps, illustrate how the model effectively associates source code tokens with meaningful words.

## 5 Conclusion and Future work

We proposed an encoder-decoder model for automatically generating natural descriptions from source code changes. We believe our current results suggest that the idea is feasible and, if improved, could represent a contribution for the understanding of software evolution from a linguistic perspective. As future work, we will consider improving the model by allowing feature learning from richer inputs, such as abstract syntax trees and also functional data, such as execution traces.

\begin{table}
\begin{tabular}{|c|l|l|} \hline  & **Reference** & **Generated** \\ \cline{2-3} \multirow{7}{*}{**Dataset**} & Fix image resizing in preprocessing/image & Fixed image preprocessing . \\ \cline{2-3}  & Fix task flakes & Fix flaky test \\ \cline{2-3}  & fix crash in the new warning message & Better warning message \\ \cline{2-3}  & remove w at used & remove not used code . \\ \hline  & Better error msg & better error message . \\ \cline{2-3}  & Merge pull request 4486 & Update Debian packaging description for new \\  & stacking description for now & bitcoin-clip (Jahnathan Corgan ) & Add i: Minor files to \\  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{2-3}  & Add two untient-related files to & Add : Minor files to \\  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{2-3}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \cline{2-3}  & Add a bunch of verbs which are & Add a bunch of verbs which \\  & more likely to be room than & Are more to be scoomb than \\  & vmod & vmod \\ \cline{2-3}  & Add a brief test for optional & make this test do something \\ \hline  & [ caruchboard] Fix uploader and & [ curchboard] Fix uplooder \\  & upload date extraction & extracting \\  & [ extractor/common] Improve & [ extractor/common] \\  & base and construction & Improve extraction \\  & [ invalcoded] Use & [ common] Use \\  & unicode_literals & unicode_literals \\ \hline  & faked geo compilation & fried compile under linux \\ \cline{2-3}  & remove unused variables in & remove unused variable in \\  & OCL\_PERF\_TEST\_P ( ) & the module \\ \hline \end{tabular}
\end{table}
Table 2: Examples of generated natural language passages v/s original ones taken from the test set.

Figure 1: Heatmaps of attention weights \(\alpha_{i,j}\).

 



## References

* Allamanis et al. (2016) Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for extreme summarization of source code. _arXiv preprint arXiv:1602.03001_ .
* Volume 1_. ACM, New York, NY, USA, ICSE '10, pages 95-104. [https://doi.org/10.1145/1806799.1806817](https://doi.org/10.1145/1806799.1806817).
* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _arXiv preprint arXiv:1409.0473_ .
* Bertoldi et al. (2009) Nicola Bertoldi, Haddow Barry, and Jean-Baptiste Fouet. 2009. Improved minimum error rate training in moses. _The Prague Bulletin of Mathematical Linguistics_ pages 1-11.
* Brandl (2016) Georg Brandl. 2016. Pygments: Python syntax high-lighter. [http://pygments.org](http://pygments.org).
* Buse and Weimer (2010) Raymond P.L. Buse and Westley R. Weimer. 2010. Automatically documenting program changes. In _Proceedings of the IEEE/ACM International Conference on Automated Software Engineering_. ACM, New York, NY, USA, ASE '10, pages 33-42. [https://doi.org/10.1145/1858996.1859005](https://doi.org/10.1145/1858996.1859005).
* Cortes-Coy et al. (2014) Luis Fernando Cortes-Coy, Mario Linares Vasquez, Jairo Aponte, and Denys Poshyvanyk. 2014. On automatically generating commit messages via summarization of source code changes. In _SCAM_. volume 14, pages 275-284.
* Gousios et al. (2014) Georgios Gousios, Martin Pinzger, and Arie van Deursen. 2014. An exploratory study of the pull-based software development model. In _Proceedings of the 36th International Conference on Software Engineering_. ACM, pages 345-355.
* Gu et al. (2016) Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016. Deep api learning. In _Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering_. ACM, New York, NY, USA, FSE 2016, pages 631-642. [https://doi.org/10.1145/2950290.2950334](https://doi.org/10.1145/2950290.2950334).
* Haiduc et al. (2010) Sonia Haiduc, Jairo Aponte, and Andrian Marcus. 2010. Supporting program comprehension with source code summarization. In _Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 2_. ACM, pages 223-226.
* Heafield et al. (2013) Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics_. Sofia, Bulgaria, pages 690-696.
* Hindle et al. (2012) Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. 2012. On the naturalness of software. In _2012 34th International Conference on Software Engineering (ICSE)_. IEEE, pages 837-847.
* Huo et al. (2016) Xuan Huo, Ming Li, and Zhi-Hua Zhou. ??? Learning unified features from natural and programming languages for locating buggy source code .
* Iyer et al. (2016) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, pages 2073-2083. [http://www.aclweb.org/anthology/P16-1195](http://www.aclweb.org/anthology/P16-1195).
* Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_. Association for Computational Linguistics, Prague, Czech Republic, pages 177-180. [http://www.aclweb.org/anthology/P07-2045](http://www.aclweb.org/anthology/P07-2045).
* Lavie and Agarwal (2007) Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In _Proceedings of the Second Workshop on Statistical Machine Translation_. Association for Computational Linguistics, Stroudsburg, PA, USA, StatMT '07, pages 228-231. [http://dl.acm.org/citation.cfm?id=1626355.1626389](http://dl.acm.org/citation.cfm?id=1626355.1626389).
* Letovsky (1987) Stanley Letovsky. 1987. Cognitive processes in program comprehension. _Journal of Systems and software_ 7(4):325-339.
* Linares-Vasquez et al. (2015) Mario Linares-Vasquez, Luis Fernando Cortes-Coy, Jairo Aponte, and Denys Poshyvanyk. 2015. Changescribe: A tool for automatically generating commit messages. In _Proceedings of the 37th International Conference on Software Engineering-Volume 2_. IEEE Press, pages 709-712.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Lisbon, Portugal, pages 1412-1421. [http://aclweb.org/anthology/D15-1166](http://aclweb.org/anthology/D15-1166).
* Marcus et al. (1993) Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatric Santorini. 1993. Building a large annotated corpus of english: The penn treebank. _Computational linguistics_ 19(2):313-330

Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional neural networks over tree structures for programming language processing. In _Proc. AAAI_. AAAI Press, pages 1287-1293.
* Och (2003) Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In _Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Sapporo, Japan, pages 160-167. [https://doi.org/10.3115/1075096.1075117](https://doi.org/10.3115/1075096.1075117).
* Panichella et al. (2013) Annibale Panichella, Bogdan Dit, Rocco Oliveto, Massimiliano Di Penta, Denys Poshyvanyk, and Andrea De Lucia. 2013. How to effectively use topic models for software engineering tasks? an approach based on genetic algorithms. In _Proceedings of the 2013 International Conference on Software Engineering_. IEEE Press, Piscataway, NJ, USA, ICSE '13, pages 522-531. [http://dl.acm.org/citation.cfm?id=2486788.2486857](http://dl.acm.org/citation.cfm?id=2486788.2486857).
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of 40th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pages 311-318. [https://doi.org/10.3115/1073083.1073135](https://doi.org/10.3115/1073083.1073135).
* Raychev et al. (2015) Veselin Raychev, Martin Vechev, and Andreas Krause. 2015. Predicting program properties from big code. In _ACM SIGPLAN Notices_. ACM, volume 50, pages 111-124.
* Raychev et al. (2014) Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In _ACM SIGPLAN Notices_. ACM, volume 49, pages 419-428.
* Rodeghero et al. (2014) Paige Rodeghero, Collin McMillan, Paul W McBurney, Nigel Bosch, and Sidney D'Mello. 2014. Improving automated source code summarization via an eyetracking study of programmers. In _Proceedings of the 36th International Conference on Software Engineering_. ACM, pages 390-401.
* Zaremba et al. (2015) Wojciech Zaremba, Ilya Sutskever, and Vinyals Oriol. 2015. Recurrent neural network regularization. In _Proceedings of the 3rd International Conference on Learning Representations_