Title: THE EFFECTIVENESS OF PRE-TRAINED CODE EMBEDDINGS Anonymous

Abstract: Word embeddings are widely used in machine learning based natural language processing systems. It is common to use pre-trained word embeddings which provide benefits such as reduced training time and improved overall performance. There has been a recent interest in applying natural language processing techniques to programming languages. However, none of this recent work uses pretrained embeddings on code tokens. Using extreme summarization as the downstream task, we show that using pre-trained embeddings on code tokens provides the same benefits as it does to natural languages, achieving: over 1.9x speedup, 5% improvement in test loss, 4% improvement in F1 scores, and resistance to over-fitting. We also show that the choice of language used for the embeddings does not have to match that of the task to achieve these benefits and that even embeddings pre-trained on human languages provide these benefits to programming languages.

1. INTRODUCTION:

One of the initial steps in a machine learning natural language processing (NLP) pipeline is converting the one-hot encoded R V tokens into dense R D representations, with V being the size of the vocabulary, D the embedding dimensions and V << D. This conversion is usually done with a single layer neural network, commonly called an embedding layer.

The parameters of the embedding layer can either be initialized randomly or initialized via "pretrained" parameters obtained from a model such as word2vec (Mikolov et al., 2013b; a) , GloVe (Pennington et al., 2014) or a language model (McCann et al., 2017; Howard & Ruder, 2018; Peters et al., 2018) .

It is common to use pre-trained parameters (most frequently the GloVe embeddings), which act as a form of transfer learning (Mou et al., 2016) similar to that of using pre-trained parameters for the convolutional kernels in a machine learning computer vision task (Huh et al., 2016; Kornblith et al., 2018) . These parameters in the embedding layer are then fine-tuned whilst training on the desired downstream task.

The use of these pre-trained embeddings over random initialization allows machine learning models to: train faster, achieve improved overall performance (Kim, 2014) , increase the stability of their training, and reduce the amount of over-fitting (Mou et al., 2016) .

Recently there has been an increased interest in applying NLP techniques to programming languages and software engineering applications (Vechev & Yahav, 2016; Allamanis et al., 2017a) , the most common of which involves predicting the names of methods or variables using surrounding source code (Raychev et al., 2014; 2015; Allamanis et al., 2015; 2017b; Alon et al., 2018b; a) .

Remarkably, none of this work takes advantage of pre-trained embeddings created on source code. From the example below in table 1, we can see how semantic knowledge (provided by the pre-trained code embeddings) of the method body would help us predict the method name, i.e. knowing how pi and radius are used to calculate an area and how height and width are used to calculate an aspect ratio. This semantic knowledge is available to us as even though computers do not need to understand the semantic meaning of a method or variable name, they are mainly chosen to be understood by other human programmers (Hindle et al., 2012) .

In this paper, we detail experiments using pre-trained code embeddings on the downstream task of predicting a method name from a method body. This task is known as extreme summarization (Allamanis et al., 2016b) as a method name can be thought of as a summary of the method body. Our experiments are focused on answering the following research questions:

1. Do pre-trained code embeddings reduce training time?

2. Do pre-trained code embeddings improve performance?

3. Do pre-trained code embeddings increase stability of training? 4. Do pre-trained code embeddings reduce over-fitting? 5. How does the choice of corpora used for the pre-trained code embeddings affect all of the above?

To answer RQ5, we gather a corpus of C, Java and Python code and train embeddings for each corpus separately, as well as comparing them with embeddings trained on natural language. We then test each of these on the same downstream task, extreme summarization, which is in Java. We also release the pre-trained code embeddings.1 

As far as we are aware, this is the first study on the effectiveness of pre-trained code embeddings applied to an extreme (code) summarization task.

2. MODELS:



2.1. LANGUAGE MODEL:

We train our embeddings using a language model (LM). We choose a LM over the word2vec or GloVe models as LMs have shown to capture long-term dependences (Linzen et al., 2016) and hierarchical relations (Gulordava et al., 2018) . We believe both of these properties are essential for predicting a method name from method body. The long-term dependencies are required due to the average length of the method body over 72 tokens2 . The hierarchical relations are needed due to the way data flows through variables within the method body, starting from the method argument(s) (at the top of the hierarchy) to the return value(s) (at the bottom of the hierarchy).

A language model is a probability distribution over sequences of tokens. Each token, x, is represented by a one-hot vector x ∈ R V , with V being the size of the vocabulary. The probability given to a sequence of tokens x 1 , ..., x T can be calculated as:

p(x 1 , ..., x T ) = T t=1 p(x t |x t-1 , ..., x 1 )

We model this probability distribution with a recurrent neural network trained to predict the next token in a sequence of given tokens. Specifically, we use the AWD-LSTM-LM model (Merity et al., 2017; 2018) due to its performance at modeling natural languages and open source implementation.

3

2.2. CONVOLUTIONAL ATTENTION MODEL:

The extreme summarization task uses the Copy Convolutional Attention Model from Allamanis et al. (2016a) . Briefly, the model takes a series of code tokens from the method body, c, as input and outputs the code tokens that form the method name, m. It generates the method name one token at a time, using a recurrent hidden state, h t , provided by a Gated Recurrent Unit (GRU) (Cho et al., 2014) and a series of convolutional filters over the embeddings of the tokens c, which produce attention (Bahdanau et al., 2014) features, L f eat . It also has a mechanism to directly copy tokens from the body to the output.

This model was chosen as it is the state-of-the-art on the extreme summarization dataset used and provided a clear improvement over baseline models. It also has an open source implementation.4 

3 EXPERIMENTAL SETUP

3.1. EMBEDDING DATASET:

The dataset used for the pre-trained code embeddings was gathered from GitHub. To ensure the quality of the data we only used projects with over 10,000 stars and manually checked each project's suitability, i.e. did not use projects which were tutorials or guides.

After scraping the appropriate projects for each of the three languages (C, Java and Python) we tokenized each, converting each token to lowercase as well as splitting each token into subtokens on camelCase and snake case, e.g. getSurfaceArea becomes get, surface and area. This was done to match the tokenization of the extreme summarization dataset. There are approximately 100 million tokens across 20 million lines of code for each language. Each of the embeddings has their own distinct vocabulary, e.g. not all tokens that appear in the C corpus appear in the Java corpus.

For the natural language embeddings we used the WikiText-103 dataset Merity et al. (2016) , as it contains a comparable 103 million tokens.

The AWD-LSTM-LM model was trained with all default parameters from the open source implementation, with the exception of: the embedding dimension changed to 128 and the hidden dimension changed to 512. The embedding dimension was changed to match that of the original Copy Convolutional Attention Model, and the hidden dimension was changed to fit in GPU memory.

Tokens that were not in the most 150,000 common or did not appear at least 3 times were converted into an <unk> token and the model was trained until the validation loss did not decrease for 5 epochs.

3.3. EXTREME SUMMARIZATION DATASET:

The extreme summarization task dataset is detailed in Allamanis et al. (2016a) . Briefly, it consists of 10 Java projects selected for their quality and diversity in application. For each project, all full Java methods are extracted with the method body used as the input and the method name used as the target. Each project has their own vocabulary, e.g. tokens that appear in one project may not appear in any others.

All tokens are formatted the same as the embedding dataset to ensure maximum vocabulary overlap between each pre-trained embeddings and each Java project.

The Copy Convolutional Attention Model was trained with all default parameters from the open source implementation, and was trained for 25 epochs.

The model was trained on each project separately and was run 5 times on each project for each of the embeddings. The results were then averaged together for each project.

4. RESULTS:

Table 2 shows the rank 1 F1 scores achieved for each of the embeddings. On average, we achieve a 4% relative improvement in F1 scores for each of the embeddings. Figure 1 shows the validation losses achieved on all 10 Java projects. Randomly initialized embeddings are shown in purple, Java embeddings in green, C in blue, Python in red and English in orange.

It can be seen that for most projects the pre-trained embeddings train faster, achieve lower losses, are more stable and over-fit less.

Table 3 shows overlap, speedup and improvement in test loss for each project-embedding combination compared to random embeddings. Overlap is the percentage of tokens in the project vocabulary that also appear in the embedding vocabulary. Speedup is calculated as:

S = N r /N e

N r is the number of epochs taken by the random embedding to reach its best validation loss and N e is the number of epochs taken by a non-random embedding to reach that same validation loss.

Improvement is calculated as: We notice that some projects, particularly the elasticsearch project (figure 1b ), did not achieve any benefits from using the pre-trained embeddings and in fact experienced a slow down when using the C embeddings. To explore this further, we plotted speedup and improvement against overlap in figure 2 . We measure the Pearson correlation coefficient of each, receiving a coefficient of 0.73 for speedup and 0.73 for improvement, a medium to strong positive correlation for each. This implies that using more of the pre-trained embeddings provides more of both the speedup and improvement in test loss benefits.

I = L r /L e L

Table 4 shows the overlap, speedup and improvement averaged over all projects. From this, we can see that we achieve around 1.9x speedup, along with a 5% relative improvement in test loss.

Embedding Intuitively, it would make sense that the Java embeddings give the best results as the summarization task is also in Java. We see this is not the case, and the Java embeddings have the same speedup and improvement as the Python embeddings and only a small speedup improvement over the C embeddings, even though the average overlap using the Java embeddings is higher. Most interesting is the fact that the English embeddings achieve the comparable speedup and performance improvement, even though they have only been trained on human languages.

One potential reason for the similar performance between the embeddings that are trained on programming languages is that even though C, Java and Python are syntactically different, the extreme summarization task does not require much of this syntactic information. Consider the examples in table 5, which are Python versions of the Java examples in table 1. Although the syntax has changed (dynamic typing, no braces or semicolons, etc.) the available semantic information from the method body has not. This would imply that the language of the dataset used to pre-train code embeddings does not matter as much as the quality of the dataset with regards to sensible method and variable names. This is further solidified in the fact that the English embeddings, which have only been trained on human languages, achieved similar performance compared to programming languages. We also look at the amount of over-fitting on each project. From figures 1c, 1d, 1f, 1g and 1i we can see how the random embeddings show a large amount of over-fitting compared to the pre-trained code embeddings. We measure how much a project over-fits as:

O = L b /L f

L b is the best validation loss achieved and L f is the final validation loss achieved. We dub this term the over-fit factor, where an O = 1 would imply the final loss is equal to the lowest loss and thus has not over-fit at all (this could also mean the model is still converging, however from figure 1 we can see all project-embedding combinations converge before 25 epochs).

Table 6 shows the over-fit factors for each project-embedding combination. We can see that the random embeddings show the worst performance on every project. Interestingly, there appears to be no correlation between the overlap and the amount of over-fitting.

Table 7 shows the over-fit factor averaged across all projects for each embedding. Again, the results for each of the pre-trained embeddings are similar, showing that the language of the dataset used to train the embeddings does not have a significant impact on performance. 

5. RELATED WORK:

Language models have been used as a form of transfer learning in natural language processing applications with great success (McCann et al., 2017; Howard & Ruder, 2018; Peters et al., 2018) .

There has also been recent work on the further analysis of language models (Merity et al., 2018; 2017) and how well they assist in transfer learning (Mou et al., 2016) .

The use of probabilistic models for source code originated from Hindle et al. (2012) . From that, work on language models of code began on both the token level (Nguyen et al., 2013; Tu et al., 2014) and syntax level (Maddison & Tarlow, 2014) .

Predicting variable and method names has become a common task for machine learning applications in recent years. Initial work was on the token level (Raychev et al., 2014; 2015; Allamanis et al., 2015) but it is beginning to become more common to represent programs as graphs using their abstract syntax tree (Allamanis et al., 2017b; Alon et al., 2018a; b) .

6. DISCUSSION & CONCLUSIONS:

We refer back to our research questions.

Do pre-trained code embeddings reduce training time? Yes, tables 3 and 4 show we get an average of 1.93x speedup. This is correlated with the amount of overlap between the task vocabulary and the embedding vocabulary, shown in figure 2a .

Do pre-trained code embeddings improve performance? Yes, tables 3 and 4 show we get an average of 5% relative validation loss improvement. Again, this is correlated with the amount of overlap between the vocabularies, shown in figure 2b .

Do pre-trained code embeddings increase stability of training? Although this is difficult to quantify due to how over-fitting interacts with the variance of the validation loss curves, from figures 1a and 1c we can see a clear increase in the variance of the validation loss curves using random embeddings compared to those using pre-trained embeddings.

Do pre-trained code embeddings reduce over-fitting? Yes, tables 6 and 7 show that the random embeddings over-fit more than the pre-trained embeddings on every project. However, this does not seem to have a correlation with the amount of vocabulary overlap and further work is needed to determine the cause of this.

How does the choice of corpora used for the pre-trained code embeddings affect all of the above? Intuitively, it would seem the best pre-trained embeddings would be those that are trained on the same language as that of the downstream task, but this is not the case. We hypothesize through the examples shown in tables 1 and 5 that the differing syntax between the languages is not as important as sensible semantic method and variable names within the dataset. This semantic information is also contained in human languages, which explains why the English embeddings also receive comparable performance.

:

Code and embeddings to be released at a later date.

We use a token to refer to an atomic part of a sequence of code.

https://github.com/salesforce/awd-lstm-lm

http://groups.inf.ed.ac.uk/cup/codeattention/

Figures and Tables:

Figure 1: Validation losses for each of the 10 projects, averaged across 5 runs. Randomly initialized embeddings shown in purple, Java in green, C in blue, Python in red, and English in orange.

(a) Overlap vs. speedup over random embeddings. Correlation coefficient of 0.72. (b) Overlap vs. improvement over random embeddings. Correlation coefficient of 0.73.

Figure 2: Overlap vs. speedup/improvement on all project-embedding combinations. Points are colored: green for Java embeddings, blue for C embeddings, red for Python embeddings and orange for English embeddings.

def get surface area (radius): return 4 * math.pi * radius * radius def get aspect ratio (height, width): return height / width

Examples showing how the semantics of the variable names within a method can be used to reason about the name of the method body

Table content: <table><tr><td>float getSurfaceArea (int radius) {</td></tr><tr><td>return 4 * Math.PI * radius * radius;</td></tr><tr><td>}</td></tr><tr><td>float getAspectRatio (int height, int width) {</td></tr><tr><td>return height / width;</td></tr><tr><td>}</td></tr></table>

Rank 1 F1 scores for each of the embeddings.

Table content: <table><tr><td>Project Name</td><td>Description</td><td colspan="5">Random F1 C F1 Java F1 Python F1 English F1</td></tr><tr><td>cassandra</td><td>Distributed Database</td><td>48.1</td><td>51.7</td><td>50.4</td><td>50.7</td><td>51.1</td></tr><tr><td>elasticsearch</td><td>REST Search Engine</td><td>31.7</td><td>31.5</td><td>31.8</td><td>31.9</td><td>32.1</td></tr><tr><td>gradle</td><td>Build System</td><td>36.3</td><td>39.8</td><td>40.1</td><td>39.6</td><td>39.5</td></tr><tr><td>hadoop-common</td><td>Map-Reduce Framework</td><td>38.4</td><td>41.7</td><td>42.5</td><td>40.9</td><td>41.4</td></tr><tr><td>hibernate-orm</td><td>Object/Relational Mapping</td><td>58.7</td><td>59.8</td><td>59.9</td><td>59.3</td><td>59.9</td></tr><tr><td colspan="2">intellij-community IDE</td><td>33.8</td><td>36.8</td><td>36.1</td><td>35.3</td><td>35.1</td></tr><tr><td>liferay-portal</td><td>Portal Framework</td><td>65.9</td><td>66.1</td><td>68.2</td><td>65.4</td><td>66.4</td></tr><tr><td>presto</td><td>Distributed SQL Query Engine</td><td>46.7</td><td>47.9</td><td>48.3</td><td>47.1</td><td>47.6</td></tr><tr><td>spring-framework</td><td>Application Framework</td><td>36.8</td><td>38.3</td><td>38.6</td><td>38.5</td><td>39.3</td></tr><tr><td>wildfly</td><td>Application Server</td><td>44.7</td><td>45.0</td><td>45.9</td><td>45.2</td><td>45.2</td></tr></table>

r is the test loss achieved using random embeddings and L e is the test loss achieved using a nonrandom embedding. An I of 1.05 would indicate a 5% relative performance improvement in test loss over random embeddings.

Table content: <table><tr><td/><td/><td>C</td><td/><td/><td>Java</td><td/><td/><td>Python</td><td/><td/><td>English</td><td/></tr><tr><td>Project Name</td><td colspan="12">Overlap Speedup Improvement Overlap Speedup Improvement Overlap Speedup Improvement Overlap Speedup Improvement</td></tr><tr><td>cassandra</td><td>0.54</td><td>2.2</td><td>1.11</td><td>0.58</td><td>1.8</td><td>1.09</td><td>0.52</td><td>2.75</td><td>1.1</td><td>0.53</td><td>2.75</td><td>1.11</td></tr><tr><td>elasticsearch</td><td>0.38</td><td>0.43</td><td>0.99</td><td>0.45</td><td>0.75</td><td>1</td><td>0.36</td><td>1.1</td><td>1</td><td>0.38</td><td>1.0</td><td>1.0</td></tr><tr><td>gradle</td><td>0.63</td><td>4.5</td><td>1.13</td><td>0.74</td><td>4.5</td><td>1.14</td><td>0.64</td><td>4.5</td><td>1.14</td><td>0.65</td><td>4.49</td><td>1.14</td></tr><tr><td>hadoop-common</td><td>0.38</td><td>1.14</td><td>1.01</td><td>0.4</td><td>1.6</td><td>1.01</td><td>0.33</td><td>1.6</td><td>1.01</td><td>0.34</td><td>1.95</td><td>1.01</td></tr><tr><td>hibernate-orm</td><td>0.46</td><td>1.6</td><td>1.06</td><td>0.54</td><td>0.8</td><td>1.04</td><td>0.48</td><td>1.33</td><td>1.03</td><td>0.52</td><td>1.51</td><td>1.06</td></tr><tr><td>intellij-community</td><td>0.33</td><td>1.17</td><td>1.05</td><td>0.4</td><td>2.33</td><td>1.05</td><td>0.32</td><td>1.75</td><td>1.04</td><td>0.36</td><td>1.41</td><td>1.07</td></tr><tr><td>liferay-portal</td><td>0.37</td><td>0.75</td><td>1.01</td><td>0.46</td><td>1.2</td><td>1.04</td><td>0.37</td><td>1.0</td><td>1.02</td><td>0.39</td><td>1.50</td><td>1.03</td></tr><tr><td>presto</td><td>0.54</td><td>2.0</td><td>1.05</td><td>0.62</td><td>1.71</td><td>1.04</td><td>0.54</td><td>2.0</td><td>1.04</td><td>0.54</td><td>2.41</td><td>1.05</td></tr><tr><td>spring-framework</td><td>0.36</td><td>1.5</td><td>1.05</td><td>0.46</td><td>2.0</td><td>1.04</td><td>0.35</td><td>1.5</td><td>1.05</td><td>0.37</td><td>1.5</td><td>1.05</td></tr><tr><td>wildfly</td><td>0.5</td><td>2.8</td><td>1.05</td><td>0.56</td><td>2.8</td><td>1.05</td><td>0.48</td><td>2.0</td><td>1.06</td><td>0.51</td><td>2.71</td><td>1.06</td></tr></table>



Table content: <table/>

Results relative to random embeddings for each of the pre-trained embeddings, averaged across all 10 projects.

Table content: <table><tr><td/><td colspan="3">Overlap Speedup Improvement</td></tr><tr><td>C</td><td>0.48</td><td>1.84</td><td>1.05</td></tr><tr><td>Java</td><td>0.54</td><td>1.98</td><td>1.05</td></tr><tr><td>Python</td><td>0.48</td><td>1.98</td><td>1.05</td></tr><tr><td>English</td><td>0.49</td><td>1.98</td><td>1.05</td></tr></table>

Python versions of the Java examples from table 1

Table content: <table/>

Over-fit factor for each project-embedding combination. Higher is better.

Table content: <table><tr><td>Project</td><td colspan="5">Random Over-fit C Over-fit Java Over-fit Python Over-fit English Over-fit</td></tr><tr><td>cassandra</td><td>0.87</td><td>0.99</td><td>0.98</td><td>0.98</td><td>0.98</td></tr><tr><td>elasticsearch</td><td>0.91</td><td>0.97</td><td>0.96</td><td>0.95</td><td>0.96</td></tr><tr><td>gradle</td><td>0.74</td><td>0.97</td><td>0.97</td><td>0.97</td><td>0.97</td></tr><tr><td>hadoop-common</td><td>0.8</td><td>0.94</td><td>0.95</td><td>0.95</td><td>0.96</td></tr><tr><td>hibernate-orm</td><td>0.8</td><td>0.95</td><td>0.98</td><td>0.99</td><td>0.98</td></tr><tr><td>intellij-community</td><td>0.67</td><td>0.85</td><td>0.87</td><td>0.91</td><td>0.88</td></tr><tr><td>liferay-portal</td><td>0.8</td><td>0.94</td><td>0.9</td><td>0.94</td><td>0.92</td></tr><tr><td>presto</td><td>0.93</td><td>0.98</td><td>0.98</td><td>0.98</td><td>0.98</td></tr><tr><td>spring-framework</td><td>0.67</td><td>0.87</td><td>0.88</td><td>0.9</td><td>0.86</td></tr><tr><td>wildfly</td><td>0.95</td><td>0.97</td><td>0.98</td><td>0.97</td><td>0.98</td></tr><tr><td/><td colspan="3">Embedding Over-fit</td><td/><td/></tr><tr><td/><td colspan="2">Random</td><td>0.81</td><td/><td/></tr><tr><td/><td>C</td><td/><td>0.94</td><td/><td/></tr><tr><td/><td colspan="2">Java</td><td>0.95</td><td/><td/></tr><tr><td/><td colspan="2">Python</td><td>0.95</td><td/><td/></tr><tr><td/><td colspan="2">English</td><td>0.95</td><td/><td/></tr></table>

Over-fit factor for each embedding averaged across all 10 projects. Higher is better.

Table content: <table/>

Peer review (iclr2019):
 THE EFFECTIVENESS OF PRE-TRAINED CODE EMBEDDINGS Summary: This work shows how pre-training word vectors using corpuses of code leads to representations that are more suitable than randomly initialized (and trained) representations for function/method name prediction (here called extreme summarization). This paper applies a standard language model to several collected corpuses of code written in different programming languages in order to pretrain the embeddings. It then uses a standard model for the extreme summarization task that takes pre-trained language model embeddings. This leads to speedups in training, improvements in validation loss, and less overfitting. I worry that these results didn*t really need much proving given that we have already seen the exact same methods work with natural languages. It would actually be more surprising if they didn*t work for programming languages, which suggests that the real question is whether code embeddings are actually more effective than natural language embeddings for this problem given that the authors show syntax of the code is far less important than the semantics of the words in the vocabulary. It is not clear that embeddings pre-trained with much more data on natural languages wouldn*t work just as well. Pros: This paper takes the time to clearly explain the effectiveness of pre-training words vectors in a new setting. It is easy to follow and understand thanks to the clear organization and exposition. Speedup and validation loss improvements are demonstrated for a variety of programming languages despite the final evaluation being only in Java, which is quite surprising. The authors discover that overlap in actual programming language syntax is less important than overlap in semantic representation. Because the models are out-of-the-box, it is easy to focus on the actual contributions of this paper related to pre-training. Does seem to clearly demonstrate that pre-trained embeddings of some form should be used for this extreme summarization task. Cons: It is not clear how big the collected corpus is. This is important because the work that this paper cites on pre-trained embeddings (Mikolov et al 2013, Pennington et al 2014, McCann et al 2017, Peters et al 2018) typically use fairly large datasets for pre-training. All of these results might be watered down by insufficient pre-training data for the language model when in fact the results could be much stronger with more data. It would be nice to show the effects of pre-training dataset size as is done in the aforementioned previous works. Without this comparison, it is hard to tell whether the paper sufficiently explores this idea. The models are both standard, out-of-the-box models. There is no novelty on the modeling side of this paper. The pre-training methods are also not novel. They are methods that have already been shown to work applied in a slightly different setting. It is not clear that the setting is actually different enough to require this pre-training. Comparing to randomly initialized embeddings is fine, but I would also like to see a comparison to other pre-trained embeddings like GloVe, GloVe+CoVe, or ELMo (Pennington et al 2014, McCann et al 2017, Peters et al 2018). Since the authors find that it is the semantics of the words that matter more than the syntax of any particular programming language, then perhaps it would actually be better to use pre-trained embeddings that tap into much larger amounts of data. At the very least, it seems it would make sense to perhaps supplement a standard pre-trained embedding with those suggested by the authors since so many of the words in the code must be English words. If this is too farfetch*d, then I would suggest that the authors provide some statistics showing why GloVe, GloVe+CoVe, and ELMo are not appropriate starting points for comparison, but the overlap from the pre-training corpuses is already so low that it seems supplementing with standard pre-trained embeddings should only help. The evaluation dataset detailed in Allamanis et al 2016 uses two metrics: an F1 metric and an exact match metric. This paper only compares on validation loss. What*s more, it reports everything in relative terms so that the raw improvement is masked until Figure 1 makes it somewhat possible to deduce. The problem here is that we don*t know how a 0.0-0.5 raw improvement in validation loss translates to the metrics established for the dataset by Allamanis et al 2016. If those are no longer the standard metrics, the authors should explain how validation loss came to supplant the original metrics proposed by Allamanis et al 2016. What*s more, there is no context for how well models typically do on this evaluation task. Without any comparisons it is impossible to tell whether any of the experiments are using models in a reasonable realm of performance on this task. Overall: All these effects have already been shown for pre-trained embeddings in the past, and the experiments involve running standard methods on newly collected datasets. This means there is no novelty in the pre-training method or the extreme summarization method. Little is known about the newly collected datasets, it is not clear how to interpret the relative improvements in validation loss compared to the original metrics of Allamanis et al 2016, and the paper lacks necessary comparisons to othef pre-trained embeddings, so though the overall claim that pre-trained embeddings should be used for this task seems to hold up, it is not clear that this is a complete argument for the method chosen by the authors.