SlowGuess/NAACL_2021_OCR huggingface dataset:
Nadezhda Chirkova  
HSE University*  
Moscow, Russia
Source code processing heavily relies on the methods widely used in natural language processing (NLP), but involves specifics that need to be taken into account to achieve higher quality. An example of this specificity is that the semantics of a variable is defined not only by its name but also by the contexts in which the variable occurs. In this work, we develop dynamic embeddings, a recurrent mechanism that adjusts the learned semantics of the variable when it obtains more information about the variable's role in the program. We show that using the proposed dynamic embeddings significantly improves the performance of the recurrent neural network, in code completion and bug fixing tasks.
Deep learning is now actively being deployed in source code processing (SCP) for solving such tasks as code completion (Li et al., 2018), generating code comments (Alon et al., 2019), and fixing errors in code (Vasic et al., 2019a). Source code visually looks like a text, motivating the wide use of NLP architectures in SCP. A lot of modern SCP approaches are based on recurrent neural networks (Le et al., 2020), other popular architectures are transformers, and convolutional and graph neural networks.
Utilizing the specifics of source code as a data domain may potentially improve the quality of neural networks for SCP. These specifics include three main aspects. Firstly, the source code is strictly structured, i.e. the source code follows the syntactic rules of the programming language. Secondly, the vocabularies may be large or even potentially unlimited, i.e. a programmer is allowed to define the identifiers of the arbitrary complexity. Thirdly, the identifiers are invariant to renaming, i.e. renaming all the user-defined identifiers
does not change the algorithm that the code snippet implements. The first two mentioned specifics have been extensively investigated in the literature. For example, the tree-based architectures, such as TreeLSTM (Chen et al., 2018) or TreeTransformer (Shiv and Quirk, 2019) allow for the utilization of the code's structure. On the other hand, using byte-pair encoding (Karampatsis et al., 2020; Sennrich et al., 2016) or the anonymization of out-of-vocabulary identifiers (Chirkova and Troshin, 2021) deals with the unlimited vocabulary problem. However, the property of source code being invariant to renaming user-defined identifiers has not been paid much attention to. In this work, we aim to close this gap for the recurrent neural networks (RNNs).
Let us take a closer look at the invariance property. In Fig. 1 (a) and (b), a code snippet implementing a simple mathematical calculation is presented with two different variable naming schemes. Both code snippets implement the same algorithm, i.e. are equivalent in the "program semantics" space, but have different text representations. Classic NLP approach implies using the embedding layer as the first layer in the network where learnable embeddings encode the global semantics of the input tokens. In example (a), the embeddings of variables  $\mathbf{x}$  and  $\mathbf{y}$  make sense, as these variables are usually used in mathematical calculations, but in example (b), the embeddings of variables  $\mathsf{foo}$  and  $\mathsf{foo2}$  do not reflect any semantics. Moreover, even in case (a), the semantics of identifier  $\mathbf{y}$  reflected by its embedding are too broad, i.e. this identifier could be used in a lot of different calculations, while variable  $\mathbf{y}$  has a much more specific role in the program, i.e. storing the result of the particular function. The key idea of this work is that the embedding of a variable in the program should reflect the variable's particular role in this program and not only its name. The name of the variable may act as the secondary
* The work was done while working at Samsung-HSE Laboratory, HSE University.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2679-2689 June 6-11, 2021. ©2021 Association for Computational Linguistics
source of information about the variable's role, but the main source of this information is the program itself, i.e. the contexts the variable is used in.
We develop the recurrent mechanism called dynamic embeddings that captures the representations of the variables in the program based on the contexts in which these variables are used. Being initialized before a program processing, the dynamic embedding of a variable is updated each time the variable has been used in the program, see the scheme in Fig. 1(g). We test the dynamic embedding approach in two settings: the standard setting with the full data, and the anonymized setting, when all variable names are replaced with unique placeholders var1, var2, var3 etc. In the full data setting, we initialize the dynamic embeddings with standard embeddings, see Fig. 1(e), to implement the idea of the variable name being a secondary source of information about the variable's semantics. In the anonymized setting, we initialize the dynamic embeddings using a constant initial embedding, the same for all identifiers, see Fig. 1(f). In this setting, the variable names are not used at all, and the model detects the role of the variable purely based on the contexts in which the variable is used in the program. Although being less practically oriented, the anonymized setting is a conceptually interesting benchmark, as it highlights the capabilities of deep learning architectures to understand the pure program structure, that is actually the main goal of SCP, without relying on the unstructured textual information contained in variable names. In addition, the anonymized setting could be the case in practice, e.g. when processing the decompiled or obfuscated code (Lacomis et al., 2019).
In the experiments, we show that using the proposed dynamic embeddings significantly outperforms the model that uses the standard embeddings, called static embeddings in our work, in both described settings in two SCP tasks, namely code completion and bug fixing.
- We propose the dynamic embeddings for capturing the semantics of the variable names in source code;
- To demonstrate the wide practical applicability of the proposed dynamic embeddings, we show that they outperform static embeddings in two different code processing tasks, namely
code completion (generative task) and bug fixing (discriminative task), in the full data setting;
- We propose the version of the dynamic embeddings approach that does not use variable names at all, and show that it achieves high results in both tasks, sometimes even outperforming the standard model trained on full data (with variable names present in the data).
Our source code is available at https://github.com/nadiinchi/dynamicembeddings.
The possibility of improving deep learning models of source code by taking into account the invariance property of variable names has been superficially discussed in the literature. Ahmed et al. (2018) replace variables with their types, while Gupta et al. (2017) and Xu et al. (2019) use the static embeddings for anonymized variables. However, the existing SCP work did not consider developing a special architecture that dynamically updates the embeddings of the variables during processing a code snippet.
Our research is also related to the field of processing out-of-vocabulary (OOV) variable names. The commonly used approaches for dealing with OOV variables are using the pointer mechanism (Li et al., 2018) or replacing OOV variables with their types (Hu et al., 2018). As we show in our work, both methods may be successfully combined with the proposed dynamic embeddings.
In the context of NLP, Kobayashi et al. (2017) use a similar model with dynamic embeddings to process OOV and anonymized named entities in natural text. In contrast to their approach, we apply dynamic embeddings to the whole vocabulary of variable names, and incorporate dynamic embeddings into the model that relies on the syntactic structure of code. This results in more meaningful dynamic embeddings.
We firstly describe what format of the model input we use, i.e. the procedure of code preprocessing, and then describe our model. At the end of this section, we discuss how we use the proposed model in two code processing tasks.
Input code snippet:
(a)  $\mathbf{y} = \mathbf{x}^{**}\mathbf{x} - \mathbf{x}$
(b) foo2 = foo**foo - foo
(c) Abstract syntax tree (AST):
1	Assign	<empty>
2	NameStore	y
3	BinOpSub	<empty>
4	BinOpPow	<empty>
5	NameLoad	x
6	NameLoad	x
7	NameLoad	x
(d) AST depth-first traversal:
(h) One timestep processing:
Figure 1: The overview of the proposed approach. (a) and (b): two variants of the input code snippet, variant (a) is used in other illustration blocks; (c) abstract syntax tree (AST); (d) AST converted to a sequence that will be passed to the RNN; (e) the static-embedding-based initialization of dynamic embeddings; (f) the constant initialization of dynamic embeddings; (h) the scheme of updating dynamic embeddings and hidden states; (g) the scheme of one timestep processing. SE: static embedding, DE: dynamic embedding.
(e)  $\begin{array}{l}\mathbf{DE}_{\mathbf{x}} = \mathbf{SE}_{\mathbf{x}}\\ \mathbf{DE}_{\mathbf{y}} = \mathbf{SE}_{\mathbf{y}} \end{array}$
or (f)  $\begin{array}{r}\mathbf{D}\mathbf{E}\mathbf{x} = \mathbf{D}\mathbf{E}_{\mathrm{initial}}\\ \mathbf{D}\mathbf{E}\mathbf{y} = \mathbf{D}\mathbf{E}_{\mathrm{initial}} \end{array}$
$$
\mathsf {H} = \mathsf {H} _ {\text {i n i t i a l}}
$$
(g) Code
processing:
1.  
2.  $DE_{\mathbf{y}}^{new} = LSTM_{\mathsf{dyn}}(\mathsf{H}, \mathsf{SE}_{\mathsf{NameStore}}; DE_{\mathbf{y}})$
3.
4.  
5.  $\mathsf{D}\mathsf{E}_{\mathbf{x}}^{\mathsf{new}} = \mathsf{L}\mathsf{S}\mathsf{T}\mathsf{M}_{\mathsf{dyn}}(\mathsf{H},\mathsf{S}\mathsf{E}_{\mathsf{NameLoad}};\mathsf{D}\mathsf{E}_{\mathbf{x}})$
6.  $\mathsf{D}\mathsf{E}_{\mathsf{X}}^{\mathsf{new}} = \mathsf{L}\mathsf{S}\mathsf{T}\mathsf{M}_{\mathsf{dyn}}(\mathsf{H},\mathsf{S}\mathsf{E}_{\mathsf{NameLoad}};\mathsf{D}\mathsf{E}_{\mathsf{X}})$
7.  $\mathsf{D}\mathsf{E}_{\mathsf{X}}^{\mathsf{new}} = \mathsf{L}\mathsf{S}\mathsf{T}\mathsf{M}_{\mathsf{dyn}}(\mathsf{H},\mathsf{S}\mathsf{E}_{\mathsf{NameLoad}};\mathsf{D}\mathsf{E}_{\mathsf{X}})$
H  $\equiv$  LSTMmain (SE<empty>, SEAssign; H)
H  $\equiv$  LSTMmain (DEy, SENameStore; H)
new H = LSTMmain (SE<empty>, SEBinOpSub; H)
H'ew LSTMmain (SE<empty>, SEBinOpPow; H)
H  $\equiv$  LSTMmain (DEx, SENameLoad; H)
H  $\equiv$  LSTMmain (DEx, SENameLoad; H)
H = LSTMmain (DEX, SENameLoad; H)
3.1 Code preprocessing
To capture the syntactic structure of an input code snippet, we convert it to an abstract syntax tree (AST), see Fig. 1(c) for the illustration. In order to process the code snippet with an RNN, we need to convert the AST into a sequence. We use the most popular approach that implies traversing the AST in the depth-first order (Li et al., 2018), see Fig. 1(d). Recent research shows that using the AST traversal may be even more effective than using specific tree-based architectures (Chirkova and Troshin, 2020).
Each node in the AST contains a type, reflecting the syntactic unit, e.g. If or NameLoad. Some nodes also contain values, e.g. a user-defined variable or a constant. We insert the <EMPTY> value in the nodes that do not have values so that the input snippet is represented as a sequence of (type, value) pairs:  $I = [(t_1, v_1), \ldots, (t_L, v_L)]$ . Here  $L$  denotes the length of the sequence,  $t_i \in T$  denotes the type and  $v_i \in V$  denotes the value. The size of the type vocabulary  $T$  is small and determined by the programming language, while the size of the value vocabulary  $V$  may be potentially large, as it contains a lot of user-defined identifiers. Given
sequence  $I$ , the RNN outputs a sequence of hidden states  $[h_1, \ldots, h_L]$ ,  $h_i \in \mathbb{R}^{d_{\mathrm{hid}}}$ ,  $i = 1, \ldots, L$ . These hidden states are used to output the task-specific prediction as described in Section 3.3.
3.2 Dynamic embeddings
We use the standard baseline recurrent architecture that initializes the hidden state with a learnable predefined vector  $h_{\mathrm{init}} \in \mathbb{R}^{h_{\mathrm{hid}}}$ :  $h_0 = h_{\mathrm{init}}$ , and then updates the hidden state at each timestep  $i = 1, \ldots, L$ :
$$
h _ {i} = \mathrm {L S T M} _ {\text {m a i n}} \left(e _ {v _ {i}}, e _ {t _ {i}}; h _ {i - 1}\right).
$$
Here,  $e_{v_i} \in \mathbb{R}^{d_{\mathrm{val}}}$  and  $e_{t_i} \in \mathbb{R}^{d_{\mathrm{type}}}$  denote the embeddings of the value and the type correspondingly. Without loss of generality, we use the Long Short-Term Memory recurrent unit (LSTM) (Hochreiter and Schmidhuber, 1997). In this work, we replace value embeddings  $e_{v_i}$  with dynamic embeddings described below.
Dynamic embeddings. The general idea of dynamic embeddings is that the variable's embedding is updated in the RNN-like manner after each occurrence of the variable. We first describe the updating
procedure and then discuss the initialization strategy. Since the dynamic embeddings change over timesteps, we use notation  $e_{v,i}$  for the dynamic embeddings of value  $v$  at timestep  $i$ . For example, for the value located at the  $i$ -th position in the input sequence,  $v_i$ , the dynamic embedding after processing the  $i$ -th step is denoted as  $e_{v_i,i}$ , and its previous state is denoted as  $e_{v_i,i-1}$ . At each timestep  $i = 1, \ldots, L$ , we update the dynamic embedding  $e_{v_i,i}$  of the current value  $v_i$  and hidden state  $h_i$  using two LSTMs:
$$
e _ {v _ {i}, i} = \mathrm {L S T M} _ {\mathrm {d y n}} (h _ {i - 1}, e _ {t _ {i}}; e _ {v _ {i}, i - 1}) \tag {1}
$$
$$
e _ {v, i} = e _ {v, i - 1}, \quad v \neq v _ {i} \tag {2}
$$
$$
h _ {i} = \operatorname {L S T M} _ {\text {m a i n}} \left(e _ {v _ {i}, i - 1}, e _ {t _ {i}}; h _ {i - 1}\right) \tag {3}
$$
An illustration of this update procedure is given in Fig. 1(h), and the example scheme of processing a code snippet is given in Fig. 1(g).  $\mathrm{LSTM}_{\mathrm{main}}$  implements the recurrence over the hidden state, while  $\mathrm{LSTM}_{\mathrm{dyn}}$  implements the recurrence over dynamic embeddings, and the same  $\mathrm{LSTM}_{\mathrm{dyn}}$  is used to update the dynamic embeddings of different values at different timesteps. We note that at timestep  $i$ , the dynamic embedding of only current value  $v_{i}$  is updated, while the dynamic embeddings of other values do not change, as stated in Eq. (2).
In practice, several dummy values, e.g. <EMPTY>, <UNK> and <EOF>, do not change their roles in different sequences. We use static embeddings for these values.
Initializing dynamic embeddings. The most reasonable strategy for initializing the dynamic embeddings is to use static embeddings:  $e_{v,0} = e_v$  where  $e_v$  are the learnable embedding vectors of all values  $v$  in vocabulary  $V$ . In this case, the model utilizes all the available information about the variable: the variable's name introduced by the programmer that is supposed to somehow reflect the mission of the variable, and the contexts in which the variable occurs (captured by hidden states). In other words, the model firstly "understands" the loose role of the variable from its name and then "finetunes" this understanding, while learning more about what the variable is used for.
Another possible strategy is to ignore all the variable names and initialize all dynamic embeddings with a constant embedding:  $e_{v,0} = e_{\mathrm{init}}$ ,
$e_{\mathrm{init}} \in \mathbb{R}^{d_{\mathrm{val}}}$ ,  $v \in V$ . Although the initial embeddings of all values are the same, they will be updated differently, as different values occur in different locations in the program, and the dynamic embeddings will characterize these locations. Interestingly, the described strategy ensures that if we rename all the variables in the program, the output of the RNN does not change. Such a behaviour is consistent with the variable invariance property: renaming all user-defined variables does not change the underlying algorithm. The common sense is that the architecture for processing source code should be consistent with the variable invariance property, and dynamic embeddings with the constant initial embedding fulfill this conceptual requirement. On the other hand, commonly-used static embeddings are not consistent with the invariance property, i.e. renaming variables scheme results in using different embeddings and changes the predictions of the RNN.
As will be shown below, in practice, using both sources of information, namely variable names and variable occurrences in the program, performs better than relying on only one source of information. In other words, dynamic embeddings with static embedding initialization outperform both static embeddings in the full data setting (relying only on variable names) and dynamic embeddings with constant initialization (relying only on variable occurrences).
3.3 Task-specific prediction
We test the proposed dynamic embeddings in two SCP tasks: code completion and variable misuse prediction. Below, we describe how we make predictions in these tasks, using the output  $[h_1,\dots ,h_L]$  of the RNN.
Code completion. In code completion, the task is to predict the next type-value pair  $(t_{i+1}, v_{i+1})$  given prefix  $[(t_1, v_1), \ldots, (t_1, v_i)]$  at each timestep  $i = 1, \ldots, L$ . In our work, we focus on value prediction, as type prediction is a simple task usually solved with high quality in practice (Li et al., 2018). We rely on the setup and the architecture of Li et al. (2018).
To predict the next value  $v_{i+1}$  based on  $[h_1, \ldots, h_i]$ , we firstly apply the standard attention mechanism (Bahdanau et al., 2015), obtaining the context vector  $c_i = \sum_{j=1}^i \alpha_j h_j$ ,  $\alpha_j$  denote attention weights, and then combine all available
representations using a fully-connected layer:
$$
\hat {h} _ {i} = W ^ {1} h _ {i} + W ^ {2} c _ {i} + W ^ {3} h _ {\mathrm {p a r e n t}},
$$
where  $h_{\mathrm{parent}}$  is the hidden state of the parent node. For computing the logit  $y_{v,i} \in \mathbb{R}$  of each value  $v$ , we reuse the dynamic embeddings  $e_{v,i}$  of the input layer, as well as the static embeddings of several dummy values:  $y_{v,i} = e_{v,i}^{T}\hat{h}_{i}$ , and apply Softmax on top of  $y_{v,i}$  to predict the probability distribution  $P_{i}^{vals} \in \mathbb{R}^{|V|}$  over next value  $v$ . Finally, we use the pointer mechanism to improve the prediction of rare values. We reuse attention scores  $[\alpha_1, \ldots, \alpha_i]$ ,  $\sum_{j=1}^{i} \alpha_j = 1$ ,  $\alpha_j \geqslant 0$  as a distribution over previous positions  $P_{i}^{pos} \in \mathbb{R}^i$ , and use switcher  $s = \sigma(w^{swit,1}h_{i} + w^{swit,2}c_{i}) \in (0,1)$  to gather two distributions into one:  $R_{i} = [sP_{i}^{vals}, (1 - s)P_{i}^{pos}]$ . To make the prediction, we select the largest element of vector  $R_{i}$ ; if it corresponds to the value from the vocabulary, we output this value, if it corresponds to the position, we copy the value from that position. To train the model, we optimize the cross-entropy loss, using as ground truth the values in the vocabulary for in-vocabulary values and the last occurrence of the value (if any) for out-of-vocabulary values.
Variable misuse prediction. The variable misuse task implies outputting two pointers: the first one points to the location  $i$  in which the wrong value  $v_{i}$  is used and the second one points to the location  $j$  that can be used to repair the bug by copying its value  $v_{j}$ . If there is no bug, the first pointer selects a special no-bug location. In this task, we rely on the approach of Vasic et al. (2019b) and its implementation of Hellendoorn et al. (2020). In addition, we change the format of the model input and use the depth-first AST traversal (Li et al., 2018).
We use the bidirectional LSTM, with each of the two LSTMs being equipped with its own dynamic embeddings. As a result, we have two sequences of hidden states:  $[h_1^{\mathrm{fw}},\dots,h_L^{\mathrm{fw}}]$  and  $[h_1^{\mathrm{bw}},\dots,h_L^{\mathrm{bw}}]$ . To make the prediction, we firstly combine two representations using a fully-connected layer:
$$
h _ {i} = \tanh (W ^ {1} h _ {i} ^ {\mathrm {f w}} + W ^ {2} h _ {i} ^ {\mathrm {b w}}), i = 1, \ldots , L
$$
and then use two other fully-connected layers to obtain logits  $y_{i}^{\mathrm{bug}} \in \mathbb{R}$  and  $y_{i}^{\mathrm{fix}} \in \mathbb{R}$  of each position  $i$ :  $y_{i}^{\mathrm{bug}} = (w^{\mathrm{bug}})^{T}h_{i}$ ,  $y_{i}^{\mathrm{fix}} = (w^{\mathrm{fix}})^{T}h_{i}$ . Finally, we apply Softmax over  $[y_1^{\mathrm{bug}},\dots ,y_L^{\mathrm{bug}},y^{\mathrm{nobug}}]$  and
over  $[y_i^{\mathrm{fix}}]_{i = 1}^L$  to obtain two distributions over positions. Here, learnable  $y^{\mathrm{nobug}}\in \mathbb{R}$  corresponds to a no-bug position. The model is trained using the cross-entropy loss.
Data and preprocessing. We conduct experiments on Python150k (Raychev et al., 2016a) and JavaScript150k (Raychev et al., 2016b) datasets. Both datasets are commonly used in SCP and were obtained by downloading repositories from GitHub. However, the train / test split released by the authors of the dataset does not follow the best practices of splitting data in SCP (Allamanis, 2019; LeClair and McMillan, 2019), so we use another train / test split released by Chirkova and Troshin (2020). This split is based on the repositories, i.e. all files from one repository go either to train or test, and was deduplicated using the tools provided by Allamanis (2019), i.e. code files in the test set that are duplicated in the train set were removed; this is a common case in source code downloaded from GitHub. In addition, the Python dataset includes only redistributable code (Kanade et al., 2020). Splitting by repository and deduplication are highly important in SCP to avoid a percentage of testing accuracy being provided by the examples the model saw during training. With the described new split, the results in our tables are not directly comparable to the results reported in other works. To validate our implementation, we compared the quality of baseline models trained in our implementation with the quality reported in the papers describing these baselines, and observed that the numbers are close to each other (see details in Section 5.4).
For the code completion task, we use the entire code files as training objects, filtering out exceptionally long files, i.e. files longer than  $3 \cdot 10^{4}$  characters. The resulting training / testing set consists of  $76\mathrm{K} / 39\mathrm{K}$  files for Python and of  $69\mathrm{K} / 41\mathrm{K}$  for JavaScript. The mean length of the code files in  $567 / 669$  AST nodes for Python / JavaScript.
For the variable misuse task, we select all top-level functions, including functions inside classes from all files, and filter out functions longer than 250 AST nodes, and functions with fewer than three positions containing user-defined variables or less than three distinct user-defined variables. The resulting training / testing set consists of  $417\mathrm{K} / 231\mathrm{K}$  functions for Python and  $202\mathrm{K} / 108\mathrm{K}$
functions for JavaScript. One function may occur in the dataset up to 6 times: 3 times with a synthetically generated bug and 3 times without bug. The buggy examples are generated synthetically by choosing random bug and positions from positions containing user-defined variables. The described strategy for injecting synthetic bugs is the same as in (Hellendoorn et al., 2020).
In both tasks, the size of the node type vocabulary is  $330 / 44$  for Python / JavaScript, the vocabulary of node values is limited to  $50\mathrm{K}$  of the most frequent values.
Metrics. Following Li et al. (2018), we use accuracy to measure model quality in the code completion task, counting all predictions of <UNK> as wrong. Following Hellendoorn et al. (2020), to measure the quality in the variable misuse task, we use the joint localization and repair accuracy (what portion of buggy values is correctly located and fixed).