SlowGuess/NAACL_2021_OCR huggingface dataset:
Nadezhda Chirkova* and Sergey Troshin*
{nchirkova, stroshin}@hse.ru
There is an emerging interest in the application of natural language processing models to source code processing tasks. One of the major problems in applying deep learning to software engineering is that source code often contains a lot of rare identifiers, resulting in huge vocabularies. We propose a simple, yet effective method, based on identifier anonymization, to handle out-of-vocabulary (OOV) identifiers. Our method can be treated as a preprocessing step and, therefore, allows for easy implementation. We show that the proposed OOV anonymization method significantly improves the performance of the Transformer in two code processing tasks: code completion and bug fixing.
Natural language processing (NLP) is widely used for source code processing (SCP), e.g. for learning the meaningful vector representations of code (Feng et al., 2020; Alon et al., 2019b; Azcona et al., 2019), that can be used in various downstream tasks, e.g. code summarization (Iyer et al., 2016; Shiv and Quirk, 2019), code completion (Kim et al., 2020), or bug fixing (Hellendoorn et al., 2020).
An important question, one should answer before building an SCP model, is how to create a vocabulary? Karampatsis et al. (2020) underline that modern source code datasets may incorporate millions of unique identifiers, of which less than  $1\%$  occur in the dataset frequently, e.g. more than 5 times. The common practice is to crop the vocabulary based on top-N identifiers and replace all occurrences of out-of-vocabulary (OOV) identifiers with an UNK identifier to avoid huge embedding matrices and the meaningless embeddings of rare tokens. But can one process rare identifiers in a better way?
Vocabulary: {np, sin}
Input: my_y = np.sin(my_x) + my_x
Standard OOV processing procedure:
UNK = np.sin(UNK) + UNK
Proposed OOV anonymization procedure:
VAR1 = np.sin(VAR2) + VAR2
Figure 1: Illustration of the proposed OOV anonymization procedure. Out-of-vocabulary identifiers my_y and my_x are replaced with anonymized identifiers VAR1 and VAR2, while in-vocabulary identifiers np and sin preserve their names.
Python150k dataset (custom train-test split)
JavaScript150k dataset (custom train-test split)
Figure 2: Results for Transformer in the variable misuse task: joint bug localization and repair accuracy, mean value  $\pm$  standard deviation (over 3 runs). Models with the proposed OOV anonymization significantly outperform the standard model (all OOV identifiers are replaced with an UNK token). The numerical data for the plots is given in Table 2 in Appendix.
There are two main directions in the NLP literature to tackle rare tokens: open vocabulary and
* The work was done while working at Samsung-HSE Laboratory, HSE University. Both authors contributed equally.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 278-288 June 6-11, 2021. ©2021 Association for Computational Linguistics
copy-based approaches. An open vocabulary solution implies splitting rare tokens into subtokens (Sennrich et al., 2016). The copy-based approaches are used in generation tasks and imply using the pointer mechanism (Gulcehre et al., 2016) to copy tokens from the input sequence.
We propose a new, simple, yet effective approach for processing OOV identifiers in source code, namely OOV anonymization. Anonymization implies replacing rare identifiers with unique placeholders, i.e. VAR1, VAR2, VAR3 etc., while preserving the names of frequent identifiers. An example of OOV anonymization is shown in figure 1. The intuition behind using anonymization is that it preserves the semantics of the algorithm that the code snippet implements, i.e. renaming user-defined identifiers does not change the underlying algorithm. By contrast, replacing all rare identifiers with an UNK identifier changes the algorithm. We underline that we propose anonymizing only rare identifiers, because frequently used identifier names may serve as an additional source of information, and neural networks are indeed capable of capturing this information.
The proposed OOV anonymization strategy allows for easy implementation as a preprocessing step, thus no modification of model code is required. Another advantage of the OOV anonymization is that it enhances both the encoder and the decoder. The proposed approach significantly outperforms the model with all rare identifiers being replaced with an UNK, in code completion and bug fixing tasks, with the Transformer (Vaswani et al., 2017) architecture being used (see example comparison in Fig. 2). Our code and data split are available at https://github.com/bayesgroup/code_transformers.
Handling OOV identifiers in source code.
Code processing often borrows ideas from NLP. Source code can be represented as a sequence of identifiers. In this case, identifiers can be further split into subtokens using byte-pair encoding (BPE) (Karampatsis et al., 2020; Sennrich et al., 2016) resulting in an open vocabulary model. This approach has several drawbacks. Firstly, splitting identifiers into subtokens increases the length of the sequence several times. This substantially slows down inference, e.g. vanilla Transformer's forward pass has a complexity quadratic w.r.t. the
input length. Secondly, splitting breaks one-to-one alignment between identifiers and nodes in the parsing tree, e.g. abstract syntax tree (AST), in other words, several subtokens correspond to one node in the AST, which makes it harder to apply structure-aware models such as (Hellendoorn et al., 2020) or (Alon et al., 2019a). To the best of our knowledge, all SCP works, proposing structure-aware models, either use entire tokens without subtokenization / BPE, or average the embeddings over subtokens (this strategy provides only a slight quality improvement compared to the first one), and the question of how to incorporate BPE in structure-aware models needs further investigation. Taking into account the described disadvantages of BPE, we do not consider BPE in this work and do not split tokens into subtokens.
An orthogonal direction for handling OOV identifiers in source code is the modification of the computational graph. For the task of code generation, the pointer mechanism is widely adapted (Li et al., 2018). Cvitkovic et al. (2019) also propose a graph-structured cache for inferring the representations of the rare identifiers in source code. The major drawback of the mentioned approaches is that they are quite hard to implement.
Identifier anonymization in source code.
Chirkova and Troshin (2020) conduct an empirical study of Transformers for source code in a setting with all identifiers being anonymized and show that Transformers can make meaningful predictions in this setting. By contrast, we propose anonymizing only OOV identifiers and show that it boosts the performance of the model in the setting with frequent identifier names being present in the data. The anonymization of all identifiers has also been used in (Gupta et al., 2017) and (Xu et al., 2019) for training recurrent neural networks. Ahmed et al. (2018) replace variables with their types, losing information about identifier repetition.
Consider a vocabulary of all identifiers in the training data. It could be a vocabulary of all tokens if we treat input code snippets as text sequences, or a vocabulary of all user-defined variables if we parse the ASTs of code snippets. Let us now select the vocabulary  $V_{full}$  of frequent identifiers and call all others OOV identifiers.
We propose an elegant way of tackling OOV identifiers based on anonymization. Particularly,
we propose replacing all OOV identifiers with placeholders VAR1, VAR2, VAR3 etc. All occurrences of one identifier in one input sequence are replaced with the same placeholder (anonymized identifier), but different identifiers are replaced with different placeholders. One identifier may be replaced with different placeholders in different input sequences. An example of OOV anonymization is presented in figure 1.
We consider two strategies for the OOV anonymization, namely ordered anonymization and randomized anonymization. The ordered anonymization implies assigning an anonymized identifier VAR1 to the first seen rare identifier, VAR2 to the next seen rare identifier, etc. For example, the snippet from Fig. 1 is transformed into VAR1 = np.sin(VAR2) + VAR2. The randomized anonymization implies fixing the placeholder vocabulary size  $|V_{an}|$  and selecting a random subset of anonymized placeholders VAR1, ..., VAR|  $V_{an}$  for each code snippet. For example, the snippet from Fig. 1 can be transformed into VAR38 = np.sin(VAR801) + VAR801.
To ensure that we can always encode identifiers in a code snippet injectively, the size  $|V_{an}|$  of the placeholder vocabulary should not be fewer than the maximum possible number of tokens per snippet. We set  $|V_{an}|$  to the maximum length of code snippets.
The proposed OOV anonymization can be seen as a preprocessing step, thus no model parts change. In the encoder, the embedding matrix contains embeddings for both anonymized and in-vocabulary identifiers:  $\{e_v\}_{v\in V_{full}}\cup \{e_{\mathrm{VARi}}\}_{i = 1}^{|V_{an}|}$ . In the decoder, when generating the next identifier, the softmax is computed over all anonymized and invocabulary identifiers. We note that the ordered OOV anonymization may need a more careful implementation, e.g. of metric computation, see details in section 4.
We conduct experiments with Transformer (Vaswani et al., 2017) on the code completion (CC) and variable misuse (VM) tasks, on Python150k (Raychev et al., 2016a) (the redistributable version of (Kanade et al., 2020)) and JavaScript150k (Raychev et al., 2016b) datasets.
We use the problem setup, metrics and loss
of Hellendoorn et al. (2020) for the VM task, and of Kim et al. (2020) for the CC task. To validate our implementation, we check that the quality we obtain with the vanilla Transformer is the same as the quality of this model reported in the corresponding works, see details in Appendix B. As a base model, we use the 6-layer Transformer equipped with the relative attention mechanism (Shaw et al., 2018) and applied over the depth-first traversal of the AST. Chirkova and Troshin (2020) show that such an approach leads to high performance and outperforms the vanilla Transformer and several techniques for capturing AST structure in Transformer. The hyperparameters are given in Appendix A. Allamanis (2019); Chirkova and Troshin (2020) emphasize the importance of the thoughtful splitting data into training and testing parts, which includes splitting by repositories and removing duplicate code. We follow the same strategy in our experiments (later referred to as custom train-test split).
Variable misuse task. For the VM task, we use the same setup as in (Hellendoorn et al., 2020), below we briefly recap this setup. In the VM task, given the code of a function, the task is to output two positions (using two pointers): in what position a wrong variable is used and which position a correct variable can be copied from (any such position is accepted). If a snippet is non-buggy, the first pointer should select a special no-bug position. We obtain two pointers by applying two position-wise fully-connected layers, and softmax over positions on top of the Transformer encoder outputs. We use the joint accuracy to assess the model quality (the portion of buggy examples for which the model correctly localizes and repairs the bug).
To obtain a dataset for the VM task, we select all top-level functions in Python150k dataset, including functions inside classes, and filter out functions longer than 250 AST nodes, and functions with less than three positions containing user-defined variables or less than three distinct user-defined variables. The resulting training / testing set consists of  $417\mathrm{K} / 231\mathrm{K}$  functions (Python) and  $202\mathrm{K} / 108\mathrm{K}$  functions (JavaScript). One function may occur in the dataset up to 6 times, 3 times with synthetically generated bug and 3 times without bug. The buggy examples are generated synthetically by choosing random bug and fix positions from positions containing user-defined variables. When using the ordered OOV anonymization, we firstly inject a synthetic bug and then perform anonymization, to
JavaScript150k dataset (custom train-test split)
Figure 3: Results for Transformer in the code completion task (value prediction): mean reciprocal rank  $\pm$  standard deviation over 3 runs. The numerical data for the plots is given in Table 3 in Appendix.
avoid data leak.
Code completion task. For the CC task, we use the setup of Kim et al. (2020), and below we briefly review it. The CC task implies predicting the type and value of the next node based on the prefix of the depth-first AST traversal. We predict the next type and value using two fully-connected heads on top of the Transformer decoder and optimize the sum of cross-entropy losses for types and values. While computing the loss, we skip the first occurrences of anonymized values and special positions, i.e. UNK and PAD. We tie the embeddings of input and output layers. In this task, we split the large AST traversals into chunks with a maximum length of 500, as described in (Kim et al., 2020). The resulting dataset includes 186K / 100K training/testing chunks for Python and 270K / 220K for JavaScript.
We use mean reciprocal rank (MRR) to measure the quality of the model:  $\mathrm{MRR} = 100\% \cdot N^{-1}\sum_{i=1}^{N}1 / \mathrm{rank}_i$ , where  $N$  is the total number of tokens in the dataset and  $\mathrm{rank}_i$  is a position of the true token in the model ranking. We assign zero scores (a) if the correct token is not in the top 10 predicted tokens, (b) if the correct token is a UNK and (c) for the first occurrences of anonymized identifiers.
For the next value prediction task, we add the pointer mechanism to the Transformer for comparison. We re-implement the pointer mechanism following the design choice of (Deaton, 2019). Given an input sequence  $[x_1,\ldots ,x_\ell ]$  of length  $\ell$ , Transformer outputs two distributions: the distribution over the fixed vocabulary  $V$ ,  $p_{\mathrm{model}}(a)$ ,  $a\in V$ , and the probability of copying an input from position  $j$ ,  $p_{\mathrm{copy}}(j)$ ,  $j = 1,\dots ,\ell$ . Then both distributions are combined to obtain the final distribution over the extended vocabulary:  $p(x_{\ell +1} = a) = p_{\mathrm{gen}}p_{\mathrm{model}}(a)[a\in V] + (1 - p_{\mathrm{gen}})\sum_{j = 1}^{\ell}p_{\mathrm{copy}}(j)[x_j = a]$ . The switcher is computed given the current input and the output of the decoder as  $p_{\mathrm{gen}}(x_{\ell},h_{\ell}) = \sigma (w_h^T h_\ell +w_i^T x_\ell +b_{\mathrm{gen}})$ . The cross entropy loss is computed over the extended vocabulary.
We compare the proposed anonymization of OOV identifiers with the following baseline approaches: (1) Standard: with all OOV identifiers being replaced with an UNK identifier; (2) training on fully anonymized data, i.e. all identifiers are anonymized. This baseline corresponds to the zero vocabulary size in all plots. For the code completion task, we also include the baseline with the pointer mechanism.
Figure 2 presents the results for the variable misuse task, for different frequent identifier vocabulary sizes. We observe that the proposed approach, with the anonymization of OOV identifiers (dark blue and blue lines), performs substantially better than the baseline models, particularly than the standard approach with OOV identifiers being replaced with an UNK identifier (orange line). The leftmost point in both blue lines corresponds to the full anonymization baseline (zero vocabulary size). The ordered OOV anonymization (dark blue line) performs slightly better or similarly to the randomized OOV anonymization (blue line). We also experimented with the frequency-based OOV anonymization, i.e. sorting rare identifiers by frequencies in the code snippet and assigning VAR1 to the most frequent one, VAR2 to the next one etc. We found that such a strategy achieves the same quality as the ordered anonymization.
Increasing the vocabulary size for the standard model does not help much and even hurts the performance, i. e. the standard model with a vocabulary of 50K identifiers outperforms the one with
the largest possible vocabulary. The reason is that the embeddings of rare identifiers are updated only several times during the training and do not change a lot after being initialized randomly. On the contrast, anonymized identifiers occur quite frequently in the data, e.g. thousands of times, so their embeddings are updated regularly. As a result, it is more beneficial to anonymize rare identifiers than to include them in vocabulary.
The intuition behind why OOV anonymization performs well is that it saves information about variable repetition and thus does not change the algorithm that the code snippet implements. For example, in the buggy snippet with open(myfnm) as myfp: data = myfnm.read(), (should be myfp.read(), the model with OOV anonymization detects that OOV variables after as and before read are different and correctly predicts the bug, while the model with OOVs replaced with UNK does not distinguish variables myfnm and myfp and cannot detect the bug.
Figure 3 presents the results for code completion (value prediction), for different frequent identifier vocabulary sizes. In this task, the ordered OOV anonymization again slightly outperforms the randomized OOV anonymization, and they both substantially outperform the standard baseline and the baseline with full anonymization. Moreover, the proposed OOV anonymization surpasses the strong pointer baseline for almost all vocabulary sizes. The advantage of the proposed OOV anonymization approach is that it helps the Transformer to distinguish OOV identifiers in the input, while the pointer mechanism enhances only the output layer. Also, in contrast to the pointer mechanism, OOV anonymization is much easier to implement. The pointer mechanism and the OOV anonymization could be straightforwardly combined, however, in our experiments, this combination did not increase the scores compared to the maximum score of the OOV anonymization and the pointer. The results for type prediction are relatively the same as for the value prediction and can be found in Appendix C. We visualize the t-SNE representations of the learned embeddings in Appendix D.
4.3 The influence of the anonymized vocabulary size
The randomized OOV anonymization strategy comprises the hyperparameter  $|V_{an}|$ , i.e. the size of the anonymized vocabulary. It should not be less than the maximum sequence length, to avoid using the same placeholder for different identifiers, and we select  $|V_{an}|$  as the maximum length of code snippets. We tried using the larger values of  $|V_{an}|$  and observed the insignificant difference in quality in the variable misuse task, and a slight drop in quality in the code completion task, as shown in Table 1.
\( \mathbf{{PY}} \)	\( \mathbf{{JS}} \)
\( |V_{full}|: \)	1k	10k	1k	10k
\( |V_{an}|=0.5K \)	63.35	64.77	66.63	68.98
\( |V_{an}|=1K \)	63.03	64.63	66.52	68.75
\( |V_{an}|=3K \)	62.79	64.34	66.22	68.60
Table 1: Increasing the size  $\left| {V}_{an}\right|$  of the anonymized vocabulary for two frequent identifier vocabulary sizes  ${V}_{full}$  ,namely  $1\mathrm{k}$  and  ${10}\mathrm{k}$  ,in the code completion task (value prediction). Metric: MRR (%),all standard deviations are less than 0.3%.
In this work, we propose the effective anonymization-based encoding of out-of-vocabulary identifiers, with two options, namely ordered and randomized OOV anonymization. Our preprocessing technique allows for easy implementation, could be easily plugged into various Transformer models and outperforms the widely used standard approach by a significant margin. The ordered anonymization performs slightly better than the randomized anonymization but requires a more careful implementation.
We would like to thank Ivan Rubachev and the anonymous reviewers for the valuable feedback. The results on the variable misuse task were supported by the Russian Science Foundation grant No 19-71-30020. The results on the code completion task were supported by Samsung Research, Samsung Electronics. The research was supported in part through the computational resources of HPC facilities at NRU HSE.
Umair Z. Ahmed, Pawan Kumar, Amey Karkare, Purushottam Kar, and Sumit Gulwani. 2018. Compilation error repair: For the student programs, from the student programs. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET '18, page 78-87, New York, NY, USA. Association for Computing Machinery.
Miltiadis Allamanis. 2019. The adverse effects of code duplication in machine learning models of code. Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019a. code2seq: Generating sequences from structured representations of code. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Uri Alon, Meital Zilberstein, Omer Levy, and E. Yahav. 2019b. code2vec: learning distributed representations of code. Proceedings of the ACM on Programming Languages, 3:1-29.
David Azcona, Piyush Arora, I-Han Hsiao, and Alan Smeaton. 2019. User2code2vec: Embeddings for profiling students based on distributional representations of source code. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, LAK19, page 86-95, New York, NY, USA. Association for Computing Machinery.
Nadezhda Chirkova and Sergey Troshin. 2020. Empirical study of transformers for source code.
Milan Cvitkovic, Badal Singh, and Animashree Anand-kumar. 2019. Open vocabulary learning on source code with a graph-structured cache. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1475-1485. PMLR.
Jon Deaton. 2019. Transformers and pointer-generator networks for abstractive summarization.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Codebert: A pre-trained model for programming and natural languages.
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 140-149, Berlin, Germany. Association for Computational Linguistics.