SlowGuess/NAACL_2021_OCR huggingface dataset:
Details. In all our models, node type embeddings have 300 units, node value embeddings have 1200 units (for static embeddings), and the one-layer LSTM's hidden state has 1500 units. The described model size matches the configuration of the model of Li et al. (2018). The proposed dynamic embeddings of values have 500 units in all experiments to show that they outperform the static embeddings with much less dimension. In the code completion task, we split the input AST traversals into the chunks, each chunk has the length of 50 AST nodes, and apply attention and pointer only over the last 50 positions. In the variable misuse task, we pass the entire function's AST traversal to the model. In code completion / variable misuse tasks, we train all models for 10 epochs with AdamW (Loshchilov and Hutter, 2019) / Adam (Kingma and Ba, 2015) with an initial learning rate of  $0.001 / 0.0001$ , a learning rate decay of 0.6 after each epoch, a batch size of  $128 / 32$ , and using weight decay of  $0.01 / 0$ . We also use early stopping for the variable misuse task. For code completion, all hyperparameters are the same as in (Li et al., 2018). We tuned hyperparameters to achieve convergence on the training set, for variable misuse. We use the same hyperparameters for static and dynamic embedding models. Both used datasets are large, which helps to avoid overfitting, thus the regularization is not much needed.
Figure 2: The visualisation of the model input in two settings considered in the paper: full data setting and anonymized setting, for the example code snippet from Figure 1(a). The leftmost column represents types (the same for both settings), two other columns visualize values for two settings.
5.1 Anonymized setting
We firstly test the proposed dynamic embeddings in the setting without using the user-defined variable names, stored in node values. Directly omitting values results in losing much information, this can be seen as replacing all the variables in a code snippet with the same variable var. To save the information about whether two AST nodes store the same value or not, we anonymize values, i.e. we map the set of all node values in the program (except dummy values, e.g. <EMPTY>) to the random subset of anonymized values var1...varK,  $K$  is a size of the anonymized value vocabulary, we use  $K = 1000$ . For example, code snippet sum = sum + lst[i] may be transformed into var3 = var3 + var8[var1], and stat = [sum / n; sum] — into var1 = [var5 / var2; var5]. All occurrences of the same value in the program, e.g. sum, are replaced with one anonymized value, but value sum may be replaced with different anonymized values in different programs. Fig. 2 visualizes how the anonymization is applied to AST. Although being not so practically oriented, the anonymized setting highlights the capabilities of the deep learning models to capture pure syntactic information from the AST, without relying on the unstructured text information laid in variable names. In our opinion, this setting should become a must for the future testing of syntax-based SCP models, and the proposed dynamic embeddings could be used as a first layer in such models to capture an equal-not-equal relationship between values.
Model	PY	JS
LSTM	LSTM+at	LSTM+pt	LSTM	LSTM+at	LSTM+pt
Stat. emb. (an. data)	55.76	59.74	60.28	51.80	56.26	57.67
Dyn. emb. (an. data)	66.35	66.79	66.90	61.69	62.86	62.85
Stat. emb. (full data)	61.62	63.73	64.69	62.03	64.28	65.05
Table 1: Anonymized setting, code completion task, accuracy (%) of the proposed dynamic embedding model and the baseline static embedding model on Python150k (Py) and JavaScript150k (JS) datasets. All standard deviations over three runs are less than  $0.05\%$ . The last row represents the conventionally used model trained on the full data (Li et al., 2018) (this model uses more information during training than the models in the first three rows). Columns list the three variants of the base architecture: LSTM, attentional LSTM (LSTM+at), and attentional LSTM with pointer (LSTM+pt).
Model	PY	JS
Stat. emb. (an. data)	25.17	13.16
Dyn. emb. (an. data)	63.64	53.53
Stat. emb. (full data)	54.78	35.06
In the described anonymized setting, we compare the proposed dynamic embeddings (constant initialization) with the static embeddings, i.e. learning the static embeddings of var1..varK.
Results for the code completion task. In the code completion task, we consider three variants of the architecture: plain LSTM, and attentional LSTM with and without pointer. We note that our goal is to compare the dynamic embeddings with the baseline in three setups, i.e. using three base architectures. We do not pursue the goal of comparing base architectures. Table 1 lists the results.
For all base architectures, the proposed dynamic embeddings outperform static embeddings by a large margin. We note that the number of parameters in both architectures is approximately the same. In the first two setups, with plain and attentional LSTMs, the models can only predict values by generating them from the vocabulary (no pointer), relying on the input and output embeddings of the values. In these setups, the difference between static and dynamic embeddings is large, indicating that dynamic embeddings capture the semantics of the variables significantly better. In the setup
Table 2: Anonymized setting, variable misuse task, joint accuracy  $(\%)$  of the proposed dynamic embedding model and the baseline static embedding model on Python150k (Py) and JavaScript150k (JS) datasets. All standard deviations over three runs are less than  $0.1\%$ . The last row represents the conventionally used model trained on the full data (Vasic et al., 2019b) (this model uses more information during training than the models in the first three rows).
Model (full data)	Code compl.	Var. misuse
PY	JS	PY	JS
Stat. emb.	64.69	65.05	54.78	35.06
Dyn. emb.	68.61	65.67	68.59	53.74
Table 3: Full data setting, two tasks, Python150k (Py) and JavaScript150k (JS) datasets. Accuracy (\%) of LSTM with pointer (code competition), joint accuracy (\%) of BiLSTM (variable misuse). All standard deviations are less than  $0.05\%$  for code completion and  $0.1\%$  for the variable misuse task. Comparing the conventionally used model (static embeddings) and the proposed dynamic embeddings (static initialization). The conventionally used model is a model of Li et al. (2018) for the code completion task and of Vasic et al. (2019b) for the variable misuse task. Note that we use custom data split, see details in Sec. 4.
with pointer LSTM as a base architecture, the static embeddings win back some percent of correct predictions by relying on the pointer mechanism. Still, the gap between them and dynamic embeddings is large. The portion of correct predictions made using the pointer is  $25\%$  for static embeddings and only  $0.01\%$  for dynamic embeddings. This shows that dynamic embeddings actually replace the pointer mechanism, performing better. This also explains why the difference in quality of dynamic embeddings between attentional LSTM and pointer LSTM is very small.
Interestingly, on the Python dataset, the model with dynamic embeddings trained in the anonymized setting outperforms the conventionally used static embedding model trained in the full data setting, although the first model uses much less information during training. The explanation is that the first model predicts rare values much better than the second model: the accuracy of rare<sup>1</sup> val
<sup>1</sup>By rare values, we mean values outside top-1000 frequent values.
ues prediction is  $27\%$  for the first model and  $11\%$  for the second, for the pointer LSTM model. On the contrary, frequent values are easier to predict with static embeddings: the accuracy of predicting frequent values is  $53\%$  for the first model and  $57\%$  for the second model. The total frequencies of rare and frequent values are approximately the same and equal to  $25\%$  (the rest  $50\%$  are EMPTY values, they are predicted with similar quality with both models). As a result, when counting accuracy over all values, the first model outperforms the second one.
However, on the JavaScript dataset, the first model does not outperform the second one. We analysed the example predictions of both models on both datasets and found that in JavaScript, there are a lot of short code snippets commonly used in different projects. This is expected since JavaScript is mostly used for one purpose, web development, while Python is used for a lot of different purposes. As a result, for JavaScript, the total frequency of top-1000 values is  $32\%$  (higher than for Python), while the total frequency of rare values is  $22\%$  (less than for Python). The commonly used code snippets are easy to predict in the full data setting but hard to predict in the anonymized setting: the accuracy of predicting frequent values is only  $44\%$  for the first model and  $54\%$  for the second model. The rare values are still better predicted with dynamic embeddings, but with the gap smaller than for Python: the accuracy of rare values prediction is  $23\%$  for the first model and  $17\%$  for the second one. The gap is smaller since rare values also occur in the commonly used code snippets which improves the performance of the second model on rare values. When counting accuracy over all values, the second model outperforms the first one.
Results for the variable misuse task. Table 2 lists the joint accuracies of the proposed model and the baseline in the anonymized setting. Again the dynamic embeddings outperform static embeddings by a large margin. Moreover, the dynamic embeddings outperform even the commonly used static embedding model trained on the full data, for both datasets. We think the reason is that we use the dynamic embeddings in two layers of bi-directional LSTMs and these bi-directional dynamic embeddings provide a rich representation of the input code snippet.
5.2 Full data setting
We now test the proposed dynamic embeddings in the full data setting, i.e. we compare a commonly used model with static embeddings and the proposed model with dynamic embeddings (static embedding initialization). The initialization of dynamic embeddings was discussed in Sec. 3.2. Both models process the full data (see illustration in Fig. 2.
The results for both tasks are presented in Table 3 and show that the dynamic embeddings outperform the static embedding model in all cases. We note that dynamic embeddings could be easily incorporated into any recurrent SCP architecture. In our experiments we incorporate them into the base models of Li et al. (2018) and Vasic et al. (2019b) and show that the dynamic embeddings significantly improve these base models. We also note that we use the dynamic embeddings of 500 units while static embeddings have 1200 units. The number of parameters in the dynamic embedding layer, 2.6M, is much smaller than that of the main LSTM layer, 13.8M, and two orders smaller than the number of parameters in the embedding layer, 134M (the numbers are given for the code completion task).
5.3 Example predictions
Figure 3 visualizes the predictions of different models for three example code snippets in Python. We highlighted three scenarios when the dynamic embedding model outperforms the static embedding model in the full data setting: 1) capturing the specific role of the variable, e.g. variable qual indexes sequence in the list comprehension in the left example; 2) associating variables with each other, e.g. in the central example, variable name always goes with 0, and variable post always goes with 1; 3) repeating variables when they occur in the similar context they have already been used, e.g. zeros in the right example. In all these examples, the proposed dynamic model makes correct predictions, while the static embedding model makes mistakes, in the full data setting. In the anonymized setting, all models tend to predict previously used variables, and again the dynamic embedding model captures the described relationships, and the static embedding model tends to simply predict the most frequent previously used variable.
Figure 3: Example predictions for code completion task on Python language. Row 1: ground truth; rows 2 and 3: model trained in the full data setting; rows 4 and 5: models trained in the anonymized setting (these models observe data in a different way, see Fig. 2). The model predicts one next token based on the prefix and does not see gray-colored code.
5.4 Validating the implementation
In our experiments, we use the setup of Li et al. (2018) in the code completion task and of Hellendoorn et al. (2020) in the variable misuse task, but with our custom data split, see details in Section 4. To maintain the possibility of comparing our results to these works, we trained the static embedding models in the full data setting, with the commonly used train / test splits of Python150k and JavaScript150k datasets. For code completion with vocabulary size 50K and pointer network, using exactly the same setup as in (Li et al., 2018), we achieved accuracy of  $69.39\% / 80.92\%$ , while the paper reports  $71\% / 81.0\%$  for Python / JavaScript: the results are close to each other. In the variable misuse task, we achieved joint accuracy of  $50.2\%$  while Hellendoorn et al. (2020) report  $44.4\%$  (Python, JavaScript was not reported in the paper). Our result is higher, since we use 1500 hidden units while Hellendoorn et al. (2020) uses 256 hidden units. In addition, we use different preprocessing and different synthetically generated bugs.
In this work, we presented dynamic embeddings, a new approach for capturing the semantics of the variables in code processing tasks. The proposed approach could be used in any recurrent architecture. We incorporated dynamic embeddings in the RNN-based models in two tasks, namely code completion and variable misuse detection, and showed that using the proposed dynamic embeddings improves quality in both full data setting and the anonymized setting, when all user-defined identifiers are removed from the data.
I would like to thank Sergey Troshin, Irina Saparina, and the anonymous reviewers for the valuable feedback. The results for the anonymized setting presented in section 5.1 were supported by Samsung Research, Samsung Electronics. The results for the full data setting presented in section 5.2 were supported by the Russian Science Foundation grant No.19-71-30020. The research was supported in part through the computational resources of HPC facilities at NRU HSE.
Umair Z. Ahmed, Pawan Kumar, Amey Karkare, Purushottam Kar, and Sumit Gulwani. 2018. Compilation error repair: For the student programs, from the student programs. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET '18, page 78-87, New York, NY, USA. Association for Computing Machinery.
Miltiadis Allamanis. 2019. The adverse effects of code duplication in machine learning models of code. Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019. code2seq: Generating sequences from structured representations of code. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations,
ICLR 2015; Conference date: 07-05-2015 Through 09-05-2015.
Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks for program translation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 2552-2562, Red Hook, NY, USA. Curran Associates Inc.
Nadezhda Chirkova and Sergey Troshin. 2020. Empirical study of transformers for source code. In CoRR.
Nadezhda Chirkova and Sergey Troshin. 2021. A simple approach for handling out-of-vocabulary identifiers in deep learning for source code. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics.
Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. Deepfix: Fixing common c language errors by deep learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, page 1345-1351. AAAI Press.
Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. 2020. Global relational models of source code. In International Conference on Learning Representations, ICLR 2020.
Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780.
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment generation. In Proceedings of the 26th Conference on Program Comprehension, ICPC '18, page 200-210, New York, NY, USA. Association for Computing Machinery.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5110-5121. PMLR.
Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, C. Sutton, and A. Janes. 2020. Big code != big vocabulary: open-vocabulary models for source code. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Sosuke Kobayashi, Naoaki Okazaki, and Kentaro Inui. 2017. A neural language model for dynamically representing the meanings of unknown words and entities in a discourse. In Proceedings of the Eighth International Joint Conference on Natural Language
Processing (Volume 1: Long Papers), pages 473-483, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Jeremy Lacomis, Pengcheng Yin, Edward J. Schwartz, Mitiadis Allamanis, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. 2019. DIRE: A neural approach to decompiled identifier naming. In 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019, pages 628-639. IEEE.
Triet Le, Hao Chen, and Muhammad Ali Babar. 2020. Deep learning for source code modeling and generation: Models, applications and challenges. Computing Research Repository, arXiv:2002.05442.
Alexander LeClair and Collin McMillan. 2019. Recommendations for datasets for source code summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3931-3937, Minneapolis, Minnesota. Association for Computational Linguistics.
Jian Li, Yue Wang, Michael R. Lyu, and Irwin King. 2018. Code completion with neural attention and pointer networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI'18, page 4159-25. AAAI Press.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Veselin Raychev, Pavol Bielik, and Martin Vechev. 2016a. Probabilistic model for code with decision trees. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, page 731-747, New York, NY, USA. Association for Computing Machinery.
Veselin Raychev, Pavol Bielik, Martin Vechev, and Andreas Krause. 2016b. Learning programs from noisy data. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL '16, page 761-774, New York, NY, USA. Association for Computing Machinery.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Vighnesh Shiv and Chris Quirk. 2019. Novel positional encodings to enable tree-based transformers. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 12081-12091. Curran Associates, Inc.
Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh Singh. 2019a. Neural program repair by jointly learning to localize and repair. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh singh. 2019b. Neural program repair by jointly learning to localize and repair. In International Conference on Learning Representations.
Shengbin Xu, Yuan Yao, Feng Xu, Tianxiao Gu, Hang-hang Tong, and Jian Lu. 2019. Commit message generation for source code changes. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 3975-3981. International Joint Conferences on Artificial Intelligence Organization.