SlowGuess/NAACL_2021_OCR huggingface dataset:
Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. Deepfix: Fixing common c language errors by deep learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, page 1345-1351. AAAI Press.
Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. 2020. Global relational models of source code. In International Conference on Learning Representations, ICLR 2020.
Srini Iyer, Ioannis Konstas, A. Cheung, and L. Zettlemoyer. 2016. Summarizing source code using a neural attention model. In ACL.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5110-5121. PMLR.
Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, C. Sutton, and A. Janes. 2020. Big code != big vocabulary: open-vocabulary models for source code. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering.
Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. 2020. Code prediction by feeding trees to transformers.
Jian Li, Yue Wang, Michael R. Lyu, and Irwin King. 2018. Code completion with neural attention and pointer networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI'18, page 4159-25. AAAI Press.
I. Loshchilov and F. Hutter. 2017. Sgdr: Stochastic gradient descent with warm restarts. In Proceedings of the International Conference on Learning Representations (ICLR).
Veselin Raychev, Pavol Bielik, and Martin Vechev. 2016a. Probabilistic model for code with decision trees. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, page 731-747, New York, NY, USA. Association for Computing Machinery.
Veselin Raychev, Pavol Bielik, Martin Vechev, and Andreas Krause. 2016b. Learning programs from noisy data. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL '16, page 761-774, New York, NY, USA. Association for Computing Machinery.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468, New Orleans, Louisiana. Association for Computational Linguistics.
Vighnesh Shiv and Chris Quirk. 2019. Novel positional encodings to enable tree-based transformers. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 12081-12091. Curran Associates, Inc.
L.J.P. van der Maaten and G.E. Hinton. 2008. Visualizing high-dimensional data using t-sne. Journal of Machine Learning Research 9:2579-2605.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Shengbin Xu, Yuan Yao, Feng Xu, Tianxiao Gu, Hang-hang Tong, and Jian Lu. 2019. Commit message generation for source code changes. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 3975-3981. International Joint Conferences on Artificial Intelligence Organization.
Passing ASTs to Transformer. To pass an AST to the Transformer, we follow the strategy of Chirkova and Troshin (2020). They converted each input code snippet to the depth-first traversal of the abstract syntax tree (AST), obtaining a sequence of pairs (node type, node value). The node types denote syntactic units of the programming language, e.g. If or For, and come from a small dictionary (up to 350 types), while the node values denote user-defined identifiers, language-specific identifiers, e.g. None in Python, and constants. Some nodes do not store any values, we insert <EMPTY> values in these nodes. We store two embedding layers, one for types and one for values, and sum the embedding of type and value in each AST node. The OOV anonymization is applied to the values. To train a model on the fully anonymized data, we anonymize all values except <EMPTY>.
Hyperparameters. We list hyperparameters for the VM / CC tasks using slashes. Our Transformer model has 6 layers,  $8 / 8$  heads,  $d_{\mathrm{model}}$  equals to  $512 / 512$ . The number of parameters of our models (excluding embeddings) is  $19\mathrm{M} / 18\mathrm{M}$ . We train all Transformers using Adam with a starting learning rate of  $0.00001 / 0.001$  and the batch size of 32 for 20 epochs (CC), 25 epochs (VM PY), or 40 epochs (VM JS). In the CC task, we use cosine learning rate schedule (Loshchilov and Hutter, 2017) with a warmup step of 2000 and zero minimal learning rate, and the gradient clipping of 0.2. In the VM task, we use a constant learning rate. We use residual, embedding and attention dropout with  $p = 0.2 / 0.1$ . We use relative attention (Shaw et al., 2018) with the maximum distance between elements of  $8 / 32$ .
B Validating our implementation
The numbers reported in our paper are not directly comparable to the works we borrow setups from, because we use our custom (and more correct) data split rather than the commonly used split (see details in Sec. 4). We ensure the validity of our results in two ways: by relying on the code of recently published works, and by comparing our numbers achieved for the commonly used data split to the numbers in the corresponding papers. Particularly, we use the model / loss / metrics and the code of (Kim et al., 2020) for the CC task, and the model / loss / metrics of (Hellendoorn et al., 2020) for
the VM task (we rewrite line-by-line their code for metrics and loss). For the vanilla Transformer in the VM task, Hellendoorn et al. (2020) report  $67.7\%$  joint accuracy and we achieved  $64.4\%$  with the similar model size. The results are close to each other. For the vanilla Transformer in the CC task (Python), for value / type prediction, (Kim et al., 2020) report  $58.0 / 87.3$  MRR ("TravTrans" model), and we achieve  $60.0 / 89.1$  MRR, again the results are close.
C Experiments with type prediction
In Fig. 4, we report the results for type prediction, code completion task. Overall, the anonymization of rare identifiers again performs better, compared to the standard Transformer with rare identifiers replaced with an UNK, and also improves over the pointer baseline for almost all vocabulary sizes.
Figure 4: Type prediction for code completion task for Transformer: mean reciprocal rank  $\pm$  standard deviation over 3 runs.
D Visualization of embeddings
In this section, we visualize the embeddings learned in the code completion task on Python dataset, vocabulary size 1k, OOV identifiers are anonymized randomly. We use t-SNE (van der Maaten and Hinton, 2008) with default hyperparameters and cosine metric to visualize the embeddings in a 2-dimensional space, see Figure 5. We
observe that the embeddings of anonymized identifiers form a well-separated cluster in the embedding space. We also measured the inter-cluster / intracluster cosine similarities: the average cosine similarity between pairs of embeddings in one cluster / two different clusters. We observe the intercluster similarity for in-vocabulary / OOV identifiers of  $0.09 / 0.05$ , and the intra-cluster similarity between in-vocabulary / OOV clusters of  $-0.06$ , which shows that these two clusters occupy different subspaces of the embedding space.
E Numerical data for the plots in the paper
Table 2 lists the numerical data for Figure 2 and Table 3 lists the numerical data for Figure 3.
For the code completion task, we also report accuracy scores of the best performing models for values prediction. We mark UNK prediction as wrong. Our random / ordered / pointer / baseline models achieve 59.31 / 59.71 / 58.88 / 50.41 accuracy  $(\%)$  on the Python dataset, and 64.08 / 64.13 / 63.58 / 58.48 on the JavaScript dataset.
Python150k embeddings (custom train-test split)
Figure 5: t-SNE visualization of Python150k embeddings. Anonymized (random) embeddings (left cluster), invocabulary embeddings (right cluster).
Variable Misuse task, Joint localization and repair accuracy  $(\%)$
PY	JS
|Vfull|	Ordered	Random	Standard	Ordered	Random	Standard
1	81.93	81.26	0.00	78.80	77.69	0.00
20	84.08	83.72	40.28	79.45	78.69	6.89
100	85.29	84.96	54.71	81.63	80.40	33.66
300	86.26	85.33	64.72	82.01	80.47	50.61
1000	86.76	86.10	72.36	83.05	82.62	62.87
10000	87.72	86.66	79.36	82.68	82.80	73.36
25000	87.29	86.85	81.01	82.66	82.65	74.27
50000	87.02	86.82	81.42	82.16	82.62	76.33
Max	N/A	N/A	80.16	N/A	N/A	73.68
Table 2: Numerical data for Figure 2. Max denotes the vocabulary without any filtering: 622K for PY and 266K for JS. The standard deviations for all models are approx.  $0.5\%$ .
Code Completion task, Values Prediction, MRR  $(\%)$
PY	JS
|Vfull|	Random	Ordered	Pointer	Standard	Random	Ordered	Pointer	Standard
1	57.00	58.61	54.77	N\A	57.93	60.77	56.61	N\A
10	58.40	59.52	56.01	N\A	59.81	61.60	57.55	N\A
50	59.45	60.74	57.68	N\A	61.81	63.19	59.63	N\A
100	60.35	61.41	58.53	N\A	62.88	64.09	60.92	N\A
200	61.31	62.26	59.72	N\A	64.23	65.25	62.36	N\A
500	62.55	63.29	61.45	N\A	65.69	66.43	64.34	N\A
1000	63.12	63.90	62.38	N\A	66.53	67.24	65.61	N\A
2000	64.05	64.51	63.28	46.0	67.46	68.04	66.53	51.85
10000	64.77	65.25	64.94	51.8	68.76	69.34	68.51	58.41
100000	63.91	64.27	65.09	55.07	69.87	70.05	70.05	63.9
Table 3: Numerical data for Figure 3. The standard deviations for all models are approx.  ${0.3}\%$  .