SlowGuess/NAACL_2021_OCR huggingface dataset:
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 7464-7473.
Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshanyak. 2019. An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(4):1-29.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. 2020. Detecting code clones with graph neural network and flow-augmented abstract syntax tree. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 261-271. IEEE.
Haoran Xu, Shuhui Fan, Yongjun Wang, Zhijian Huang, Hongzuo Xu, and Peidai Xie. 2020. Tree2tree structural language modeling for compiler fuzzing. In International Conference on Algorithms and Architectures for Parallel Processing, pages 563-578. Springer International Publishing.
Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440-450, Vancouver, Canada. Association for Computational Linguistics.
Hao Yu, Wing Lam, Long Chen, Ge Li, Tao Xie, and Qianxiang Wang. 2019. Neural detection of semantic code clones via tree-based convolution. In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC), pages 70-80. IEEE Press.
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019. A novel neural source code representation based on abstract syntax tree. In Proceedings of the 41st International Conference on Software Engineering, ICSE '19, page 783-794. IEEE Press.
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019. Design: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Advances in Neural Information Processing Systems, volume 32, pages 10197-10207. Curran Associates, Inc.
Hyper-parameter	RoBERTa*	CodeGPT-2	CodeBERT	GraphCodeBERT	PLBART
vocab size	50,265	50,001	50,265	-	50,004
n_positions	514	1024	514	256	1024
model size	768	768	768	768	768
# layers	12	12	12	12	6
# heads	12	12	12	12	12
dff	3072	3072	3072	-	3072
dropout	0.1	0.1	0.1	-	0.1
optimizer	Adam	Adam	Adam	Adam	Adam
learning rate	5e-5	5e-5	5e-5	1e-4	5e-5
batch size	32	32	32	64	32
Table 10: Details of the hyper-parameters used during fine-tuning for sequence generation tasks. * indicates pretrained from scratch using source code-text pairs.
Figure 4: Qualitative examples of "text to code" generation using PLBART, where generated code does not exactly match the reference.
Figure 5: Qualitative examples of source code translation between Java and C# using PLBART.