Before revision:  BERT and other works have further shown that pre-trained contextual embeddingscan be extremely powerful and can be fine-tuned effectively for a variety of downstream supervised tasks . Inspired by these developments, we present the first attempt to replicate this success on source code. We curate a massive corpus of Python programs from GitHub to pre-train a BERT model, which we call Code Understanding BERT (CuBERT).
After revision:  BERT and other works have further shown that pre-trained contextual embeddingscan be extremely powerful and can be fine-tuned for downstream tasks with less labeled data and training budget, while achieving better accuracies. However, there is no attempt yet to obtain a high-quality contextual embedding of source code, and to evaluate it on multiple program-understanding tasks simultaneously; that is the gap that this paper aims to mitigate. Specifically, first, we curate a massive corpus of Python programs from GitHub to pre-train a BERT model, which we call Code Understanding BERT (CuBERT).