SlowGuess/NAACL_2021_OCR huggingface dataset:
Wasi Uddin Ahmad\*, Saikat Chakraborty†*, Baishakhi Ray†, Kai-Wei Chang
$^{\S}$ University of California, Los Angeles,  $^\dagger$ Columbia University
$\S$  {wasiahmad, kwchang} @cs.ucla.edu, †{saikatc, rayb} @cs.columbia.edu
Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks. PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding. Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program repair, clone detection, and vulnerable code detection, demonstrate PLBART's effectiveness in program understanding. Furthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow (e.g., if block inside an else block is equivalent to else if block) that are crucial to program semantics and thus excels even with limited annotations.
Engineers and developers write software programs in a programming language (PL) like Java, Python, etc., and often use natural language (NL) to communicate with each other. Use of NL in software engineering ranges from writing documentation, commit messages, bug reports to seeking help in different forums (e.g., Stack Overflow), etc. Automating different software engineering applications, such as source code summarization, generation, and translation, heavily rely on the understanding of PL and NL—we collectively refer them as PLUG (stands for, Program and Language Understanding and Generation) applications or tasks.
Program snippet in Python
1 def sort_list(uns):
2 return sorted(uns, key=lambda x:x[0])
Program snippet in Java
1 static Tuple[] sortArray(Tuple[] uns) {
2 return Arrays.sort(
3 uns, new Comparator<Tuple>(); 
4 public int compare(
5 Tuple o1, Tuple o2) {
6 return o1.get(0) == o2.get(0);
7 }
8 });
9 }
Summary: sort a list of tuples by first element
Figure 1: Example motivating the need to understand the association of program and natural languages for code summarization, generation, and translation.
Note that the use of NL in software development is quite different than colloquially written and spoken language. For example, NL in software development often contains domain-specific jargon, e.g., when software engineers use Code Smell<sup>1</sup>, it means a potential problem in code (something other than Smell in regular English language).
In this work, our goal is to develop a general-purpose model that can be used in various PLUG applications. Recent advancements in deep learning and the availability of large-scale PL and developers' NL data ushered in the automation of PLUG applications. One important aspect of PLUG applications is that they demand a profound understanding of program syntax and semantics and mutual dependencies between PL and NL. For example, Figure 1 shows two implementations of the same algorithm (sorting) in two PL and corresponding NL summary. An automatic translation tool must understand that function sorted in Python acts similar to Arrays.sort in Java and the lambda
<sup>1</sup>https://en.wikipedia.org/wiki/Code_smell
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655-2668 June 6-11, 2021. ©2021 Association for Computational Linguistics
operation in Python is equivalent to instantiating a Comparator object in Java. Similarly, a tool that summarizes either of these code must understand that x[0] in Python or Tuple.get(0) in Java refers to the first element in the tuple list.
Most of the available data in PL and NL are unlabeled and cannot be trivially used to acquire PLUG task-specific supervision. However, PLUG tasks have a common prerequisite — understanding PL and NL syntax and semantics. Leveraging unlabeled data to pretrain a model to learn PL and NL representation can be transferred across PLUG tasks. This approach reduces the requirement of having large-scale annotations for task-specific fine-tuning. In recent years we have seen a colossal effort to pretrain models on a massive amount of unlabeled data (e.g., text, images, videos) (Devlin et al., 2019; Liu et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020; Li et al., 2019; Sun et al., 2019) to transfer representation encoders across a wide variety of applications. There are a few research efforts in learning general purpose PL-NL representation encoders, such as CodeBERT (Feng et al., 2020) and GraphCodeBERT (Guo et al., 2021) that are pretrained on a small-scale bimodal data (code-text pairs). Such models have been found effective for PLUG tasks, including code search, code completion, etc.
Language generation tasks such as code summarization is modeled as sequence-to-sequence learning, where an encoder learns to encode the input code and a decoder generates the target summary. Despite the effectiveness of existing methods, they do not have a pretrained decoder for language generation. Therefore, they still require a large amount of parallel data to train the decoder. To overcome this limitation, Lewis et al. (2020) proposed de-noising sequence-to-sequence pre-training where a Transformer (Vaswani et al., 2017) learns to reconstruct an original text that is corrupted using an arbitrary noise function. Very recently, Lachaux et al. (2020) studied denoising pre-training using a largescale source code collection aiming at unsupervised program translation and found the approach useful. This raises a natural question, can we unify pretraining for programming and natural language? Presumably, to facilitate such pre-training, we need unlabeled NL text that is relevant to software development. Note that unlike other bimodal scenarios (e.g., vision and language), PL and associated NL text share the same alphabet or uses anchor tokens
Java	Python	NL
All Size	352 GB	224 GB	79 GB
All - Nb of tokens	36.4 B	28 B	6.7 B
All - Nb of documents	470 M	210 M	47 M
Table 1: Statistics of the data used to pre-train PLBART. "Nb of documents" refers to the number of functions in Java and Python collected from Github and the number of posts (questions and answers) in the natural language (English) from StackOverflow.
(e.g., "sort", "list", "tuple" as shown in Figure 1) that can help to learn alignment between semantic spaces across languages.
We introduce PLBART (Program and Language BART), a bidirectional and autoregressive transformer pre-trained on unlabeled data across PL and NL to learn multilingual representations applicable to a broad spectrum of PLUG applications. We evaluate PLBART on code summarization, generation, translation, program repair, clone detection, and vulnerability detection tasks. Experiment results show that PLBART outperforms or rivals state-of-the-art methods, e.g., CodeBERT and GraphCodeBERT, demonstrating its promise on program understanding and generation. We perform a thorough analysis to demonstrate that PLBART learns program syntax, logical data flow that is indispensable to program semantics, and excels even when limited annotations are available. We release our code<sup>2</sup> to foster future research.
2 PLBART
PLBART uses denoising sequence-to-sequence pretraining to utilize unlabeled data in PL and NL. Such pre-training lets PLBART reason about language syntax and semantics. At the same time, PLBART learns to generate language coherently.
2.1 Denoising Pre-training
Data & pre-processing We pre-train PLBART on a large-collection of Java and Python functions and natural language descriptions from Github and StackOverflow, respectively. We download all the GitHub repositories associated with Java and Python languages available on Google BigQuery. We extract the Java and Python functions following the pre-processing pipeline from Lachaux et al. (2020). We collect the StackOverflow posts (include both questions and answers, exclude code
$^{2}$ https://github.com/wasiahmad/PLBART
<sup>3</sup>https://console.cloud.google.com/
marketplace/details/gitpub/gitpub-repos
PLBART Encoder Input	PLBART Decoder Output
Is 0 the [MASK] Fibonacci [MASK] ? <En>	<En> Is 0 the first Fibonacci number ?
public static main ( String args [ ]) { date = Date ( ) ; System . out . ( String . format (" Current Date : % tc ",)) ; } <java>	<java> public static void main ( String args [ ]) { Date date = new Date ( ) ; System . out . printf ( String . format (" Current Date : % tc ", date ) ) ;}
def addThreeNumbers ( x , y , z ) : NEW_LINE INDENT return [MASK] <python>	<python> def addThreeNumbers ( x , y , z ) : NEW_LINE INDENT return x + y + z
Table 2: Example encoder inputs and decoder outputs during denoising pre-training of PLBART. We use three noisy strategies: token masking, token deletion, and token infilling (shown in the three examples, respectively).
snippets) by downloading the data dump (date: 7th September 2020) from stackexchange. Statistics of the pre-training dataset are presented in Table 1. We tokenize all the data with a sentencepiece model (Kudo and Richardson, 2018) learned on  $1/5$ 'th of the pre-training data. We train sentencepiece to learn 50,000 subword tokens.
One key challenge to aggregate data from different modalities is that some modalities may have more data, such as we have 14 times more data in PL than NL. Therefore, we mix and up/down sample the data following Conneau and Lample (2019) to alleviate the bias towards PL. We sample instances for pre-training according to a multinomial distribution with probabilities  $(q_{1}, q_{2}, \ldots, q_{N})$ :
$$
q _ {i} = \frac {1}{p _ {i}} \cdot \frac {p _ {i} ^ {\alpha}}{\sum_ {j = 1} ^ {N} p _ {j} ^ {\alpha}}, p _ {i} = \frac {n _ {i}}{\sum_ {j = 1} ^ {N} n _ {j}},
$$
where  $N$  is the total number of languages and  $n_i$  is the total number of instances in language  $i$ . We set the smoothing parameter  $\alpha$  to 0.3.
Architecture PLBART uses the same architecture as  $\mathrm{BART}_{\mathrm{base}}$  (Lewis et al., 2020), it uses the sequence-to-sequence Transformer architecture (Vaswani et al., 2017), with 6 layers of encoder and 6 layers of decoder with model dimension of 768 and 12 heads (~140M parameters). The only exception is, we include an additional layer-normalization layer on top of both the encoder and decoder following Liu et al. (2020), which is found to stabilize training with FP16 precision.
Noise function,  $f$  In denoising autoencoding, a model learns to reconstruct an input text that is corrupted by a noise function. Reconstruction of the original input requires the model to learn language syntax and semantics. In this work, we use three noisy strategies: token masking, token deletion,
and token infilling (Lewis et al., 2020). According to the first two strategies, random tokens are sampled and replaced with a mask token or deleted from the input sequence. In token infilling, a number of text spans are sampled and replaced with a single mask token. The span lengths are drawn from a Poisson distribution  $(\lambda = 3.5)$ . We mask  $35\%$  of the tokens in each instance.
Input/Output Format The input to the encoder is a noisy text sequence, while the input to the decoder is the original text with one position offset. A language id symbol (e.g., <java>, <python>) is appended and prepended to the encoder and decoder inputs, respectively. We provide a few examples in Table 2. The input instances are truncated if they exceed a maximum sequence length of 512.
Learning PLBART is pre-trained on  $N$  languages (in our case,  $N = 3$ ), where each language  $N_{i}$  has a collection of unlabeled instances  $\mathcal{D}_i = \{x_1,\ldots ,x_{n_i}\}$ . Each instance is corrupted using the noise function  $f$  and we train PLBART to predict the original instance  $x$  from  $f(x)$ . Formally, PLBART is trained to maximize  $\mathcal{L}_{\theta}$ :
$$
\mathcal {L} _ {\theta} = \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {m _ {i}} \log P (x _ {j} | f (x _ {j}); \theta)
$$
where  $m_{i}$  is the number of sampled instances in language  $i$  and the likelihood  $P$  is estimated following the standard sequence-to-sequence decoding.
Optimization We train PLBART on 8 Nvidia GeForce RTX 2080 Ti GPUs for 100K steps. The effective batch size is maintained at 2048 instances. We use Adam  $(\epsilon = 1\mathrm{e} - 6, \beta_{2} = 0.98)$  with a linear learning rate decay schedule for optimization. We started the training with dropout 0.1 and reduced it to 0.05 at 50K steps and 0 at 80K steps. This is done to help the model better fit the data (Liu et al., 2020). The total training time was approximately
<sup>4</sup>https://archive.org/download/stackexchange
PLBART Encoder Input	PLBART Decoder Input
S	def maximum(a, b, c): NEW_LINE INDENT return max([a, b, c]) <python>	<En> Find the maximum of three numbers
G	Find the maximum of three numbers <En>	<java> public int maximum (int a, int b, int c) { return Math. max(a, Math. max(b, c))}
T	public int maximum (int a, int b, int c) { return Math. max(a, Math. max(b, c))} <java>	<python> def maximum(a, b, c): NEW_LINE INDENT return max([a, b, c])
Table 3: Example inputs to the encoder and decoder for fine-tuning PLBART on sequence generation tasks: source code summarization (S), generation (G), and translation (T).
276 hours (11.5 days). All experiments are done using the Fairseq library (Ott et al., 2019).
2.2 Fine-tuning PLBART
We fine-tune PLBART for two broad categories of downstream applications.
Sequence Generation PLBART has an encoder-decoder architecture where the decoder is capable of generating target sequences autoregressively. Therefore, we can directly fine-tune PLBART on sequence generation tasks, such as code summarization, generation, and translation. Unlike denoising pre-training, the source sequence is given as input to the encoder during fine-tuning, and the decoder generates the target sequence. The source and target sequence can be a piece of code or text sequence. Table 3 shows a few examples of input and output to and for PLBART for different generation tasks. Note that PLBART prepends a language id to the decoded sequence; it enables fine-tuning PLBART in a multilingual setting (e.g., code generation in multiple languages).
Sequence Classification We fine-tune PLBART on sequence classification tasks following Lewis et al. (2020). The input sequence is fed into both the encoder and decoder. For a pair of inputs, we concatenate them but insert a special token  $(\mathrm{~~\text{一}~ / ~s~ > ~})$  between them. A special token is added at the end of the input sequence. This last token's representation from the final decoder layer is fed into a linear classifier for prediction.
Optimization We fine-tune PLBART for a maximum of 100K steps on all the downstream tasks with 2500 warm-up steps. We set the maximum learning rate, effective batch size, and dropout rate to 3e-5, 32 and 0.1, respectively. The final models are selected based on the validation BLEU (in generation task) or accuracy (in classification tasks).
Fine-tuning PLBART is carried out in one Nvidia GeForce RTX 2080 Ti GPU.
To understand PLBART's performance in a broader context, we evaluate PLBART on several tasks. Our evaluation focuses on assessing PLBART's ability to capture rich semantics in source code and associated natural language text.
We divide the evaluation tasks into four categories. The evaluation task datasets are summarized in Table 4. We use CodeXGLUE (Lu et al., 2021) provided public dataset and corresponding train-validation-test splits for all the tasks.
Code Summarization refers to the task of generating a natural language (English) summary from a piece of code. We fine-tune PLBART on summarizing source code written in six different programming languages, namely, Ruby, Javascript, Go, Python, Java, and PHP.
Code Generation is exactly the opposite of code summarization. It refers to the task of generating a code (in a target PL) from its NL description. We fine-tune PLBART on the Concode dataset (Iyer et al., 2018), where the input is a text describing class member functions in Java and class environment, the output is the target function.
Code Translation requires a model to generate an equivalent code in the target PL from the input code written in the source PL. Note that the source and target PL can be the same. Hence, we consider two types of tasks in this category.
The first task is a typical PL translation task, translating a code i.e., from Java code to C#, and vice versa. In this task, the semantic meaning of the translated code should exactly match the input
<sup>5</sup>We do not perform multilingual fine-tuning in this work.
Task	Dataset	Language	Train	Valid	Test
Summarizaion	Husain et al. (2019)	Ruby	24,927	1,400	1,261
Javascript	58,025	3,885	3,291
Go	167,288	7,325	8,122
Python	251,820	13,914	14,918
Java	164,923	5,183	10,955
PHP	241,241	12,982	14,014
Generation	Iyer et al. (2018)	NL to Java	100,000	2,000	2,000
Translation	Code-Code (Lu et al., 2021)	Java to C#	10,300	500	1,000
C# to Java	10,300	500	1,000
Program Repair (Tufano et al., 2019)	Java small	46,680	5,835	5,835
Java medium	52,364	6,545	6,545
Classification	Vulnerability Detection (Zhou et al., 2019)	C/C++	21,854	2,732	2,732
Clone Detection (Wang et al., 2020)	Java	100,000	10,000	415,416
Table 4: Statistics of the downstream benchmark datasets.
code. Thus, this task evaluates PLBART's understanding of program semantics and syntax across PL. The second task we consider is program repair. In this task, the input is a buggy code, and the output is a modified version of the same code which fixes the bug. This task helps us understand PLBART's ability to understand code semantics and apply semantic changes in the code.
Code Classification aims at predicting the target label given a single or a pair of source code. We evaluate PLBART on two classification tasks. The first task is clone detection, where given a pair of code, the goal is to determine whether they are clone of each other (similar to paraphrasing in NLP). The second task is detecting whether a piece of code is vulnerable. This task helps us gauging PLBART's effectiveness in program understanding in an unseen PL since the code examples in this task are written in C/C++.
BLEU computes the n-gram overlap between a generated sequence and a collection of references. We use corpus level BLEU (Papineni et al., 2002) score for all the generation tasks, except code summarization where we use smoothed BLEU-4 score (Lin and Och, 2004) following Feng et al. (2020).
CodeBLEU is a metric for measuring the quality of the synthesized code (Ren et al., 2020). Unlike BLEU, CodeBLEU also considers grammatical and logical correctness based on the abstract syntax tree and the data-flow structure.
Exact Match (EM) evaluates if a generated sequence exactly matches the reference.
We compare PLBART with several state-of-the-art models and broadly divide them into two categories. First, the models that are trained on the evaluation tasks from scratch, and second, the models that are pre-trained on unlabeled corpora and then finetuned on the evaluation tasks.
3.3.1 Training from Scratch
Seq2Seq (Luong et al., 2015) is an LSTM based Seq2Seq model with attention mechanism. Vocabulary is constructed using byte-pair encoding.
Transformer (Vaswani et al., 2017) is the base architecture of PLBART and other pre-trained models. Transformer baseline has the same number of parameters as PLBART. Hence, a comparison with this baseline demonstrates the direct usefulness of pre-training PLBART.
3.3.2 Pre-trained Models
As described in section 2, PLBART consists of an encoder and autoregressive decoder. We compare PLBART on two categories of pre-trained models. First, the encoder-only models (e.g., RoBERTa, CodeBERT, and GraphCodeBERT) that are combined with a randomly initialized decoder for task-specific fine-tuning. The second category of baselines include decoder-only models (CodeGPT) that can perform generation autoregressively.
Methods	Ruby	Javascript	Go	Python	Java	PHP	Overall
Seq2Seq	9.64	10.21	13.98	15.93	15.09	21.08	14.32
Transformer	11.18	11.59	16.38	15.81	16.26	22.12	15.56
RoBERTa	11.17	11.90	17.72	18.14	16.47	24.02	16.57
CodeBERT	12.16	14.90	18.07	19.06	17.65	25.16	17.83
PLBART	14.11	15.56	18.91	19.30	18.45	23.58	18.32
RoBERTa, RoBERTa (code) are RoBERTa (Liu et al., 2019) model variants. While RoBERTa is pre-trained on natural language, RoBERTa (code) is pre-trained on source code from CodeSearchNet (Husain et al., 2019).
CodeBERT (Feng et al., 2020) combines masked language modeling (MLM) (Devlin et al., 2019) with replaced token detection objective (Clark et al., 2020) to pretrain a Transformer encoder.
GraphCodeBERT (Guo et al., 2021) is a concurrent work with this research which improved CodeBERT by modeling the data flow edges between code tokens. We report GraphCodeBERT's performance directly from the paper since their implementation is not publicly available yet.
GPT-2, CodeGPT-2, and CodeGPT-adapted are GPT-style models. While GPT-2 (Radford et al., 2019) is pretrained on NL corpora, CodeGPT-2 and CodeGPT-adapted are pretrained on CodeSearchNet (Lu et al., 2021). Note that, CodeGPT-adapted starts from the GPT-2 checkpoint for pre-training.
We aim to address the following questions.
1. Does PLBART learn strong program and language representations from unlabeled data?
2. Does PLBART learn program characteristics, e.g., syntax, style, and logical data flow?
3. How does PLBART perform in an unseen language with limited annotations?
4.1 Code Summarization
Table 5 shows the result of code summarization. PLBART outperforms the baseline methods in five out of the six programming languages with an overall average improvement of 0.49 BLEU-4 over CodeBERT. The highest improvement  $(\sim 16\%)$  is in the Ruby language, which has the smallest amount of training examples. Unlike CodeBERT, PLBART is not pretrained on the Ruby language; however,
Table 5: Results on source code summarization, evaluated with smoothed BLEU-4 score. The baseline results are reported from Feng et al. (2020).
Methods	EM	BLEU	CodeBLEU
Seq2Seq	3.05	21.31	26.39
Guo et al. (2019)	10.05	24.40	29.46
Iyer et al. (2019)	12.20	26.60	-
GPT-2	17.35	25.37	29.69
CodeGPT-2	18.25	28.69	32.71
CodeGPT-adapted	20.10	32.79	35.98
PLBART	18.75	36.69	38.52
PLBART10K	17.25	31.40	33.32
PLBART20K	18.45	34.00	35.75
PLBART50K	17.70	35.02	37.11
Table 6: Results on text-to-code generation task using the CONCODE dataset (Iyer et al., 2018).
the significant performance improvement indicates that PLBART learns better generic program semantics. In contrast, PLBART performs poorly in the PHP language. The potential reason is syntax mismatch between the pre-trained languages and PHP. Surprisingly, RoBERTa performs better than PLBART on the PHP language. We suspect that since RoBERTa is pre-trained on natural language only, it does not suffer from the syntax mismatch issue. Overall in comparison to the Transformer baseline, PLBART improves with an average of 2.76 BLEU-4, and we credit this improvement to the pre-training step.
4.2 Code Generation
Table 6 shows the evaluation result on code generation from NL description. PLBART outperforms all the baselines in terms of BLEU and CodeBLEU. While CodeGPT-adapted (Lu et al., 2021) achieves the best Exact Match (EM) score, PLBART outperforms CodeGPT-adapted by a large margin in terms of CodeBLEU. This result implies that PLBART generates significantly more syntactically and logically correct code than all the baselines.
Figure 2 shows an example of code generated by PLBART. The difference between the reference code and the generated code is in line 6 onward. In the reference code,  $\mathtt{loc0}$  is returned, however