CodeT5Mix: A Pretrained Mixture of Encoder-decoder Transformers for Code Understanding and Generation
Authors: Yue Wang
Hung Le
Akhilesh Deepak Gotmare
Junnan Li
Steven Hoi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Language model pretraining
multimodal learning
code understanding and generation

Abstract: Pretrained language models (LMs) trained on vast source code have achieved prominent progress in a wide range of code intelligence tasks. Despite their success, they either adopt specific types of network architectures (encoder-only or decoder-only) for different downstream tasks or rely on a single architecture (encoder-decoder or UniLM-style encoder) for all tasks. The latter approach usually results in a sub-optimal performance on a subset of tasks. To address these limitations, we propose “CodeT5Mix”, a  mixture of encoder-decoder Transformers for code where its components can be flexibly combined based on the target  tasks during finetuning, while still enjoying the mutual benefits from the joint pretraining. To endow the model with both code understanding and generation capabilities, we pretrain CodeT5Mix  using a mixture of denoising, contrastive learning, matching, and Causal Language Modeling (CLM) tasks  on  large-scale multilingual code corpora in nine programming languages. Additionally, we design a weight sharing strategy in  decoders except the feedforward layers, which act as task-specific experts to reduce the  interference across tasks of various types. We extensively evaluate CodeT5Mix on seven tasks in four different modes and achieve state-of-the-art (SoTA) performance on most tasks such as text-to-code retrieval,  code completion and generation, and math programming. Particularly, we demonstrate that CodeT5Mix can be used as a unified semi-parametric retrieval-augmented generator with SoTA code generation performance. 

Under review as a conference paper at ICLR 2023
CODET5MIX: A PRETRAINED MIXTURE OF ENCODER-DECODER TRANSFORMERS FOR CODE UN- DERSTANDING AND GENERATION
Anonymous authors Paper under double-blee blind review
ABSTRACT
Pretrained language models (LMs) trained on vast source code have achieved prominent progress in wide range of code intelligence tasks. Despite their success, they either adopt specific types of network architectures (encoder-only or decoder-only) for different downstream tasks or rely on a single architecture (encoder-decoder or UniLM-style encoder) for all tasks. The latter approach usu- ally results sub-optimal performance on a subset of tasks. To address these limitations, we propose "Code'TSMix", a mixture of encoder-decoder Transform- ers for code where its components can be flcxibly combined based on the target tasks during finetuning, while still enjoying the mutual benefits from the joint pretraining. To endow the model with both code understanding and generation ca- pabilities, pretrain CodeT55ix using mixture of denoising, contrastive learn- ing, matching, and Causal Language Modeling (CLM) tasks on large-scale mul- tilingual code corpora in stage-wise manner. Additionally, sharing strategy in decoders except the feedforward layers, which act as task- specific experts to reduce the interference across tasks of various types. We exten- sively evaluate CodeT5Mix on seven code-related tasks over twenty datasets and show achieves saate-of-the-aee (SoTA) performance on most tasks such as text- to-code retrieval, code completion and generation, and math programming. Partic- ularly, we demonstrate that CodeT5Mix can be used unified semi- cerimerreettiic retrieval-augmented generator with SoTA code generation performance.
design weight
INTRODUCTION
Language model pretraining (Chen et al. 2021 Wang et al., 2021c Feng al. 20200) has recently demonstrated remarkable success in various downstream tasks in the code domain (Husain et al. 2019 Lu et al. 2021 Hendrycks et al.) 2021). By pretraining large-scale language models on massive code-based data (e.g. GitHub public data), these models
learn rich contextual represen- tations which can be transferred to related downstream tasks. However, we found that many of the existing models are specifically designed perform well only in a subset of tasks (e.g. generative- only tasks or retrieval-only tasks). On other tasks, their performance is suboptimal and the models often require substantial modifications to the architectural features or learning objectives.
Existing models have two main limitations. First, current models follow either encoder-only (Feng al. 2020 Guo al. 202T) or decoder-only (Chen al. 2021 Nijkamp et al. 2022) architectures which are suitable for only subset of tasks. SSecifiially, encoder-only models are often used to facilitate retrieval-based tasks such as text-to-code retrieval (Lu al. (02T). For generative tasks such code generation (Chen et al. 2021 Hendrycks et al., 2021). decoder-only models are more appropriate. Several approaches have adopted encoder-decoder architectures to adapt to multiple types of tasks (Wang et al.) 2021c Ahmad et al.) 2222I. While these models can achieve good performance overall, they still fail to beat state-of-the- encoder-only or decoder-only baselines in some tasks, e.g., retrieval and code completion tasks respectively (Guo et al.) 2022). Moreover, Li jet al. (2022b) observes that encoder-decoder models do not perform well with in co-tontext learning compared to GPT-style models like Codex (Chen et al. 2021) in code synthesis tasks.

---

Under review as a conference paper at ICLR 2023
Code-only
Code-Text
Understanding Tasks
Generation Tasks Code Sunmarization Coda Completion Generation, Math Pregramming
Data
Pair
Text Co-eo Retrivall, Defect Detection, Clane Detection
pretraining
pretraining
finetuning
finetuning
Mixture
Encoder- Decoder LM
Encoder-
Encoder-
Decoder- only
Ercoder- Decoder
Decoder
only
Transformes
Task Experts
Span
Causal LM
Contrastive Loss
Matching
Matching FFN
Code Generation FFN
Text Generation FFN
Denaising
Loss
Figure I: An overview of our CodeT5Mix approach
Secondly, current models are trained in self-supervised learning objectives that might not be ap- propriate transfer the models some downstream tasks. For instance, T5-based models such as (Wang et al. 2021c) are often trained with a span denoising objective. However, in downstream tasks such code generation (Chen al., 2021 Hendrycks et al.) (02T), most state-of-the-art mod- els are pretrained with next-token prediction objective which auto-rogressively predicts a program token by token. Furthermore, most models do no have specific pretraining tasks ensure the sharp text/code representation learning which is vital for understanding tasks like text-to-code retrieval. Although recent attempts (Guo al. 2022) introduce contrastive learning pretraining tasks to cope with this, the performance is still limited by neglecting the fine-grained cross-modal alignments.
To address the above issues, we introduce "CodeT5Mix", a new pretrained language framework for both code understanding and generation (See Fig. 1 for an overview). Specifically, CodeT5Mix includes the following contributions:
A mixture of encoder-decoder Transformers: we introduce a new architectural design for multi-task pretraining and flexible finetuning for both code understanding and generation tasks. CodeT5Mix consists of multimodal encoder and decoder modules, which, downstream tasks, can be directly repurposed and combined to suit different functionalities.
A mixture of self-supervised pretraining tasks: we adopt diverse set of pretraining objectives to learn rich representations from both code and text data. We design a stage-wiss pretraining strategy to first train code-only data with span denoising and causal language modeling (CLM) tasks, and then train on text-code data with cross-modal contrastive learning, matching, and CLM tasks, where the matching task is crucial to capture the fine-grained text-code interactions.
A weight sharing stralegy through task-specific experts: to optimize multi-task learning while keeping the model parameters affordable, we propose task-specific experts which are designed for different learning tasks while sharing the same backbone contextual representations.
A unified model for remi-parametric retrieval-augmenntr generation: as CodeT55Mx is capable of both retrieval and generation tasks, we demonstrated that it can seamlessly adopted as a semi semi-parametric retrieval-augne -ugmentedd generator to achieve SoTA code generation performance.
Thorough evaluation and SoTA performance: our extensive evaluations show that M5dx yields significant performance gains on most downstream tasks compared to their SoTA base- lines, e.g., 8 text-to-code retrieval tasks (+3.16 avg. MRR). 2 line-level code completion tasks (+2.56 avg. exact match), retrieval-augmented code generation tasks (+5.78 avg. BLEU-4).
Open source:
implementation code, data, and pretrained models will be made publicly available.
RELATED WORK
Typically, code-based language models (LMs) can be categorized into three architectures: encoder-
only models like CodeBERT (Feng et al. (020). GRRRhCodeBERT (Guo et al. 2021) ), and CVd (Wang et al. (202)) ) decoder-only models like CodeGPT (Lu al. 2021), Codex (Chen et al. 202T), InCoder (Fried et al. 2022) and CodeGen (Nijkamp et al. 2022). and encoder-decoder models like
2

---

Under review as a conference paper at ICLR 2023
PLBART (Ahmad al. 2021), CodeT5 (Wang et al. 202 SPT-Code (Nuu et al. 2022). and NatGen Chakraborty et al. (2022). For encoder-only and decoder-only models, they are often ideal for either understanding tasks such as code retrieval (Husain et al. 2019) generation tasks such as code synthesis (Chen et al. 2021 Hendrycks et al. 2021) respectively. For encoder-decoder models, they can be easily adapted for both code understanding and generation but do not always achieve better performance (Wang et al., [021c] AAhmad et al., (021)) than decoder-only or encoder-only models. In this work, we propose a flexible mixture architecture of Transformers that can operate in various modes: encoder-only, decoder-only. and encoder-decoder.
Prior models employ limited set of pretraining tasks, which might not be appropriate to transfer the models to some downstream tasks. For instance, there are no specific pretraining tasks in CodeT5 (Wang et al. 2021c) to ensure the learning of a sharp code representation that can distinguish code samples of different semantics, leading to sub-optimal performance on code understanding tasks like code retrieval (Husain al. 2019). In light of this, we introduce a contrastive learning task to learn better unimodal representation and a matching task to learn richer bimodal representation, which has been shown to be very effective in text-image retrieval tasks (Li et al. [02T). Additionally, encoder-decoder models (Wang et al. al.]220222 202 AAhad et al.) 2021 Niu al. |022||Chakaabbr Chakraborty et al., 2022) are not ideal for retto-regressive generation tasks like next-line code completion (Lu et al,
202 Svyatkovskiy et al., 2020b) I and program synthesis (Chen et al. 2021), these models trained to recover short spans of limited lengths rather than a whole program sequences. Inspired by recent advances in related NLP domains TTy et al.] (2022): [Soltan et al. (2022), we propose a mixture of span denoising and CLM tasks to improve the model with better causal generation capability.
More related our work is UniXcoder (Guo al. 2022) that adopts UniLM-style (Dong al. 2019) model to support various tasks. However, Unide attempts to employ a fixed model with fully shared parameters (but different attention masks) to support many different tasks. The model might suffer from inter-task interference, resulting in unstable model optimization and sub-optimal performance. CodeT5Mix, we employ partial weight-sharing approach with a mixture of en- coder and decoder Transformers, pretrained with diverse set of learning objectives. Compared to prior work (Wang et al. [202Ic] Guo et al. 2022; Wang et al. 2022). CodeT5Mix also does not rely on any engineering features such as abstract syntax tree or identifier information.
CODETSMIX: A MIXTURE OF ENCODER TRANSFORMERS
To develop CodeT5Mix, we extend CodeTS (Wang al., 2021c), code-aware encoder-decoder language model, with flexible mixture architecture of encoder-decoded Transformers (Sec. 3.1). In this mixture architecture, we pretrain the models with enhanced learning objectives, with diverse set of self supervisdd tasks over two major stages of pretraining with unimodal and multimodal data (Sec. 3.2). Finally, using the pretrained models, different components can be transferred and activated to serve different functionalities for different downstream tasks (Sec. 3.3). An overview of
our pretraining and finetuning process can be seen in Fig. I
and more details in Fig. and Fig. 3
3.1 MODEL ARCHITECTRRE
CodeT5Mix consists of multiple Transformers-bawee encoders and decoders. Each of them can operate specific functionality and in combination, they serve unified multi-task model:
Bimodal encoder, which encodes a text code smippet into continuous representation through bidirectional self-attention (Vaswani et al. 2017). Similar to BERT (Devlin et al.) 2019), we prepend a special token [CLS] ] to the input and regard its output embedding the final Trans- former layer as the representations of the corresponding input text or code. We further add a linear layer map the output to 56-dimensional vectors together with the L2 normalization.
Bimodal matching decoder. which aims predict whether a text and code snippet share the same semantics. Given a code sample, this decoder first passes it to an embedding layer and a causal self-attention layer. The self-attention representations are then passed to a cross-altention
'Recent work such as (Tabachnyk& & Nikolov 2022 Fried et al. 2022) demonstrated success using encoder- decoder models for infilling-hasse hased code completion, in which the context after the cursor (or after the completed code) employed. Such code completion setting is not the focus of our paper.
3

---

Under review as a conference paper at ICLR 2023
gu- [MASKO] lacs, E, sun): Count pairs [HASKI] to "euos
[NABK0] BairsCouns [MASK1] with sun (NASK2) [MASK3] arzii]
[HASK2], E):
geLPa fazz, oy sun):
for range 11 + nI:
Count paire with zun equaa to 'sun!
ic [MAS33]
0
COunt
for in zange 10, D)3
retuEn count
ror 3 in zange 1, nl:
ase
[CLM]
CodeT5Mix
count
zeluza count
[CIM]
dEc gctPalcscount (azE, (um) paiza with
for in range (1 D):
Tause
if azet]] count
count
Ioz in range (0, al:
AMLUED Count
Figure 2: First-stage pretraining objectives of CodeT5Mix on code-only unimodal data
layer which queries relevant signals from the text representations (received from the bimodal encoder). A task-specific [Match] token is prepended to the code input sequence to inform the decoder of the text-code matching functionality, and a [EOS] token is appended to the end of the code input. The output embedding of [EOS] at the last decoder layer is used as the text-code cross-modal alignment representation?
Unimodal generation decoders, which can generate an output sequence in programming lan- guage/natural language. These decoders follow the same design as the bimodal matching decoder with causal attention and pross-attention layers. When the input is text sample, we use code generation decoder and prepend a [CDec] token as the first token in the input sequence. This decoder operates in code generation functionality. When the input is code sample, we use text generation decoder and prepend a [TDec] token to the input sequence. This decoder operates in text generation (i.e. code summarization) functionality.
3.2 PRETRAINING OBJECTIVES
Given the above model architecture, we develop both understanding and generation-based self- supervised tasks over two major pretraining stages. In the first stage, we pretrain a vanilla encoder- decoder Transformer with massive code-only data using computationally efficient objectives (see Fig. 2). In the second stage, we the prior model backbone to initialize mixture encoder- decoder Transformers (as described in Sec. 3.T) and train with a smaller set of code-text data with cross-modal learning objectives (see Fig. 3} We found that this stage-wise training approach can efficiently expose our models to more diverse data to learn rich contextual representations. Note that for each stage, we jointly optimize the model with its pretraining objectives with equal weights.
3.2.1 1 PRETRAINING ON CODE-ONLY UNIMODAL DATA
In the first stage, we pretrain CodeT5Mix large-scale code-only unimodal data, which can be easily obtained from open-source platforms like GitHub. Note that in this stage, we pretrain the model from scratch using mixture of span denoising and CLM tasks:
Span Denoising. Similar to T5 (Raffel et al. 2020), we randomly replace 15% of the tokens into indexed sentinel tokens (like [MASKO]) in the encoder input, and require the decoder recover them via generating a combination of these spans. We follow Code'T5 to employ the whole-word masking by sampling spans before subword tokenization avoid masking partial subtokens.
Causal Language Modeling. To optimize our model for auto-regressive generation, we introduce two variants of CLM. In the first variant, randomly select a pivot location and regard the context before it as the source sequence and the sequence after it as the target output. We restrict the pivot location to be uniformly sampled between 10% and 90% of the whole sequence. We prepend special token [CLM] to the input sequence. The second CLM variant is decoder-only generation task and can be viewed as an extreme case of the first variant. In this task, we always pass a single [CLM] token to the encoder input and require the decoder to generate the full code sequence.
Note that we choose the embedding of [ROS] instead of [ [Match] as the decoder employs causal self- attention masks and only the last decoder token can attend all the contexts.
4

---

Under review as a conference paper at ICLR 2023
Text Code Curtret
TrahCode
Codee Corprraionn
Trat Exneration
Matining
Feed Forward
Feed Forward
Feed Forward
Feed Forward
Feed Forward
Cross Attention
Cross Attention
Cross Attention)
Bi Self-Att
Bi SelF-Att
Causal Self-Att
Causal Self-Att
Causal Self-Att
Encoder
|[CSS]+
Increment
[CLS]+
Decoder
"[Match|+
*[CDac]+
*[TDac]+
Figure 3: Second-stage pretraining objectives of CodeT5Mix on text-code bimodal data. Model components with the same color have tied weights: encoders are fully shared and 3 decoders except the Feed Forward layers (FNN) shared. Although this mixture architecture incurs extra parameter cost of two FNN experts compared to a standard mmcoder-decoden such cost can be waived in finetuning as only FFN expert is activated for task and incurs no extra computational cost.
Compared to the first variant, this task aims to provide more dense supervision signals to train the decoder independent full-fledged code generation module.
3.2.2 PRETRAINING ON TEXT-CODE BIMODAL DATA
In the second stage, we use text-code bimodal data the function level curated by Husain et al. (2019) In this setting, each text-code pair is code function and its corresponding docstring de- scribing its semantics. Such bimodal data format facilitates the exploration of cross-modal retrieval and generation tasks. Note that we adopt the CodeTTMi architecture (see Fig. 3) in which the en- coder and decoder components are initialized from the checkpoints after the first pretraining stage.
Text-Code Contrastive Learning. This task aims to align the feature space of text and code en- coder by pulling together the representations of positive text-code pairs and pulling apart the negative pairs. This task has been shown be effective for code understanding (Guo al. 2022). This task only activates the bimodal encoder to produce text/code embedding.
To enrich the negative samples, we use momentum encoder to store embeddings of samples from previous mini-batches, as similarly adopted by He et al. (2020): : Li et al. (2022a). Specifically, the momentum encoder maintains queuing system that enqueues the samples in the current mini-batch and dequeues the samples in the oldest mini-batch. To ensure the consistency of representations across training steps, we update the momentum encoder by linear interpolation of the original en- coder and the momentum encoder. Besides, since text and code samples might be loosely paired and each text/code sample can have multiple positive pairs, we also use the momentum encoder to create soft labels and consider the potential positives in the negative pairs.
Text-Codk Matching, This task activates the bimodal matching decoder, which aims to learn bet- ter bimodal representations that capture the fine-grained alignment between text and code modalities. We build linear layer top of the output embedding of the decoder for binary classification task, which predicts whether a text-code pair is positive (matched) or negative (unmatched).
In order to find more informative negatives, employ a hard negative mining strategy (Li et al. 2021). Specifically, we sample hard negatives based on the contrastive- based similarity scores be- tween the current sample and previous samples in the queue maintained by the momentum encoder. As such, harder negatives are more likely to be selected. For a batch of positive pairs, we construct two batches of negative pairs by mining negatives from the text/code queue with code/text query.
Text-Code Dual Generation. This task focuses a cross-modal generative objective between text and code through dual multimodal conversion: text-to-code generation and code-to-text gener- ation (i.e. code summarization). Each conversion separately activates the corresponding (unimodal) code/text generation decoder. This task is an effective learning objective to close the gap between the pretraining and finetuning stage generation-basee downstream tasks (Wang et al. 2021c).
5

---

Under review as a conference paper at ICLR 2023
3.2.3 EFFICIENT WEIGHT-SHARING STRATEGY WITH SPER-IC EXPERTS
While we only use a single encoder to process bimodal data, we adopt a partially sharing scheme among the three decoders. Specifically, among the decoders, we share the parameters of the self- attention and cross-attention layers. We found that the contextual representations learned from these layers should be shared across text-code matching and text-code dual generation tasks, enabling cross-task generalization at the context level. Intuitively, these multimodal tasks benefit from shar- ing contextual representations, e.g., a text-lo-code generation decoder could benefit from semantic- aware code representations well learned from text-code matching task.
While the contextual representations can learned across tasks, we deploy different feed-forward layers with separate weights, similar to the modality -speciffc experts in rision-languss encoders by Wang et al.] (202Ia). These layers acts as task-specific experts in order to reduce the task in- terference during pretraining. Another benefit of this approach is keeping the model parameters affordable, even with a mixture architecture like CodeTT5Mix. Note that a weight-sharing scheme that fully shares all parameters among the decoders can save more parameter costs, but will result in serious interference and performance drops in downstream tasks (see (see/Sec. 4.4p. Moreover, the extra parameter cost of the introduced task experts can be waived during finetuning, only a single task expert will be activated for one downstream task, thereby incurring no extra computational cost.
3.3 FINETUNING ON DOWNSTREAM TASKS
Understanding Tasks. We employ three tasks to evaluate the code understanding capability of models: text-to-code retrieval, defect detection, and clone detection. CodeT5Mix support these understanding tasks in two ways: first, it employs the bimodal encoder to obtain text/code embeddings, which can be either passed to a binary classifier for defect detection, used to compute a similarity score for clone detection and code retrieval tasks; secondly, CodeT55ix can additionally the text-code matching decoder to predict the matching probabilities.
Decoder-only Tasks, We use next-line code completion tasks to evaluate the decoder-only genera- tion capability of CodeT5Mix. In this setting, we always feed [CLM] token the encoder input and pass the source sequence to the decoder. The decoder is required to understand the given context and complete the next line of code. We freeze the weights of the encoder and the cross-attention layers at the decoder to reduce the number of trainable parameters for efficient training.
Seq2Seq Generation Tasks. CodeT5Mix can be transferred various (Seq2Seq) generation tasks such as code summarization, code generation, and math programming. For math programming tasks, as their domains are much different from our text-code pretraining data, we employ models pretrained from the first pretraining stage for evaluation.
EXPERIMENTS
We conduct comprehensive experiments seven code understanding and generation tasks across nine programming languages (PLs). Detailed experimental setups can be found the Appendix.
Baselines. We develop two variants of CodeT5Mix: base (220M) and large (770M) models. We compare CodeT5Mix with state-of-the-art code-based pretrained LMs. For encoder-only models, consider RoBERTa (Liu al. 2019). CodeBERT (Feng al. 2020 trained with masked lan- guage modeling, CraphCodeBERT (Guo et al. [02TT) using data flow extracted from abstract syntax tree (AST) of code, and SYNCOBERT (Wang al. 2021b) that incorporates AST and contrastive learning. For decoder-only models, we employ GPT-2 (Radford et al. 2019) and CodeGPT (Lu al. 202T). : where both are pretrained using CLM objective. For encoder-decoder models, we consider PLBART (Ahmad et al., (021) and CodeT5 (Wang et al.) 2021c) that employ unified framework to support both understanding and generation tasks. We further include CodeT5-large re- sults from Le et al. (2022). We also compare another unified model UniXcoder (Guo et al. (022)) that employs UniLle masking (Dong et al.) 2019). For model sizes, CodeBERT, GraphCodeBERT, SYNCOBERT, UniXcoder are based on RoBERTa-base and have 125M parameters, while GPT-2 and CodeGPT has 124M and PLBART has 140M. Notably, CodeT5Mix only employs roughly half of its parameters when acted in encoder-only and decoder-only modes.

---

Under review as a conference paper at ICLR 2023
4.1 EVALUATION ON UNDERSTANDING TASKS
Text-to-Code Retrieval.
Table I: Text-to-Code Retrieval results (MRR) on CodeXGLUE
This task aims find the
most semantically related
Model
CodeSearch
Vet
CosQA
AdvTest
Ruby
Go
Fython
Java
PTP
Overall
code snippet at the func-
RoHERIa
58.7
51.7
85.01
38.7
59.9
56.0
61.7
60.3
18.3
tion level from a collec- CodeBERT
67.9
62.0
88.2
67.2
67.6
62.8
69.3
65.7
27.2
tion of candidate codes based on a natural lan- guage query. We con- sider three datasets for evaluation: CodeSearch-
GraphCoteBERT
SYNCOBERT
72.2
70.3
67.7
64.4
89.7
91.3
72.4
69.2
72.3
69.1
67.8
64.9
71.3
74.0
68.4
38.3
35.2
PLBART
67.5
61.6
88.7
66.3
66.3
61.1
68.6
65.0
34.7
CodeTS-basc
71.9
65.5
88.8
69.8
68.6
64.5
71.5
67.8
39.3
Unidender
91.5
72.0
Code 15Mix-base
71.7
70.8
92.4
75.6
76.1
69.8
77.1
72.7
43.3
Mode-SMix-larg
92.7
75.8
74.0
Net (Husain et al., 2019).
CosQA (Huang et al., (222), and AdvTest (Lu et al. 202I), which are curated from the original
NeaeSearchNet by filtering data with low-quality queries, adopting real-world queries from mod- em search engine, and obfuscating identifiers to normalize the code. In this task, we activate the encoder and matching decoder of CodeT5Mix and use Mean Reciprocal Rank (MRR) the metric.
From Table I, our CodeT5Mixxbase significantly outperforms all existing encoder-only and encoder- decoder models and large variant further sets new SoTA results, surpassing the previous SoTA UniXcoder by more than 3 absolute MRR points on all 3 tasks across 8 datasets. This implies CodeT5Mix is a robust code retriever model to handle queries with diverse formats and PLs. Be- sides, CodeTT5Mix-ber -baee yields substantial performance gains over CodeTT5-bae,, which can be at- tributed the text-code contrastive learning and matching objectives that facilitate better unimodal and bimodal representation learning. Particularly, compared to prior contrastive-leane leaaninn- approaches like SYNCOBERT and UniXcoder, our models achieve much better results, which can be attributed to our bimodal matching decoder that allows for more fine-grained text-code alignments.
Defect Detection and Clone Detection. Defect de- tection is to predict whether code is vulnerable soft- ware systems or not, while clone detection aims to mea- sure the similarity between two code snippets and pre- dict whether they have a common functionality. We use benchmarks from CodeXGLUE (Lu al., 202T) and use accuracy and F1 score as the metrics. In Table we can see CodeT5Mix models achieve new SoTA accu- racy on defect detection. On clone detection, our model achieves comparable results to SoTA models, where the performance increase tend to be saturated.
Table 2: Results on understanding tasks:
code defect detection and clone detection
Defect
Clone Detection
Model
Acc
Rec
Pree
Fl
RoBERTa
61.1
95.1
8788
91.3
CodeBERT
94.7
93.4
94.1
GraphCodeBERT
94.8
95.2
95.0
PLBART
63.2
94.8
92.5
93.6
CodcT5-base
65.8
95.1
94,9
95.0
UniXender
92.9
97.6
95.2
CodeT5Mix-base
66.1
96.4
94.1
95.2
CodeT5Mix-Iarge
66.7
96.7
93.5
95.1
4.2 EVALUATION ON CODE COMPLETION TASKS
We evaluate the decoder-only generation capa- bility of CodeTT5Mix through line-level code completion task, which aims to complete the next whole line code based on the previous code contexts. We employ PY150 (Raychev et al. 2016) and GitHub JavaCorpus (Allamanis & Sutton 2013) from CodeXGLUÉ, and use ex- act match (EM) accuracy and Levenshtein edit similarity (Svyatkovskiy et al.] 2020a) eval- uation metrics. Typically, this task requires decoder-only model for efficient training.
Table 3: Results on line-level code complction
PY150
JavaCorpus
Model
EM
Edit Sim
EM
Edit Sim
Transformer
38.51
69.01
17.00
50.23
GPT-2
41.73
70.60
27.50
60.36
CodeGPT
42.37
71.59
30.60
63.45
PLBART
38.01
68.46
26.97
61.59
CodeT5-base
36.97
67.12
24.80
58.31
UniXeoder
43.12
72.00
32,90
65.78
CodeT3Mix-busc
43.42
73.69
34.23
68.75
CodeT5Mixxlarge
44.86
74.22
36.27
70.33
As shown Table 3 and encoder-decoder models in both metrics. In particular, -oxeT5Mix-base yields substantial im- provements over C5deT5-bbsse by 6.45 and 9.43 EM scores on PY150 and JavaCorpus respectively. This is mainly due the causal language modeling task introduced in first-stage pretraining, which allows the decoder to see longer sequences instead of a combination of discrete spans in CodeT5, leading to better causal sequence generation capability.
our CodeT5Mix achieves new SoTA results compared to both decoder-only

---

Under review as a conference paper at ICLR 2023
Table 5: Results math programming tasks. denotes few-shot learning.
Results MathQA-Python
(b) Results on GSM8K-Pytee
Model
Pass 80
Model
Pass @
Pass 100
Codex Davinci
OpenAI 6B
21.8
70.9
LaMDA 8B
74.7
Codex Davinci
17
71
LaMDA
79.5
LaMDA 137B
LaMDA 137B
81.2
PaLM-Coder 540B
50,9
GPT-Neo+sampling 125M
84.7
GPT-Neo+sampling 2.7B
19.5
41.4
CodeTS-base 220M
71.5
CodeTS-base 220M
13.5
58.4
CodeTS--arge 770M
72.3
CodeTT5-arree 770M
19.3
61.4
CodeTSMix-buse 220M
85.6
CodeT5Mix- base 220M
22.4
70,5
CodeI'Mii--arge 770M
87.4
IodeT5Mix-large 770M
26.2
73.8
4.3 EVALUATION ON SEQ2SEQ GENERATION TASKS
Code Summarization and Code Generation. Code summarization aims to summarize code snippet into docstrings while code generation produce a function based on natural language description. We employ the clean version of CodeSearchNet dataset in six PLs for code summariza- tion and Java ConCode (lyer et al. 2018 for code generation. For evaluation metric, we employ BLEU-4 (B4), exact match (EM) accuracy, and CodeBLEU (CB) (Ren et al.. 2020) which accounts for syntactic and semantic matches based on the code structure addition to the n-gram match. From CodeT5Mix, we activate the encoder with the text generation decoder for code summariza- tion, and with the code generation decoder for code generation. We report the results in Table 4]
Table 4: Results on code summarization (aver-
age BLEU-4 over PLs) and code generation
Sum.
Generation
Model
EM
B4
CB
Overall, found that encoder-decoder models (CodeT5 and CodeTT5Mix) generally outperform both encoder-only and decoder-only models, well as the unified UniXcoder with controlled masks on both tasks. This implies that encoder- decoder models can better support Seq2Seq gen- eration tasks. Our CodeT5Mix-larg achieves new SoTA results on both tasks across various metrics.
CodeBERT
17.83
CodeGPT
20.10
32.79
35,98
PLBART
18.32
18.75
36.69
38,52
CoTexT
18.54
20.10
37.40
40.14
UniXcnder
19.30
22.60
38.23
CodeT5-base
19.55
22.30
40.73
43.20
CodeT5-
19.87
22.65
42.66
45.08
CodeT3Mix-baae
19.81
22.30
41.23
44.03
CodeT5Mix-large
20.15
22.75
42.73
45.56
Math Programming. To evaluate models for code generation, exact match or BLEU scores might be limited there be multiple forms of correct program solutions. Besides, Chen et al. (2021) found that the functional correctness of generated codes correlates poorly with their BLEU scores. Therefore, further consider math programming tasks, namely MathQA-Python (Austin al.) 2021] and GSM8K-Python (Cobbe et al. 2021I. where code correctness can be measured based on the execution outputs of code programs. The task to generate Python programs solve mathe- matical problems described in natural language descriptions. We employ pass @ k, which measures the percentage of problems solved using ls generated programs.
Apart from CodeTS, we compare with very large-scale decoder-only models including Codex Chen al.] 2021). LaMDA (Austin al. (02T), PaLM-Codee Codder (Chowdhery et al.) 2022). and GPT-
Neo Black et al.) with self-sampled learning strategy (Ni al.) (022). As shown Tables 5a
and [b) we found that CodeT5Mix achieves significant performance gains, outperforming many CodeT5Mixx
pretrained models of much larger sizes. Specifically,
achieves new SoTA results of 87.4 pass @ 80 on MathQA-Python and 73.8 pass @ 100 on GSM8K- ESMB8KPyyhon. On GSM8K- Python, Our model achieves the second best result of 26.2 pass@ @ I, only behind PaLM-Coder which has a much larger size than M5dxx (at 540B) and was exposed to much larger pretraining data.
4.4 ABLATION STUDY
We conduct an ablation study to analyze the impacts of different architectural designs (removing either the matching decoder or both code and text generation decoder) and weight-sharing strategies (fully separate or shared decoders compared to shared decoders except FFNs). Table reports the results three representative tasks, namely text-to-code retrieval (MRR), code summarization (BLEU) and generation (CodeBLEU) Note that for code retrieval and summarization that share the same dataset source of clean NodeSearchNet (Lu al. 2021),
CodeT5Mix-honl checkpoint
8

---

Under review as a conference paper at ICLR 2023
Table 6: Ablation results of M5ix with varying model designs and weight sharing strategies.
Text-to-Code Retrieval
Model
Summarization
Generation
Ruby
JavaSeripi
Go
Python
Java
PHP
Overall
Code 15MMx-xaaas
75.85
69.91
91.33
74.45
75.28
68.22
75.84
19.53
44.03
no match decoder
74,51
69.09
90.69
71.36
71.81
67.70
74.19
19.49
43,76
no code/text decoder
74.39
68.68
91.36
73,46
74,60
67.71
75.03
separale devöder
74.72
69.46
91,45
73.85
74.55
67.60
75.27
79.38
43.63"
shured decoder
75.84
69.68
91.07
73.38
74.31
67.43
75.29
19.24
43,44
that was jointly trained on all six PLs in a multitask setting for evaluation to remove the impact of task-specific finetuning. For the code generation task, we report the task-specific finetuning results.
Overall, in CodeT5Mix, we found that all components of our mixture architecture can complement to each other and lead to the best results on all tasks. This is observed by a performance drop by removing the matching decoder or code/text generation decoder. Particularly, we found that the rex--to-code retrieval performance (1.65 average MRR over 6 datasets
matching decoder is critical dropped by removing it), as this decoder can learn a better bimodal representation that captures the fine-grained alignment between text and code, while for code summarization and generation tasks it has only slight benefits. For the weight sharing strategy, we found that our proposed task-specific FFN experts allow the model to balance various tasks with insignificant extra overhead of FFN parameters. This is demonstrated by the consistent performance gains over the model variants (the last two rows in Table 6) with a fully separate decoder or shared decoder design.
4.5 RETRIEVAL-AUGGENTTE CODE GENERATION
As our model is capable of both code
Table 7: Results of retrieval-augmmeted code generation
retrieval and generation, it can be nat-
Java
Python
Model
urally exploited as a unified semi- parametric retrieval-augmented gen- crator. To explore such setting, we follow [arvez et al. (202T) to eval- uate two code generation tasks by reversing the input and output der of code summarization on Java and Python and using their released deduplicated retrieval codebase. We evaluate our model in three set- tings: retrieval-baseee generative, and retrieval- augmented (RA) generative. For the retrieval -bssd setting, we activate our encoder to retrieve the top-1 code sample as the pre- diction given a text query, while for the RA generative setting, we append the combination of top-k retrieved samples (k=1 in our work) to the input and activate the code generation decoder.
EM
B4
CB
EM
B4
CB
Retrieval- based
BM25
0.00
4,90
16.00
0.00
6.63
13.49
SCODE-R
0.00
25.34
26.68
0.00
22.75
23.92
Code Tase-
0.00
28.74
31.00
0.00
27.30
26.51
Generative
CodeBERT
0.00
8.38
14.52
0.00
4.06
10.42
GraphCodeBERT
0.00
7.86
14.53
0.00
3.97
10.55
CodeGPT PLBART
0.00
7.10
14.90
0.01
3.11
11.31
0.00
10.10
14.96
0.00
4.89
12.01
bode
0.00
10.33
20.54
0.00
4.40
13.88
Retrieval Augmented Generative
REDCODER
10.21
28.98
33.18
9.61
24.43
30.21
CodeTSMix-buse
11.66
33.83
40.60
11.83
31.14
36.39
we found that our CodeT5Mix achieves significantly better results in all cate- gories, especially in the retrieval-based and RA generative setting. Compared to the previous SoTA model of REDCODER-EE EXXT (Parvez et al.) 202 I) that separately employs GraphCodeBERT as the retriever and PLBART as the generator, model can seamlessly operate as a better code retriever and generator, leading to superior retrieval-augmented generative performance.
As shown in Table 7,
CONCLUSION
We present CodeT5Mix, a new code-based pretrained model with a mixture of encoder-decoder Transformer modules that can be flexibly combined for a wide range of code understanding and generation tasks. To train CodeT55Mxx we introduce a diverse set of pretraining tasks including denoising, causal language modeling, contrastive learning and matching to learn rich representations from both code-only data and bimodal code-text data. We design an efficient weight-sharing strategy with task-specific FFN experts to reduce inter-task interference while keeping the model parameters affordable. Extensive experiments We further showcase that CodeT5Mix can fully support a semi-part carimmeraettiic aetrieval-uugment code generation approach the model can effectively operate in both code retrieval and generation tasks.
seven downstream tasks verify the superiority of model.

---

Douglas Under review as a conference paper at ICLR 2023
6 ETHICS STATEMENT
Improving code understanding and generation systems has the potential to create substantial positive impacts on the society, by making programming more accessible and developers more productive via natural language interfaces. However, as discussed in detail by Chen et al. (02T), several ethical considerations have to be made when deploying such systems scale. One such limitation is the risk of generated code summaries or comments containing toxic and insensitive language. Several studies have discussed the problem of ensuring non-toxic natural language generation with some solutions ranging from the use of RL (Ouyang et al.. 2022), weighted decoding (Krause et al. 2021) to safety-specific control tokens (uu et al., 2020).
Besides safety, substantial consideration is also needed to account for the broader intellectual prop- crty implications of code generation and search systems before deployment. For instance, with code search, appropriate attribution should be provided the source along with the retrieved re- sults. Code generated from deep learning models can also contain security vulnerabilities, requiring expert review before adoption.
REFERENCES
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for program understanding and generation. In NAACL-HLT, pp. 2655-2668. Association for Computational Linguistics, 2021.
Miltiadis Allamanis and Charles Sutton. Mining source code repositories at massive scale using language modeling. In MSR, pp. 207-216. IEEE Computer Society, 2013.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation- formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2357-2367 Minneapolis, Minnesota, June 2019. Association for Computational Lin- guistics. doi: 10.18653/v1/N19-11 1245. URL utthttss::/ ac http://///acaattooooooooooooe orthology..rr//1115 1245
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2 077307 2021.
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow, march 2021. URI. https://doi. wrg/10.5281/zenodl 5297715.
Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Prem Devanbu, and Baishakhi Ray. Nat- gen: generative pre-training by "naturalizing" source code. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software En- gineering. 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint vrXiv:2107.03377 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton. Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyyashchekko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski. Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov. Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy
10

---

Under review as a conference paper at ICLR 2023
Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reicchiro Nakano, Christo- pher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, 0os/2110.14161 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Li Dong, Nan Yang. Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation.
Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Sys- tems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13042-13054, 2019.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjum Shou, Bing Qin. Ting Liu, Daxin Jiang, and Ming Zhou, Codebert: A pre-trained model for programming and natural languages. In EMNLP (Findings), volume EMNLP 2020 of Findings of ACL, pp- 1536-1547. Association for Computational Linguistics, 2020.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. CoRR, abs/2204.0520 2022.
Daya Guo. Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng. Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. Graphcodebert: Pre-training code representations with data flow. In ICLR. OpenReview.net, 2021.
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. Unixcoder: Unified cross-modal pre-training for code representation. In ACL (1), pp. 7212-7225. Association for Computational Linguistics, 2022.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for un- supervised visual representation learning. In CVPR, pp. 9726-9735. Computer Vision Foundation / /EEE, 2020.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurIPS, 2021.
Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. Cosqa: 20, 000+ web queries for code search and question answering. In ACL/J/CNPP (1), pp. 5690-5700. Association for Computational Linguistics, 2021.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search. CoRR, abs/1909 09946,996 2019.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language code in programmatic context. In EMNLP, pp. 1643-1652, Association for Computational Linguistics, 2018.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547, 2019.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu. Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6789-6781. Association for Computational Linguistics, 2020.
11

---

Under review as a conference paper at ICLR 2023
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4929-4952, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/w1/2021. htwp:///w/.2...ddnnnneeemmm//// findings-emmlp.422.. URL httpp:::/aclaaaahhll.ogo.oo/ orgph/2000org/22111
Findings em..p. 424
Hung Le. Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. CoRR, abs/2207.01700, 2022.
Junnan Li, Ramprasaath R. Selvaraju. Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu-Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS, pp. 9694-9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping manguage-image pre- vre-training for unified vision-language understanding and generation. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 1 12900. PMLR, 2022a.
Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago. Thomas Hubert, Peter Choy, Cy- prien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. CoRR, abs/2203.078 2022b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CORR, abs/1907. 1692, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster). Open- Review.net, 2019.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation. In NeurlPS Datasets and Benchmarks, 2021.
Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Oleksandr Polozov, Christopher Meek, Dragomir R. Radev, and Jianfeng Gao. Learning from self-sampled correct and partially-correct programs. CoRR, abs/2205514318, 2022.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi. Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. CoRR, abs/2203. 13474, 2022.
Changan Niu, Chuanyi Li, Vincent Ng. Jidong Ge, Liguo Huang, and Bin Luo. Spt-code: Sequence- to-sequence pre-training for learning source code representations. In ICSE, pp. 1-13. ACM, 2022.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02/55 2022.
Md. Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Retrieval augmented code generation and summarization. In EMNLP (Findings), pp. 2719-2734. Association for Computational Linguisties, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:1 10:14:171144. 2020.
12

---

Under review as a conference paper at ICLR 2023
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In KDD, pp- 3505-3506. ACM, 2020.
Veselin Raychev, Pavol Bielik, and Martin T. Vechev. Probabilistic model for code with decision trees. In OOPSIA, pp. 731-747. ACM, 2016.
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. CoRR, 0bos/209.11027,, 2020.
Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gökhan Tür, and Prem Natarajan. Alexatm 20b: Few-shot learning using large-scale multilingual seq2seq model. CoRR, 0obs/2208.01444 2022.
Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution, pp. 476-480, IEEE, 2014.
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose:
code generation using transformer. In ESEC/SIGSOFT FSE, pp. 1433-1443. ACM, 2020a.
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose: code generation using transformer. In ESEC/SIGSOFT FSE, pp. 1433-1443. ACM, 2020b.
M Tabachnyk and S Nikolov. MI-enhanced code completion improves developer productivity, 2022.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. CoRR, abs/2205.05131, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of-mooalty--xxppe CoRR, abs/2111.02358, 2021a.
Xin Wang. Yasheng Wang, Fei Mi, Pingyi Zhou, Yao Wan, Xiao Liu, Li, Hao Wu, Jin Liu, and Xin Jiang. Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation. arXiv preprint arXiv:2108 04556, 2021b.
Xin Wang, Yasheng Wang, Yao Wan, Jiawei Wang. Pingyi Zhou, Li,,, Hao Wu, and Jin Liu. CODE- MVP: Learning to represent source code from multiple views with contrastive pre-training. In Findings of the Association for Computational Linguistics: NAACI.2022, pp. 1066-1077 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/1/2022.
andings-naacl.co URL https: attpss//ccllatttoooogg wwwp:/oooooyyg//222...oo. /ttpg/2222.c2 windingsiiiiinn
80
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. Codet5; Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In EMNLP (I), pp- 8696-8708. Association for Computational Linguistics, 2021c.
Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079, 2020.
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulner- ability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems, 32, 2019.
13

---

Under review as a conference paper at ICLR 2023
Text-Code Contrastive
Text Code
Code
Text Generation
Matching
Generation
M-FFN
CG-FFN
TG-FFN
Feed Forward
NX
Nx
Cross Attention
Bi Self-Att
Encoder
Causal Self-Att
Increment list
Decoder
or def inc poc_list(list_):
Figure 4: Another view of Codx as encoder-decoder model with task-specific FFN experts: M-FEN for text-code matching loss, TG-FFN for text generation, and CG-FFN for code generation.
A FURTHER EXPLANATION OF CODET5MIX
A.1 ANOTHER VIEW OF CODETSMIX
The mixture architecture CodeT5Mix can be interpreted from another view as a unified encoder- decoder with spek-sp FFN experts in shared decoder, which is illustrated in |Fg.. Note that while we introduce extra parameter cost of two FFN layers compared to standard encoder- decoder model during pretraining, only FFN layer is activated when finetuning downstream task, thereby incurring no additional computational cost in fact.
specific
A.2 TEXT-CODE CONTRASTIVE LEARNING AND MATCHING PRETRAINING TASK
To expose the model
diverse set of pretraining data, employ stage-wise pretraining process to first train CodeT5Mix on large-scale code-only data with span denoising and causal lan- guage modeling (CLM) tasks, then train on smaller set of text-code bimodel data using text-code contrastive learning, matching, and CLM tasks. Below, we provide detailed formulas for text-code contrastive learning and matching tasks at the second-stage pretraining on text-code pairs.
Text-Code Contrastive Learning activates bimodal encoder to learn better unimodal (text/code) representations by computing a similarity score such that parallel text-code pairs have higher scores. Given a text and a code C, we first learn representations h' for text and he for code C' by mapping the [CLS] embeddings to normalized lower-dimensional (256-d) representa- tions from the bimodal encoder. Given a batch of N text-code pairs, we obtain text vectors (h$ m and code vectors {h N to compute text-to-code and code-to-text and similarities:
$
h
h
Si.
c2t
h h
220 (T)
N Σn exp ////T) p (C)
exp
122// /T) (
exp (s /T)
Σn exp (s /T)
where 22e represents text-to-code similarity of text of i-th pair and code of j-th pair, and s the code-to-text similarity, 7 is learned temperature parameter. (T) and (C² are the softmax- normalized text-to-code and code-to-text similarities for the i-th text and code.
Let (T) and (C) denote the ground truth one-hot similarity, where negative pairs have a probability of () and the positive pair has probability of 1. The text-code contrastive loss from
14

---

Under review as a conference paper at ICLR 2023
corpus D of text-code pairs is defined as the cross-entropy H between p and y:
1 (llc) (T)) + Hy (C), ( (C))
Leontra
(3)
Text-Code Matching activates the bimodal matching decoder to predict whether a pair of text and code is positive (matched) or negative (unmatched). We employ the output embedding of the [ECS] token as the fused bimodal representation for a text-code pair (T' C). Followed by linear layer and softmax, we compute two-class probability pmateh (T, C) and define the matching loss:
Lmatch E(r,C)-D(H metch (T,C), C), match (T, CI,
(4)
where y match (T, о) is 2-dimensional one-hot vector representing the ground-truth label.
B PRETRAINING
B. PRETRAINING DATASET
We enlarge the pretraining dataset of CodeSearchNet (Husain et al. 2019 with the recently released GitHub Code datase f
We select nine PLs (Python, Java, Ruby, JavaScript, Go, PHP, C, C++, C#) and filter the dataset by preserving only permissively licensed code") and files with 50 to 2000 tokens. Besides, we filter out the overlapped subset with CodeSearchNet and other downstream tasks by checking their GitHub repositories. Note that although we employ the duplicated data version in which duplicates filtered out based on the exact match ignoring the whitespaces, there might be some potential remaining duplicates. However, we do not expect any remaining duplication will impact model performance significantly. We the CodeTT5 tokenizer and the resulting multilingual dataset has 51.5B tokens, more than 50x larger than CodeSearchNet.
Table 8: Data statistics of both unimodal and bimodal pretraining data
Dataset
Language
# Sample
Total size
Ruby
2,119.741
JavaScript
5,856.984
Go
1,501,673
Python
3,418.376
Ours
Java
10,851,759
37,274,876 files
PHP
4,386,876
C
4,187,467
C++
2,951,945
C#
4,119.796
Ruby
49,009
JavaScript
125,166
Go
319,132
1,929.817 text-code
CodeSearchNet
Python
453,772
pairs function level
Java
457,381
PHP
525,357
We report the data statistics of both code-only and text-code pretraining datasets in Table 8 From the table, we can see that our curated dataset from GitHub code has much larger data size the file level than the CodeSearchNet bimodal data at the function level, allowing our model to learn rich representation the first stage of pretraining. Different from CodeT5 (Wang et al.] (021c) that employs both unimodal and bimodal data in CodeSearchNet Husain et al. (2019), we only employ its bimodal subset for the second stage pretraining of our C5de which aims to adapt our model to better support text-code related tasks like text-to-code retrieval and generation. For data tokenization, we employ the code-specific subword tokenizer as described by Wang et al. (202cc).
https: : httggiugiiiia waw.cee.acccccoooooo wtt.c////aaata.e. datasatss//aaaaa htw.aaaeeeeoeppprrotgggigahaaaaa Perrisssive licenses: "mit", "apache-2" "hsd-3-clause" "bsd-2-clause", "cc0-1.0". "unlicense", "isc"
15

---

Under review as a conference paper at ICLR 2023
Table 9: Data statistics of the CSN dataset (Husain et
2019
Language
Training
Validation
Test
#Candidates
Go
167,288
7,325
8,122
28,120
Java 164,923
5,183
10,955
40,347
JavaScript
58,025
3,885
3,291
13,981
PHP 241,241
12,982
14,014
52,660
Python 251,820
13,914
14,918
43,827
Ruby 24,927
1,400
1,261
4,360
B.2 PRETRAINING DETAILS
We pretrained a Code TMMi-bsseas (220M) and CodeT5Mix-large (770M) from scratch following T5's architecture (Raffel al., 2020) on cluster with 16 A100-40G GPUs on Google Cloud Platform. Note that we introduce task-specific FFN experts in decoders, where each FFN expert incurs insignificant extra parameter costs 28M and 101M for the base and large model respec- tively. These insignificant extra parameter costs can be waived when applied a specific task as we only activate one specifcc FFN expert. We adopt a stage-wise pretraining process to pretrain CodeT5Mix first the large-scale unimodal dataset and then the smaller bimodal dataset.
In the first stage, we warm up the model with the span denoising task for 10k training steps, and then joint training with another a linear decay learning rate (LR) scheduler with peak learning rate of 2e-4 and set the batch size 2048 for denoising and 512 for CLM. To prepare the input and output data, we set the maximum length to 512 for the denoising task, and set the maximum length to 768 and 600 for source and target sequences for the code completion CLM, and 1024 for the decoder-only generation CLM.
CLM tasks with equal probability for 100k steps. We employ
In the second stage, we jointly optimize four losses of contrastive learning, matching, and two CLM losses with equal weights for 10 epochs with batch size of 256. We employ peak learning rate of lc-4 and set the maximum sequence length to 420 and 128 for code and text sequences. For all experiments, we employ an AdamW optimizer (Loshchilov Hutter 2019) with a 0.1 weight decay. We also employ the DeepSpeed's ZeRO Stage 2 (Rasley et 2020) with mixed precision training FP16 for training acceleration.
C FINETUNING TASKS
C.I TEXT-TO-CODE RETRIEVAL
Text-to-code retrieval (or code search), the task of finding the best code sample that most rele- vant to a natural language query. from collection of code candidates. We experiment CodeT5Mix with three major benchmarks: CodeSearchNet (CSN) (Husain et al., 2019). CosQA (Huang et al.) 2021], and AdvTTest (Lu et al. 2021). CSN consists of six programming languages in total, and the dataset is curated by filtering low-quality queries through handcrafted rules, following (Guo et al.) 2021). For instance, an example handcraft rule is filter examples in which the number of tokens in query is shorter than 3 or more than 256. The resulting dataset statistics can be seen in Table 9
CosQA and AdvTest two related benchmarks that derived from the CSN data. Specifically, instead of natural language queries, CosQA uses logs from Microsoft Bing search engine as queries, each of which is annotated by 3 human annotators (Huang et al.) (021). AdvTest is created from the Python split of the CSN data but the code samples are normalized with obfuscated variable names to better evaluate the understanding abilities of current models.
For training, we set the maximum sequence to 350 and 64 for code and text. We set the learning rate as 2e-5 and finetune the model for 10 epochs. We employ distributed training using Pytorch DistributedDattaaralll on 8 A100s and the total batch size is 64, For momentum encoders, we maintain a separate text/code queue with a size of 57600, and allow the matching decoder to retrieve 64 hard negatives from the queues for hard negative mining.
https://pytorchh.org/
16

---

Under review as a conference paper at ICLR 2023
C.2 DEFECT DETECTION
Defect detection is the task of classifying whether a code sample contains vulnerability points or not. We adopt the defect detection benchmark from CodeXGLUE (Lu al. 202T which curated data from the Devign dataset (Zhou et al. (019)). The dataset contains in total more than 27,000 annotated functions in programiiing language. All samples are collected from popular open-source projects such QEMU and FFmpeg. We follow Lu et al. (2021) and adopt 80%/10%
of the dataset as the irining/validatiionnest split. For training, we set the learning rate as 2e-5, the batch size as 32,
and the max sequence length as 512 to finetune the model for 10 epochs.
C.3 CLONE DETECTION
The task of clone detection aims to detect whether any two code samples have the same functional- ity or semantics. We conduct experiments using the clone detection benchmark from CodeXGLUE
(Lu et al., 202I). The benchmark is curated from the BigCloneBenchmark dataset (Svajlenko examples for train- ing/ valivaaioooi/sssttttt splits respectively. All samples are categorized into 10 different functionalities.
al. (2014) and the resulting curated data consists of 11.724/441.32884116
To finetune CodeTSMix on this task, we set the learning rate as 2e-5, the batch size as 10, and the max sequence length as 400. We use the Adam optimizer to finetune the model for epochs.
C.4 CODE COMPLETION
In code completion, a model is given a source sequence, containing partial code sample, and is required to generate the remaining part of the code sample. We conduct experiments on line-level code completion. using two major benchmarks: PY150 (Raychev et al. (216)) and Github Java Corpus (Allamanis & Sutton 2013).
PY150 (Raychev et al. 2016) consists of 150,000 Python source files collected from Github. Among these samples, et al. (2021) selected 10,000 samples from different files from the test set 150 and then randomly sampled lines to be predicted for the code completion task. The average numbers of tokens the source sequence and target sequence are 489.1 and 6.6 respectively.
Github Java Corpus (Allamanis & Sutton 2013) contains over 14,000 Java projects collected from GitHub. Similarly to PYI50, Lu et al. (2021) selected 3,000 samples from different files from the test set of the dataset and randomly sampled lines to be predicted for the code completion task. The average numbers tokens in the source and target sequence are 350.6 and 10.5 respectively.
For both tasks, set learning rate as 2e-5 and batch size 32, and finetune the model for 30 epochs. We set the maximum sequence length of 1024 for the decoder. During inference, we employ beam search with beam size of 5.
C.5 CODE SUMMARIZATION
Code summarization is the task of generating a natural language summary of a code snippet. We use the task dataset from Code XGL.E (Lu et al. (00T)) which curated a code summarization benchmark from CSN data (Husain et al. 2019). The benchmark consists of six programming languages: Ruby, JavaScript, Go, Python, Java, and PHP. text-to-code retrieval tasks. For training, as 256 and 128, respectively. We use learning rate of 2e-5, the batch size as 64 for 10 epochs of finetuning. We set the beam size as 5 in inference.
the same clean version of CSN data that we use for set the maximum sequence length of source and target
C.6 CODE GENERATION
The task of code generation requires model to generate corresponding code snippet given natu- ral language description. We experiment on the Java ConCode benchmark (Iyer et al. 2018) which consists of 33,000 Java projects collected from Github. The benchmark has 100k/2k/2k samples in the training/validationnness split. Each code generation sample consists a natural language de- scription, code environment, and the corresponding code snippet. For training, set the maximum sequence length of source and target as 320 and 200. We use a learning rate of 2c-5, the batch size as 32 for 30 epochs of finetuning. We the beam size in inference.
17

---

Under review as a conference paper at ICLR 2023
Natalia sold clips to 48 of her friends in April, and then she many clips May. many clips did Natalla sell allogether in April and May?
Question:
Python Solution:
no 48 n1 2
Natalia sold 48/2 = <<48/2=24==24 clips May, Natalia #### 72
nD n1 n0
<<48+24=72>>72 clips allogether April and May:
Figure 5: Example of how convert natural language solution into Python program on GSM
Problem
Problem
Problem
Toulouse has twice as many sheap as Charleston. Charleston has 4 Scattio. How many sheep do Toulouse, Charieston, and Seattle have togather if Scattle has 20 shoop?
Janat's ducks lay eggs per day. She eats three for breakfasi every morning and bakes muftins for friands every with four. farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every
Eliza's rate per hour for the first 40 hours she works each week is $10. She also
times many
avartime pay 1.2 times her regular hourly If Eliza worked for 45 hours this week, how much are her carnings week?
remainder
farmers' market?
Generated Program
Generated Program
Generated Program
n0 n1
n2 =
= 16
n2
10 n0 nf
n3 2
=
n2
=0 no 11 =1=
10 nō *11
answer
I1
answer =11 '3
12 n3-11
=2
Figure Predictions of our model on GSM8K-Python
C.7 MATH PROGRAMMING
Math Programming is the task of solving maths-based problems with programming. Compared to conventional code generation tasks such as ConCode (Iyer et al.) 2018), this task focuses more computational reasoning skills. The problem descriptions in this type of task are also more complex than conventional code generation tasks. We employ two major benchmarks for this tasks: Python (Austin et al. (021) and GradeSchool-Math (Cobbe et al. 2021).
MathQA-
MathQA-Python (Austin et al. 2021 is developed from the MathQA dataset (Amini et al. 2019) where given a mathematical problem description in natural language, a system is required to solve this problem using domain-specific language. Austtn et al. (202T) translated these programs into Python programs and filtered for cleaner problems. In total, MathQA-Pytho contains almost 24,000 problems, including 19,209/2,88 222//1,222111811 ,883 samples for training/validdationnnes splits.
GradeSchool-Math (Cobbe et al.] 2021 (also known as GSM8K) has similar nature as MathQA. The benchmark focuses on problems with moderate difficulty that an average middle school student should be able to solve. In total, GSM data contains 8,500 problems, divided into 7,500 training and 1,000 testing problems. We translate the solution described in natural language to Python programs by following Austin et al. (2021) 1 in the construction of MathQA-Pythe converted 5,861 training samples. One conversion example can be found in Fig. For qualitative results of CodeT5Mix,
Finally, we successfully
provide model predictions in Fig. 6 and Fig.
For training, we set the maximum sequence length of source and target as 256 and 256 for MathQA- Python, and 246, 138 for GSM8k-Python. We use a learning rate of 2e-5, the batch size 32 for 30 epochs of finetuning. During inference, we employ the heam size as 5 to get pass@1 results. For pass@80 and pass&100, found they quite sensitive to the diversity of the generation. We employ nucleus sampling with temperature of 1.2 and top-p=0.95.
D RETRIEVAL--UGGENTTE CODE GENERATION
Developers often search for relevant code snippets from sources on the web such as GitHub or Stack- Overflow references to aid their software development process. Motivated by this behaviour,
18

---

Under review as a conference paper at ICLR 2023
Problem
Problem
Problem
A shopkeeper sold an article offering diecuni of 5% and earned profit been percentage of profit eamed discount hed been offered?
What will be the difference belween simple and compound
full siationery of lank that is right circuler cyfinder hes radiue of 100 feel and a height of stationary tank
What
feet. pumped an cyfinder uniil the truck lank is completely filed. if the truck tank has radius of faot and a height of 10 feet, how far (in foor) did the all level tank?
sum nO 14.0 n1 1000.0 n2
years?
right
no 5.0 n1 31.1
the 25
100.0
10.0
Generated Program
Generated Program
Genorated Program
m0=550
no 14,0
import 100,0
n1.=
1000.0
100.0
n2
t1 100.0 -D
10 no 100.0
r2 m.0
12- 10 100.0
[1 10 1.0
r3 1
13 (211
-2-m0 10
malhhpp "0012
answer 13 100.0
math.pi "22"2 n3 answer
11*"min(22,
i5
16 15 -n
answer = 16
Figure 7: Predictions of our model on MathQA-Python
explore retrieval-uugmented code generation setting, where given natural language description, a retriever first retrieves similar candidates in a search codebase and then augments the input for the generator to produce the target code. Such cetrieval-augmente generation (or retrieve-then generate) paradigm has been widely used in open-domain question answering (Karpukhin et al.) 2020 in NLP and recently extended some code-related tasks such code generation and summa- rization Parvez et al., (021) with significant improvements. As our CodeT5Mix is capable of both retrieval and generation, can be seamlessly adapted as a unified uitrivall-augmente
generator.
This can bring unique benefits such as less computational cost compared to prior work that employs a different retriever and generator. We evaluate M5de on two Java and Python code generation datasets from the CodeXGLUE Lu et al. (202T) benchmark following Parvez et al. 202TT.
Specifically, we leverage the bimodal encoder to encode the code snippet in the retrieval base and build a search index with the faiss library (Johnson et al.] 2019). The search index is set of representations (of 256 dimensions) for all the code snippets in the retrieval codebase. Let (x4, v:) denote one training instance where is the input text description and y, is the corresponding target code snippet. we employ the same bimodal encoder to obtain the embedding of x, and retrieve top-k similar code samples from the search base using the L-2 similarity metric, with k being hyperparameter. We ensure that the training example's target string (y)) is not present in any of these k retrieved samples.
After retrieving these top-k relevant code samples, we combine them with a special token [SEP] and concatenate it to the end of the source input w;. Unlike Parvez al. (2021), we do augment doestrings or text descriptions and only augment the code snippet for simplicity. We then finetune CodeT5Mix on this augmented dataset. During inference, we retrieve similar code samples from the search base and augment these to input z;. For training, we set the maximum sequence length of source and target as 600 and 320. We use a learning rate of 2e-5, the batch size as 32 to finetune the model for 10 epochs. We set the beam size as 5 in inference.
We provide case study in Fig. 8 for qualitative evaluation, where retrieval- augmeeted generator predicts the ground-truth function while both retriever and generator fail to do so. We further conduct an ablation study to analyze the effects of top-k retrievals in Table 10 We found that increasing the number of retrievals can boost model performance which becomes saturated when k=5. This saturation is due to the maximum sequence length of 600, which might not be able to accommodate large number of retrieved code samples.
E ADDITIONAL EXPERIMENTAL RESULTS
We provide the full results of code summarization in Table 11 and full ablation results of code summarization in Table 12
19

---

Under review as a conference paper at ICLR 2023
Downloade JTTP resource from uel and save to dest. Capable of dealing with conpewxzed content.
Generate
vish open foxak, "wb") for chink in tear tearilanods urlcper (owl) -reddsto2t, br Ayziberchunks t.riuhh
domiced
Retrieve
serrive-tteengenerae
def di Fileteck, deat, Sunk_sizerit33p Ieport ur311b3
der doenlead, este cuni, desti:
pequest ueilibz. Dequue (uel) request- Seader ('Acopptenceddee "gxip"] spenes url:ib2 peaponse
boko urllib3. (c
htrp. cunbeniefulae)
opene lequuas))
opener
with dest..ooe cosst.openl ('sb") whils Yruus
data responde if zumponas Seadaca, streae = Beoringt yTo. Ser www.eggt (detaa quippeee e quip Caipelle (iliiooaatteeun ggt gaipper-resdd
d'eententt ben.riinnneneiiinnn 11) 'gzip's
data = E.dead/churk, sibel If data Hone or Tenidaca)
beeak
opun jounk, Evolesett
Figure 8: One example of retrieval-augnee -agmented code generation in Python dataset, where + denotes the concatenation operation. With the help of the top retrieved code, our Code ModeeT55xx-bbeee retrieval- augmented generator produces the correct function.
Table 10: Effects of varying top-k retrievals in retreeeal-augmmented code generation
Java
Python
Model
EM
B4
CB
FM
B4
CB
REDCODER-EXT (top-10)
10.21
28.98
33.18
9.61
24.43
30.21
MoxeTSMix-base (top-l)
11.66
33.83
40.60
11.83
31.14
36.39
top-2
11.57
33.26
40.74
11.78
31.21
36.58
top-3
12.29
33.10
41.71
12.48
30.92
37.31
top-4
12.42
32.08
41.94
12.73
30.40
37.60
top-5
13.02
32.42
42.28
12.93
30.52
37.87
top-10 10
12.86
31.38
42.24
12.84
29.79
37.79
Table 11: Full results (smoothed BLEU-4) on code summarization
Model
Ruby
JavaScript
Go
Python
Java
PHP
Overall
RoBERTa
11.17
11.90
17.72
18.14
16.47
24.02
16.57
CodeBERT
12.16
14.90
18.07
19.06
17.65
25.16
17.83
PLBART
14.11
15.56
18.91
19.30
18.45
23.58
18.32
CoTexT
14.02
14.96
18.86
19.73
19.06
24.58
18.54
UniXcoder
14.87
15.85
19.07
19.13
20.31
26.54
19.30
CodeT5 CodeeS-base
15.24
16.16
19.56
20.01
20.31
26.03
19.55
CodeTT--arge
15.58
16.17
19.69
20.57
20.74
26.49
19.87
-Mx
15.51
16.27
19.60
20.16
20.53
26.78
19.81
laxx
15.63
17.93
19.64
20.47
20.83
26.39
20.15
Table 12: Full ablation results (smoothed BLEU-4 scores) code summarization
Model
Ruby
JavaScript
Go
Python
Java
PHP
Overall
CodeT5MMxx-bse
15.47
16.27
19.44
19.56
20.23
26.19
19.53
no match decoder
15.21
15.84
19.43
199.44
20.43
26.57
79.49"
no code/text decoder
separate decoder
15.01
16.22
19.33
19.31
20.16
26.23
19.38
shared decoder
15.39
16.06
19.27
18.89
19.97
25.84
19.24
20


Reviews:
[{"review_id": "wKT23-28mu", "reviewer": "Reviewer_stVy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-task training approach based on CodeT5 for solving multi-modality code related tasks. The model has multiple general and specialized components based on the task. The training contains two stages. In the first stage, the model trains on span denoising and CLM objectives. In the second stage, the model trains on contrastive learning, matching and generation objectives. Evaluation results show that the proposed model outperforms CodeT5 in similar conditions. ", "summary_of_the_review": "I lean towards an acceptation for this paper given the extensive study of model performance in experiments. However, on the significance and impact of the contribution, the modification to the pre-training objectives seem to be a natural extension (by training on multiple code related tasks) of the CodeT5 objective.", "strength_and_weaknesses": "Strength:\n• The thorough evaluation shows that the proposed CodeT5Mix has strong performance against the CodeT5 baseline \n\nWeakness:\n• The contribution can be seem as an incremental extension to CodeT5 for enhancing the training with multi-task objectives. For the evaluation results, comparing to CodeT5 baseline, the gain is more incremental rather than break-through. Such a gain is largely expected when extending the original masked span prediction loss of CodeT5", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X_Pw0lKxODu", "reviewer": "Reviewer_EgG1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper uses a mixture of encoder-decoder transformers for code understanding and generation tasks. It in addition uses multiple pretraining objectives to train the mixture. The following pretraining objectives are standard and have been explored in previous works -- in particular (1) span denoising, (2) causal language . They also have (3) contrastive learning though which they apply on text-code pairs which is novel. They have a (4) text matching objective where they try and predict whether a particular text-code pair is a positive or negative. They also have the pretraining task of (5) predicting code from text and vice versa. For task specific decoders, they do weight sharing\n\nThey show good improvements for multiple baselines across multiple tasks. They also do an ablation study where they individually remove (4) and (5) and show minor drops in performance. They also have an ablation with and without weight sharing.", "summary_of_the_review": "The ideas have some marginal novelty but the ablations are not thorough and seem to indicate that the novel objectives at best have very minor benefits. Motivation for the matching objective is not provided and error bars are not given. It's also not clear what the model sizes of the baseline models as well as training data for them was. It is also unclear whether the test data was present or not in the training data.", "strength_and_weaknesses": "Strengths:\n\nThe idea of exploring multiple pretraining objectives as well as weight sharing for different tasks is interesting. The evaluation is done on multiple tasks which is good.\n\nWeaknesses:\n\n(1) Causal language modeling and masked language modeling have been shown to be a very strong baselines for code generation and understanding tasks. It's unclear whether (3), (4), (5) add much. Indeed the ablation they do indicates that at least (4), (5) as well as weight sharing do not make a big performance difference. Thus we don't have good evidence of performance improvement from the more novel pretraining objectives.\n\n(2) No error bars are given making it hard to assess the significance of improvements, especially for ablation experiments where the difference is minor\n\n(3) Perhaps I missed it but I do not see the model sizes for the baseline models mentioned. Do they have the same number of parameters as the CodeT5Mix models? If not, then a smaller parameter count for those models would also account for the performance difference given the known language model scaling laws.\n\n(4) What was the training data for the baseline models?\n\n(5) Again perhaps something I missed, but I do not see any mention of the training data being deduped against the test data. Was that done?\n\n(6) It is not clear to me why the text matching objective is necessary given the contrastive objective? In both cases you are trying to make the embeddings of the matching pairs closer and push the non-matching pairs further. Motivation for this should be provided.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "SPfHWRlUzl", "reviewer": "Reviewer_3yBT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a mixture of encoder-decoder transformers, called CodeT5Mix, for code understanding and generation. It also proposes a number of pretraining tasks including denoising, causal LM, constrastive loss, matching to pretrain the CodeT5Mix. Extensive experiments on various tasks have been well performed.", "summary_of_the_review": "Overall, I think this is a decent paper with novel method proposed and well-performed experiments. However, I have concerns on the proposed mixture of decoders, and would like to see more experiments to compare with simple baseline to justify the advantanage of mixture of decoders.", "strength_and_weaknesses": "Strength:\n- The paper is well-written. The method is well explained with examples. Experiment results are extensive with analysis.\n- The proposed method provides multiple encoder/decoders to solve different tasks in a mixture of expert manner, and could have the potential to avoid task interference.\n\n\nWeakness:\n- It is not clear to me that how many encoders and decoders there are in the proposed CodeT5Mix. Inferring from Section 3.1, there should be one encoder, three decoders (one for matching decoder, one for text generation decoder and one for code generation decoder). And 1st stage of pretraining is to pretrain one encoder-decoder, from 2nd stage, the decoders are then separated out. Is my understanding correct? Please provide clarifications on the architectures of the proposed CodeT5Mix.\n- One simple baseline to CodeT5Mix is that, we can use three separate encoder-decoders initialized from 1st stage pretraining for subsequent stage of pretraining/finetuning tasks. Could the authors demonstrate the benefit of CodeT5Mix by comparing the results of this simple baseline?\n- First stage of pretraining objectives seems the same as CodeT5. For later stages, how large gain can we get by using separate decoders compared with using only one decoder but with different sequence formatting for different tasks? I.e. are the separate decoders necessary? The paper also lacks this comparison.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MzfoyqN8sHv", "reviewer": "Reviewer_eBFF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed CodeT5-Mix - a pretrained model consisting of mixture of encoders and decoders for code understanding and generation tasks. The authors argue that encoder-decoder based pretrained models such as PLBART, CodeT5 performs suboptimally in different tasks. To overcome such problem, the authors proposed mixing and matching different encoders and decoder for different information modalities (i.e., code, text) and flexibly combine them based on the specifics of the finetuning tasks. The ensemble of the encoder-decoders is trained jointly based on several training objectives. Some of these modules (the authors talked about the decoders) share weights. The authors performed thorough evaluation on a large number of downstream tasks and showed SOTA performances. ", "summary_of_the_review": "While the idea of the paper is innovative and the results are impressive, there are some flaws in the explanation and scholarly arguments (please see the details in the evaluation above). \n\n", "strength_and_weaknesses": "**Strengths**\n1. The idea of **Ensemble of different models** and sharing the weights between them is interesting and innovative. Each of these models are equipped to deal with a specific information source/task. \n2. Very thorough evaluation on downstream tasks. \n3. Impressive results.\n4. Fairly well written paper. \n5. Trained two different sized models. \n\n**Weakness**\n1. While I agree with the motivation of the paper, there are few things that are confusing. First, the authors argue that Encoder-Decoder models are suboptimally equipped to do autoregressive generation tasks. That argument depends on what is the pretraining objective of such models. For instance, I agree that CodeT5 does not teach to generate the whole code autoregressively, but PLBART, SPT-code [1], NatGen [2] they are equipped with autoregressive generation capacity. The authors should compare with these models. Moreover, I am not understanding what part of CodeT5-mix is overcoming the problem the authors argued that PLBART, SPT-code, NatGen cannot. I don't just mean comparing the results, but also comparing on a conceptual level. \n2. I commend the authors on the impressive results throughout the paper. However, the retrieval augmented generation part looks a bit disconnected from the paper. I believe it could be better connected to the paper since it shows off the capacity of both code understanding (retrieval) and generation. Also, Parvez et al. showed evaluation on Retrieval Augmented Code Summarization ([table 4 from here](https://arxiv.org/pdf/2108.11601.pdf)). Why not show such evaluation using CodeT5-mix. Since the input of summarization is code, this evaluation will really show CodeT5-mix's understanding capability. \n3. It is not clear whether we need multiple decoders. What would happen if you trained a single encoder-decoder with all the training objectives you have on the same pretraining dataset you have? I think you have shown something very close in the ablation, but the options in ablation are not clear. In light with that, I request the authors clearer explanation on the ablation. \n\n\n**References**\n\n[1] Changan Niu, Chuanyi Li, Vincent Ng, Jidong Ge, Liguo Huang, and Bin Luo. 2022. [SPT-code: sequence-to-sequence pre-training for learning source code representations](https://arxiv.org/abs/2201.01549). In Proceedings of the 44th International Conference on Software Engineering (ICSE '22). \n\n[2]  Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar Devanbu, and Baishakhi Ray. 2022. [NatGen: Generative pre-training by\" Naturalizing\" source code](https://arxiv.org/abs/2206.07585). In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE' 22).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8w4H4Lchqqe", "reviewer": "Reviewer_ZFgG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A pretraining strategy for models performing different tasks on code is presented. It follows the idea of implementing multiple related architectures using partial weight-sharing. In particular, the core architecture is an encoder-decoder Transformer model. Different tasks share all model parameters apart from the feed-forward layers in the Transformer (i.e., the weights for computing attention scores are shared). Pretraining proceeds by first considering causal language modeling and span denoising tasks. In a second stage, it employs the code/text pairs from the CodeSearchNet dataset to train on four related tasks (contrastive learning, matching, text->code and code->text). Finally, the model is fine-tuned on different downstream tasks.\nIn an evaluation, the model shows modest improvements over baselines on a range of tasks from the literature. An additional ablation study is performed to analyze the effect of different choices in pretraining/parameter sharing.", "summary_of_the_review": "A reasonably-well executed paper with minor problems in clarity. In practice, I have doubts that the proposed method is going to be adopted by a wider set of researchers, given the complexities of managing the proposed parameter-sharing scheme and the modest gains. I'd tend to accept this because there's nothing _wrong_ with the paper, but I would also not champion it as an important contribution that needs to be presented at ICLR.\n\nUpdate after discussion period: while my overall opinion hasn't shifted substantially, I'm now leaning towards rejecting the paper, given that the validity of the empirical results remains somewhat in doubt given the lack of deduplication of the data and the numerous small concerns I and the other reviewers voiced on the presented experiments (primarily the lack of error bars, but also a lack of deeper investigation of the architecture and objective choices).", "strength_and_weaknesses": "* (+) Substantial experiments covering a wide variety of different tasks\n* (~) Improvements over simpler baselines are quite modest\n* (-) No experiments validating core architectural choices (e.g., using task-specific expert layers vs. using same number of parameters in fully shared model)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject