Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed CodeT5-Mix - a pretrained model consisting of mixture of encoders and decoders for code understanding and generation tasks. The authors argue that encoder-decoder based pretrained models such as PLBART, CodeT5 performs suboptimally in different tasks. To overcome such problem, the authors proposed mixing and matching different encoders and decoder for different information modalities (i.e., code, text) and flexibly combine them based on the specifics of the finetuning tasks. The ensemble of the encoder-decoders is trained jointly based on several training objectives. Some of these modules (the authors talked about the decoders) share weights. The authors performed thorough evaluation on a large number of downstream tasks and showed SOTA performances. 
strength_and_weaknesses: **Strengths**
1. The idea of **Ensemble of different models** and sharing the weights between them is interesting and innovative. Each of these models are equipped to deal with a specific information source/task. 
2. Very thorough evaluation on downstream tasks. 
3. Impressive results.
4. Fairly well written paper. 
5. Trained two different sized models. 

**Weakness**
1. While I agree with the motivation of the paper, there are few things that are confusing. First, the authors argue that Encoder-Decoder models are suboptimally equipped to do autoregressive generation tasks. That argument depends on what is the pretraining objective of such models. For instance, I agree that CodeT5 does not teach to generate the whole code autoregressively, but PLBART, SPT-code [1], NatGen [2] they are equipped with autoregressive generation capacity. The authors should compare with these models. Moreover, I am not understanding what part of CodeT5-mix is overcoming the problem the authors argued that PLBART, SPT-code, NatGen cannot. I don't just mean comparing the results, but also comparing on a conceptual level. 
2. I commend the authors on the impressive results throughout the paper. However, the retrieval augmented generation part looks a bit disconnected from the paper. I believe it could be better connected to the paper since it shows off the capacity of both code understanding (retrieval) and generation. Also, Parvez et al. showed evaluation on Retrieval Augmented Code Summarization ([table 4 from here](https://arxiv.org/pdf/2108.11601.pdf)). Why not show such evaluation using CodeT5-mix. Since the input of summarization is code, this evaluation will really show CodeT5-mix's understanding capability. 
3. It is not clear whether we need multiple decoders. What would happen if you trained a single encoder-decoder with all the training objectives you have on the same pretraining dataset you have? I think you have shown something very close in the ablation, but the options in ablation are not clear. In light with that, I request the authors clearer explanation on the ablation. 


**References**

[1] Changan Niu, Chuanyi Li, Vincent Ng, Jidong Ge, Liguo Huang, and Bin Luo. 2022. [SPT-code: sequence-to-sequence pre-training for learning source code representations](https://arxiv.org/abs/2201.01549). In Proceedings of the 44th International Conference on Software Engineering (ICSE '22). 

[2]  Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar Devanbu, and Baishakhi Ray. 2022. [NatGen: Generative pre-training by" Naturalizing" source code](https://arxiv.org/abs/2206.07585). In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE' 22).

clarity,_quality,_novelty_and_reproducibility: **Clarity**

There are few questions that needs to be clearly answered in the paper. 
1. What is the justification for text-code matching decoder? Can't you do that task using the [CLS] token representation from encoder?
2. Why two stage training? What happens when you combine all the objectives in on stage (say, stage 2)? Specially, since two CLM objectives are also in 2nd stage? If you really insist on two stage training, what is the point of using those two CLM is 2nd stage, those tasks are already sufficiently trained in first stage?
3. How are the losses combined in the second stage?
4. What are different baselines for the ablation studies?

**Quality and Novelty** 
1.  **+** Such an ensemble type model for code is novel. 
2.  **+** Main concepts of the paper are well written and is an easy read. 
3.  **-** Empirical results are not very well explained to show how CodeT5-mix is solving the problem the authors stated in their introductory arguments. 

**Reproducibility**
While the authors provide some detail of the pretraining and finetuning, the code and data are not made public. (the authors promised to do so, upon acceptance, perhaps)
 
summary_of_the_review: While the idea of the paper is innovative and the results are impressive, there are some flaws in the explanation and scholarly arguments (please see the details in the evaluation above).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper uses a mixture of encoder-decoder transformers for code understanding and generation tasks. It in addition uses multiple pretraining objectives to train the mixture. The following pretraining objectives are standard and have been explored in previous works -- in particular (1) span denoising, (2) causal language . They also have (3) contrastive learning though which they apply on text-code pairs which is novel. They have a (4) text matching objective where they try and predict whether a particular text-code pair is a positive or negative. They also have the pretraining task of (5) predicting code from text and vice versa. For task specific decoders, they do weight sharing

They show good improvements for multiple baselines across multiple tasks. They also do an ablation study where they individually remove (4) and (5) and show minor drops in performance. They also have an ablation with and without weight sharing.
strength_and_weaknesses: Strengths:

The idea of exploring multiple pretraining objectives as well as weight sharing for different tasks is interesting. The evaluation is done on multiple tasks which is good.

Weaknesses:

(1) Causal language modeling and masked language modeling have been shown to be a very strong baselines for code generation and understanding tasks. It's unclear whether (3), (4), (5) add much. Indeed the ablation they do indicates that at least (4), (5) as well as weight sharing do not make a big performance difference. Thus we don't have good evidence of performance improvement from the more novel pretraining objectives.

(2) No error bars are given making it hard to assess the significance of improvements, especially for ablation experiments where the difference is minor

(3) Perhaps I missed it but I do not see the model sizes for the baseline models mentioned. Do they have the same number of parameters as the CodeT5Mix models? If not, then a smaller parameter count for those models would also account for the performance difference given the known language model scaling laws.

(4) What was the training data for the baseline models?

(5) Again perhaps something I missed, but I do not see any mention of the training data being deduped against the test data. Was that done?

(6) It is not clear to me why the text matching objective is necessary given the contrastive objective? In both cases you are trying to make the embeddings of the matching pairs closer and push the non-matching pairs further. Motivation for this should be provided.
clarity,_quality,_novelty_and_reproducibility: Clarity:

Key details about the baseline models and training data seem to be missing

Quality:

Error bars are not given

Novelty:

The text-code contrastive learning/matching objective is marginally novel but it's not clear whether any significant gains are realized from adding it. Furthermore I can't find the motivation for having the matching objective given that the contrastive learning objective is already there.
summary_of_the_review: The ideas have some marginal novelty but the ablations are not thorough and seem to indicate that the novel objectives at best have very minor benefits. Motivation for the matching objective is not provided and error bars are not given. It's also not clear what the model sizes of the baseline models as well as training data for them was. It is also unclear whether the test data was present or not in the training data.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A pretraining strategy for models performing different tasks on code is presented. It follows the idea of implementing multiple related architectures using partial weight-sharing. In particular, the core architecture is an encoder-decoder Transformer model. Different tasks share all model parameters apart from the feed-forward layers in the Transformer (i.e., the weights for computing attention scores are shared). Pretraining proceeds by first considering causal language modeling and span denoising tasks. In a second stage, it employs the code/text pairs from the CodeSearchNet dataset to train on four related tasks (contrastive learning, matching, text->code and code->text). Finally, the model is fine-tuned on different downstream tasks.
In an evaluation, the model shows modest improvements over baselines on a range of tasks from the literature. An additional ablation study is performed to analyze the effect of different choices in pretraining/parameter sharing.
strength_and_weaknesses: * (+) Substantial experiments covering a wide variety of different tasks
* (~) Improvements over simpler baselines are quite modest
* (-) No experiments validating core architectural choices (e.g., using task-specific expert layers vs. using same number of parameters in fully shared model)
clarity,_quality,_novelty_and_reproducibility: The paper is largely well-written, though some technical details are somewhat obfuscated. In particular:
* Sect 3.2.1 and 3.2.2 should state more clearly what encoder/decoder outputs are.
* Sect. 3.2.3 could profit from more clarity on parameter sharing arrangements: it starts by stating that the decoders share model parameters, but then walks this back. It's unclear what the weight-sharing scheme on the encoder side is, but from Fig. 3, it seems that all parameters are shared. The paper would be strengthened by an explanation on how the final scheme was decided on.

The experiment design seems largely solid, with three notes:
* The github-code data has substantial duplication issues (see https://twitter.com/miltos1/status/1497126435261083649). Have you done any work to mitigate these?
* The experimental results should state parameter counts for all models, to make it easier to disentangle size from performance.
* No error bars are reported, which is problematic given that differences between model results are often very, very small. This is especially noteworthy for the ablation results in Table 6.

Overall, the proposed pretraining multi-task method is not particularly novel, but the idea of performing limited parameter sharing for this in the code space is.

Notes:
* page 3: "Additionally, encoder-decoder models are not ideal for autoregressive tasks like code completion (Lu et al., 2021), in which existing SoTA methods are mostly based on decoder-only models": this may be an artefact of how code completion models are benchmarked (usually with context only "to the left"). Papers such as https://arxiv.org/pdf/2204.05999.pdf indicate that there are substantial differences in this setting, and encoder-decoder models support this case well. Industry reports such as https://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html also indicate that encoder-decoder models may be more suitable for the code completion setting. 
* Fig. 2 and 3 contain information that is not accessible to the colourblind (or reviewers working on greyscale copies of the paper). Please fix this this.
* Table 2: Any reason why the $95.2$ for UniXcoder is not bolded?
summary_of_the_review: A reasonably-well executed paper with minor problems in clarity. In practice, I have doubts that the proposed method is going to be adopted by a wider set of researchers, given the complexities of managing the proposed parameter-sharing scheme and the modest gains. I'd tend to accept this because there's nothing _wrong_ with the paper, but I would also not champion it as an important contribution that needs to be presented at ICLR.

Update after discussion period: while my overall opinion hasn't shifted substantially, I'm now leaning towards rejecting the paper, given that the validity of the empirical results remains somewhat in doubt given the lack of deduplication of the data and the numerous small concerns I and the other reviewers voiced on the presented experiments (primarily the lack of error bars, but also a lack of deeper investigation of the architecture and objective choices).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a mixture of encoder-decoder transformers, called CodeT5Mix, for code understanding and generation. It also proposes a number of pretraining tasks including denoising, causal LM, constrastive loss, matching to pretrain the CodeT5Mix. Extensive experiments on various tasks have been well performed.
strength_and_weaknesses: Strength:
- The paper is well-written. The method is well explained with examples. Experiment results are extensive with analysis.
- The proposed method provides multiple encoder/decoders to solve different tasks in a mixture of expert manner, and could have the potential to avoid task interference.


Weakness:
- It is not clear to me that how many encoders and decoders there are in the proposed CodeT5Mix. Inferring from Section 3.1, there should be one encoder, three decoders (one for matching decoder, one for text generation decoder and one for code generation decoder). And 1st stage of pretraining is to pretrain one encoder-decoder, from 2nd stage, the decoders are then separated out. Is my understanding correct? Please provide clarifications on the architectures of the proposed CodeT5Mix.
- One simple baseline to CodeT5Mix is that, we can use three separate encoder-decoders initialized from 1st stage pretraining for subsequent stage of pretraining/finetuning tasks. Could the authors demonstrate the benefit of CodeT5Mix by comparing the results of this simple baseline?
- First stage of pretraining objectives seems the same as CodeT5. For later stages, how large gain can we get by using separate decoders compared with using only one decoder but with different sequence formatting for different tasks? I.e. are the separate decoders necessary? The paper also lacks this comparison.
clarity,_quality,_novelty_and_reproducibility: This paper is well-writtern and is of high quality. The proposed method is novel. And I expect that the results should be easy to reproduce.

summary_of_the_review: Overall, I think this is a decent paper with novel method proposed and well-performed experiments. However, I have concerns on the proposed mixture of decoders, and would like to see more experiments to compare with simple baseline to justify the advantanage of mixture of decoders.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-task training approach based on CodeT5 for solving multi-modality code related tasks. The model has multiple general and specialized components based on the task. The training contains two stages. In the first stage, the model trains on span denoising and CLM objectives. In the second stage, the model trains on contrastive learning, matching and generation objectives. Evaluation results show that the proposed model outperforms CodeT5 in similar conditions. 
strength_and_weaknesses: Strength:
• The thorough evaluation shows that the proposed CodeT5Mix has strong performance against the CodeT5 baseline 

Weakness:
• The contribution can be seem as an incremental extension to CodeT5 for enhancing the training with multi-task objectives. For the evaluation results, comparing to CodeT5 baseline, the gain is more incremental rather than break-through. Such a gain is largely expected when extending the original masked span prediction loss of CodeT5
clarity,_quality,_novelty_and_reproducibility: The paper is written in a clear and organized way. There are some technical details such as the equation for each loss readers can’t find in the paper. However, it’s not critical for reading.
summary_of_the_review: I lean towards an acceptation for this paper given the extensive study of model performance in experiments. However, on the significance and impact of the contribution, the modification to the pre-training objectives seem to be a natural extension (by training on multiple code related tasks) of the CodeT5 objective.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper makes the following contributions:
- Presents a pretraining strategy for transformer-based models of code.
- An encoder-decoder Transformer model where different tasks share all model parameters apart from the feed-forward layers of the transformer.
- An empirical evaluation over a large range of tasks.

## Strengths and Weaknesses
(+) Substantial experiments covering a wide variety of different tasks.

(+) An interesting set of ideas wrt to weight sharing and pretraining objectives

(-) Empirical comparison often conflates the advantages one gets from the different pretraining objective, the weight sharing/separation, and the model architecture. Specifically:

* the total amount of compute and parameters is not always comparable as the task-specific experts regime introduces more weights that could have been used in a standard transformer architecture (e.g. to add an additional layer or increase the hidden size).
* Similarly, the evaluation seems to conflate the (dis)advantages of an encoder-decoder architecture, with decoder-only, or UniLM-style (UniXCoder) and pretraining objectives and weight sharing. In particular, the similar pretraining objectives and weight sharing could be used with decoder-only or Uni-LM-style models. Would the advantages observed in CodeT5Mix transfer to those models too (e.g., if weight sharing is introduced or the same pretraining objective is used)?


Given the above, I suggest to not to accept this paper at its current state. While it represents a substantial amount of work (and compute), the evaluation needs to be refined to ensure that the contributions of this work generalize effectively and can be used by others.

justification_for_why_not_higher_score: It is possible that a reader generalizes the findings of this work in a way that is not warranted. I believe that for this paper to be published in ICLR the issues in its evaluation need to be addressed first.
Decision: Reject