Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-task pretraining objective that outperforms and unifies prior objectives, it attempts to decouple the modeling architecture and the pretraining objective. The proposed objective shows strong performance gain over SOTA.

strength_and_weaknesses: Strength
This paper is well written, the experiments are extensive, and detailed ablation study is presented.

Weakness
While the experiments are extensive, there are questions about the presentation and the conclusion.
- relative performance comparison is not a good metric when the baseline is bad. It leads to 1k+ gain in table 4, skewing the averaging comparison.
- “The best decoder baseline model here is the Prefix-LM decoder model, which is about 10% worse than the T5 baseline.” While this statement is true on average, the performance difference is not true for all tasks, notably LM. It contradicts the universal claim (“ It is clear from these results that encoder-decoder models should be preferred over decoder-only models if and only if there is no concern about storage”) this paper is making.
- “RegardingPrefix-LM pre-training,  it is interesting that Prefix-LM actually outperforms the T5 span corrupt setup by+16.7%.” Is this finding contrary to Raffel et al., 2019? 
- Why are some numbers missing in table 6?

clarity,_quality,_novelty_and_reproducibility: The paper is well written, the experiments is extensive, and it improves over existing multi-task pretraining objectives. Code and model are publicly available.

summary_of_the_review: While the experiment is extensive, some of the strong claims are not supported by the experiment.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper investigates the effect of the concrete pretraining task (and, to a lesser extend), the model architecture, on the effectiveness of the resulting pretrained language representation model. The latter is measured in multiple ways, performing zero-shot, one-shot and normal finetuning experiments.

The paper further introduces Mixture-of-Denoisers (MoD), a pretraining objective that combines multiple pretraining objectives as well as *mode switching*, which associates downstream finetuning with specific ways of pretraining by introducing special tokens, indicating the structure of the (pretraining or finetuning) task to the model.

Comparing T5-style pretraining with GPT-3-style pretraining, the authors find that the former performs best. Their newly proposed model, UL2, however, outperforms the T5-style model in nearly all settings. Looking at the ablation studies, MoD seems clearly beneficial, while mode switching seems helpful (only) in some settings. 


strength_and_weaknesses: Strengths:
- The paper's idea is very clear and will be useful for other people in the field.
- The authors will release new pretrained models with strong performance. 
- I was initially worried that the authors wouldn't back up all their initial claims, but the experiments and ablation studies support the main arguments extremely well.

Weaknesses:
- The novelty (in the sense of *creativity*) of this paper is limited (but the insights are still useful!).
- There are a couple of typos, but nothing major.
- The ablation study regarding mode switching should be in the main part of the paper (as it's needed to back up one of the main contributions). 
clarity,_quality,_novelty_and_reproducibility: This work is basically incremental (combining existing pretraining objectives), but the paper does a good job at answering research questions regarding the effectiveness of pretraining objectives and model architectures, so I don't see this as a big problem. 
The writing is very clear.
summary_of_the_review: The paper asks clear research questions and answers them in a convincing way, so I believe that it should be accepted to the conference.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes the mixture-of-denoisers as the training objective for pertained language models. The mixture consists of two types of denoising pradigms of different noise configurations and a prefix language model-like next token prediction. Authors hint that each paradigm is suitable for a particular kind of downstream tasks, such as language understanding and generation. During training, special tokens are prepended to the input such that the paradigm can be switched during test time depending on the downstream tasks.
The authors compare UL2 against other objectives at a small scale (~335M parameters) and showed overall better performance, as well as scaling up to 20B parameters and demonstrated better zero-shot performance compared to notable large models like GPT-3.
strength_and_weaknesses: The authors compare UL2 against other objectives at a small scale (~335M parameters) and showed overall better performance, as well as scaling up to 20B parameters and demonstrated better zero-shot performance compared to notable large models like GPT-3.

The paper has a value in terms of the comprehensive comparison across objectives with relevant discussions, however, there are several concerns listed in the following:

1) Evaluation
The evaluation seems a bit arbitrary, also as the author written in appendix, "We arbitrarily select XSUM, ToTTo and SGD from the GEM benchmark". Is there a reason for this "arbitrary" selection instead of testing on more datasets from GEM? Also since the Rouge-L scores are quite low in all the experiments, it would be great to show some qualitative examples to demonstrate that the model generated results are actually meaningful. Because those metrics are not very representative when the score is very low.

The normalized percentage gain is not representative of the overall performance. Look at Table 2 and 3, the major relative gain all comes from the one-shot experiments, in particular, the ones with very bad baseline performance and achieve well above 100% relative performance improvements. However, if we instead look at the actual performance numbers on those tasks, it is also not in a reasonable range. Therefore, I think the gain over pervious models are over claimed. Additionally, the one-shot experiments would exhibit high variance based on the one-shot example selection, it would be great if the authors run with several different samples and report the overall result with the variance.

In addition, there are a couple of notably inconsistent numbers listed below.
- On one shot XSum, SC (i.e., T5, 335M) in Table 2 achieves 7.49 ROUGE-L, while a much larger T5-XXL 11B achieves 0.6 ROUGE-L in Table 6.
- PaLM 8B in Table 6 reports 4.5 ROUGE-2, while the PaLM paper (Table 16) reports 7.9 ROUGE-2. In fact “4.5" appearing in PaLM paper is for WebNLG (ru) task, not XSum.
- Abstract says “tripling the performance of T5-XXL” but UL2 has double the number of parameters. The comparison isn’t entirely fair.

(2) Utility of paradigm tokens are not demonstrated
Authors suggest that each denoising paradigm has a suitable kind of downstream tasks. In particular, it is said that R-denoiser is like T5 objective, and S or X-denoisers are more suitable for generating fluent texts. However, XSUM results in Table 8 suggest otherwise — T5-like R-denoiser is the most suitable. In addition, compare the results with different paradigm tokens and the ones without in table 8 indicates that the paradigm token has minimal (or negative) effect on these tasks, so it is unclear whether those tokens are actually useful.

(3) Technical Novelty
As I understand, UL2 combines different type of span noise with different level of noise and restrictions (e.g. random locations vs. only in the latter part of the sequence). The only thing that differentiate a bit from existing work is the paradigm token, however, the effectiveness of this token is also not clear. 



clarity,_quality,_novelty_and_reproducibility: It is not clearly written how different denoising scheme is combined during training, from the implementation in the appendix it seems that they are equally weighted, however, it is not clearly indicated in the main text. Also the exact model configuration for the smaller models are not specified, which would be hard for reproducing the results. 
summary_of_the_review: Please refer to Strength And Weaknesses
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a unified pre-training model framework for the problems of existing pre-training models, which are usually targeted at specific categories of problems and have poor performance in cross-task domains. First, the authors use R-Denoiser, S-Denoiser and X-Denoiser to define multiple pre-training tasks uniformly, and propose Mixture-of-Denoisers (MoD) that frame multiple pretraining tasks as span corruption, diversifies and then mixes them. Then, the authors introduce the notion of paradigm-shifting via mode switching. During pre-training, they feed the model an extra paradigm token, {[R], [S], [X]} that helps the model switch gears and operate on a mode that is more suitable for the given task. Next, the authors do a large number of ablation experiments, and discuss 'Decoder Vs Encoder-Decoder' and 'Is GPT and/or T5 the optimal setup?' questions separately. They experimentally demonstrate the performance of the UL2 model. Finally, the authors also train a 20B parameters UL2 model, and conduct experiments over 50 NLP tasks under the supervised finetuning paradigm. They also apply chain of thought on the model, and show that CoT is also suitable for UL2.
strength_and_weaknesses: Strengths
1. The idea of this paper is simple, but very effective. Through the MoD method and mode switch, several pre-training tasks with different forms are successfully unified.
2. In our intuition, the various pre-training tasks seem to be independent of each other, and it seems reasonable to pre-train models for these tasks separately. However, the author can jump out of this mindset, observe the common points of these tasks, and propose a unified pre-training model framework by introducing effective methods, and finally achieve good results. Solving big problems with small methods inspires us a lot.
3. This paper is well-written. From the proposal, to the solution, to the experimental setup and results, and finally to scaling to 20B parameters and testing the effectiveness of the chain of thought method.
4. The experiment of this paper is very detailed. Although the main text is only 9 pages, the experimental part of the appendix is as many as 13 pages. It can be seen that the author is very comprehensive when considering the problem and is very patient when researching the problem. Although there are many experimental results, the layout is very neat, well organized, and the levels are rigorous. This scientific research spirit is worth learning.

Weakness:
1. I personally suggest that the authors need to modify Figure 2. The text is too small to recognize each denoiser’s feature.
2. Will the settings of denoisers, like different combinations, different hyper-parameters,  exert huge influence on UL2’s performance? More experiments are required to prove it.
3. In the comparisons between decoder and encoder-decoder architecture, UL2 decoder doesn’t outperform UL2 encoder-decoder. But in the paper, “However, this reinforces our point that the self-supervision objective may be intrinsically more important than the backbone architecture and negotiating architectural choices is mainly about efficiency trade-offs that can be studied independently.” is somewhat confusing. The authors need to give more detailed explanations.

clarity,_quality,_novelty_and_reproducibility: This paper is well-written and easy to follow. The proposed method is straightforward and effective. The experimental comparisons and analysis are detailed. The pretrained checkpoints will be released for reproducibility.
summary_of_the_review: To sum up, this paper is well-written and easy to follow. The proposed method is straightforward and effective. The experimental comparisons and analysis are detailed. I recommend accepting this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a unified perspective of different self-supervised pretraining objectives. More importantly, the paper shows how different objectives can be cast into each other and interpolated effectively. On the practical side, the paper proposes Mixture-of-Denoisers (MoD) pretraining and the notion of mode switching that allows models to be adapted to their downstream fine-tuning use. The paper shows experimentally that the proposed method can outperform T5 and/or GPT-like models across several different setups. The paper trains a 20B-parameter model that achieves SOTA performance on multiple well-established NLP tasks.

The paper does not contain novel creative ideas, but the experimental evaluation is very strong. The way that the space of pretraining methods is categorized and explored leads to new insights that, in terms, lead to a new state-of-the-art performance. The paper will likely impact how others think about pretraining objectives in the future.
justification_for_why_not_higher_score: The does not contain innovative methods or ideas.
justification_for_why_not_lower_score: The paper has important practical consequences and can help other researchers in their work.
The authors will release new pretrained models that perform well and will be useful to others.
Strong experimental and ablation studies support the main arguments.
Decision: Accept: poster