Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper propose to continue pre-train a pre-trained seq2seq model using annotated data for generation tasks. The authors collected 77datasets over 11 diverse NLG tasks for the continued train step. Evaluation shows good results.

strength_and_weaknesses: ### Strengths
- Collect a significant amount of training data for seq2seq tasks.
- Reconfirm empirically that using more data is always useful

### Weaknesses
- The contribution of the paper is too thin in my opinion (i.e., showing that using more data is useful)

- On *Generaliability* study: I don’t really understand this setup. The authors call it unseen tasks but then the  model is fine-tuned using the annotated data of the task. How is this different from taking any pretrained model and finetuning it on the same data? My understanding here is that the “unseen” part means that the tasks and data weren’t in the continued train step. But calling that _generaliability_ seems to be an overstatement to me. Is that what we all do with pretrained models?
- There are no human evaluation results. Evaluating language generation is tricky, using automatic metric alone is not sufficient.
While measuring by automatic metric, it shows that supervised pre-training with  more data is useful. There are many datasets with different characteristics. Does it hurts the model in a way that we don’t know yet? What type of hallucination in language generation when the model is trained on diverse datasets? Does the nature of informal text in the Personal Chat dataset have negative influences on formal text CNN/Daily Mail. I think there are many important questions about mixing random datasets in the context of language generation that have not been answered. The small improvement in automatic metrics do not fully justify the approach.


clarity,_quality,_novelty_and_reproducibility: - There is some part quite hard to follow. How do you train  (MVP+R) model? 
- the originality is marginal

summary_of_the_review: In summary, the contribution of the paper is thin. The evaluation is not convincing (lack human evaluation). There are potential issues of concatenating multiple language generation datasets in the supervised pretraining step.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: Considers supervised pre-training as an alternative to unsupervised pre-training, which is more common with state-of-the-art NLP models. Many (77) NLG supervised datasets are curated to create MVPCorpus which is used for supervised pre-training. However, the training setup is not designed in a way that allows us to compare unsupervised vs supervised pre-training since the backbone model is initialized with BART parameters. Thus the paper cannot answer the question which is posed, and instead shows that continued supervised pre-training using related tasks after starting with an unsupervised pre-trained model helps with downstream tasks. 

The model is initialized with BART and is further 'pre-trained' using supervised/multi-task learning. The main baseline is BART of the same architecture and some variants are compared.

Fine-tuning is done in two settings: full fine-tuning, and with prefix tuning. They show that MVP does better in both settings compared to BART and achieves some SoTA results in NLG in the FT setting. Finally, it is shown that zero-shot performance is better than BART as well.

strength_and_weaknesses: # Strengths
- Improvements over BART baseline,  which has the same architecture, in some tasks. 
- In some NLG cases achieves SoTA, although I've not verified every task.

# Weaknesses
- There is a major mismatch in the way the paper is written and the experimental setup. It is written to suggest that supervised pre-training can replace unsupervised pre-training, but does not have experiments to back this up. Consequently it has highly misleading claims on merits of supervised pre-training since the models are primarily pretrained in an unsupervised way.

clarity,_quality,_novelty_and_reproducibility: - Clarity: It should be emphasized from the beginning (including abstract) that this is a study of continued multi-task supervised pre-training, and not a replacement for unsupervised pre-training.
- Reproducibility: Missing details about how different tasks are mixed in supervised pre-training.
- Novelty: continued pre-training with related tasks is not new, and is sometimes called intermediate fine-tuning. It is already known to improve performance in certain tasks.

summary_of_the_review: This paper should be re-written as a study of how to continue pre-training with multi-task supervised learning or properly study supervised pre-training by initializing the model from random parameters. At best, if re-written this way the paper shows how to achieve slightly better results on certain NLG tasks compared to BART with intermediate fine-tuning. The novelty is limited and the claims are incorrect.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on multi-task pretraining via supervised data in NLG area. They collected 77 NLG datasets and applied different variants of multi-task pretraining on top. The pretrained models are evaluated on both seen and unseen tasks, comparing to the prior STOAs and BART finetune baselines.  
strength_and_weaknesses: Strengths:
- A collection of data mixtures over 77 NLG datasets, which are publicly released.
- A exploration effort on studying multi-task pretraining on the NLG area.
- Code are released and could be easy to reproduce the results.

Weaknesses:
- The idea of multi-task pretraining is not novel, as it mostly follows Aribandi et al. (2022).
- Results are mixed. On Table 2, it seems the multi-task pretraining has minor gains over BART finetuning or Single; while on Table 9, the improvements are indeed significant. 
- The proposed variants of pretraining stages (e.g., MVP+S, MVP+R, MVP+M) don't seem effective as MVP (the results are on-par on most datasets from Table 2 & 3)
clarity,_quality,_novelty_and_reproducibility: - On table 10, it seems there are a large improvement against ExT5 on the GEM benchmark. Since the ExT5 paper doesn't release their model checkpoints, and the evaluation script of GEM benchmark can make a big difference on the results, it would be good to add BART finetune baseline on Table 10 too -- so that we can exclude the effect of evaluation difference. 
summary_of_the_review: Overall, the paper makes a significant contribution on the collection of a large amount of NLG datasets and study the multi-task pretraining behaviors for NLG. Their code are released and the results can be reproduced. The idea presented in this paper is only marginally novel and mostly follow the prior work. The results of their proposed method also seems mixed in different datasets.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Reviewers were not supportive of the paper and scores are pretty low.

I also skimmed the paper myself. Many of the usefulness of multi-task pretraining or finetuning have already been shown in ExT5/T0/Flan so I was also not able to find anything interesting in this paper.


justification_for_why_not_higher_score: Reviewers are not supportive of the paper. Contribution is also quite weak here. 
justification_for_why_not_lower_score: There isn't really anything lower than a reject.
Decision: Reject