MVP: Multi-task Supervised Pre-training for Natural Language Generation
Year: 2023

Abstract:
Pre-trained language models (PLMs) have achieved remarkable success in natural language generation (NLG) tasks. Up to now, most NLG-oriented PLMs are pre-trained in an unsupervised manner using the large-scale general corpus. In the meanwhile, an increasing number of models pre-trained with labeled data (i.e., “supervised pre-training”) showcase superior performance compared to unsupervised pre-trained models. Motivated by the success of supervised pre-training, we propose Multitask superVised Pre-training (MVP) for natural language generation. We collect a large-scale natural language generation corpus, MVPCorpus, from 77 datasets over 11 diverse NLG tasks. Then we unify these examples into a general text-to-text format to pre-train the text generation model MVP in a supervised manner. For each task, we further pre-train specific soft prompts to stimulate the model’s capacity to perform a specific task. Our MVP model can be seen as a practice that utilizes recent instruction tuning on relatively small PLMs. Extensive experiments have demonstrated the effectiveness and generality of our MVP model in a number of NLG tasks, which achieves state-of-the-art performance on 13 out of 17 datasets, outperforming BART by 9.3% and Flan-T5 by 5.8%.

heading: 1 Introduction
text: Natural language generation (NLG, also known as text generation) is a crucial capacity for language intelligence, which aims to generate human-like texts on demand (Garbacea and Mei, 2020). Since the emergence of the pre-training and fine-tuning paradigm, pre-trained language models (PLMs) have dominated mainstream approaches for NLG tasks (Lewis et al., 2020; Brown et al., 2020). With a large-scale general corpus, the majority of PLMs are pre-trained in an unsupervised (self-supervised) manner by leveraging intrinsic data correlations as
B Corresponding author
supervision signals. However, unsupervised pretraining is likely to incorporate noise that affects the performance of downstream tasks (Feng et al., 2022), also leading to a slower rate of acquiring knowledge (Zhang et al., 2021).
In the meanwhile, more and more large-scale labeled datasets have become easily accessible (Deng et al., 2009; Liu et al., 2020). There is growing evidence that pre-training with labeled data can further improve the performance of PLMs, both in the fields of computer vision (He et al., 2016; Dosovitskiy et al., 2021) and natural language processing (Lin et al., 2020b; Su et al., 2022). These promising developments motivate us to consider pre-training text generation models with labeled data, which is called “supervised pretraining” (Feng et al., 2022). Existing work has shown that supervised pre-training can explicitly learn task-specific characteristics and alleviate the discrepancy between unsupervised pre-training and supervised fine-tuning (Lin et al., 2020b).
Furthermore, most NLG systems are often trained in a supervised way, requiring supervision signals to learn the input-to-output transformation. For example, dialogue systems learn to generate appropriate responses based on historical utterances, and text summarization systems learn to extract essential information from long documents according to human-written summaries. Therefore, we suspect that supervised pre-training is more suited for NLG-oriented PLMs in essence since it can provide task-related instructions early in the pre-training stage instead of a later fine-tuning stage.
Inspired by the recent success of supervised pre-training, we propose Multi-task superVised Pre-training (MVP) for natural language generation by leveraging a variety of labeled text generation datasets. Specially, we collect a largescale labeled corpus, MVPCorpus, consisting of 77 datasets over 11 text generation tasks. Since recent research shows that an extensive scale of
8758
Settings Supervised Pre-training Unsupervised Pre-training
NLG MVP (ours) GPT-2, MASS, BART, T5
NLU FLAN, T0, Muppet, ExT5 BERT, XLNet, RoBERTa, T5
show that the supervised pre-trained MVP outperforms its unsupervised pre-trained counterpart BART in both full tuning (+9.3% in ratio) and parameter-efficient tuning (+4.3% in ratio) settings. Our MVP model achieves state-of-the-art performance on 13 out of 17 datasets and outperforms Flan-T5 (Chung et al., 2022) by 5.8%. Our zero-shot performance also surpasses T011B (Sanh et al., 2022) by a large margin. Furthermore, the experiments on unseen NLG and NLU tasks demonstrate that our supervised MVP model has a strong generality for unseen tasks.
For reproducing and reusing our work, we release the MVPCorpus collection, all the MVP model variants, and accordingly codes at the link: https://github.com/RUCAIBox/MVP.

heading: 2 Related Work
text: Pre-trained Language Models. Pre-trained language models have achieved exceptional success in a wide range of tasks, and the majority of them are pre-trained in an unsupervised manner (Devlin et al., 2019; Brown et al., 2020). For example, with large-scale plain texts as the unsupervised pre-training corpus (570GB), GPT-3 (Brown et al., 2020) employs language modeling as the pretraining task, i.e., predicting the next token conditioned on previous tokens. In the meanwhile, the computer vision community benefits a lot from the labeled dataset ImageNet (Deng et al., 2009). Influential models, such as ResNet (He et al., 2016) and ViT (Dosovitskiy et al., 2021), leverage ImageNet for pre-training. Inspired by the success of pretraining with labeled data, machine translation researchers explore supervised pre-training (McCann et al., 2017; Lin et al., 2020b). Lin et al. (2020b) attempt to pre-train a translation model with parallel data in multiple languages. Despite using much less pre-trained data, mRASP still achieves better performance than translation models pre-trained in an unsupervised manner (Liu et al., 2020). In this paper, we propose to pre-train a universal NLG model in a supervised manner with collections of labeled datasets (23GB).
Multi-task Learning. Our pre-training process is also related to multi-task learning (MTL), a method of mixing multiple tasks into a single training process (Collobert and Weston, 2008). A model trained with MTL can benefit from helpful knowledge of relevant tasks, resulting in improved perfor-
Stage 1: Multi-task Supervised Pre-training
MVP
Summarization
...
Stage 2: Task-specific Prompt Pre-training
Dialogue
Data-to-textMVP
Summarization
... Dialogue
Data-to-text
soft prompt

heading: 3.1 Data Collection

heading: 3 The MVP Model
text: instance, we linearize structured data (e.g., knowledge graph or table) by concatenating triples or key-value pairs using the special token “[SEP]” for data-to-text generation, and we utilize the special token “[X_SEP]” to separate answer and paragraph for question generation. The transformed input format for each task can be found in Appendix E.
We divide MVPCorpus into two parts, which are used for pre-training and fine-tuning (evaluation), respectively. For supervised pre-training, we utilize 50 datasets from 7 tasks, including data-to-text generation, open-ended dialogue system, question answering, question generation, story generation, task-oriented dialogue system, and text summarization. We also eliminate pre-training examples overlapping with evaluation data to avoid data leakage (more details in Appendix A.2). Finally, we have a 25GB supervised pre-training corpus containing 32M examples. The statistics of the datasets for pre-training are listed in Table 9.
For evaluation, we utilize the rest of the 27 datasets, which are more commonly used in the literature. Among these datasets, 23 datasets are from the 7 tasks used in pre-training. We refer to them as seen tasks and use them to test the effectiveness of our model. The remaining 4 datasets are from the tasks of commonsense generation, paraphrase generation, simplification, and style transfer, respectively. We call them unseen tasks and use them to examine the generality of our model.

heading: 3.2 Model Architecture
text: Our MVP model is built on the standard Transformer encoder-decoder architecture (Vaswani et al., 2017). Compared to decoder-only PLMs such as GPT-3 (Brown et al., 2020) and prefix LMs such as UniLM (Dong et al., 2019), the encoderdecoder architecture is more effective for text generation tasks (Raffel et al., 2020). In the first stage, we pre-train the MVP backbone using a mixture of labeled datasets from seven tasks. To indicate each task, we apply human-written instructions to each task instance. For example, we write “Summarize:” as the prompt for summarization tasks. The manual instructions for each task are shown in Appendix E.
In the second stage, we freeze the MVP backbone and pre-train a set of task-specific prompts (i.e., continuous vectors) to stimulate the model’s capacity to perform some specific task. Specially, we follow prefix-tuning (Li and Liang, 2021) to insert continuous vectors at each Transformer layer
and learn them using a mixture of corresponding intra-task datasets (i.e., datasets under the same task2). Compared to prompt tuning (Lester et al., 2021), which only adds prompts to the input layer, layer-wise prompts are more effective and stable (Liu et al., 2022), especially for NLG tasks. These soft prompts, which are not shared between tasks, encode task-specific semantic knowledge to alleviate the blurring-out problem induced by multitask learning (He and Choi, 2021).

heading: 3.3 Training Details
text: Our MVP model adopts a Transformer with 12 layers in both the encoder and decoder (406M parameters), the same as the model size of BARTLARGE (Lewis et al., 2020). We initialize the backbone with the BART parameters to provide a good starting point for NLG tasks following previous work (Dong et al., 2019; Zhang et al., 2020). We pre-train the model with a batch size of 8,192 and adopt a temperature-scaled mixing strategy (Raffel et al., 2020) with a rate of T = 2 to mitigate the disparity in tasks and datasets.
We follow prefix-tuning (Li and Liang, 2021) to pre-train task-specific prompts by prepending trainable vectors to multi-head attention modules at each layer. The prompt length is set to 100, and we utilize the MLP reparameterization function with a hidden size of 800 to improve the training robustness and performance (Li and Liang, 2021). Hence, every task prompts have approximately 62M parameters. Then, we freeze the MVP model and train seven groups of task-specific prompts, each of which corresponds to a different task.
In the two stages, the maximum length of both input and output sequences is set to 1,024 for supporting examples to contain more tokens. We optimize the model with a constant learning rate of 3× 10−5 using standard sequence-to-sequence cross-entropy loss. We apply the AdamW optimizer with β1 = 0.9, β2 = 0.98, ϵ = 1× 10−6 to improve training stability (Liu et al., 2019b). The weight decay coefficient is 0.1. For testing, we select the checkpoint with the highest validation performance. All the experiments are conducted on 32 NVIDIA Tesla V100 32GB GPUs. We implement our model using the text generation library TextBox (Tang et al., 2022a).
2For instance, we train summarization-specific prompts using summarization datasets, e.g., Newsroom (Grusky et al., 2018), WikiHow (Koupaee and Wang, 2018), and MSNews (Liu et al., 2021a).
Methods CNN/DailyMail WebNLG SQuAD (QG) CoQA
R-1 R-2 R-L B-4 ME R-L B-4 ME R-L F1 EM
MVP 44.52 21.62 41.10 67.82 47.47 76.88 26.26 27.35 53.49 86.43 77.78 BART 44.16e 21.28 40.90 64.55b 46.51 75.13 22.00f 26.40 52.55 68.60f – Flan-T5 43.45 21.01 40.03 66.60 46.93 75.76 25.55 26.90 53.51 84.18 75.44 Single 44.36 21.54 40.88 67.74 46.89 76.94 26.09 27.15 53.29 86.20 77.26
MVP+S 44.63 21.72 41.21 68.19 47.75 76.81 25.69 27.04 53.20 86.65 77.93 MVP+R 44.14 21.45 40.72 67.61 47.65 76.70 25.71 27.03 53.09 85.95 77.22 MVP+M 43.97 21.16 40.46 67.45 47.57 76.81 25.46 26.79 52.95 86.28 77.26
SOTA 47.16a 22.55 43.87 66.14b 47.25 76.10 25.97c 27.33 53.43 84.50d –
Methods ROCStories PersonaChat MultiWOZ
B-1 B-2 D-1 D-4 B-1 B-2 D-1 D-2 B-4 Success Inform
MVP 33.79 15.76 3.02 75.65 50.73 40.69 1.65 11.23 20.26 76.40 85.00 BART 30.70g 13.30 – 69.90 49.90f 40.00 1.30 8.00 17.89j 74.91 84.88 Flan-T5 32.72 15.23 2.97 68.97 48.55 40.22 1.40 7.85 19.73 70.20 78.70 Single 32.67 15.29 2.72 72.97 49.96 40.53 1.27 7.63 19.73 75.60 83.70
MVP+S 33.92 15.60 3.44 80.58 47.91 39.97 1.52 9.54 20.32 79.90 86.80 MVP+R 32.93 15.32 2.88 73.83 48.45 40.09 1.30 7.95 19.02 73.30 81.80 MVP+M 33.30 15.51 2.71 74.24 46.26 39.30 1.36 8.07 19.93 72.70 79.70
SOTA 33.40g 15.40 – 69.30 49.90f 40.00 1.50h 9.40 20.50i 85.30 94.40

heading: 4.1 Full Tuning Performance

heading: 4 Experiment Results
text: Methods CNN/DailyMail WebNLG SQuAD (QG) CoQA
R-1 R-2 R-L B-4 ME R-L B-4 ME R-L F1 EM
FT BART 44.16 21.28 40.90 64.55 46.51 75.13 22.00 26.40 52.55 68.60 – FT MVP 44.52 21.62 41.10 67.82 47.47 76.88 26.26 27.35 53.49 86.43 77.78
T0-3B – – – 01.40 10.20 18.43 3.06 12.43 14.91 13.30 06.60 T0-11B – – – 00.26 06.13 14.12 2.63 07.00 15.25 09.18 04.36 MVP 29.50 11.29 25.92 34.42 31.33 52.33 2.90 13.94 15.48 29.40 18.20 MVP+S 25.60 09.51 22.67 39.43 34.32 55.34 2.96 15.23 18.23 52.40 37.30
Methods ROCStories PersonaChat MultiWOZ
B-1 B-2 D-1 D-4 B-1 B-2 D-1 D-2 B-4 Success Inform
FT BART 30.70 13.30 – 69.90 49.90 40.00 1.30 8.00 17.89 74.91 84.88 FT MVP 33.79 15.76 3.02 75.65 50.73 40.69 1.65 11.23 20.26 76.40 85.00
T0-3B 08.69 3.02 04.37 35.49 23.20 23.57 2.56 12.06 0.02 2.50 22.10 T0-11B 00.63 0.16 12.41 92.86 32.17 28.35 1.56 07.19 0.00 3.90 22.10 MVP 01.01 0.31 07.18 86.26 35.54 32.71 2.87 16.38 3.08 2.50 22.20 MVP+S 10.52 3.54 02.13 69.55 37.04 33.38 2.66 14.84 0.38 2.50 22.10
Methods CoLA SST-2 MRPC STS-B QQP MNLI QNLI RTE Average Matt. Acc. F1/Acc. P/S Corr. F1/Acc. m./mm. Acc. Acc.
BART 60.30 96.30 90.47 / 86.70 90.97 / 90.30 73.03 / 89.87 90.03 / 89.27 94.60 79.83 85.17 MVP 59.87 96.43 92.07 / 89.43 91.37 / 90.90 73.20 / 90.13 89.70 / 88.73 95.10 82.87 85.88

heading: 4.3 Generality to Unseen Tasks

heading: 4.2 Zero-shot Performance

heading: 4.4 Parameter-Efficient Tuning Performance
text: In the lightweight fine-tuning setting, we only tune the prompts while freezing the backbone MVP model to verify its effectiveness in resourceconstrained situations. Besides our MVP+S model, we consider comparing the following methods:
• Prefix-tuning (Li and Liang, 2021): Prefixtuning is a popular prompt-based lightweight tuning method for text generation. We employ BART as its backbone, denoted as BART+R.
• Only tuning randomly initialized prompts (MVP+R): This variant only tunes the randomly initialized prompts of MVP+R, and it shares a similar idea with prefix-tuning.
• Only tuning multi-task pre-trained prompts (MVP+M): This variant only tunes the multi-task pre-trained prompts of MVP+M. Such an idea has been used in SPoT (Vu et al., 2022).
From the experimental results in Table 6, we can see that: the good performance of the MVP model in lightweight settings further demonstrates the effectiveness of supervised pre-training. By comparing two randomly initialized prompting methods (BART+R and MVP+R), we can see that MVP+R achieves superior performance to BART+R (+2.0%) due to its multi-task supervised backbone. Furthermore, when initialized with pretrained prompts, MVP+S and MVP+M achieve improved results over MVP+R, which is consistent with the findings of SPoT (Vu et al., 2022).
Datasets MVP wins (%) Ties (%) BART wins (%)
CNN/DM 46.50 10.67 42.83 WebNLG 32.17 45.67 22.17 ROCStories 46.50 11.33 42.17 PersonaChat 35.33 34.00 30.67

heading: 4.5 Human Evaluation
text: Methods #NLG (PT) #NLU (PT) #NLG (FT) #NLU (FT) SP model SP prompts Open source
FLAN 3 9 2 9 ✓ ✗ ✗ T0 2 6 0 4 ✓ ✗ ✓ Muppet 1 3 1 3 ✓ ✗ ✓ ExT5 3 8 6 8 ✓ ✗ ✗ SPoT 1 4 0 6 ✗ ✓ ✗ MVP (ours) 7 0 11 3 ✓ ✓ ✓

heading: 6 Conclusion

heading: 5 Discussion

heading: Acknowledgements
text: This work was partially supported by National Natural Science Foundation of China under Grant No. 62222215, Beijing Natural Science Foundation under Grant No. 4222027, and Beijing Outstanding Young Scientist Program under Grant No. BJJWZYJH012019100020098. Xin Zhao is the corresponding author.

heading: Limitations
text: Despite our efforts to collect as many generation tasks and datasets as possible, we only evaluate the generation quality and generality of our models on a small number of tasks and datasets. The interpretability and robustness of our models require further analysis. Besides, there exists subjectivity when collecting downstream tasks and intratask datasets, albeit our attempts to employ widelyrecognized categorizations from the literature. Due to the limitation of computing power, we do not study the performance of our method at different model scales. The effectiveness of multi-task pretraining from scratch, similar to ExT5 (Aribandi et al., 2022), also merits an in-depth study.
Broader Impacts
In this paper, we pre-trained a language model MVP using labeled NLG datasets. According to the research (Bender et al., 2021; Bommasani et al., 2021), PLMs tend to “remember” what they have “seen” in the pre-training corpus. This could result in the reproduction of undesirable biases from pretraining data on downstream tasks. Training data intervention could be a solution to alleviate this issue (Lu et al., 2020). It is also interesting to investigate whether supervised pre-training produces fewer biases than unsupervised pre-training.
Environmental impact is another factor we should consider. We attempt a more efficient pretraining strategy and released our PLM for future work. In contrast to large PLMs with tens of billions of parameters, such as T5 (Raffel et al., 2020) and GPT-3 (Brown et al., 2020), we pre-train only a small model with hundreds of millions of parameters. In addition, we utilize supervised pretraining data and initialize our model with pretrained BART, both of which improve the convergence of our model. Ultimately, our model is pretrained for about 20, 000 steps, whereas the BART of the same size is pre-trained for 500, 000 steps.

heading: Reproducibility
text: For reproducing and reusing our work, we have released the collection MVPCorpus, the models (e.g., MVP, task-specific prompts, and multitask variants), intermediate results (e.g., the generated texts), and source codes for pre-training and fine-tuning at the link: https://github.com/ RUCAIBox/MVP. The detailed settings of the experiments are listed in Appendix B. We hope that these open-source resources will facilitate future work on supervised pre-training and contribute to the advancement of NLG research.

heading: A Tasks and Datasets

heading: A.1 Description of Tasks and Datasets
text: We provide the details of the tasks and datasets used in our paper for pre-training and fine-tuning in Tables 9 and 10. If the dataset for pre-training does not have a valid set, we divide 10% of the training set for validation.
We list the licenses for all datasets if they have them. All datasets are publicly available. The majority of them can be directly downloaded from GitHub or Google Drive. ROCStories (Mostafazadeh et al., 2016) and CommonGen (Lin et al., 2020a) can be obtained after filling out a form. GYAFC (Rao and Tetreault, 2018) is accessible after requesting Yahoo and the authors of the dataset.
The tasks and datasets we use in this paper are as follows:
• Data-to-text generation aims to generate descriptive text about structured data, such as the knowledge graph and the table. We use the following datasets for pre-training:
1. AGENDA (Koncel-Kedziorski et al., 2019); 2. ENT-DESC (Cheng et al., 2020); 3. GenWiki (Jin et al., 2020); 4. LogicNLG (Chen et al., 2020a); 5. TEKGEN (Agarwal et al., 2021); 6. WEATHERGOV (Liang et al., 2009); 7. WikiTableT (Chen et al., 2021).
We utilize the following datasets for fine-tuning evaluation:
1. WebNLG (Gardent et al., 2017), we utilize version 2.1;
2. WikiBio (Lebret et al., 2016).
• Open-ended dialogue system, also known as chatbots, is focused on daily communication. We use the following datasets for pre-training:
1. Cleaned OpenSubtitles Dialogs (Cleaned OS Dialogs) (Welivita et al., 2021), which is a cleaned variant of OpenSubtitles Dialogs (Lison et al., 2018);
2. CMU Document Grounded Conversations (CMUDog) (Zhou et al., 2018);
3. Curiosity (Rodriguez et al., 2020); 4. DREAM (Sun et al., 2019); 5. Empathetic Dialogues (Rashkin et al.,
2019);
6. Movie Dialog (Dodge et al., 2016); 7. MuTual (Stratos, 2019); 8. OpenDialKG (Moon et al., 2019); 9. Topical-Chat (Gopalakrishnan et al., 2019);
10. Wizard of Wikipedia (Dinan et al., 2019).
We utilize the following datasets for fine-tuning evaluation:
1. DailyDialog (Li et al., 2017); 2. DSTC7-AVSD (Alamri et al., 2018); 3. PersonaChat (Zhang et al., 2018).
• Paraphrase generation involves rewriting a sentence with the same semantic meaning but a different syntactic or lexical form. We utilize the following datasets for fine-tuning evaluation:
1. Quora (also known as QQP-Pos) (Kumar et al., 2020), which is a subset of Quora Question Pairs3.
• Question answering requires the model to answer a question based on optional background information. Note that we conduct this task in a generative way in our paper. We use the following datasets for pre-training:
1. HotpotQA (Yang et al., 2018); 2. MS MARCO (Nguyen et al., 2016); 3. MSQG (Liu et al., 2021a), since it is de-
signed for QG, we reverse the question and answer to enrich QA examples;
4. NarrativeQA (Kočiský et al., 2018); 5. Natural Questions (Kwiatkowski et al.,
2019); 6. NewsQA (Trischler et al., 2017); 7. QuAC (Choi et al., 2018); 8. TriviaQA (Joshi et al., 2017); 9. WebQuestions (Berant et al., 2013).
We utilize the following datasets for fine-tuning evaluation:
1. CoQA (Reddy et al., 2019); 2. SQuAD (Rajpurkar et al., 2016), we utilize
version 1.1.
• Question generation generates a coherent question given a passage and its corresponding answer. We use the following datasets for pretraining:
3https://www.kaggle.com/c/ quora-question-pairs
1. HotpotQA (Yang et al., 2018); 2. MS MARCO (Nguyen et al., 2016); 3. MSQG (Liu et al., 2021a); 4. NarrativeQA (Kočiský et al., 2018); 5. NewsQA (Trischler et al., 2017); 6. QuAC (Choi et al., 2018).
Most of them are QA tasks, and we invert the question and answer to enrich QG examples.
We utilize the following datasets for fine-tuning evaluation:
1. CoQA (Reddy et al., 2019); 2. SQuAD (Rajpurkar et al., 2016), we utilize
version 1.1.
• Story generation creates a long and informative text with a short title. We use the following datasets for pre-training:
1. ChangeMyView (Hua and Wang, 2020); 2. English Gigaword (Rush et al., 2015); 3. Hippocorpus (Sap et al., 2020); 4. WikiPlots (Markriedl); 5. WritingPrompts (Fan et al., 2018), we split
the original training set for pre-training and corresponding validation.
Considering English Gigaword is a large summarization dataset, we use the summary as the title to generate the passage in turn to enrich the examples of story generation.
We utilize the following datasets for fine-tuning evaluation:
1. ROCStories (Mostafazadeh et al., 2016); 2. WritingPrompts (Fan et al., 2018), we use
the sets created by Guan et al. (2021) (who split the original valid and test sets for training, validation, and testing) to fine-tune our model for a fair comparison.
• Task-oriented dialogue system meets the reallife needs of users, such as restaurant reservations and airplane bookings. We use the datasets for pre-training, following Su et al. (2022):
1. CamRest676 (Wen et al., 2017); 2. Frames (El Asri et al., 2017); 3. KVRET (Eric et al., 2017); 4. MetaLWOZ (Lee et al., 2019); 5. MSR-E2E (Li et al., 2018); 6. MultiWOZ (Budzianowski et al., 2018);
7. Schema-Guided (Rastogi et al., 2020a); 8. TaskMaster (Byrne et al., 2019); 9. WOZ (Mrkšić et al., 2017).
We utilize the following datasets for fine-tuning evaluation:
1. MultiWOZ (Budzianowski et al., 2018), we utilize version 2.0.
• Text style transfer modifies the style (e.g., sentiment and formality) of given texts while retaining their style-independent content. We utilize the following datasets for fine-tuning evaluation:
1. GYAFC (Rao and Tetreault, 2018), which has two sub-domains: “Entertainment and Music” (E&M) and “Family and Relationships” (F&R).
• Text summarization condenses a long document into a brief text while retaining the essential details. We use the following datasets for pre-training:
1. English Gigaword (Graff et al., 2003), we use the variant provided by Rush et al. (2015);
2. MediaSum (Zhu et al., 2021); 3. MSNews (Liu et al., 2021a); 4. Newsroom (Grusky et al., 2018); 5. WikiHow (Koupaee and Wang, 2018).
We utilize the following datasets for fine-tuning evaluation:
1. CNN/DailyMail (Hermann et al., 2015), we use the variant provided by See et al. (2017);
2. SAMSum (Gliwa et al., 2019); 3. XSum (Narayan et al., 2018).
To better compare with ExT5 (Aribandi et al., 2022), we utilize the language generation benchmark GEM (Gehrmann et al., 2021) for fine-tuning evaluation. GEM includes five tasks:
• Commonsense generation:
1. CommonGen (CG) (Lin et al., 2020a).
• Data-to-text generation:
1. DART (Nan et al., 2021); 2. E2E NLG cleaned (Novikova et al., 2017); 3. ToTTo (Su et al., 2021); 4. WebNLG (Gardent et al., 2017).
• Dialogue system:
1. Schema-Guided Dialog (SGD) (Rastogi et al., 2020b).
• Text simplification:
1. WikiAuto + Turk/ASSET (WiA-T/A) (Jiang et al., 2020; Xu et al., 2016; Alva-Manchego et al., 2020).
• Text summarization:
1. Wiki-Lingua (WLE) (Ladhak et al., 2020).
To test the generalization ability of our model, we also utilize the natural language standing benchmark GLUE (Wang et al., 2019), which is composed of three tasks:
• Natural language inference:
1. MNLI (Williams et al., 2018); 2. QNLI (Rajpurkar et al., 2016; Wang et al.,
2019); 3. RTE (Dagan et al., 2006; Haim et al., 2006;
Giampiccolo et al., 2007; Bentivogli et al., 2009).
• Paraphrase detection:
1. MRPC (Dolan and Brockett, 2005); 2. QQP 3; 3. STS-B (Cer et al., 2017).
• Text classification:
1. CoLA (Warstadt et al., 2019); 2. SST-2 (Socher et al., 2013).

heading: A.2 Data Leakage
text: Since our model is pre-trained on a large number of labeled datasets, it may have “seen” examples from fine-tuning test sets during pre-training, which leads to an unfair comparison with other methods. Hence, we eliminate the pre-training examples that share n-gram overlap with either of the test datasets. Following Brown et al. (2020), n is the 5th percentile example length in words, and the maximum value of n is set to 13. Finally, we have removed 17, 848 examples from the pre-training datasets. The number of “cleaned” examples for each dataset can be found in Table 9.

heading: Task Dataset #Train #Valid #Test Input Output License
text: Commonsen generation CommonGen 67,389 993 – 5.5 11.6 MIT
Data-to-text generation
DART 62,659 2,768 – 27.5 21.5 MIT E2E 33,525 4,299 – 9.5 20.6 CC BY-SA 4.0 ToTTo 120,761 7,700 – 37.8 18.0 CC BY-SA 3.0 WebNLG 34,338 4,313 4,222 18.0 19.9 CC BY-NA-SA 4.0 WebNLG (GEM) 35,426 1,667 – 17.7 22.7 CC BY-NA-SA 4.0 WikiBio 582,659 72,831 72,831 81.6 26.1 CC BY-SA 3.0
Open-ended dialogue
DailyDialog 76,052 7,069 6,740 72.5 13.9 CC BY-NC-SA 4.0 DSTC7-AVSD 76,590 17,870 1,710 148.2 11.5 MIT PersonaChat 122,499 14,602 14,056 132.1 11.9 MIT SGD 164,982 10,000 – 134.7 11.3 CC BY-SA 4.0
Natural language inference
MNLI-m 392,702 9,815 9,796 29.8 – Mixed MNLI-mm 9,832 9,847 QNLI 104,743 5,463 5,463 36.6 – CC BY-SA 4.0 RTE 2,490 277 3,000 51.0 – N/A
Paraphrase generation Quora 137,185 3,000 3,000 10.9 10.8 N/A
Paraphrase detection MRPC 3,668 408 1,725 43.8 – N/A QQP 363,846 40,430 390,965 22.3 – N/A STS-B 5,749 1,500 1,379 20.3 – N/A
Question answering CoQA 107,286 31,621 – 349.4 2.6 Mixed SQuAD 75,722 10,570 11,877 156.2 3.6 CC BY-SA 4.0
Question generation CoQA 107,286 31,621 – 346.6 5.5 Mixed SQuAD 75,722 10,570 11,877 148.3 11.6 CC BY-SA 4.0
Story generation ROCStories 176,688 9,816 4,909 9.0 40.7 N/A WritingPrompts 53,516 4,000 2,000 25.5 150.4 MIT
Task-oriented dialogue MultiWOZ 170,220 22,074 22,116 128.3 11.3 MIT
Text classification CoLA 8,551 1,043 1,063 7.7 – N/A SST-2 67,349 872 1,821 9.8 – N/A
Text simplification WiA-A 483,801 20,000 359 26.2 21.5 Mixed WiA-T 359
Text style transfer GYAFC-E&M 52,595 11,508 1,416 9.9 10.6 N/A GYAFC-F&R 51,967 11,152 1,332 10.7 11.3
Text summarization
CNN/DailyMail 287,227 13,368 11,490 679.8 48.3 MIT SAMSum 14,732 818 819 103.4 20.3 CC BY-NC-ND 4.0 WLE 99,020 28,614 – 367.6 33.4 CC0 1.0 XSum 204,045 11,332 11,334 373.7 21.1 MIT
Table 10: The statistics and licenses of datasets for evaluating our MVP model. The license of the MNLI dataset is composed of OANC, CC BY-SA 3.0, and CC BY 3.0. The license of the CoQA dataset is composed of CC BY-SA 4.0, MSR-LA, and Apache 2.0. The license of the WiA-A/T datasets is composed of CC BY-NC 3.0, CC BY-NC 4.0, and GNU General Public License v3.0.
Methods XSum SAMSum CoQA QG
R-1 R-2 R-L R-1 R-2 R-L B-4 ME R-L
BART 45.14d 22.27 37.25 51.74b 26.46 48.72 12.34c 35.78 46.88 MVP 45.60 22.47 37.42 53.78 29.12 49.37 23.48 47.79 55.09 MVP+S 45.67 22.63 37.50 53.81 29.75 49.43 23.43 47.49 55.25
SOTA 49.57a 25.08 41.81 53.89b 28.85 49.29 15.78c 40.15 50.98
Methods WritingPrompts DailyDialog WikiBio
B-1 B-2 D-1 D-4 B-1 B-2 D-1 D-2 B-4
BART 22.40e 8.40 – 31.30 44.30f 39.20 3.90 21.10 – MVP 32.34 13.11 2.12 64.58 46.19 41.81 4.61 25.06 48.42 MVP+S 30.12 11.46 3.97 83.70 45.71 42.92 5.10 27.14 48.19
SOTA 22.40e 8.40 – 31.30 46.10f 40.70 4.10 22.20 45.10g
Methods DSTC7-AVSD SQuAD
B-1 B-2 B-3 B-4 ME R-L CIDEr F1 EM
BART 82.40f 69.10 58.20 48.70 31.30 63.50 1.38 91.56i 84.23 MVP 83.75 70.89 60.19 50.94 32.12 65.04 1.45 93.45 87.20 MVP+S 83.81 71.07 60.45 51.20 31.77 64.76 1.44 93.45 87.17
SOTA 83.20f 70.50 59.80 50.60 31.40 63.80 1.39 96.22h 91.26

heading: B Fine-tuning and Evaluation Details
text: Methods DART E2E ToTTo
B-4 R-2 ME B-4 R-2 ME B-4 R-2 ME
T5.1.1 34.31 45.22 36.30 42.57 46.60 38.20 39.79 49.90 36.80 ExT5 36.62 48.14 37.60 42.25 46.70 38.10 40.14 50.33 36.90
MVP 39.13 48.92 38.53 37.38 47.96 39.39 50.58 55.24 41.27 MVP+S 38.83 48.49 38.41 37.32 47.40 38.90 50.69 55.52 41.29
Methods WebNLG CommonGen SGD
B-4 R-2 ME B-4 R-2 ME B-4 R-2 ME
T5.1.1 31.67 43.31 34.40 8.38 17.01 20.20 33.15 36.17 32.40 ExT5 35.03 48.17 36.50 9.68 19.04 21.40 34.74 37.77 33.00
MVP 47.03 59.00 42.34 32.59 37.71 33.00 45.63 48.29 38.48 MVP+S 47.03 59.03 42.28 34.10 37.87 33.11 45.24 48.25 38.47
Methods WiA-A WiA-T WLE
B-4 R-2 ME B-4 R-2 ME B-4 R-2 ME
T5.1.1 29.30 38.37 30.10 42.12 50.52 36.2 15.55 20.47 19.60 ExT5 29.23 37.98 30.00 41.39 50.38 35.8 16.64 21.16 20.40
MVP 71.55 70.88 48.19 91.73 83.46 57.34 18.80 22.84 21.95 MVP+S 70.37 70.65 47.70 91.12 83.59 56.95 18.52 22.57 22.02

heading: C.2 Results on the GEM Benchmark

heading: C.1 Results of Common Datasets

heading: C Additional Results

heading: D Human Evaluation
text: We hired six English-proficient college students with TOEFL or IELTS scores greater than 110 or 7.0. We paid 0.2$ per judge for each instance, for a total budget of 320$ for 400 instances. The text instructions we provided for each judge are shown in Figure 2.

heading: E Qualitative Examples
text: In this section, we showcase the linearized inputs, human-written task instructions, and corresponding outputs of a single dataset for tasks in Section 4. We provide the results of BART, MVP, and MVP+S under full tuning settings. To minimize human intervention, we select the first and second instances of the test set.

heading: Input
text: Summarize: Marseille, France (CNN)The French prosecutor leading an investigation into the crash of Germanwings Flight 9525 insisted Wednesday that he was not aware of any video footage from on board the plane. Marseille prosecutor Brice Robin told CNN that "so far no videos were used in the crash investigation." He added, "A person who has such a video needs to immediately give it to the investigators." Robin’s comments follow claims by two magazines, German daily Bild and French Paris Match, of a cell phone video showing the harrowing final seconds from on board Germanwings Flight 9525 as it crashed into the French Alps. All 150 on board were killed. Paris Match and Bild reported that the video was recovered from a phone at the wreckage site. The two publications described the supposed video, but did not post it on their websites. The publications said that they watched the video, which was found by a source close to the investigation. "One can hear cries of ’My God’ in several languages," Paris Match reported. "Metallic banging can also be heard more than three times, perhaps of the pilot trying to open the cockpit door with a heavy object. Towards the end, after a heavy shake, stronger than the others, the screaming intensifies. Then nothing." "It is a very disturbing scene," said Julian Reichelt, editor-in-chief of Bild online. An official with France’s accident investigation agency, the BEA, said the agency is not aware of any such video. Lt. Col. Jean-Marc Menichini, a French Gendarmerie spokesman in charge of communications on rescue efforts around the Germanwings crash site, told CNN that the reports were "completely wrong" and "unwarranted." Cell phones have been collected at the site, he said, but that they "hadn’t been exploited yet." Menichini said he believed the cell phones would need to be sent to the Criminal Research Institute in Rosny sous-Bois, near Paris, in order to be analyzed by specialized technicians working hand-in-hand with investigators. But none of the cell phones found so far have been sent to the institute, Menichini said. Asked whether staff involved in the search could have leaked a memory card to the media, Menichini answered with a categorical "no." Reichelt told "Erin Burnett: Outfront" that he had watched the video and stood by the report, saying Bild and Paris Match are "very confident" that the clip is real. He noted that investigators only revealed they’d recovered cell phones from the crash site after Bild and Paris Match published their reports. "That is something we did not know before. ... Overall we can say many things of the investigation weren’t revealed by the investigation at the beginning," he said. What was mental state of Germanwings co-pilot? German airline Lufthansa confirmed Tuesday that co-pilot Andreas Lubitz had battled depression years before he took the controls of Germanwings Flight 9525, which he’s accused of deliberately crashing last week in the French Alps. Lubitz told his Lufthansa flight training school in 2009 that he had a "previous episode of severe depression," the airline said Tuesday. Email correspondence between Lubitz and the school discovered in an internal investigation, Lufthansa said, included medical documents he submitted in connection with resuming his flight training. The announcement indicates that Lufthansa, the parent company of Germanwings, knew of Lubitz’s battle with depression, allowed him to continue training and ultimately put him in the cockpit. Lufthansa, whose CEO Carsten Spohr previously said Lubitz was 100% fit to fly, described its statement Tuesday as a "swift and seamless clarification" and said it was sharing the information and documents – including training and medical records – with public prosecutors. Spohr traveled to the crash site Wednesday, where recovery teams have been working for the past week to recover human remains and plane debris scattered across a steep mountainside. He saw the crisis center set up in Seyne-les-Alpes, laid a wreath in the village of Le Vernet, closer to the crash site, where grieving families have left flowers at a simple stone memorial. Menichini told CNN late Tuesday that no visible human remains were left at the site but recovery teams would keep searching. French President Francois Hollande, speaking Tuesday, said that it should be possible to identify all the victims using DNA analysis by the end of the week, sooner than authorities had previously suggested. In the meantime, the recovery of the victims’ personal belongings will start Wednesday, Menichini said. Among those personal belongings could be more cell phones belonging to the 144 passengers and six crew on board. Check out the latest from our correspondents. The details about Lubitz’s correspondence with the flight school during his training were among several developments as investigators continued to delve into what caused the crash and Lubitz’s possible motive for downing the jet. A Lufthansa spokesperson told CNN on Tuesday that Lubitz had a valid medical certificate, had passed all his examinations and "held all the licenses required." Earlier, a spokesman for the prosecutor’s office in Dusseldorf, Christoph Kumpa, said medical records reveal Lubitz suffered from suicidal tendencies at some point before his aviation career and underwent psychotherapy before he got his pilot’s license. Kumpa emphasized there’s no evidence suggesting Lubitz was suicidal or acting aggressively before the crash. Investigators are looking into whether Lubitz feared his medical condition would cause him to lose his pilot’s license, a European government official briefed on the investigation told CNN on Tuesday. While flying was "a big part of his life," the source said, it’s only one theory being considered. Another source, a law enforcement official briefed on the investigation, also told CNN that authorities believe the primary motive for Lubitz to bring down the plane was that he feared he would not be allowed to fly because of his medical problems. Lubitz’s girlfriend told investigators he had seen an eye doctor and a neuropsychologist, both of whom deemed him unfit to work recently and concluded he had psychological issues, the European government official said. But no matter what details emerge about his previous mental health struggles, there’s more to the story, said Brian Russell, a forensic psychologist. "Psychology can explain why somebody would turn rage inward on themselves about the fact that maybe they weren’t going to keep doing their job and they’re upset about that and so they’re suicidal," he said. "But there is no mental illness that explains why somebody then feels entitled to also take that rage and turn it outward on 149 other people who had nothing to do with the person’s problems." Germanwings crash compensation: What we know. Who was the captain of Germanwings Flight 9525? CNN’s Margot Haddad reported from Marseille and Pamela Brown from Dusseldorf, while Laura Smith-Spark wrote from London. CNN’s Frederik Pleitgen, Pamela Boykoff, Antonia Mortensen, Sandrine Amiel and Anna-Maja Rappard contributed to this report.

heading: Gold
text: Match are "very confident" the video clip is real, an editor says. Andreas Lubitz had informed his Lufthansa training school of an episode of severe depression, airline says.

heading: BART
text: Prosecutor: "A person who has such a video needs to immediately give it to the investigators" French Gendarmerie spokesman: Cell phones have been collected at the site, but haven’t been exploited.

heading: MVP
text: Prosecutor: "So far no videos were used in the crash investigation" An official with France’s accident investigation agency says the agency is not aware of any video.

heading: MVP+S
text: Prosecutor: "So far no videos were used in the crash investigation" Two publications claim to have found a cell phone video of the crash. "It is a very disturbing scene," says editor-in-chief of Bild online.
Table 13: The first instance from the CNN/Daily Mail dataset. Human-written task instructions are labeled in italics.

heading: Input
text: Summarize: The Palestinian Authority officially became the 123rd member of the International Criminal Court on Wednesday, a step that gives the court jurisdiction over alleged crimes in Palestinian territories. The formal accession was marked with a ceremony at The Hague, in the Netherlands, where the court is based. The Palestinians signed the ICC’s founding Rome Statute in January, when they also accepted its jurisdiction over alleged crimes committed "in the occupied Palestinian territory, including East Jerusalem, since June 13, 2014." Later that month, the ICC opened a preliminary examination into the situation in Palestinian territories, paving the way for possible war crimes investigations against Israelis. As members of the court, Palestinians may be subject to counter-charges as well. Israel and the United States, neither of which is an ICC member, opposed the Palestinians’ efforts to join the body. But Palestinian Foreign Minister Riad al-Malki, speaking at Wednesday’s ceremony, said it was a move toward greater justice. "As Palestine formally becomes a State Party to the Rome Statute today, the world is also a step closer to ending a long era of impunity and injustice," he said, according to an ICC news release. "Indeed, today brings us closer to our shared goals of justice and peace." Judge Kuniko Ozaki, a vice president of the ICC, said acceding to the treaty was just the first step for the Palestinians. "As the Rome Statute today enters into force for the State of Palestine, Palestine acquires all the rights as well as responsibilities that come with being a State Party to the Statute. These are substantive commitments, which cannot be taken lightly," she said. Rights group Human Rights Watch welcomed the development. "Governments seeking to penalize Palestine for joining the ICC should immediately end their pressure, and countries that support universal acceptance of the court’s treaty should speak out to welcome its membership," said Balkees Jarrah, international justice counsel for the group. "What’s objectionable is the attempts to undermine international justice, not Palestine’s decision to join a treaty to which over 100 countries around the world are members." In January, when the preliminary ICC examination was opened, Israeli Prime Minister Benjamin Netanyahu described it as an outrage, saying the court was overstepping its boundaries. The United States also said it "strongly" disagreed with the court’s decision. "As we have said repeatedly, we do not believe that Palestine is a state and therefore we do not believe that it is eligible to join the ICC," the State Department said in a statement. It urged the warring sides to resolve their differences through direct negotiations. "We will continue to oppose actions against Israel at the ICC as counterproductive to the cause of peace," it said. But the ICC begs to differ with the definition of a state for its purposes and refers to the territories as "Palestine." While a preliminary examination is not a formal investigation, it allows the court to review evidence and determine whether to investigate suspects on both sides. Prosecutor Fatou Bensouda said her office would "conduct its analysis in full independence and impartiality." The war between Israel and Hamas militants in Gaza last summer left more than 2,000 people dead. The inquiry will include alleged war crimes committed since June. The International Criminal Court was set up in 2002 to prosecute genocide, crimes against humanity and war crimes. CNN’s Vasco Cotovio, Kareem Khadder and Faith Karimi contributed to this report.

heading: Gold
text: Membership gives the ICC jurisdiction over alleged crimes committed in Palestinian territories since last June. Israel and the United States opposed the move, which could open the door to war crimes investigations against Israelis.

heading: BART
text: Palestinian Authority becomes 123rd member of the International Criminal Court. The move gives the court jurisdiction over alleged crimes in Palestinian territories. Israel and the United States opposed the Palestinians’ efforts to join the body.

heading: MVP
text: "Today brings us closer to our shared goals of justice and peace," foreign minister says. The Palestinians signed the ICC’s founding Rome Statute in January. The move gives the court jurisdiction over alleged crimes in Palestinian territories.

heading: MVP+S
text: "Today brings us closer to our shared goals of justice and peace," foreign minister says. The United States says it "strongly" disagrees with the decision. The Palestinian Authority is the 123rd member of the International Criminal Court.
Table 14: The second instance from the CNN/Daily Mail dataset.

heading: Input
text: Describe the following data: Abilene,_Texas | cityServed | Abilene_Regional_Airport

heading: Gold
text: Abilene, Texas is served by the Abilene regional airport. Abilene Regional Airport serves the city of Abilene in Texas.

heading: BART
text: Abilene Regional Airport serves the city of Abilene in Texas.

heading: MVP
text: Abilene Regional Airport serves the city of Abilene, Texas.

heading: MVP+S
text: Abilene Regional Airport serves the city of Abilene, Texas.

heading: Input
text: Describe the following data: "Madrid, Paracuellos de Jarama, San Sebastián de los Reyes and Alcobendas" | location | Adolfo_Suárez_Madrid–Barajas_Airport

heading: Gold
text: Adolfo Suárez Madrid–Barajas Airport can be found in Madrid, Paracuellos de Jarama, San Sebastián de los Reyes and Alcobendas. Adolfo Suarez Madrid-Barajas airport is located at Madrid, Paracuellos de Jarama, San Sebastián de los Reyes and Alcobendas. Adolfo Suarez Madrid-Barajas Airport is located in Madrid, Paracuellos de Jarama, San Sebastian de los Reyes and Alcobendas.

heading: BART
text: Adolfo Suárez Madrid–Barajas Airport can be found in Madrid, Paracuellos de Jarama, San Sebastián de los Reyes and Alcobendas.

heading: MVP
text: Adolfo Suárez Madrid–Barajas Airport can be found in Madrid, Paracuellos de Jarama, San Sebastián de los Reyes and Alcobendas.

heading: MVP+S
text: Adolfo Suárez Madrid–Barajas Airport is located in Madrid, Paracuellos de Jarama, San Sebastián de los Reyes and Alcobendas.
Table 16: The second instance from the WebNLG dataset, which has three golden target sentences.

heading: Input
text: Generate the question based on the answer: Saint Bernadette Soubirous [SEP] Architecturally , the school has a Catholic character . Atop the Main Building ’ s gold dome is a golden statue of the Virgin Mary . Immediately in front of the Main Building and facing it , is a copper statue of Christ with arms upraised with the legend " Venite Ad Me Omnes " . Next to the Main Building is the Basilica of the Sacred Heart . Immediately behind the basilica is the Grotto , a Marian place of prayer and reflection . It is a replica of the grotto at Lourdes , France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858 . At the end of the main drive ( and in a direct line that connects through 3 statues and the Gold Dome ) , is a simple , modern stone statue of Mary .

heading: Gold
text: To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France ?

heading: BART
text: Who is believed to have appeared to the Virgin Mary at Lourdes ?

heading: MVP
text: Who did the Virgin Mary appear to in Lourdes ?

heading: MVP+S
text: The Grotto is a replica of the grotto at Lourdes , France where the Virgin Mary appeared to whom ?
Table 17: The first instance from the SQuAD dataset.

heading: Input
text: Generate the question based on the answer: a copper statue of Christ [SEP] Architecturally , the school has a Catholic character . Atop the Main Building ’ s gold dome is a golden statue of the Virgin Mary . Immediately in front of the Main Building and facing it , is a copper statue of Christ with arms upraised with the legend " Venite Ad Me Omnes " . Next to the Main Building is the Basilica of the Sacred Heart . Immediately behind the basilica is the Grotto , a Marian place of prayer and reflection . It is a replica of the grotto at Lourdes , France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858 . At the end of the main drive ( and in a direct line that connects through 3 statues and the Gold Dome ) , is a simple , modern stone statue of Mary .

heading: Gold
text: What is in front of the Notre Dame Main Building ?

heading: BART
text: What is in front of the Main Building and facing it ?

heading: MVP
text: What is immediately in front of the Main Building ?

heading: MVP+S
text: What is immediately in front of the Main Building ?
Table 18: The second instance from the SQuAD dataset.

heading: Input
text: Answer the following question: what color was cotton ? [X_SEP] once upon a time , in a barn near a farm house , there lived a little white kitten named cotton . cotton lived high up in a nice warm place above the barn where all of the farmer ’ s horses slept . but cotton wasn ’ t alone in her little home above the barn , oh no . she shared her hay bed with her mommy and 5 other sisters . all of her sisters were cute and fluffy , like cotton . but she was the only white one in the bunch . the rest of her sisters were all orange with beautiful white tiger stripes like cotton ’ s mommy . being different made cotton quite sad . she often wished she looked like the rest of her family . so one day , when cotton found a can of the old farmer ’ s orange paint , she used it to paint herself like them . when her mommy and sisters found her they started laughing . " what are you doing , cotton ? ! " " i only wanted to be more like you " . cotton ’ s mommy rubbed her face on cotton ’ s and said " oh cotton , but your fur is so pretty and special , like you . we would never want you to be any other way " . and with that , cotton ’ s mommy picked her up and dropped her into a big bucket of water . when cotton came out she was herself again . her sisters licked her face until cotton ’ s fur was all all dry . " don ’ t ever do that again , cotton ! " they all cried . " next time you might mess up that pretty white fur of yours and we wouldn ’ t want that ! " then cotton thought , " i change my mind . i like being special " .
Gold white
BART white
MVP white
MVP+S white

heading: Input
text: Answer the following question: what color was cotton ? [SEP] white [X_SEP] where did she live ? [X_SEP] once upon a time , in a barn near a farm house , there lived a little white kitten named cotton . cotton lived high up in a nice warm place above the barn where all of the farmer ’ s horses slept . but cotton wasn ’ t alone in her little home above the barn , oh no . she shared her hay bed with her mommy and 5 other sisters . all of her sisters were cute and fluffy , like cotton . but she was the only white one in the bunch . the rest of her sisters were all orange with beautiful white tiger stripes like cotton ’ s mommy . being different made cotton quite sad . she often wished she looked like the rest of her family . so one day , when cotton found a can of the old farmer ’ s orange paint , she used it to paint herself like them . when her mommy and sisters found her they started laughing . " what are you doing , cotton ? ! " " i only wanted to be more like you " . cotton ’ s mommy rubbed her face on cotton ’ s and said " oh cotton , but your fur is so pretty and special , like you . we would never want you to be any other way " . and with that , cotton ’ s mommy picked her up and dropped her into a big bucket of water . when cotton came out she was herself again . her sisters licked her face until cotton ’ s fur was all all dry . " don ’ t ever do that again , cotton ! " they all cried . " next time you might mess up that pretty white fur of yours and we wouldn ’ t want that ! " then cotton thought , " i change my mind . i like being special " .
Gold in a barn
BART in a barn
MVP in a barn
MVP+S in a barn

heading: Input
text: Given the story title: male was out jogging one morning .

heading: Gold
text: the weather was crisp and cool . male felt good and energetic . he decided to keep jogging longer than normal . male went several more miles out of his way .

heading: BART
text: he saw a man struggling to walk across a field . male stopped to help him . he saw a guy struggling to walking across a fields . the man thanked male profusely !

heading: MVP
text: he ran into a man who was holding a knife . male was scared , so he ran away . the man threatened male . male ran back into his house .

heading: MVP+S
text: he noticed a sound coming from a tree . male went to investigate . he found a small baby bird stuck in the trunk . male took the bird to the vet and they released it .
Table 21: The first instance from the ROCStories dataset.

heading: Input
text: Given the story title: ken was driving around in the snow .

heading: Gold
text: he needed to get home from work . he was driving slowly to avoid accidents . unfortunately the roads were too slick and ken lost control . his tires lost traction and he hit a tree .

heading: BART
text: he hit a patch of black ice . ken slid and hurt his knee . he went to the doctor for it . krampus was sent home with a concussion .

heading: MVP
text: he was in his car and it was very cold . ken started to back up . the car started to make a loud noise . kena had to back out and call for help .

heading: MVP+S
text: he was having trouble starting his car . he had to push it a couple of blocks . his car would n’t start . ken had to call a tow truck for help .
Table 22: The second instance from the ROCStories dataset.

heading: Input
text: Given the dialog: i love to meet new people . [SEP] i have a turtle named timothy . [SEP] my favorite sport is ultimate frisbee . [SEP] my parents are living in bora bora . [SEP] autumn is my favorite season . [X_SEP] hello , how are you doing tonight ?

heading: Gold
text: i am well an loving this interaction how are you ?

heading: BART
text: i am doing well , how are you ?

heading: MVP
text: i am doing well . how are you ?

heading: MVP+S
text: i am doing well . how are you ?
Table 23: The first instance from the PersonaChat dataset.

heading: Input
text: Given the dialog: i just bought a brand new house . [SEP] i like to dance at the club . [SEP] i run a dog obedience school . [SEP] i have a big sweet tooth . [SEP] i like taking and posting selkies . [X_SEP] hello , how are you doing tonight ? [SEP] i am well an loving this interaction how are you ?

heading: Gold
text: i am great . i just got back from the club .

heading: BART
text: i am doing well , thanks for asking . what do you do for fun ?

heading: MVP
text: i am doing well . i just got home from dancing at the club .

heading: MVP+S
text: i am doing well . i just bought a new house .
Table 24: The second instance from the PersonaChat dataset.

heading: Input
text: Given the task dialog: Belief state [X_SEP] i would like a taxi from saint john ’s college to pizza hut fen ditton . Given the task dialog: Dialogue action [X_SEP] [db_nores] [X_SEP] i would like a taxi from saint john ’s college to pizza hut fen ditton . Given the task dialog: System response [X_SEP] [db_nores] [X_SEP] i would like a taxi from saint john ’s college to pizza hut fen ditton .

heading: Gold
text: [taxi] destination pizza hut fen ditton departure saint john ’s college [taxi] [request] leave arrive what time do you want to leave and what time do you want to arrive by ?

heading: BART
text: [taxi] destination pizza express fen ditton departure saint johns college [taxi] [request] leave what time would you like to leave or arrive ?

heading: MVP
text: [taxi] destination pizza hut fen ditton departure saint john ’s college [taxi] [request] leave sure ! when would you like to arrive ?

heading: MVP+S
text: [taxi] destination pizza hut fen ditton departure saint john ’s college [taxi] [request] leave arrive what time would you like to leave ?
Table 25: The first instance from the MultiWOZ dataset. The task-oriented dialogue is composed of dialogue state tracking, dialogue action learning, and system response generation.

heading: Input
text: Given the task dialog: Belief state [X_SEP] i would like a taxi from saint john ’s college to pizza hut fen ditton . [SEP] what time do you want to leave and what time do you want to arrive by ? [SEP] i want to leave after 17:15 . Given the task dialog: Dialogue action [X_SEP] [db_nores] [X_SEP] i would like a taxi from saint john ’s college to pizza hut fen ditton . [SEP] what time do you want to leave and what time do you want to arrive by ? [SEP] i want to leave after 17:15 . Given the task dialog: System response [X_SEP] [db_nores] [X_SEP] i would like a taxi from saint john ’s college to pizza hut fen ditton . [SEP] what time do you want to leave and what time do you want to arrive by ? [SEP] i want to leave after 17:15 .

heading: Gold
text: [taxi] destination pizza hut fen ditton departure saint john ’s college leave 17:15 [taxi] [inform] car phone booking completed ! your taxi will be [value_car] contact number is [value_phone]

heading: BART
text: [taxi] destination pizza hut fen ditton departure saint john ’s college leave 17:15 [taxi] [inform] car phone booking completed ! booked car type : [value_car] contact number : [value_phone]

heading: MVP
text: [taxi] destination pizza hut fen ditton departure saint john ’s college leave 17:15 [taxi] [inform] car phone booking completed ! booked car type : [value_car] contact number : [value_phone]

heading: MVP+S
text: [taxi] destination pizza hut fen ditton departure saint john ’s college leave 17:15 [taxi] [inform] car phone booking completed ! booked car type : [value_car] contact number : [value_phone]
Table 26: The second instance from the MultiWOZ dataset.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section Broader Impacts

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: See below

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section A.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section A.1 and Tables 8 and 9

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5 - Applicability

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Sections Limitations, A.1, and A.2

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section A.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Sections A.1 and E, Tables 8 and 9
C 3 Did you run computational experiments?

heading: See below

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Sections 3.2 and 3.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Sections 3.3 and B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Sections 4 and B
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: See below

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? See Figure 2

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? See Section D

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? See Figure 2
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: See Section D

heading: Future Work
text: We hope that these open-source resources will facilitate future work on supervised pre-training and contribute to the advancement of NLG research