MVP: Multi-task Supervised Pre-training for Natural Language Generation
Authors: Tianyi Tang
Junyi Li
Xin Zhao
Ji-Rong Wen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Natural language generation
pretrained language models
multi-task learning
prompt learning

Abstract: Pre-trained language models (PLMs) have achieved remarkable success in natural language generation (NLG) tasks. Up to now, most NLG-oriented PLMs are pre-trained in an unsupervised manner using the large-scale general corpus. In the meanwhile, an increasing number of models pre-trained with labeled data (i.e., “supervised pre-training”) showcase superior performance compared to unsupervised pre-trained models. Motivated by the success of supervised pre-training, we propose Multi-task superVised Pre-training (MVP) for natural language generation. We collect a large-scale natural language generation corpus, MVPCorpus, from $77$ datasets over $11$ diverse NLG tasks. Then we unify these examples into a general text-to-text format to pre-train the text generation model MVP in a supervised manner. For each task, we further pre-train specific soft prompts to stimulate the model’s capacity to perform a specific task. Extensive experiments have demonstrated the effectiveness and generalizability of our MVP model in a number of NLG tasks, which achieves state-of-the-art performance on $13$ out of $17$ datasets.

Under review as a conference paper at ICLR 2023
MVP: MULTI-TASK SUPERVISED PRE-TRAINING FOR NATURAL LANGUAGE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-trained language models (PLMs) have achieved remarkable success in natu- ral language generation (NLG) tasks. Up to now, most NLG-oriented PLMs are pre-trained in an unsupervised manner using the large-scale general corpus. In the meanwhile, an increasing number of models pre-trained with labeled data (i.e., "su- pervised pre-trainin pre-training") showcase superior performance compared to unsupervised pre-trained models. Motivated by the success of supervised pre- pre-training, we pro- pose Multi-task super Pre-training (MVP) for natural language generation. We collect a large-scale natural language generation corpus, MVPCorpus, from 77 datasets over 11 diverse NLG tasks. Then we unify these examples into a general text-to-text format to pre-train the text generation model MVP in a supervised manner. For each task, we further pre-train specific soft prompts to stimulate the model's capacity to perform specific task. Extensive experiments have demon- strated the effectiveness and generalizability of our MVP model in number of NLG tasks, which achieves state-of-the-art performance 13 out of 17 datasets.
INTRODUCTION
Natural language generation (NLG, also known as text generation) is a crucial capacity for language intelligence, which aims to generate numan-like texts on demand (Garbacea & Mei 2020p. Since the emergence of the pre-training and fine-tuning paradigm, pre-trained language models (PLMs) have
dominated the mainstream approaches for NLG tasks (Lewis et al.) 2020; Brown et al. 2020). With large-scale general corpus, the majority of PLMs are pre-trained in unsupervised (self-supervised) manner by leveraging intrinsic data correlations as supervision signals. However, unsupervised pre-training is likely incorporate noise that affects the performance of downstream tasks (Feng
ct al. 2022). also leading to a slower rate of acquiring knowledge (Zhang et al., (021).
In the meanwhile, more and more large-scale labeled datasets have become easily accessible (Deng et al. 2009 Liu et al., 2020). There is growing evidence that pre-training with labeled data can further improve the performance of PLMs, both in the fields of computer vision (He et al. 2016
Dosovitskiy et al. 2021) and natural language processing (Lin et al. (2020b) Su et al. 2022). These promising developments motivate
consider pre-training text generation models with labeled data, which is called "supervised pre-training" (Feng et al. (022). Existing work has shown that supervised pre-training can explicitly learn task-specific characteristics and alleviate the discrepancy
between unsupervised pre- pre-training and supervised fine-tuning (Sanh et al. 2022 Lin et al.) 2020b).
Furthermore, most NLG systems are often trained a supervised way, requiring supervision signals to learn the input-to-outper transformation. For example, dialogue systems learn to generate appropriate responses based on historical utterances, and text summarization systems learn extract essential information from long documents according to human- written summaries. Therefore, we suspect that supervised pre-training is more suited for NI.G-oriented PLMS in essence since can provide task-related instructions early in the pre-training stage instead of later fine-tuning stage.
Inspired by the recent success of supervised pre-training, we propose Multi-task super Pre- training (MVP) for natural language generation by leveraging variety of labeled text generation datasets. Specially, we collect a large-scale labeled corpus, MVPCorpus, consisting of 77 datasets over 11 text generation tasks. Since recent research shows that extensive scale of multi-tase pre- training (Aribandi et al.; 2022) is the key to generalizing to new tasks for large PLMs, we combine these labeled datasets for multi-task pre-training. Existing popular works, as shown in Table mainly

---

Under review as a conference paper at ICLR 2023
Table 1: Representative PLMs for NLG and NLU tasks using (un)supervised pre-training. We present a more detailed comparison and discussion about supervised pre-training in Section
Settings
Supervised Pre-training
Unsupervised Pre-training
NLG
MVP (ours)
GPT-2, GPT-3, BART, TS, UniLM, MASS, PEGASUS
NLU FLAN, TO, Muppet, ExT5
BERT, RoBERTa, T5, UniLM, XLNeI, ELECTRA
2022: Arihandi et al. 2022 or use unsupervised pre-training Lewis et al. 2020 Raffel al., 2020). with consideration of supervised pre-training NLG tasks. To fill this gap. we explore supervised pre-training and multi-task learning for deriving both effective and general NLG models.
focus on NLU tasks (Sanh et al.
To develop our approach, we adopt a Transformer-based (Vaswani et al. 2017) sequence-to-sequence
model as the pre-training backbone. In multi-task training, different tasks may "neutralize" the ability learned through other tasks (He & Choi 202I). To mitigate this potential issue, we propose to learn task-specific prompts based on the MVP model, following the structure of prefix-tuning (Li & Liang ). Task-specific pre-training enables prompts to "store" specialized knowledge for each corresponding task. Integrating MVP with task-specific prompts can further stimulate the model's capacity to perform some specific tasks.
To summarize, our main contributions center around the following research questions:
How to train an NLG-oriented PLM in supervised pre-training way? In order to prepare the supervised corpus, we collect massive labeled MVPCorpus, consisting 77 datasets over 11 NLG tasks across various domains and specific objectives. To the best of our knowledge, MVPCorpus is the largest collection of NLG datasets. Firstly, we formulate different NLG tasks as a general text-to-text form so that the supervised corpus can be used in unified way for pre-tratn pre-training NLG model. Our work presents a simple yet general approach for pre- gre-training a more capable NLG model by leveraging various labeled NLG datasets.
Can supervised pre-trained NLG models be both effective and general? Extensive experiments show that the supervised pre-trained MVP outperforms its unsupervised pre-trained counterpart BART in both full tuning (+7.00 7.0% on avarege) and parameter-efficieet tuning (+4.3% on avarege) settings. Our MVP model achieves state-of of-the-f- performance on 13 out of 17 datasets. Furthermore, the experiments on unseen NLG and NLU tasks demonstrate that our supervised MVP model has strong generalization ability for unseen tasks.
For reproducing and reusing our work, we release the collection MVPCorpus, the models (e.g. MVP. task-specific prompts, and multi-task variants), and codes for pre-training and fine-tuning at the link: httpttt::///oonmmoouuii 4open. www.:ssceeeeeecccneeeee Pttp:c//ee//cccce2002e2eeeee1111
RELATED WORK
Pre-trained Language Models. Pre-trained language models have achieved exceptional success in a wide range of tasks, and the majority of them are pre-trained in an unsupervised manner (Brown al. 2020 Devlin et al. 2019 Lewis et al. 2020 Raffel et al. 2020). For example, with large-scale scale plain texts as the unsupervised pre-training corpus, GPT-3 (Brown et al. 2020) employ language modeling as the pre-training task, i.e., predicting the next token conditioned on previous tokens;
BART (Lewis et al.) 2020) learns to recover the original text from corrupted text which has been altered by arbitrary noise transformations, GPT-3 and BART use 570GB and 160GB of unlabeled text as the pre-training corpora, respectively. In the meanwhile, the computer vision community benefits lot from the labeled dataset ImageNet (Deng et al. 2009). Influential models, such as ResNet (He et al., 2016) and VIT (Dosovitskiy et al.] 2021). leverage ImageNet for pre-training. Inspired by the success of pre-training with labeled data, machine translation researchers explore supervised pre-training (McCann et al. 2017 Lin al. 2020b). Lin et al. (2020b) attempt to pre-train a translation model with parallel data in multiple languages. Despite using much less pre-trained data, mRASP still achieves better performance than translation models pre-traie pre-trained in unsupervised manner (Lample & Conneau 2019; Liu et al. 2020). In this paper, we propose to pre-train a universal NLG model in supervised manner with collections of labeled datasets (23GB).
2

---

Under review as a conference paper at ICLR 2023
Stage 1: Multi-task Supervised Pre-training
Stage 2: spss-sppccficc Prompt Pre-training
Summarization
Summurization
MVP
MVP
Data-to-text
Data-to-texee
Dialogue
Dialogue
soft prompl
Figure 1: The overview of the pre-training process of our MVP model and task-specific prompts.
Multi-task Learning. Our pre-training process is also related to multi-task learning (MTL), method of mixing multiple tasks into single training process (Collobert Weston 2008).
model trained with MTL can benefit from helpful knowledge of relevant tasks, resulting in improved performance (McCann et al.] 2018; Subramamian et al. 2018). Recently, MT-DNN (Liu et al.) 2019a)
and Muppet (Aghajanyan al. (021) collect tens of datasets in the multi-task procedure and achieve downstream tasks. The pre-finetuning schema proposed in Muppet shares a similar idea with our study. Arrianndi et al.] (2022) further combine the denoising pre-training task of TS (Raffel et al. (020) and multi-task learning to pre-train a new model, ExT5. MTL has also contributed sub-fields of text generation, such as open-ended dialogue system (Zhang et al. 22220),
better performance
lask-triented dialogue system (Su et al. 2022), text style transfer (Bujnowski et al. (020)) and question answering (Khashabi et al.) 2020). At the same time, researchers explore the transferability
of models trained on multi-task datasets (Mishra et al.) 2022). FLANN (Wei et al., 2022), TO (Sanh al., 2022), and ZeroPrompt (Xu er al. 2022) investigate the zero-shot generalization abilities of large PLMs trained numerous task datasets with well-designed prompts. Ye et al. (202I) develop a benchmark CrossFit to study the few-shot learning ability of models. Compared with these works,
aim to explore multi-task learning to derive both effective and general NLG models.
Prompt Learning. Prompt learning is thriving method in the field of NLP. Prompt learning con- verts fine-tuning text into a format similar to pre-training to leverage implicit pre- pre-traing training knowledge and alleviate the discrepancy between pre-training and fine-tuning (Liu al. 2021b). GPT-2 (Rad- ford et al. 2019) and T5 (Raffel et al. 2020) add human-written task prompts to the input text. For instance, 15 prepends "Summarize:"
the input document for summarization tasks; GPT-3 (Brown) et al. 2020) further combines several demonstrations to input to learn task patterns, which is called in-context learning. Some researchers also design elaborate prompts or demonstrations for each task and dataset and investigate their effectiveness and robustness (Wei et al. 2022) Sanh et al. 2022; et al. [2022] Mishra al. (002).. To overcome the constraints of manually constructed prompts, researchers develop continuous (soft) prompts that can be optimized in the continuous space (Lester et al.) 2021 Qin Eisner (02T). Prefix-tuning (Li & Liang 2021) increases
number of parameters in prompts and employs prompting each Transformer layer. Gu et al. (2022) propose PPT to pre-train continuous prompts using unlabeled data. SPoT (Vu et al.) (2022) and UnifiedSKG (Xie et al., 2022 learn the prompts on related tasks and transfer them to new tasks.
THE MVP MODEL
This section introduces our MVP model: Multi-task superYised Pre-trained model for natural language generation. We first collect a large-scale NLG corpus, MVPCorpus, from 77 datasets over 11 diverse NLG tasks. After that, pre-train our MVP model using mixture of labeled data from MVPCorpus. We further learn the task-specific prompts to stimulate the MVP model to perform a certain task. The overview of our model is illustrated in Figure] 1
3

---

Under review as a conference paper at ICLR 2023
3.1
DATA COLLECTION
Formally, the natural language generation (NLG) task aims to generate a sequence of tokens 3/ (y1. N2..y..... ................................ .n)..)) conditioned on input data X' (e.g., a piece of text or structured data) (Li et al. 2022). Typically, NLG tasks are categorized according to the data format of X' and V. For example, text summarization condenses a long document into brief text containing essential information; data- to-text generation produces descriptive text about structured input; and dialogue system creates pertinent responses given multiple dialog utterances.
In this paper, we collect large-scale labeled MVPCorpus consisting of 77 labeled datasets from 11 including commonsense generation, data-to-text generation, open-ended dialogue system, paraphrase generation, question answering, question generation, story generation, task-oriented dialogue system, text simplification, text style transfer, and text summarization. These datasets come from various domains and are of different sizes. Some datasets elaborately hand- crafted and thus relatively small in size, while others are created for large-scale weak supervision. Despite originating from various tasks, these diverse labeled datasets contain rich task-specific supervision signals for establishing global -equence-t--sequencc mapping relations. The detailed descriptions of these tasks can be found in Appendix [..
representative NLG tasks !
Next, we transform all tasks into a unified text-to-text format and convert different input data X into text format. For instance, we linearize structured data (e.g., knowledge graph or table) by concatenating triples or key-value pairs using the special token " [[EPP] ] for data-to-text generation, and we utilize the special token " [x SEP] ]" to separate answer and paragraph for question generation. The transformed input format for each task can be found in Appendix [
We divide MVPCorpus into two parts, which are used for pre-training and fine-tuning (evaluation), respectively. For supervised pre-training, we utilize 50 datasets from 7 tasks, including data-to-text generation, open-ended dialogue system, question answering, question generation, story generation, task-oriented dialogue system, and text summarization. To enrich reverse the input and output of some tasks for obtaining new datasets (e.g., story generation and summarization, question generation and question answering). We also eliminate pre-training examples overlapping with evaluation data to avoid data leakage (more details in Appendix [B.2). Finally, we have a 25GB supervised pre-training corpus containing 32M examples. The statisties of datasets for pre-training are listed in Table 7
pre-training corpus,
For evaluation, we utilize the rest 27 datasets which are more commonly used in the literature. Among these datasets, 23 datasets are from the 7 tasks used in pre-training. We refer to them as seen tasks and use them to test the effecttiveness of our model. The remaining 4 datasets are from the tasks of commonsense generation, paraphrase generation, simplification, and style transfer, respectively. We call them unseen tasks and use them to examine the generalization ability of our model.
3.2 MODEL ARCHITECTURE
We pre-train our MVP model and task aask-specific prompts in two stages. In the first stage, we tre-train the MVP backbone using mixture of labeled datasets from seven tasks to learn general text-to-text relationships and transferable semantic information across tasks. To indicate each task, we apply Human--ritten prompts to each task instance. For example, we write "Summarize:" as the prompt for summarization tasks. The manual prompts for each task are shown in Appendix EE]
In the second stage, we freeze our MVP backbone and pre-train set of task task-specific soft prompts (iee, continuous vectors) to stimulate the model's 's capacity to perform some specific tasks. We learn them using a mixture of corresponding intra-task datasets (i.e., datasets under the same task F. These soft prompts, which are not shared between tasks, encode the task-specific semantic knowledge to alleviate the blurring-out problem induced by multi-task learning (He & Choi 2021).
Specifically, we employ the standard Transformer encoder-decoder /Vaswani et al. 2017) as our backbone. Compared decoder-only architectures such GPT-3 (Brown et al. 2020) and prefix LMs such as UniLM (Dong et al. 2019). the encoder-decoder architecture is more effective for
Wee do not consider machine translation tasks hut only focusing on English tasks in this work. For instance, we train summarization-specfffc prompts using summarization datasets (e.g., News- (Grusky al. 2018), WikiHow (Koupaee & Wang 2018), and MSNews (Liu et al. 2021a)).
4

---

Under review as a conference paper at ICLR 2023
text generation tasks (Raffel et al., 2020). As for task-specific soft prompts, we insert continuous vectors at each Transformer layer, following prefix-tuning (Li & Liang 202I). Compared to prompt tuning (Lester et al. 202I), which only adds trainable embeddings to the input layer, the layer-wa layer-wise prompting of prefix-tuning is more effective and stable (Liu et al. 2022). especially for NLG tasks.
3.3 TRAINING DETAILS
Our MVP model adopts Transformer with 12 layers in both encoder and decoder (406M parameters), the same as the model size of BART (Lewis et al. 2020). The hidden size is 1,024 and the inner hidden size of the feed-forward network is 4,096. We employ the byte-pair-ene pat---aiii--iinggggg (BPE) tokenizer, and the vocabulary size is 50,267. We initialize the backbone with the BART parameters to provide good starting point for NLG tasks following previous work (Dong et al. 2019 Zhang et al., 2020). We pre-train the model with batch size of 8,192 and adopt cemperature-scalle mixing strategy (Raffel et al. 2020) with a rate of T = 2 to mitigate the disparity in tasks and datasets.
We follow prefix-tuning (Li & Liang 202 to pre-train task-specific prompts by prepending trainable continuous vectors to the keys and values of the multi-head attention module at each layer. The prompt length is set to 100, and we utilize the MLP reparameterization function with a hidden size of 800 to improve the training robustness and performance (Li & Liang 2021). Hence, every task prompts have approximately 62M parameters. Then, we freeze the MVP model and train seven groups of task-specific prompts, each of which corresponds to a different task. The batch size is set to 8,192, and we leverage the mixing strategy with rate of T = 2.
In the two stages, the maximum length of both input and output sequences is set to 1,024 for supporting examples contain more tokens. We optimize the model with constant learning rate of 3 X 10-5 using standard sequence-to-sequeee
cross-entropy loss. We apply the AdamW optimizer (Loshchilov & Hutter 2019) with 31 0.9, 32 = 0.98, c = 1 10-6 to improve training stability (Liu et 2019bb. The weight decay coefficient is 0.1. For testing, we select the checkpoint with the highest validation performance. All the experiments are conducted on 32 NVIDIA Tesla V100 32GB GPUs. We implement our model using the library Hugging Face (Wolf et al. (020))
In summary, we pre-train a 406M text generation model MVP and seven groups of 62M task-specific prompts. For each downstream task, users can either utilize the MVP backbone (406M) directly or further combine MVP with task-specific prompts (468M).
EXPERIMENT RESULTS
In this section, we mainly investigate the effectiveness of our proposed supervised pre-training for NLG. Specifically, we fine-tune our MVP model on new datasets for pre-trained (seen) generation tasks under full tuning and carameter-efficien tuming settings.
For the full tuning setting, fine-tune the entire model (including the backbone MVP and prompts), while for the parameter-efficiee of MVP. We optimize the model via the seq2seq loss with label smoothing (Szegedy et al.) 2016) factor of 0.1 and the AdamW W optimizer with default hyper-parametto
tuning, we only fine-tune prompts but freeze the parameter weights
We sweep over the batch size 3 x 10-53 to find the optimal hyper-
in {16, 64, 256} and the learning rate in (5 10 6 10 parameters for each evaluation task. We utilize the checkpoint with the best validation performance for test set inference. During inference, we set the beam size to 5 and the no-repetitive ngram size to 3. For evaluation, we leverage the automatic generation metrics BLEU (Papineni et al., 2002). ROUGE (Lin) 2004), and METEOR (Banerjee & Lavie 2005) to measure the quality of the generated text and employ Distinct (Li al. 2016) to evaluate its diversity. Details regarding fine-tuning and evaluation can be found in Appendix C
We conduct extensive experiments with in different settings. Under full tuning scenarios, we employ the 23 datasets from 7 seen tasks for evaluation. Section 4.1 and Appendix D]analyze the performance of our methods these datasets. To better compare with ExT5 (Aribandi et al. 2022). we conduct experiments on the GEM benchmark (Gehrmann et al., 2021) in Appendix D.2 Under parameter- efficient tuning settings,
utilize the same datasets as Section 4.1 and the results can be found in Section 4.2 Furthermore, we evaluate our models without fine- tuning and compare them with TO (Sanh et al. 2022) in Appendix D.3 These extensive results show that our MVP model consistently
5

---

Under review as a conference paper at ICLR 2023
Table 2:
The main results seven tasks under full tuning settings. The best and second-best results among all the methods are marked in bold and underlined. respectively. The SQuAD dataset
here is used for the question generation task. The letters B,
and ME denote BLEU, ROUGE,
Distinct, and METEOR, respectively. "_" means the work does not compute the corresponding result.
These setups and abbreviations are the same below. (Ravaut al.
2022
(Ke et al.
2021
(Bao et al. 2021 (Xiao et al. 2020) (Lewis et al. 2020)
Liu et al.
202
(Gunn
et al. 2021
(Chen et al. 2022)
(He et al. 2022) (Lin et al. (020c)
CNN/DailyMai
WebNL.G
SQuAD (QG)
CoQA
Methods
R-1
R-2
R-L
B-4
ME
R-L
B-4
ME
R-L
F1
EM
SOTA
47.16'
22.55
43.87
66.14
47.25
76.10
25.97
27.33
53.43
84.50"
MVP
44.44
21.61
40.99
67.76
47.74
77.04
26.21
27.20
53.46
86.51
77.62
BART
44.16
21.28
40.90
64.55
46.51
75.13
22.00'
26.40
52.55
68. 68.60V
Single
44.36
21.54
40.88
67.74
46.89
76.94
26.09
27.15
53.29
86.20
77.26
MVP+S
44.30
21.53
40.83
68.21
47.77
77.09
26.13
27.18
53.44
86.75
77.97
MVP+R
44.14
21.45
40.72
67.61
47.65
76.70
25.71
27.03
53.09
85,95
77.22
MVP+M
43.97
21.16
40.46
67.45
47.57
76.81
25.46
26.79
52.95
77.26
ROCStories
Persona
MultiWOZ
Methods
B-1
B-2
D-1
D-4
B-1
B-2
D-1
D-2
B-4
Success
Inform
SOTA
33.40
15.40
69.30
49.90'
40.00
1.50
9.40
20,50'
85.30
94.40
MVP
33.42
15.54
2.92
75.06
50.07
40.54
1.54
9.86
20.04
79.20
87.20
BART
30.70
13.30
69.90
49.90'
40.00
1.30
8.00
17.89
74.91
84.88
Single
32.67
15.29
2.72
72.97
49.96
40.53
1.27
7.63
19.73
75.60
83.70
MVP+S
33.92
15.60
3,44
80.58
47.64
39.84
1.40
8.10
19.91
77.60
84.60
MVP+R
32.93
15.32
2.88
73.83
48 45
40.09
1.30
7.95
19.02
73.30
81.80
MVP+M
33.30
15.51
2.71
74.24
46.26
39.30
1.36
8.07
19.93
72.70
79.70
outperforms various baselines in different scenarios, which demonstrates the effectiveness of our supervised pre-training method for NLG.
4.1 FULL TUNING PERFORMANCE
We design several model variants verify the effectiveness of two-stage pre-training method proposed in Section 3.2 For the first-stage model MVP that uses multi-task supervised pre-training, we compare with competitive backbones using different pre-training strategies:
BART (Lewis al. 2020): BART widely-used PLM for natural language generation using unsupervised pre-training task, i.e, denoising auto encoder.
Single-task pre-training (Single): We individually train single model for each task using intra- task datasets under the same pre-training settings in multi-task training. For instance, we pre-train summarization model using summarization datasets (e.g., Newsroom, WikiHow, and MSNews). Therefore, we have seven single-task pre-trate pre-trained models in total.
For the second-stage model that integrates pre-trained task-specific prompts (denoted by MVP+S), we compare with two variants using different prompts:
Randomly initialized prompts (MVP+R): The layer-wise prompts for the MVP model are randomly initialized without pre-training.
Multi-Task pre-trained prompts (MVP+M): We only pre-train one group of prompts for all tasks, using the same mixed datasets as in the backbone pre-training.
Besides these variants, further include the best-reported results from original papers in the we can see that:
literature for comparison (denoted as SOTA). From the results in Table 2
First, supervised pre-training models (i.e., MVP and Single) achieve better performance than the unsupervised pre-tre rreitraieed model BART, yielding an average improvement 7.0% and 4.4% (in ratio), respectively. This finding demonstrates the effectiveness of our supervised pre-training method. With

---

Under review as a conference paper at ICLR 2023
Table 3: The main results seven tasks under parameter-efficient settings. We also include the results of BART and MVP under the full tuning setting (denoted as FT) for comparison.
CNN/DailyMail
WebNLG
SQuAD (QG)
CoQA
Methods
R-1
R-2
R-L
B-4
ME
R-L
B-4
ME
R-L
F1
EM
FT BART
44.16
21.28
40.90
64.55
46.51
75.13 13
22.00
26.40
52.55
68.60
MVP
44.44
21.61
40.99
67.76
47.74
77.04
26.21
27.20
53.46
86.51
77.62
MVP4S
43.03
20.27
39.72
66.73
47.42
76.36
25.28
26.66
52.69
86.44
76.84
BART+R
42.47
19.82
39.15
65.54
46.86
75.24
24.27
26.07
52.03
82.22
71.92
MVP+R
42.84
20.21
39.61
66.12
47.12
75.83
25.05
26.34
52.57
85.51
75.56
MVP+M
42.99
20.36
39.70
66.40
47.16
75.89
25.24
26.49
52.88
85.90
76.34
ROCStories
Persona
MultiWOZ
Methods
B-1
B-2
D-1
D-4
B-1
B-2
D-1
D-2
B-4
Success
Inform
FT BART
30.70
13.30
69.90
49.90
40.00
1.30
8.00
17.89
74.91
84.88
FT MVP
33.42
15.54
2.92
75.06
50.07
40.54
1.54
9.86
20.04
79.20
87.20
MVP4S
32.94
15.12
2.98
71.09
47.11
39.51
1.39
7.28
19.24
71.40
77.80
BART+R
32.14
14.71
2.85
68.94
46.23
38.98
1.30
6.82
17.94
62.20
69.20
MVP+R
32.28
14.85
2.97
70.29
46.70
39.23
1.31
6.98
18.86
64.40
71.40
MVP+M
32.62
15.28
2.95
69.58
46.78
39.40
1.33
7.13
19.13
67.20
72.90
labeled datasets, supervised pre-training enables the model to acquire more task-specific information, thus leading to improved results downstream tasks. Regarding multi-task pre-training (MVP) and single-task (Single), our MVP model outperforms its single-task counterparts by 2.7%. This result indicates that the proposed multi-task learning approach enhance single-task performance by learning transferable semantic information across tasks.
Second, task-specific prompt learning is effective alleviate the "blurring-out" issue of multi-task learning. For tasks such as data-to-text generation and question answering, MVP with single-task prompt pre-training (MVP+S) consistently outperforms the other two variants (MVP+R. MVP+M). This verifies that task-specific prompts can acquire specialized knowledge of each task and stimulate the capacity of the MVP model to perform certain tasks.
Finally, our supervised pre-training approach achieves five new SOTA results data-to-text genera- tion, question generation, question answering, story generation, and open- andedended dialogue tasks in T We also achieve SOTA performance in six out of eight datasets in Table which shows the strong text generation capability our MVP model. As for the remaining tasks, the SOTA models incorporate specific techniques tailored to the tasks, e.g. the re-ranking framework (Ravaut et al.) 2022) and various task-specific objectives (He et al.) 2022). which yield better performance than our models. In contrast, the results of our models are very competitive, which is developed based general architecture and a unified learning objective.
4.2 PARAMETER- TUNING PERFORMANCE
In the lightweight fine-tuning setting, we only tune the prompts while freezing the backbone MVP model. Besides our MVP+S model, we consider comparing the following methods:
Prefix-tuning Li & Liang 202T): Prefix-tunin tiinix-tuinnggg popular prompt-based based lightweight tuning
method for text generation. We employ BART LaRge as its backbone, denoted as BART+R.
Only tuning randomly initialized prompts (MVP+R): This variant only tunes the randomly initialized prompts of MVP+R, and shares a similar idea with prefix-tuning.
Only tuning multi-task pre-trained prompts (MVP+M): This variant only tunes the multi-task pre-trained prompts of MVP+M. Such an idea has been used SPoT (Yu et al. 2022).
From the experimental results in Table lightweight settings further demonstrates the effectiveness of supervised pre-training. By comparing that MVP+R
can see that: the good performance of the MVP model in
two randomly initialized prompting methods (BART+R and MVP+R), achieves superior performance to BART+R (+2.0% due to its multi-task supervised backbone. Furthermore, when initialized with pre-trained prompts, MVP+S and MVP+M achieve improved
7

---

Under review as a conference paper at ICLR 2023
Table 4: The main results of unseen NLG tasks. We use AESOP and SC & BLEU denote the methods proposed by Sun et al. (2021) and Lai et al. (2021), respectively. Accuracy is calculated by a pre-trained TexCCNN evaluate the style strength, and HM denotes the harmonic mean of BLEU-4 and style accuracy. (Sun et al. 2021)
(Lai et al.
2021
Quora
GYAFCE&M
GYAFC F&R
AESOP
SC & BLEU
B-1
R-1
R-2
R-L
ME
B-4
Accuracy
HM
B-1
Accuracy
HM
+BART
47.30"
73.30
54.10
75.10
49.70
+BART +MVP
76,50*
93.70
83,90
79.30
92.00
85,20
+MVP
49,86
74.93
56.55
76.56
52.27
77.01
94.66
84.92
79.70
93.07
85.87
Table 5: The main results of NLU tasks the GLUE benchmark. We evaluate the results the official website hht : http:///quubbnnbne cow.ec.aaaaa.ommmmmmm....m.. Matt. means the Matthews correlation coefficient. Acc. stands for the accuracy rate. P/S Corr. denote Pearson and Spearman correlation coefficients. m./mm. refer to the accuracy of the matched and mismatched domains. Avg. is macro-average of scores defined in WWan al. (2019).
CoLA
SST-2
MRPC
STS-B
QQP
MNLI
QNLI
RTE Acc.
Methods
Average
Matt.
Acc.
FI/Ace.
P/S Cor
FI/Acc.
m/mm.
Acc.
BART
60,30
96.30
90.47 86.70
90.97 / 90.30
73.03 89.87
90.03 / 89.27
94.60
79.83 82.87
85.17
MVP
59.87
96.43
92.07 89.43
91.37/ / 90.90
73.20 90.13
89.70/ 88.73
95.10
85.88
results over MVP+R, which is consistent with the findings of SPoT Vu et al. (022). When compared with MVP+M, MVP+$ performs marginally better by 1.2%, indicating that task-specific prompts are useful improve the model in specific generation tasks.
Surprisingly, our lightweight MVP+S can even outperform fully tuned BART on tasks such as question generation and question answering, showcasing the effectiveness of the proposed supervised pre-training approach. Another note is that lightweight prompting methods (Lester et al. 2021 Vu
al., 2222) that work on NLU tasks cannot achieve competitive performances when compared to full tuning methods on NLG tasks.
GENERALIZATION ABILITY
In this section, we test our MVP model on unseen NLG and NLU tasks to verify the generalizability.
Generalization to Unseen NLG Tasks. According to Deng et al. (021), an NLG task can be assigned to one of the following three categories: compression (e.g., summarization), transduction (e.g., translation), or creation (e.g., story generation). Since we do not include any transduction tasks during pre-training, paraphrase generation and text style transfer. We select the SOTA methods for these two tasks, i.e., AESOP (Sun et al. 202I) for paraphrase generation and SC & BLEU (Lai al. 2021 for text style transfer, and replace their backbone BART with our MVP model for comparison. The experimental setup remains the same as described in Section 4 and details are reported in Appendix From the results in Table 4 we can see that our model outperforms BART by ratio of 2.2% and achieves two new SOTA results, which verifies the strong generalizability of our model. This finding shows that our MVP model is more capable than BART and can serve as general yet effective backbone to solve more specific tasks by providing superior parameter initialization.
evaluate our MVP model using two unseen transduction NL.G tasks:
Generalization to Unseen NLU Tasks. Although MVP is designed especially for NLG tasks, we also evaluate its performance on unseen NLU tasks using the widely-used GLUE benchmark (Wang) ct al.) 2019). We compare our model to BART using its original sequence classification method (Lewis et al. 2020). The detailed settings can be found in Appendix C According to the results presented in Table 5 our MVP model outperforms BART on 9 of 12 metrics and has a superior overall performance of O. 0.71%. This result indicates the strong generalization ability of our MVP model and further demonstrates that our supervised pre- pre-training not only learns generation ability but also improves the overall semantic representations.
8

---

Under review as a conference paper at ICLR 2023
Table 6: Comparison of our work with existing supervised pre-training methods. #NLG/#NLU denote the number of NLG and NLU tasks, respectively. PT denotes pre-training, FT denotes fine-tuning and SP denotes supervised pre-training.
Methods
#NLG (PT) #NLU (PT)
#NLG (FT)
#NLU (FT)
SP model
SP prompts
Open source
FLAN
X
TO
Muppet
/
X
ExT5
/
X
X
SPoT
X
MVP (ours)
11
/
DISCUSSION
Differences with Existing Methods. To the best of our knowledge, existing supervised pre-training works mainly focus NLU tasks (Aghajanyan et al. 2021 Aribandi et al. 2022) or small number
of NLG tasks (Lin et al. 2020b Su et al.) 2022). Given the superior performance achieved by supervised pre-traiming approaches, it important to explore supervised pre- pre-training for deriving both effective and general NLG models. Our work makes a significant contribution in this direction, achieving SOTA performance with single model 13 of 17 datasets. Compared with its strong counterpart ExT5 (Aribandi et al. 2022). our MVP model outperforms it in 26 out of 27 metrics (detailed in Appendix D.2). In order better understand the difference between our paper with previous supervised (multi-task) pre-trate pre-training studies, we present a detailed comparison in Table 6 As we can see, our work conducts the study with the largest number of NL. tasks for both supervised pre-training and fine-tuning, incorporates task-specific prompts, and also releases all the important resources for reproducing or reusing our work.
Applicability. To facilitate the application of our work, we have released the collection corpus, pre-trained models, task-specific prompts, and the generated texts. Our collected MVPCorpus the largest NLG task collection. We use all the data pre-train a general model or select subset to continue pre-training a domain- or task- specific model (Gururangan et al. 2020). Our MVPCorpus the evaluation benchmark for different NLG tasks. Furthermore,
also be considered MVP model can be used to achieve new state-of saute-of-thh-ar results in various NLG tasks. Users can either fine-tune the MVP model integrate it with task-specific prompts achieve better results based on sufficient labeled data. Even data-scarce domains, our MVP model can be also directly employed to obtain good performance without fine-tuning. In addition, our MVP model can provide effective parameter initialization for improving existing methods, as described in Section 5 Finally, the pre-trained task-specific prompts and the generated texts can be further used to study the task similarity and their effect on the multi-task pre-training.
7 CONCLUSION
In this paper, we present Multi-task superVised Pre-training (MVP) for natural language generation. Firstly, we collect large-scale NLG corpus, MVPCorpus, from 77 datasets over 11 diverse NLG tasks. After converting various NLG tasks into unified text-to-text format, we propose multi-task supervised pre-training to learn an effective and general model MVP with task-specific prompts for NLG tasks. Extensive experiments have demonstrated that: (1) supervised pre-training is beneficial for NLG tasks as a general solution. Our MVP model outperforms the unsupervised pre-trained model BART and even achieves SOTA performance on 13 out of 17 datasets; (2) supervised pre-trained models have strong generalization ability on unseen generation or even understanding tasks.
In future work, we will explore the multilingual version of our MVP model by covering more datasets in other languages. Such a model is expected to capture Ianguage-indeennde and improve the generation tasks in the minority language. Besides, it is interesting to study how different tasks relate to each other in the unified semantic space the MVP model, which can inspire methods that incorporate task relations prior.
task characteristics

---

Under review as a conference paper at ICLR 2023
BROADER IMPACTS
In this paper, we pre-trained a language model MVP using labeled NLG datasets. According to the research (Bender et al. 2021 BBommasani et al. 202T). ) PLMs tend to "remember" what they have "seen" in pre-training corpus. This could result in the reproduction of undesirable biases from pre-training data on downstream tasks. Training data intervention could solution to alleviate this issue (Lu et al., 2020). It is also interesting to investigate whether supervised pre-training produces fewer biases than unsupervised pre-trainn training in the future.
Environmental impact is another factor should consider. We have attempted more efficient pre-training strategy and released our PLM for future work. In contrast to large PLMs with tens of billions of parameters, such as T5 (Raffel al. 2020) and GPT-3 (Brown et al. 2020), pre-train only a small model with hundreds of millions of parameters. In addition, we utilize supervised pre- training data and initialize our model with pre-trained BART, both of which improve the convergence of our model. Ultimately, our model is pre-trained for about 20, 000 steps, whereas BART of the same size is pre-trained for 500, 000 steps.
REPRODUCIBILITY
For reproducing and reusing our work, we have released the collection MVPCorpus, the models (e.g., MVP, task-specific prompts and multi-task variants), intermediate results (e.g., the generated texts), and source codes for pre- pre-training and fine-tuning at the link: https: https::/anonymous anonymooonoosouomoa 4open| ccience/r/ICLA 2022--2002--2 P223-Pape33318/ The detailed settings of experiments listed in Appendix C We hope that these open-source resources will facilitate future work on supervised pre-training and contribute to the advancement of NLG research.
REFERENCES
URL.
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. Knowledge graph based synthetic corpus generation for knowledge-enhanee
language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3554-3566 4365,, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2011 maacl-main.271 URL htthhtts:://// htt.lhtaaa/accaaattllooo org/
2021. nww.aaaalllmm..mmm - mwin. 278
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5799-5811, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 202 emm ww-.mmmwmmmmmmmmmmmmmmmmmmmmmmmm 4688
http////02...mmmmmiii.n..... URL /aattss://acllis antho /o org/
Huda Alamri, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das, Jue Wang, Irfan Essa, Dhruy Batra, Devi Parikh, Anoop Cherian, Tim K Marks, et al. Audio visual scene-aware dialog (avsd) challenge at dstc7. arXiv preprint arXiv:1806 00525, 2018. URL httt:tt///aaxixorrr
1806. 00525
Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, Benoît Sagot, and Lucia Specia. ASSET: dataset for tuning and evaluation sentence simplification models with multiple rewriting transformations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4668-4679, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.188533/1l 1atpl////220aaee 1/wa2/.acll-miii.222...2 URL https: / //aaaaaanti wanthooohhyyy..... org a www..ainnn....... 424.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni. Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In International
10

---

Under review as a conference paper at ICLR 2023
Conference Learning Representations, 2022. URL http:tpss//ppenrreeeeeeee//////p id=Vzh1BFUCiTX
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72,
Ann Arbor, Michigan, June 2005, Association for Computational Linguisties. URL https
1/aclanthology -lw.ooooy..////--000000
Hangbo Bao, Li Dong, Wenhui Wang, Nan Yang, and Furu Wei. s2s-ft: Fine-tuning pretrained transformer encoders for sequence-----eqeeeeee learning. arXiv preprint arXiv:21 10.13400, 2021.
13640
Emily M. Bender, Timnit Gebru, Angelina McMillan-Majon and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAceT '21, pp. 610-623, New York, NY, USA, 2021. Association for Computing Machinery. doi: 10.1 N0.1145/3442180 (/55/3442188..33 3445922. 15843444228833445522 URL pettps:/ddo..rgg/ https: doi http://i..ogggg///// 1040/ 3442184 3445922
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In In Proc Text Analysis Conference (TAC'09, 2009.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533- 1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL
logy. www.o//.//1111111111111
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the op- portunities and risks of foundation models. arXiv preprint arXiv:2108.0722 2021. URL
http: org/ absp////b////ee 0t..b///1088025511
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- cc cttp:c//pppppaeece wwtpp//pppppp//.......
ciates, Inc., 2020. URL https: htt.t::pp///.ooeeiiiii..ooo
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašié. MultiWOZ. large-scale multi-domai Wizard-of-OZ dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5016-5026, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/ 1547.
URL https
Pawel Bujnowski, Kseniia Ryzhova, Hyungtak Choi, Katarzyna Witkowska, Jaroslaw Piersa, Ty- moteusz Krumholc, and Katarzyna Beksa. An empirical study on multi-task learning for text style transfer and paraphrase generation. In Proceedings of the 28th International Conference on Computational Linguistics: Industry Track, pp. 50 63, Online, December 2020. Interna- tional Committee on Computational Linguistics. doi: 10.188633/112000.oolii -onioiiuttnn. URL industry.6.
settps://aclanttien http://ch.aatttooooooooo//////o/
wwwpoogggg22......oo col ing nddustr.
Bill Byrne Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu- Young Kim, and Andy Cedilnik. Taskmaster- 1: Toward realistic and diverse dialog dataset. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.--ICCNIP)) pp. 4516-4525. Hong Kong, China, November
11

---

Under review as a conference paper at ICLR 2023
2019. Association for Computational Linguistics. doi: 10.188653/11D119-4445 URL https
oz
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio. and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1 11th International Workshop on Semantic Evaluation (SemEval-2011)) pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 1/018653/1/1177-0001 URL btttps://allanthoooog..oo
Mingda Chen, Sam Wiseman, and Kevin Gimpel. WikiTableT: A large-scale data-to-text dataset for generating Wikipedia article sections. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 193-209, Online, August 2021. Association for Computational Linguistics. doi: 10. 100.18653/v1/// 1/t.3/////221..2/ findiigss 2021. windingsnnnlll....
acl.17. fiidd....niignal...... URL https: /ttpas://ccaaaaa wwtaaaaaannttoooooo /rg/
Wei Chen, Yeyun Gong, Song Wang, Bolun Yao, Weizhen Qi. Zhongyu Wei, Xiaowu Hu, Bartuer Zhou, Yi Mao, Weizhu Chen, Biao Cheng, and Nan Duan. DialogVED: A pre-trained latent variable encoder-decoder model for dialog response generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1852-4866 Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https::/aclanthool
org wawlcclloong 333
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. Logical natural language generation from open-domain tables. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 7929-7942, Online. July 2020a. Association for Computational Linguistics. doi: 2..18633/1/22000 www.://.....mmmi...... main.708. URL /ettps:/accaattiin
org
2020. 708.
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. KGPT: Knowledge-grounded pre- training for data-to-text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 0635-8648. Online, November 2020b.
Association for Computational Linguistics. doi: 10. 10.18863/11/220..00 www.0020e.mmmmmmmeemm mmnlp-mmin.6997 URL
https::/aclanthol.oo org/2020.0222.. 2020. emnlp-ma 697
Liying Cheng, Dekun Wu, Lidong Bing, Yan Zhang. Zhanming Jie, Wei Lu, and Luo Si. ENT-
DESC: Entity description generation by exploring knowledge graph. In Proceedings of the 2020
Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1187-1197, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020
emolp-main.90 90. URL https:/ / ac www.ancctllloooooooooooooocccooe org 2020 emn ma main.. 90
Eunsol Choi, He He, Mohit lyyer. Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke
Zettlemoyer. QuAC: Question answering context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2174-2184 Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v1/Dis 1241. URL
https://acl antho org/D1r 1241
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In Machine Learning. Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008, volume 307 of ACM International Conference Proceeding Series, pp. 160-167 ACM, 2008. doi: 10.1 10.1145/139015 56. 1390177. URL httph:tt/ wrr.//rgg/.. /00. 11 /111/11911111 1390177
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Ob- ject Classification, and Recognising Tectual Entailment, pp. 177-190, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Sociery Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), pp. 248-255, Los Alamitos, CA, USA, jun 2009. IEEE Computer Society. doi: 10. 10.11009/PP..20000050 5206848 URL fattps://dd/ fieeccomptterscieeyyyooo 111.:....111111111 2009. 5206848
12

---

Under review as a conference paper at ICLR 2023
Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In Proceedings of the 2021 Conference
Empirical Methods in Natural Language Processing, pp. 7580-7605. Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 2t055///122222..mmpmmmmaiaa 2021. emnlp main. maaan.5999 599
//cl http:////a/aaaaoaoolooooooo org/
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: tre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186 Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18655//1/119-14223 URL https
//aclanthology. Org/N19- -4233423
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: showledge-poweedd conversational agents. In International Conference on Learning Representations, 2019. URLatttpss//ppeeeevvee wet http://wwwww..aeeeoeo.oooooaoaaa
Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander H. Miller, Arthur Szlam, and Jason Weston. Evaluating prerequisite qualities for learning end-to-end dialog systems. In 4th International Conference on Learning Representations, ICLR 2016, 2016. URL
httpxxiv.rrrg///oooeeooooo.oo..t abs httt:://axxtt v. http://w...///b//111.11... .0613111.3111
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (/WP2005), 2005. URL
htttps://aclantho..o
org/ 105-502
Li Dong, Nan Yang. Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wius Hon. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL http:::ttt//ppt httpppss//prcceeigggss. newrei.isnnnnnn.nnn.p.......... http:/pss..cppppe///////// (1G062205S0bcc111113333eeeeeeee3
Paper. pdf
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL URL htttps:/ net ftr.:/orumm?m. wtw.uummid-iioooo wid.i11bddYYiTTT1
Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. Frames: corpus for adding memory to goal-oriented dialogue systems. In Proceedings of the 18th Anmual SIGdial Meeting on Discourse and Dialogue, pp. 207-219, Saarbrücken, Germany, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/117755222 URL
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. Key-value re- trieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 37-49, Saarbrücken, Germany, August 2017. Association for Computational Linguisties. doi: 10.18653w1//117-5506. URL
orq/ orq/w17-5506
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P188 1082. htt petttps://aclanthee hwtp://aaaaaanooy...oo.o org/P18 1082
Yutong Feng, Jianwen Jiang, Mingqian Tang, Rong Jin, and Yue Gao. Rethinking supervised pre-training for hetter downstream transferring. In International Conference on Learning Repre- sentations, 2022. http:/ttts///peeeeeette
new. forum? -idiiddjjsss JJev9MTqhcc
Cristina Garbacea and Qiaozhu Mei. Neural language generation: Formulation, methods, and evaluation. arXiv preprint arXiv:2007.15788, 2020. URL http:://awxwv..o//a///////////// crq/abs/2007
15780
13

---

Under review as a conference paper at ICLR 2023
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachine Creating training corpora for NLG miero-planners. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 179-188, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1.11866/11/117-10177 URL
/app:://aclanttho www.tthhooo......oog.... logy. 00g.00g//11111117
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, An- uoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušck. Chris Chinenye Emezue. Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank San- thanam, João Sedoc. Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pp. 96-120, Online, August 2021. Association for Computational Linguistics. doi: 1.10. /RLppttps///aeeettbooooyyy
gem 1.10
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 1-9, Prague. June 2007. Association for Computational Linguistics. URL
https::/aclaathhll www.aaaccaantttooooooooooooooooe NO0 -001
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human- annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 70-79, Hong Kong. China, November 2019, Association for Computational Linguistics. doi: 10. 10..8665////111991
URLAtttss//accaatho..o
Drg/D19-5409
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinglang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tite Topical-chat: Towards knowledge- grounded open-domain conversations. In Interspeech 2019, 20th Anmual Conference of the International Speech Communication Association, pp. 1891 1895- ISCA, 2019. doi: 10.214377/nterspeechh 2019-3079.
knowledge-grounded
10. 21437/ /ttpe///...c..ooo....emee/eeeeee 2019 90099-300 3079
David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword. Linguistic Data Consortium, Philadelphia, 4(1):3,, 2003.
Max Grusky, Mor Naaman, and Yoav Artzi. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long Papers), pp. 708-719, New Orlcans, Louisiana, June 2018. Association for Computational org/ 1065
Linguistics. doi: 10.18653/v 1/N11///////////1 1065.
Jing Gu, Mostafa Mirshekari, Zhou Yu, and Aaron Sisto. ChainCQG: Flow- conversational question generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2061-2070, Online, April 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 eacl-main.117 URL https!
//aclanthology wwt.or//////.
177
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. PPT: Pre-trained prompt tuning for few- shot learning. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8410-8423, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https: /ttp///aaatttttt cenethooogy...oo logy. .rayy.org/aae 2022 acl long. 576
Jian Guan, Xiaoxi Mao. Changjie Fan, Zitao Liu, Wenbiao Ding, and Minlie Huang. Long text generation by modeling sentence-leved and discourse-level coherence. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1thh International Joint
14

---

Under review as a conference paper at ICLR 2023
Online, August 2021. Association for Computational Linguistics. doi: 10. watp5//w/22221.allooooo.... URL httpts:://aaaaattlllnnne wrtp:/oooooo..g////////
Conference on Natural Language Processing (Volume 1: Long Papers), pp.
499
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-8360, Online, July 2020, Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 acl- acl.
acl-main. 740
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, 2006.
Han He and Jinho D. Choi. The stem cell hypothesis: Dilemma behind multi-task learning with transformer encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5555-5577 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. (0.18633/11/2011 www.l-mmin..e.ee. URL https: ac: http:p/////alaaaaaolooo..o.o/o// hwwphhholoo......./o2eo www.2201111mmmem.mm emn
451
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, Los Alamitos, CA, USA, jun 2016. IEBE Computer Society, doi: 10.1109/CVPR.2016. URL http::///da...eccmmttteeeeoooeop
1tt.:0..1111vvvv 2016. 90
Wanwei He, Yinpei Dai, Yinhe Zheng. Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang, Min Yang, Fei Huang, Luo Si, Jian Sun, and Yongbin Li. Galaxy: A generative pre-trained model for task- oriented dialog with supervise learning and explicit policy injection. Proceedings of the AAAI Conference on Artificial Intelligence, 36 36(10): 2022. doi: 10. 16.1/a9/ v36. 10.21320. URL httph:tt///offtt aaai. www//rrggneeeeee http://nnw...p//aaaaaaaaaaaaeeee
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and compre- hend. In Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL ptttp:://proceeeiingg www.:ooeeeeiinn....iiii.....pppp neurips cc
pdf
Xinyu Hua and Lu Wang. PAIR: Planning and iterative refinement in pre-trained transformers for long text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing (EMNLPP), pp. 781-793, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653/861112// 122200/112200..mmmmmaman.. URL https:tp//// http::://///aaaaaooooo.oooo 2020. www.pmmlppmmme. maapp-mman.... .7
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. Neural CRF model for sentence alignment in text simplification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7943-7960, Online, July 2020. Asso- ciation for Computational Linguistics. doi: 16653/v1/2022.acc./ 7/acl aachannhoolooooooo wow.ooo.o..oooo............o.... wwwpoogg2222200 acl-mate in.iin.... 709
URL https
Zhijing Jin, Qipeng Guo, Xipeng Qiu, and Zheng Zhang. GenWiki: A dataset of 1.3 million content- sharing text and graphs for unsupervised graph-to-text generation. Proceedings of the 28th Computational Linguistics, pp. 2398-2409, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.1 (0.18633/v1/0200 coling- mainngmmii..111 217. URL ht htttt::s/aclaantollogooo org/:/20rrrg/00 wttp:///22....coooooooooo main.217
International Conference
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601- 1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 18653/ 11/P17-1147. http:::s//aclaattaa www.aattoooooy......oooo hww.hhooolyooooooooooooo http:///oaaaaollllooooo/////1111
15

---

Under review as a conference paper at ICLR 2023
Pei Ke, Haozhe Ji. Yu Ran, Xin Cui, Liwei Wang. Linfeng Song, Xiaoyan Zhu, and Minlie Huang.
JointGT: Graph-text joint representation learning for text generation from knowledge graphs. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2526- 2538, Online, August 2021. Association for Computational Linguistics. doi: 10. 12.18633/1122211 findings-acl.222 223. /tt.httts:://aclmmchool www.hhlllloy.org/////
ndi findings-aclll 223
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- nanch Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1896-1907, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. /188//613/122200 finding-emmllp.... org/ 2020. Findingg emnlp 171
URL
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The Aa reading comprchension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018. doi: 10.1162/tacl a 00023. URL
https: hettps://aclantoon wacchologg .og.. ortp://-1/11112222
Rik
Dhanush Bekal, Yi Luan, Mirella Lapata, and Hamnaneh Hajishirzi. Text Generation from Knowledge Graphs with Graph Transformers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2284-2292 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19. 1238.
URL https: /attps://aclsse http///aaaaaahccccao htw.hhoooooooorg//1111 1238
Mahnaz Koupaee and William Yang Wang. Wikihow: A large scale text summarization dataset. .03005
arXiv preprint arXiv: 1800.0905 2018. URL http://axvvvo.oraaaaaaaaa
Ashutosh Kumar, Kabir Ahuja, Raghuram Vadapalli, and Partha Talukdar. Syntax-guided controlled generation of paraphrases. Transactions of the Association for Computational Linguistics, 8: 329-345, 2020. doi: 10.1162/acl.a.00315 URL BtLhttts::/ http://aaaaaoool..ogg/////// tacl-1.22
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wey Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, http//acaaaa http://aaaanttoloy..ooooo wtt.oorr//11120010
2019. doi: 10.1 a a.00276. URL
Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. WikiLingua: new benchmark dataset for cross-lingual abstractive summarization. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2020, pp. 4034-4048, Online, November 2020. findings-emnlp3660.. URL
Association for Computational Linguistics. doi: 10. 10.18653/v1/2001 122553//11220.... 10.16653w/2000.mmmmmmmmmmmm
oro. 2020. findings-emas emm. 360
Huiyuan Lai, Antonio Toral, and Malvina Nissim. Thank you BART! rewarding pre-trained models improves formality style transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 2: Short Papers). pp. 484 494, Online, August 2021. Association for Computational Linguistics. doi: 10. (0.18653/v1/2022 acl-short.62. URL org/ httpp::/7cch antho ooy. wrg. 2021 -shorshort....... 62
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL hoLLittttes//prreeiiigsgco
pdf
Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. for Computational Linguistics. doi: 10.18653/v 1/D16- 1128. URL https: :/tps:://ccciiee ////aaaanttt //t///aaaaanloooyyo/ooo
Austin, Texas, November 2016. Association
Brg/DD6-1122
16

---

Under review as a conference paper at ICLR 2023
Sungjin Lee, Hannes Schulz, Adam Atkinson, Jianfeng Gao, Kaheer Suleman, Layla El Asri, Mahmoud Adada, Minlie Huang, Shikhar Sharma, Wendy Tay, and Xiujun Li. Multi- domain task-completion dialog challenge. In Dialog System Technology Challenges. volume 8, 2019. URL http:::tttt:/sssteetttttst mul multidomaa in wwtttass_popppc .ddf
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 1/266///1/222lmmmllppp 2433. URL https //aclanthologe org/2021 emn 243
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10. 20018863/11/2000.aa. mainn,703. URL
https://acl antho ogy www. 20r0g//020.ccco
ma -ma in. 703
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-proomtting
objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110-119, San Diego, California, June 2016. Association for Computational 1014
Linguistics. doi: 10.18653/w1/N16- 1014. URL httphttp//// wwtp:///occaattoooyoyoooooo
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. A survey of pretrained language models based text generation. arXiv preprint 3xiv:2200.00277, 2022. (thhtt
//arxiv. ora/ abs 2201222112222 05273
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume 1:: Long Papers), pp. 4582-4597, Online, August 2021, Association for Computational Linguistics. doi: 10.18653/ v1/2021 aacol.2..--- URL / wtt.:/oooooo.o.//////........... acll-lnng. 1on..3331333333
Xiujun Li, Yu Wang, Siqi Sun, Sarah Panda, Jingjing Liu, and Jianfeng Gao. Microsoft dialogue chal- 112.1
lenge: Building end-to-end task-completion dialogue systems. arXiv preprint
2018. URL htt.:::://aaai...r////aaa//////e
1807 11125
Yanran Li, Hui Su. Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog A manually labelled multi-turn dialogue dataset. In Proceedings the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 986-995, Taipei, Taiwan, November 2017, Asian Federation of Natural Language Processing. URL https: htttps://aclanttttt org/T17-1099
Percy Liang, Michael Jordan, and Dan Klein. Learning semantic correspondences with less su- pervision. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACI. and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 91-99, Suntec, Singapore, August 2009. Association for Computational Linguistics. URL ftttps//acllanthoooryoooog org/p09 1011
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative com- monsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1823 1840, Online, November 2020a. Association for Computational Linguistics. doi: 10.18653/ 1/2020.22022m http:/222220iinnnnnnneoee findings-emmlp.165.. URL
Sindings-ema 1p. 1.p...1.. 165
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
URL htkl:sttt:://ee antho ogy. www.oogg//000000
-
17

---

Under review as a conference paper at ICLR 2023
Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, and Lei Li. Pre-training multilingual neural machine translation by leveraging alignment information. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 2649- 2663, Online, November 2020b. Association for Computational I Linguistics. doi: 10.18653/v1/ 0020.emnlp-main.211
htRt:ttts:////t anthooogyy.....o. Seantholog..org/20000 org. emn p-ma a i 2100
Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. MinTL: Minimalist transfer learning for task-oriented dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3391-3405, Online, November 2020c. Association for Computational Linguistics. doi: 10. /10////////200///////o/oo
URL
htapp:::/alaantclooo...ooo org. att.://......22222..mo.mmmmaaa..
emnlp-main. 273
Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. OpenSubrittes20188 8: Statistical rescoring of sentence alignments large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL httpt::/
1275
Dayiheng Liu, Yu Yan, Yeyun Gong. Weizhen Qi. Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Ruofci Zhang, Winnie Wu, Ming Zhou, and Nan Duan. GLGE: A new general language generation evaluation benchmark. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 408-420, Online, August 2021a. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.findings-acl.66 URL BtLhhtts::// http:///olcaaaaooooooo.ooooo
http://ooo./ttttp//////aaaaoooo/ wttp:yyy.rg//221.1.11 http:/oooooooor//222...o.aannnnn findings acl. acl.36 36
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint ar arXiv:2107. 13586, 2021b. httt:://arrii..org/ab////// 13586
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https: http:///caaatthloogo... wwwp:/y.........o.. aww.::2..cccl300
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4487-4496, Florence, Italy, July 2019a. Association for Com- putational Linguistics. doi: 10.18653/w1/P19- 1441. URL https: https////aaaaatolgagooe /rg/ P19-1441
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /692, 2019b. URL http: /ttp///axvvvv////////// org abs wwwp//ggg//g/gooo/e 10683
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguisticcs, 8:726-742, 2020. doi: 10.1162/
tacl a 00343. uutp:httess//ellaaahoooooo.oro//
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https:://pperee// http://wwwwww.....e.eeee.eeeee.e forum? id=
Bkg6RicqY?
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender Bias in Neural Natural Language Processing, pp. 189-202. Springer International Publishing, Cham, 2020. doi:
14. URL
ort./0..007..
18

---

Under review as a conference paper at ICLR 2023
Bryan McCann, James Bradbury. Caiming Xiong, and Richard Socher. Learned in translation: Con- textualized word vectors. In Advances in Neural Information Processing Systems, volume 30. Cur- Associates, Inc., 2017. URL http:: http:///cc..eeee......o...
newrips....ppppppp CC ct/paper/201..
fi
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning question answering. arXiv preprint arXiv:1806 08730, 2018.
URL http:: org/ http://aas/11...... 08730
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task gener- alization via natural language crowdsourcing instructions. In Proceedings of the 60th An- mual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 3470-3487, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL /tttps://aclanto/// wrw. 20222 acl- 244
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. OpenDialKG: Explainable con- versational reasoning with attention brestion-based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 845-854, Florence. Italy, July 2019. Association for Computational Linguistics. doi: 10. 1..11663/11/111-1000 URL /ttta:///eclathoooooooooor////--
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper under- standing of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839 849, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10. 10.18653/v 1/116///////11111 11166-1999 1098. URL wttpt:ttp//..m...... //acl antho I .eg.yorg//e 6- 1098 1098
Nikola Mrk3ić, Diarmuid Ó Séaghdha, Tsung- Hsien Wen, Blaise Thomson, and Steve Young. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1777-1788, Vancouver, Canada, July 2017, Association for Computational Linguistics. doi: 10. 10.18653//1/111-1163. URL
https: ac
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsich, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. DART: Open-domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 432-447, Online, June 2021. Association for Computational Linguistics. doi: 11186633/1/222/ www.al.....aa.l-mm......... maacl-main.31) URL httpp:://aclaatiollgyy org/ 2021 www.c-mminnn.....
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October- 2018. Association for Computational Linguistics. doi: 10.18653/v1/ D18- URL http::::///llaattoooooooo / -
Thong Nguyen, Anh Tuan Luu, Truc Lu, and Tho Quan. Enriching and controlling global semantics for text summarization. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 9443-9454 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 1/2.82 wem.ll-mmnll..n.... 744. URL org/ 2021 www.:2ee.mmmmmm.mmmmmmm ma mainn. 744
https://aclantheet
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and
Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@NIPS, volume 1773 of CEUR Workshop Proceedings. CEUR BEUR-WS.orr .Soog, 2016. URL
htttp://ceurwws www.:/..rrg/eo.. wttp://wwww.///o1-//000 11773/////CNIS_222. SaciiISS_216161 2atppp____2oo_ppp_r___ee
19

---

Under review as a conference paper at ICLR 2023
Jekaterina Novikova, Ondřej Dušck, and Verena Rieser. The E2E dataset: New challenges for end-to- end generation. In Proceedings of the 18th Anmual SIGdial Meeting on Discourse and Dialogue, pp. 201-206, Saarbrücken, Germany, August 2017. Association for Computational Linguistics. doi: N0.18633/11/117-555 URL htttts:s///cacachlllllloooo l
5555
Kishore Papineni, Salim Roukos, 'odd Ward, and Wei-Jing Zhu. Bleu: method for auto- matic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073333.1033333 URL
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. In
Advances in Neural Information Processing Systems, volume 34, pp. 11054-11070 Curran As- sociates, Inc., 2021. URL
wtw.:eeeeediinn....nni...e. nwwrips httpr//is.cpppppee//////////////
ppf
Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts.
In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5203-5212, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v L Laaacl-mai..440 URL https
//aclanthology.oo www.oyy.orgg///. 2021 naacl wea-mmai.....
Alec Radford, Jeffrey Wu. Rewon Child, David Luan, Dario Amodei, llya Sutskever, et Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
2019. URL http:t:ttt::/....ottmt openai www.wttotbbetooo..e..eeootoo...e languageggggggaae model s/ s/language. models 1s_ ae__aree_ussuee wwwrre_ssuperriied_mm.. satreedddulttttttttt learners..............
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http:// www.pppprs..aaooooooooooop waarrr/vv11//// htm.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v 1/D16- 1264. URL /LLpsttss///acaattooooo.
org/Di6-13 1100061220 1264
Sudha Rao and Joel Tetreault. Dear sir or madam, may introduce the GYAFC dataset: Cor- pus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Papers), pp. 129-140, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18- 1012. URL /Etts::/aclantholoogy
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5370-5381, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 1..18663//1/P11-155 URL otg/P19......9.91. 1534
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. volume 34, www00/90.a...4.......
pp. 8689-8696, Apr. 2020a. doi: 16../69/ .34.0056639444. URL https::///// //ts:///jss..// aaai. orgg
Lndex.pppp http:p//wAAAaaaaccclee.e////////
6394
Abhinay Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards
scalable multi-domain conversational agents: The schema-guided dialogue dataset. volume 34, pp. 8189-6886688 Apr. 2020b. doi: 10. www.1660/a....4444 v34105.6394 URL sattps://of) aaai. org/
[index
Sindex.php/AAAT//e httph//AAATA
ew/639 6394
20

---

Under review as a conference paper at ICLR 2023
Mathieu Ravaut, Shafiq Joty, and Nancy Chen. SummaReranker: A multi-task mixture-of- experts re-ranking framework for abstractive summarization. In Proceedings of the 60th An- mual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 3504-4524 Dublin, Ireland, May 2022. Association for Computational Linguisties. URL
https::/acclanee www.anthhoooy.....ooom wgw.ooog......oo.... acl.lonn.lllonn.... 3009
Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: conversational question answering
challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019. doi: 10.1 16.2tacl 00266. Utt.ttts:://acl anthel wwwphellllooooooooooooooe org//rgggggooooo Q10-01999000 9-10
Pedro Rodriguez, Paul Crook, Seungwhan Moon, and Zhiguang Wang. Information seeking in the
spirit of learning: A dataset for conversational curiosity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8153-8172, Online, November 2020. Association for Computational Linguisties. doi: 10. 10211166/1122000.0.0 www.:2000.mmlmmmmmmmmm mat.llppmmii.65... URL
http:///loclaatoooo.....oooo http://..rrr/2222..meee-----aa inniiii...n 655
Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sen- tence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 379-389, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/Dii 1044.
Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. A survey of evaluation metrics used for nlg systems. ACM Comput. Surv. 55(2), jan 2022. ISSN 0360-0300. doi: 10.1445/348
URL http:: /tt:::/doooiooo wttp:/o.......115/..////// 34857766666
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry. Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL www.https:s./ htttt:::///perrvviiii
net http://oorummmdd--1edeee
Maarten Sap. Eric Horvitz, Yejin Choi, Noah A. Smith, and James Pennebaker. Recollection versus imagination: Exploring human memory and cognition via neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1970-1978, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/1/2022..cc 11120..cl-mainn 178. URL https:ss:: a/acl anthologyy...... org wtw.g///00000 2220 acl mainn. 178
Abigail See, Peter J. Liu, and Christopher D. Manning. Get the point: Summarization with pointer-generater networks. In Proceedings of the 55th Anmual Meeting of the Association for Computationnll Linguistics (Volume Long Papers). pp. 1073-1083, Vancouver, Canada, July 2017, Association for Computational Linguistics. doi: 10.18653/v1/P11 1/P77-1099
URL https
//aclanthe wwwthhooooy..ooo wwwpyorgg/PP1100000
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality
sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
URL ht..pt:::/ /tt:tttss::/aa //ac antto ogy. 3- 1170
Karl Stratos. Mutual information maximization for simple and accurate pfrt-of-speech induction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Humun Language Technologies, Volume (Long and Short Papers), pp. 1095-1104, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1/N1911131111111.. URL https: https::///aaaaathhaeae 0rg.N19-1g///11111111111
Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang, and Nigel Collier. Plan-then then-generate Con- trolled data-to-text generation via planning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 895-909, Punta Cana, Dominican Republic, November 2021.
21

---

Under review as a conference paper at ICLR 2023
Association for Computational Linguistics. doi: 10.18653//1/2221..nnmll www.::/i-iiininnnnnnn...... URL htw.aaaanttoooooo.or//2.......
Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. Multi- task pre-training for plug-and-play task-oriented dialogue system. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4661-4676, Dublin, Ireland, May 2022. Association for Computational Linguisties. URL
acl long. 319
Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. Learning general purpose distributed sentence representations via large scale multi-task learning. In International Conference Learning Representations, 2018. URL httpt:ttt:::/eeeettttte www.r//owwweteeeeenttt... forum?
1d=B1BWgG-Ba
Jiao Sun, Xuezhe Ma, and Nanyun Peng. AESOP: Paraphrase generation with adaptive syntactic control. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5176-5189, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10.18653/v 1/202/.emmlp-mmmmeeeeee 1aippp-mai..220.. URL https?
//aclanthologh org/ 2021. emnip---- melommain.... 420
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. DREAM: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217-231, 2019, doi: 10.1 1662/taclaa00026666 URL https
coolanthology.org/O1..
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826, Los Alamitos, CA, USA, jun 2016. IEEE Computer URL https://dol: rtw.eeppeeeeeooooeeoooo.......e
Society. doi: 10. 100.100//CP..2216....2 10g/10.1109///NT 2t1.///............ 2016.308
Xiangru Tang, Arjun Nair, Borui Wang. Bingyao Wang. Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, and Dragomir Radev. CONFIT: Toward faithful dialogue summarization with lingustticall--infrmeee contrastive fine-tuning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, pp. 5657-5668, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10188866111220 1ttp////2222aaae// wat.222nacllmmii....
URL
ptttp:://aclanthology org 2022. www.nanclllaaamm
415
Adam Trischler, Tong Wang. Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman,
and Kaheer Suleman. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 191-200, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 10.18853/11/11-22000 URL https
/acclanchology. org/w17-2623
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL htttps://proceeddnggs neurips.eer...... wtwpuripss.cppapee////... cc/ppaappt///11//iile Paper. Papeeeaaaappperpp.p. pdf
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4576-4575, Los Alamitos, CA, USA, jun 2015. IEEE Computer Society. doi: URL https: /dol http:////oo...eeeccoppprrrrooooo
10.11099CVVP..0055.22900 1109/CVPR. 72990077
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5039- 5059, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https
7/aclantho ogy. ogg. 2022. acl 1ong
22

---

Under review as a conference paper at ICLR 2023
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In International Conference Learning Representations, 2019. URL http::///peneeeiier
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019. doi: 10.1162/
tacl a 00290. URL https: http::///ccaaaaalooloooo////////
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe Le. Finetuned language models are zero-shot learners. In International Conference Learning Representations, 2022.
http::::s///peeeeveeeee net forum?
id=gE2r3CozddR
Anuradha Welivita, Yubo Xie, and Pearl Pu. A large-scale dataset for empathetic response generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 1251-1264, Online and Punta Cana, Dominican Republic, November LinguutttttiianLLniiiisiiii
2021. Assoociaionn for Computational Linguistics. doi: 1118653/v12221..crr 112222.eemmlmmmmee... main. 96. URL
https://aclanthoo. http://rrg///// 2021. emnlp momamnppmaaa ma www.amni......... 96
Tsing-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašié, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chupter of the Association for Computational Linguistics: Volume 1. Long Papers, pp. 4384444888 449, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https: http///aaaannntoooo.o..... org/
E17-1042
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N11 1101. URL https! 7 /aclanthology. org N111111
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. 1/266///1/020..mmmp--2o6 https::/ aclatt/oolla org/ 2020. www.:.0eemm0mmmmmemmmm.m0 cennppddmoos.
URL
Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-gen gent An enhanced multi-flow pre-training and fine-tuning framework for natural language generation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI- 20, pp. 3997-4003 International Joint Conferences on Artificial Intelligence Organization, 7 2020. doi: 10. 10.24963/jjcai.22000 Main track.
URL bttpp:://ooii http:///......1...9993/6/ httpj/aa......533...3. 2020/553
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida Wang, et al. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint arXiv:2201.05966, 2022. URL BtLltttp:/aaxxivoort aww......iii.............. http:/bb///222250/6..
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang. Haiyu L., and Zhilin Yang. Zeroprompt: Scaling prompt-based pretraining to 1,000 tasks improves zero generalization. arXiv preprint arXiv:2201.066 2022.
abs/ 2ttp://222...9900. 06910
Peng Xu, Davis Liang, Zhiheng Huang, and Bing Xiang. Attention guidedd generative models for extractive question answering. arXiv preprint arX0v:21 2021. URL tattp://arxx. orgg/bb// 2200. 06393
23

---

Under review as a conference paper at ICLR 2023
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Buree Optimiz- ing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics. 4:401-415, 2016. doi: 10.1 00107. |
//aclantho www.chhooooy...oo owgp/1o0 6-
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: dataset for diverse, explainable multi-hop question answer- ing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 2369-2380, Brussels, Belgium, October-November 2018. Association for Computational I Lin- guistics. doi: N0.18653/v1/DD/ 8-15998--9---
1259. URL httpt::tp//............p ac antho log.. 8- -
Qinyuam Ye. Bill Yuchen Lin, and Xiang Ren. CrossFit: A few-shot learning challenge for cross-task generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 7163-71888 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 00.18833/11/021111 wet.:22...mmlpmm..... https:
URL
oro.oog/////// 2021 emnlp main. 572
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: have dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 2204 2213, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18 1205. UtLhttss::/ http::/:s///aaaaahoooooooooooooo .rgg/orgg... 1205
Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. When do you need bil- lions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Associ- ation for Computational Linguistics and the 11th International Joins Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1112-1125, Online, August 2021. As- sociation for Computational Linguistics. doi: 10. 111665//112022.aclllooo //aclanthology wwt.:yy..rr..2.......o acl-long..9999
URL https
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. DIALOGPT Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics: System Demonstrations, pp. 270- 270-278, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ acl del-demo..30. URL https: //acl antho ogy. org 2020 wtt.:000..cllleee acw--lmos............... 30
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. A dataset for document grounded conver- sations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- cessing. pp. 708-713, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/w1/DA 1076. URL [rRLattpp./////ltttooooooo.////p
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. MediaSum: A large-scale media interview dataset for dialogue summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5927-5934, Online, June 2021. Association for Computational Linguistics. doi: 18.18653/v1/ naaclnacc-mmin.... URL htt.tt :/acl anth ogy otg//221 naacl -ma n, 474
24

---

Under review as a conference paper at ICLR 2023
A LIMITATIONS
Despite our efforts to collect as many generation tasks and datasets as possible, we only evaluate the generation quality and generalization ability of our models small number of tasks and datasets. The interpretability and robustness of our models require further analysis. Besides, there exists subjectivity when collecting intra-task datasets, albeit our attempts to employ widely-recogmizze categorizations from the literature. Due to limitation of computing power, we do not study the performance of our method at different model scales. The effectiveness of multi-task pre-training from scratch, similar to ExT5 (Aribandi et al. (022), also merits in-depth study. Regarding evaluation methods, we only consider basic automatic metrics such as BLEU (Papineni et al.) (002) and ROUGE (Lin 2004). However, there is still certain gap between these metrics and human judgments (Sai et al 2022).
B TASKS AND DATASETS
B. DESCRIPTION OF TASKS AND DATASETS
We provide the details of the tasks and datasets used in our paper for pre-training and fine-tuning in Tables and If the dataset for pre-trane pre-training does not have a valid set, we divide 10% of the training set for validation.
We list the licenses for all datasets if them have. All datasets are publicly available. The majority of them can be directly downloaded from GitHub or Google Drive. ROCStories (Mostafazadeh et 2016) and CommonGen (Lin et al. 2020a) can be obtained after filling out form. GYAFC (Rao & Tetreault 2018) is accessible after requesting Yahoo and the authors of the dataset.
The tasks and datasets we use in this paper are as follows:
Data-to-text generation aims to generate descriptive text about structured data, such as the knowledge graph and the table. We use the following datasets for pre-training:
1. AGENDA (Koziol-Kedzioskii et al. 2019)
2. ENT-DESC (Cheng et al. 2020);
3. GenWiki (Jin et al.
2020);
4. LogicNLG (Chen et al.
2020a):
5. TEKGEN (Agarwal al.
202
6. WEATHERGOV (Liang et
2009);
WikiTableT (Chen al. 2021p.
We utilize the following datasets for fine-tuning evaluation:
1. WebNLG (Gardent al. 2017), we utilize the version 2.1;
2. WikiBio (Lebret et al. 2016).
Open-ended dialogue system, also known as chatbots, focused on daily communication. We use the following datasets for pre-training:
1. Cleaned OpenSubtitles Dialogs (Cleaned OS Dialogs) (Welivita et al.
2021 which is
cleaned variant of OpenSubtitles Dialogs (Lison al. 2018):
2. CMU Document Grounded Conversations (CMUDog) Zhouu et al. 2018
3. Curiosity (Rodriguez et al. 2020);
4. DREAM (Sun et al.
2019)
5. Empathetic Dialogues RRshhin al. 2019
Movie Dialog (Dodge et al. 2016);
7. MuTual (Stratos 2019);
8. OpenDialKG (Moon et al. 2019);
9. Topical-Chat (Gopalakrishnan al. 2019);
10. Wizard of Wikipedia (Dinan et 2019)
We utilize the following datasets for fine-tuning evaluation:
25

---

Under review as a conference paper at ICLR 2023
1. DailyDialgg (Li et al.
2017);
2. DSTC7-AVSD (Alamri et al.
2018);
3. PersonaChat (Zang et al. 2018).
Paraphrase generation involves rewriting a sentence with the same semantic meaning but different syntactic lexical form. We utilize the following datasets for fine-tuning evaluation:
1. Quora (also known QQP-Pos) (Kumar et al. 2020), which is subset of Quora Question Pairs
Question answering requires the model to answer a question based on optional background information. Note that conduct this task in generative way in our paper. We the following datasets for pre-training:
1. HotpotQA (Yang et al.
2018);
2. MS MARCO Nguyen et al. 2016);
3. MSQG (Liu 2021a), since is designed for QG, we reverse the question and answer enrich QA examples;
4. NarrativeQA (Kočiský et al. 2018);
5. Natural Questions (Kwiatkwsski et al. 2019):
6. NewsQA (Trischler et al. 2017);
7. QuAC (Choi et al. 2018);
8. TriviaQA (Joshi et al. 2017;
WebQuestions (Berant al.
2013).
We utilize the following datasets for fine-tuning evaluation:
1. CoQA (Reddy al.) 2019;
SQuAD (Rajpurkar et al. 2016), we utilize the version 1.1.
Question generation generates a coherent question given passage and its corresponding answer. We use the following datasets for pre-training:
1. HotpotQA (Yang et al. 2018);
2. MS MARCO (gguyen et 2016):
3. MSQG (Liu et al. 2021a);
4. NarrativeQA (Kočiský et al.
2018)
5. NewsQA (Trischler et al. 2017)
QuAC (Choi et al. 2018);
Most of them are QA tasks, and we invert the question and answer to enrich QG examples.
We utilize the following datasets for fine-tuning evaluation:
1. CoQA (Reddy et al. 2019
2. SQuAD (Rajpurkar et al. 2016), we utilize the version 1.1.
Story generation creates long and informative text with a short title. We use the following datasets for pre-training:
ChangeMy View (Hua & Wang 2020
2. English Gigaword (Rush et al. 2015):
3. Hippocorpus (Sap et al. 2020);
4. WikiPlots (wik):
WritingPrompts (Fan et al. 2018), we split the original training set for pre-training and corresponding validation.
Considering English Gigaword is large summarization dataset, we use the summary the title to generate the passage turn to enrich the examples of story generation.
We utilize the following datasets for fine-tuning evaluation:
1. ROCStories (Mostafazadeh et al. 2016);
https:: httbpp:://ww.wwee httpsg/lle.coo////////////oo cucoraaauusiiionn
26

---

Under review as a conference paper at ICLR 2023
2. WritingPrompts (Fan et al. 2018). the sets created by Guan et al. (2021) (who split the original valid and test sets for training, validation, and testing) to tine-tune our model for fair comparison.
Task-oriented dialogue system meets real-life needs of users, such as restaurant reservations and airplane bookings. We use the datasets for pre-training, following Su et al. (2022):
1. CamRest676 (Wen et al.
2017);
2. Frames (El Asri et al. 2017;
3. KVRET Eric et al.
(017);
4. MetaLWOZ (Lee et al.
2019);
5. MSR-E2E (Li al. 2018):
MultiWOZ (Budzianowski et al.
2018 8
7. Schema-Guided (Rastogi et al. 2020a):
8. TaskMaster (Byrne et al.) 2019):
9. wOZ (Mrkšić et al. (2017).
We utilize the following datasets for fine-tuning evaluation:
1. MultiWOZ (Budzianowski et al. 2018). we utilize the version 2.0;
Text style transfer modifies the style (e.g., sentiment and formality) of given texts while retaining their eyle-inddepnndent content. We utilize the following datasets for fine-tuning evaluation:
1. GYAFC (Rao & Tetreault 2018), which has sub-domains "Entertainment and Music" (E&M) and "Family and Relationships" (F&R).
Text summarization condenses long document into a brief text while retaining the essential details. We use the following datasets for pre-training:
1. English Gigaword (Graff et al. 2003), use the variant provided by Rush et al. (0155):
2. MediaSum (Zhu et al.
2021)
MSNews (Liu et al.
2021a);
4. Newsroom (Grusky et al. 2018):
5. WikiHow (Koupaee & Wang 2018
We utilize the following datasets for fine-tuning evaluation:
1. CNN/DailyMail (Hermann et al. 2015), we use the variant provided by See et al. (2017);
2. SAMSum (Gliwa et al. 2019);
3. XSum (Narayan et al.
2018
To better compare with ExT5 Aribandi et al. 2022). we utilize the language generation benchmark
GEM (Gehrmann et al. 2021 for fine-tuning evaluation. GEM includes five tasks:
Commonsense generation:
1. CommonGen (CG) (Lin et al. 2020a).
Data-to-text generation:
1. DART (Nan et al.
2021
2. E2E NLG cleaned Novikkvaa et al. 2017):
3. ToTTo (Su al.
2021);
WebNLG (Gardent et 2017
Dialogue system:
1. Sehema-Guides Dialog (SGD) (Rastogi et al. 2020b
Text simplification:
1. WikiAuto Turk/ASSET (WiA-T/A) (Jiang et al.
2020 Xu et al. 2016 Alva- Ila--Machheg
al. 2020
Text summarization:
1. Wiki-LLnguaa (WLE) (Ladhak al. 2020
27

---

Under review as a conference paper at ICLR 2023
To test the generalization ability of our model, also utilize the natural language standing benchmark GLUE (Wang et al. 2019), which is composed of three tasks:
Natural language inference:
1. MNLI (Williams al.
2018);
2. QNLI (Rajpurkar et
2016
Wang al.
2019);
3. RTE Dagan et al. 2006
Haim et al. 2006
Giampiccolo et al. 2007 Bentivogli et al. 2009).
Paraphrase detection:
1. MRPC (Dolan & Brocket 2005
2. QQP
3. STS-B (Cer et al.
2017
Text classification:
1. CoLA Warstadt et al.
2019
SST-2 (Socher et
2013
B.2 DATA LEAKAGE
Since model is pre-trained large number of labeled datasets, it may have "seen" examples from fine-tuning test sets during pre- pre-training, which leads to an unfair comparison with other methods. Hence, eliminate the pre-training examples that share n-gram overlap with either of the test datasets. Following Brown et al. (2020). n is the 5th percentile example length in words, and the maximum value set to 13. Finally, have removed 17, 848 examples from the pre-training datasets. The number of "cleaned" examples for each dataset can be found in Table 7
28

---

Under review as a conference paper at ICLR 2023
Table 7: The statistics and licenses of datasets for pre-training our MVP model. The #Train, #Valid, and #Test denote the number of examples in the train, valid, and test sets, respectively. Cleaned # #Tran represents the number of training examples after filtering. Input and Output are the average number of words (split by space) in the input and output sequences, respectively. These setups and
abbreviations are the same below.
Dataset
#Train
Cleaned #Train
#Valid
#Test
Input
Output
License
AGENDA
38,720
38,720
1,000
1,000
52.1
141.2
N/A
ENT-DESC
88,652
88,652
11,081
11,081
279.9
31.0
N/A
GenWiki
681,436
681,436
75,716
1,000
21.4
29.5
MIT
LogieNLG
28,450
4,260
178.4
14.2
TEKGEN
6,3 6,310,061
6,307.995
788,746
796,982
17.0
21.2
CC 2.0
WEATHERGOV
25,000
25,000
1,000
3,528
148.7
30.6
N/A
WikiTableT
1,453,794
1.452,778
4,533
4,351
81.0
99.7
MIT
Cleaned Dialogs
13,355.487
13.355,366 368
1483,944
75.5
16.7
N/A
CMUDeG
82,818
82,818
5,555
14,510
433.0
12.2
N/A
Curiosity
64,930
64,551
8,539
8,495
144.4
20.2
CC BY-NC4400
DREAM
14,264
14,242
4,709
4,766
75.66
13.6
N/A
Empathetic Dialogues
64.636
64,636
9,308
8,426
$2.7
12,9
CC BY-NC4.0
Movie Dialog
762.751
762.711
8.216
8,066
126.9
44.0
N/A
MuTual
33.691
33.691
4,090
3,248
53.6
14.5
N/A
Ope
69,680
69,680
7,743
54.2
12.4
BY-NC.0
Topical- Thpical-Chat Chat
179,750
179,750
22,295
22,452
223.3
20.0
CDI.A-Shringg 1.0
Wizard of Wikipedia
148,357
147,702
15,767
15,564
297.0
16.7
MIT
HotpotQA
90.447
87,815
7,405
187.9
2.2
BC
MARCO
681,445
681,226
77,580
68.7
13.3
N/A
MSQG
198,058
198,029
11,008
48.1
3.7
CC BY-SA 4.0
NarrativeQAA
65,494
65,494
6,922
21.114
584.1
4,2
Apache 2.0
Natural Questions
96,676
96,676
10,693
6,490
9.0
2.1
CC BY-SA 3.0
NewsQA
97,850
97,700
5,486
5,396
MII'
QuAC
83,568
83,485
31,906
487.9
12.5
CC 4.0
TriviaQA
78.785
78,785
8,837
11,313
14.0
2.0
Apache 2.0
WebQuestions
8,933
8,933
4,863
4,863
6.7
CC
HotpotQA
90,440
6,972
19.8
BY-SA
MS MARCO
681,445
681,228
77,580
75.9
6.0
N/A
MSQG
198.058
198,029
11,008
11,022
45.9
6.0
CC BY-SA 4.0
NarrativeQA
65,494
6,922
21.114
579.7
Apache
NewsQA
97.850
97,700
5,486
5,396
724.2
MIT
QuAC
69,109
69,026
26,301
496.7
6.5
4.
ChangeMy View
42,462
42,459
6,480
7.562
17.9
104.1
MII'
English Gigaword
3,803.957
3,802,620
189,651
1,951
8.8
33.3
MIT
Hippocorpus
6,168
6,168
686
34.1
262.6
CDLA-Permisive
WikiPlots
101.642
101,641
11,294
3.4
338.5
N/A
WritingPrompts
272,600
272,518
15,620
15,138
28.4
630.8
MIT
CamRest676
4,872
4,872
616
55.3
9.4
N/A
Frames
26,631
26,631
2,106
13.0
MIT
KVRET
14.136
14.136
1,616
30.5
9.3
N/A
MctaLWOZ
176,073
176,073
17,912
45.6
8.0
N/A
MSR-E2E
103.362
103.362
5,235
51.3
12.8
Microsoft
Scheema-Guided
494,946
494,933
73,089
120.8
12.5
CC BY-SA 4.0
TuskMaster
249.664
249.662
20,680
95.6
12.0
CC 4,0
WOZ
6,364
6,359
1,260
47.0
10.6
N/A
English Gigaword
3,803.957
3,802,620
189.651
1,951
33.3
MIT
MediaSum
443,596
442,021
10,000
10,000
1641.0
14.4
N/A
MSNews
136,082
135,937
7,496
7,562
309.9
9.8
CC 4.0
Newsroom
995,041
989,351
108,837
108,862
642.4
26.7
N/A
WikiHow
157,252
157,247
5,599
CC
29

---

Under review as a conference paper at ICLR 2023
Table 8: The statistics and licenses of datasets for evaluating MVP model. The license of the MNLI dataset is composed of OANC, CC BY-SA 3.0, and CC BY 3.0. The license of the CoQA
dataset is composed of CC BY-SA 4.0, MSR-LA, and Apache 2.0. The license of the WiA-A/T
datasets is composed of CC BY-NC 3.0, CC BY-NC 4.0, and GNU General Public License v3.0.
Task
Dataset
#Train
#Valid
#Test
Input
Output
License
Commonsen generation
CommonGien
67,389
993
5.5
11.6
MIT
DART
62,659
2.768
27.5
21.5
MIT
E2E
33,525
4,299
9.5
20.6
CC BC-Y-
ToTTo
120,761
7,700
37.8
18.0
CC BY-SA
Data-to-text generation
WebNLG
34.338
4,313
4.222
18.0
19.9
CC BCBY-NA-SA 4.0
WchNLG (GEM)
35,426
1.667
17.7
22.7
CC BCBNNA-S 4.0
WikiBio
582,659
72,831
72,831
81.6
26.1
CC BY-SA
DailyDialog
76,052
7.069
6,740
72.5
13.9
CCBY-NC-SA 4.0
DSTC7-AVSD
76,590
17,870
1,710
148.2
11.5
MIT
Open-ended dialogue
PersonaChat
122,499
14,602
14,056
132.1
11.9
MIT
SGD
164,982
10,000
134.7
11.3
BC
MNLI-m
9,815
9,796
392,702
29.8
Mixed
MNLI-mm
9,832
9.847
Natural language inference
QNLI
104,743
5,463
5,463
36.6
CC
RTE
2,490
277
3,000
51.0
N/A
Paraphrase generation
Quora
137.185
3,000
3,000
10.9
10.8
N/A
MRPC
3,668
408
1,725
43.8
N/A
Paraphrase detection
QQP
363.846
40,430
390,965
22.3
N/A
STS-B
5.749
1,500
1,379
20.3
N/A
CoQA
107.286
31,621
349.4
Question answering
SQuAD
75,722
10,570
11,877
156.2
3.6
CC BC-YSAA4, 4.0
Question generation
CoQA
107,286
31,621
346.6
Mixed
SQuAD
75,722
10,570
11,877
148.3
11.6
CC CCBYSAA4 4.0
ROCStories
176,688
9,816
4,909
9.0
40.7
N/A
Story generation
WritingPrompis
53,516
4,000
2,000
25.5
150.4
MIT
Task-oriented dialogue
MultiWOZ
170,220
22.074
22,
128.3
11.3
MIT
CoLA
8,551
1.043
1,063
7.7
Text classification
SST-2
67,349
872
9.8
N/A
WiA-A
359
Text simplification
483,801
20,000
26.2
21.5
Mixed
WiA-T
359
TYAPC-E&M
52,595
11,508
1,416
9.9
10.6
Text style transfer
GYAPC-F&R
51,967
11,152
1,332
10.7
11.3
CNN//ailyMail
287.227
13,368
11,490
679.8
48.3
MIT
SAMSum
14.732
20.3
CC CCB-NNC-DD40
Text summarization
WLE
99.020
28.614
367.6
33.4
CCO 1.0
XSum
204.045
11,332
11.334
373.7
21.1
MIT
30

---

Under review as a conference paper at ICLR 2023
C FINE-TUNING AND EVALUATION DETAILS
In this section, we introduce the details for fine-tuning and evaluating each downstream task.
For the experiments Section (Tables 2 and 3 and Appendix D.I (Table D. the fine-tuning details
introduced in Section 4 and the evaluation details are presented follows:
For data-to-text generation tasks. use the script provided by Chen et al. (2020b
BLEU(-4), ROUGE-L, and METEOR for evaluation. We
For open-ended dialogue system tasks, we use BLEU-1, BLEU-2, Distinct-1, and Distinct-2 for evaluation. For DSTC7-AVSD we also utilize CIDEr (Vedantam et al. 2015). We employ NLTK
3.5 with smoothing function 7 to compute BLEU for PersonaChat and DailyDialog, and utilize the script I to evaluate DSTC7-AVSD;
For question answering tasks, we Exact Match (EM) and Macro-averaged Fl score (F1) for evaluation. We use the provided script for CoQA and SQuAD
For question generation tasks, we use BLEU-4, ROUGE-L, and METEOR for evaluation. We use the script provided by Dong et al. (2019)
employ nucleus sampling with 0.9 and temperature of 0.7 follow- ing Guan et al. (202T). We use corpus BLEU-1, BLEU-2, Distinct-1, and Distinct-4 for evaluation. We use NLTK 3.5 to calculate corpus BLEU following Guan al. (2021):
For story generation,
For task-oriented dialogue system tasks, we use BLEU(-4), inform (rate), success (rate), and combined score for evaluation. Inform and success two specially designed accuracy metrics for task-oriented dialogue system, and the combined score defined as (Inform + Success) x 0.5
BLEU (Budzianowski et al. 2018). We use the script provided by Su et al. (022) [
For text summarization tasks, we use ROUGE-1, 1, ROUGE-2, and ROUGE-L L for evaluation. We use the toolkit Files2rouge
For the experiments in Section 5 (Tables[aadd] the fine-tuning and evaluation details follows:
For paraphrase generation tasks, we employ the fine-tuning and evaluation scripts provided by AESOP (Sun et al. 2021 The evaluation metrics are BLEU-4, ROUGE-1, ROUGE-2, ROUGE-
L., and METEOR.
For text style transfer tasks, we employ the fine-tuning and evaluation scripts provided by SC & BLEU (Lai al. 22021) 12
We conduct the informal-to-formal transfer and train the model on the data from both the E&M and F&R domains following Lai et al. (202T). The evaluation metrics are BLEU-4, accuracy, and HM. Accuracy is calculated by a pre-trained TexICNN evaluate the style strength, and HM denotes the harmonic mean of BLEU-4 and style accuracy (Lai et al. 2021)).
For GLUE tasks, we utilize the fine-tuning code provided by Hugging Face The hyper-
parameters are consistent with original BART (Lewis et al. 2020 14 The evaluation is computed
by the official website
eva uuate.yy
httpf//hhhhhoooaaaaaaa/r/////rrr https:://g coga/eval.pp http:////hhtcooooaaaa/////////er
pyt
REA READM-. qlu......uemdd
https://g
wewencharrk.ccmm
31

---

Under review as a conference paper at ICLR 2023
Table 9: The results on six seen tasks under full tuning settings.
Nguven et al.
2021
(Tag
al. 2022)
(Gu al. 2021) (Lewis al. 2020
"Guan
2021
Chen al.
2022
(Chen et al. 2020b)
(Raffel et al. (0200)
(uu et al. 2021
XSum
SAMSum
CoQA QG
Methods
R-1
R-2
R-L
R-1
R-2
R-L
B-4
ME
R-L
SOTA
49.57
25.08
41.81
53.89
28.85
49.29
15.78
40.15
50.98
BART
45.14
22.27
37.25
51.74
26.46
48.72
12.34
35.78
46.88
MVP
45.67
22.53
37.41
53.91
29.28
49.40
22.96
47.14
54.84
MVP+S
45,59
22.54
37.39
53.97
29.44
49.69
23.04
47.01
55.14
WritingPrompts
DailyDialoy
WikiBio
Methods
B-1
B-2
D-1
D-4
B-1
B-2
D-1
D-2
B-4
SOTA
22.40
8.40
31.30
46.10'
40.70
4.10
22.20
45.10F
BART
22.40
8.40
31.30
44.30'
39.20
3,90
21.10
MVP
31.81
12.80
2.58
69.45
52.34
43.93
6.39
35.65
48.42
MVP+S
29.18
11.11
3.71
78.02
51.04
42.87
6.70
36.84
48.19
DSTC7-AVSD
SQuAD
Methods
B-1
B-2
B-3
B-4
ME
R-L
CIDEr
F1
EM
SOTA
83.20'
70.50
59.80
50.60
31.40
63.80
1.391
96.22
91.26
BART
82.40
69.10
58.20
48.70
31.30
63.50
1.382
91.56'
84.23
MVP
83.92
70.83
60.06
50.64
31.95
64.87
1.429
93.04
86.44
MVP+S
83.82
70.95
60.23
50.87
31.49
64.29
1.409
93.21
86.78
For the experiments of the GEM benehmark in Appendix D.2 (Table T0p. the same as those described in Section 4 use the GEM evaluation scripts
the fine-tuning settings are We use BLEU-4, ROUGE-2, and METEOR for evaluation.
D ADDITIONAL RESULTS
In this section, we provide additional results of our MVP model and other baselines.
D.1 RESULTS OF COMMON DATASETS
We also conduct experiments on eight common datasets under full tuning settings. Due to space limits in Section these results are shown in Table We can see that these results share a similar trend to those in Section and we achieve SOTA performances in of 8 datasets.
D.2 RESULTS ON THE GEM BENCHMARK
To better compare with ExT5 (Aribandi al. 2022). we conduct experiments on the GEM bench- mark (Gehrmann al.) 202T). For "unseen" commonsense generation and text simplification tasks, we utilize prompts of data-to-text generation and summarization, respectively. The results are presented in Table 10 and our MVP models outperform ExT5 in 26 out of 27 metries.
D.3 RESULTS WITHOUT FINE-TUNING
Considering our MVP model has already been pre-trained on several tasks, we conduct experiments on these "seen" tasks without fine fine-tuning model. To some degree, this setting be viewed as
32

---

Under review as a conference paper at ICLR 2023
Table 10: The results the GEM benchmark under full tuning settings. We utilize the large version of TS.1.1 and ExT5, and all the results of them are from Aribandi et al. (0022).
DART
E2E
ToTTo
Methods
B-1
R-2
ME
B-1
R-2
ME
B-1
R-2
ME
T5.1. ExT5
34.31
45.22
36.30
42.57
46.60
38.20
39.79
49.90
36.80
36.62
48.14
37.60
42.25
46.70
38.10
40.14
50.33
36.90
MVP MVP+S
39.13
48.92
38.53
37.38
47.96
39.39
50.58
55.24
41.27
38.83
48.49
38.41
37.32
47.40
38.90
50.69
55.52
41.29
WebNLG
CommonGen
SGD
Methods
B-4
R-2
ME
B-4
R-2
ME
B-4
R-2
ME
15.1.1
31.67
43.31
34.40
8.38
17.01
20.20
33.15
36.17
32.40
ExT5
35.03
48.17
36.50
9.68
19.04
21.40
34.74
37.77
33.00
MVP MVP+S
47.03
59.00
42.34
32.59
37.71
33.00
45.63
48.29
38.48
47.03
59.03
42.28
34.10
37.87
33.11
45.24
48.25
38.47
WiA-A
WiA-T
WLE
Methods
B-4
R-2
ME
B-4
R-2
ME
B-4
R-2
ME
T5.1.
29.30
38.37
30.10
42.12
50.52
36.2
15.55
20.47
19.60
ExT5
29.23
37.98
30.00
41.39
50.38
35.8
16.64
21.16
20.40
MVP
71.55
70.88
48.19
91.73
83.46
57.34
18.80
22.84
21.95
MVP+S
70.37
70.65
47.70
91.12
83.59
56.95
18.52
22.57
22.02
Table 11: The results on seven seen tasks without fine-tuning, Given that TO has been pre-trained on the CNli dataset, exclude their results to provide fair comparison (denoted "-").
CNN/DailyMail
WebNLG
SQuAD (QG)
CoQA
Methods
R-1
R-2
R-L
B-4
ME
R-L
B-4
ME
R-L
Fl
EM
FT BART
44.16
21.28
40.90
64.55
46.51
75.13
22.00
26.40
52.55
68 88.60
FT MVP
44.44
21.61
40.99
67.76
47.74
77.04
26.21
27.20
53.46
86.51
77.62
TO
1.40
10.20
18.43
3.06
12.43
14.91
13.30
6.60
MVP MVP+S
29.50
11.29
25.92
34.42
31.33
52.33
2.90
13.94
15.48
29.40
18.20
25.60
9.51
22.67
39.43
34.32
55.34
2.96
15.23
18.23
52.40
37.30
ROCStories
Persona
MultiWOZ
Methods
B-1
B-2
D-1
D-4
B-1
B-2
D-1
D-2
B-4
Success
Inform
FT BART
30.70
13.30
69.90
49.90
40.00
1.30
8.00
17.89
74.91
84.88
MVP
33.42
15.54
75.06
50.07
40.54
1.54
9.86
20.04
79.20
87.20
TO
8.69
3.02
4.37
35.49
23.20
23.57
2.56
12.06
0.02
2.50
22.10
MVP
1.01
0.31
7.18
86.26
35.54
32.71
2.87
16.38
3.08
2.50
22.20
MVP+S
10.52
3.54
2.13
69.55
37.04
33.38
2.66
14.84
0.38
2.50
22.10
zero-shot learning. Nonetheless, it does not conform the definition of true zero-shot settings (Perez al. 2021). To avoid controversy, refer to this without fine-tuning.
include TO-3B (Sanh et al. 2022) as our baseline. The results are listed in Table II model outperforms TO in all metrics with large margin. However, all tasks demonstrate that methods without fine- line-tuning perform significantly worse than those with full tuning settings. This suggests that zero-shot strategies that effective for NLU tasks may not produce satisfactory results for NLG tasks. Even though our model has acquired task knowledge, it struggles to perform well in new domain without being fine-tuned. Thus, focus mainly full tuning settings in this paper.
Our MVP
33

---

Under review as a conference paper at ICLR 2023
E QUALITATIVE EXAMPLES
In this section, we showcase the linearized inputs, human-writeer task prompts, and corresponding outputs of single dataset for tasks in Section We provide the results of BART, MVP, and MVP+S under full tuning settings. To minimize human intervention, we select the first and second instances of the test set.
34

---

Under review as a conference paper at ICLR 2023
Table 12: The first instance from the CNN/Daily Mail dataset. Sluman-writeee task prompts are labeled in italic. The setting is the same below.
Input
Summmrizee: Marseille, France (CNN)The French prosecutor leading an investigation into the crash of Cermanwings Hight 9525 insisted Wednesday that be was not of any video footage from board the plane Mansrille prosecutvr Brice Robin told CNN that "so far no videos used in the crash invesigation" He asdled, "A person who such video needs immediately give the imvesigatoes." " Robis's
follow claims by two magazizes, Gemman daily and Freach phone video showing harowing final seconds from on bourd Germawwnns Flight 9525 as il crasted into the Frrenhh Alps. 150 buard were killed. Paris Motch and Rild reported that the video was recovesed from a phone the wreckage The two publications descrited supposed video, but did not post it on their wehsites, The publicstions said that they watched the video, which was found by a source close to the invesigation. "Oue can hear cries of "My God' in several languages," Puris Match reported. "Metallic bunging can also be heard more than three times, pechags of the pilot trying open the cockpit door with a heney object Towards the end, after hency stake, strogger than the others, the scresming inteneifies. Then nothing" "'t is very disturhnn scene"" said Julian Reichelt, editor in- chaef of Bild online. An official with France's accidem investigation agency, BEA, stid the ageney Mern-Mare Menichini, French Gendarmoric spokesman charge of communicaions rescue afforts around Gemmawings crash told CNN reports were "complisely wrong" and "unwarrated" Cell phones have collected at the snc, said, but that they "hado't 't been exploited yea" Menichini the cell phones would need to be sent to the Criminal Research Instiiule in Rosny soue Bois new Paris, in order to be anilyzed by specialized techniciums working hand-in-llnnd with invesigutoes. But none of the cell phones found so far have besn sent to the instituee Menichini said. Asked whether suff inwalved in the search could have lesked a memory card to the media, Menichini answered with a categorical "no." Roichelt told "Eáin Burnett Outfrcar" that he had watched the video and stoud by the repoet, saying Bild and Paris Mutch "very confident" that the clip real. Ile noted that investigators only revealed they's ''d recovered cell phones from the crash site after Rild and Paris Mutch published their reports. "That samething did not know before. Overall
Paris Match,
of any video. Col.
believed
say many things of investigarion
weren't 't revealed by the invesigation at beginning." said. co-pilot Andreas Labitz had buttled depression years before he took the controls of Gemanwings Flight 9525, which he's accused of delibentely crashing last week in the French Alps. Lubitz told his Lufthussa flihht training school in 2009 that he had a "previous qpisode of severe depression." the airtine said Tuesday, Eimall vorrespnndence between Labitz and the school discovered in an internal investigatin, Lublaassa said, included medical documents be submited in connection with resuming his flight tezining. The aenouncement indicates that Lufthanso, the parent company of Gentarwings, knew of Labetz's battle with depcession, allowed him contine training and ultimately put
mental state of Germanwings to pilot? German airline Lufthansa confirmed Tuesday that
cockpit. Lufthessa whose CBO Carsten Speht previoulyy said Lubitz 100% to fly, described its statement Tuesday "swift seumless clarificatioon and
sharing the information documents- inchading training and medical reconds- traveled to the crush site Wodnesday, where recovery teams have been weeking for the post week to reoover human remains and plane debeis seattred across steep mountainside. He saw the crisis center set up in Scyne les Alpes. liid a wreath in the village of Le Vernet, closer to the crush site, where greving families have left flowers at a simple stone memorial. Menichini wold CNN late Tuesday that so visible humon remiiss were left at the site but recowery tesms would keep seanching. French President Francois Hollande, speaking Tuesday, said that should be possible to identity all the vietims using DNA analysis by the end of the week, sconer than authrriies had provicusly sugested. the meaniime the necoery of the victims" personal belongings will start Wedneskay Menichini vaid. Among thuse penconal oa board Check
public prosecutuns Spotr
betongings could be cell phoses belonging the 144 passeagers comrsponnlene the llight school during his training were amung several developments as investigators continued to possible motive for downing the jet. A Lufthama spokespenson told CNN on Tuesday that Labitz had a valid medical vertificate, had pussed all his examinations and "held all the lieenses required." Earlier, a spokesman for the prosecutoe's i office in Dusseldorf, Christoph Kamça, said madical recoeds reveal Lubitz sufforad from suicidal sendencies some point before his aviation career and underwent psychotherapy before he got his pilot's license. Kumpu emphasized there's is no evidence suggesting Labitt was snicidal acting aggrosiively hefore the crash. Investigators are looking into whether Lubitz feared his medical condition would cause him lose his pilot's licesse, Europeun government official briefed only theoey being considened. Another source, a enforcement official briefed on investigation also told CNN authorites believe the primory motive Labitz bring down the plase was leared he would not allowed to dy becouse of medical problems. Labut''s girlfriand told investigators he seen eye ductor and a neumpeychologii, boch of whom deemed him unfit to woek recently and cuncluded he had psychological issues, the European government official said. But no matier what details emerge about his previous mental hoalth singgles, there's 's more to the story, said Brisn Russell, a furensic psychologis.' "Psychology cun explain why somchody would tamn mge inward on thenselves about the fact that mayhe they weren't going keep doing their joh and they'r 'e upser about that and they te suicidal." " he said. "But there mental illmess that explains why somebody then feels entiled to also take that and it outward on 149 other people who nothing de with persen's prohlems." " Germanwings crssh compensation: What we know: Who the captain Germanwings Flight 95257 CNN's Margot Haddad reported from Marseille and Pamela Brown Dusseloof, while Laura Smith Spuzk wrote London. CNN's Frederik Pleitgen, Pamela Boykoff, Antonia Mortensen, Sandrine Amiel and Area- Maja Raguard contributed this report.
latest our conespandenss. The details about Lubinz's
caused crach and Lubitz's
invesiigatio told CNN Tuesday. While Syyng was "a big part his life," the source said, it's
Gold
Match are "very confident" the video clip is real, an editor says. Andreas Lubitz had informed his Lufthansa training school of an episode of severe depression, airline says.
BART
Prosecutor: "A person who has such a video needs to immediately give it to the investigators" French Gen- darmerie spokesman: Cell phones have been collected at the site, but haven't 't been exploited.
MVP
Prosecutor: "So far no videos were used in the crash investigation" An official with France's accident investigation agency says the agency is not aware of any video.
MVP+S
Prosecutor: "So far no videos were used in the crash investigation" Two publications claim to have found cell phone video of the crash. "It is very disturbing scene," says editor-in-chief of Bild online.
35

---

Under review as a conference paper at ICLR 2023
Table 13: The second instance from the CNN/Daily Mail dataset.
Input
Summurizz:: The Palestinian Authority officially became the 123rd member of the International Criminal Court on Wednesday, a step that gives the court accession was marked with a ceremony The Hague, in the Netherlands, where the court based. The Palestinians signed the ICC's founding Rome Statute in January, when they also accepted its jurisdiction over alleged crimes committed "in the occupied Palestinian territory, including East Jerusalem, since June 13, 2014." Later that month, the ICC opened preliminary examination into the situation in Palestinian territories, paving the way for possible war crimes investigations against Israelis. As members of the court, Palestinians may he subject counter-charges as well. Israel and the United States, neither of which is ICC member, opposed the Palestinians' efforts to join the body. But Palestinian Foreign Minister Riad al-Malki, speaking at Wednesday's 's ceremony, said it was a move toward greater justice. "As Palestine formally becomes a State Party to the Rome Statute today, the world is also step closer to ending long era of impunity and injustice." he said, according to ICC news release. "Indeed, today brings us closer to our shared goals of justice and peace." " Judge Kuniko Ozaki, vice president of the ICC, said acceding to the treaty was just the first step for the Palestinians. "As the Rome Statute today enters into force for the State of Palestine, Palestine acquires all the rights as well as responsibilities that come with being a State Party to the Statute. These are substantive commitments, which cannot be taken lightly." she said. Rights group Human Rights Watch welcomed the development. "Governments seeking to penalize Palestine for joining the ICC should immediately end their pressure, and countries that support universal acceptance of the court's 's treaty should speak out to welcome its membership." said Balkees Jarrah, international justice counsel for the group. "What's objectionable is the attempts to undermine international justice, not Palestine's 's decision to join treaty to which over 100 countries around the world are members." In January. when the preliminary ICC examination was opened, Israeli Prime Minister Benjamin Netanyahu described as an outrage, saying court was overstepping boundaries. The United States also said it "strongly" disagreed with the court's ' decision. "As we have said repeatedly, we do not believe that Palestine is state and therefore we do not helieve that is cligible to join the ICC," the State Department said in statement. It urged the warring sides to resolve their differences through direct negotiations. "We will continue oppose actions against Israel the ICC as counterproductive said. But the ICC begs to differ with the definition a state for purposes and refers to the territories as "Palestine." While a preliminary examination is not a formal investigation, it allows the court to review evidence and determine whether to investigate suspects on both sides. Prosecutor Fatou Bensouda said her office would "conduct its analysis in full independenee and impartiality." The war between Isracl and Hamas militants in Gaza last summer left more than 2,000 people dead. The inquiry will include alleged war crimes committed since June. The International Criminal Court was set up in 2002 to prosecute genocide, crimes against humanity and war crimes. CNN's Vasco Cotovio, Kareem Khadder and Faith Karimi contributed to this report.
court.juriddittii jurisdiction over alleged crimes in Palestinian territories. The formal
the cause of peace."
Gold
Memhership gives the ICC jurisdiction over alleged crimes committed in Palestinian territories since last June. Israel and the United States opposed the move, which could open the door to war crimes investigations against Israclis.
BART
Palestinian Authority becomes 123rd member of the International Criminal Court. The move gives the court jurisdiction over alleged crimes in Palestinian territorics. Isracl and the United States opposed the Palestinians' efforts to join the body.
MVP
"Today brings us closer to our shared goals of justice and peace," foreign minister says. The Palestinians signed the ICC's founding Rome Statute in January. The move gives the court jurisdiction over alleged crimes in Palestinian territories.
MVP4S
"Today brings closer our shared goals of justice and peace," foreign minister says. The United States says it "strongly" disagrees with the decision. The Palestinian Authority is the 123rd member of the International Criminal Court.
36

---

Under review as a conference paper at ICLR 2023
Table 14: The first instance from the WebNLG dataset, which has two gold target sentences.
Input
Describe the following data: Abilene,TTexas cityServed Abilene A Regional_Airpon
Gold
Abilene, Texas is served by the Abilene regional airport.
Abilene Regional Airport serves the city of Abilene in Texas.
BART
Abilene Regional Airport serves the city of Abilene in Texas.
MVP
Abilene Regional Airport serves the city of Abilene, Texas.
MVP+S
Abilene Regional Airport serves the city Abilene, Texas.
Table 15: The second instance from the WebNLG dataset, which has three gold target sentences.
Input
Describe the following data: "Madrid, Paracuellos de Jarama, San Sebastián de los Reyes and www.SSuurrr.dddddde Airpos
Alcobendas" - location
Gold
Adolfo Suárez BadriddBarajas Airport be found in Madrid, Paracuellos de Jarama, San Sebastián de los Reyes and Alcobendas.
Adolfo Suarez Madrid Bariida airport is located at Madrid, Paracuellos de Jarama, San Sebastián de los Reyes and Alcobendas. Adolfo Suarez Madrid-Barajaa Airport is located in Madrid, Paracuellos de Jarama, San Sebastian de los Reyes and Alcobendas.
BART
Adolfo Suárez Madrid -Brrdjas Airport can be found Madrid, Paracuellos de Jarama, San Sebastián de los Reyes and Alcobendas.
MVP
Adolfo Suárez Madrid-Barajas Airport can be found in Madrid, Paracuellos de Jarama, San Sebastián de los Reyes and Alcobendas.
MVP+S
Adolfo Suárez Madrid Madrid-Barajas Airport is located in Madrid, Paracuellos de Jarama, San Sebastián de los Reyes and Alcobendas.
37

---

Under review as a conference paper at ICLR 2023
Table 16: The first instance from the SQuAD dataset.
Input
Generate the question based on the answer: Saint Bernadette Soubirous [SEP] Architecturally the school has Catholic character. Atop the Main Building gold dome is golden statue of the Virgin Mary- Immediately in front of the Main Building and facing it is a copper statue of Christ with arms upraised with the legend Venite Ad Me Omnes Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection It is replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive ( and in direct line that connects through statues and the Gold Dome ), a simple modern stone statue of Mary
Gold
To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? ?
BART
Who is believed to have appeared to the Virgin Mary at Lourdes ?
MVP
Who did the Virgin Mary appear to Lourdes
MVP+S
The Grotto is replica of the grotto at Lourdes France where the Virgin Mary appeared to whom
Table 17: The second instance from the SQuAD dataset.
Input
Generate the question based on the answer: copper statue of Christ [SEP] Architecturally the school has Catholic character Atop the Main Building gold dome is golden statue of the Virgin Mary Immediately in front of the Main Building and facing it is a copper statue of Christ with arms upraised with the legend Venite Ad Me Omnes Next the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection It is replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive and in a direct line that connects through 3 statues and the Gold Dome ), simple, modern stone statue of Mary
Gold
What is in front of the Notre Dame Main Building ?
BART
What is in front of the Main Building and facing it ?
MVP
What is immediately in front of the Main Building ?
MVP+S
What is immediately in front of the Main Building ?
38

---

Under review as a conference paper at ICLR 2023
Table 18: The first instance from the CoQA dataset.
Input
Answer the following question: what color was cotton ? [X_SEP] once upon a time in barn near farm house, there lived a little white kitten named cotton cotton lived high up in a nice warm place above the barn where all of the farmer horses slept. but cotton wasn alone in her little home above the barn, oh no she shared her hay bed with her mommy and 5 other sisters all of her sisters were cute and fluffy. like cotton. but she was the only white one in the bunch. the rest her sisters were all orange with beautiful white tiger stripes like cotton s mommy being different made cotton quite sad she often wished she looked like the rest of her family so one day. when cotton found a can of the old farmer' orange paint, she used it to paint herself like them when her mommy and sisters found her they started laughing what are you doing cotton ? 1"" i only wanted to be more like you" cotton mommy rubbed her face cotton and said oh cotton, but your fur is so pretty and special, like you. we would never want you to be any other way " and with that, cotton" mommy picked her up and dropped her into a big bucket of water, when cotton came out she was herself again. her sisters licked her face until cotton s fur was all all dry "don" ever do that again, cotton! they all cried. next time you might mess up that pretty white fur of yours and we wouldn want that ! then cotton thought, i change my mind.. like being special"
Gold
white
BART
white
MVP
white
MVP+S
white
Table 19; The second instance from the CoQA dataset.
Input
Answer the following question: what color was cotton ? [SEP] white [X SEP) where did she live ? [X_SEP] once upon time, in a barn near a farm house, there lived little white kitten named cotton cotton lived high up in a nice warm place above the barn where all of the farmer s horses slept. but cotton wasn t alone in her little home above the barn oh no she shared her hay bed with her mommy and 5 other sisters all of her sisters were cute and fluffy like cotton. but she was the only white one in the bunch. the rest of her sisters were all orange with beautiful white tiger stripes like cotton s mommy. being different made cotton quite sad. she often wished she looked like the rest of her family so one day, when cotton found can of the old farmer orange paint, she used it to paint herself like them.. when her mommy and sisters found her they started laughing you doing, cotton ? ! ""; only wanted be more like you cotton s mommy rubbed her face on cotton and said oh cotton, but your fur is so pretty and special like you. we would never want mommy picked her up and dropped her into
what are
you to be any other way" " and with that, cotton big bucket of water. when cotton came out she was herself again. her sisters licked her face until cotton s fur was all all dry don I ever do that again. cotton ! " they all cried.. " next time you might mess up that pretty white fur of yours and we wouldn t want that then cotton thought, change my mind. like being special"
Gold
in barn
BART in barn
MVP
in barn
MVP+S
barn
39

---

Under review as a conference paper at ICLR 2023
Table 20: The first instance from the ROCStories dataset.
Input
Given the story title: male was out jogging one morning.
Gold
the weather was crisp and cool male felt good and energetic he decided to keep jogging longer than normal. male went several more miles out of his way.
BART
he saw man struggling to walk across a field. male stopped to help him. he saw a guy struggling to walking across a fields. the man thanked male profusely
MVP
he ran into a man who was holding knife. male was scared, so he ran away the man threatened male... male ran back into his house.
MVP+S
he noticed sound coming from tree male went to investigate he found small baby bird stuck the trunk. male took the bird to the vet and they released it.
Table 21: The second instance from the ROCStories dataset.
Input
Given the story title: ken was driving around in the snow
Gold
he needed to get home from work he driving slowly to avoid accidents. unfortunately the roads were too slick and ken lost control. his tires lost traction and he hit tree
BART
he hit a patch of black ice ken slid and hurt his knee. he went to the doctor for it krampus was sent home with concussion.
MVP
he was in his car and it was very cold. ken started to back up the car started to make loud noise. kena had to back out and call for help.
MVP+S
he was having trouble starting his car. he had to push it couple of blocks. his car would n't start.
ken had to call a tow truck for help.
40

---

Under review as a conference paper at ICLR 2023
Table 22: The first instance from the PersonaChat dataset.
Input
Given the dialog: love to meet people... [SEP] have turtle named timothy [SEP] my favorite sport is ultimate frisbee [SEP] my parents are living in bora bora. [SEP] autumn is my favorite season. [X SEP] hello, how you doing tonight ?
Gold
i am well an loving this interaction how are you
BART
i am doing well, how are you ?
MVP
am doing well. how are you?
MVP+S
i am doing well. how are you?
Table 23: The second instance from the PersonaChat dataset.
Input
Given the dialog: i just bought brand new house [SEP] i like to dance at the club. [SEP] run a dog obedience school. [SEP] have a big sweet tooth [SEP] like taking and posting selkies [X SEP] hello, how are you doing tonight [SEP] am well loving this interaction how you"
Gold
i am great i just got back from the club.
BART
i am doing well, thanks for asking. what do you do for fun ?
MVP
am doing well just got home from dancing the club.
MVP+S
am doing well just bought new house
41

---

Under review as a conference paper at ICLR 2023
Table 24: The first instance from the MultiWOZ. dataset. The task-oriented dialogue is composed of dialogue state tracking, dialogue action learning and system response generation.
Input
Given the task dialog: Belief state [X SEP] would like taxi from saint john 's college pizza hut fen ditton.
Given the task dialog: Dialogue action [X_SEP] [db_ _ores]] [X_SEP] would like taxi from saint john is college to pizza hut fen ditton.
Given the task dialog: System response [X.SEP] nore] [X SEP) would like a taxi from saint
john 's college to pizza hut fen ditton.
Gold
[taxi] destination pizza hut fen ditton departure saint john college
[taxi] [request] leave arrive
what time do you want to leave and what time do you want to arrive by
BART
[taxi] destination pizza express fen ditton departure saint johns college
[taxi] [request] leave
what time would you like to leave or arrive ?
MVP
[taxi] destination pizza hut fen ditton departure saint john 's college
[taxi] [request] leave
sure when would you like arrive
MVP+S
[taxi] destination pizza hut fen ditton departure saint john 's college
[taxi] [request] leave arrive
what time would you like to leave
Table 25: The second instance from the MultiWOZ dataset.
Input
Given the task dialog: Belief state [X_SEP] would like a taxi from saint john 's college to pizza hut fen ditton... [SEP] what time do you want leave and what time do you want to arrive by [SEP] want to leave after 17:15
Given the task dialog: Dialogue action [X SEP] [db _nore]] [X SEP) would like taxi from saint john 's college to pizza hut fen ditton. [SEP] what time do you want to leave and what time do you want arrive by [SEP] want leave after 17:15.
Given the task dialog: System response IX SEP] [db_nores] IX SEP] would like taxi from saint john is college to pizza hut fen ditton [SEP] what time you want to leave and what time do you want arrive by [SEP] want to leave after 17:15
Gold
[taxi] destination pizza hut fen ditton departure saint john 's college leave 17:15
[taxi] [inform] car phone
booking completed your taxi will be [value_car] contact number is evalue_phone
BART
[taxi] destination pizza hut fen ditton departure saint john college leave 17:15
[taxi] [inform] car phone
booking completed booked car type: [value_cat contact number [valuc.phone
MVP
[taxi] destination pizza hut fen ditton departure saint john college leave 17:15
[taxi] [inform] car phone
booking completed! booked car type [value.car contact number [value phone]
MVP+S
[taxi] destination pizza hut fen ditton departure saint john 's college leave 17:15
[taxi] [inform] car phone
booking completed booked car type: [value car] contact number:
[value phone]
42


Reviews:
[{"review_id": "pkH4NajyEGO", "reviewer": "Reviewer_AMUT", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Considers supervised pre-training as an alternative to unsupervised pre-training, which is more common with state-of-the-art NLP models. Many (77) NLG supervised datasets are curated to create MVPCorpus which is used for supervised pre-training. However, the training setup is not designed in a way that allows us to compare unsupervised vs supervised pre-training since the backbone model is initialized with BART parameters. Thus the paper cannot answer the question which is posed, and instead shows that continued supervised pre-training using related tasks after starting with an unsupervised pre-trained model helps with downstream tasks. \n\nThe model is initialized with BART and is further 'pre-trained' using supervised/multi-task learning. The main baseline is BART of the same architecture and some variants are compared.\n\nFine-tuning is done in two settings: full fine-tuning, and with prefix tuning. They show that MVP does better in both settings compared to BART and achieves some SoTA results in NLG in the FT setting. Finally, it is shown that zero-shot performance is better than BART as well.\n", "summary_of_the_review": "This paper should be re-written as a study of how to continue pre-training with multi-task supervised learning or properly study supervised pre-training by initializing the model from random parameters. At best, if re-written this way the paper shows how to achieve slightly better results on certain NLG tasks compared to BART with intermediate fine-tuning. The novelty is limited and the claims are incorrect.\n", "strength_and_weaknesses": "# Strengths\n- Improvements over BART baseline,  which has the same architecture, in some tasks. \n- In some NLG cases achieves SoTA, although I've not verified every task.\n\n# Weaknesses\n- There is a major mismatch in the way the paper is written and the experimental setup. It is written to suggest that supervised pre-training can replace unsupervised pre-training, but does not have experiments to back this up. Consequently it has highly misleading claims on merits of supervised pre-training since the models are primarily pretrained in an unsupervised way.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "HSABvCXx9Mt", "reviewer": "Reviewer_VRii", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propose to continue pre-train a pre-trained seq2seq model using annotated data for generation tasks. The authors collected 77datasets over 11 diverse NLG tasks for the continued train step. Evaluation shows good results.\n", "summary_of_the_review": "In summary, the contribution of the paper is thin. The evaluation is not convincing (lack human evaluation). There are potential issues of concatenating multiple language generation datasets in the supervised pretraining step.", "strength_and_weaknesses": "### Strengths\n- Collect a significant amount of training data for seq2seq tasks.\n- Reconfirm empirically that using more data is always useful\n\n### Weaknesses\n- The contribution of the paper is too thin in my opinion (i.e., showing that using more data is useful)\n\n- On *Generaliability* study: I don’t really understand this setup. The authors call it unseen tasks but then the  model is fine-tuned using the annotated data of the task. How is this different from taking any pretrained model and finetuning it on the same data? My understanding here is that the “unseen” part means that the tasks and data weren’t in the continued train step. But calling that _generaliability_ seems to be an overstatement to me. Is that what we all do with pretrained models?\n- There are no human evaluation results. Evaluating language generation is tricky, using automatic metric alone is not sufficient.\nWhile measuring by automatic metric, it shows that supervised pre-training with  more data is useful. There are many datasets with different characteristics. Does it hurts the model in a way that we don’t know yet? What type of hallucination in language generation when the model is trained on diverse datasets? Does the nature of informal text in the Personal Chat dataset have negative influences on formal text CNN/Daily Mail. I think there are many important questions about mixing random datasets in the context of language generation that have not been answered. The small improvement in automatic metrics do not fully justify the approach.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "546Kq14BEE", "reviewer": "Reviewer_4XxP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on multi-task pretraining via supervised data in NLG area. They collected 77 NLG datasets and applied different variants of multi-task pretraining on top. The pretrained models are evaluated on both seen and unseen tasks, comparing to the prior STOAs and BART finetune baselines.  ", "summary_of_the_review": "Overall, the paper makes a significant contribution on the collection of a large amount of NLG datasets and study the multi-task pretraining behaviors for NLG. Their code are released and the results can be reproduced. The idea presented in this paper is only marginally novel and mostly follow the prior work. The results of their proposed method also seems mixed in different datasets.", "strength_and_weaknesses": "Strengths:\n- A collection of data mixtures over 77 NLG datasets, which are publicly released.\n- A exploration effort on studying multi-task pretraining on the NLG area.\n- Code are released and could be easy to reproduce the results.\n\nWeaknesses:\n- The idea of multi-task pretraining is not novel, as it mostly follows Aribandi et al. (2022).\n- Results are mixed. On Table 2, it seems the multi-task pretraining has minor gains over BART finetuning or Single; while on Table 9, the improvements are indeed significant. \n- The proposed variants of pretraining stages (e.g., MVP+S, MVP+R, MVP+M) don't seem effective as MVP (the results are on-par on most datasets from Table 2 & 3)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject