SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 1: Cross-task transferability performance comparisons of different methods in fully-supervised setting. Bold and underline fonts denote the best and the second best methods (the same as below).
Target Dataset	CNN/DAILY MAIL	PERSONACHAT
#Metrics	ROUGE-1	ROUGE-2	ROUGE-L	BLEU-1	BLEU-2	Distinct-1	Distinct-2
GPT-2Large	30.30	7.66	26.40	36.07	22.64	1.57	8.54
BARTLarge	41.37	21.16	38.36	40.48	26.48	1.42	7.60
T5Large	40.47	20.30	37.57	42.23	27.36	1.39	7.63
PREFIXTUNING	41.79	20.69	38.50	41.87	27.28	1.33	7.20
SPOT	39.85	18.21	36.33	40.39	26.34	1.32	7.60
MT MODELTUNING	41.71	21.41	38.67	42.53	27.83	1.39	7.86
PTG	42.68	21.63	39.45	45.47	29.52	1.43	8.34
Target Dataset	XSUM	DAILYIALOG
#Metrics	ROUGE-1	ROUGE-2	ROUGE-L	BLEU-1	BLEU-2	Distinct-1	Distinct-2
GPT-2Large	28.28	9.17	22.29	29.14	18.01	5.78	21.52
BARTLarge	43.93	20.78	35.94	32.62	21.77	5.16	25.08
T5Large	41.01	17.84	32.60	31.54	20.08	5.70	29.25
PREFIXTUNING	42.87	19.98	34.82	34.00	21.63	4.31	19.95
SPOT	41.43	17.56	31.33	30.22	20.11	4.91	25.56
MT MODELTUNING	43.75	20.70	35.66	34.41	23.08	5.46	27.23
PTG	44.21	20.99	36.00	42.72	28.75	5.36	29.48
Table 2: Cross-dataset transferability performance comparisons of different methods in fully-supervised setting.
target prompt and parameters of the backbone PLM but only tune the multi-head attention parameters in adaptive attention mechanism (Eq. 4).
In particular, we adopt BART-LARGE to learn a set of source prompts. The length of prompt is set to 200 and the learning rate is set to  $1 \times 10^{-3}$ . For the target generation task, we utilize BART-LARGE as the generation backbone and frozen BERT-LARGE to obtain the instance-level query  $q^{ins}$ . The dimension  $d$  is set to 1024, which is the same as the embedding size  $e$  of the BERT/BART-LARGE. The multi-head attention in adaptive attention mechanism has 16 heads. During fine-tuning, the learning rate of BART is set to  $3 \times 10^{-5}$  and the learning rate of cluster key  $k^c$ , prompt key  $k^p$ , task key  $q^{task}$  and multi-head attention is set to  $1 \times 10^{-3}$ . The value of  $\lambda$  is set to 0.5 based on the performance in validation set. The training details of baselines can be found in Appendix B.
Evaluation Metrics. For performance comparison, we adopt three automatic evaluation metrics widely used by previous works, i.e., BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and Distinct (Li et al., 2016). Specifically, BLEU-  $n$  measures the ratios of the co-occurrences of  $n$ -grams between the generated and real text; ROUGE-  $n$  measures the text quality by counting the overlapping  $n$ -grams between the generated and real text; and Distinct-  $n$  measures the degree of diversity by calculating the number of distinct  $n$ -grams in generated text.
Table 1 and Table 2 present the fully-supervised results of cross-task and cross-dataset transferability, respectively, for our model and baselines. In fully-supervised setting, we use all training instances of the target task to train our model.
For the cross-task experiment, we consider two pairs of source and target tasks transfer: 1) the tar
Target Task	SUMMARIZATION (R-1/R-2/R-L)	DIALOG (B-1/B-2/D-1/D-2)
#Instances	50	100	200	500	50	100	200	500
GPT-2Large	19.8/3.1/17.5	20.6/3.9/18.4	26.1/6.1/23.3	29.2/7.3/26.0	27.7/10.4/2.2/10.9	27.6/10.4/2.2/11.4	29.3/11.1/2.1/11.5	31.4/12.0/1.9/10.7
BARTLarge	37.5/16.9/34.4	38.8/17.9/35.6	39.3/18.4/36.1	39.9/19.0/36.7	22.7/9.0/1.3/5.4	30.0/11.9/1.3/5.2	32.4/12.8/1.3/5.7	31.7/12.6/1.3/5.6
T5Large	39.1/18.3/36.2	39.9/18.5/36.8	40.0/18.7/37.0	39.6/19.2/36.7	41.7/15.5/0.9/6.6	42.1/15.7/0.8/5.4	43.1/16.3/0.7/4.6	45.1/17.4/0.8/4.4
PREFIX	32.2/12.4/28.5	32.3/12.5/28.5	34.0/13.7/30.9	37.5/16.3/34.7	39.6/23.9/0.6/3.4	39.7/24.0/0.5/3.1	36.4/22.4/0.8/3.7	25.7/16.1/1.1/4.1
SPOT	31.3/11.8/27.5	31.9/11.8/27.5	33.6/12.6/29.3	36.5/16.0/33.6	38.3/22.1/0.5/3.0	38.2/22.0/0.5/3.0	39.0/23.2/0.8/4.1	41.1/23.5/1.0/4.5
MODEL	36.2/15.6/32.8	37.8/16.6/34.4	38.6/17.3/35.2	39.3/17.9/35.8	24.9/9.9/1.5/6.6	24.8/9.8/1.6/6.6	27.8/11.0/1.6/7.1	28.9/11.4/1.7/7.8
PTG	37.8/16.7/34.5	39.0/17.5/35.6	39.3/17.7/36.2	40.1/19.1/36.8	37.3/22.6/1.1/6.2	39.9/21.2/1.1/5.3	37.7/23.6/1.1/4.9	37.7/24.2/1.4/6.3
Table 3: Cross-task transferability performance comparisons of different methods in few-shot setting. B-n, R-n, D-n, and MODELT are short for BLEU, ROUGE, Distinct and MULTI-TASK MODELTUNING (the same as below).
Target Data	CNN/DAILY MAIL (R-1/R-2/R-L)	PERSONACHAT (B-1/B-2/D-1/D-2)
#Instances	50	100	200	500	50	100	200	500
GPT-2LARGE	19.8/3.1/17.5	20.6/3.9/18.4	26.1/6.1/23.3	29.2/7.3/26.0	27.7/10.4/2.2/10.9	27.6/10.4/2.2/11.4	29.3/11.1/2.1/11.5	31.4/12.0/1.9/10.7
BARTLARGE	37.5/16.9/34.4	38.8/17.9/35.6	39.3/18.1/36.1	39.9/19.0/36.7	22.7/9.0/1.3/5.4	30.0/11.9/1.3/5.2	32.4/12.8/1.3/5.7	31.7/12.6/1.3/5.6
T5LARGE	39.1/18.3/36.2	37.9/18.5/36.8	39.0/18.7/36.0	39.6/19.2/36.7	31.7/15.5/0.9/6.6	32.1/15.7/0.8/5.4	33.1/16.3/0.7/4.6	35.1/17.4/0.8/4.4
PREFIX	32.2/12.4/28.5	32.3/12.5/28.5	34.0/13.7/30.9	37.5/16.3/34.7	39.6/23.9/0.6/3.4	39.7/24.0/0.5/3.1	36.4/22.4/0.8/3.7	25.7/16.1/1.1/4.1
SPOT	31.9/11.5/26.8	31.9/11.4/26.8	33.0/12.8/29.3	36.6/15.5/33.2	37.6/22.0/0.5/3.1	37.6/22.2/0.5/3.2	35.0/20.2/0.7/3.2	21.2/15.6/1.0/3.8
MODELT	37.7/17.0/34.5	38.8/17.9/35.6	39.3/18.2/36.0	40.5/19.0/36.1	32.0/13.1/2.4/12.4	34.2/13.9/2.2/11.9	35.9/14.7/2.1/11.7	35.5/14.7/2.0/10.8
PTG	37.9/16.5/34.5	38.7/17.5/35.8	39.5/18.3/36.2	39.9/18.7/36.6	34.6/21.5/1.1/4.5	36.9/19.3/1.0/5.5	38.6/24.1/1.0/4.4	36.7/23.0/1.2/5.5
Target Data	XSUM (R-1/R-2/R-L)	DAILYIALOG (B-1/B-2/D-1/D-2)
#Instances	50	100	200	500	50	100	200	500
GPT-2LARGE	12.2/1.5/9.8	11.3/1.1/9.1	11.1/1.1/8.9	12.9/1.7/10.2	18.5/7.0/5.9/23.3	19.3/7.3/5.6/22.8	20.9/7.9/5.4/22.0	22.0/8.3/5.5/2.9
BARTLARGE	33.2/10.3/25.2	32.8/11.0/26.6	34.5/11.6/25.5	36.4/13.2/28.2	22.0/8.5/3.5/15.6	22.2/8.5/3.3/14.5	24.8/9.6/3.4/14.9	24.3/9.4/3.8/11.4
T5LARGE	23.2/5.0/16.6	23.4/5.3/17.1	26.0/7.1/19.5	30.8/10.3/24.2	30.6/14.8/2.5/14.9	41.0/15.0/2.4/14.1	30.9/15.1/2.8/15.4	30.6/15.1/3.2/17.7
PREFIX	25.0/8.3/17.9	25.0/8.2/17.9	25.1/8.2/18.1	27.5/9.8/19.7	38.1/22.6/2.8/14.1	38.4/22.8/2.5/12.0	35.2/21.0/2.5/11.6	21.8/13.5/3.8/16.1
SPOT	23.4/6.6/16.6	23.4/6.5/16.6	23.5/6.8/17.0	25.5/7.5/18.6	35.5/20.6/2.5/13.2	35.7/20.8/2.3/12.8	33.6/18.9/2.2/11.9	25.0/13.2/3.7/16.1
MODELT	35.6/13.1/27.8	35.7/13.3/28.0	36.0/13.6/28.4	36.1/13.8/28.5	28.2/11.1/5.4/24.3	30.4/11.8/5.2/23.9	29.8/11.8/4.9/23.0	29.5/11.7/4.7/22.3
PTG	33.6/10.9/25.4	33.8/11.2/25.9	34.7/12.0/26.8	36.8/13.6/27.7	31.8/19.4/2.5/11.6	30.9/18.9/2.8/12.8	31.5/19.3/2.9/13.9	31.0/19.0/3.1/14.9
Table 4: Cross-dataset transferability performance comparisons of different methods in few-shot setting.
get task is summarization (CNN/Daily Mail), and the source tasks are the mixture of other five tasks; and 2) the target task is dialog (PersonChat), and the source tasks are other five tasks. For the cross-dataset experiment, we consider datasets within summarization and dialog. For summarization, the target dataset is CNN/Daily Mail or XSum, and the source datasets are the mixture of other four summarization datasets. For dialog, the target dataset is PersonaChat or DailyDialog, and the source datasets are other four dialog datasets.
First, by transferring prompts from source tasks to the target task, PTG outperforms GPT-2, BART, T5 and PREFIXTUNING. The results suggest that prompt transfer in PTG provides an effective means of improving the performance of typical fine-tuning and prompt methods since our method utilizes the knowledge learned from source tasks.
Second, PTG performs better than the prompt-based transfer method, SPOT. While transferring prompts, SPOT considers each source task equally and ignored the specific instance information. And SPOT only learns a common prompt for source tasks to directly initialize the target prompt. By
constraint, PTG clusters diverse source prompts and uses an adaptive attention mechanism considering both task- and instance-level characteristics.
Finally, PTG produces competitive performance or even exceeds the strong MULTI-TASK MODEL-TUNING. Different from most NLU tasks sharing some common knowledge to understand the semantics and syntax of surface words, text generation tasks need to generate diverse text based on different input data, thus having large task boundaries. Thus, in cross-task transfer, simply tuning PLMs on a mixture of tasks without considering the task similarity leads to a performance decrease. While, our prompt-based transfer learning approach can still achieve the best performance, showing that PTG improves stability across tasks and datasets.
5.3 Few-Shot Setting
In few-shot setting, we only sample a handful of training instances of the target task to train our model. Specifically, we subsample the target task dataset to obtain small training datasets of size \{50, 100, 200, 500\}. For each size, we sample 5 different datasets and average over 2 training random
Target Task	SUMMARIZATION (CNN/Daily Mail)	DIALOG (PersonaChat)
Model	ROUGE-1	ROUGE-2	ROUGE-L	BLEU-1	BLEU-2	Distinct-1	Distinct-2
PTG w/o Prompt Pool	41.46	20.40	38.40	39.70	24.45	0.77	4.00
PTG w/o Prompt Cluster	42.10	21.15	38.86	44.63	29.20	1.34	7.78
PTG w/o Multi-Key Memory	42.12	21.14	38.85	44.67	29.34	1.42	8.23
PTG w/o Instance-level Query	42.16	21.22	38.93	44.74	29.28	1.36	7.80
PTG	42.40	21.35	39.14	45.46	29.52	1.46	8.43
Table 5: Ablation analysis on cross-task transferability experiments.
seeds. Thus, we average over 10 models for each few-shot setting. In few-shot setting, we adopt the same cross-task and cross-dataset experiments with the fully-supervised setting. Table 3 and 4 shows the few-shot results of our model and baselines.
We can clearly see that PTG achieves competitive (underline fonts) or better performance (bold fonts) than the strong baseline (i.e., MULTI-TASK MODELTUNING) in most low-data regimes, but the gap narrows as the training dataset size increases. In addition, our model outperforms most of vanilla PLMs in most cases. The reason behind this might be that large PLMs can easily suffer from overfitting during few-shot learning due to their massive parameters (Gao et al., 2020). While, in our framework, we adopt a lightweight technique, i.e., prompting, to learn source prompts, which can provide the previously learned knowledge in source tasks to PLMs and serve as a better starting point when solving the target tasks.
5.4 Effectiveness of Core Designs
We further conduct ablation studies to demonstrate the effectiveness of the core designs of PTG.
Source Prompt Pool. To confirm the importance of the prompt pool, we design a counterpart of our method with only training a sharing prompt for all source tasks. From Table 5 (row 1), we can see that PTG significantly outperforms its counterpart with a single prompt, suggesting that the prompt pool encodes task-specific knowledge well.
Source Prompt Cluster. We remove the step of grouping source prompts into different clusters and directly lookup source prompts based on queries (see in Table 5 row 2). The decrease in performance demonstrates that when tasks are diverse, clustering task prompts can identify the similarity between source tasks, thus promoting effective knowledge transfer.
Multi-Key Memory Network. We remove the
Figure 2: Similarity analysis of 14 datasets within our six generation tasks.
learnable key vector associated with prompts and directly transfer the mean of the source prompts to the target task. From Table 5 (row 4), we can see this results in a significant drop, demonstrating the importance of introducing learnable keys to dynamically select prompts through query-key matching.
Instance-level Query. The instance-level query is used in adaptive attention mechanism. When we remove it (Table 5 row 3), we only use the task-level query to select source prompts. The declined performance demonstrates that incorporating the instance-level features can indeed help to transfer the most helpful knowledge to the specific instances in target tasks.
5.5 Task Similarity Analysis
Figure 2 shows a clustered heatmap of cosine similarities between the source prompts of the 14 public datasets within our six text generation tasks using the position-agnostic Euclidean distances defined by Eq. 1. We can clearly observe that our learned 14 source prompts are roughly grouped into three clusters. Similar tasks and datasets are grouped together into clusters in this heatmap, and these
clusters capture many intuitive task relationships. Specifically, these three clusters mainly focus on compression, transduction, and creation tasks respectively. For example, story generation (WritingPrompts) and dialog (PersonaChat) are grouped together into the third cluster. This observation further verifies our conclusion that text generation tasks can help each other within our approach by learning task-specific prompts and then transferring them to the target task. The results also suggest that our method can serve as an effective means of predicting task transferability.
This paper presented a prompt-based transfer learning approach for text generation. We learn a set of prompts from a number of representative source generation tasks and then transfer these prompts as target prompts to perform the target generation tasks. In our model, we design an adaptive attention mechanism considering both task- and instance-level information to construct the target prompts. Experiments in fully-supervised and few-shot settings demonstrate the effectiveness of our prompt-based transfer learning model. In future work, we will consider incorporating more kinds of text generation tasks.
7 Ethical Concerns
Text generation techniques has been applied to a wide range of meaningful applications for society, such as game narrative generation, news report generation, and weather report generation. However, this technique may be potentially utilized for harmful applications. Our work improves the quality of generated text compared with traditional methods. Thus, the high-quality text generated by our work makes it difficult to distinguish synthetic text from human-written text, such as fake news and stories. Here we are primarily concerned with two potential ethical issues: the possibility of deliberate misuse of our methodology and the issue of bias.
First, it is somewhat challenging to anticipate the harmful usages of our method since they often involve repurposing our model in a totally different setting or for an unexpected purpose than we planned. To alleviate this issue, we can ask for the assistance of classic security risk assessment frameworks such as detecting threats. Second, biases in training data may cause our model to generate stereotyped or prejudiced texts. This
is a worry since the model bias has the potential to hurt some persons in relevant groups in unforeseen ways. To avoid prejudice, it may be useful to develop a common vocabulary that connects the normative, technological, and empirical difficulties of bias reduction for our model.
This work was partially supported by Beijing Natural Science Foundation under Grant No. 4222027, National Natural Science Foundation of China under Grant No. 61872369, Beijing Outstanding Young Scientist Program under Grant No. BJWZYJH012019100020098, and the Outstanding Innovative Talents Cultivation Funded Programs 2021 of Renmin University of China. Xin Zhao is the corresponding author.
Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Stefan Lee, Peter Anderson, Irfan Essa, Devi Parikh, Dhruv Batra, Anoop Cherian, Tim K. Marks, and Chiori Hori. 2019. Audio-visual scene-aware dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Ultes Stefan, Ramadan Osman, and Milica Gašić. 2018. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Zhiyu Chen, Harini Eavani, Wenhu Chen, Yinyin Liu, and William Yang Wang. 2020. Few-shot NLG with pre-trained language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 183-190. Association for Computational Linguistics.
Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. 2021. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7580-7605. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Chris H. Q. Ding, Xiaofeng He, Hongyuan Zha, Ming Gu, and Horst D. Simon. 2001. A min-max cut algorithm for graph partitioning and data clustering. In Proceedings of the 2001 IEEE International Conference on Data Mining, 29 November - 2 December 2001, San Jose, California, USA, pages 107-114. IEEE Computer Society.
Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 1074-1084. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann N. Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 889-898. Association for Computational Linguistics.
Dan Friedman, Ben Dodge, and Danqi Chen. 2021. Single-dataset experts for multi-dataset question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November; 2021, pages 6128-6137. Association for Computational Linguistics.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In ICML.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019. Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations. In Proc. Interspeech 2019, pages 1891-1895.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018,
New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 708-719. Association for Computational Linguistics.
Minbyul Jeong, Mujeen Sung, Gangwoo Kim, Donghyeon Kim, Wonjin Yoon, Jaehyo Yoo, and Jaewoo Kang. 2020. Transferability of natural language inference to biomedical question answering. In CLEF.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In *NAACL HLT* 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 110-119. The Association for Computational Linguistics.
Junyi Li, Tianyi Tang, Gaole He, Jinhao Jiang, Xiaoxuan Hu, Puzhao Xie, Zhipeng Chen, Zhuohao Yu, Wayne Xin Zhao, and Ji-Rong Wen. 2021a. Textbox: A unified, modularized, and extensible framework for text generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 30-39.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2022. A survey of pretrained language models based text generation. CoRR, abs/2201.05273.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan, and Ji-Rong Wen. 2021b. Few-shot knowledge graph-to-text generation with pretrained language models. In *Findings of the Association for Computational Linguistics: ACL/IJCNLP* 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of *Findings of ACL*, pages 1558-1568. Association for Computational Linguistics.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2021c. Pretrained language model for text generation: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event/Montreal, Canada, 19-27 August 2021, pages 4492-4499. ijcai.org.
Junyi Li, Wayne Xin Zhao, Ji-Rong Wen, and Yang Song. 2019. Generating long and informative reviews with aspect-aware coarse-to-fine decoding. In