SlowGuess/NAACL_2022_OCR huggingface dataset:
Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 1969-1979. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4582-4597. Association for Computational Linguistics.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 - December 1, 2017 - Volume 1: Long Papers, pages 986-995. Asian Federation of Natural Language Processing.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Ruofei Zhang, Winnie Wu, Ming Zhou, and Nan Duan. 2021a. GLGE: A new general language generation evaluation benchmark. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 408-420. Association for Computational Linguistics.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In *NAACL-HLT*.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021b. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. CoRR, abs/2107.13586.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021c. Gpt understands, too. arXiv preprint arXiv:2103.10385.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1797-1807. Association for Computational Linguistics.
Kartikey Pant, Tanvi Dadu, and Radhika Mamidi. 2020. Towards detection of subjective bias using contextualized word embeddings. In *Companion of The 2020 Web Conference* 2020, Taipei, Taiwan, April 20-24, 2020, pages 75-76. ACM / IW3C2.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392. The Association for Computational Linguistics.
Tven Le Scao and Alexander M. Rush. 2021. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2627-2636. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1073-1083. Association for Computational Linguistics.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4222-4235. Association for Computational Linguistics.
YuSheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. 2021. On transferability of prompt tuning for natural language understanding. CoRR, abs/2111.06719.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information
Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. 2021. Spot: Better frozen model adaptation through soft prompt transfer. CoRR, abs/2110.07904.
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 4144-4150. ijcai.org.
Georg Wiese, Dirk Weissenborn, and Mariana L. Neves. 2017. Neural domain adaptation for biomedical question answering. In CoNLL.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2204-2213. Association for Computational Linguistics.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. 2021. A comprehensive survey on transfer learning. Proc. IEEE, 109(1):43-76.
Xu Zou, Da Yin, Qingyang Zhong, Hongxia Yang, Zhilin Yang, and Jie Tang. 2021. Controllable generation from pre-trained language models via inverse prompting. arXiv preprint arXiv:2103.10685.
We provide some experiment-related information as supplementary materials. The appendix is organized into three sections:
- Statistics of each dataset are presented in Appendix A;
- Training settings of baselines and our model PTG are presented in Appendix B.
The detailed information of the dataset for each task is listed in Table 6, including summarization (CNN/Daily Mail, XSum, MSNews, Multi-News and NEWSROOM), question generation (SQuAD), style transfer (Wiki Neutrality), text paraphrase (Quora), dialog (PersonaChat, TopicalChat, Daily-Dialog, DSTC7-AVSD and MultiWOZ) and story generation (WritingPrompts). These datasets are utilized under MIT license.
B Configuration of Models
The learning rate of other baselines is set to  $3 \times 10^{-5}$ , which is the same as our backbone BART. The other settings of baselines and our model are set the same for fair comparison. And we do not utilize special tricks such as label smoothing, warm-up learning rate and length penalty. We apply the Adam optimizer and set  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.98$ ,  $\epsilon = 1 \times 10^{-6}$ . We set the accumulated batch size of each model to 96 using accumulated gradients. Furthermore, we use the model with the best performance on validation set for generation. During inference, we apply the beam search method with a beam size of 5 and a no repeat ngram size of 3. We train our models using NVIDIA A100 GPUs and PyTorch 1.9.0 upon Ubuntu 20.04.2 LTS.
Dataset	#Train	#Valid	#Test	#Input	#Output
CNN/Daily Mail	287113	13368	11490	790.2	58.4
Xsum	204017	11327	11333	358.8	21.1
MSNews	136082	7496	7562	311.6	24.8
Multi-News	44972	5622	5622	2291.9	263.1
NEWSROOM	995040	108837	108862	658.5	26.7
SQuAD	75722	10570	11877	148.3	11.6
Wiki Neutrality	145197	18149	18150	29.1	27.3
Quora	119410	14927	14926	9.8	9.9
PersonaChat	122499	14602	14056	122.1	11.9
TopicalChat	179750	11142	11221	216.6	20.3
DailyDialog	76052	7069	6740	68.4	13.9
DSTC7-AVSD	145521	33953	11780	90.7	9.5
MultiWOZ	105115	13748	13744	110.7	13.2
WritingPrompts	67765	3952	3784	25.7	232.3
Table 6: Statistics of our datasets after preprocessing. #Train, #Valid and #Test denote the number of examples in training, valid and test datasets, respectively. #Input and #Output denote the average number of tokens in the input text and output text.