SlowGuess/ACL_2020_OCR huggingface dataset:
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. arXiv:1910.13461.
Chunyuan Li, Xiang Gao, Yuan Li, Xiujun Li, Baolin Peng, Yizhe Zhang, and Jianfeng Gao. 2020. Optimus: Organizing sentences via pre-trained modeling of a latent space. arXiv:2004.04092.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Jiwei Li, Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1106-1115, Beijing, China. Association for Computational Linguistics.
Dayiheng Liu, Jie Fu, Pengfei Liu, and Jiancheng Lv. 2019. TIGS: An inference algorithm for text infilling with gradient search. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4146-4156, Florence, Italy. Association for Computational Linguistics.
Sourab Mangrulkar, Suhani Shrivastava, Veena Thenkanidiyoor, and Dileep Aroor Dinesh. 2018. A context-aware convolutional natural language generation model for dialogue systems. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 191-200, Melbourne, Australia. Association for Computational Linguistics.
Tomas Mikolov and Geoffrey Zweig. 2012. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 234-239.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. https://d4mucfpkseywv.cloudfront.net/better-language-models/language_models—are_unsupervisedMULTITASK_learners.pdf.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv:1910.10683.
Juan Ramos. 2003. Using TF-IDF to determine word relevance in document queries. https://www.cs.rutgers.edu/~mlittman/courses/ml03/iCML03/papers/ramos.pdf.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Dinghan Shen, Asli Celikyilmaz, Yizhe Zhang, Liquin Chen, Xin Wang, Jianfeng Gao, and Lawrence Carin. 2019. Towards generating long and coherent text with multi-level latent variable models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2079-2089, Florence, Italy. Association for Computational Linguistics.
Tianxiao Shen, Victor Quach, Regina Barzilay, and Tommi Jaakkola. 2020. Blank language models. arXiv:2002.03079.
Yong-Siang Shih, Wei-Cheng Chang, and Yiming Yang. 2019. XL-Editor: Post-editing sentences with XLNet. arXiv:1910.10479.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5926-5936, Long Beach, California, USA. PMLR.
Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and JianYun Nie. 2015a. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM '15, pages 553-562, New York, NY, USA. ACM.
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015b. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 196-205, Denver, Colorado. Association for Computational Linguistics.
Jian Tang, Yifan Yang, Sam Carton, Ming Zhang, and Qiaozhu Mei. 2016. Context-aware natural language generation with recurrent neural networks. arXiv:1611.09900.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pages 1096-1103, New York, NY, USA. Association for Computing Machinery.
Oriol Vinyals and Quoc V. Le. 2015. A neural conversational model. arXiv:1506.05869.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019. Context-aware monolingual repair for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 877-886, Hong Kong, China. Association for Computational Linguistics.
Hongning Wang, Yue Lu, and Chengxiang Zhai. 2010. Latent aspect rating analysis on review text data: A rating regression approach. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '10, pages 783-792, New York, NY, USA. Association for Computing Machinery.
Tian Wang and Kyunghyun Cho. 2016. Larger-context language modelling with recurrent neural network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1319-1329, Berlin, Germany. Association for Computational Linguistics.
Wenlin Wang, Zhe Gan, Wenqi Wang, Dinghan Shen, Jiaji Huang, Wei Ping, Sanjeev Satheesh, and Lawrence Carin. 2018. *Topic compositional neural language model*. In *Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics*, volume 84 of *Proceedings of Machine Learning Research*, pages 356–365, Playa Blanca, Lanzarote, Canary Islands. PMLR.
Tsung-Hsien Wen, Milica Gašić, Dongho Kim, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Stochastic language generation in dialogue using recurrent neural networks with convolutional sentence reranking. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 275-284, Prague,
Czech Republic. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 5753-5763. Curran Associates, Inc.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489, San Diego, California. Association for Computational Linguistics.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. 2018. Generating informative and diverse conversational responses via adversarial information maximization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 1810-1820. Curran Associates, Inc.
Yizhe Zhang, Xiang Gao, Sungjin Lee, Chris Brockett, Michel Galley, Jianfeng Gao, and Bill Dolan. 2019. Consistent dialogue generation with self-supervised feature learning. arXiv:1903.05759.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. *DiaoGPT: Large-scale generative pre-training for conversational response generation*. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations*. Association for Computational Linguistics.
Wanrong Zhu, Zhiting Hu, and Eric Xing. 2019. Text infilling. arXiv:1901.00158.
A Additional Generated Examples
Tables 6, 7 show some additional examples generated by our model (without keywords) on the TripAdvisor and Recipe datasets, respectively. The results are semantically informative and can fit the surrounding context reasonably well. Table 8 provides additional examples to Table 5. Our model can incorporate keywords into the generated sentence in a smart way, adapting to the context.
Table 6: Generated examples by our model on the TripAdvisor dataset
example 1	example 2
preceding context	Roll up rectangles width-wise and pinch ends to seal. Bake for 12 minutes or until the tops begin to brown.	Drizzle each potato cup with 1 teaspoon browned butter. Cover muf-fin tin tightly with aluminium foil and place in oven.
following context	Best when served warm. For added flavor, serve with strawberry jelly.	Remove from oven and turn broiler on high. Sprinkle potato rounds evenly with remaining parmesan cheese.
ground truth	Let cool on baking sheet.	Bake for 25 minutes.
INSET	Cool on wire rack and remove.	Bake for 20 minutes or until potatoes are tender.
example 3	example 4
preceding context	Preheat oven to 425 degrees Fahrenheit. Line a baking sheet with a SILPAT mat.	Heat the oil in a pan at medium. Add the mushrooms and saute until tender, about 7-10 minutes.
following context	With a pastry cutter, cut in the coconut oil and the butter. Make a well and add in the milk 1/2 cup at a time, stirring gently with a wooden spoon.	Add the reserved water and simmer at medium-high until reduced by half, about 10 minutes. Meanwhile cook the pasta as directed on the package.
ground truth	In a bowl, mix the flour, baking powder, baking soda and sea salt.	Add shallots, garlic, thyme, salt and pepper and saute for 2 minutes.
INSET	In a medium bowl, mix together the flour, baking powder, sugar, salt and cinnamon.	Add the garlic and sautee until fragrant, about 2 minutes.
example 5	example 6
preceding context	After another 15 minutes or so the mixture should thicken up. The mixture will continue to thicken as it cools.	Bake the graham cracker crust for 10 minutes. Remove from oven and allow to cool to room temperature.
following context	Sterilize your jars and lids and while still hot fill with the jam leaving about a 1/2 inch headspace. Place lids onto the jars and boil in a water bath with jars covered by 3 inches of water for 10 minutes.	Stir in the lime zest and lime juice. Stir until mixture is smooth and begins to slightly thicken.
ground truth	Remove from the heat and stir in your amaretto.	Meanwhile, combine the egg yolks and condensed milk in a medium bowl.
INSET	Carefully remove the jars from hot water and keep going until a thick sauce is formed.	In a medium bowl, combine the cream cheese and powdered sugar, stirring until smooth.
Table 7: Generated examples by our model on the Recipe dataset
preceding context	Also has a safe. The hotel is in a good location, beside the City Centre and there are a nice selection of shops within the Monte Carlo. Service was very good but avoid the concierge in the morning when people are booking tours, the queues are long.
following context	No wi-fi in the room which is a bit annoying but they have it in the foodcourt by Starbucks and McDs. Also we were disappointed to see the $15/night resort fee was charged to our credit card after our stay. I don’t recall them mentioning this at check-in.
human oracle	CVs is next door and it’s 24/7 so you can buy snacks and anything else you fancy.
+ (breakfast, cereal)	Breakfast is included with cereal, muffins and breads.
+ (food, expensive)	Prices are expensive but food in the hotel is very cheap.
Table 8: Examples generated by our model in the same context but with different keywords. “+ (…)” is keywords.