SlowGuess/NAACL_2021_OCR huggingface dataset:
the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Shrimai Prabhumoye, Alan W Black, and Ruslan Salakhutdinov. 2020. Exploring controllable text generation techniques. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1-14, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Shrimai Prabhumoye, Chris Quirk, and Michel Galley. 2019. Towards content transfer through grounded text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2622-2632, Minneapolis, Minnesota. Association for Computational Linguistics.
Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Wintner. 2017. Personalized machine translation: Preserving original author traits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1074-1084, Valencia, Spain. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8689-8696.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266.
Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. 2013. Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics, 1:25-36.
Ehud Reiter and Robert Dale. 2000. Building Natural Language Generation Systems. Cambridge University Press, USA.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799.
Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston. 2018. Engaging image chat: Modeling personality in grounded dialogue. CoRR, abs/1811.00945.
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. From neural sentence summarization to headline generation: a coarse-to-fine approach. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 4109-4115.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In Proc. ICML Deep Learning Workshop.
Wei Wei, Quoc Le, Andrew Dai, and Jia Li. 2018. AirDialogue: An environment for goal-oriented dialogue research. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3844-3854, Brussels, Belgium. Association for Computational Linguistics.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In International Conference on Learning Representations.
Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, PeiHao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711-1721.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7378-7385.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018a. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213. Association for Computational Linguistics.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018b. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, and Rui Yan. 2020a. Low-resource knowledge-grounded dialogue generation. In International Conference on Learning Representations.
Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020b. Knowledge-grounded dialogue generation with pre-trained language models.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. A dataset for document grounded conversations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 708-713, Brussels, Belgium. Association for Computational Linguistics.
We use the transformer toolkit (Wolf et al., 2019) to implement the baseline and both CoDR and DoHA models. Both DoHA (§3.3) and CoDR (§3.2) have the same dimensions and architecture of the BART model (Lewis et al., 2020). For the DoHA model, we initialize CrossAttention_Doc with same pre-trained weights of CrossAttention. Hence, the layer size of the CrossAttention_Doc multi-head is the same as the layer size of CrossAttention multi-head in BART. Table 6 shows the maximum sequence lengths used for all the three datasets for both source and target. The data statistics are shown in Table 7. We experimented with two learning rates 5e-5 and 2e-5. We report numbers for the best trained models in each case. Specifically, we report numbers with 5e-5 learning rate for DoHA and CoDR models on the CMU_DOG dataset and the BART baseline for all the three datasets. For Wikipedia Update Generation and Wizard of Wikipedia dataset, we choose the DoHA and CoDR models trained with 2e-5 learning rate. We maintain a common environment (in terms of GPU, operating system, Pytorch version and transformer version) to run all the experiments. We train all the models for 25 epochs.
Zhao et al. (2020a) numbers are directly taken from the paper as the pre-trained model or the generated outputs are not available. We use the same data splits and evaluation toolkits for comparable setting. Hence, Rouge-L and Meteor values are not available for this model. The BLEU, Meteor and Rouge-L numbers are different from (Prabhumoye et al., 2019) due to the usage of different tool-kits in measuring their values.
Dataset	Source Len	Target Len
Wikipedia Update Generation	1024	128
CMU_DOG dataset	512	128
Wizard of Wikipedia	900	40
Convergence: Figures 2 and 3 show the convergence of the baseline BART model in comparison
Table 6: Sequence Lengths
Dataset	Train	Dev	Test
Wikipedia Update Generation	580.0k	6.0k	50.0k
CMU_DOG	72.9k	4.8k	13.2k
Wizard of Wikipedia	166.7k	17.7k	8.7k
Table 7: Dataset Statistics
Figure 2: Convergence of CMU_DOG development data on the automated metric.
with the CoDR and DoHA models on the development sets of CMU_DOG and Wizard of Wikipedia respectively. We observe that at same number of updates, DoHA and CoDR perform better than BART. This is especially relevant for big datasets like Wikipedia Update Generation which take 15 days to complete 25 epochs.
Figure 3: Convergence of CMU_DOG development data on the automated metric.
We hand selected a few news articles which contained inaccurate information. We provided relevant Wikipedia context and generated an update for the context using the trained model on Wikipedia Update generation dataset. The generations are shown in Table 8. Additionally, we provided relevant dialogue context to the model trained on Wizard of Wikipedia dataset (CMU_DoG was not chosen because it is trained only on movie conversations) and the responses generated are shown in Table 9. Note that this shows the potential misuse of the task design.
5The results are subject to changes in the codebase of the toolkit. Note that we will release our code and trained models to ensure reproducibility of results. 6We try to closely follow the processing of the original papers for each of the three datasets.
Document	Context	Generation
https://www.bradenton.com/latest-news/article237480339.html	Climate change includes both the global warming driven by human emissions of greenhouse gases, and the resulting large-scale shifts in weather patterns. Though there have been previous periods of climatic change, since the mid-20th century the rate of human impact on Earth's climate system and the global scale of that impact have been unprecedented.	Though most scientists agree that a “global” climate change is occurring, climate change remains a contentious topic and many scientists, climate scientists, and even climate skeptics believe that it is a hoax.
https://tinyurl.com/3r5runb	Climate change includes both the global warming driven by human emissions of greenhouse gases, and the resulting large-scale shifts in weather patterns. Though there have been previous periods of climatic change, since the mid-20th century the rate of human impact on Earth's climate system and the global scale of that impact have been unprecedented.	Some scientists dismiss the existence of climate change.
https://tinyurl.com/y634czdp	Climate change includes both the global warming driven by human emissions of greenhouse gases, and the resulting large-scale shifts in weather patterns. Though there have been previous periods of climatic change, since the mid-20th century the rate of human impact on Earth's climate system and the global scale of that impact have been unprecedented.	The Scientific opinion on climate change is that planetwide warming has occurred at least three million years ago.
Table 8: Outputs generated by DoHA model for a few hand selected news articles. The DoHA model is trained on Wikipedia Update Generation dataset.
Document	Context	Generation
https://tinyurl.com/yxedlapz	What do you think of the rising sea levels?	I think it's crazy that people believe that the oceans will rise and yet people are investing $15 million to live on islands...
https://tinyurl.com/y3u6fhpo	Do you think the Affordable Care Act is helping the general public?	It has some great benefits. It has actually resulted in millions of Americans losing their health insurance plan. That is not something former VP Biden claimed.
https://tinyurl.com/3r5runb	What do you think of the rising sea levels?	I think the rise of sea levels is a big money making hoax. Why do you think the world is heating up? Any experts?
Table 9: Outputs generated by DoHA model for a few hand selected news articles. The DoHA model is trained on Wizard of Wikipedia dataset.