SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 4: Response generation results ROUGE-1 (R1), ROUGE-2 (R2), ROUGE-L (RL) and knowledge selection accuracy (Acc%) on HollE. † results taken from (Kim et al., 2020). Other results with citations are taken from their respective papers.
Model	Preferred	Approp.	Know.	Engaging
Ours	69%	3.54	3.42	3.32
Ranking	56%	3.47	3.39	3.28
MIKe	34.5%	2.88	3.02	2.82
Table 5: Human evaluation results. "Preferred" includes cases where annotators choose multiple systems as the best. 'Approp.' is short for Appropriate, 'Know.' is short for Knowledgeable.
Model	R1	R2	RL
GPT2 + Ranking	19.95	4.70	16.33
GPT2 + Graph Paths	19.83	4.89	16.37
GPT2 + Graph Selection	20.43	5.31	16.97
GPT2 + Gold knowledge	30.53	11.94	25.61
lection task and our edge-aware graph attention network is more effective in utilizing the graph structure compared to simply enumerating graph paths. In particular, when the graph is used, there is a large improvement in MRR when multiple gold-standard references are provided, showing that in cases where the top 1 result does not match the reference, we are able to rank the gold-standard knowledge at a high position.
For the end-to-end evaluation in Table 4, our model stands favorably among previous published results, with improvements in both knowledge selection accuracy and response quality.
We report human evaluation results in Table 5. Our system scores the best in all aspects and is voted by annotators as the most preferred response in the majority of the cases.
Table 6: End-to-end results (in %) on the unseen split of WoW using first turn retrieved passages as background knowledge.
Model	Acc(%)	MAP	Concept MAP	Concept MRR
Full	37.7	0.513	0.420	0.495
Sent. graph	35.6	0.494	-	-
Coref. graph	37.0	0.510	0.420	0.421
Homog. graph	37.3	0.516	0.409	0.398
Sent. loss	36.0	0.500	0.063	0.151
Table 7: Model ablations for knowledge selection on Holl-E using single reference.
On the WoW dataset (Table 3 and Table 6), the basic ranking model performs slightly better on the seen split and our graph-based knowledge selection method shows benefits for generalizing to unseen topics.
Model Ablations. We investigate whether our design of the document semantic graph is effective by exploring different variants of the document graph, including: (1) sentence graph with only sentence nodes and source nodes, (2) coreference graph that removes all AMR role edges, and (3) homogeneous graph that treats all edges and nodes as the same type. The results are presented in Table 7. In particular, the sentence graph does not make use of AMR parsing nor coreference resolution, so it only reflects the document structure. This makes it the least effective in knowledge selection and unable to perform concept selection at all. The coreference graph does not perform as well as the full graph, but largely closes the gap. This suggests that entity recognition and coreference resolution are essential to the effectiveness of the document graph. When using the homogeneous graph, our
edge-aware graph attention network falls back to a regular graph attention network. We can see that without edge and node semantics, both sentence and concept selection are negatively impacted.
An important characteristic of our model is that it is trained to perform joint sentence selection and concept selection through a multi-task objective. We compare our full model with a variant, which is only trained with sentence-level supervision signal. Our results show that adding the concept selection loss not only enables concept-level knowledge selection, but also improves sentence-level knowledge selection.
Case Studies. We present some examples of the generated responses on HollE in Table 8. In the first example, the system started out with a comment on the character "Morpheus", the user agreed, and then shifted the topic towards a general comment on the movie. Both our model and the ranking model are able to follow the user's topic and make comments on the movie while the MIKe model continues the previously initiated topic. In the second example, we see that our model and the ranking model both capture the "viral fame" keyword in the user's response, but our model is able to produce a more appropriate response instead of directly copying the plot. In the last example, the ranking model repeats what the user said while our model and MIKe pick knowledge that is relevant to the rating of the movie. In this case, our model produces a more engaging response.
Figure 5 visualizes an example from the WoW dataset about the topic "Football". In this conversation, although the knowledge selected by our model is not the same as the ground truth, it is relevant to the user's question of "where and how the game (of football) got started". The ground truth, on the other hand, follows up on the wizard's own initiated topic of "college football".
Discussions on Limitations. (1) Concept selection. Current datasets were annotated with sentence selection in mind and only provided sentence level references. This makes it hard to directly demonstrate the utility of concept selection. (2) Better utilization of history. We have used the dialog history in a primitive way by concatenating the latest turns with the candidate knowledge. This ignores earlier turns, and leads to cases of repetition of history, or contradiction of persona. (3) Limitation of preprocessing tools. Our document semantic
graphs rely on AMR parsing, which might not be available for other languages, or not be of high quality.
Knowledge Selection for Dialog. Knowledge selection can be tightly coupled with the response generator (Ghazvininejad et al., 2018) or performed separately prior to response generation. Some approaches adapted question answering models (Moghe et al., 2018; Qin et al., 2019; Wu et al., 2021) or summarization models (Meng et al., 2020a) for knowledge selection. With a pool of knowledge candidates, knowledge selection has been commonly set up as a sentence classification or ranking problem (Dinan et al., 2019; Lian et al., 2019; Kim et al., 2020; Chen et al., 2020; Meng et al., 2020b; Zhao et al., 2020). Some work has modeled the underlying knowledge as a latent variable (Lian et al., 2019; Kim et al., 2020; Chen et al., 2020). Others have explored modeling the knowledge transition over dialog turns to improve selection accuracy (Kim et al., 2020; Meng et al., 2020b; Zheng et al., 2020; Zhan et al., 2021). In comparison, we model knowledge selection as a node selection task on the document semantic graph.
Graph-based Knowledge Sources. Knowledge graphs are popular choices for integrating knowledge into dialog systems (Liu et al., 2018; Moon et al., 2019; Xu et al., 2020; Jung et al., 2020; Zhou et al., 2020). However, their applicability is limited by the coverage of both entities and relations. For example in (Moon et al., 2019), for books and movies, the knowledge base only contains metadata such as title and genre, making it impossible to conduct conversation about the actual content. The closest work to ours is AKGCM (Liu et al., 2019b), which starts from an existing general knowledge graph and then augments the knowledge graph with unstructured text by performing entity linking on the sentences. In comparison, our document semantic graph is created from knowledge documents and during knowledge selection we select both sentences and concept nodes.
Application of Document Graphs. Document-level AMR graphs have been used for summarization (Liu et al., 2015; Dohare et al., 2018; Hardy and Vlachos, 2018; Lee et al., 2021) and document generation (Fung et al., 2021). Graphs constructed using OpenIE (Banko et al., 2008) have been ap
Topic	The Matrix	Bad Ass	Rocky V
History	S: My favorite character was Mor-
pheus, because he truly believed in
what he was doing. U: Yeah. It was
a story of a reluctant Christ-like per-
son ... The Matrix ... should be
the benchmark for all sci-fi films to
come.	S: What was your favorite scene in
this movie? U: I liked the one in
which Frank fights the skinheads. S:
Yes. They messed with the wrong
senior citizen! U: No doubt! Gotta
love the viral fame angle they had
as well.	U: My favorite character was
Adrian ... S: She sure did protect her
family in this great movie. U: I have
watched the rocky series probably
100 times each ... I would feel of-
fended that Rocky V is rated lower
than Rockys III and IV.
Ours	And so much interpretation in these
movies. Have the directors ever
come out and just explained it?
That would make things a lot easier.	Oh yeah, it goes viral & he is pro-
claimed by the public as “Bad Ass.”
Right. It wasn’t all that bad when
Frank becomes an instant celebrity,
and he and Mrs Vega (Tonita Ca-
tro) are interviewed at the morning
programme LA Mornings by a jour-
nalist (Sam Rubin).	Yeah. This movie is a 10, but I feel
I need to give it that bit extra due to
the very low rating this movie has
polled on this site.
Ranking	Yeah. With The Matrix, both ele-
ments are perfectly en sync.	My favorite character is Adrian too.
MIKe	I agree , and I loved the scene
where he throws Neo into the sub-
way tracks then drops down there.	Danny Trejo’s Grand Torino.	A wonderful movie about father and
son.
Table 8: Generated responses from our system and baselines on HollE. S stands for system turn and U for user turn.
Figure 5: An example of selected knowledge and generated responses from our model on WoW.
plied to long-form question answering and multi-document summarization (Fan et al., 2019).
In this paper, we introduce document semantic graphs for knowledge selection. Compared to existing document-based knowledge selection methods that typically treat sentences independently, our automatically-constructed document semantic graphs explicitly represent the semantic connections between sentences while preserving sentence-level information. Our experiments demonstrate that our semantic graph-based approach shows advantages over various sentence selection baselines in both the knowledge selection task and the end-to-end response generation task.
The paper focuses on improving the knowledge selection component for dialog systems.
Intended use. The intended use of this grounded dialog system is to perform chit-chat with the user on topics such as books and movies. We also hope that our released system can help research in knowledge selection.
Bias. Our model is developed with the use of large pretrained language models such as RoBERTa (Liu et al., 2019a) and GPT2 (Radford et al., 2019), both of which are trained on large scale web data that is known to contain biased or discriminatory content. The datasets that we train on also include subjective knowledge (comments on movies) that may express the bias of the writers.
Misuse potential. Although our system is knowledge-grounded, the output from our system should not be treated as factual knowledge. It should also not be considered as advice for any critical decision-making.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In LAW@ACL.
Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2008. Open information extraction from the web. In CACM.
Xiuyi Chen, Fandong Meng, Peng Li, Feilong Chen, Shuang Xu, Bo Xu, and Jie Zhou. 2020. Bridging the gap between prior and posterior knowledge selection for knowledge-grounded dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3426-3437, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and J. Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. ICLR.
Shibhansh Dohare, Vivek Gupta, and Harish Karnick. 2018. Unsupervised semantic abstractive summarization. In ACL.
Angela Fan, Claire Gardent, C. Braud, and Antoine Bordes. 2019. Using local knowledge graph construction to scale seq2seq models to multi-document inputs. In EMNLP/IJCNLP.
Ramón Fernandez Astudillo, Miguel Ballesteros, Tahira Naseem, Austin Blodgett, and Radu Florian. 2020. Transition-based parsing with stack-transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1001-1007, Online. Association for Computational Linguistics.
Yi Fung, Christopher Thomas, Revanth Gangi Reddy, Sandeep Polisetty, Heng Ji, Shih-Fu Chang, Kathleen McKeown, Mohit Bansal, and Avi Sil. 2021.
InfoSurgeon: Cross-media fine-grained information consistency checking for fake news detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1683-1698, Online. Association for Computational Linguistics.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, W. Dolan, Jianfeng Gao, Wen tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In AAAI.
Karthik Gopalakrishnan, Behnam Hedayatnia, Q. Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and D. Hakkani-Tur. 2019. Topical-chat: Towards knowledge-grounded open-domain conversations. In INTERSPEECH.
Hardy Hardy and Andreas Vlachos. 2018. Guided neural language generation for abstractive summarization using abstract meaning representation. In EMNLP.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. ArXiv, ICLR.
Jaehun Jung, Bokyung Son, and Sungwon Lyu. 2020. AttnIO: Knowledge Graph Exploration with In-and-Out Attention Flow for Knowledge-Grounded Dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3484-3497, Online. Association for Computational Linguistics.
Byeongchang Kim, Jaewoo Ahn, and G. Kim. 2020. Sequential latent knowledge selection for knowledge-grounded dialogue. *ICLR*, abs/2002.07510.
Fei-Tzin Lee, Christopher Kedzie, Nakul Verma, and Kathleen McKeown. 2021. An analysis of document graph construction methods for amr summarization. *ArXiv*, abs/2111.13993.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Soren Auer, et al. 2015. Dbpedia-a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167-195.
Rongzhong Lian, Min Xie, Fan Wang, Jinhua Peng, and Hua Wu. 2019. Learning to select knowledge for response generation in dialog systems. *IJCAI*.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Fei Liu, Jeffrey Flanigan, Sam Thomson, Norman M. Sadeh, and Noah A. Smith. 2015. Toward abstractive summarization using semantic representations. In NAACL.
Shuman Liu, Hongshen Chen, Zhaochun Ren, Yang Feng, Qun Liu, and Dawei Yin. 2018. Knowledge diffusion for neural dialogue generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1498, Melbourne, Australia. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Zhibin Liu, Zheng-Yu Niu, Hua Wu, and Haifeng Wang. 2019b. Knowledge aware conversation generation with explainable reasoning over augmented graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1782-1792, Hong Kong, China. Association for Computational Linguistics.
Chuan Meng, Pengjie Ren, Zhumin Chen, Christof Monz, Jun Ma, and M. de Rijke. 2020a. Refnet: A reference-aware network for background based conversation. In AAAI.
Chuan Meng, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tengxiao Xi, and M. de Rijke. 2021. Initiative-aware self-supervised learning for knowledge-grounded conversations. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval.
Chuan Meng, Pengjie Ren, Zhumin Chen, Weiwei Sun, Zhaochun Ren, Zhaopeng Tu, and M. de Rijke. 2020b. Dukenet: A dual knowledge interaction network for knowledge-grounded conversation. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.
Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018. Towards exploiting background knowledge for building conversation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2322-2332, Brussels, Belgium. Association for Computational Linguistics.
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019. OpenDialKG: Explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 845-854, Florence, Italy. Association for Computational Linguistics.
Prasanna Parthasarathi and Joelle Pineau. 2018. Extending neural generative conversational model using external knowledge sources. In Proceedings of
the 2018 Conference on Empirical Methods in Natural Language Processing, pages 690-695, Brussels, Belgium. Association for Computational Linguistics.
Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, and Jianfeng Gao. 2019. Conversing by reading: Contentful neural conversation with on-demand machine reading. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5427-5436, Florence, Italy. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Simon Razniewski, Fabian M. Suchanek, and Werner Nutt. 2016. But what do we actually know? In AKBC@NAACL-HLT.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric Michael Smith, Y.-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In EACL.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In EMNLP.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. AAAI.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio', and Yoshua Bengio. 2018. Graph attention networks. ICLR.
Denny Vrandecić and Markus Krötzsch. 2014. Wiki-data: a free collaborative knowledgebase. Commun. ACM, 57:78-85.
Haoyang Wen, Ying Lin, Tuan Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu, Alexander Dong, Zhenhailong Wang, Yi Fung, Piyush Mishra, Qing Lyu, Didac Suris, Brian Chen, Susan Windisch Brown, Martha Palmer, Chris Callison-Burch, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, and Heng Ji. 2021. RESIN: A dockerized schemaguided cross-document cross-lingual cross-media information extraction and event tracking system. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, pages 133-143, Online. Association for Computational Linguistics.
Zeqiu Wu, Bo-Ru Lu, Hannaneh Hajishirzi, and Mari Ostendorf. 2021. DIALKI: Knowledge identification in conversational systems through dialogue-document contextualization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1852-1863, Online and
Punta Cana, Dominican Republic. Association for Computational Linguistics.
J. Xu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, and Wanxiang Che. 2020. Knowledge graph grounded goal planning for open-domain conversation generation. In AAAI.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 535-546, Online. Association for Computational Linguistics.
Tom Young, E. Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Aug-mentation end-to-end dialog systems with commonsense knowledge. In AAI.
Haolan Zhan, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Yongjun Bao, and Yanyan Lan. 2021. Aug-mentation knowledge-grounded conversations with sequential knowledge transition. In NAACL.
Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020. Knowledge-grounded dialogue generation with pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3377-3390, Online. Association for Computational Linguistics.
Chujie Zheng, Yunbo Cao, Daxin Jiang, and Minlie Huang. 2020. Difference-aware knowledge selection for knowledge-grounded conversation generation. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 115-125, Online. Association for Computational Linguistics.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018a. Commonsense knowledge aware conversation generation with graph attention. In *IJCAI*.
Hao Zhou, Chujie Zheng, Kaili Huang, Minlie Huang, and Xiaoyan Zhu. 2020. KdConv: A Chinese multi-domain dialogue dataset towards multi-turn knowledge-driven conversation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7098-7108, Online. Association for Computational Linguistics.
Kangyan Zhou, Shrimai Prabhumoye, and A. Black. 2018b. A dataset for document grounded conversations. EMNLP.
Our experiments were run on a single V100 or RTX2080 GPU. We use gradient accumulation to reach an effective batch size of 16.
On average, the semantic graph construction takes 40s per document for the AMR parsing and 30s per document for coreference resolution. All documents were constructed before running knowledge selection experiments. Our knowledge selection model requires 10G of GPU memory and 6 hours to finish training. Our response generation model takes 1.5 hours to finish training.
We tuned our learning rate in the range of  $[3e - 6,1e - 5,3e - 5,5e - 5]$  and our batch size in the range of [4,8,16]. For the EGAT model, we experimented with hidden dimensions of [50, 100, 200] and layers from [2,3,4].
B Extra Case Studies
In Figure 6 we present an instance where the question from the user is quite open-ended and while our model's selection does not match the ground truth, it is still relevant to the dialog and can serve as the basis for an appropriate response.
In Figure 7 we show an example where knowledge selection performance does not directly translate to better dialog due to response generation errors. The selected knowledge from our model follows up on the "set routines" mentioned by the user but the response's stance is wrong. The baseline model selects a general statement about cheerleading as the relevant knowledge but the response is logically incorrect as the difficulty of cheerleading is not due to its geographical origin, but due to the moves.
C Human Eval Details
We show an example of the information provided to annotators in Figure 8. Annotators have access to the dialog history and the ground truth responses. System outputs are anonymized.
Figure 6: A case of mismatched but acceptable knowledge selection on WoW.