SlowGuess/NAACL_2022_OCR huggingface dataset:
We conducted an ablation study to investigate the contribution of major components of DialogoKG. The results on CamRest in Table 4 demonstrates that the ses2seq approach achieves the lowest scores, which represents the DialogoKG model without the embedding layers: entity embedding, triple embedding, and type embedding. Inclusion of the entity and triple embedding layers significantly improved model's performance in both BLEU and Entity F1 scores. The type embedding further improved DialogoKG's performance. The significant difference in results shows the effectiveness of the proposed embedding technique. Finally, we observed a remarkable improvement in DialogoKG's overall performance after the inclusion of knowledge attention mask. Question-aware weighted-graph computation used to construct knowledge attention mask, helped the model focus on the task at the inference time.
4.4 Effectiveness of Knowledge Embedding
The proposed graph embedding technique works best in combination with the knowledge attention mask. The graph embedding design allows DialoKG to handle disconnected graphs and
Table 4: Ablation study.
Top-k (entity)	Top-k (relation)	BLEU	MoverScore	Entity F1
3	5	10.8	65.3	48.2
3	7	11.0	65.4	48.9
5	5	16.9	68.0	62.1
5	7	17.4	68.1	62.5
7	5	19.3	70.4	64.4
7	7	20.0	70.6	65.9
All	All	15.9	67.2	59.0
Table 5: Effect of triple selection on the performance.
triples. This makes DialogKG suitable for large-scale graphs, where a cosine-similarity based triple selection may be used to fit the graph triples inside the model's input capacity. The entity and triple embedding layers allow the model to preserve the structural information of a particular triple even though triples from different parts of the input sequence are selected based on the top- $k$  entities and relations to construct the knowledge attention mask. Overall, the graph embedding technique improves the Entity F1 score by 5.4, 9.0, and 3.7 points on SMD, CamRest, and MWOZ, respectively. This indicates the effectiveness of the proposed embedding techniques for capturing graph triples.
4.5 Impact of Knowledge Attention Mask
To understand the effect of the knowledge-graph weighted attention mask, we experiment with the triple selection process described in DialogKG's approach. Table 5 shows the performance of DialogKG with selected top- $k$  entities and relations on the SMD dataset. We observe that DialogKG achieves the best performance on SMD when the top 7 entities and relations are chosen to construct the knowledge mask. Consider the question "Do you have any local coffee shops?" the ground truth is "There is Coupa, it's just 6 miles away but there is heavy traffic on our way". The ground truth contains traffic information in addition to the distance and name of the coffee shop. Selecting a high number of entities and relations increases the chance of generating such additional information related to the subject of the question. However, choosing too many entities harms the model since it is more likely to add irrelevant noise (see Table 5). Particularly, the entities that are irrelevant for answering the current user utterance act as a noise. For MWOZ, six entities and seven relations, and for CamRest, seven entities and five relations result in the best performance.
Knowledge Triples
Case 1
Subject	Relation	Object
Charlie Chan	food	chinese
Charlie Chan	address	Regent street city centre
Charlie Chan	pricerange	cheap
Charlie Chan	area	center
Yu Garden	address	529 Newmarket road fen ditton
Yu Garden	pricerange	expensive
Yu Garden	food	chinese
Rice House	address	88 mill road city centre
Rice House	food	chinese
Rice House	pricerange	cheap
Figure 6: Case study: comparison between ground truth and system-generated responses.
Figure 6 shows two cases from the MWOZ dataset given a subset of the knowledge graph. In Case 1, we observe that in answering the user question, DialogKG correctly picked Rice House that serves cheap and Chinese food. However, in this case, multiple correct answers exist, e.g. Charlie Chan also falls into the same category of restaurant. Despite generating the correct answer based on the given knowledge and the user question, DialogKG receives a low Entity F1 score since the generated response entity does not match the ground truth. In Case 2, where the baseline systems focus on imitating the ground truth, DialogKG generates a fluent and engaging response. Despite generating a meaningful and semantically similar sentence, it obtained a BLEU score of 0.0 because of the low overlap with the ground truth response. However, a high MoverScore in both cases indicates DialogKG's ability to generate a semantically similar response. Overall, we observe that DialogKG can generate human-like, engaging, and informative responses in a multi-turn dialogue setting.
5.1 Influence of Dialogue History
Dialogue history is particularly crucial since it gives the model the context for generating the response. In some cases where the entity information is missing in the current user utterance, the dialogue context provides the model with enough information to perform the inference and generate the correct response. For instance, for the question, What is the food type they serve?, the name of the restaurant is not given in the question, but the system can infer it from the dialogue history. However, from the experiments, we found that too much dialogue context may inject noisy and irrelevant information to answer the current question, in particular for knowledge-grounded responses
in MWOZ. To quantify this, we selected different numbers of dialogue turns as history for the model's input depending on the characteristics of the dataset and visualised the result in Figure 7.
Figure 7: DialogKG's performance on benchmark datasets for different number of dialogue contexts.
Task-Oriented Dialogue Systems. Recent dialogue systems mainly leverage Memory Pointer Networks (Sukhbaatar et al., 2015; Madotto et al., 2018; Wu et al., 2019), Copy mechanisms (Gu et al., 2016; Lin et al., 2020; Chaudhuri et al., 2019) and similarity-based knowledge distillation techniques (Wen et al., 2018; Raghu et al., 2021) for the knowledge selection and dialogue generation task. In this research direction, learning to generate template responses and fill in the slot is a common practice (Wu et al., 2019). Dialogue history and knowledge entities are stored in shared memory, facilitating these systems to apply copy mechanisms over the memory space. A multi-level memory architecture is proposed by (Gangi Reddy et al., 2019) that handles the dialogue history and knowledge entries separately.
Knowledge-Structure Aware Dialogue Generation. Recently, several knowledge-grounded dialogue systems have attempted to capture structural knowledge to improve the performance of dialogue generation. A sequence-to-sequence model is proposed by (Liu et al., 2018b) that employs global and local attention to understand structural information. Several studies found GCN to be effective for modeling graph-based data. Hence, they construct a graph from a document (Moghe et al., 2020) or the interaction between two speakers (Wu et al., 2021) for generating informative dialogues. In a different approach, an enhanced entity representation is proposed by (He et al., 2020c) by considering the entity information and the structural and relational information of the knowledge entries. In contrast to the previous works, our proposed approach represents the graph's structural information such as entities and triples through multiple embedding layers.
Language Model Based Dialogue Generation. Pre-training a language model with dialogue datasets (Zhang et al., 2020; Bao et al., 2020; Gu et al., 2021) and fine-tuning an already pre-trained model for various dialogue-related sub-tasks such as dialogue state tracking, action decision and response generation (Hosseini-Asl et al., 2020; Wu et al., 2020a; Galetzka et al., 2021) has received much attention in recent years. Recently, (Madotto et al., 2020) proposed a new method to embed the knowledge into the language model parameters. However, the authors noticed that the generated dialogues are sometimes noisy and requires high fine-tuning costs. Despite the success of language model-based approaches, integrating structured knowledge into the dialogue generation process remains a challenging task. Unlike the previous approaches, we designed a structure-aware embedding method and exploit GPT-2 to generate dialogues.
We have presented DialogoKG, a novel knowledge-grounded task-oriented dialogue system improving the state-of-the-art across multiple benchmark datasets. DialogoKG focuses on capturing the underlying semantics of the knowledge graph and pays attention to the relevant graph triples to understand the task and generate correct and human-like responses. The key contributions of DialogoKG include 1) Knowledge embedding technique, that
embeds the structural information of a knowledge graph effectively, and 2) Knowledge graph-weighted attention masking, which guides the masked language model to attend to the relevant knowledge entries for generating correct and informative responses. Finally, we showed DialoKG's ability to generate accurate, diverse, and human-like dialogues through quantitative and qualitative analysis. We performed an ablation study and studied the effect of dialogue history, knowledge embedding and knowledge attention masking.
We acknowledge the support of the following projects: SPEAKER (BMWi FKZ 01MK20011A), JOSEPH (Fraunhofer Zukunftsstiftung), OpenGPTX (BMWK FKZ 68GX21007A), the excellence clusters ML2R (BmBF FKZ 01 15 18038 A/B/C), ScaDS.AI (IS18026A-F) and TAILOR (EU GA 952215). The authors also acknowledge the financial support by the Federal Ministry for Economic Affairs and Energy of Germany in the project CoyPu (project number 01MK21007G).
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450.
Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2020. PLATO: Pre-trained dialogue generation model with discrete latent variable. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 85-96, Online. Association for Computational Linguistics.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Debanjan Chaudhuri, Md Rashad Al Hasan Rony, Simon Jordan, and Jens Lehmann. 2019. Using a kg-copy network for non-goal oriented dialogues. In International Semantic Web Conference, pages 93-109. Springer.
Debanjan Chaudhuri, Md Rashad Al Hasan Rony, and Jens Lehmann. 2021. Grounding dialogue systems via knowledge graph aware decoding with pretrained transformers. In European Semantic Web Conference, pages 323-339. Springer.
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. 2017. Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 37-49, Saarbrücken, Germany. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833.
Fabian Galetzka, Jewgeni Rose, David Schlangen, and Jens Lehmann. 2021. Space efficient context encoding for non-task-oriented dialogue generation with graph attention transformer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7028-7041, Online. Association for Computational Linguistics.
Revanth Gangi Reddy, Danish Contractor, Dinesh Raghu, and Sachindra Joshi. 2019. Multi-level memory for task oriented dialogs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3744-3754, Minneapolis, Minnesota. Association for Computational Linguistics.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631-1640, Berlin, Germany. Association for Computational Linguistics.
Jing Gu, Qingyang Wu, Chongruo Wu, Weiyan Shi, and Zhou Yu. 2021. PRAL: A tailored pre-training model for task-oriented dialog generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 305-313, Online. Association for Computational Linguistics.
Wanwei He, Min Yang, Rui Yan, Chengming Li, Ying Shen, and Ruifeng Xu. 2020a. Amalgamating knowledge from two teachers for task-oriented dialogue system with adversarial training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3498-3507, Online. Association for Computational Linguistics.
Zhenhao He, Yuhong He, Qingyao Wu, and Jian Chen. 2020b. Fg2seq: Effectively encoding knowledge for
end-to-end task-oriented dialog. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8029-8033.
Zhenhao He, Jiachun Wang, and Jian Chen. 2020c. Task-oriented dialog generation with enhanced entity representation. In INTERSPEECH, pages 3905-3909.
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. arXiv preprint arXiv:2005.00796.
Byeongchang Kim, Jaewoo Ahn, and Gunhee Kim. 2020. Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue. In ICLR.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR).
Xiexiong Lin, Weiyu Jian, Jianshan He, Taifeng Wang, and Wei Chu. 2020. Generating informative conversational response using recurrent knowledge-interaction and knowledge-copy. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 41-52, Online. Association for Computational Linguistics.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132, Austin, Texas. Association for Computational Linguistics.
Shuman Liu, Hongshen Chen, Zhaochun Ren, Yang Feng, Qun Liu, and Dawei Yin. 2018a. Knowledge diffusion for neural dialogue generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1498.
Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018b. Table-to-text generation by structure-aware seq2seq learning. In Thirty-Second AAAI Conference on Artificial Intelligence.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Andrea Madotto, Samuel Cahyawijaya, Genta Indra Winata, Yan Xu, Zihan Liu, Zhaojiang Lin, and Pascale Fung. 2020. Learning knowledge bases with parameters for task-oriented dialogue systems. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 2372-2394, Online. Association for Computational Linguistics.
Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018. Mem2seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1468-1478. Association for Computational Linguistics.
Nikita Moghe, Priyesh Vijayan, Balaraman Ravindran, and Mitesh M. Khapra. 2020. On incorporating structural information to improve dialogue response generation. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 11-24, Online. Association for Computational Linguistics.
Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252, Copenhagen, Denmark. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Libo Qin, Yijia Liu, Wanxiang Che, Haoyang Wen, Yangming Li, and Ting Liu. 2019. Entity-consistent end-to-end task-oriented dialogue system with KB retriever. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 133-142, Hong Kong, China. Association for Computational Linguistics.
Libo Qin, Xiao Xu, Wanxiang Che, Yue Zhang, and Ting Liu. 2020. Dynamic fusion network for multi-domain end-to-end task-oriented dialog. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6344-6354, Online. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Dinesh Raghu, Atishya Jain, Mausam, and Sachindra Joshi. 2021. Constraint based knowledge base distillation in end-to-end task oriented dialogs. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5051-5061, Online. Association for Computational Linguistics.
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS'15, page 2440-2448, Cambridge, MA, USA. MIT Press.
Shikhar Vashishth, Prateek Yadav, Manik Bhandari, and Partha Talukdar. 2019. Confidence-based graph convolutional networks for semi-supervised learning. In Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pages 1792-1801. PMLR.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Haoyang Wen, Yijia Liu, Wanxiang Che, Libo Qin, and Ting Liu. 2018. Sequence-to-sequence learning for task-oriented dialogue with dialogue state representation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3781-3792, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438-449, Valencia, Spain. Association for Computational Linguistics.
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020a. TOD-BERT: Pretrained natural language understanding for task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 917-929, Online. Association for Computational Linguistics.
Chien-Sheng Wu, Richard Socher, and Caiming Xiong. 2019. Global-to-local memory pointer networks for task-oriented dialogue. In Proceedings of the International Conference on Learning Representations (ICLR).
Han Wu, Kun Xu, and Linqi Song. 2021. Csagn: Conversational structure aware graph network for conversational semantic role labeling. arXiv preprint arXiv:2109.11541.
Sixing Wu, Ying Li, Dawei Zhang, and Zhonghai Wu. 2020b. Improving knowledge-aware dialogue response generation by using human-written prototype
dialogues. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1402-1411, Online. Association for Computational Linguistics.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 535-546, Online. Association for Computational Linguistics.
Hao-Tong Ye, Kai-Lin Lo, Shang-Yu Su, and Yun-Nung Chen. 2020. Knowledge-grounded response generation with deep attentional latent-variable model. Computer Speech & Language, 63:101069.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. Dialogpt: Large-scale generative pre-training for conversational response generation. In ACL, system demonstration.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563-578, Hong Kong, China. Association for Computational Linguistics.
We report the hyper-parameters used to train DialoKG in Table 6 for SMD, CamRest, and MWOZ. GPT-2 specific hyper-parameters are also reported in Table 6. All the hyper-parameters are found after a grid search and evaluation on the validation set. We sample learning rate from  $\{6.25\mathrm{e} - 01,6.25\mathrm{e} - 04,$ $6.25\mathrm{e} - 05\}$  and maximum history token and knowledge token from  $\{128,256,384,512\}$ .
SMD	CamRest	MWOZ
Learning rate	6.25e-05	6.25e-04	6.25e-05
Adam epsilon	1e-08	1e-08	1e-08
Batch size	4	4	4
Gradient accumulation steps	4	4	4
Max history turn	4	4	1
Maximum history token	128	256	128
Maximum knowledge token	384	256	384
Top relations	7	7	6
Top entities	7	5	7
Epochs	40	25	30
For both training and evaluation, we use a batch size of 4. Hyper-parameters used during the inference are reported in Table 7. We used 12 NVIDIA TitanX GPUs, each with 12GB of memory to train models. It took 30, 18 and 45 minutes to train on SMD, CamRest and MWOZ data.
Table 6: Training parameters.
SMD	CamRest	MWOZ
Temperature	0.68	0.85	0.18
Top-k	6	8	10
Top-p	0.9	0.9	0.9
Maximum response length	100	80	120
Top entities	7	7	6
Top relations	7	5	7
Table 7: Decoding parameters.