SlowGuess/NAACL_2022_OCR huggingface dataset:
We test our proposed technique on two knowledge-grounded dialogue generation benchmark datasets, viz. Wizard of Wikipedia (Dinan et al., 2018) and CMU Document Grounded Conversations (Zhou et al., 2018b). The WoZ and CMU_DOG datasets consist of approximately  $\approx 22\mathrm{K}$  and  $\approx 4\mathrm{K}$  dialogs, respectively, covering more than 1,365 and 90 topics. The datasets are summarized in Appendix B. The statistics of the datasets are shown in Table 4 of the Appendix.
4.1.2 Commonsense Knowledge Base
We use ConceptNet, an open-domain repository of commonsense knowledge. It includes the relationships between concepts that are commonly used in everyday situations, such as "Mango is a fruit." This function is desirable in our experiments because it is critical to be able to identify the informal relationships between common concepts in an open-domain conversation setting. We remove triples containing multi-word entities when filtering words based on dataset vocabulary, and 147,676 triples were retained with 27,468 entities and 44 relations for Wizard of Wikipedia dataset. For CMU_DOG dataset, we have a total of 14,689 entities, 74,485 triples and 42 relations.
We use the following models as the baselines:
1. Transformer Memory Network (TMN) (Dinan et al., 2018): To encode dialogue, a shared transformer-based encoder is used. After knowledge selection, memory networks are used to reencode the dialogue information. Finally, a transformer decoder is used to decode the responses.
2. DialogGPTfinetune(Zhao et al., 2020d): It utilises a DialogGPT (345M) model fine-tuned on training examples from the Topical Chat dataset to determine whether the pre-trained models can serve as knowledge bases for open-domain dialogue generation.
3. Incremental Transformer with Deliberation Decoder (ITDD) (Li et al., 2019): It uses an incremental transformer-based model to encode utterances and documents and a deliberation decoder to decode responses.
4. Disentangled Response Decoder (DRD) (Zhao et al., 2019): It is made up of three modules: a language model, a context processor, and a
Wizard of Wikipedia	CMU_DoG
Models	PPL (Seen/Unseen)	F1% (Seen/Unseen)	BLEU-4 (Seen/Unseen)	Embedding Average (Seen/Unseen)	Vector Extrema (Seen/Unseen)	Greedy Matching (Seen/Unseen)	PPL	F1%	BLEU-4	Embedding Average	Vector Extrema	Greedy Matching
TMN	66.5 / 103.6	15.9 / 14.3	0.017 / 0.009	0.844 / 0.839	0.427 / 0.408	0.658 / 0.645	75.2	9.9	0.007	0.789	0.399	0.615
ITDD	17.8 / 44.8	16.2 / 11.4	0.025 / 0.011	0.841 / 0.826	0.425 / 0.364	0.654 / 0.624	26.0	10.4	0.009	0.748	0.390	0.587
\( DialogGPT_{finetune} \)	16.2 / 20.4	19.0 / 17.6	0.023 / 0.017	0.871 / 0.869	0.461 / 0.451	0.683 / 0.674	15.9	13.7	0.015	0.812	0.430	0.641
DRD	19.4 / 23.0	19.3 / 17.9	0.044 / 0.037	0.864 / 0.862	0.455 / 0.444	0.679 / 0.671	54.4	10.7	0.012	0.809	0.413	0.633
ConKADI	89.4 / 93.0	13.3 / 15.9	0.016 / 0.014	0.726 / 0.662	0.355 / 0.324	0.599 / 0.601	84.4	8.7	0.006	0.768	0.326	0.600
KnewledGPT	19.2 / 22.3	22.0 / 20.5	0.058 / 0.047	0.872 / 0.870	0.463 / 0.452	0.682 / 0.674	20.6	13.5	-	0.837	0.437	0.654
CNTF	24.4 / 28.6	32.5 / 31.4	0.119 / 0.110	0.911 / 0.910	0.577 / 0.570	0.758 / 0.752	46.0	14.6	0.018	0.882	0.518	0.708
CNTF-DKIC	24.3 / 28.5	33.1 / 32.9	0.118 / 0.117	0.913 / 0.913	0.582 / 0.581	0.761 / 0.758	44.5	15.1	0.018	0.882	0.518	0.708
CNTF-DKI	26.8 / 31.8	32.4 / 31.5	0.114 / 0.110	0.911 / 0.912	0.576 / 0.575	0.758 / 0.754	45.3	14.2	0.015	0.881	0.514	0.707
CNTF-DK	25.9 / 31.1	30.9 / 29.8	0.105 / 0.101	0.909 / 0.909	0.567 / 0.564	0.752 / 0.746	45.9	14.1	0.014	0.880	0.505	0.700
CNTF-D	47.5 / 96.3	15.3 / 13.5	0.022 / 0.015	0.884 / 0.883	0.456 / 0.440	0.689 / 0.679	47.9	11.8	0.013	0.880	0.492	0.693
Table 1: Automatic evaluation results marked in bold fonts indicate the best outcome for the measure and improvement over the best baseline, and is statistically significant (t-test with p-value at 0.05 significance level). The scores on the ablation models are shown in the last four rows. The values for baseline models are derived from Zhao et al., 2020c) and (Zhao et al., 2020d). (-) indicates that the value was not reported.
Models	Fluency (Seen/Unseen)	Adequacy (Seen/Unseen)	Knowledge Existence (Seen/Unseen)	Knowledge Correctness (Seen/Unseen)	Knowledge Relevance (Seen/Unseen)	Kappa (Seen/Unseen)
TMN	1.314 / 1.197	1.262 / 0.934	1.046 / 0.811	1.005 / 0.691	0.867 / 0.487	0.931 / 0.888
ITDD	1.135 / 1.290	0.545 / 0.965	0.515 / 0.382	0.301 / 0.188	0.184 / 0.101	0.940 / 0.930
KnowledGPT	1.813 / 1.817	1.568 / 1.556	1.493 / 1.139	1.430 / 1.390	1.172 / 1.040	0.810 / 0.811
CNTF	1.561 / 1.554	1.647 / 1.469	1.653 / 1.285	1.770 / 1.422	1.732 / 1.376	0.830 / 0.869
Gold Response	1.865 / 1.883	1.891 / 1.883	1.825 / 1.864	1.908 / 1.916	1.903 / 1.904	0.890 / 0.854
Table 2: Human assessment results for the baseline and proposed model on WoZ dataset. Bolded results of the proposed model against the baselines are statistically significant using t-test at  $0.05\%$  significance level.
knowledge processor for decoding responses. The response decoder is broken down into independent components in this case to investigate knowledge-based dialogue generation.
5. ConKADI (Wu et al., 2020a): It includes a Felicitous Fact mechanism to help the model focus on knowledge facts that are highly significant; additionally, two techniques, Context-Knowledge Fusion and Flexible Mode Fusion, are proposed to assist ConKADI in integrating the knowledge information.
6. **KnowledGPT** (Zhao et al., 2020c): This model implements response generation by combining a pre-trained language model with a knowledge selection module, and it intends to jointly optimize knowledge selection and response generation with unlabeled dialogues using an unsupervised approach.
To evaluate the predicted responses, we choose BLEU (Papineni et al., 2002), PPL, F1 and Embedding-based metrics (Liu et al., 2016). For human evaluation, we use fluency, adequacy, knowledge existence, knowledge correctness and knowledge relevance. Appendix D provides more infor
mation on these metrics.
5.1 Results of Automatic Evaluation
Table 1 shows the results on automatic evaluation metrics on Wizard of Wikipedia and CMU_DOG datasets. On WoZ, CNTF gives a significant rise of  $48\%$  on Test Seen and  $53\%$  on Unseen in F1 score and around two times more on both Seen and Unseen, in terms of BLEU-4, compared to the strongest baseline, KnownGPT. On CMU_DOG too, where the average turn length is roughly 2.5 times that of WoZ, CNTF surpasses the previous best on F1 and BLEU-4 by  $8\%$  and  $20\%$  respectively. Hence, CNTF achieves new state-of-the-art on both datasets.
Existing models struggle to generate engaging responses for dialogues based on new topics that were not encountered during the training phase, which most likely explains the observed low performance on Test Unseen. On the contrary, CNTF is capable of capturing the dialogue context and effectively utilizing external commonsense knowledge and parse the implicit mentions made to various entities through the conversation to produce accurate responses, as evidenced by the magnitude
of improvement achieved. On embedding-based metrics, all three measures have significantly improved, demonstrating the efficacy of our methodology. Comparison to more baseline models can be found in Appendix E.1.
5.2 Human Evaluation Results
Human evaluation results are shown in Table 2. We only compare our proposed model against KnowledGPT, ITDD and TMN on WoZ, as manual evaluation is expensive. It is clear that CNTF outperforms the baselines on both adequacy and knowledge-related criteria, demonstrating consistency with the results of automatic evaluation, and has comparable fluency performance. It is important to note that, despite providing contextually appropriate responses, KnowledGPT failed to capture the accurate knowledge associated with the input sequences, resulting in low scores. The strength of CNTF can be seen from the knowledge existence, correctness and relevance scores. This can be attributed to the fact that the multi-hop attention module and the interactive attention module incorporate the knowledge bases efficiently. The knowledge in the generated response is relevant with the contexts and is factually correct. Furthermore, the responses are more effective at exchanging information than at casual chat. The proposed model also makes good use of commonsense knowledge and named entities due to attention module as explained in Section 3.5. All of the kappa values are greater than 0.75, indicating that the annotators agree.
In Table 3, we present a few example conversations as predicted by the proposed (CNTF) and the strongest baseline (KnowledGPT) on Test Seen from Wizard of Wikipedia. In utterance 3, CNTF is able to decipher that the context of the conversation is dr. pepper using the triple (drink, RelatedTo, pepper) obtained using the mechanism explained in Section 3.5 unlike KnowledGPT which starts talking about 7up. Additionally, CNTF efficiently utilizes the commonsense knowledge triples by correctly copying the entities in the triples associated with the word flavor. As seen in the fourth utterance, the model correctly decodes the response using more detailed knowledge from the topic-specific knowledge base as opposed to KnowledGPT. Triples such as (1904, RelatedTo, pepper), (sold, RelatedTo, Europe) which were created using Section 3.4 have aided it in understanding the context better.
To analyze the impact of the constituent modules in our model on performance (Table 1), we compare CNTF with the following variants:
(i) CNTF-D: This configuration only employs the dialogue encoder with multi-hop attention to demonstrate the significance of employing a knowledge encoder. This results in a  $53\%$  decrease in F1 score on Test Seen, demonstrating the effectiveness of our knowledge module with multi-hop attention. The score reduction in CMU_DoG is less severe because workers do not rely as heavily on external knowledge as the Wizard does, where it is highly correlated with available knowledge. (ii) CNTFDK: Interactive attention is essential for generating insightful responses while decoding the answer. We remove the Interactive Dialogue-Knowledge module, as explained in Section 3.6.1, to demonstrate its utility. This results in a significant decrease in both BLEU and F1 scores. (iii) CNTFDKI: We conduct experiments with only the dialogue and knowledge modules, as well as the interactive module, to demonstrate the benefit of using structured knowledge in the form of triples for gauging the implicit references made throughout the conversation. We see a significant drop in scores here as well. (iv) CNTF-DKIC: This model is used to show the effectiveness of co-reference based named entity triples. We see a drop in BLEU-4 scores for the seen testset, but we see an improvement on the unseen testset by using only commonsense knowledge. This could be attributed to the fact that for unseen data, the same entities are usually not present because they have conversations on topics that are rarely seen in the training set.
We may note that CNTF beats the SOTA models on every metric however due to the addition of new triples (more than  $60\%$  increment in triples on an average for both the datasets) which may have added to noise in the model, and hence it shows lower scores on some metrics than CNTF-DKIC.
Using the generated responses, we perform a thorough examination of our proposed model and categorize the errors it encounters as follows:
1. Repetition: There are some instances where certain words are repeated in the generated responses. For example, Predicted response: "i'm not sure, but it is similar to violet, violet and violet."
Utterance 1
Knowledge 1
Triples	Agent 1: i love dr pepper
dr pepper is a carbonated soft drink marketed as having a unique flavor.
(pepper, RelatedTo, soda), (soda, IsA, drink), (flavor, RelatedTo, drink), (carbonated, Synonym, soda), (soft, RelatedTo, drink), (drink, RelatedTo, like)...
CNTF
KnowledGPT	yeah i love dr pepper. it is marketed by having a unique flavor.
i love dr pepper too! it's a carbonated soft drink marketed as having a unique flavor.
Utterance 2
Knowledge 2
Triples	Agent 2: Oh same, it's my favorite Dr Pepper is a carbonated soft drink Agent 1: oh yea? its such a good soda
7 up (stylized as 7 up outside the u.s.) is a brand of lemon-lime flavored, non-caffeinated soft drink.
(flavor, RelatedTo, drink), (pepper, RelatedTo, soda), (soda, IsA, drink), (carbonated, Synonym, soda), (soft, RelatedTo, drink), (US, RelatedTo, pepper), (great, RelatedTo, super) ...
CNTF
KnowledGPT	yea 7 up (stylized as 7up outside the u.s.) is a brand of lemon-line.
it was created in the 1880s by pharmacist charles alderton in waco, texas and first served around 1885.
Utterance 3
Knowledge 3
Triples	Agent 2: oh yeah, i also like 7-up, 5 a brand of lemon-lime flavor. Agent 1: oh yeah thats great too
the drink was created in the 1880s by pharmacist charles alderon in waco, texas and first served around 1885.
(flavor, RelatedTo, drink), (1880s, RelatedTo, wild), (1880s, RelatedTo, created), (1880s, RelatedTo, pharmacist), (drink, RelatedTo, pepper) ...
CNTF
KnowledGPT	yeah, the drink was created in the 1880s by pharmacist charles alderton in waco.
yeah, 7 up was created by charles leiper grigg, who launched his st. louis based company the howdy corporation in 1920
Utterance 4
Knowledge 4
Triples	Agent 2: but, dr pepper was created in 1880s by a pharmacist. isn't that wild? Agent 1: yea wow it is so old
dr pepper was first nationally marketed in the united states in 1904, and is now also sold in europe, asia, canada, mexico, australia, and south america, as well as new zealand and south africa as an imported good.
(1904, RelatedTo, pepper), (sold, RelatedTo, Europe), (pepper, RelatedTo, Australia), (pepper, RelatedTo, Canada) ...
CNTF
KnowledGPT	yea dr pepper was first nationally sold in the us in 1904, and is now also sold in europe.
it was first served in waco, texas in 1885. i wonder how long it has been around?
Utterance 5
Knowledge 5
Triples	Agent 2: oh yeah, and it didn't make it into the us until it was first nationally marketed in the united states in 1904 Agent 1: thats super crazy
dr pepper was first nationally marketed in the united states in 1904, and is now also sold in europe, asia, canada, mexico, australia, and south america, as well as new zealand and south africa as an imported good.
(1904, RelatedTo, marketed), (States, RelatedTo, 1904), (United, RelatedTo, 1904), (1904, RelatedTo, US), (1904, RelatedTo, first), (first, RelatedTo, marketed) ...
CNTF
KnowledGPT	yea dr pepper was first nationally marketed in the us in 1904, and is now also sold in
it was formulated by a pharmacist named charles alderton in morrison's old corner drug store in waco, texas.
Table 3: Samples from Test Seen of WoZ dataset. The gold response for the  $(k)$ -th example is Agent 2's utterance in the  $(k + 1)$ -th example. The displayed knowledge is the supporting sentence for the gold response to that utterance.
2. Incomplete response: As shown in the response for the last example in Table 3, incomplete responses result in lower fluency scores. We discovered that the ground truth responses in the dataset are generated by copying incomplete sentences from the document knowledge. Since our model augments knowledge, it learns to produce responses in the same manner. For example: Document knowledge: "there is no scientifically precise definition of genius, and the question of whether the notion itself has any real meaning has long been a subject of debate, although psychologists are converging on a definition that emphasizes creativity and eminent achievement"; Gold Response: "there is no scientifically precise definition of genius". As can be seen, the response picked is incomplete and less fluent if it is compared to the knowledge sentence. We have evaluated our gold responses considering this in Table 2. We observed that the fluency score is  $1.865 / 1.883$  for both the test seen / unseen set. A few more error cases with examples are shown in the Appendix E.2.
We present a Commonsense, Named Entity, and Topical Knowledge Fused neural network (CNTF) to address reasoning over multiple knowledge bases in this paper. We propose, in particular, multi-hop attention over both structured and unstructured knowledge. Unlike previous approaches in Dialog, CNTF can find relevant supporting named entities in dialogs at each step of multi-hop attention in addition to already present commonsense knowledge. We test CNTF on WoZ and CMU_DOG and achieve excellent results. Furthermore, our analysis shows that CNTF can generate consistent results.
In the future, we hope to expand our work to build models which include emotions for knowledge grounded dialogues. Also, to tackle repetition and incomplete response, we aim to introduce rewards functions for these factors. Currently, our model does not consider the relation attribute in our proposed framework and hence the use of "RelatedTo" relation is not really affecting the performance of the proposed approach. We aim to incorporate relation attributes for triple representations.
7 Ethical Declaration
Our work relies solely on publicly available data. We followed the policies for using the data and did not violate any copyright issues.
Asif Ekbal acknowledges the Young Faculty Research Fellowship (YFRF), supported by Visvesvaraya Ph.D. scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia).
Yu Cao, Wei Bi, Meng Fang, and Dacheng Tao. 2020. Pretrained language models for dialogue generation with multiple input sources. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 909–917, Online. Association for Computational Linguistics.
Xiuyi Chen, Jiaming Xu, and Bo Xu. 2019. A working memory model for task-oriented dialog response generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2687-2693, Florence, Italy. Association for Computational Linguistics.
Maxime De Bruyn, Ehsan Lotfi, Jeska Buhmann, and Walter Daelemans. 2020. Bart for knowledge grounded conversations. In Converse@ KDD.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.
Sergey Edunov, Alexei Baevski, and Michael Auli. 2019. Pre-trained language model representations for language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4052-4059, Minneapolis, Minnesota. Association for Computational Linguistics.
Kate Ehrlich. 1981. Search and inference strategies in pronoun resolution: An experimental study. In 19th Annual Meeting of the Association for Computational Linguistics, pages 89-93.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. *Psychological bulletin*, 76(5):378.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Thirty-Second AAAI Conference on Artificial Intelligence.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinglang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, Dilek Hakkani-Tür, and Amazon Alexa AI. 2019. Topical-chat: Towards knowledge-grounded open-domain conversations.
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 140-149.
Baorong Huang, Zhuosheng Zhang, and Hai Zhao. 2021. Tracing origins: Coref-aware machine reading comprehension. arXiv preprint arXiv:2110.07961.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016b. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 994-1003, Berlin, Germany. Association for Computational Linguistics.
Zekang Li, Cheng Niu, Fandong Meng, Yang Feng, Qian Li, and Jie Zhou. 2019. Incremental transformer with deliberation decoder for document grounded conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 12-21.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132, Austin, Texas. Association for Computational Linguistics.