SlowGuess/NAACL_2022_OCR huggingface dataset:
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019. OpenDialKG: Explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 845-854, Florence, Italy. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, and Jianfeng Gao. 2019. Conversing by reading: Contentful neural conversation with on-demand machine reading. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5427-5436, Florence, Italy. Association for Computational Linguistics.
Jun Quan, Deyi Xiong, Bonnie Webber, and Changjian Hu. 2019. GECOR: An end-to-end generative ellipsis and co-reference resolution model for task-oriented dialogue. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4547-4557, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Dinesh Raghu, Nikhil Gupta, and Mausam. 2019. Disentangling Language and Knowledge in Task-Oriented Dialogs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1239-1255, Minneapolis, Minnesota. Association for Computational Linguistics.
Revanth Gangi Reddy, Danish Contractor, Dinesh Raghu, and Sachindra Joshi. 2019. Multi-level memory for task oriented dialogs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3744-3754.
Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable
encoder-decoder model for generating dialogues. In Thirty-First AAAI Conference on Artificial Intelligence.
Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1577-1586.
Robyn Speer and Catherine Havasi. 2012. Representing general relational knowledge in ConceptNet 5. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 3679-3686, Istanbul, Turkey. European Language Resources Association (ELRA).
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Chongyang Tao, Shen Gao, Mingyue Shang, Wei Wu, Dongyan Zhao, and Rui Yan. 2018. Get the point of my utterance! learning towards effective responses with multi-head attention mechanism. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4418-4424.
Yi-Lin Tuan, Yun-Nung Chen, and Hung-yi Lee. 2019. DyKgChat: Benchmarking dialogue generation grounding on dynamic knowledge graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1855-1865, Hong Kong, China. Association for Computational Linguistics.
Deeksha Varshney and Asif Ekbal Anushkha Singh. 2021. Knowledge grounded multimodal dialog generation in task-oriented settings. In Proceedings of the 35th Pacific Asia Conference on Language, Information and Computation, pages 425-435.
Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869.
Jian Wang, Junhao Liu, Wei Bi, Xiaojiang Liu, Kejing He, Ruifeng Xu, and Min Yang. 2020. Dual dynamic memory network for end-to-end multi-turn task-oriented dialog systems. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4100-4110, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Sixing Wu, Ying Li, Dawei Zhang, Yang Zhou, and Zhonghai Wu. 2020a. Diverse and informative dialogue generation with context-specific commonsense knowledge awareness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5811-5820, Online. Association for Computational Linguistics.
Sixing Wu, Ying Li, Dawei Zhang, Yang Zhou, and Zhonghai Wu. 2020b. Topicka: Generating commonsense knowledge-aware dialogue responses towards the recommended topic fact. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI, pages 3766-3772.
Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential Reasoning Learning for Language Representation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7170-7186, Online. Association for Computational Linguistics.
Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Augmenting end-to-end dialogue systems with commonsense knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Hainan Zhang, Yanyan Lan, Liang Pang, Jiafeng Guo, and Xueqi Cheng. 2019. Recosa: Detecting the relevant contexts with self-attention for multi-turn dialogue generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3721-3730.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213.
Wei-Nan Zhang, Yue Zhang, Hanlin Tang, Zhengyu Zhao, Caihai Zhu, and Ting Liu. 2021. What did you refer to? Evaluating co-references in dialogue. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5075-5084, Online. Association for Computational Linguistics.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online. Association for Computational Linguistics.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654-664.
Xiangyu Zhao, Longbiao Wang, Ruifang He, Ting Yang, Jinxin Chang, and Ruifang Wang. 2020a. Multiple knowledge syncretic transformer for natural dialogue generation. In Proceedings of The Web Conference 2020, pages 752-762.
Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, and Rui Yan. 2019. Low-resource knowledge-grounded dialogue generation. In International Conference on Learning Representations.
Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, and Rui Yan. 2020b. Low-resource knowledge-grounded dialogue generation. In International Conference on Learning Representations.
Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020c. Knowledge-grounded dialogue generation with pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3377-3390, Online. Association for Computational Linguistics.
Yufan Zhao, Wei Wu, and Can Xu. 2020d. Are pre-trained language models knowledgeable to ground open domain dialogues? arXiv preprint arXiv:2011.09708.
Wen Zheng, Natasa Milic-Frayling, and Ke Zhou. 2021. Knowledge-grounded dialogue generation with term-level de-noising. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2972-2983, Online. Association for Computational Linguistics.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018a. Common-sense knowledge aware conversation generation with graph attention. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4623-4629.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018b. A dataset for document grounded conversations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 708-713, Brussels, Belgium. Association for Computational Linguistics.
To update  $D_{S}$ , we use another Gated Recurrent Unit (GRU) network to emulate the decoder at round  $r$ , obtaining the "intermediate" hidden states,  $\tilde{s}_t^{(r)}$ .
$$
\tilde {s} _ {t} ^ {(r)} = \operatorname {G R U} \left(c _ {t} ^ {(r)}, q _ {t}\right) \tag {15}
$$
$$
\tilde {u} _ {t} ^ {(r)} = u _ {t} ^ {(r - 1)} \left(1 - \tilde {a} _ {t} ^ {(r)} F _ {t} ^ {(r)}\right) \tag {16}
$$
$$
F _ {t} ^ {(r)} = \operatorname {S i g m o i d} \left(W _ {3} ^ {(r)}, \tilde {s} _ {t} ^ {(r)}\right) \tag {17}
$$
$$
D _ {S, t} ^ {(r)} = \tilde {u} _ {t} ^ {(r)} + \tilde {a} _ {t} ^ {(r)} A _ {t} ^ {(r)} \tag {18}
$$
$$
A _ {t} ^ {(r)} = \operatorname {S i g m o i d} \left(W _ {4} ^ {(r)}, \tilde {s} _ {t} ^ {(r)}\right) \tag {19}
$$
$W_{3}^{(r)}$  and  $W_{4}^{(r)}$  are the learnable parameters.  $\tilde{a}_t^{(r)}$  is computed similar to the manner defined in Eq 1.
Experiments are carried out on two benchmark datasets, viz. Wizard of Wikipedia (Dinan et al., 2018) and CMU_DOG (Zhou et al., 2018b).
Wizard of Wikipedia (WoZ) is one of the most comprehensive knowledge-based conversation datasets, covering 1,365 open-domain topics. Each conversation takes place between a wizard who can retrieve knowledge about a specific topic and form a response based on it and an apprentice who is simply eager to speak with the wizard but lacks access to external knowledge. The test set is further divided into two parts: Test Seen and Test Unseen. The former contains conversations about topics that have previously been seen in the training set, whereas the latter contains conversations about topics that have never been seen in either the training or validation sets.
CMU_DOG focuses on the movie domain, and the conversations take place between two users who both have access to the relevant documents. Every document includes information, such as the title of the film, the cast, an introduction, ratings, and a few scenes. We consider subsequent utterances by the same person as a single one. ConceptNet database can be downloaded from https://conceptnet.io.
Wizard of Wikipedia	CMU_DOG
Train	Valid	Test Seen	Test Unseen	Train	Valid	Test
#Conversation	18,430	1,948	965	968	3,373	229	619
#Utterances	166,787	17,715	8,715	8,782	74,717	4,993	13,646
Avg. # of Turns	9.0	9.1	9.0	9.1	22.2	21.8	22.0
#TopicsDocuments	1,247	599	533	58	30	30	30
Table 4: Dataset Statistics
For our proposed CNTF model, we set the word embedding dimension as 300, and use GloVe word embeddings. The hidden size of GRU is sampled from  $\{128,256\}$ . Both the number of rounds  $R$ , the number of hops  $K$  are sampled from  $\{2,3\}$ , and the sliding window size is sampled from  $\{1,2\}$ . We use the ADAM optimizer (Kingma and Ba, 2014) whose learning rate is fixed to 0.0005 and set the beam size to 4, while decoding the responses. We truncate utterances to a max token count of 200 and knowledge base to 400. To handle the long-text knowledge base of CMU_DOG,
for every utterance and knowledge sentence we compute a TF-IDF vector. We then compute the cosine similarity between an utterance and every sentence in the knowledge base and retain the top-2 knowledge sentences, similar to the procedure adopted in Enriched Topical Chat dataset (Gopalakrishnan et al., 2019). The conversation and knowledge base vocabulary is shared and comprises of 30,004 words, while common sense vocabulary is maintained separately. We choose batch size as 2 and 8 for CMU_DOG and Wizard of Wikipedia, respectively, for training the models. There are roughly 83M parameters for our model when trained on Wizard of Wikipedia, and 38M on CMU_DOG, the difference in size is due to the vocabulary variation. These are much lesser than large pre-trained models which have much greater parameters (KnowledGPT which uses GPT-2). It is trained for 10-15 epochs. We choose the best model when the loss on the validation set does not decrease. The variances of the results are at most 1e-3 after three runs with random initialization for each method, and they have no effect on the trend. We have adapted the code framework from DDMN (Wang et al., 2020). We have used GeForce GTX 1080 Ti as the computing infrastructure. We used the AllenNLP co-reference resolution module (https://github.com/allenai/allennlp-models) for coreference resolution. We used the spacy toolkit (https://github.com/huggingface/neuralcoref) to identify named entities in the text.
D.1 Automatic Evaluation:
For evaluating our baseline and proposed models, we used F1 $^1$ , BLEU (Papineni et al., 2002), PPL and Embedding-based metrics $^2$  (Liu et al., 2016) such as Vector Extrema, Greedy Matching and Embedding Average for evaluation. Perplexity (PPL) is a metric used to assess how well a probability model predicts a sentence. The term intersection between the gold response and output response by the model is calculated using BLEU (BLEU-4) and the unigram F1-score. Word-matching-based metrics are an alternative to embedding-based metrics. These metrics allocate a vector to each term in the sentence in order to truly understand the intended meaning of the predicted sentence, as described by
<sup>1</sup>https://github.com/facebookresearch/ParlAI/blob/master/parlai/core/metrics.py
$^{2}$ https://github.com/Maluuba/nlg-eval
word embedding. Using the above standard metrics, we evaluate our models on both the seen and unseen test sets of the Wizard of Wikipedia dataset, as well as the test set of the CMU_DOG dataset.
D.2 Human Evaluation:
However apart from the automatic evaluation metrics, for evaluating samples from human perspective we randomly selected 100 samples from the Wizard of Wikipedia's Test Seen and Test Unseen sets. We hire four professionals, each with a postgraduate degree and experience, to serve as human judgment annotators. The annotators are regular employees (paid monthly in accordance with university policy) earning Rs 35,000 per month. The annotators are members of our research team and have been working on similar projects for the past three years. For each example, we provide our annotators with model responses and human groundtruth. We use the following metrics for evaluation:
(i) Fluency: It is a metric that measures whether or not a sentence is comprehensible. (ii) Adequacy: This metric assesses the cohesiveness of the generated response with respect to the conversation context. (iii) Knowledge Existence (KE): This metric determines whether the response contains knowledge or not. (iv) Knowledge Correctness (KC): This metric determines whether the knowledge in the predicted response is correct. (v) Knowledge Relevance (KR): This metric is used to determine whether the knowledge is correct and relevant to the topic of the conversation. The annotators assign a score of 0 to 2 to each response (representing "incorrect," "moderately correct," and "perfect"). Fleiss' kappa (Fleiss, 1971) is used to calculate the annotators' agreement.
E.1 Automatic Evaluation
We also compare our proposed CNTF model to (Zhao et al., 2020a) and (Zheng et al., 2021). MKST (Zhao et al., 2020a) obtains a F1-score of 22.2 / 21.3 and BLEU-4 score of 0.077 / 0.072 on test seen / unseen of WoZ dataset. KTWM (Zhao et al., 2020a) obtains a BLEU-4 score of 0.033 / 0.022 with an embedding average, extrema and greedy score of 0.682 / 0.668, 0.394 / 0.379, 0.574 / 0.542 respectively. Our model clearly outperforms these baselines by obtaining a BLEU-4 score of 0.119 / 0.110, F1-score of 32.5 / 31.4 with an embedding average, extrema and greedy score of
0.911 / 0.910, 0.577 / 0.570, 0.758 / 0.752, respectively. In addition, our model clearly outperforms the BART based models for knowledge grounded generation (De Bruyn et al., 2020) on F1-score (Test Seen - 12.2 / 20.1; Test Unseen 14.9 / 19.3) by a huge margin on both the test set of WoZ dataset.
E.2 Error Analysis
For a dialogue with no topic specific knowledge sentences usually our model fails to keep the conversation going by generating inadequate responses and also misses several entities. For example, Input utterance: that's not uncommon! there are rescue groups that specialize in finding homes for retired sled dogs. I bet they retire them at a certain age then they need a home huh; Predicted Response (CNTF): that's cute! i'm sure they're cute!; Gold Response: yes. huskies got their name from the word referring to eskimos. As it can be clearly seen the model fails to capture the entity huskies and instead generates a generic response.