SlowGuess/ACL_2020_OCR huggingface dataset:
BERT (Devlin et al., 2019): We adapted this deep bidirectional transformers (Vaswani et al., 2017) as a retrieval-based model. For each utterance (except the first one in a dialog), we extracted keywords in the same way as Wu et al. (2017) and retrieved 10 response candidates, including the golden truth based on the BM25 algorithm (Robertson et al., 1995). The training task is to predict whether a candidate is the correct next utterance given the context, where a sigmoid function was used to output the probability score  $\hat{y} = \mathcal{P}(y = 1)$  and the cross-entropy loss was optimized:
$$
\mathcal {L} _ {0} ^ {(r)} = - y \log \hat {y} - (1 - y) \log (1 - \hat {y}),
$$
where  $y \in \{0,1\}$  is the true label. For the test, we selected the candidate response with the largest probability.
4.1.3 Knowledge-aware Models
A key-value memory module (Miller et al., 2016) is introduced to the aforementioned models to utilize the knowledge information. We treated all knowledge triples mentioned in a dialogue as the knowledge information in the memory module. For a triple that is indexed by  $i$ , we represented the key memory and the value memory respectively as a key vector  $\pmb{k}_i$  and a value vector  $\pmb{v}_i$ , where  $\pmb{k}_i$  is the average word embeddings of the head entity and the relation, and  $\pmb{v}_i$  is those of the tail entity. We used a query vector  $\pmb{q}$  to attend to the key vectors  $\pmb{k}_i (i = 1, 2, \dots)$ :  $\alpha_i = \text{softmax}_i (\pmb{q}^T \pmb{k}_i)$ , then the weighted sum of the value vectors  $\pmb{v}_i (i = 1, 2, \dots)$ ,  $\pmb{v} = \sum_{i} \alpha_i \pmb{v}_i$ , was incorporated into the decoding process (for the generation-based models, concatenated with the initial state of the decoder) or the classification (for the retrieval-based model, concatenated with the <CLS> vector). For Seq2Seq,  $\pmb{q}$  is the final hidden state of the encoder. For HRED, we treated the context vector as the query, while for BERT, the output vector of <CLS> was used.
Note that our dataset has a sentence-level annotation on the knowledge triples that each utterance uses. To force the knowledge-aware models to attend to the golden KG triples, we added an extra attention loss (for retrieval-based models, this loss was computed only on the positive examples):
$$
\mathcal {L} _ {\mathrm {a t t}} = - \frac {1}{| \{\mathrm {t r u t h} \} |} \sum_ {i \in \{\mathrm {t r u t h} \}} \log \alpha_ {i},
$$
where  $\{\mathrm{truth}\}$  is the set of indexes of triples that are used in the true response. The total loss are the weighted sum of  $\mathcal{L}_0^{(l)}$  and  $\mathcal{L}_{\mathrm{att}}$ :
$$
\mathcal {L} _ {\mathrm {t o t}} ^ {(l)} = \mathcal {L} _ {0} ^ {(l)} + \lambda \mathcal {L} _ {\mathrm {a t t}}, l \in \{g, r \}.
$$
Note that the knowledge-enhanced BERT was initialized from the fine-tuned BERT discussed in Section 4.1.2, and the parameters of the transformers were frozen during training the knowledge-related modules. The purpose was to exclude the impact of the deep transformers but only examine the potential effects introduced by the background knowledge.
We implemented the above models with TensorFlow (Abadi et al., 2016), PyTorch (Paszke et al.,
2017) and CoTK (Huang et al., 2020). The Jieba Chinese word segmenter $^{7}$  was employed for tokenization. The 200-dimensional word embeddings were initialized by Song et al. (2018), while the unmatched ones were randomly sampled from a standard normal distribution  $\mathcal{N}(0,1)$ . The type of RNN network units was all GRU (Cho et al., 2014) and the number of hidden units of GRU cells were all set to 200. ADAM (Kingma and Ba, 2014) was used to optimize all the models with the initial learning rate of  $5 \times 10^{-5}$  for BERT and  $10^{-3}$  for others. The mini-batch sizes are set to 2 dialogues for LM and 32 pairs of post and response for Seq2Seq and HRED.
4.3 Automatic Evaluation
4.3.1 Metrics
We measured the performance of all the retrieval-based models using Hits@1 and Hits@3, same as Zhang et al. (2018) and Wu et al. (2019). We adopted several widely-used metrics to measure the quality of the generated response. We calculated Perplexity (PPL) to evaluate whether the generation result is grammatical and fluent. BLEU-1/2/3/4 (Papineni et al., 2002) is a popular metric to compute the  $k$ -gram overlap between a generated sentence and a reference (Sordoni et al., 2015; Li et al., 2016b). Distinct-1/2/3/4 (Li et al., 2016b) is also provided to evaluate the diversity of generated responses.
The results are shown in Table 5. We analyze the results from the following perspectives:
The influence of knowledge: after introducing the knowledge, all the models were improved in terms of all the metrics except PPL in all the domains. First, all the models obtain higher Hits@1 scores (in the music domain, BERT obtains an improvement of 0.4 on Hits@1). After incorporating the knowledge into BERT, the performance of Hits@1 improves slightly, because the memory network which models knowledge information is rather shallow, compared to the deep structure in BERT. Second, Seq2Seq and HRED both have better BLEU- $k$  scores (in the travel domain, Seq2Seq obtains an improvement of 7.2 on BLEU-4), which means a better quality of generated responses. Third, the two generation-based models
<sup>7</sup>https://github.com/fxsjy/jieba
For generative models, the rank is decided by the PPL values of candidate responses.
7104
Model	Hits@1/3	PPL	BLEU-1/2/3/4	Distinct-1/2/3/4
Film
LM	14.30	35.70	21.91	24.22	12.40	7.71	4.27	2.32	6.13	10.88	16.14
Seq2Seq	17.54	40.57	23.88	26.97	14.31	8.53	5.30	2.51	7.14	13.62	21.02
HRED	16.45	40.62	24.74	27.03	14.07	8.30	5.07	2.55	7.35	14.12	21.86
BERT	65.36	91.79	-	81.64	77.68	75.47	73.99	8.55	31.28	51.29	63.38
Seq2Seq + know	17.77	41.66	25.56	27.45	14.51	8.66	5.32	2.85	7.98	15.09	23.17
HRED + know	17.38	39.79	26.27	27.94	14.69	8.73	5.40	2.86	8.08	15.81	24.93
BERT + know	65.67	91.79	-	81.98	78.08	75.90	74.44	8.59	31.47	51.63	63.78
Music
LM	18.09	39.36	14.61	25.80	13.93	8.61	5.57	2.72	7.31	12.69	18.64
Seq2Seq	22.65	44.43	16.17	28.89	16.56	10.63	7.13	2.52	7.02	12.69	18.78
HRED	21.20	42.84	16.82	29.92	17.31	11.17	7.52	2.71	7.71	14.07	20.97
BERT	55.64	86.90	-	78.71	73.61	70.55	68.43	6.57	26.75	44.75	55.85
Seq2Seq + know	22.90	47.14	17.12	29.60	17.26	11.36	7.84	3.93	12.35	23.01	34.23
HRED + know	21.82	45.33	17.69	29.73	17.51	11.59	8.04	3.80	11.70	22.00	33.37
BERT + know	56.08	86.87	-	78.98	73.91	70.87	68.76	6.59	26.81	44.84	55.96
Travel
LM	22.16	41.27	8.86	27.51	17.79	12.85	9.86	3.18	8.49	13.99	19.91
Seq2Seq	27.07	46.34	10.44	29.61	20.04	14.91	11.74	3.75	11.15	19.01	27.16
HRED	25.76	46.11	10.90	30.92	20.97	15.61	12.30	4.15	12.01	20.52	28.74
BERT	45.25	71.87	-	81.12	76.97	74.47	72.73	7.17	22.55	34.03	40.78
Seq2Seq + know	29.67	50.24	10.62	37.04	27.28	22.16	18.94	4.25	13.64	24.18	34.08
HRED + know	28.84	49.27	11.15	36.87	26.68	21.31	17.96	3.98	13.31	24.06	34.35
BERT + know	45.74	71.91	-	81.28	77.17	74.69	72.97	7.20	22.62	34.11	40.86
Table 5: Automatic evaluation. The best results of generative models and retrieval models are in **bold** and **underlined** respectively. “+ know” means the models enhanced by the knowledge base.
also gain larger Distinct- $k$  values (in the music domain, HRED obtains an improvement of 12.4 on Distinct-4), which indicates a better diversity of the generated results.
Comparison between models: In all the three domains, the knowledge-aware BERT model achieves the best performance in most of the metrics, as it retrieves the golden-truth response at a fairly high rate. HRED performs best in BLEU- $k$  and Distinct- $k$  among all the generation-based baselines without considering the knowledge. Knowledge-aware HRED has better results of BLEU- $k$  in the film and music domains and better results of Distinct- $k$  in the film domain, while the knowledge-enhanced Seq2Seq achieves the best Hits@1/3 scores among all the generation-based models.
Comparison between domains: For retrieval-based models, the performance is best in the film domain but worst in the travel domain, largely affected by the data size (see Table 3). For generation-based models, however, the performance improves from the film domain to the travel domain, as the average number of utterances per dialogue decreases
from 24.4 in the film domain to 16.1 in the travel domain (see Table 3). The more utterances a dialogue contains, the more difficulties in conversation modeling for generation-based models. Besides, the more diverse knowledge (1,837 entities and 318 relations in the film domain, vs. 699 entities and 7 relations in the travel domain) also requires the models to leverage knowledge more flexibly. The difference between different domains can be further explored in the setting of transfer learning or meta learning in the following research.
4.4 Manual Evaluation
To better understand the quality of the generated responses from the semantic and knowledge perspective, we conducted the manual evaluation for knowledge-aware BERT, knowledge-aware HRED, and HRED, which have achieved advantageous performance in automatic evaluation<sup>9</sup>.
4.4.1 Metrics
Human annotators were asked to score a generated response in terms of the fluency and coherence
<sup>9</sup>We omitted the BERT model because it performs similarly to knowledge-aware BERT as shown in automatic evaluation.
Model	Fluency	Coherence
Film \ κ	0.50	0.61
HRED	1.64	1.19
HRED + know	1.78	1.28
BERT + know	2.00	1.79
Music \ κ	0.37	0.57
HRED	1.90	1.30
HRED + know	1.86	1.36
BERT + know	2.00	1.80
Travel \ κ	0.55	0.74
HRED	1.77	1.10
HRED + know	1.78	1.31
BERT + know	2.00	1.76
Table 6: Manual evaluation. The best results (t-test,  $p$ -value  $< 0.005$ ) are in bold. Between two generative models, the significantly better results are italic underlined (t-test,  $p$ -value  $< 0.005$ ) or underlined (t-test,  $p$ -value  $< 0.05$ ).  $\kappa$  is the Fleiss' kappa value. "+ know" means the models enhanced by knowledge information.
metrics. The fluency score (rating scale is 0,1,2) is defined as whether the response is fluent and natural. The coherence (rating scale is 0,1,2) is defined as whether a response is relevant and coherent to the context and the knowledge information.
4.4.2 Annotation Statistics
We randomly sampled about 500 contexts from the test sets of the three domains and generated responses by each model. These 1,500 context-response pairs in total and related knowledge graphs were presented to three human annotators.
We calculated the Fleiss' kappa (Fleiss, 1971) to measure inter-rater consistency. Fleiss' kappa for Fluency and Coherence is from 0.37 to 0.74, respectively. The overall  $3/3^{10}$  agreement for Fluency and Coherence is from  $68.14\%$  to  $81.33\%$  in the three domains.
4.4.3 Results
The results are shown in Table 6. As can be seen, knowledge-aware BERT outperforms other models significantly in both metrics in all the three domains, which agrees with the results of automatic evaluation. The Fluency is 2.00 because the retrieved responses are all human-written sentences.
The Fluency scores of both generation-based models are close to 2.00 (in the music domain, the Fluency of HRED is 1.90), showing that the generated responses are fluent and grammatical. The Coherence scores of both HRED and knowledge-aware HRED are higher than 1.00 but still have a huge gap to 2.00, indicating that the generated responses are relevant to the context but not coherent to knowledge information in most cases. After incorporating the knowledge information into HRED, the Coherence score is improved significantly in all the three domains, as the knowledge information is more expressed in the generated responses.
In this paper, we propose a Chinese multi-domain corpus for knowledge-driven conversation generation, KdConv. It contains 86K utterances and 4.5K dialogues, with an average number of 19.0 turns. Each dialogue contains various topics and sentence-level annotations that map each utterance with the related knowledge triples. The dataset provides a benchmark to evaluate the ability to model knowledge-driven conversations. In addition, KdConv covers three domains, including film, music, and travel, that can be used to explore domain adaptation or transfer learning for further research. We provide generation- and retrieval-based benchmark models to facilitate further research. Extensive experiments demonstrate that these models can be enhanced by introducing knowledge, whereas there is still much room in knowledge-grounded conversation modeling for future work.
This work was jointly supported by the NSFC projects (Key project with No. 61936010 and regular project with No. 61876096), and the National Key R&D Program of China (Grant No. 2018YFC0830200). We thank THUNUS NExT Joint-Lab for the support.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly
103/3 means all the three annotators assign the same label to an annotation item.
7106
learning to align and translate. arXiv preprint arXiv:1409.0473.
Hannah Bast, Florian Bäurle, Björn Buchhold, and Elmar Haußmann. 2014. Easy access to the freebase dataset. In WWW, pages 95-98. ACM.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137-1155.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In AAAI.
John J Godfrey, Edward C Holliman, and Jane McDaniel. 1992. Switchboard: Telephone speech corpus for research and development. In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517-520. IEEE.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-TAijr. 2019. Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations. In Proc. Interspeech 2019, pages 1891-1895.
Jiatao Gu, Yong Wang, Yun Chen, Victor OK Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In EMNLP, pages 3622-3631.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
Fei Huang, Dazhen Wan, Zhihong Shao, Pei Ke, Jian Guan, Yilin Niu, Xiaoyan Zhu, and Minlie Huang. 2020. Cotk: An open-source toolkit for fast development and fair evaluation of text generation. arXiv preprint arXiv:2002.00583.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In *NAACL*, pages 110-119.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016b. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Shuman Liu, Hongshen Chen, Zhaochun Ren, Yang Feng, Qun Liu, and Dawei Yin. 2018. Knowledge diffusion for neural dialogue generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1498.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. arXiv preprint arXiv:1506.08909.
Fei Mi, Minlie Huang, Jiyong Zhang, and Boi Faltings. 2019. Meta-learning for low-resource natural language generation in task-oriented dialogue systems. In *IJCAI*.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In EMNLP, pages 1400-1409.
Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M Khapra. 2018. Towards exploiting background knowledge for building conversation systems. In EMNLP, pages 2322-2332.
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019. Opendialkg: Explanable conversational reasoning with attention-based walks over knowledge graphs. In ACL, pages 845-854.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318.
7107
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch.
Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, and Jianfeng Gao. 2019. Conversing by reading: Contentful neural conversation with on-demand machine reading. arXiv preprint arXiv:1906.02738.
Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of twitter conversations. In *NAACL*, pages 172-180.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109.
Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C. Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In AAAI, pages 3776-3784.
Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In ACL, pages 1577-1586.
Louis Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Generating long and diverse responses with neural conversation models. arXiv preprint arXiv:1701.03185.
Yan Song, Shuming Shi, Jing Li, and Haisong Zhang. 2018. Directional skip-gram: Explicitly distinguishing left and right context for word embeddings. In NAACL, pages 175–180.
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 196-205, Denver, Colorado. Association for Computational Linguistics.
Robert Speer and Catherine Havasi. 2013. Conceptnet 5: A large semantic network for relational knowledge. In *The People's Web Meets NLP*, pages 161-176. Springer.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104-3112.
Yi-Lin Tuan, Yun-Nung Chen, and Hung-yi Lee. 2019. Dykgchat: Benchmarking dialogue generation grounding on dynamic knowledge graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1855-1865.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998-6008.
Zhigang Wang, Juanzi Li, Zhichun Wang, Shuangjie Li, Mingyang Li, Dongsheng Zhang, Yao Shi, Yongbin Liu, Peng Zhang, and Jie Tang. 2013. Xlore: A large-scale english-chinese bilingual knowledge graph. In International semantic web conference (Posters & Demos), volume 1035, pages 121-124.
Tsung Hsien Wen, Milica Gasic, Nikola Mrksic, Pei Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In EMNLP, pages 1711-1721.
Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019. Proactive human-machine conversation with explicit conversation goal. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3794-3804, Florence, Italy. Association for Computational Linguistics.
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 496-505, Vancouver, Canada. Association for Computational Linguistics.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Hao Zhou, Minlie Huang, and Xiaoyan Zhu. 2016. Context-aware natural language generation for spoken dialogue systems. In COLING, pages 2032-2041.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018a. Commonsense knowledge aware conversation generation with graph attention. In *IJCAI*, pages 4623-4629.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018b. A dataset for document grounded conversations. In EMNLP, pages 708-713.
Wenya Zhu, Kaixiang Mo, Yu Zhang, Zhangbin Zhu, Xuezheng Peng, and Qiang Yang. 2017. Flexible end-to-end dialogue system for knowledge grounded conversation. arXiv preprint arXiv:1709.04264.