SlowGuess/NAACL_2021_OCR huggingface dataset:
For DuConv, we set the vocabulary size to  $21,128^2$ . To fairly compare our model with all baselines, the number of hidden nodes is all set to 512 and the batch size set to 128. The max length of sentence is set to 30 and the max number of dialogue turns is set to 8. The topic size of LDA for WoW dataset is set as 50. We use Adam (Kingma and Ba, 2014) for gradient optimization in our experiments. The learning rate is set to 0.001. We run all models on the Tesla P40 GPU.
Evaluation Measures. We use both quantitative evaluation and human judgements in our experiments. Specifically, we use the indicators including BLEU-1/2 and distinct-1/2, Embedding metrics (average, extrema and greedy) $^3$ . We also measure the knowledge selection precision and F1 score between the prediction and ground-truth knowledge. For human evaluation, we randomly sampled 300 generated response and invited six annotators (all CS majors students) to give their rating score based on the relevant, informative and natural of the generated response with respect to the contexts. The rating ranges from 0 to 3 for relevance, informativeness and natural, respectively.
The metric-based evaluation results are shown in Table 1 and Table 2. From the results, we can see that the sequential knowledge models, i.e., SLKS and our SKT models, perform better than the traditional knowledge-grounded dialogue models, i.e., MemNet and PostKS models, in terms of BLEU and Distinct measures. That's because the sequential characteristic in knowledge is significant and beneficial for the knowledge selection process. Our proposed SKT-KG model obtains good results. Tak
$^{2}$ https://github.com/ymcui/Chinese-BERT-wwm
$^{3}$ https://github.com/Maluuba/nlg-eval
$^{1}$ https://github.com/google-research/bert
Dataset	WoW Test Seen	WoW Test Unseen
Model	BLEU-1 / 2	Dist-1	Dist-2	Avg.	Ext.	Gre.	BLEU-1 / 2	Dist-1	Dist-2	Avg.	Ext.	Gre.
Transformer	15.76/6.45	2.97	10.72	46.21	34.45	40.98	15.16/5.45	2.45	6.62	41.13	34.71	37.29
MemNet(soft)	16.67/6.67	3.65	11.28	48.23	40.37	44.19	14.72/4.81	2.15	16.18	42.38	35.74	38.53
PostKS(fusion)	17.21/6.98	5.67	21.85	53.36	39.25	45.17	15.61/5.38	2.87	15.18	44.24	38.69	40.38
SLKS	18.91/7.64	7.35	26.59	53.98	43.57	51.20	15.91/6.14	2.35	16.59	42.02	39.15	43.66
SKT	19.16/7.32	7.65	27.46	55.99	44.74	47.03	13.50/6.96	3.08	16.04	46.29	39.70	42.43
SKT-KG	20.62/7.36	7.79	28.32	59.71	48.87	54.26	16.26/6.99	3.69	16.83	52.85	41.07	45.39
Table 2: Automatic evaluation results on Wizard of Wikipedia datasets. The metrics Distinct, Average, Extrema, and Greedy are abbreviated as Dist, Avg., Ext., and Gre., respectively. The best results are highlighted with bold.
Dataset	Model	F1	klg Acc.
DuConv	MemNet (soft)	15.49	0.22
PostKS (fusion)	16.38	0.23
SLKS	17.62	0.26
SKT	17.75	0.29
SKT-KG	19.26	0.29
WoW Test Seen	MemNet (soft)	17.23	0.19
PostKS (fusion)	16.36	0.21
SLKS	18.91	0.23
SKT	19.25	0.26
SKT-KG	19.73	0.26
ing the BLEU-2 value on the DuConv dataset as an example, the BLEU-2 value of SKT-KG is 26.31, which is better than that of baseline models. The distinct-2 value of our model is also higher than other baseline models, indicating that our model can generate more diverse responses. For the unigram F1 score of the knowledge selection in Table 3, the F1 score of SKT-KG is 19.26, which is better than other models, showing that our model can extract more relevant and natural knowledge than baseline models. Compared with the ablation model SKT, we find that the pre-trained knowledge-aware response generator in our model can improve distinct measure and unigram F1 score, indicating that the model with pre-trained generator has ability to generate more diverse response. We also conducted a significant test. The experimental results show that the improvement of our model is significant in both datasets, i.e.,  $p$ -value  $< 0.01$ . In summary, our SKT-KG model is able to generate higher relevant and more diverse responses than the baselines.
The results of human evaluation are shown in Table 4. The rating scores are given to evaluate the relevance, informativeness and natural of the gen
Table 3: The unigram F1 score and knowledge selection accuracy between SKT-KG and other base-lines on two datasets. The klg stands for knowledge here.
Dataset	Model	Rel	Info	Nat	kappa
DuConv	MemNet(soft)	1.7	1.9	1.6	0.49
PostKS(fusion)	2.1	2.0	1.7	0.59
SLKS	1.9	2.2	2.1	0.42
SKT	2.2	2.1	1.9	0.47
SKT-KG	2.3	2.6	2.3	0.58
WoW Test Seen	MemNet(soft)	1.6	1.8	1.4	0.45
PostKS(fusion)	1.7	2.0	1.6	0.51
SLKS	1.9	2.3	1.7	0.49
SKT	2.0	1.9	1.6	0.44
SKT-KG	2.0	2.2	1.9	0.46
Table 4: Human evaluation between SKT-KG and other baselines on DuConv and WoW test seen datasets.
erated responses. From the experimental results, the relevance (Rel), information (Info) and natural (Nat) score for our model is greater than that of MemNet, PostKS and SLKS, indicating that our SKT-KG model is better than the baseline methods. Taking DuConv as an example, the score of relevance and informativeness in SKT-KG are 2.3 and 2.6, respectively, while the SLKS are 2.2 and 2.1, indicating that our model can generate more informative response than SLKS. In addition, for the natural comparison, the score of SKT-KG is 2.3, which is larger than SLKS i.e.,2.1, showing that the high-level knowledge transition is effective for the knowledge-grounded dialogue generation task and our SKT-KG model can generate more natural response with more information. The Kappa (Fleiss, 1971) value demonstrates the consistency of different annotators. We also conducted a significant test, and the improvement of our model is significant on both datasets, i.e.,  $p$ -value  $< 0.01$ .
4.2.3 Case study
To facilitate a better understanding of our model, we present some examples in Figure 5. From the multi-turn dialogues, we can see that the knowledge topic is from 'reviews of Mengyao Xi', to the 'master work of her', and then to the 'master work of Sui He'. The knowledge tag of ground-truth
Model	DuConv
P@1	R@1	F@1	P@2	R@2	F@2	P@5	R@5	F@5
PostKS(fusion)	0.23	0.19	0.21	0.23	0.33	0.27	0.22	0.71	0.34
SLKS	0.26	0.21	0.23	0.25	0.35	0.29	0.25	0.74	0.37
SKT-KG	0.29	0.22	0.25	0.27	0.38	0.32	0.26	0.77	0.39
Model	WoW Test Seen
P@1	R@1	F@1	P@2	R@2	F@2	P@5	R@5	F@5
PostKS(fusion)	0.21	0.17	0.19	0.22	0.29	0.25	0.19	0.67	0.30
SLKS	0.23	0.19	0.20	0.21	0.34	0.26	0.19	0.69	0.30
SKT-KG	0.26	0.21	0.23	0.25	0.35	0.29	0.21	0.73	0.33
Table 5: The ranking evaluation of knowledge selection on DuConv and WoW datasets.
Knowledge Pool	奚梦瑶 Meng-yao Xi	代表作 Masterpiece	维多利亚的秘密 Victoria's Secret	Multi-turn Dialogue
评论 Review	演技一般 Not good skill	A:奚梦瑶你认识吗?演技好一般啊。(Meng-yao Xi, Do you know her? Not very good skill.)B:认识,和你说的一样哈哈。(Yes, I know her, as you said.)A:不管怎么说,她也还是参加过维多利亚秘密秀的人。(However, she has attended the Victoria's Secret show.)B:对了,我记得一个浙江温州的女孩也参加过这个。(Yes, I remember another girl from Wenzhou, Zhejiang also attended.)A:是的,叫何穗,厉害许多。(Yes, it's Sui He, much better!)
祖籍 Homeland	中国上海 Shanghai, China
毕业院校 School	东华大学 East China Univ.
性别 Sex	女 Female
身高 Height	178厘米 178 cm	Ground Truth	哈哈,她可被誉为中国天使呢。(Haha, she was named as the Angel of China.)
何穗 Sui He	代表作 Masterpiece	维多利亚的秘密 Victoria's Secret	MemNet	她还是一个明星呢。(She is also a super star.)
评论 Review	中国天使 Angel of China	PostKS	是的,她的身高是178厘米.(Yes, her height is 178 cm.)
职业 Job	明星 Star	SLKS	对,她是中国天使。(Yes, she is the angel of China.)
祖籍 Homeland	浙江温州 Wenzhou, Zhejiang	SKT-KG	何穗可是被称作中国天使的女孩呢。(Sui He was the girl named as the angel of China.)
Figure 5: The case of generated response from different models on DuConv.
is the 'reviews of Sui He'. From the generation results, we can see that the sequential-based model performs better than the selection model, i.e., MemNet and PostKS. Taking an example in Figure 5, an un-natural response is generated by MemNet and PostKS, such as 'Area of Sui He' and 'Height of Sui He'. However, the sequential model can generate more natural and relevant responses, such as 'Yes, she is the angel of China' and 'He Sui was the girl named as the angel of China'. This is mainly because the sequential model is able to locate the 'reviews' knowledge which is more natural for the contexts. Moreover, our high-level transition model with pre-trained knowledge-aware response generator can generate more informative response than SLKS, as shown in Figure 5.
4.3 Analysis on Knowledge Selection
To verify whether the performance improvements are owing to the knowledge transition module, we conduct a further data analysis. Specifically, we randomly sample 300 examples from the DuConv dataset and WoW dataset, to evaluate the performance of the knowledge selection process in base
lines and our model. As knowledge-grounded dialogue models will select the relevant knowledge from the candidate knowledge set based on the dialogue contexts, we can treat it as a ranking task. Ranking evaluation measures, such as the precision, recall and F1 score, are used for quantitative evaluations. Then we calculate the precision, recall and F1 score of the top 1,2,5 for PostKS, SLKS and our SKT-KG model. The results are shown in Table 5. We can see that the sequential knowledge selection models, such as SLKS and SKT-KG, perform better than traditional selection model, i.e., PostKS, validating the effectiveness of sequential knowledge model. These results indicate that our proposed knowledge sequential transition module is capable to select out more relevant knowledge content than baseline models.
In this paper, we propose a sequential knowledge transition model with knowledge-aware response generator to model the high-level knowledge transition and fully utilize the low-resource knowledge data. SKT-KG models can abstract knowledge into
tags which leads our model easily to apply into both the structured and unstructured knowledge-grounded conversations. Besides, we propose a pre-trained knowledge-aware response generator, aiming at generating a natural sentence based on a given knowledge, to make full use of the limited data. Experimental results on both structured and unstructured knowledge-grounded dialogue datasets show that our SKT-KG model outperforms baseline models. As for future work, we intend to apply variational autoencoder to unstructured dataset, in order to empower models to learn the knowledge topic by themselves.
The authors would like to thank all the anonymous reviewers for their constructive comments and suggestions. This work was partially supported by the National Key R&D Program of China under Grants No. 2019AAA0105200, 2016QY02D0405, the Beijing Academy of Artificial Intelligence (BAAI) (No. BAAI2020ZJ0303), the National Natural Science Foundation of China (NSFC) (No. 61722211, 61773362, 61872338, 61906180).
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder for english. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169-174.
Hongshen Chen, Zhaochun Ren, Jiliang Tang, Yihong Eric Zhao, and Dawei Yin. 2018. Hierarchical variational memory network for dialogue generation. In Proceedings of the 2018 World Wide Web Conference, pages 1653-1662.
Meng Chen, Ruixue Liu, Lei Shen, Shaozu Yuan, Jingyan Zhou, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020. The JDDC corpus: A large-scale multi-turn Chinese dialogue dataset for E-commerce customer service. In LREC, pages 459-466.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. ICLR.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. American Psychological Association.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Thirty-Second AAAI Conference on Artificial Intelligence.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
Byeongchang Kim, Jaewoo Ahn, and Gunhee Kim. 2020. Sequential latent knowledge selection for knowledge-grounded dialogue. arXiv preprint arXiv:2002.07510.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055.
Linxiao Li, Can Xu, Wei Wu, Yufan Zhao, Xueliang Zhao, and Chongyang Tao. 2020. Zero-resource knowledge-grounded dialogue generation. Advances in Neural Information Processing Systems, 33.
Rongzhong Lian, Min Xie, Fan Wang, Jinhua Peng, and Hua Wu. 2019. Learning to select knowledge for response generation in dialog systems. arXiv preprint arXiv:1902.04911.
Ruixue Liu, Meng Chen, Hang Liu, Lei Shen, Yang Song, and Xiaodong He. 2020. Enhancing multi-turn dialogue modeling with intent information for e-commerce customer service. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 65-77. Springer.
Shuman Liu, Hongshen Chen, Zhaochun Ren, Yang Feng, Qun Liu, and Dawei Yin. 2018. Knowledge diffusion for neural dialogue generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1498.
Yinong Long, Jianan Wang, Zhen Xu, Zongsheng Wang, Baoxun Wang, and Zhuoran Wang. 2017. A knowledge enhanced generative conversational service agent. In Proceedings of the 6th Dialog System Technology Challenges (DSTC6) Workshop.
J Mayo, O White, and Hans J Eysenck. 1978. An empirical study of the relation between astrological factors and personality. The Journal of Social Psychology, 105(2):229-236.
Laura Miller. 2014. The divination arts in girl culture.
Zheng-Yu Niu, Hua Wu, Haifeng Wang, et al. 2019. Knowledge aware conversation generation with explainable reasoning over augmented graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1782-1792.
Haoruo Peng, Qiang Ning, and Dan Roth. 2019. Knowsemlm: A knowledge infused semantic language model. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 550-562.
Pengjie Ren, Zhumin Chen, Christof Monz, Jun Ma, and Maarten de Rijke. 2020. Thinking globally, acting locally: Distantly supervised global-to-local knowledge selection for background based conversation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8697-8704.
Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Now Publishers Inc.
Lei Shen and Yang Feng. 2020. CDL: Curriculum dual learning for emotion-controllable response generation. In ACL, pages 556-566.
Lei Shen, Yang Feng, and Haolan Zhan. 2019. Modeling semantic relationship in multi-turn conversations with hierarchical latent variables. In ACL, pages 5497-5502.
Lei Shen, Haolan Zhan, Xin Shen, and Yang Feng. 2021. Learning to select context in a hierarchical and global perspective for open-domain dialogue generation. arXiv preprint arXiv:2102.09282.
Hui Su, Xiaoyu Shen, Sanqiang Zhao, Zhou Xiao, Pengwei Hu, Randy Zhong, Cheng Niu, and Jie Zhou. 2020. Diversifying dialogue generation with non-conversational text. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7087-7097.
Yajing Sun, Yue Hu, Luxi Xing, Jing Yu, and Yuqiang Xie. 2020. History-adaption knowledge incorporation mechanism for multi-turn dialogue system. In AAAI, pages 8944-8951.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Pavlos Vougiouklis, Jonathon Hare, and Elena Simperl. 2016. A neural network approach for knowledge-driven response generation. In COLING.
Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019. Proactive human-machine conversation with explicit conversation goal. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3794-3804.
Jun Xu, Haifeng Wang, Zhengyu Niu, Hua Wu, and Wanxiang Che. 2020. Knowledge graph grounded goal planning for open-domain conversation generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9338-9345.
Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Aug-mentation end-to-end dialogue systems with commonsense knowledge. In Thirty-Second AAAI Conference on Artificial Intelligence.
Hainan Zhang, Yanyan Lan, Jiafeng Guo, Jun Xu, and Xueqi Cheng. 2018a. Reinforcing coherence for sequence to sequence model in dialogue generation. In *IJCAI*, pages 4567-4573.
Hainan Zhang, Yanyan Lan, Jiafeng Guo, Jun Xu, and Xueqi Cheng. 2018b. Reinforcing coherence for sequence to sequence model in dialogue generation. In International Joint Conference on Artificial Intelligence, pages 4567-4573.
Hainan Zhang, Yanyan Lan, Liang Pang, Hongshen Chen, Zhuoye Ding, and Dawei Yin. 2020. Modeling topical relevance for multi-turn dialogue generation. arXiv preprint arXiv:2009.12735.
Hainan Zhang, Yanyan Lan, Liang Pang, Jiafeng Guo, and Xueqi Cheng. 2019. Recosa: Detecting the relevant contexts with self-attention for multi-turn dialogue generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3721-3730.
Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, and Rui Yan. 2020. Low-resource knowledge-grounded dialogue generation. arXiv preprint arXiv:2002.10348.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In *IJCAI*, pages 4623-4629.