SlowGuess/ACL_2020_OCR huggingface dataset:
Hao Zhou\*, Chujie Zheng\*, Kaili Huang, Minlie Huang† Xiaoyan Zhu
Department of Computer Science and Technology,
Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems,
{tuxchow, chujiezhengchn}@gmail.com, aihuang@tsinghua.edu.cn
The research of knowledge-driven conversational systems is largely limited due to the lack of dialog data which consists of multi-turn conversations on multiple topics and with knowledge annotations. In this paper, we propose a Chinese multi-domain knowledge-driven conversation dataset, KdConv, which grounds the topics in multi-turn conversations to knowledge graphs. Our corpus contains 4.5K conversations from three domains (film, music, and travel), and 86K utterances with an average turn number of 19.0. These conversations contain in-depth discussions on related topics and natural transition between multiple topics. To facilitate the following research on this corpus, we provide several benchmark models. Comparative results show that the models can be enhanced by introducing background knowledge, yet there is still a large space for leveraging knowledge to model multi-turn conversations for further research. Results also show that there are obvious performance differences between different domains, indicating that it is worth further explore transfer learning and domain adaptation. The corpus and benchmark models are publicly available<sup>1</sup>.
It has been a long-term goal of artificial intelligence to deliver human-like conversations, where background knowledge plays a crucial role in the success of conversational systems (Shang et al., 2015; Li et al., 2016a; Shao et al., 2017). In task-oriented dialog systems, background knowledge is defined as slot-value pairs, which provides key information for question answering or recommendation, and has been well defined and thoroughly studied (Wen et al., 2015; Zhou et al., 2016). In
open-domain conversational systems, it is important but challenging to leverage background knowledge, which is represented as either knowledge graphs (Zhu et al., 2017; Zhou et al., 2018a) or unstructured texts (Ghazvininejad et al., 2018), for making effective interactions.
Recently, a variety of knowledge-grounded conversation corpora have been proposed (Zhou et al., 2018b; Dinan et al., 2018; Moghe et al., 2018; Moon et al., 2019; Wu et al., 2019; Liu et al., 2018; Tuan et al., 2019; Qin et al., 2019) to fill the gap where previous datasets do not provide knowledge grounding of the conversations (Godfrey et al., 1992; Shang et al., 2015; Lowe et al., 2015). CMU DoG (Zhou et al., 2018b), India DoG (Moghe et al., 2018), and Wizard of Wikipedia (Dinan et al., 2018) demonstrate attempts for generating informative responses with topic-related Wikipedia articles. However, these datasets are not suitable for modeling topic transition or knowledge planning through multi-turn dialogs based on the relations of topics. OpenDialKG (Moon et al., 2019) and DuConv (Wu et al., 2019) use knowledge graphs as knowledge resources. Nevertheless, the number of topics is limited to one (Moon et al., 2019) or two (Wu et al., 2019), which is not sufficient for diversified topic transition in human-like conversations. Therefore, these knowledge-grounded dialog datasets still have limitations in modeling knowledge interactions<sup>2</sup> in multi-turn conversations.
In this paper, we propose KdConv, a Chinese multi-domain dataset towards multi-turn Knowledge-driven Conversation, which is suitable for modeling knowledge interactions in multi-turn human-like dialogues, including knowledge planning, knowledge grounding, knowledge adaptations, etc. KdConv contains 86K utterances and
2Refer to knowledge planning, knowledge grounding, knowledge adaptations in dialog systems.
<https://github.com/thu-coai/KdConv>
7098
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7098-7108 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Conversation (Music)	Knowledge Triple
Head Entity	Relation	Tail Entity
User1:知道《飞得更高》这首歌吗?Do you know the song 'Flying Higher'?
User2:知道呀,这首歌入选了中歌榜中国年度最受华人欢迎十大金曲。Yes, this song has been selected in the top ten most popular songs in China.	FlyingHigher	Information	... selected in the top ten mostpopular songs in China...
...
User1:具体的发行时间你记得吗?Do you remember the exact release date?
User2:记得,是在 2005 年 3 月 19 日。Yes. It is March 19, 2005.	FlyingHigher	Release date	March 19, 2005
User1:我觉得这首歌算是 汪峰的经典之曲。I think it is one of the classic songs of Wang Feng.	Originalsinger	Wang Feng
User2:我也那么认为,编曲填词都由他自己完成,真的算是经典之作了。So do I. The arrangement and lyrics of the music are all completed by himself.It's really a classic.	Arrangement
Lyrics
User1:说到他真的很了不起,在音乐方面获得很多大奖,我能说上来的就有第 12 届音乐风云榜年度最佳男歌手奖。He is really amazing and has won many awards in music, such as the 12th Music Awards of the Year Award for Best Male Singer.	Wang Feng	Mainachievements	The 12th Music Awards of theYear Award for Best Male Singer
...
User1:那他的歌曲除飞得更高,你还喜欢哪首?So which song do you like besides 'Flying Higher'?
User2:再喜欢的就是《怒放的生命》这首歌了,听的感觉特别好,减压。I like 'Blooming Life'. I feel great and decompression.	Wang Feng	Representativeworks	Fireworks, Brave Heart, FlyingHigher, Blooming Life...
User1:啊,这首歌我也很喜欢,也都是由他自己作词作曲并演唱。Oh, I like this song, too. He wrote and sang it by himself.	BloomingLife	Information	'Blooming Life' is a song sung, written and composed by Wang Feng... The song won the BestSong of the Year Award in the13th Global Chinese Music List.
User2:是的,该曲也获得了 13 届全球华语音乐榜中榜年度最佳歌曲奖。Yes, and the song also won the Best Song of the Year Award in the 13th Global Chinese Music List.
Knowledge Graph
Flying HigherOriginal singerWang FengRepresentative workBlooming LifeInformationMain achievementsInformation... selected in the topten most popular songsin China in 2015...March 19, 2005The 12th Music Awardsof the Year Award forBest Male Singer‘Blooming Life'is a song sung, written and composedby Wang Feng... The song won the Best Song of theYear Award in the 13th Global Chinese Music List.
Figure 1: An example in KdConv from the music domain. The underlined text is the related knowledge that is utilized in conversation. The italic text and circles are topics (refer to the distinct head entities in the knowledge triples and the central nodes with degree greater than 1 in the knowledge graph) in this dialogue.
4.5K dialogues in three domains, 1.5K dialogues for each domain (an example is shown in Figure 1). Each utterance is annotated with related knowledge facts in the knowledge graph, which can be used as supervision for knowledge interaction modeling. Furthermore, conversations of KdConv contain diversified topics ranged from one to four, without any pre-defined goals or constraints, which are closer to real human-human conversations than other datasets. The relations of topics are explicitly defined in the knowledge graph. Moreover, KdConv covers three domains, including film, music, and travel, which can be used to explore knowledge adaptation between different domains. We provide a benchmark to evaluate both generation- and retrieval-based conversational models on the
proposed dataset with/without access to the corresponding knowledge. Results show that knowledge grounding contributes to the improvement of these models, while existing models are still not strong enough to deliver knowledge-coherent conversations, indicating a large space for future work.
- We collect a new dataset, KdConv, for knowledge-driven conversation generation in Chinese. KdConv contains 86K utterances and 4.5K dialogues in three domains (film, music, and travel). The average turn number is about 19, remarkably longer than those in other corpora.
7099
Dataset	Language	Knowledge Type	Annotation Level	Domain	Avg. # turns	Avg. # topics	#utters
CMU DoG	English	Text	Sentence	Film	22.6	1.0	130K
WoW	English	Text	Sentence	Multiple	9.0	2.0	202K
India DoG	English	Text & Table	Sentence	Film	10.0	1.0	91K
OpenDialKG	English	Graph	Sentence	Film, Book, Sport, Music	5.8	1.0	91K
DuConv	Chinese	Text & Graph	Dialog	Film	9.1	2.0	270K
KdConv (ours)	Chinese	Text & Graph	Sentence	Film, Music, Travel	19.0	2.3	86K
Table 1: Comparison between our corpus and other human-labeled knowledge-grounded dialogue corpora.
- KdConv provides a benchmark to evaluate the ability of generating conversations with access to the corresponding knowledge in three domains. The corpus can empower the research of not only knowledge-grounded conversation generation, but also domain adaptation or transfer learning between similar domains (e.g., from film to music) or dissimilar domains (e.g., from music to travel).
- We provide benchmark models on this corpus to facilitate further research, and conduct extensive experiments. Results show that the models can be enhanced by introducing background knowledge, but there is still much room for further research. The corpus and the models are publicly available<sup>3</sup>.
Recently, open-domain conversation generation has been largely advanced due to the increase of publicly available dialogue data (Godfrey et al., 1992; Ritter et al., 2010; Shang et al., 2015; Lowe et al., 2015). However, the lack of annotation of background information or related knowledge results in significantly degenerated conversations, where the text is bland and strangely repetitive (Holtzman et al., 2019). These models produce conversations that are substantially different from those humans make, which largely rely on background knowledge.
To facilitate the development of conversational models that mimic human conversations, there have been several knowledge-grounded corpora proposed. Some datasets (Zhou et al., 2018b; Ghazvininejad et al., 2018; Liu et al., 2018; Tuan et al., 2019; Qin et al., 2019) collect dialogues and label the knowledge annotations using NER, string match, artificial scoring, and filtering rules
based on external knowledge resources (Liu et al., 2018). However, mismatches between dialogues and knowledge resources introduce noises to these datasets. To obtain the high-quality knowledge-grounded datasets, some studies construct dialogues from scratch with human annotators, based on the unstructured text or structured knowledge graphs. For instance, several datasets (Zhou et al., 2018b; Dinan et al., 2018; Gopalakrishnan et al., 2019) have human conversations where one or both participants have access to the unstructured text of related background knowledge, while OpenDialKG (Moon et al., 2019) and DuConv (Wu et al., 2019) build up their corpora based on structured knowledge graphs. In Table 1, we present a survey on existing human-labeled knowledge-grounded dialogue datasets.
CMU DoG (Zhou et al., 2018b) utilizes 30 Wikipedia articles about popular movies as grounded documents, which explores two scenarios: only one participant has access to the document, or both have. Also using Wikipedia articles, however, Wizard of Wikipedia (WoW) (Dinan et al., 2018) covers much more dialogue topics (up to 1,365), which puts forward a high demand for the generalization ability of dialog generation models. One other difference from CMU DoG is that in WoW, only one participant has access to an information retrieval system that shows the worker paragraphs from Wikipedia possibly relevant to the conversation, which is unobservable to the other. In addition to the unstructured text, India DoG (Moghe et al., 2018) uses fact tables as background resources.
The idea of using structured knowledge to construct dialogue data is also adopted in OpenDialKG (Moon et al., 2019), which has a similar setting to KdConv. OpenDialKG contains chit-chat conversations between two agents engaging in a dialog about a given topic. It uses the Freebase knowl
<sup>3</sup>https://github.com/thu-coai/KdConv
7100
Domain	Film	Music	Travel	Total
# entities	7,477	4,441	1,154	13,072
# start	559	421	476	1,456
# extended	6,917	4,020	678	11,615
# relations	4,939	4,169	7	9,115
# triples	89,618	56,438	10,973	157,029
Avg. # triples per entity	12.0	12.7	9.5	12.0
Avg. # tokens per triple	20.5	19.2	20.9	20.1
Avg. # chars per triple	51.6	45.2	39.9	48.5
edge base (Bast et al., 2014) as background knowledge. In OpenDialKG, the entities and relations that are mentioned in the dialog are annotated, and it also covers multiple domains (film, books, sports, and music). However, the limitation is that there are much fewer turns in a conversation, and the whole dialogue is restricted to only one given topic, which is not suitable for modeling topic transition in human-like conversations.
To the best of our knowledge, DuConv (Wu et al., 2019) is the only existing Chinese human-labeled knowledge-grounded dialogue dataset. DuConv also utilizes unstructured text like short comments and synopsis, and structured knowledge graphs as knowledge resources. Given the knowledge graph, it samples two linked entities, one as the transitional topic and the other as the goal topic, to construct a conversation path. This path is used to guide participants toward the goal of the dialogue, which, as argued in Wu et al. (2019), can guide a model to deliver proactive conversations. However, the existence of the target path is inconsistent with an open dialogue in reality because humans usually do not make any assumption about the final topic of a conversation. Beyond that, the knowledge graph and the goal knowledge path are only annotated for the whole dialogue, which cannot provide explicit supervision on knowledge interactions for conversational models.
KdConv is designed to collect open-domain multi-turn conversations for modeling knowledge interactions in human-like dialogues, including knowledge planning, knowledge grounding, knowledge adaptations, etc. However, the open-domain background or commonsense knowledge is too large in scale (e.g., there are over 8 million concepts and 21 million relations in ConceptNet (Speer and Havasi, 2013)). Thus, it is costly and time-consuming to
Table 2: Statistics of the knowledge graphs used in constructing KdConv (char represents character).
Domain	Film	Music	Travel	Total
# dialogues	1,500	4,500
# dialogues in Train/Dev/Test	1,200/150/150	3,600/450/450
# utterances	36,618	24,885	24,093	85,596
Avg. # utters per dialogue	24.4	16.6	16.1	19.0
Avg. # topics per dialogue	2.6	2.1	2.2	2.3
Avg. # tokens per utter	13.3	12.9	14.5	13.5
Avg. # characters per utter	20.4	19.5	22.9	20.8
Avg. # tokens per dialogue	323.9	214.7	233.5	257.4
Avg. # chars per dialogue	497.5	324.0	367.8	396.4
# entities	1,837	1,307	699	3,843
# start entities	559	421	476	1,456
# relations	318	331	7	656
# triples	11,875	5,747	5,287	22,909
Avg. # triples per dialogue	16.8	10.4	10.0	10.1
Avg. # tokens per triple	25.8	29.7	31.0	28.3
Avg. # chars per triple	49.4	56.8	57.4	53.6
Table 3: Statistics of KdConv (utter/char represents utterance/character respectively).
collect multi-turn conversations from scratch based on such large-scale knowledge. KdConv is proposed as one small step to achieve this goal, where we narrowed down the scale of background knowledge to several domains (film, music, and travel) and collected conversations based on the domain-specific knowledge. KdConv contains similar domains (film and music) and dissimilar domains (film and travel) so that it offers the possibility to investigate the generalization and transferability of knowledge-driven conversational models with transfer learning or meta learning(Gu et al., 2018; Mi et al., 2019).
In the following subsections, we will describe the two steps in data collection: (1) Constructing the domain-specific knowledge graph; (2) Collecting conversation utterances and knowledge interactions by crowdsourcing.
3.1 Knowledge Graph Construction
As the sparsity and the large scale of the knowledge were difficult to handle, we reduced the range of the domain-specific knowledge by crawling the most popular films and film stars, music and singers, and attractions as start entities, from several related websites for the  $\mathrm{film}^4/\mathrm{music}^5/\mathrm{travel}^6$  domain. The knowledge of these start entities contains both structured knowledge triples and unstructured knowledge texts, which make the task more general but challenging. After filtering the start entities which have few knowledge triples, the film/music/travel domain contains 559/421/476
4https://movie.douban.com/top250
5https://music.douban.com/top250
<https://travel.qunar.com/
p-cs299914-beijing-jingdian
7101
start entities, respectively.
After crawling and filtering the start entities, we built the knowledge graph for each domain. Given the start entities as seed, we retrieved their neighbor entities within three hops from XLORE, a large-scale English-Chinese bilingual knowledge graph (Wang et al., 2013). We merged the start entities and these retrieved entities (nodes in the graph) and relations (edges in the graph) into a domain-specific knowledge graph for film and music domains. For the travel domain, we built the knowledge graph with the knowledge crawled only from the Web, because XLORE provides little knowledge for start entities in the travel domain. There are two types of entities in the knowledge graph: one is the start entities crawled from the websites, the other is the extended entities that are retrieved from XLORE (film/music), or websites (travel) to provide related background knowledge. The statistics of the knowledge graphs used in constructing KdConv are provided in Table 2.
3.2 Dialogue Collection
We recruited crowdsourced annotators to generate multi-turn conversations that are related to the domain-specific knowledge graph without any predefined goals or constraints. During the conversation, two speakers both had access to the knowledge graph rather than that only one participant had access to the knowledge, as proposed in WoW (Dinan et al., 2018) where one party always leads the conversation with an expert-apprentice mode. Allowing two participants to access the knowledge, in our corpus the two parties can dynamically change their roles, as either leader or follower, which is more natural and real to human conversations. In addition to making dialogue utterances, the annotators were also required to record the related knowledge triples if they generated an utterance according to some triples. To increase the knowledge exposure in the collected conversations, the annotators were instructed to start the conversation based on one of the start entities, and they were also encouraged to shift the topic of the conversation to other entities in the knowledge graph. Thus, the topics of conversations and the knowledge interactions in KdConv are diversified and unconstrained. In order to ensure the naturalness of the generated conversations, we filtered out low-quality dialogues, which contain grammatical errors, inconsistencies of knowledge facts, etc. The distinct-4 score is
Figure 2: Statistics of the number of dialogues where at least  $k(k = 2,3,4)$  topics have been discussed in the first  $n$  turns. The proportions of dialogues that contain 3 or 4 topics become larger when the dialog turn becomes longer.
0.54/0.51/0.42 for the film/music/travel domain, which is comparable to the score of DuConv (Wu et al., 2019), 0.46. The distinct-4 score decreases, due to the decrease of knowledge triples and utterances in three domains, as shown in Table 3.
The detailed statistics of KdConv are shown in Table 3. We collect 1,500 dialogues for each domain. The training, validation, and test sets are partitioned with the ratio of 8:1:1. Note that the number of conversation turns in the film domain is larger than those in the music/travel domains (24.4 vs. 16.6/16.1), while the utterance lengths are similar (13.3 vs. 12.9/14.5 at the token level, and 20.4 vs. 19.5/22.9 at character level). As aforementioned, the dialogues in the real world are not limited to one or two topics, while discussing multiple topics in depth usually requires a conversation having enough number of turns. In order to verify this point, we analyze the relationship between the number of turns and the number of topics. Note that the topics are defined as the distinct head entities in the knowledge triples and the central nodes with a degree greater than 1 in the knowledge graph.
The results of three domains are shown in Figure 2. Given a number  $k(k = 2,3,4)$  of topics and a number  $n$  of conversation turns, we count the number of dialogues where at least  $k$  topics have been
7102
discussed in the first  $n$  turns. It can be observed that more topics tend to appear in a dialogue only if there are enough conversation turns. For instance, most dialogues involve at least 2 topics when the number of turns exceeds 15. This is consistent with the fact that if a conversation is very short, speakers will not be able to discuss in detail, let alone natural transition between multiple topics.
Topic Transition
1 Hop	T1-Major Work→T2T1-Star→T2T1-Director→T2
2 Hop	T1-Major Work→T2-Star→T3T1-Major Work→T2-Director→T3T1-Star→T2-Major Work→T3
3 Hop	T1-Major Work→T2-Star→T3-Major Work→T4T1-Star→T2-Major Work→T3-Director→T4T1-Major Work→T2-Star→T3-Information→T4
Table 4: Top-3 topic transition of the film domain, where  $T_{n}$  denotes the  $n$ -th topic of a dialog and  $T_{n} - X \rightarrow T_{n+1}$  represents the relation  $X$  between  $T_{n}$  and  $T_{n+1}$ .
To analyze topic transition in our dataset, we provide top-3 topic transition in the film domain, as shown in Table 4. As can be seen, topic transition has diverse patterns conditioned on different hops. With the increase of the hops of topic transition, the complexity of topic transition goes up. Compared to DuConv (Wu et al., 2019), the dialogues of KdConv contain multiple and diverse topics instead of fixed two topics, leading to diverse and complex topic transition, which are more suitable for the research of knowledge planning in human-like conversations. Note that the relation “ $-\text{Information} \rightarrow$ ” appeared in the last row is different from the other relations, which means the target topic is mentioned in unstructured texts describing the information about the source topic. The low frequency of the relation “ $-\text{Information} \rightarrow$ ” demonstrates that people prefer to shift the topic according to the structured relations rather than unstructured texts, as adopted in WoW (Dinan et al., 2018).
To provide benchmark models for knowledge-driven conversation modeling, we evaluated both generation- and retrieval-based models on our corpus. In order to explore the role of knowledge
annotation, we evaluated the models with/without access to the knowledge graph of our dataset.
4.1.1 Generation-based Models
Language Model (LM) (Bengio et al., 2003): We trained a language model that maximizes the log likelihood:  $\log \mathcal{P}(\pmb{x}) = \sum_{t}\log \mathcal{P}(x_{t}|x_{< t})$ , where  $\pmb{x}$  denotes a long sentence that sequentially concatenates all the utterances of a dialogue.
Seq2Seq (Sutskever et al., 2014): An encoder-decoder model augmented with attention mechanism (Bahdanau et al., 2014). The input of the encoder was the concatenation of the past  $k - 1$  utterances, while the target output of the decoder was the  $k$ -th utterance.  $k$  was set to 8 in the experiment. If there were fewer than  $k - 1$  sentences in the dialogue history, all the past utterances would be used as input.
HRED (Serban et al., 2016): A hierarchical recurrent encoder-decoder model that has a specific context RNN to incorporate historical conversational utterances into a context state, which is used as the initial hidden state of the decoder. The adapted model generates the  $k$ -th utterance based on the past  $k - 1$  utterances, where  $k$  was also set to 8, for fair comparison with Seq2Seq.
All the generative models were trained by optimizing the cross-entropy loss:
$$
\mathcal {L} _ {0} ^ {(g)} = - \frac {1}{T} \sum_ {t = 1} ^ {T} \log \mathcal {P} (\hat {x} _ {t} = x _ {t}),
$$
where  $\hat{x}_t$  denotes the predicted token at the time step  $t$ , while  $x_t$  is the  $t$ -th token of the target sentence.
4.1.2 Retrieval-based Model