SlowGuess/ACL_2020_OCR huggingface dataset:
Sixing  $\mathbf{W u}^{1}$ , Ying  $\mathbf{L i}^{2*}$ , Dawei Zhang $^{1}$ , Yang Zhou $^{3}$  and Zhonghai  $\mathbf{W u}^{2}$
$^{1}$ School of Electronics Engineering and Computer Science, Peking University, Beijing, 100871, China
$^{2}$ National Research Center of Software Engineering,
Peking University, Beijing, 100871, China
3Auburn University, Auburn, Alabama, 36849, USA
{wusixing, li.ying, daweizhang}@pku.edu.cn
yangzhou@auburn.edu, zhwu@ss.pku.edu.cn
Generative dialogue systems tend to produce generic responses, which often leads to boring conversations. For alleviating this issue, Recent studies proposed to retrieve and introduce knowledge facts from knowledge graphs. While this paradigm works to a certain extent, it usually retrieves knowledge facts only based on the entity word itself, without considering the specific dialogue context. Thus, the introduction of the context-irrelevant knowledge facts can impact the quality of generations. To this end, this paper proposes a novel commonsense knowledge-aware dialogue generation model, ConKADI. We design a Felicitous Fact mechanism to help the model focus on the knowledge facts that are highly relevant to the context; furthermore, two techniques, Context-Knowledge Fusion and Flexible Mode Fusion are proposed to facilitate the integration of the knowledge in the ConKADI. We collect and build a large-scale Chinese dataset aligned with the commonsense knowledge for dialogue generation. Extensive evaluations over both an open-released English dataset and our Chinese dataset demonstrate that our approach ConKADI outperforms the state-of-the-art approach CCM, in most experiments.
Nowadays, open-domain dialogue response generation systems have shown impressive potential, to endow a machine with the ability to converse with a human, using natural language (Chen et al., 2017). Although such models have achieved promising performance, they still suffer from generating generic and boring responses, such as "I don't know." Such low-quality responses always reduce the attractiveness of generative dialogue systems to end-users. Researchers have tried to
tackle it from multiple aspects; for example, using the enhanced objective function (Li et al., 2016a); introducing additional contents (Xu et al., 2019). However, these methods haven't solved the issue thoroughly. Different from a human being, who is capable of associating the dialogue with the background knowledge in his/her mind, a machine can merely capture limited information from the surface text of the query message (Ghazvininejad et al., 2018). Consequently, it is difficult for a machine to understand the query fully, and then to generate diverse and informative responses (Zhou et al., 2018).
To bridge the gap of the knowledge between the human and the machine, researchers have begun to introduce large-scale knowledge graphs for enhancing the dialogue generation (Zhu et al., 2017; Zhou et al., 2018; Liu et al., 2018), and they have obtained lots of impressive results. Generally, the retrieval of knowledge facts is based on the entity name; in detail, the first step is to recognize entity words in the given query message, and then facts that contain the mentioned entities can be retrieved as candidates<sup>1</sup>. Subsequently, a knowledge-aware response can be generated based on the query message and previously retrieved facts. Although such a straightforward paradigm works to a certain extent, some challenges in knowledge-aware dialogue generation still keep unsolved. 1) An entity word usually can refer to different concepts, i.e., an entity has multiple meanings, but only one specific concept is involved in a particular context. Without considering this, some pre-fetched knowledge fact candidates can be irrelevant to the context. 2) Even if we only consider a particular entity meaning, the related knowledge facts may cover various target topics. However, some of those topics do not con
For example, for a mentioned entity "apple" in a query, the fact (apple, is a type of, fruit) or (fruit, related to, apple) can be retrieved.
Corresponding author: Ying Li, li.ying@pku.edu.cn
5811
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5811-5820 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Message: Apple's new product is awesome!
Figure 1: An illustrative example. #1 shows the response generated with a highly relevant fact, #2 shows the response generated with irrelevant facts.
tribute to the dialogue generation. Figure 1 presents an illustrative example to demonstrate such two issues. Here, a subgraph is retrieved based on the entity word "Apple" in the query. In general, "Apple" can be interpreted as either a type of fruit or a brand name. In this context, it is evident that "Apple" refers to a brand name. However, some knowledge facts concerning a type of fruit are retrieved too. If a model makes an inappropriate choice of irrelevant facts, the generated response will make no sense to the query message. In our example, even for the entities in blue circle related to the brand name "Apple", only some of them have a positive effect in the dialogue generation, e.g., "Jobs" should not make any contribution to the "#1". 3) The integration of the knowledge and the dialogue generation in previous approaches is insufficient, including the way of integration, as well as the types of knowledge.
To tackle such challenges, this paper proposes a Context Knowledge-Aware Diverse and Informative conversation generation model, ConKADI. First, we design a Felicitous Fact mechanism to help the model highlight the knowledge facts that are highly relevant to the context, that is, "Felicitous Facts". Felicitous Fact mechanism generates a felicitous fact probability distribution over the retrieved facts. For improving the selection of felicitous facts, human-generated answers (i.e., the ground-truth responses) are used as the posterior context knowledge to supervise the training of the prior felicitous fact probability distribution. Next, Context-Knowledge Fusion is proposed to lift the role of knowledge facts in the dialogue generation, by fusing the context and the felicitous knowledge before the decoding. Last, ConKADI can generate three types of words owing to the Flexible Mode
Fusion module, which aims at simultaneously fusing multiple types of knowledge. To summarize, Felicitous Fact mechanism can alleviate the first two issues, and the next two techniques solve the last issue. Consequently, our approach can improve the utilization rate of knowledge graphs, as well as can promote the diversity and informativeness of the generated responses.
In the experiments, a large-scale Chinese Weibo dataset is collected and aligned with the commonsense knowledge for dialogue generation. We perform extensive evaluations on two large-scale datasets: an open-released English Reddit dataset and our proposed Chinese Weibo dataset. The experimental results demonstrate that our proposed ConKADI model significantly outperforms representative methods in knowledge utilization, diversity, and informativeness. Especially, ConKADI exceeds the latest knowledge-aware dialogue generation model, CCM (Zhou et al., 2018), in most experiments.
Seq2Seq (Sutskever et al., 2014; Vinyals and Le, 2015) has been widely used in the open-domain dialogue generation. However, models tend to generate generic responses (Serban et al., 2016). To tackle this issue, researchers have proposed new objectives (Li et al., 2016a), enhanced decoding algorithms (Li et al., 2016b), latent-variable based methods (Zhao et al., 2017, 2018; Gao et al., 2019). Introducing additional contents into the dialogue generation is also helpful. (Xu et al., 2019) uses meta-words; (Zhu et al., 2019) uses the retrieved existing dialogues. However, the leading cause of generating generic responses is that the model can not obtain enough background knowledge from the query message (Ghazvininejad et al., 2018; Liu et al., 2019).
Recently, to alleviate the lack of background knowledge, researchers have begun to introduce the knowledge into the generation. The knowledge can be the unstructured knowledge texts (Ghazvininejad et al., 2018), the structured knowledge graphs (Zhou et al., 2018), or the hybrid of them (Liu et al., 2019). The structured knowledge has the best quality, because it is generally extracted and summarized by the human. The structured knowledge graph can be either domain-specific (Zhu et al., 2017; Liu et al., 2018) or open-domain (Young et al., 2018; Zhou et al., 2018). ConceptNet (Speer
5812
et al., 2017) is a multilingual open-domain commonsense knowledge graph, which is designed to represent the general knowledge and to improve understanding of the meanings behind the words people use. Two previous studies (Young et al., 2018; Zhou et al., 2018) have proved the feasibility of introducing commonsense knowledge into dialogue systems. The first work (Young et al., 2018) is designed for retrieval-based systems; therefore, only the current state-of-the-art CCM (Zhou et al., 2018) is our direct competitor. In comparison with CCM, 1) ConKADI is aware of the context when using the knowledge. 2) ConKADI uses human's responses as posterior knowledge in training.
In addition, our Felicitous Fact mechanism is different from the word/knowledge selection mechanisms previously proposed in related tasks; for example, selecting a cue word (Mou et al., 2016; Yao et al., 2017) or selecting a knowledge (Liu et al., 2019). First, ConKADI can access more contextual information because our model is fully end-to-end, while previous works use independent and external modules. Second, our Felicitous Fact outputs a probabilistic distribution instead of a hard singleton value, as did the previous works.
3.1 Task Formulation and Model Overview
Formally, given a training data  $\mathcal{D}$  of triplets, where each triplet includes a query message  $X = (x_{1},\ldots ,x_{n})$ , a response  $Y = (y_{1},\ldots ,y_{m})$ , and a set of commonsense knowledge facts  $F = \{f_1,\dots ,f_l\}$ . The training goal of knowledge-aware dialogue generation is to maximize the probability  $\sum_{(X,Y,F)\in \mathcal{D}}\frac{1}{|\mathcal{D}|} p(Y|X,F)$ ; the inference goal is to find  $Y^{*} = \arg \max_{Y}p(Y|X,F)$ . Knowledge facts  $F$  are retrieved from the knowledge graph  $\mathcal{G}$ ; each fact is organized as a triplet  $(h,r,t)$ .
The overview of ConKADI has been shown in Figure 2. Knowledge fact set  $F$  is retrieved by the Knowledge Retriever given the query message  $X$ . The Context Encoder summarizes an utterance into contextual representations. The Felicitous Fact Recognizer calculates the felicitous fact probability distribution  $\mathbf{z}$  over the  $F$ , which is used to initialize the Decoder and guide the generation. The Triple Knowledge Decoder can generate three types of words: vocabulary words, entity words, and copied words, with the Flexible Mode Fusion.
3.2 Felicitous Fact mechanism
Knowledge Retriever: Given a query message  $X$ , if a word  $x_{i} \in X$  is recognized as an entity word and can be matched to a vertex  $e_{src}$  in the knowledge graph  $\mathcal{G}$ , then, each neighbour  $e_{tgt} \in Neighbour(e_{src})$  and the corresponding directional relation  $r$  is retrieved as a candidate fact  $f$ .  $e_{src} / e_{tgt}$  is called as source/target entity. If a word can't match any vertex, a special fact  $f_{NAF}$  will be used.
Context Encoder: The Context Encoder is a bidirectional GRU network (Cho et al., 2014), which reads  $X$  or  $Y$  and outputs a contextual state sequence. For simplicity, we take  $X$  as an example. At the time step  $t$ , the Encoder outputs a forward state and a backward state, the concatenation of such two states  $\mathbf{h}_{\mathbf{t}}^{\mathbf{x}} = [\mathbf{h}_{\mathbf{t}}^{\mathbf{fw}}; \mathbf{h}_{\mathbf{t}}^{\mathbf{bw}}] \in \mathbb{R}^{2d_h \times 1}$  is regarded as the contextual state:
$$
\begin{array}{r} \mathbf {h} _ {\mathbf {t}} ^ {\mathbf {f w}} = G R U ^ {f w} (\mathbf {h} _ {\mathbf {t - 1}} ^ {\mathbf {f w}}, \mathbf {x} _ {\mathbf {t}}, \mathbf {e} _ {\mathbf {x} _ {\mathbf {t}}}) \\ \mathbf {h} _ {\mathbf {t}} ^ {\mathbf {b w}} = G R U ^ {b w} (\mathbf {h} _ {\mathbf {t - 1}} ^ {\mathbf {b w}}, \mathbf {x} _ {\mathbf {n - t + 1}}, \mathbf {e} _ {\mathbf {x} _ {\mathbf {n - t + 1}}}) \end{array} \tag {1}
$$
where  $\mathbf{x_t}$  is the word embedding of  $x_{t}$ . To enhance the semantic information, the matched entity embedding  $\mathbf{e}_{\mathbf{x_t}}$  of  $x_{t}$  is also involved. Finally, the contextual state sequence of  $X / Y$  is denoted as  $H^{x / y} = (h_1^{x / y},\dots ,h_{n / m}^{x / y})$ . Specifically,  $H^{x}$  is the prior context;  $H^{y}$  is the posterior context that is only available in the training stage.
Felicitous Fact Recognizer: Recall the example illustrated in Figure 1, some preliminary retrieved knowledge facts may be inappropriate in the dialogue context. The Felicitous Fact Recognizer is designed to detect the facts that highly coincide with the dialogue context, i.e., Felicitous Facts. The Felicitous Fact Recognizer reads the contextual information, then outputs a probability distribution  $\mathbf{z} \in \mathbb{R}^{l \times 1}$  over the  $F$ ; therefore, the  $i$ -th dimension value  $\mathbf{z}[i]$  indicates the weight of  $f_{i}$ . In the training stage, the high-quality human-generated response  $Y$  is served as the posterior knowledge; hence, the posterior  $\mathbf{z}_{\mathrm{post}}$  is adopted in training, the prior  $\mathbf{z}_{\mathrm{prior}}$  is adopted in inference:
$$
\mathbf {z _ {p o s t}} = \eta (\varphi (\mathbf {F} \cdot \mathbf {W _ {f t}}) \cdot \varphi ([ \mathbf {h _ {n} ^ {x}} ^ {\top}; \mathbf {h _ {m} ^ {y}} ^ {\top} ] \cdot \mathbf {W _ {p o s t}})) ^ {\top}
$$
$$
\mathbf {z} _ {\text {p r i o r}} = \eta \left(\varphi \left(\mathbf {F} \cdot \mathbf {W} _ {\mathrm {f t}}\right) \cdot \varphi \left(\mathbf {h} _ {\mathbf {n}} ^ {\mathbf {x} ^ {\top}} \cdot \mathbf {W} _ {\text {p r i o r}}\right)\right) ^ {\top} \tag {2}
$$
where  $\mathbf{F} \in \mathbb{R}^{l \times (d_e + d_r + d_e)}$  is the embedding matrix of the retrieved facts  $F$ ,  $\mathbf{W}_{\mathrm{ft}}$ ,  $\mathbf{W}_{\mathrm{post}}$  and  $\mathbf{W}_{\mathrm{prior}}$  are trainable parameters,  $\eta$  is softmax activation,
5813
Figure 2: An overview of the proposed approach ConKADI.
$\varphi$  is tanh activation. Kullback-Leibler Divergence (Kullback and Leibler, 1951) (KLD) is used to force two distributions to become as close as possible.
$$
\mathcal {L} _ {k} = \operatorname {K L D} \left(\mathbf {z} _ {\text {p o s t}}, \mathbf {z} _ {\text {p r i o r}}\right) \tag {3}
$$
Context-Knowledge Fusion: To enhance the Decoder's understanding of the background knowledge, the Decoder is initialized based on the fused knowledge  $\mathbf{f}_{\mathbf{z}}^{\top} = \mathbf{z}^{\top}\cdot \mathbf{F}$  and the query context:
$$
\mathbf {h} _ {\mathbf {0}} ^ {\mathbf {y} ^ {\top}} = \tanh  ([ \mathbf {h} _ {\mathbf {n}} ^ {\mathbf {x} ^ {\top}}; \mathbf {f} _ {\mathbf {z}} ^ {\top} ] \cdot \mathbf {W} _ {\mathbf {i n i t}}) \qquad (4)
$$
where  $\mathbf{W}_{\mathrm{init}}$  is a trainable parameter.
Following the previous work (Zhao et al., 2017), we adopt the Bag-of-Words Loss to ensure the accuracy of the input of the Context-Knowledge Fusion, namely,  $\mathbf{h}_{\mathbf{n}}^{\mathbf{x}}$  and  $\mathbf{f}_{\mathbf{z}}$ . Meanwhile, we construct a 0-1 indicator vector  $\mathbf{I}_{\mathbf{f}} \in \mathbb{R}^{l \times 1}$  to supervise the training of  $\mathbf{z}_{\mathrm{post}}$ , where  $\mathbf{I}_{\mathbf{f}}[i]$  is set to 1 if the target entity of the  $i$ -th fact  $f_{i}$  appears in the  $Y$ , otherwise 0. Thus, the objective is to minimize the  $\mathcal{L}_f$  given by:
$$
- \frac {\sum_ {y _ {b} \in B} \log p _ {b} \left(y _ {b} \left| \mathbf {h} _ {\mathbf {n}} ^ {\mathbf {x}}, \mathbf {f} _ {\mathbf {z}}\right)\right)}{| B |} - \frac {\mathbf {I} _ {\mathbf {f}} ^ {\top} \cdot \log \left(\mathbf {z} _ {\text {p o s t}}\right)}{\left| \mathbf {I} _ {\mathbf {f}} \right|} \tag {5}
$$
where  $\mathbf{B}$  is the word bag of  $Y$ ,  $p_b$  is a 2-layer  $MLP_{bow}$  activated by softmax, which outputs the probability distribution over the vocabulary  $V$ .
3.3 Triple Knowledge Decoder
The Decoder is another GRU network. At each time step, the Decoder can generate one of three
types of words: vocabulary words, knowledgeable entity words, and copied words. ConKADI first updates the internal state:
$$
\mathbf {h} _ {\mathbf {t}} ^ {\mathbf {y}} = g (\mathbf {h} _ {\mathbf {t} - \mathbf {1}} ^ {\mathbf {y}}, \mathbf {u} _ {\mathbf {t} - \mathbf {1}}, \mathbf {c} _ {\mathbf {t} - \mathbf {1}}) \qquad (6)
$$
where  $\mathbf{u}_{\mathbf{t} - 1}^{\top} = [\mathbf{y}_{\mathbf{t} - 1}^{\top};\mathbf{e}_{\mathbf{y}_{\mathbf{t} - 1}}^{\top};\mathbf{h}_{\mathbf{y}_{\mathbf{t} - 1}}^{\mathbf{x}}]^{\top}]$ , and  $\mathbf{y}_{\mathbf{t} - 1}$ ,  $\mathbf{e}_{\mathbf{y}_{\mathbf{t} - 1}}$ ,  $\mathbf{h}_{\mathbf{y}_{\mathbf{t} - 1}}^{\mathbf{x}}$  are the word embedding, the entity embedding and the pointed-then-copied source state of the last predicted token  $y_{t - 1}$ , respectively; and  $\mathbf{c}_{\mathbf{t} - 1}$  is the Attention  $^2$ .
Vocabulary Words: The probability distribution  $p_{w,t} \in \mathbb{R}^{|V| \times 1}$  over the  $V$  is given by:
$$
p _ {w, t} ^ {\top} = \eta (e l u ([ \mathbf {h} _ {\mathbf {t}} ^ {\mathbf {y} ^ {\top}}; \mathbf {u} _ {\mathbf {t} - \mathbf {1}} ^ {\top}; \mathbf {c} _ {\mathbf {t}} ^ {\top} ] \cdot \mathbf {W} _ {\mathbf {v 1}}) \cdot \mathbf {W} _ {\mathbf {v 2}}) \tag {7}
$$
where  $\mathbf{W}_{\mathbf{v1} / 2}$  are trainable parameters, and the non-linear activation elu is proposed by (Clevert et al., 2016).
Knowledgeable Entity Words: An entity word can be generated by extracting the target entity of the best-matched fact  $f$  at each time step. The corresponding probability distribution  $p_{k,t} \in \mathbb{R}^{l \times 1}$  over the  $F$  is calculated as:
$$
\mathbf {z} _ {\mathbf {d}, \mathbf {t}} = \eta (\varphi (\mathbf {F} \cdot \mathbf {W} _ {\mathbf {f d}}) \cdot \varphi ([ \mathbf {h} _ {\mathbf {t}} ^ {\mathbf {y} ^ {\top}}; \mathbf {u} _ {\mathbf {t} - \mathbf {1}} ^ {\top} ] \cdot \mathbf {W} _ {\mathbf {d}}) ^ {\top})
$$
$$
\gamma_ {t} = s i g m o i d ([ \mathbf {h _ {t} ^ {y}} ^ {\top}; \mathbf {u _ {t}} ^ {\top}; \mathbf {c _ {t}} ^ {\top} ] \cdot \mathbf {W _ {g a t e}}) \in \mathbb {R} ^ {1}
$$
$$
p _ {k, t} = \gamma_ {t} \times \mathbf {z} + (1. 0 - \gamma_ {t}) \times \mathbf {z} _ {\mathbf {d}} \tag {8}
$$
2We have omitted the description of Attention. Please see (Luong et al., 2015) for the detail.
where the previous  $\mathbf{z}$  here serves as a static global distribution (denoted as GlFact),  $\mathbf{z}_{\mathbf{d},\mathbf{t}}$  is the dynamic distribution, and  $\gamma_{t}$  is a gate to control the contribution of each distribution.
Copied Words: The Decoder can further point out a word  $x$  from  $X$ , and then copies the  $x$ . The corresponding probability distribution  $p_{c,t} \in \mathbb{R}^{n \times 1}$  over the query message  $X$  is calculated as:
$$
\begin{array}{r l} p _ {c, t} = & \eta (\varphi \left(\mathbf {H} ^ {\mathbf {x}} \cdot \mathbf {W} _ {\mathbf {c s}}\right) \cdot \varphi \left(\mathbf {u} _ {\mathbf {t}} ^ {\mathbf {c} ^ {\top}} \cdot \mathbf {W} _ {\mathbf {c t}}\right) ^ {\top}) \\ & \mathbf {u} _ {\mathbf {t}} ^ {\mathbf {c} ^ {\top}} = \left[ \mathbf {h} _ {\mathbf {t}} ^ {\mathbf {y} ^ {\top}}; \mathbf {u} _ {\mathbf {t} - \mathbf {1}} ^ {\mathbf {y} ^ {\top}}; \mathbf {c} _ {\mathbf {t}} ^ {\top} \right] \end{array} \tag {9}
$$
Flexible Mode Fusion: Previous three distributions can be fused by the  $MF(\mathbf{h}_{\mathbf{t}}^{\mathbf{y}}, \mathbf{u}_{\mathbf{t}-1}, \mathbf{c}_{\mathbf{t}})$ , a 2-layer MLP activated by softmax.  $MF$  can outputs a probability distribution  $(\gamma_{w,t}, \gamma_{k,t}, \gamma_{c,t})$  over three modes at each time step:
$$
p _ {\text {o u t}, t} = \gamma_ {w, t} \times p _ {w, t} + \gamma_ {k, t} \times p _ {k, t} + \gamma_ {c, t} \times p _ {c, t} \tag {10}
$$
The proposed  $MF$  can be regarded as a multiclass classifier; therefore, the advantage of  $MF$  is the flexibility, we can additionally integrate more modes or remove existing modes by simply changing the number of classes. For a more reasonable fusion, the Cross-Entropy between the ground-truth mode and the predicted distribution by  $MF$  is used to supervise the training; the corresponding Cross-Entropy loss is denoted as  $\mathcal{L}_m$ . Next, we optimize the fused output distribution  $p_{out}(Y|X,F)$  by minimizing the  $\mathcal{L}_n$ , which is given by:
$$
- \sum_ {t} \lambda_ {t} \log p _ {\text {o u t}, t} \left(y _ {t} \mid y _ {t - 1: 1}, X, F\right) + \frac {\mathcal {L} _ {m}}{2} \tag {11}
$$
where  $\lambda_{t}$  is a normalization term to penalize the out-of-vocabulary words,  $\lambda_{t} = \frac{1}{\#(unk\in Y)}$  if  $y_{t}$  is an unk, otherwise  $\lambda_{t} = 1$ .
Training Objective: Finally, the ConKADI can be trained by minimizing the following objective:
$$
\mathcal {L} = \mathcal {L} _ {n} + \mathcal {L} _ {k} + \mathcal {L} _ {f} \tag {12}
$$
To verify the generalization among different languages, we evaluate models not only on a public English Reddit dataset (Zhou et al., 2018), but we also collect and construct a Chinese Weibo dataset. Both datasets are aligned with the commonsense knowledge graph ConcetNet (conceptnet.io), the statistics have been reported in Table 1.
Reddit	Weibo
#Train	1,352,961	1,019,908
#Dev/#Test	40,000	56,661
#Vocab	30,000	50,000
Batch Size	100	50
#Entity/#Relation	21,471/44	27,189/26
#Fact	149,803	696,466
Table 1: The statistics of two datasets.
The English Reddit: We did some filtering on the raw data: Utterances that are too short (< 4 words) or too long (>30 words) were removed, and each message can be associated with at most 300 related fact triplets.
The Chinese Weibo: We first collected three open-sourced Weibo (weibo.com) datasets, which originally contained 4.44M (Shang et al., 2015), 1.96M (Ke et al., 2018) and 10.48M (Li and Yan, 2018) pairs of dialogue, respectively. Jieba<sup>4</sup> was used to segment; utterances that are too short/long were removed as well. Next, we crawled 4.48M entities and 13.98M facts from the ConceptNet. Stop entities, and low-frequent entities are excluded. For a dialogue pair, if one entity in the message and another entity in the response can be connected by a 1-hop edge in the knowledge graph, this dialogue was kept. In comparison with the English Reddit, our dataset has more facts, but the relation types are quite limited; hence, we set the limit that a message can be associated with at most 150 fact triplets.
For two datasets, the embedding of entities and relations are learned by using TransE (Bordes et al., 2013); then, they are kept fixed in training. Our experimental resources are available at the web  $^{5}$ .
4.2 Settings
Baselines: The widely used S2S (Sutskever et al., 2014), and its Attentive version ATS2S (Luong et al., 2015). We further add the bidi-MMI (Li et al., 2016a) or the diverse decoding (Li et al., 2016b) to improve the diversity of ATS2S, which are denoted as  $\mathbf{ATS2S}_{MMI}$  and  $\mathbf{ATS2S}_{DD}^6$ . Copy mechanism (Gu et al., 2016; Vinyals et al., 2015) allows Decoder to point then copy a source word. GenDS is a knowledge-aware model, which can generate responses with the utilizing of entity words. (Zhu et al., 2017). CCM is the current state-of-the-art approach in the task of response generation with
$^{4}$ https://pypi.python.org/pypi/jieba/  
 $^{5}$ https://github.com/pku-orangecat/  
ACL2020-ConKADI
The best  $k$  was searched form [0.1, 3.0].
$^3\# (\cdot)$  is the count of  $\cdot$
5815