SlowGuess/ACL_2020_OCR huggingface dataset:
Houyu Zhang $^{1}$  *† Zhenghao Liu $^{2*}$  Chenyan Xiong $^{3}$  Zhiyuan Liu $^{2}$
$^{1}$ Department of Computer Science, Brown University, Providence, USA
$^{2}$ Department of Computer Science and Technology, Tsinghua University, Beijing, China  
 $^{3}$ Institute for Artificial Intelligence, Tsinghua University, Beijing, China
State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China  
<sup>3</sup>Microsoft Research AI, Redmond, USA
Human conversations naturally evolve around related concepts and scatter to multi-hop concepts. This paper presents a new conversation generation model, ConceptFlow, which leverages commonsense knowledge graphs to explicitly model conversation flows. By grounding conversations to the concept space, ConceptFlow represents the potential conversation flow as traverses in the concept space along commonsense relations. The traverse is guided by graph attentions in the concept graph, moving towards more meaningful directions in the concept space, in order to generate more semantic and informative responses. Experiments on Reddit conversations demonstrate ConceptFlow's effectiveness over previous knowledge-aware conversation models and GPT-2 based models while using  $70\%$  fewer parameters, confirming the advantage of explicit modeling conversation structures. All source codes of this work are available at https://github.com/thunlp/ConceptFlow.
The rapid advancements of language modeling and natural language generation (NLG) techniques have enabled fully data-driven conversation models, which directly generate natural language responses for conversations (Shang et al., 2015; Vinyals and Le, 2015; Li et al., 2016b). However, it is a common problem that the generation models may degenerate dull and repetitive contents (Holtzman et al., 2019; Welleck et al., 2019), which, in conversation assistants, leads to off-topic and useless responses. (Tang et al., 2019; Zhang et al., 2018; Gao et al., 2019).
Conversations often develop around Knowledge. A promising way to address the degeneration prob
Figure 1: An Example of Concept Shift in a Conversation. Darker green indicates higher relevance and wider arrow indicates stronger concept shift (captured by ConceptFlow).
lem is to ground conversations with external knowledge (Xing et al., 2017), such as open-domain knowledge graph (Ghazvininejad et al., 2018), commonsense knowledge base (Zhou et al., 2018a), or background documents (Zhou et al., 2018b). Recent research leverages such external knowledge by using them to ground conversations, integrating them as additional representations, and then generating responses conditioned on both the texts and the grounded semantics (Ghazvininejad et al., 2018; Zhou et al., 2018a,b).
Integrating external knowledge as extra semantic representations and additional inputs to the conversation model effectively improves the quality of generated responses (Ghazvininejad et al., 2018; Logan et al., 2019; Zhou et al., 2018a). Never
† Part of work is conducted at Tsinghua University.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2031-2043 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
theless, some research on discourse development suggests that human conversations are not "still": People chat around a number of related concepts, and shift their focus from one concept to others. Grosz and Sidner (1986) models such concept shift by breaking discourse into several segments, and demonstrating different concepts, such as objects and properties, are needed to interpret different discourse segments. Attentional state is then introduced to represent the concept shift corresponding to each discourse segment. Fang et al. (2018) shows that people may switch dialog topics entirely in a conversation. Restricting the utilization of knowledge only to those directly appear in the conversation, effective as they are, does not reach the full potential of knowledge in modeling human conversations.
To model the concept shift in human conversations, this work presents ConceptFlow (Conversation generation with Concept Flow), which leverages commonsense knowledge graphs to model the conversation flow in the explicit concept space. For example, as shown in Figure 1, the concepts of a conversation from Reddit evolves from "chat" and "future", to adjacent concept "talk", and also hops to distant concept "dream" along the commonsense relations—a typical involvement in natural conversations. To better capture this conversation structure, ConceptFlow explicitly models the conversations as traverses in commonsense knowledge graphs: it starts from the grounded concepts, e.g., "chat" and "future", and generates more meaningful conversations by hopping along the commonsense relations to related concepts, e.g., "talk" and "dream".
The traverses in the concept graph are guided by graph attention mechanisms, which derives from graph neural networks to attend on more appropriate concepts. ConceptFlow learns to model the conversation development along more meaningful relations in the commonsense knowledge graph. As a result, the model is able to "grow" the grounded concepts by hopping from the conversation utterances, along the commonsense relations, to distant but meaningful concepts; this guides the model to generate more informative and on-topic responses. Modeling commonsense knowledge as concept flows, is both a good practice on improving response diversity by scattering current conversation focuses to other concepts (Chen et al., 2017), and an implementation solution of the attentional
state mentioned above (Grosz and Sidner, 1986).
Our experiments on a Reddit conversation dataset with a commonsense knowledge graph, ConceptNet (Speer et al., 2017), demonstrate the effectiveness of ConceptFlow. In both automatic and human evaluations, ConceptFlow significantly outperforms various seq2seq based generation models (Sutskever et al., 2014), as well as previous methods that also leverage commonsense knowledge graphs, but as static memories (Zhou et al., 2018a; Ghazvininejad et al., 2018; Zhu et al., 2017). Notably, ConceptFlow also outperforms two finetuned GPT-2 systems (Radford et al., 2019), while using  $70\%$  fewer parameters. Explicitly modeling conversation structure provides better parameter efficiency.
We also provide extensive analyses and case studies to investigate the advantage of modeling conversation flow in the concept space. Our analyses show that many Reddit conversations are naturally aligned with the paths in the commonsense knowledge graph; incorporating distant concepts significantly improves the quality of generated responses with more on-topic semantic information added. Our analyses further confirm the effectiveness of our graph attention mechanism in selecting useful concepts, and ConceptFlow's ability in leveraging them to generate more relevant, informative, and less repetitive responses.
Sequence-to-sequence models, e.g., Sutskever et al. (2014), have been widely used for natural language generation (NLG), and to build conversation systems (Shang et al., 2015; Vinyals and Le, 2015; Li et al., 2016b; Wu et al., 2019). Recently, pretrained language models, such as ELMO (Devlin et al., 2019), UniLM (Dong et al., 2019) and GPT-2 (Radford et al., 2018), further boost the NLG performance with large scale pretraining. Nevertheless, the degenerating of irrelevant, off-topic, and non-useful responses is still one of the main challenges in conversational generation (Rosset et al., 2020; Tang et al., 2019; Zhang et al., 2018; Gao et al., 2019).
Recent work focuses on improving conversation generation with external knowledge, for example, incorporating additional texts (Ghazvininejad et al., 2018; Vougiouklis et al., 2016; Xu et al., 2017; Long et al., 2017), or knowledge graphs (Long et al., 2017; Ghazvininejad et al., 2018). They have
shown external knowledge effectively improves conversation response generation.
The structured knowledge graphs include rich semantics represented via entities and relations (Hayashi et al., 2019). Lots of previous studies focus on task-targeted dialog systems based on domain-specific knowledge bases (Xu et al., 2017; Zhu et al., 2017; Gu et al., 2016). To generate responses with a large-scale knowledge base, Zhou et al. (2018a) and Liu et al. (2018) utilize graph attention and knowledge diffusion to select knowledge semantics for utterance understanding and response generation. Moon et al. (2019) focuses on the task of entity selection, and takes advantage of positive entities that appear in the golden response. Different from previous research, ConceptFlow models the conversation flow explicitly with the commonsense knowledge graph and presents a novel attention mechanism on all concepts to guide the conversation flow in the latent concept space.
This section presents our Conversation generation model with latent Concept Flow (ConceptFlow). Our model grounds the conversation in the concept graph and traverses to distant concepts along commonsense relations to generate responses.
Given a user utterance  $X = \{x_{1},\dots,x_{m}\}$  with  $m$  words, conversation generation models often use an encoder-decoder architecture to generate a response  $Y = \{y_{1},\dots,y_{n}\}$ .
The encoder represents the user utterance  $X$  as a representation set  $H = \{\vec{h}_1,\dots,\vec{h}_m\}$ . This is often done by Gated Recurrent Units (GRU):
$$
\vec {h} _ {i} = \operatorname {G R U} \left(\vec {h} _ {i - 1}, \vec {x} _ {i}\right), \tag {1}
$$
where the  $\vec{x}_i$  is the embedding of word  $x_i$ .
The decoder generates  $t$ -th word in the response according to the previous  $t - 1$  generated words  $y_{<t} = \{y_1, \dots, y_{t-1}\}$  and the user utterance  $X$ :
$$
P (Y | X) = \prod_ {t = 1} ^ {n} P \left(y _ {t} \mid y _ {<   t}, X\right). \tag {2}
$$
Then it minimizes the cross-entropy loss  $L$  and optimizes all parameters end-to-end:
$$
L = \sum_ {t = 1} ^ {n} \operatorname {C r o s s E n t r o p y} \left(y _ {t} ^ {*}, y _ {t}\right), \tag {3}
$$
where  $y_{t}^{*}$  is the token from the golden response.
Figure 2: The Architecture of ConceptFlow.
The architecture of ConceptFlow is shown in Figure 2. ConceptFlow first constructs a concept graph  $G$  with central graph  $G_{\text{central}}$  and outer graph  $G_{\text{outer}}$  according to the distance (hops) from the grounded concepts (Sec. 3.2).
Then ConceptFlow encodes both central and outer concept flows in central graph  $G_{\text{central}}$  and outer graph  $G_{\text{outer}}$ , using graph neural networks and concept embedding (Sec. 3.3).
The decoder, presented in Section 3.4, leverages the encodings of concept flows and the utterance to generate words or concepts for responses.
3.2 Concept Graph Construction
ConceptFlow constructs a concept graph  $G$  as the knowledge for each conversation. It starts from the grounded concepts (zero-hop concepts  $V^0$ ), which appear in the conversation utterance and annotated by entity linking systems.
Then, ConceptFlow grows zero-hop concepts  $V^0$  with one-hop concepts  $V^1$  and two-hop concepts  $V^2$ . Concepts from  $V^0$  and  $V^1$ , as well as all relations between them, form the central concept graph  $G_{\mathrm{central}}$ , which is closely related to the current conversation topic. Concepts in  $V^1$  and  $V^2$  and their connections form the outer graph  $G_{\mathrm{outer}}$ .
3.3 Encoding Latent Concept Flow
The constructed concept graph provides explicit semantics on how concepts related to commonsense knowledge. ConceptFlow utilizes it to model the conversation and guide the response generation. It starts from the user utterance, traversing through central graph  $G_{\text{central}}$ , to outer graph  $G_{\text{outer}}$ . This is modeled by encoding the central and outer concept flows according to the user utterance.
Central Flow Encoding. The central concept graph  $G_{\mathrm{central}}$  is encoded by a graph neural network that propagates information from user utterance  $H$  to the central concept graph. Specifically, it encodes concept  $e_i \in G_{\mathrm{central}}$  to representation  $\vec{g}_{e_i}$ :
$$
\vec {g} _ {e _ {i}} = \operatorname {G N N} \left(\vec {e} _ {i}, G _ {\text {c e n t r a l}}, H\right), \tag {4}
$$
where  $\vec{e}_i$  is the concept embedding of  $e_i$ . There is no restriction of which GNN model to use. We choose Sun et al. (2018)'s GNN (GraftNet), which shows strong effectiveness in encoding knowledge graphs. More details of GraftNet can be found in Appendix A.3.
Outer Flow Encoding. The outer flow  $f_{e_p}$ , hopping from  $e_p \in V_1$  to its connected two-hop concept  $e_k$ , is encoded to  $\vec{f}_{e_p}$  by an attention mechanism:
$$
\vec {f} _ {e _ {p}} = \sum_ {e _ {k}} \theta^ {e _ {k}} \cdot [ \vec {e} _ {p} \circ \vec {e} _ {k} ], \tag {5}
$$
where  $\vec{e}_p$  and  $\vec{e}_k$  are embeddings for  $e_p$  and  $e_k$ , and are concatenated (○). The attention  $\theta^{e_k}$  aggregates concept triple  $(e_p, r, e_k)$  to get  $\vec{f}_{e_p}$ :
$$
\theta^ {e _ {k}} = \operatorname {s o f t m a x} \left(\left(w _ {r} \cdot \vec {r}\right) ^ {\top} \cdot \tanh  \left(w _ {h} \cdot \vec {e} _ {p} + w _ {t} \cdot \vec {e} _ {k}\right)\right), \tag {6}
$$
where  $\vec{r}$  is the relation embedding between the concept  $e_p$  and its neighbor concept  $e_k$ .  $w_r$ ,  $w_h$  and  $w_t$  are trainable parameters. It provides an efficient attention specifically focusing on the relations for multi-hop concepts.
3.4 Generating Text with Concept Flow
To consider both user utterance and related information, the texts from the user utterance and the latent concept flows are incorporated by decoder using two components: 1) the context representation that combines their encodings (Sec. 3.4.1); 2) the conditioned generation of words and concepts from the context representations (Sec. 3.4.2).
3.4.1 Context Representation
To generate  $t$ -th time response token, we first calculate the output context representation  $\vec{s}_t$  for  $t$ -th
time decoding with the encodings of the utterance and the latent concept flow.
Specifically,  $\vec{s}_t$  is calculated by updating the  $(t - 1)$ -th step output representation  $\vec{s}_{t-1}$  with the  $(t - 1)$ -th step context representation  $\vec{c}_{t-1}$ :
$$
\vec {s} _ {t} = \operatorname {G R U} \left(\vec {s} _ {t - 1}, \left[ \vec {c} _ {t - 1} \circ \vec {y} _ {t - 1} \right]\right), \tag {7}
$$
where  $\vec{y}_{t-1}$  is the  $(t-1)$ -th step generated token  $y_{t-1}$ 's embedding, and the context representation  $\vec{c}_{t-1}$  concatenates the text-based representation  $\vec{c}_{t-1}^{\text{text}}$  and the concept-based representation  $\vec{c}_{t-1}^{\text{concept}}$ :
$$
\vec {c} _ {t - 1} = \operatorname {F F N} \left(\left[ \vec {c} _ {t - 1} ^ {\text {t e x t}} \circ \vec {c} _ {t - 1} ^ {\mathrm {c p t}} \right]\right). \tag {8}
$$
The text-based representation  $\vec{c}_{t-1}^{\text {text }}$  reads the user utterance encoding  $H$  with a standard attention mechanism (Bahdanau et al., 2015):
$$
\vec {c} _ {t - 1} ^ {\text {t e x t}} = \sum_ {i = 1} ^ {m} \alpha_ {t - 1} ^ {j} \cdot \vec {h} _ {j}, \tag {9}
$$
and attentions  $\alpha_{t - 1}^{j}$  on the utterance tokens:
$$
\alpha_ {t - 1} ^ {j} = \operatorname {s o f t m a x} \left(\vec {s} _ {t - 1} \cdot \vec {h} _ {j}\right). \tag {10}
$$
The concept-based representation  $\vec{c}_{t - 1}^{\mathrm{concept}}$  is a combination of central and outer flow encodings:
$$
\vec {c} _ {t - 1} ^ {\mathrm {c p t}} = \left(\sum_ {e _ {i} \in G _ {\text {c e n t r a l}}} \beta_ {t - 1} ^ {e _ {i}} \cdot \vec {g} _ {e _ {i}}\right) \circ \left(\sum_ {f _ {e _ {p}} \in G _ {\text {o u t e r}}} \gamma_ {t - 1} ^ {f} \cdot \vec {f} _ {e _ {p}}\right). \tag {11}
$$
The attention  $\beta_{t - 1}^{e_i}$  weights over central concept representations:
$$
\beta_ {t - 1} ^ {e _ {i}} = \operatorname {s o f t m a x} \left(\vec {s} _ {t - 1} \cdot \vec {g} _ {e _ {i}}\right), \tag {12}
$$
and the attention  $\gamma_{t - 1}^{f}$  weights over outer flow representations:
$$
\gamma_ {t - 1} ^ {f} = \operatorname {s o f t m a x} \left(\vec {s} _ {t - 1} \cdot \vec {f} _ {e _ {p}}\right). \tag {13}
$$
3.4.2 Generating Tokens
The  $t$ -th time output representation  $\vec{s}_t$  (Eq. 7) includes information from both the utterance text, the concepts with different hop steps, and the attentions upon them. The decoder leverages  $\vec{s}_t$  to generate the  $t$ -th token to form more informative responses.
It first uses a gate  $\sigma^{*}$  to control the generation by choosing words  $(\sigma^{*} = 0)$ , central concepts  $(V^{0,1}, \sigma^{*} = 1)$  and outer concept set  $(V^{2}, \sigma^{*} = 2)$ :
$$
\sigma^ {*} = \operatorname {a r g m a x} _ {\sigma \in \{0, 1, 2 \}} \left(\mathrm {F F N} _ {\sigma} \left(\vec {s} _ {t}\right)\right), \tag {14}
$$
The generation probabilities of word  $w$ , central concept  $e_i$ , and outer concepts  $e_k$  are calculated
over the word vocabulary, central concept set  $V^{0,1}$ , and outer concept set  $V^2$ :
$$
y _ {t} \sim \left\{ \begin{array}{l} \operatorname {s o f t m a x} \left(\vec {s} _ {t} \cdot \vec {w}\right), \sigma^ {*} = 0 \\ \operatorname {s o f t m a x} \left(\vec {s} _ {t} \cdot \vec {g} _ {e _ {i}}\right), \sigma^ {*} = 1 \\ \operatorname {s o f t m a x} \left(\vec {s} _ {t} \cdot \vec {e} _ {k}\right), \sigma^ {*} = 2, \end{array} \right. \tag {15}
$$
where  $\vec{w}$  is the word embedding for word  $w$ ,  $\vec{g}_{e_i}$  is the central concept representation for concept  $e_i$  and  $\vec{e}_k$  is the two-hop concept  $e_k$ 's embedding.
The training and prediction of ConceptFlow are conducted following standard conditional language models, i.e. using Eq. 15 in place of Eq. 2 and training it by the Cross-Entropy loss (Eq. 3). Only ground truth responses are used in training and no additional annotation is required.
4 Experiment Methodology
This section describes the dataset, evaluation metrics, baselines, and implementation details of our experiments.
Dataset. All experiments use the multi-hop extended conversation dataset based on a previous dataset which collects single-round dialogs from Reddit (Zhou et al., 2018a). Our dataset contains 3,384,185 training pairs and 10,000 test pairs. Preprocessed ConceptNet (Speer et al., 2017) is used as the knowledge graph, which contains 120,850 triples, 21,471 concepts and 44 relation types.
Evaluation Metrics. A wide range of evaluation metrics are used to evaluate the quality of generated responses: PPL (Serban et al., 2016), Bleu (Papineni et al., 2002), Nist (Doddington, 2002), ROUGE (Lin, 2004) and Meteor (Lavie and Agarwal, 2007) are used for relevance and repetitiveness; Dist-1, Dist-2 and Ent-4 are used for diversity, which is same with the previous work (Li et al., 2016a; Zhang et al., 2018). The metrics above are evaluated using the implementation from Galley et al. (2018). Zhou et al. (2018a)'s concept PPL mainly focuses on concept grounded models and this metric is reported in Appendix A.1.
The Precision, Recall, and F1 scores are used to evaluate the quality of learned latent concept flow in predicting the golden concepts which appear in ground truth responses.
Baselines. The six baselines compared come from three groups: standard Seq2Seq, knowledge-enhanced ones, and fine-tuned GPT-2 systems.
Seq2Seq (Sutskever et al., 2014) is the basic encoder-decoder for language generation.
Knowledge-enhanced baselines include MemNet (Ghazvininejad et al., 2018), CopyNet (Zhu
et al., 2017) and CCM (Zhou et al., 2018a). MemNet maintains a memory to store and read concepts. CopyNet copies concepts for the response generation. CCM (Zhou et al., 2018a) leverages a graph attention mechanism to model the central concepts. These models mainly focus on the grounded concepts. They do not explicitly model the conversation structures using multi-hop concepts.
GPT-2 (Radford et al., 2019), the pre-trained model that achieves the state-of-the-art in lots of language generation tasks, is also compared in our experiments. We fine-tune the 124M GPT-2 in two ways: concatenate all conversations together and train it like a language model (GPT-2 lang); extend the GPT-2 model with encode-decoder architecture and supervise with response data (GPT-2 conv).
Implement Details. The zero-hop concepts are initialized by matching the keywords in the post to concepts in ConceptNet, the same with CCM (Zhou et al., 2018a). Then zero-hop concepts are extended to their neighbors to form the central concept graph. The outer concepts contain a large amount of two-hop concepts with lots of noises. To reduce the computational cost, we first train ConceptFlow (select) with  $10\%$  random training data, and use the learned graph attention to select top 100 two-hop concepts over the whole dataset. Then the standard train and test are conducted with the pruned graph. More details of this filtering step can be found in Appendix A.4.
TransE (Bordes et al., 2013) embedding and Glove (Pennington et al., 2014) embedding are used to initialize the representation of concepts and words, respectively. Adam optimizer with the learning rate of 0.0001 is used to train the model.
Five experiments are conducted to evaluate the generated responses from ConceptFlow and the effectiveness of the learned graph attention.
5.1 Response Quality
This experiment evaluates the generation quality of ConceptFlow automatically and manually.
Automatic Evaluation. The quality of generated responses is evaluated with different metrics from three aspects: relevance, diversity, and novelty. Table 1 and Table 2 show the results.
In Table 1, all evaluation metrics calculate the relevance between the generated response and the
Model	Bleu-4	Nist-4	Rouge-1	Rouge-2	Rouge-L	Meteor	PPL
Seq2Seq	0.0098	1.1069	0.1441	0.0189	0.1146	0.0611	48.79
MemNet	0.0112	1.1977	0.1523	0.0215	0.1213	0.0632	47.38
CopyNet	0.0106	1.0788	0.1472	0.0211	0.1153	0.0610	43.28
CCM	0.0084	0.9095	0.1538	0.0211	0.1245	0.0630	42.91
GPT-2 (lang)	0.0162	1.0844	0.1321	0.0117	0.1046	0.0637	29.08*
GPT-2 (conv)	0.0124	1.1763	0.1514	0.0222	0.1212	0.0629	24.55*
ConceptFlow	0.0246	1.8329	0.2280	0.0469	0.1888	0.0942	29.90
Table 1: Relevance Between Generated and Golden Responses. The PPL results* of GPT-2 is not directly comparable because of its different tokenization. More results can be found in Appendix A.1.
Diversity(↑)	Novelty w.r.t. Input(↓)
Model	Dist-1	Dist-2	Ent-4	Bleu-4	Nist-4	Rouge-2	Rouge-L	Meteor
Seq2Seq	0.0123	0.0525	7.665	0.0129	1.3339	0.0262	0.1328	0.0702
MemNet	0.0211	0.0931	8.418	0.0408	2.0348	0.0621	0.1785	0.0914
CopyNet	0.0223	0.0988	8.422	0.0341	1.8088	0.0548	0.1653	0.0873
CCM	0.0146	0.0643	7.847	0.0218	1.3127	0.0424	0.1581	0.0813
GPT-2 (lang)	0.0325	0.2461	11.65	0.0292	1.7461	0.0359	0.1436	0.0877
GPT-2 (conv)	0.0266	0.1218	8.546	0.0789	2.5493	0.0938	0.2093	0.1080
ConceptFlow	0.0223	0.1228	10.27	0.0126	1.4749	0.0258	0.1386	0.0761
Table 2: Diversity (higher better) and Novelty (lower better) of Generated Response. Diversity is calculated within generated responses; Novelty compares generated responses to the input post. More results are in Appendix A.1.
Model	Parameter	Average Score	Best@1 Ratio
App.	Inf.	App.	Inf.
CCM	35.6M	1.802	1.802	17.0%	15.6%
GPT-2 (conv)	124.0M	2.100	1.992	26.2%	23.6%
ConceptFlow	35.3M	2.690	2.192	30.4%	25.6%
Golden	Human	2.902	3.110	67.4%	81.8%
Table 3: Human Evaluation on Appropriate (App.) and Informativeness (Inf.). The Average Score takes the average from human judgments. Best@1 Ratio indicates the fraction of judges consider the case as the best. The number of parameters are also presented.
Model	App.	Inf.
ConceptFlow-CCM	0.3724	0.2641
ConceptFlow-GPT2	0.2468	0.2824
Table 4: Fleiss' Kappa of Human Agreement. Two testing scenarios Appropriate (App.) and Informativeness (Inf.) are used to evaluate the the quality of generated response. The Fleiss' Kappa evaluates agreement from various annotators and focuses on the comparison of two models with three categories: win, tie and loss.
golden response. ConceptFlow outperforms all baseline models by large margins. The responses generated by ConceptFlow are more on-topic and match better with the ground truth responses.