SlowGuess/ACL_2020_OCR huggingface dataset:
where  $\circ$  is concatenate operator.  $\vec{g}_{e_j}^{l - 1}$  is the concept  $e_j$ 's representation of  $(l - 1)$ -th layer.  $\vec{p}^{l - 1}$  is the user utterance representation of  $(l - 1)$ -th layer.
The  $(l - 1)$ -th layer user utterance representation is updated with the zero-hop concepts  $V^0$ :
$$
\vec {p} ^ {l - 1} = \operatorname {F F N} \left(\sum_ {e _ {i} \in V ^ {0}} \vec {g} _ {e _ {i}} ^ {l - 1}\right). \tag {18}
$$
$f_{r}^{e_{j}\rightarrow e_{i}}(\vec{g}_{e_{j}}^{l - 1})$  aggregates the concept semantics of relation  $r$  specific neighbor concept  $e_j$ . It uses attention  $\alpha_r^{e_j}$  to control concept flow from  $e_i$ :
$$
f _ {r} ^ {e _ {j} \rightarrow e _ {i}} (\vec {e} _ {j} ^ {l - 1}) = \alpha_ {r} ^ {e _ {j}} \cdot \operatorname {F F N} (\vec {r} \circ \vec {g} _ {e _ {j}} ^ {l - 1}), \tag {19}
$$
where  $\circ$  is concatenate operator and  $\vec{r}$  is the relation embedding of  $r$ . The attention weight  $\alpha_{r}^{e_{j}}$  is computed over all concept  $e_{i}$ 's neighbor concepts according to the relation weight score and the Page Rank score (Sun et al., 2018):
$$
\alpha_ {r} ^ {e _ {j}} = \operatorname {s o f t m a x} (\vec {r} \cdot \vec {p} ^ {l - 1}) \cdot \operatorname {P a g e R a n k} \left(e _ {j} ^ {l - 1}\right), \tag {20}
$$
where  $\mathrm{PageRank}(e_j^{l - 1})$  is the page rank score to control propagation of embeddings along paths starting from  $e_i$  (Sun et al., 2018) and  $\vec{p}^{l - 1}$  is the  $(l - 1)$ -th layer user utterance representation.
The 0-th layer concept representation  $\vec{e}_i^0$  for concept  $e_i$  is initialized with the pre-trained concept
Model	Bleu-1	Bleu-2	Bleu-3	Nist-1	Nist-2	Nist-3	Concept-PPL
Seq2Seq	0.1702	0.0579	0.0226	1.0230	1.0963	1.1056	-
MemNet	0.1741	0.0604	0.0246	1.0975	1.1847	1.1960	46.85
CopyNet	0.1589	0.0549	0.0226	0.9899	1.0664	1.0770	40.27
CCM	0.1413	0.0484	0.0192	0.8362	0.9000	0.9082	39.18
GPT-2 (lang)	0.1705	0.0486	0.0162	1.0231	1.0794	1.0840	-
GPT-2 (conv)	0.1765	0.0625	0.0262	1.0734	1.1623	1.1745	-
ConceptFlow	0.2451	0.1047	0.0493	1.6137	1.7956	1.8265	26.76
Table 7: More Metrics on Relevance of Generated Responses. The relevance is calculated between the generated response and the golden response. Concept-PPL is the method used for calculating Perplexity in CCM (Zhou et al., 2018a), which combines the distribution of both words and concepts together. The Concept-PPL is meaningless when utilizing different numbers of concepts (more concepts included, better Perplexity shows).
Novelty w.r.t. Input(↓)
Model	Bleu-1	Bleu-2	Bleu-3	Nist-1	Nist-2	Nist-3	Rouge-1
Seq2Seq	0.1855	0.0694	0.0292	1.2114	1.3169	1.3315	0.1678
MemNet	0.2240	0.1111	0.0648	1.6740	1.9594	2.0222	0.2216
CopyNet	0.2042	0.0991	0.056	1.5072	1.7482	1.7993	0.2104
CCM	0.1667	0.0741	0.0387	1.1232	1.2782	1.3075	0.1953
GPT-2 (lang)	0.2124	0.0908	0.0481	1.5105	1.7090	1.7410	0.1817
GPT-2 (conv)	0.2537	0.1498	0.1044	1.9562	2.4127	2.5277	0.2522
ConceptFlow	0.1850	0.0685	0.0281	1.3325	1.4600	1.4729	0.1777
Table 8: More Metrics on Novelty of Generated Responses. The novelty is calculated between the generated response and the user utterance, where lower means better.
Version	Bleu-1	Bleu-2	Bleu-3	Bleu-4	Nist-1	Nist-2	Nist-3	Nist-4
Base	0.1705	0.0577	0.0223	0.0091	0.9962	1.0632	1.0714	1.0727
Rand	0.1722	0.0583	0.0226	0.0092	1.0046	1.0726	1.0810	1.0823
Distract	0.1734	0.0586	0.0230	0.0097	1.0304	1.0992	1.1081	1.1096
Full	0.2265	0.0928	0.0417	0.0195	1.4550	1.6029	1.6266	1.6309
Table 9: The Generation Quality of Different Outer Hop Concept Selectors. Both Bleu and Nist are used to calculate the relevance between generated responses and golden responses.
embedding  $\vec{e}_i$  and the 0-th layer user utterance representation  $\vec{p}^0$  is initialized with the  $m$ -th hidden state  $h_m$  from the user utterance representation set  $H$ . The GNN used in ConceptFlow establishes the central concept flow between concepts in the central concept graph using attentions.
A.4 Concept Selection
With the concept graph growing, the number of concepts is increased exponentially, which brings lots of noises. Thus, a selection strategy is needed to select high-relevance concepts from a large number of concepts. This part presents the details of our concept selection from ConceptFlow (select).
The concept selector aims to select top K related
two-hop concepts based on the sum of attention scores for each time  $t$  over entire two-hop concepts:
$$
\alpha_ {n} = \sum_ {t = 1} ^ {n} \operatorname {s o f t m a x} \left(\vec {s} _ {t} \cdot \vec {e} _ {k}\right), \tag {21}
$$
where  $\vec{s}_t$  is the  $t$ -th time decoder output representation and  $\vec{e}_k$  denotes the concept  $e_k$ 's embedding.
Then two-hop concepts are sorted according to the attention score  $\alpha_{n}$ . In our settings, top 100 concepts are reserved to construct the two-hop concept graph  $V^{2}$ . Moreover, central concepts are all reserved because of the high correlation with the conversation topic and acceptable computation complexity. Both central concepts and selected two-hop concepts construct the concept graph  $G$ .