SlowGuess/ACL_2020_OCR huggingface dataset:
Jun Xu $^{1*}$ , Haifeng Wang $^{2}$ , Zheng-Yu Niu $^{2}$ , Hua Wu $^{2}$ , Wanxiang Che $^{1\dagger}$ , Ting Liu $^{1}$
<sup>1</sup>Research Center for Social Computing and Information Retrieval,
Harbin Institute of Technology, Harbin, China
$^{2}$ Baidu Inc., Beijing, China
{jxu, car, tliu}  $@$  ir.hit.edu.cn, {wanghaifeng, niuzhengyu, wu_hua}  $@$  baidu.com
To address the challenge of policy learning in open-domain multi-turn conversation, we propose to represent prior information about dialog transitions as a graph and learn a graph grounded dialog policy, aimed at fostering a more coherent and controllable dialog. To this end, we first construct a conversational graph (CG) from dialog corpora, in which there are vertices to represent "what to say" and "how to say", and edges to represent natural transition between a message (the last utterance in a dialog context) and its response. We then present a novel CG grounded policy learning framework that conducts dialog flow planning by graph traversal, which learns to identify a what-vertex and a how-vertex from the CG at each turn to guide response generation. In this way, we effectively leverage the CG to facilitate policy learning as follows: (1) it enables more effective long-term reward design, (2) it provides high-quality candidate actions, and (3) it gives us more control over the policy. Results on two benchmark corpora demonstrate the effectiveness of this framework.
How to effectively learn dialog strategies is an enduring challenge for open-domain multi-turn conversation generation. To address this challenge, previous works investigate word-level policy models that simultaneously learn dialog policy and language generation from dialog corpora (Li et al., 2016b; Zhang et al., 2018b). But these word-level policy models often lead to a degeneration issue where the utterances become ungrammatical or repetitive (Lewis et al., 2017). To alleviate this issue, utterance-level policy models have been proposed to decouple policy learning from response generation, and they focus on how to incorporate
Figure 1: Our system (1) understands the user message by linking it to CG. We call the linked vertices as hit what-vertices (green color); (2) selects a what-vertex ("sleepy") and a how-vertex (responding mechanism  $M_3$ , a MLP network) from one-hop neighbors of hit vertices; (3) generates a coherent response with two sub-steps: firstly, obtains a response representation  $\bar{r}$  using both  $M_3$  and a message representation (from a message-encoder); Next, produces a response "It's so ..." with "sleepy" and  $\bar{r}$  as input. Notice all the how-vertices are from the same set rather than completely independent of each other.
high-level utterance representations, e.g., latent variables or keywords, to facilitate policy learning (He et al., 2018; Yao et al., 2018; Zhao et al., 2019).
However, these utterance-level methods tend to produce less coherent multi-turn dialogs since it is quite challenging to learn semantic transitions in a dialog flow merely from dialog data without the help of prior information. In this paper, we propose to represent prior information about dialog transition (between a message and its response) as a graph, and optimize dialog policy based on the graph, to foster a more coherent dialog.
To this end, we propose a novel conversational graph (CG) grounded policy learning frame
*This work was done at Baidu.
† Corresponding author: Wanxiang Che.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1835-1845
work for open-domain multi-turn conversation generation (CG-Policy). It consists of two key components, (1) a CG that captures both local-appropriateness and global-coherence information, (2) a reinforcement learning (RL) based policy model that learns to leverage the CG to foster a more coherent dialog. In Figure 1, given a user message, our system selects a what-vertex ("sleepy") and a how-vertex (responding mechanism  $M_3$ ) to produce a coherent response.
We first construct the CG based on dialog data. We use vertices to represent utterance content, and edges to represent dialog transitions between utterances. Specifically, there are two types of vertices: (1) a what-vertex that contains a keyword, and (2) a how-vertex that contains a responding mechanism (from a multi-mapping based generator in Section 3.1) to capture rich variability of expressions. We also use this multi-mapping based method to build edges between two what-vertices to capture the local-appropriateness between the two keywords as a message and a response respectively. It can be seen that the what-vertices from the same highly connected region are more likely to constitute coherent dialog.
We then present a novel graph grounded policy model to plan a long-term success oriented vertex sequence to guide response generation. Specifically, as illustrated by the three pink lines in Figure 1, given a user message, CG-Policy first links its keywords to CG to obtain hit what-vertices. Next, the policy model learns to select a what-vertex from one-hop what-vertex neighbors of all hit what-vertices, and then select a how-vertex from how-vertex neighbors of the chosen what-vertex. Finally, the two selected vertices are utilized to guide response generation. Thus we leverage the prior dialog-transition information (as graph edges) to narrow down candidate response content for more effective policy decision, instead of using the whole set of keywords as candidate actions. Moreover, to facilitate the modeling of long-term influence of policy decisions in an ongoing dialog, we first present novel CG based rewards to better measure the long-term influence of selected actions. We then employ a graph attention mechanism and graph embedding to encode global structure information of CG into dialog state representations, enabling global information aware decisions.
- This work is the first attempt that represents dialog transitions as a graph, and conducts graph grounded policy learning with RL. Supported by CG and this policy learning framework, CG-Policy can respond better in terms of local appropriateness and global coherence.
- Our study shows that: (1) one-hop what-vertex neighbors of hit what-vertices provide locally-appropriate and diverse response content; (2) the CG based rewards can supervise the policy model to promote a globally-coherent dialog; (3) the use of how-vertices in CG can improve response diversity; (4) the CG can help our system succeed in the task of target-guided conversation, indicating that it gives us more control over the dialog policy.
Policy learning for chitchat generation To address the degeneration issue of word-level policy models (Li et al., 2016b; Zhang et al., 2018b), previous works decouple policy learning from response generation, and then use utterance-level latent variables (Zhao et al., 2019) or keywords (Yao et al., 2018) as RL actions to guide response generation. In this work, we investigate how to use prior dialog-transition information to facilitate dialog policy learning.
Knowledge aware conversation generation There are growing interests in leveraging knowledge bases for generation of more informative responses (Dinan et al., 2019; Ghazvininejad et al., 2018; Moghe et al., 2018; Zhou et al., 2018; Liu et al., 2019; Bao et al., 2019; Xu et al., 2020). In this work, we employ a dialog-modeling oriented graph built from dialog corpora, instead of a external knowledge base, in order to facilitate multi-turn policy learning, instead of dialog informativeness improvement.
Specifically, we are motivated by (Xu et al., 2020). The method in (Xu et al., 2020) has the issue of cross-domain transfer since it relies on labor-intensive knowledge graph grounded multiturn dialog datasets for model training. Compared with them, our conversational graph is automatically built from dialog datasets, which introduces very low cost for training data construction. Furthermore, we decouple conversation modeling into two parts: "what to say" modeling and "how to
<sup>1</sup>Each mechanism is a MLP network to model how to express response content (Chen et al., 2019).
Figure 2: The architecture of our CG-Policy that consists of NLU, state/action, policy, and NLG. We first construct conversational graph from dialog corpus. Then we train CG-Policy with RL. The upper-right part shows the details of input/output of each module.
say" modeling. It is reasonable to only adjust the "what-" part when transfer to different domains which further reduces the domain transfer cost.
The overview of CG-Policy is presented in Figure 2. Given a user message, to obtain candidate actions, the NLU module attempts to retrieve contextually relevant subgraphs from CG. The state/action module maintains candidate actions, history keywords that selected by policy at previous turns or mentioned by user, and the message. The policy module learns to select a response keyword and a responding mechanism from the above subgraphs. The NLG module first encodes the message into a representation using a message encoder and the selected mechanism, and then employs a Seq2BF model $^{2}$  (Mou et al., 2016) to produce a response
Figure 3: The Multi-mapping based generator for NLG in which we use a Seq2BF based model (Mou et al., 2016) as the decoder.
with the above representation and the selected keyword as input. The models used in CG construction/policy/NLG/reward are trained separately.
3.1 Background: Multi-mapping Generator for NLG
To address the "one-to-many" semantic mapping problem for conversation generation, Chen et al.(2019) proposed an end-to-end multi-mapping model in which each responding mechanism (a MLP network) models how to express response content (e.g. responding with a specific sentence function). In test procedure, they randomly select a mechanism for response generation.
As shown in Figure 3, the generator consists of a RNN based message encoder, a set of responding mechanisms, and a decoder. First, given a dialog message, the message-encoder represents it as a vector  $\mathbf{x}$ . Second, the generator uses a responding mechanism (selected by policy) to convert  $\mathbf{x}$  into a response representation  $\bar{r}$ . Finally,  $\bar{r}$  and a keyword (selected by policy) are fed into the decoder for response generation. To ensure that the given keyword will appear in generated responses, we introduce another Seq2BF based decoder (Mou et al., 2016) to replace the original RNN decoder. Moreover, this generator is trained on a dataset with pairs of [the message, a keyword extracted from a response]-the response. $^3$
3.2 CG Construction
Given a dialog corpus  $D$ , we construct the CG with three steps: what-vertex construction, how-vertex construction, and edge construction.
3If multiple keywords are extracted from the response, we randomly choose one; and if no keyword exists in the response, we randomly sample a word from the response to serve as "keyword".
2It decodes a response starting from the input keyword, and generates the remaining previous and future words subsequently. In this way, the keyword will appear in the response.
What-vertex construction To extract content words from  $D$  as what-vertices, we use a rule-based keyword extractor to obtain salient keywords from utterances in  $D$ . After removing stop words, we obtain all the keywords as what-vertices.
How-vertex construction We obtain a set of  $N_r$  responding mechanisms from the generator described in Section 3.1. Then they are used as how-vertices. Notice that all the how-vertices in CG share the same set of responding mechanisms.
Edge construction There are two types of edges in CG. One is to join two what-vertices and the other is to join a what-vertex and a how-vertex. To build the first type of edges, we first construct another dataset that consists of keyword pairs, where each pair consists of any two keywords extracted from the message and the response respectively in  $D$ . To capture natural transitions between keywords, we train another multi-mapping based model on this new dataset. For each what-vertex  $v^w$ , we find appropriate keywords as its responses by selecting top five keywords decoded (decoding length is 1) by each responding mechanism, and then connect  $v^w$  to vertices of these keywords.
To build the second type of edges, for the [message-keyword]-response pair in  $D$  (described in Section 3.1), we use the ground-truth response to select the most suitable mechanism for each keyword. Then, given a what-vertex  $v^w$ , we select top five mechanisms that are frequently selected for  $v^w$ 's keyword. Then we build edges to connect  $v^w$  to each of the top ranked how-vertices. These edges lead to responding mechanisms that are suitable to generate  $v^w$ .
3.3 NLU
To obtain subgraphs to provide high-quality candidate actions, we first extract keywords in the last utterance of the context (message) using the same tool in CG construction, and then link each keyword to the CG through exact string matching, to obtain multiple hit what-vertices. Then we retrieve a subgraph for each keyword, and use vertices (exclude hit what-vertices) in these subgraphs as candidate actions. Each subgraph consists of three parts: the hit what-vertex, its one-hop neighboring
0. Prepare dataset D and pretrained embedding.
1. Construct the what-vertex set. (3.2)
2. Train a multi-mapping based generator for NLG. (3.1) Responding mechanisms constitute the how-vertex set.
3. Construct edges between two what-vertices or a what-vertex and a how-vertex. (3.2)
4. Train a scoring model for local relevance. (3.6)
5. Train TransE based embedding and PageRank scores for what-vertices. (3.6)
6. Calculate shortest path distances between any two what-vertices. (3.6)
7. Train a original multi-mapping based with a RNN decoder on D for user-simulator. (4.3)
8. Optimize policy with reinforcement learning, where parameters in other modules stay intact. (3.7)
Table 1: The training procedure of CG-Policy.
what-vertices, and how-vertices being connected to the above neighbors. If there are no keywords to be extracted from the message or to be linked to CG, we reuse the retrieved subgraphs at the last time.6 Thus we leverage the CG to provide high-quality candidate actions, instead of using the whole set of candidates as done in previous work (Yao et al., 2018).
3.4 State/Action
This module maintains candidate actions, history keywords that selected by the policy or mentioned by user, and the message. Moreover, we use the message-encoder from Section 3.1 to represent the message as a vector  $\mathbf{x}$ , and then we use all the responding mechanisms from Section 3.1 to convert  $\mathbf{x}$  into  $N_{r}$  candidate response representations  $\{\mathbf{r}_j\}_{j=1}^{N_r}$ , which will be used in the policy.
3.5 Policy
State representation The state representation  $\mathbf{s}_t$  at the  $t$ -th time step is obtained by concatenating a message representation  $\mathbf{s}_t^M$  and a history keywords representation  $\mathbf{s}_t^V$  that are encoded by two RNN encoders respectively. Formally,
$$
\mathbf {s} _ {t} = \left[ \mathbf {s} _ {t} ^ {M}; \mathbf {s} _ {t} ^ {V} \right]. \tag {1}
$$
To enable global information aware policy decisions, we employ a graph attention mechanism and graph embedding to encode global structure information into state representation.
Recall that we have a subgraph for each keyword in the message obtained by NLU. Here each subgraph  $g_{i}$  consists of a hit what-vertex,
<sup>6</sup>If we encounter this case at the first time step, hit what-vertices are set as what-vertices that contain the top-5 high-frequency keywords in  $D$ .
$^{4}$ github.com/squareRoot3/Target-Guided-Conversation
<sup>5</sup>We ever tried other methods for edge construction, e.g., PMI (Yao et al., 2018). Finally we found that our method can provide more diverse response keyword candidates, while PMI tends to provide high-frequency keyword candidates. Here we use a RNN based decoder to replace the Seq2BF.
its what-vertex neighbors (here we remove howvertices) and edges between them. Formally,  $g_{i} = \{\tau_{k}\}_{k = 1}^{N_{g_{i}}}$  , where each  $\tau_{k}$  is a triple with  $\tau_{k} = (\text{head}_{k},\text{rel}_{k},\text{tail}_{k})$  , and  $N_{g_i}$  is the number of triples in  $g_{i}$  . For non keywords in the message, a NULL subgraph is used.
Then we calculate a subgraph vector  $\mathbf{g}_i$  as a weighted sum of head vectors and tail vectors in the triples.
$$
\mathbf {g} _ {i} = \sum_ {k = 1} ^ {N _ {g _ {i}}} \alpha_ {k} \left[ \mathbf {e} _ {\text {h e a d} _ {k}}; \mathbf {e} _ {\text {t a i l} _ {k}} \right],
$$
$$
\alpha_ {k} = \frac {\exp \left(\beta_ {k}\right)}{\sum_ {m = 1} ^ {N _ {g _ {i}}} \exp \left(\beta_ {m}\right)}, \tag {2}
$$
$$
\beta_ {k} = \mathbf {e} _ {r e l _ {k}} ^ {T} \tanh (\mathbf {W} _ {h} \mathbf {e} _ {h e a d _ {k}} + \mathbf {W} _ {t} \mathbf {e} _ {t a i l _ {k}}).
$$
Here  $\mathbf{e}_{*}$  represents pretrained graph embedding (TransE (Bordes et al., 2013)) that are not updated during RL training.  $\mathbf{W}_h$  and  $\mathbf{W}_t$  are parameters.
$\mathbf{s}_t^M$  is obtained by recursively feeding a concatenated vector  $\mathbf{e}_i = [\mathbf{w}_i^c;\mathbf{g}_i]$  into a vanilla RNN unit, where  $\mathbf{w}_i^c$  (as model parameters) is the embedding of the keyword  $w_{i}^{c}$ . Thus we encode the global graph structure information into RL state representations, enabling a global-information aware policy model. Moreover, we calculate  $\mathbf{s}_t^V$  in a similar way.
Policy decision Each decision consists of two sequential sub-decisions. First the what-policy selects a what-vertex from candidate what-vertices, and then the how-policy selects a how-vertex from how-vertex neighbors of the selected what-vertex.
With  $\mathbf{s_t}$  as the state representation, the what-policy  $\mu^{what}$  is defined by:
$$
\mu^ {\text {w h a t}} \left(\mathbf {s} _ {t}, \mathbf {v} _ {j} ^ {w}\right) = \frac {\exp \left(\mathbf {s} _ {t} ^ {T} \mathbf {v} _ {j} ^ {w}\right)}{\sum_ {l = 1} ^ {N _ {w - a c t}} \exp \left(\mathbf {s} _ {t} ^ {T} \mathbf {v} _ {l} ^ {w}\right)}, \tag {3}
$$
where  $\mathbf{v}_j^w$  (as model parameters, different from both  $\mathbf{w}_i^c$  and  $\mathbf{e}_*$ ) is the embedding of the  $j$ -th candidate what-vertices, and  $N_{w\_act}$  is the number of candidate what-vertices.
The how-policy  $\mu^{how}$  is defined by:
$$
\mu^ {h o w} (\mathbf {s} _ {t}, \mathbf {r} _ {i}) = \frac {\lambda_ {i} \exp \left(\mathbf {s} _ {t} ^ {T} \mathbf {r} _ {i}\right)}{\sum_ {j = 1} ^ {N _ {r}} \lambda_ {j} \exp \left(\mathbf {s} _ {t} ^ {T} \mathbf {r} _ {j}\right)}, \tag {4}
$$
where  $\mathbf{r}_i$  is a candidate response representation in the state module, and  $\lambda_{i}$  is mechanism mask.  $\lambda_{i}$  is set as 1 if the  $i$ -th responding mechanism is one of neighbors of the selected what-vertex, otherwise 0.
3.6 Rewards
Following previous works, we consider these utterance-level rewards:
Local relevance We use a state-of-the-art multi-turn response selection model, DualEncoder in (Lowe et al., 2015), to calculate local relevance.
Repetition Repetition penalty is 1 if the generated response shares more than  $60\%$  words with any contextual utterances, otherwise 0.
Target similarity For target-guided conversation, we calculate cosine similarity between the chosen keyword and the target word in pretrained word embedding space as target similarity.
To leverage the global graph structure information of CG to facilitate policy learning, we propose the following rewards:
Global coherence We calculate the average cosine distance between the chosen what-vertex and one of history what-vertices (selected or mentioned previously) in TransE based embedding space (also used in Equation 2) as coherence reward.
Sustainability It is reasonable to promote what-vertices with a large number of neighbors to generate more sustainable, coherent, and diverse dialogs. For this reward, we calculate a PageRank score (calculated on the full CG) for the chosen what-vertex.
Shortest path distance to the target For target-guided conversation, if the chosen what-vertex is closer to the target what-vertex in terms of shortest path distance when compared to the previously chosen what-vertex, then this reward is 1, or 0 if the distance does not change, otherwise -1.
Moreover, we define the final reward as a weighted sum of the above-mentioned factors, where the weight of each factor is set as [0.5, -5, 0, 3, 8000, 0] by default. We see that our rewards can fully leverage dialog transition information in training data by using not only utterance based rewards (e.g., local relevance), but also graph based rewards (e.g., coherence, sustainability).
3.7 Policy Optimization
To make training process more stable, we employ the A2C method (Sutton and Barto, 2018) for optimization. Moreover, we only update policy pa
<sup>7</sup>If no keyword is chosen, as in baseline models, we calculate target similarity for each word in response and select the closest one.
<sup>8</sup>We optimize these values on Weibo dataset by grid search. The weights of the third/sixth factors are set as 0 by default because they are proposed for target-guided conversation.
rameters, and the parameters of other modules stay intact during RL training.
3.8 NLG
As described in Section 3.1, we use the mechanism selected by how-policy to convert x into a response representation  $\bar{r}$ . Then we feed the keyword in the selected what-vertex and  $\bar{r}$  into a Seq2BF decoder (Mou et al., 2016) for response generation.
We conduct experiments on two widely used open-domain dialog corpora.
Weibo corpus (Shang et al., 2015). This is a large micro-blogging corpora. After data cleaning, we obtain 2.6 million pairs for training, 10k pairs for validation and 10k pairs for testing. We use publicly-available lexical analysis tools to obtain POS tag features for this dataset and then we further use this feature to extract keywords from utterances. We use Tencent AI Lab Embedding<sup>11</sup> for embedding initialization in models.
Persona dialog corpus (Zhang et al., 2018a). This ia a crowd-sourced dialog corpora where each participant plays the part of an assigned persona. To evaluate policy controllability brought by CGPolicy, we conduct an experiment for target-guided conversation on the Persona dataset as done in (Tang et al., 2019). The training set / validation set / testing set contain 101,935 / 5,602 / 5,371 utterances respectively. Embeddings are initialized with Glove (Pennington et al., 2014).
Conversational Graph The constructed CG on Weibo corpus contains 4,000 what-vertices and 74,362 edges among what-vertices, where  $64\%$  edges are evaluated as suitable for chatting by three human annotators.[12] The constructed CG on Persona corpus contains 1,500 what-vertices and 21,902 edges among what-vertices, where  $67\%$  edges are evaluated as suitable for chatting by three human annotators.
We carefully select three SOTA methods that focus on dialog policy learning as baselines.
LaRL It is a latent variable driven dialog policy model (Zhao et al., 2019). We use their released codes and choose the multivariate categorical latent variables as RL actions since it performs the best. For target-guided conversation, we implement another model LaRL-Target, where we add the "target similarity" factor into RL rewards, and its weight is set as 4 by grid search.
ChatMore We implement the keyword driven policy model (Yao et al., 2018) by following their original design. For target-guided conversation, we implement ChatMore-Target, where we add the "target similarity" factor into RL rewards, and its weight is set as 4 by grid search.
TGRM It is a retrieval based model for target-guided conversation, where the keyword chosen at each turn must move strictly closer (in embedding space) to a given target word (Tang et al., 2019). For target-guided conversation, we use the codes released by the original authors, denoted as TGRM-Target, and we use their kernel version since it performs the best. To suit the task of open-domain conversation on Weibo, we remove the unnecessary constraint on keyword's similarity with the target word, denoted as TGRM.