SlowGuess/ACL_2020_OCR huggingface dataset:
CG-Policy It is our system presented in Section 3. For target-guided conversation, we implement another system CG-Policy-Target, where we use an additional feature, the "shortest path distance to the target" factor, to augment the original what-vertex representation  $\mathbf{v}_j^w$  in the what-policy  $\mu^{what}$ . Formally,  $\bar{\mathbf{v}}_j^w = \mathbf{W}_1*[\mathbf{v}_j^w;\mathbf{e}_{d_j}]$ , where  $\bar{\mathbf{v}}_j^w$  is the augmented representation,  $\mathbf{W}_1$  is a weighting matrix,  $\mathbf{e}_{d_j}$  is an embedding for the distance value  $d_j$ , and  $\bar{\mathbf{v}}_j^w$  has the same size with  $\mathbf{v}_j^w$ . We also use this factor in reward estimation and its weight is set as 5 by grid search, and we don't use the "target similarity" factor. Moreover, we use the same dialog corpora to construct CG, train user simulator, reward functions, and the NLG module for CG-Policy.
4.3 User Simulator
We use the same user simulator for RL training of LaRL, ChatMore and CG-Policy. The user simulator is the original multi-mapping based generator with a RNN decoder, which is pretrained on dialog corpus and not updated during policy training. Please refer to (Chen et al., 2019) for more details. During testing, all the systems share this simulator.
$^{13}$ github.com/squareRoot3/Target-Guided-Conversation
9Please see the supplemental material for more details.
10 ai.baidu.com/
ai.tencent.com/ailab/nlp/embedding.html
12We randomly sample 500 edges for evaluation.
4.4 Evaluation Settings
Conversation with user simulator Following previous work (Li et al., 2016b; Tang et al., 2019), we use a user simulator to play the role of human and let each of the models converse with it. Given a randomly selected model, we randomly select an utterance from all the utterances (at the starting position of sessions) in test set for the model to start a conversation. Moreover, we set a maximum allowed number of turns, which is 8 in our experiment. Finally, we collect 100 model-simulator dialogs for evaluation. For single-turn level evaluation, we randomly sample 100 message-response pairs from the dialogs for each model.
Conversation with human Following previous work (Tang et al., 2019), we also perform human evaluation for a more reliable system comparison. Given a model to be evaluated, we randomly select a dialogue from test set and pick its first utterance for the model to start a conversation with a human. Then the conversation will continue till 8 turns are reached. Finally, we obtain 50 dialogs for evaluation. For single-turn level evaluation, we randomly sample 100 message-response pairs from the dialogs for each model.
4.5 Evaluation Metrics
Metrics such as BLEU and perplexity have been widely used for dialog evaluation (Li et al., 2016a; Serban et al., 2016), but it is widely debated how well these automatic metrics are correlated with true response quality (Liu et al., 2016). Since the proposed system does not aim at predicting the highest-probability response at each turn, but rather the long-term success of a dialog (e.g., coherence), we do not employ BLEU or perplexity for evaluation, and we propose the following metrics.
4.5.1 Multi-turn Level Metrics
Global coherence We define incoherence problems as follows: (1) Inconsistent dialogs where the model contradicts with itself, e.g., the model says he is a driver before and then says he is a doctor; (2) One-side dialogs in which the model ignores the user's topics with two or more consecutive turns. A session will be rated "0" if it contains more than three incoherence cases, or "+" if a session contains 2 or 3 cases, otherwise "+"
Distinct The metric Dist-i calculates the ratio of distinct i-gram in generated responses (Li et al., 2016a). We use Dist-2 to measure the diversity of generated responses.
Methods	Cohe.	Dist-2	Appr.	Infor.
LaRL	0.85	0.12	0.55	0.77
ChatMore	0.95	0.05	0.58	0.93
TGRM	0.79	0.42	0.68	1.00
CG-Policy	1.33	0.31	0.73	1.00
Table 2: Results for dialogs with simulator on Weibo.
Dialog-target success rate For target-guided conversation, we measure the success rate of generating the target word within 8 turns.
4.5.2 Single-turn Level Metrics
Local appropriateness<sup>14</sup> A response will be rated “0” if it is inappropriate as an reply to the given message, otherwise “1”.
Informativeness "0" if a response is a "safe" response, e.g. "I don't know", otherwise "1".
4.6 Evaluation Results
4.6.1 Setting
We ask three annotators to judge the quality of each dialog (at multi-turn level) or utterance pair (at single-turn level) for each model. Notice that model identifiers are masked during evaluation.
4.6.2 Conversation with simulator
As shown in Table 2, CG-Policy significantly outperforms (sign test,  $\mathbf{p}$ -value  $< 0.01$ ) baselines in terms of global coherence and local appropriateness. It indicates that the CG can effectively facilitate policy learning (see the ablation study for further analysis). For LaRL, its single-turn response quality is worse than other models. It might be explained by that their latent variables are not fine-grained enough to provide sufficient information to guide response generation. ChatMore tends to select high-frequency or generic keywords, resulting in its worst performance in terms of Dist-2. TGRM performs the best in terms of Dist-2 and informativeness, indicating that retrieval-based models can produce more diverse responses than generation based models. It is consistent with the conclusions in previous work (Chen et al., 2017; Zhang et al., 2018a). However, TGRM performs the worst in terms of coherence, since TGRM does not use RL framework. It indicates the importance of RL framework for multi-turn dialog modeling. Here the Kappa value for inter-annotater agreement is above 0.4, indicating moderate agreement.
14We do not consider if a response is appropriate or not for the selected responding mechanism.
Methods	Cohe.	Dist-2	Appr.	Infor.
LaRL	0.82	0.22	0.52	0.74
ChatMore	0.88	0.15	0.54	0.94
TGRM	0.77	0.63	0.61	1.00
CG-Policy	1.26	0.47	0.67	1.00
4.6.3 Conversation with human
As shown in Table 3, CG-Policy outperforms baselines in terms of both global coherence and local appropriateness (sign test,  $\mathbf{p}$ -value  $< 0.01$ ), which is consistent with the results in Table 2. The Kappa value is above 0.4, indicating moderate agreement.
4.6.4 Ablation study
We conduct an ablation study for CG-Policy on Weibo corpus to investigate why CG-Policy performs better. First, to evaluate the contribution of CG, we remove the CG from CG-Policy, denoted as CG-Policy-noCG, where we do not use graph structure information for action space pruning and reward design. Moreover, we attempt to use the CG (without how-vertices) to augment the ChatMore model for action space pruning and reward design, denoted as Chatmore-CG. As shown in Table 4, the performance of CG-Policy-noCG drops dramatically in terms of coherence, Dist-2 and appropriateness when compared to the original model. Moreover, CG can boost the performance of ChatMore in terms of most of metrics. It indicates that the use of CG is crucial to the superior performance of CG-Policy, and it can also help other models, e.g., ChatMore. Second, to evaluate the contribution of CG for action space pruning or reward design respectively, we implement two system variants: (1) we use all the what-vertices in CG as action candidates at each turn, denoted as CG-Policy-noCGact; (2) we remove all the CG-based factors from RL rewards, denoted as CG-Policy-noCGrwd. As shown in Table 4, the performance of CG-Policy-noCGact drops significantly in terms of Dist-2 as it tends to select high-frequency keywords like ChatMore, indicating the importance of graph paths to provide both locally-appropriate and diverse response keywords. Moreover, the performance of CG-Policy-noCGrwd drops significantly in terms of coherence, indicating that CG based rewards can effectively guide CG-Policy to promote coherent dialogs. Third, we remove how-vertices from CG, denoted as CG-Policy-noCGhow. As shown in Table 4, how-vertex removal hurts its per
Table 3: Results for dialogs with human on Weibo.
Methods	Cohe.	Dist-2	Appr.	Infor.
CG-Policy	1.33	0.31	0.73	1.00
ChatMore	0.95	0.05	0.58	0.93
ChatMore-CG	1.15	0.14	0.65	0.91
CG-Policy-noCG	1.03	0.07	0.62	1.00
CG-Policy-noCGact	1.11	0.08	0.68	1.00
CG-Policy-noCGrewd	1.06	0.19	0.64	1.00
CG-Policy-noCGhow	1.21	0.13	0.65	1.00
formance in Dist-2, indicating the importance of how-vertices for response diversity.
4.7 The Task of Target-guided Conversation
Besides maintaining coherence, CG grounded policy learning can enable more control over dialog models, which is important to achieve certain goals for chatbot, e.g. proactive leading to certain chatting topics (keywords) or certain products.
4.7.1 Setting
Following the setting in (Tang et al., 2019), where we randomly sample a keyword as the target word for each session in testing procedure. Here we use a multi-mapping based user simulator trained on the Persona dataset for evaluation.
Table 4: Ablation study for CG-Policy on Weibo.
Methods	Succ.(%)	Cohe.	Appr.	Infor.
LaRL-Target	1	0.91	0.62	0.91
ChatMore-Target	6	0.93	0.65	0.97
TGRM-Target	69	0.96	0.67	1.00
CG-Policy-Target	98	1.17	0.75	1.00
Table 5: Results for target-guided dialogs on Persona.
4.7.2 Results
Table 5 presents the results on 100 dialogs for each model. We see that CG-Policy-Target can significantly outperform baselines in terms of dialog-target success rate (sign test,  $\mathbf{p}$ -value  $< 0.01$ ). It can be seen that that CG-Policy can successfully lead the dialog to a given target word by learning to walk over the CG, indicating that this graph gives us more control over the policy. LaRL-Target and ChatMore-Target perform badly in terms of success rate. It may be explained by that they lack the ability of proactive dialog content planning.
4.8 Analysis of Responding Mechanisms
Figure 4 provides representative words of each mechanism. For example, for Mech-1, its keywords are mainly subjective words (e.g. think) for
15We select words that occur frequently in responses guided by this mechanism but rarely occur with other mechanisms.
generation of responses with respect to personal opinion or intention. For Mech-2, it tends to respond with a specific type of mood.
Mech-1	Mech-2	Mech-3	Mech-4	Mech-5
以为think	哈哈haha	哪where	漂亮beautiful	别no
想want	哇 wow	什么what	可爱cute	还是or else
信believe	好吧alright	?	萌cuddly	没有no
Figure 4: Representative words of responding mechanisms.
In this paper we present a novel graph grounded policy learning framework for open-domain multi-turn conversation, which can effectively leverage prior information about dialog transitions to foster a more coherent and controllable dialog. Experimental results demonstrate the effectiveness of this framework in terms of local appropriateness, global coherence and dialog-target success rate. In the future, we will investigate how to extend the CG to support hierarchical topic management in conversational systems.
We are grateful for the support from Yan Zeng at the initial stage of this work. We also thank the anonymous reviewers for their helpful comments and suggestions. This work is supported by the National Key Research and Development Project of China (No.2018AAA0101900) and the National Natural Science Foundation of China (NSFC) via grant 61976072.
Siqi Bao, Huang He, Fan Wang, Rongzhong Lian, and Hua Wu. 2019. Know more about each other: Evolving dialogue strategy via compound assessment. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5382-5391.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795.
Chaotao Chen, Jinhua Peng, Fan Wang, Jun Xu, and Hua Wu. 2019. Generating multiple diverse re
sponses with multi-mapping and posterior mapping selection. Proceedings of IJCAI.
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Recent advances and new frontiers. SIGKDD Explorations.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In Proceedings of ICLR. Association for Computational Linguistics.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Proceedings of AAAI 2018, pages 5110-5117. Association for the Advancement of Artificial Intelligence.
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2333-2343.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443-2453.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119.
Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016b. Deep reinforcement learning for dialogue generation. In Proceedings of EMNLP, pages 1192-1202.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132.
Zhibin Liu, Zheng-Yu Niu, Hua Wu, and Haifeng Wang. 2019. Knowledge aware conversation generation with explainable reasoning over augmented graphs. In EMNLP-IJCNLP.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 285-294.
Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018. Towards exploiting background knowledge for building conversation systems. In Proceedings of EMNLP, pages 2322-2332. Association for Computational Linguistics.
Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. 2016. Sequence to backward and forward sequences: A content-introducing approach to generative short-text conversation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3349-3358.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C. Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of AAAI, pages 3776-3784.
Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of ACL-IJCNLP, volume 1, pages 1577-1586.
Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT Press.
Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric P. Xing, and Zhiting Hu. 2019. Target-guided open-domain conversation. In Proceedings of ACL.
Jun Xu, Haifeng Wang, Zhengyu Niu, Hua Wu, and Wanxiang Che. 2020. Knowledge graph grounded goal planning for open-domain conversation generation. In Thirty-Fourth AAAI Conference on Artificial Intelligence.
Lili Yao, Ruijian Xu, Chao Li, Dongyan Zhao, and Rui Yan. 2018. Chat more if you like: Dynamic cue words planning to flow longer conversations. arXiv preprint arXiv:1811.07631.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018a. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213.
Wei-Nan Zhang, Lingzhi Li, Dongyan Cao, and Ting Liu. 2018b. Exploring implicit feedback for open domain conversation generation. In Thirty-Second AAAI Conference on Artificial Intelligence.
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement
learning in end-to-end dialog agents with latent variable models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1208-1218.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In Proceedings of IJCAI-ECAI.
Training Details and Two Conversation Cases For fair comparison, all models share the same vocab (maximum size is 50000 for the Weibo corpus or 20000 for the Persona corpus), initialized word embedding (size is 200), and keyword set. Further, One layer bidirectional GRU-RNN (hidden size is 512) is utilized for all encoders. Dropout rate is 0.3, and optimizer is Adam(lr=2le-3) for all models.
We initialize each session with a starting utterance chosen from the training dataset randomly. Maximum turns is set as 8 and the discounting weight for rewards is set as 0.95.
LaRL
Categorical z	M=10,K=20
β	0.01
CG-Policy
TransE Embedding Size	100
Number of Responding Mechanisms Nr	10
Table 6: Training details for models.
B:	晚饭吃了么 [Did you have dinner?]
U:	我想吃排骨了，晚上炖排骨吃吧 [I want to have spareribs. Let's stew spareribs in the evening.]
B:	我请你吃猪蹄吧，我也要吃 [I'll treat you to pig hooves. I'll eat them, too.]
U:	炸猪蹄很好吃啊 [Fried pig hooves are delicious.]
B:	这是什么类型啊，你吃过么？[What kind of food is it? Have you eaten it?]
U:	我还是吃炸酱面吧，最近减肥 [I'd better eat fried sauce noodles. I've lost weight recently.]
B:	正宗了，我也想吃 [It's authentic. I want to eat it, too.]
U:	我想跟你一起跳舞 [I want to dance with you.]
B:	好萌啊，好可爱，好喜欢 [how budding! You are lovely and I like it.]
U:	我想起我家那条狗了，泰迪狗，给你看看照片 [I think of my dog, Teddy. Show you the picture.]
B:	我想养狗了，我也要养一只 [I want to have a dog. I want to have one, too.]
U:	我想回家吃狗肉 [I want to go home and eat dog meat.]
B:	可怜的泰迪啊，哈哈哈 [Poor Teddy, haha]
U:	我不是要回家吃我的泰迪！[I'm not going home to eat my Teddy!]
Figure 5: Case 1: One conversation between CG-Policy and human, where “B” is CG-Policy and “U” is human. The red words are keywords. We translate the original Chinese utterances into English.
B:	昨天去菜市场买了一大袋土豆，累死我了
[Yesterday I went to the vegetable market and bought a big bag of potatoes. I am exhausted.]
U:	话说楼下新开了个超市 [I heard that there is a new supermarket downstairs.]
B:	我去了，不打折了，哈哈 [I've been there. No discount yet, haha.]
U:	里面的水果新鲜么 [Is the fruit fresh?]
B:	我都没看出是有多难吃 [It's untasty, even more than I expected.]
U:	刚开业就不打折,真抠门,我去淘宝买吧
[It's really stingy that have no discount just after opening. I'll go to Taobao to buy fruit.]
B:	在哪里买啊，低价代购？求地址 [Which shop? Substitute purchase at low price? Give me the link address.]
U:	很多店铺吧,搜一下就好了 [A lot of shops can be your choice. Just search them.]
B:	万能的淘宝网店，天猫要火啊
[Taobao shops are really omnipotent, TianMao is becoming more and more famous]
U:	双十一要到了，马云赚钱好开心
[The Double Eleventh Shopping Festival is coming. Ma Yun is happy to earn lots of money.]
B:	我去年双十一购买了5件商品 [I bought five items last year in the Double Eleventh Shopping Festival.]
U:	美死了,哈哈,好可爱,哈哈 [Haha, you are lovely, haha]
B:	你也很可爱 [You're lovely, too]
U:	我想买一只萨摩耶，好可爱 [I want to buy a Samoa. It's cute.]
Figure 6: Case 2: One conversation between CG-Policy and human, where “B” is CG-Policy and “U” is human. The red words are keywords. We translate the original Chinese utterances into English.