SlowGuess/ACL_2020_OCR huggingface dataset:
IterDPL Iterative Dialog Policy Learning (Liu and Lane, 2017) updates two agents iteratively using single RL training to reduce the risk of nonstationarity when jointly training the two agents.
5 Automatic Evaluation
5.1 Interaction between Two Agents
A set of 1,000 user goals are used for automatic evaluation as shown in Table 1. When the dialog is launched, two agents interact with each other around a given user goal. The performance of interaction between the two trained policies are shown in Table 2. MADPL reaches the highest match rate and task success among all the methods. It manages to improve the success rate of the pretrained policies from  $49.7\%$  to  $70.1\%$ . Single RL policies (row 2 to 4) have limited improvement, and even decline in match rate since they assume a stationary environment. The comparison between CRL and IterDPL indicates the effectiveness of CTDE in the multi-agent task. The superiority of MADPL against CRL shows that two agents benefit from the role-aware reward decomposition in HVN. The learning curves in Fig. 3 illustrates that the success rate grows rapidly in MADPL, and it always improves the success rate as the training proceeds.
The average reward of each component reward is shown in 4. We run 10 different instances of MADPL with different random seeds. The solid curves correspond to the mean and the shaded region to the standard deviation of rewards over the
System	User	Turns	Inform	Match	Success
SL	SL	6.34	73.08	82.58	49.7
SL	RL	8.75	76.86	76.28	60.2
RL	SL	6.20	72.84	79.15	51.1
RL	RL	7.92	75.96	70.37	58.7
CRL	8.13	68.29	89.71	66.6
IterDPL	8.79	74.01	81.04	64.6
MADPL	8.96	76.26	90.98	70.1
Table 2: Performance of the interaction between the user agent and the system agent.
Figure 3: Learning curves of the interaction between the user agent and the system agent.
10 trials. We can observe that all the rewards increase steadily during the training process, which implies that HVN has estimated a proper return for policy training.
5.2 Interaction with Benchmark Policies
It is essential to evaluate a multi-agent dialog system whether all the agents understand the semantic interaction rather than invent an uninterpretable language (Kottur et al., 2017; Lee et al., 2019a). To this end, we use two benchmark policies in the standardized task-oriented dialog system platform Convlab (Lee et al., 2019b) to examine all the methods. Each benchmark is a strong rule-based system policy or user policy at the dialog act level, which is used as the simulated evaluation in the DSTC-8 Track 1 competition and show a high correlation with real user interaction (Li et al., 2020). The trained system/user policy in each method is directly deployed to interact with the benchmark user/system policy during the test without any other finetuning, which can be regarded as a weakly zero-shot experiment. The same goal set in Table 1 is used here.
Table 3 and Fig. 5 show the results of the interac
Figure 4: Learning curves of MADPL on system reward (top), user reward (middle) and global reward (bottom).
tion between the benchmark user policy and the system agent of each model. The SOTA performance from GDPL (Takanobu et al., 2019) that directly trains with benchmark user policy is also presented as the soft performance upper bound. Among all the methods, MADPL has achieved the highest task success and the second-highest match rate. All the methods experience a decline in inform F1 after the RL training. Fig. 5 also shows that the success rate is unstable during training. This is because the action space of the system policy is much larger, thus more challenging to learn. In spite of that, the success rate of MADPL shows a rising trend.
Table 4 and Fig. 6 show the results of the interaction between the user agent of each method and the benchmark system policy. Among all the methods, MADPL has achieved the highest inform F1 and task success. Though CRL improves the performance at the beginning, the success rate fails to increase further afterwards, while MADPL continues to improve all the time. This also indirectly indicates the advantage of using role-aware reward decomposition in HVN.
System	Turns	Inform	Match	Success
SL	7.76	83.33	85.84	84.2
RL	7.53	82.06	85.77	84.3
CRL	8.38	72.43	89.48	86.4
IterDPL	7.74	79.68	82.49	82.5
MADPL	7.63	79.93	89.24	87.7
GDPL	7.62	92.10	91.50	92.1
Table 3: Performance of the interaction between the benchmark user policy and each system agent.
Figure 5: Learning curves of the interaction between the benchmark user policy and each system agent.
User	Turns	Inform	Match	Success
SL	8.64	78.64	87.84	51.7
RL	11.18	85.69	92.13	77.2
CRL	11.31	86.58	92.89	74.7
IterDPL	12.53	84.68	92.57	75.5
MADPL	13.25	87.04	90.81	83.7
Figure 6: Learning curves of the interaction between each user agent and the benchmark system policy.
In summary, each policy trained from MADPL can interact well with the benchmark policy, which
Table 4: Performance of the interaction between each user agent and the benchmark system policy.
VS.	System Q	User Q	Success
W	D	L	W	D	L	W	D	L
SL/SL	55	22	23	61	25	14	68	26	6
RL/RL	49	23	28	52	28	20	70	19	11
IterDPL	50	27	23	56	30	14	64	24	12
Table 5: Human preference on dialog session pairs that MADPL wins (W), draws with (D) or loses to (L) baselines with regard to quality (Q) and success by majority voting.
implies that MADPL learns a reasonable dialog strategy.
5.3 Goal across Multiple Domains
We also investigate the domains in the user goals to observe the scalability of each method in the complex tasks. 200 goals are randomly sampled under each setting. Fig. 7 presents the results of the interaction between two agents in different numbers or classes of domains. The success rate decreases substantially as the number of domains increases in the goal. When there are 3 domains in the goal, RL/RL gets a high inform F1 but a low match rate, IterDPL gets a high match rate but a low inform F1, while MADPL can still keep a high inform F1 and match rate, and obtains the highest task success. In terms of the class of domains, there are 7/10/6 inormable slots that needs to be tracked in the Restaurant/Hotel/Train domain respectively. Among these, MADPL outperforms other baselines in the Restaurant and Hotel domains, and performs comparably in the Train domain. In brief, all the results indicate that MADPL has good scalability in multi-domain dialog.
For human evaluation, we hire Amazon Mechanical Turkers to conduct pairwise comparison between MADPL and baselines. Since all the policies work at the dialog act level, we generate the texts from dialog acts using hand-crafted templates to make the dialog readable. Each Turker is asked to read a user goal first, then we show 2 dialog sessions around this user goal, one from MADPL and the other from another baseline. We randomly sample 100 goals for each baseline. For each goal, 5 Turkers are asked to judge which dialog is better (win, draw or lose) according to different subjective assessments independently: (I) system quality, (II) user quality,
Figure 7: Performance of dialog agents according to the different number (left) or class (right) of domains in the dialog.
and (III) task success. The system quality metric evaluates whether the system policy provides the user with the required information efficiently, and the user quality metric evaluates whether the user policy expresses the constraints completely in an organized way. Note that we do not evaluate the quality of language generation here.
Table 5 shows the results of human preference by majority voting. We can observe that the high win rate of MADPL on the task success is consistent with the results of automatic evaluation, and MADPL outperforms three baselines significantly in all aspects (sign test, p-value  $< 0.01$ ) except for the system quality against RL/RL policies.
The proportion of the pairwise annotations in which at least 3 of 5 annotators assign the same label to a task is  $78.7\% / 77.3\% / 83.3\%$  for system quality/user quality/task success, respectively. This indicates that annotators have moderate agreements. The human judgements align well with the results of automatic evaluation, which also indicates the reliability of the metrics used in task-oriented dialog.
We present a multi-agent dialog policy algorithm, MADPL, that trains the user policy and the sys
tem policy simultaneously. It uses the actor-critic framework to facilitate pretraining and bootstrap RL training in multi-domain task-oriented dialog. We also introduce role-aware reward decomposition to integrate the task knowledge into the algorithm. MADPL enables the developers to set up a dialog system rapidly from scratch. It only requires the annotation of dialog acts in the corpus for pretraining and does not need to build a user simulator explicitly beforehand. Extensive experiments<sup>1</sup> demonstrate the effectiveness, reasonableness and scalability of MADPL.
As future work, we will apply MADPL in the more complex dialogs and verify the role-aware reward decomposition in other dialog scenarios.
This work was jointly supported by the NSFC projects (Key project with No. 61936010 and regular project with No. 61876096), and the National Key R&D Program of China (Grant No. 2018YFC0830200). We would like to thank THUNUS NExT Joint-Lab for the support. The code is available at https://github.com/truthless11/MADPL.
<sup>1</sup>We provide implementation details and case studies in appendix.
Andrew G Barto, Richard S Sutton, and Charles W Anderson. 1983. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, 13(5):834-846.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. 2002. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4):819-840.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz: A large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026.
Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. 2017. Learning cooperative visual dialog agents with deep reinforcement learning. In 2017 IEEE International Conference on Computer Vision, pages 2951-2960.
Layla El Asri, Jing He, and Kaheer Suleman. 2016. A sequence-to-sequence model for user simulation in spoken dialogue systems. 17th Annual Conference of the International Speech Communication Association, pages 1151-1155.
Mehdi Fatemi, Layla El Asri, Hannes Schulz, Jing He, and Kaheer Suleman. 2016. Policy networks with two-stage training for dialogue systems. In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 101-110.
Jun Feng, Heng Li, Minlie Huang, Shichen Liu, Wenwu Ou, Zhirong Wang, and Xiaoyan Zhu. 2018. Learning to collaborate: Multi-scenario ranking via multi-agent reinforcement learning. In 27th International Conference on World Wide Web, pages 1939-1948.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. 2018. Counterfactual multi-agent policy gradients. In 32nd AAAI Conference on Artificial Intelligence, pages 2974-2982.
Kallirroi Georgila, Claire Nelson, and David Traum. 2014. Single-agent vs. multi-agent techniques for concurrent reinforcement learning of negotiation dialogue policies. In 52nd Annual Meeting of the Association for Computational Linguistics, pages 500-510.
Izzeddin Gur, Dilek Hakkani-Tur, Gokhan Tur, and Pararth Shah. 2018. User modeling for task oriented dialogues. In 2018 IEEE Spoken Language Technology Workshop, pages 900-906.
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and generation
in negotiation dialogues. In 2018 Conference on Empirical Methods in Natural Language Processing, pages 2333-2343.
Simon Keizer, Milica Gašić, Filip Jurčíček, François Mairesse, Blaise Thomson, Kai Yu, and Steve Young. 2010. Parameter estimation for agenda-based user simulation. In 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 116-123.
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. 2017. Natural language does not emerge naturally in multi-agent dialog. In 2017 Conference on Empirical Methods in Natural Language Processing, pages 2962-2967.
Florian Kreyssig, Iñigo Casanueva, Paweł Budzianowski, and Milica Gasic. 2018. Neural user simulation for corpus-based policy optimisation of spoken dialogue systems. In 19th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 60-69.
Jason Lee, Kyunghyun Cho, and Douwe Kiela. 2019a. Countering language drift via visual grounding. In 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, pages 4376-4386.
Sungjin Lee, Qi Zhu, Ryuichi Takanobu, Zheng Zhang, Yaoqin Zhang, Xiang Li, Jinchao Li, Baolin Peng, Xiujun Li, Minlie Huang, and Jianfeng Gao. 2019b. Convlab: Multi-domain end-to-end dialog system platform. In 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 64-69.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443-2453.
Jinchao Li, Baolin Peng, Sungjin Lee, Jianfeng Gao, Ryuichi Takanobu, Qi Zhu, Minlie Huang, Hannes Schulz, Adam Atkinson, and Mahmoud Adada. 2020. Results of the multi-domain task-completion dialog challenge. In 34th AAAI Conference on Artificial Intelligence, Eighth Dialog System Technology Challenge Workshop.
Bing Liu and Ian Lane. 2017. Iterative policy learning in end-to-end trainable task-oriented neural dialog models. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop, pages 482-489.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. In 31st Annual Conference on Neural Information Processing Systems, pages 6379-6390.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533.
Alexandros Papangelis, Yi-Chia Wang, Piero Molino, and Gokhan Tur. 2019. Collaborative multi-agent dialogue model training via reinforcement learning. In 20th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 92-102.
Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. 2017. Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. In 2017 Conference on Empirical Methods in Natural Language Processing, pages 2231-2240.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. 2018. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In 35th International Conference on Machine Learning, pages 4292-4301.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In 2007 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 149-152.
Pararth Shah, Dilek Hakkani-Tür, Bing Liu, and Gokhan Tur. 2018. Bootstrapping a neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning. In 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 41-51.
Weiyan Shi, Kun Qian, Xuewei Wang, and Zhou Yu. 2019. How to build user simulators to train rl-based dialog systems. In 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, pages 1990-2000.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George Van Den Driessche, Thore Graepel, and Demis Hassabis. 2017. Mastering the game of go without human knowledge. Nature, 550(7676):354-359.
Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul
Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meeter. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339-373.
Pei-Hao Su, Milica Gašić, Nikola Mrkšić, Lina M Rojas Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. On-line active reward learning for policy optimisation in spoken dialogue systems. In 54th Annual Meeting of the Association for Computational Linguistics, pages 2431-2441.
Shang-Yu Su, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Yun-Nung Chen. 2018. Discriminative deep dyna-q: Robust planning for dialogue policy learning. In 2018 Conference on Empirical Methods in Natural Language Processing, pages 3813-3823.
Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. 2019. Guided dialog policy learning: Reward estimation for multi-domain task-oriented dialog. In 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, pages 100-110.
Harm Van Seijen, Mehdi Fatemi, Joshua Romoff, Romain Laroche, Tavian Barnes, and Jeffrey Tsang. 2017. Hybrid reward architecture for reinforcement learning. In 31st Annual Conference on Neural Information Processing Systems, pages 5392-5402.
Jason D Williams, Antoine Raux, and Matthew Henderson. 2016. The dialog state tracking challenge series: A review. Dialogue & Discourse, 7(3):4-33.
Zhirui Zhang, Xiujun Li, Jianfeng Gao, and Enhong Chen. 2019. Budgeted policy learning for task-oriented dialogue systems. In 57th Annual Meeting of the Association for Computational Linguistics, pages 3742-3751.
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. In 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1208-1218.
Both the system policy  $\pi$  and the user policy  $\mu$  are implemented with two hidden layer MLPs. The action space of system policy and user policy is 172 and 80 respectively. For Hybrid Value Network  $V$ , all neural network units  $f(\cdot)$  are two hidden layer MLPs. The activation function is all  $Relu$  for MLPs.
We use RMSprop as the optimization algorithm. The batch size is set to 32. The weighted pretraining factor  $\beta$  is 2.5, 4 for the system policy and user policy respectively. The learning rate for two policies is 1e-3 when pretraining. As for RL training, the learning rate is 1e-4, 5e-5 for the system policy and the user policy respectively, and 3e-5 for Hybrid Value Network. The discount factor  $\gamma$  is 0.99, and the target network is updated every  $C = 400$  training iterations.
In terms of reward design, the empty action penalty is set to -5, and penalties of other types are set to -1. The sub-goal completion reward is set to 5. The task success and the user goal reward are set to 20 if triggered, otherwise they are set to -5.
To illustrate the superiority of our model, we show a pair of sample dialog sessions between the user and the system, trained with RL/RL and MADPL, respectively. As shown in Table 6, the user goal contains two domains (i.e., train and restaurant). The user agent is supposed to inform the constraints displayed in Info, request information displayed in Reqt, and express the intent to book the train and restaurant with the information in Book. The system agent is required to answer the request, as well as book the train and the restaurant according to the user's demands. Table 7 shows the sampled dialog sessions. For RL/RL, the user sometimes talks about both the train and the restaurant simultaneously, making it difficult for the system to respond (highlighted in red color). Moreover, the system agent in RL/RL sometimes fails to answer the requested information or book the restaurant as the user demands (highlighted in blue color). As a result, the dialog in RL/RL fails as the user does not request the arrival time of the train, and the system does not book the restaurant for the user. In comparison, both the user and the system in MADPL fulfill their own task, and the whole dialog is successful. Table 8 shows the same sample dialog sessions with the original dialog acts.
Train	Restaurant
Info	Reqt	Book	Info	Reqt	Book
destination=peterborough day=thursday departure=cambridge leaveAt=19:45	duration arriveBy	people=3	food=euclidean pricerange=expensive	phone postcode address	time=19:30 people=5 day=tuesday
Table 6: User goal used in the case study.
RL/RL	MADPL
U: Hi, I'd like to take a train from cambridge to peterborough on thursday. I want to leave at about 19:45.
S: You may consider TR7324, which leaves at 21:40. Shall I book tickets for you?	U: Hi, I'd like to take a train from cambridge to peterborough on thursday. I want to leave at about 19:45.
S: You may consider TR7324, which leaves at 21:40, and arrives at cambridge at 24:23. Shall I book tickets for you?
U: Sounds good! Please book tickets to peterborough for 3 people.
S: I've booked your train tickets. The price is 10.10 pounds and your reference number is train-00002321. Is there anything else I can help you with today?	U: Sounds good! Please book tickets for 3 people.
S: I've booked your train tickets. The price is 10.10 pounds and your reference number is train-00002321. Is there anything else I can help you with today?
U: Could you please tell me the train ID and travel time? I also want to book an european restaurant with expensive price.
S: There are 4 choices for expensive european restaurant. What area of town would you prefer?	U: I also want to book an european restaurant with expensive price on Tuesday.
S: There are 4 choices found. City Stop Restaurant is a good choice.
U: I don't care its name, but make sure it's an expensive restaurant. Can you help me with a reservation for 5 people at 19:30 this coming Tuesday?
U: I don't care about the location. Could you please tell me the address, phone number and post of the restaurant?	S: Booking was successful. Reference number is: restaurant-00000083.
S: Sure, City Stop Restaurant is located at Cambridge City Football Club Milton Road Chesterton and the post is cb41uy. The phone number is 01223363270. Is there anything else I can help you with today?	U: Although I don't care its name, I'd like to know its phone number and post.
S: Sure, the phone number is 01223368786 and the post is cb23rh. Anything else?
U: Can you help me with a reservation for 5 people at 19:30 this coming Tuesday?	U: What's the address of the restaurant?
S: Is there anything else I can help you with?	S: The restaurant is located at Cambridge Lodge Hotel 139 Huntingdon Road City Centre. Is there anything else I can help you with?
U: Could you please tell me the travel time of the train I just booked?	U: Could you please tell me the travel time of the train I just booked? Thanks very much!
S: The travel time is 60 minutes. Anything else I can look up for you?	S: The travel time is 105 minutes. Anything else I can look up for you?
U: Thanks! When will the train arrive?	U: Thank you. That's all I needed.
S: You are very welcome. Goodbye.
S: You're welcome! Anything else? Bye!
U: Thank you very much!
S: You are very welcome. Goodbye.
Failure: User does not request the arrive time of the train, and the system does not book the restaurant for the user.	Success
Table 7: A sample dialog session comparison between RL/RL and MADPL in natural language.
Table 8: A sample dialog session comparison between RL/RL and MADPL in dialog acts.