SlowGuess/NAACL_2021_OCR huggingface dataset:
From Fig. 3 (a),  $\mathrm{DQN}_{SeqPrd}$  achieves the best performance with a success rate of 0.990 and converges after  $130\mathrm{K}$ , which speeds up the training process by almost  $300\%$  compared to  $\mathrm{DQN}_{vanilla}$ . Compared with  $\mathrm{DQN}_{vanilla}$ , the methods using pre-trained reward functions  $R_{offgan}$ ,  $R_{SeqArg}$ ,  $R_{SeqPrd}$  are better than vanilla in terms of both convergence speed and success rate. This phenomenon suggests that these three reward estimators could speed up dialog policy training.
Different from  $\mathrm{DQN}_{offgan}$ , whose reward function is also learned by adversarial training, we further apply disentangled learning and multi-view discriminator to obtain fine-grained rewards. The performance of  $\mathrm{DQN}_{SeqPrd}$  and  $\mathrm{WDQN}_{SeqPrd}$  gains received in convergence speed and final performance of our methods confirm the superiority of the hierarchical reward.
For WDQN agent, since first warmed up with human dialogs, the WDQN-based methods share a similar success rate (around  $6\%$ ) before training and consistently converge faster than DQN-based models. However, the usage of warm-up operation will mislead the model to local optimum and deteriorate the final success rate. This phenomenon can be found in the last 100 frames, the performances of WDQN_vanilla and WDQN_offgan drop significantly. Another attractive property of our method, compared with WDQN_vanilla and WDQN_offgan, is the variance of success rate is obviously small, which strongly supports the remarkable benefit of exploiting disentangled representation to learn prof
(a) DQN agents results
(b) WDQN agents results
(c) PPO agents results
Figure 3: The training process of dialog agents based on different reinforcement learning algorithms.
Model	SRate	RScore	ATurn
DQNvanilla	0.843	55.18	8.02
DQNoffgan	0.964	71.87	5.86
DQNSeqAvg	0.981	74.21	5.57
DQNSeqPrd	0.990	75.43	5.40
WDQNvanilla	0.678	32.53	10.84
WDQNoffgan	0.760	43.88	9.33
WDQNSeqAvg	0.798	40.01	8.79
WDQNSeqPrd	0.960	71.05	6.09
AIRL	0.793	50.63	10.14
GAIL	0.832	51.65	10.01
PPOvanilla	0.861	56.93	8.30
PPOoffgan	0.860	56.96	8.20
PPOSeqPrd	0.806	49.86	9.34
PPOSeqAvg	0.879	59.04	8.07
itable sequential reward in dialog management.
4.4 Results with PPO-based Agents
For the policy gradient-based agents, we compare our models with two other strong baselines, i.e., GAIL and AIRL, whose reward functions are updated during RL training. Similar to DQN-based methods, we employ PPO algorithms to train dialog agents with different reward functions. Before training a PPO agent, we perform imitation learning with human dialogs to warm-up PPO agents, achieving around  $33\%$  success rate. For fair comparisons, we also pretrain the discriminator in GAIL and reward model in AIRL by feeding positive samples and negative samples from pretrain process of dialog agents.
As demonstrated in Fig. 3 (c), although AIRL rises faster than others during the first 50 frames, it converges to a worse result, compared with  $\mathrm{PPO}_{SeqAvg}$ . An interesting observation is that  $\mathrm{PPO}_{vanilla}$  even performs better than AIRL. This
Table 1: The performance of dialog agents with different reward functions. SRate, Rscore and ATurn represent success rate, reward score and average turn.
Model	Acc	Prec	Rec	F1	JS
Roffgan	0.79	0.84	0.76	0.80	1.39
Rd	0.86	0.97	0.80	0.88	0.69
Ra	0.71	0.91	0.65	0.76	0.14
Rs	0.77	0.95	0.69	0.80	0.33
RSeqAvg	0.87	0.91	0.85	0.88	1.00
RSeqPrd	0.87	0.87	0.87	0.87	3.73
Table 2: The accuracy, precision, recall, F1 and JS divergence scores on test dataset with equal number of positive and negative samples.
may be due to the fact that adversarial learning is extremely unstable in RL. Therefore, we aim to learn an off-line reward function to guide the evolution of agents, as we motivate in the introduction. In the comparison between  $\mathrm{PPO}_{\text{offgan}}$  and  $\mathrm{PPO}_{\text{SeqAvg}}$ , the performance gains obtained by our model verifies the advantage of exploiting multi-level reward signals. Moreover, it can be seen that, in the PPO-based RL algorithm, the performance of the agent with the reward function  $R_{\text{SeqPrd}}$  is worse than that of  $R_{\text{SeqAvg}}$ , but the opposite is true in the DQN and WDQN-based methods. This may be caused by that the multiplicative reward (i.e.,  $R_{\text{SeqPrd}}$ ) may cause the gradient to be very steep, which makes the training of the policy gradient-based model unstable. However, in the value-based RL method, an average reward (i.e.,  $R_{\text{SeqAvg}}$ ) might degenerate the performance, as a hierarchical reward is more general and intuitive, which has access to precise intermediate reward signals. The performances of the last frame in terms of success rate, reward score and average turn are shown in Table 1, in which we could claim again that our method  $\mathrm{PPO}_{\text{SeqAvg}}$  outperforms all baseline models by a substantial margin.
(a)  $R_{SeqAvg}$
(b)  $R_{SeqPrd}$
Figure 4: Histogram of distribution about fake/real state-action pairs in different rewards setting. Horizontal axis means the reward score, and vertical axis means frequency.
(c)  $R_{offgan}$
4.5 Analysis of Different Rewards
To visualize the model performance and what benefits a sequential reward will bring, we view the evaluation as a binary classification and distribution distance problem. We use accuracy, precision, recall, and F1 to find out how good this binary model is, and use JS divergence to evaluate the ability of the reward model to divide positive and negative distributions, the larger the better. We construct a test dataset with equal numbers (7, 372) of positive and negative samples from the test dataset. All positive samples are original state-action pairs. For negative samples, we fix states and randomly pick actions from those with different domains. We evaluate three reward models separately in Table 2.  $R_{d}$  is the best one among the three with the highest five scores. This is pretty straightforward since domain is the first identity to divide action space into groups. And for  $R_{a}$  and  $R_{s}$ , the JS divergence is lower, this is because some actions could have different domains with the same action-slot. For example, action "Train-Inform-Arrive" and "Hotel-Inform-Arrive" have the same action-slot with different domains. Thus,  $R_{a}$  and  $R_{s}$  will only give an ambiguous decision boundary. But from a sequential view, we make a new combination of  $R_{SeqAvg}$  and  $R_{SeqPred}$ , which gives good results.
$R_{offgan}$  could give the right rewards to some extent, but from Fig. 4 (c), there is a large intersection between fake and real distributions among three, which means it wrongly classifies fake action as right. And this is the reason why its F1 score is lower. Besides, this reward model is a biased model, its ratio of true negative and true positive samples is 0.89 thus it tends to give fake results. For both of our model, there is little bias,  $R_{SeqPrd}$  is 0.99 and  $R_{SeqAvg}$  is 0.98, which benefits from
sequential combination.
For  $R_{SeqPrd}$ ,  $R_{SeqAvg}$  and  $R_{offgan}$ ,  $R_{SeqPrd}$  perform the best, no matter from the view of binary classification or JS divergence. And the distribution is much sharper than  $R_{SeqAvg}$  with prediction score centering at 0 or 1. For  $R_{SeqAvg}$ , the distribution is softer than  $R_{SeqPrd}$  as shown in Fig. 4. Although there is no exact evaluation to say how bad one action is, from the good results of  $\mathrm{PPO}_{SeqAvg}$ , nearly the same binary classification score with  $R_{SeqPrd}$  as well as lower JS divergence, we could get the conclusion that it is the most accurate rewards among the three.
We propose a multi-level and sequential reward modeling mechanism that models expert state-action pairs in terms of domain, act, and slot. Our approach combines a disentangled auto-encoder and a generator-discriminator framework to model the distribution of expert state-action pairs. The learned discriminators can thereby serve as a multilevel reward estimator. Experimental results show that our three-level modeling mechanism gives more accurate and explainable reward estimations and significantly boosts the performance of a variety of RL-based dialog agents, as well as accelerating the convergence speed of training.
This work was supported by the Tencent Jarvis Lab. We thank Ziming Li, Zhanpeng Huang for the helpful discussions and insightful comments. Thank Kallirroi for leading me into the field of dialogue systems.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.00278.
Inigo Casanueva, Paweł Budzianowski, Pei-Hao Su, Stefan Ultes, Lina Rojas-Barahona, Bo-Hsiang Tseng, and Milica Gašić. 2018. Feudal reinforcement learning for dialogue management in large domains. arXiv preprint arXiv:1803.03232.
Lu Chen, Zhi Chen, Bowen Tan, Sishan Long, Milica Gašić, and Kai Yu. 2019a. Agentgraph: Toward universal dialogue management with structured deep reinforcement learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(9):1378-1391.
Meng Chen, Ruixue Liu, Lei Shen, Shaozu Yuan, Jingyan Zhou, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2019b. The jddc corpus: A large-scale multi-turn chinese dialogue dataset for e-commerce customer service. arXiv preprint arXiv:1911.09969.
Ting Fang, Tingting Qiao, and Duanqing Xu. 2019. Hagan: Hierarchical attentive adversarial learning for task-oriented dialogue system. In International Conference on Neural Information Processing, pages 98-109. Springer.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680.
Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. In Advances in neural information processing systems, pages 4565-4573.
Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114.
Sungjin Lee, Qi Zhu, Ryuichi Takanobu, Xiang Li, Yaoqin Zhang, Zheng Zhang, Jinchao Li, Baolin Peng, Xiujun Li, Minlie Huang, et al. 2019. Convlab: Multi-domain end-to-end dialog system platform. arXiv preprint arXiv:1904.08637.
Ziming Li, Sungjin Lee, Baolin Peng, Jinchao Li, Shahin Shayandeh, and Jianfeng Gao. 2020. Guided dialog policy learning without adversarial learning in the loop. arXiv preprint arXiv:2004.03267.
Bing Liu and Ian Lane. 2018. Adversarial learning of task-oriented neural dialog models. arXiv preprint arXiv:1805.11762.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature, 518(7540):529-533.
Tim Paek and Roberto Pieraccini. 2008. Automating spoken dialogue management design using machine learning: An industry perspective. Speech communication, 50(8-9):716-729.
Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. 2017. Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. arXiv preprint arXiv:1704.03084.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 149-152.
Pei-Hao Su, David Vandyke, Milica Gasic, Nikola Mrkusic, Tsung-Hsien Wen, and Steve Young. 2015. Reward shaping with recurrent neural networks for speeding up on-line policy learning in spoken dialogue systems. arXiv preprint arXiv:1508.03391.
Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. 2019. Guided dialog policy learning: Reward estimation for multi-domain task-oriented dialog. arXiv preprint arXiv:1908.10719.
Dirk Väth Ngoc Thang Vu. 2019. To combine or not to combine? a rainbow deep reinforcement learning agent for dialog policy. In 20th Annual Meeting of the Special Interest Group on Discourse and Dialogue, page 62.
Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuan-Jing Huang, Kam-Fai Wong, and Xiang Dai. 2018. Task-oriented dialogue system for automatic diagnosis. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201-207.
Zheng Zhang, Ryuichi Takanobu, Qi Zhu, MinLie Huang, and XiaoYan Zhu. 2020. Recent advances and challenges in task-oriented dialog systems. Science China Technological Sciences, pages 1-17.