SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {L} (u _ {t}, a) = \mathbb {E} _ {q _ {\phi} (z | u _ {t}, a)} [ p _ {\theta} (u _ {t} | z, u _ {t - 1}, u _ {t + 1}) ] \\ - \mathrm {K L} \left(q _ {\phi} (z | a, u _ {t}) \| p (z)\right) \tag {22} \\ \end{array}
$$
The objective to jointly train the prediction model and action embeddings is the same as Eqn. 11, where the terms for fully and partially labeled dialogues are replaced with the ones in Eqn. 22 and 21, respectively. Such expanding also enables a sufficient semi-supervised learning when expert demonstrations include all types of labeled dialogues:  $D_{\mathcal{F}}$ ,  $D_{\mathcal{P}}$  and  $D_{\mathcal{U}}$ . We notice that the posterior approximation  $q_{\phi}(z|u_t,a)$  and action embedding function  $f_{E}$  can be sharing between the process of state transitions and response generation. Thus, by treating semi-supervised learning in  $D_{\mathcal{F}}$  and  $D_{\mathcal{P}}$  as auxiliary constraints, the learning over unlabeled corpus can also benefit from dialogues state information.
To show the effectiveness of the proposed model (denoted as Act-VRNN), we experiment on a multi-domain dialogue environment under semi-supervised setting (Sec. 4.1). We compare against state-of-the-art approaches, and their variants enhanced by semi-supervised learning techniques (Sec. 4.2). We analyze the effectiveness of action learning and reward estimation of Act-VRNN under different supervision ratios (Sec. 4.3).
We use MultiWOZ (Budzianowski et al., 2018), a multi-domain human-human conversational dataset in our experiments. It contains in total 8438 dialogues spanning over seven domains, and each dialogue has 13.7 turns on average. MultiWOZ also contains a larger dialogue state and action space compared to former datasets such as movie-ticket booking dialogues (Li et al., 2017), and thus it is a much more challenging environment for policy learning. To use MultiWOZ for policy learning, a user simulator that initializes a user goal at the
beginning and interacts with dialogue policy is required. For a fair comparison, we adopt the same procedure as Takanobu et al. (2019) to train the user simulator based on auxiliary user action annotations provided by ConvLab (Lee et al., 2019).
To simulate semi-supervised policy learning, we remove system action and dialogue states annotations to obtain partially labeled and unlabeled expert demonstrations, respectively. Fully labeled expert demonstrations are randomly sampled from all training dialogues with different ratios (5%, 10%, and 15% in our experiments). Note that the absence of action or state annotations only applies for expert demonstrations, while interactions between policy and user simulator are in dialogue-act level as (Takanobu et al., 2019) and not affected by semi-supervised setting.
We use a three-layer transformer (Vaswani et al., 2017) with a hidden size of 128 and 4 heads as our base model for action embedding learning, i.e.,  $g(\cdot)$  in Eqn. 12. We use grid search to find the best hyperparameters for the models. We choose the action embedding dimensionality among {50, 75, 100, 150, 200}, the stochastic latent state size in VRNN among {16, 32, 64, 128, 256}, and the deterministic latent state size among {25, 50, 75, 100, 150}.
We use Entity-F1 and Success Rate to evaluate dialogue task completion. Entity-F1 computes the F1 score based on whether the requested information and indicated constraints from users are satisfied. Compared to inform rate and match rate used by Budzianowski et al. (2018), Entity-F1 considers both informed and requested entities at the same time and balances the recall and precision. Success rate indicates the ratio of successful dialogues, where a dialogue is regarded as successful only if all informed and requested entities are matched of the dialogue. We use Turns to evaluate the cost for task completion, where a lower number indicates the policy performs tasks more efficiently.
We compare Act-VRNN with three policy learning baselines: (1) PPO (Schulman et al., 2017) using hand-crafted rewards setting; (2) ALDM(Liu and Lane, 2018); (3) GDPL (Takanobu et al., 2019); We further consider using semi-supervised techniques to enhance the baselines under semi-supervised setting, and denote them as SS-PPO, SS-ALDM, and SS-GDPL. Specifically, we first train a prediction model based on semi-supervised VAE (Kingma et al., 2014), and use the predic
Table 2: Semi-Supervised Policy Learning Results  $\left( {{D}_{\mathcal{F}}\text{and}{D}_{\mathcal{P}}}\right)$
MODEL	DF(5%) + DP(95%)	DF(10%) + DP(90%)	DF(20%) + DP(80%)
Entity-F1	Success	Turns	Entity-F1	Success	Turns	Entity-F1	Success	Turns
Handcrafted	PPO	41.8	34.1	13.3	45.3	36.7	12.5	50.6	41.2	11.2
Reward	ALDM	38.7	35.6	15.2	42.1	38.6	14.9	44.9	42.1	13.7
Learning	GDPL	49.5	47.5	12.8	54.9	53.2	12.1	60.4	59.1	10.8
Semi-VAE	SS-PPO	45.2	36.2	13.6	47.4	37.2	12.4	53.1	43.6	11.5
Enhanced	SS-ALDM	39.6	38.8	14.7	44.7	43.8	13.2	47.8	51.3	12.4
SS-GDPL	53.7	51.2	11.1	61.3	58.4	10.5	66.5	68.7	9.2
Proposed	SS-VRNN	68.7	63.2	9.4	75.1	68.5	8.6	77.3	72.4	8.2
Act-GDPL	70.6	65.6	9.5	78.8	71.1	8.4	80.9	78.0	8.2
Act-VRNN	76.2	72.7	9.1	83.0	81.8	8.0	85.5	86.7	7.9
tion results as action annotations for expert demonstrations. We also compare the full model Act-VRNN with its two variants: (1) SS-VRNN uses a VRNN that consumes predicted action labels instead of action embeddings; (2) Act-GDPL feeds expert demonstrations enriched by action embeddings to the same reward function as GDPL
4.2 Overall Results
Table 2 shows that our proposed model consistently outperforms other models in the setting that uses fully and partially annotated dialogues  $(D_{\mathcal{F}}$  and  $D_{\mathcal{P}})$ . Act-VRNN improves task completion (measured by Entity-F1 and Success) while requiring less cost (measured by Turns). For example, Act-VRNN (81.8) outperforms SS-GDPL (60.4) by  $35.4\%$  under Success when having  $10\%$  fully annotated dialogues, and requires the fewest turns. Meanwhile, we find that both action learning and dynamics model are essential to the superiority of Act-VRNN. For example, Act-VRNN achieves  $19.8\%$  and  $11.2\%$  improvements over SS-VRNN and Act-GDPL, respectively, under Success when having  $20\%$  fully annotated dialogues. This validates that the learned action embeddings well capture similarities among actions, and VRNN is able to exploit such similarities for reward estimation.
We further find that the improvements brought by semi-VAE enhancement is limited for baselines, especially when the ratio of fully annotated dialogues is low. For example, SS-PPO and SS-GDPL achieve  $6\%$  and  $7\%$  improvements over their counterparts under Success when having  $5\%$  fully annotated dialogues. Similar results are also observed for pseudo-label approach. In general, the pseudo
label methods are outperformed by the counterparts of Semi-VAE and are even worse than the baselines without enhancement when the ratio of fully annotated dialogues is low. For example, in setting  $D_{\mathcal{F}} + D_{\mathcal{P}}$ , pseudo-label enhanced PPO performs worse than PPO under Entity-F1 when the ratio of fully annotated dialogues is  $5\%$  and  $10\%$  (37.2 vs 41.8, 39.2 vs 45.3), and only achieves slightly gain when the ratio is  $20\%$  (51.0 vs 50.6). This is largely because the prediction accuracy of Semi-VAE and pseudo-label approach might be low with a small amount of fully annotated dialogues, and the expert dialogues with mispredicted actions impinge reward function learning of baselines. Act-VRNN overcomes this challenge with the generalization ability brought by modeling dialogue progress in an action embedding space for reward estimation.
The results for policy learning using unlabeled dialogues  $(D_{\mathcal{U}})$  are shown on Table 3. We consider two settings: (1) having fully labeled and unlabeled dialogues, i.e.,  $D_{\mathcal{F}} + D_{\mathcal{U}}$ ; (2) having all three types of dialogues, i.e.,  $D_{\mathcal{F}} + D_{\mathcal{P}} + D_{\mathcal{U}}$ . We can see that Act-VRNN significantly outperforms the baselines in both settings. For example, in setting  $D_{\mathcal{F}} + D_{\mathcal{U}}$ , Act-VRNN outperforms SS-GDPL by  $43\%$  and  $44\%$  under Entity-F1 and Success, respectively. Similar results are also observed in setting  $D_{\mathcal{F}} + D_{\mathcal{P}} + D_{\mathcal{U}}$ . We further find that SS-VRNN outperforms Act-GDPL in these two settings while the results are opposite in setting  $D_{\mathcal{F}} + D_{\mathcal{P}}$ , and we will conduct a detailed discussion in the following section. By comparing results of Act-VRNN and baselines in these two settings, we can see that Act-VRNN can better exploit the additional partially labeled dialogues. For example, SS-GDPL only achieves  $2.3\%$  under Success while Act-VRNN achieves more than  $5\%$ .
<sup>1</sup>We also experimented with the pseudo-label approach (Lee, 2013), and the empirical results were worse than Semi-VAE. Thus, we only report the Semi-VAE enhancement results in the table for simplicity.
Table 3: Semi-Supervised Policy Learning Results  $(D_{\mathcal{F}},D_{\mathcal{P}},$  and  $D_{\mathcal{U}})$
SUPERVISION	MODEL	Entity-F1	Success	Turns
DF(10%) +
DU(90%)	ALDM	40.0	34.9	15.9
SS-PPO	44.7	33.8	12.9
SS-ALDM	42.1	36.4	14.9
SS-GDPL	56.3	50.2	11.8
SS-VRNN	74.1	67.1	9.1
Act-GDPL	72.9	66.7	8.5
Act-VRNN	80.6	72.4	8.4
DF(10%) +
DP(10%) +
DU(80%)	ALDM	41.7	35.2	15.7
SS-PPO	44.9	34.6	12.8
SS-ALDM	42.5	40.1	14.7
SS-GDPL	57.1	51.4	10.7
SS-VRNN	75.6	67.9	8.8
Act-GDPL	73.3	67.1	8.5
Act-VRNN	81.1	76.3	8.2
* Note that PPO and GDPL achieve the same results as  $D_{\mathcal{F}}(10\%) + D_{\mathcal{P}}(90\%)$  in Table 2 since they can only utilize dialogues in  $D_{\mathcal{F}}$
4.3 Discussions
We first study the effects of action learning module in Act-VRNN. We compare Act-VRNN with SS-VRNN, and their counterparts that do not use state transition based objective in semi-supervised learning (i.e., optimizing Eqn. 3 instead of Eqn. 7). These two variants are denoted as Act-VRNN (no state) and SS-VRNN (no state). For a thorough investigation, under each setting, we further show the performances under dialogues spanning over different number of domains. Dialogues spanning over more domains are considered more difficult. The results under two supervision ratio setting are shown in Fig. 2(a) and Fig. 2(b). We can see that Act-VRNN outperforms other variants in each configuration, especially in the dialogues that include more than one domains. This is largely because the learned action embeddings effectively discover the similarities between actions across domains, and thus lead to better generalization of reward estimation. We further find that the state transition based objective we formulated fits well with the VRNN based reward estimator. Both Act-VRNN and SS-VRNN optimized considering state transitions achieve performance gains.
Last, we study the effects of dynamics model based reward function in Act-VRNN. We consider four different models as reward function: (1) our full dynamics model VRNN; (2) a dynamics model having only deterministic states (Eqn. 17); (3) a dynamics model having only stochastic states (Eqn. 15); (4) GDPL. All four models are learned based
(a)  $D_{\mathcal{F}}(5\%) + D_{\mathcal{P}}(95\%)$
(b)  $D_{\mathcal{F}}(20\%) + D_{\mathcal{P}}(80\%)$
Figure 2: Effects of action learning  $(D_{\mathcal{F}}$  and  $D_{\mathcal{P}})$
(a)  $D_{\mathcal{F}} + D_{\mathcal{P}}$
Figure 3: Effects of dynamics model
(b)  $D_{\mathcal{F}} + D_{\mathcal{U}}$
on action embedding learned in the action learning module. The results under  $D_{\mathcal{F}} + D_{\mathcal{P}}$  and  $D_{\mathcal{F}} + D_{\mathcal{U}}$  are shown in Fig. 3(a) and Fig. 3(b), respectively. We can see that both stochastic and deterministic states in VRNN are important, since VRNN outperforms its two variants and GDPL in each configuration. We further find that the contribution of stochastic and deterministic states may vary in different settings. For example, VRNN (stochastic only) consistently outperforms VRNN (deterministic only) in  $D_{\mathcal{F}} + D_{\mathcal{U}}$  while opposite results are observed in  $D_{\mathcal{F}} + D_{\mathcal{P}}$  when ratio of  $D_{\mathcal{F}}$  is over  $20\%$ . This is largely because modeling dialogue progress using stochastic states can provide more stable with less supervision signals, while the incorporation of deterministic can lead to more precise estimation can when more information of expert demonstrations are available.
Reward learning aims to provide more effective and sufficient supervision signals for dialogue policy. Early studies focus on learning reward function utilizing external evaluations, e.g., user experience feedbacks (Gašić et al., 2013), objective ratings (Su et al., 2015; Ultes et al., 2017), or a combination of multiple evaluations (Su et al., 2016; Chen et al., 2019). These approaches often assume a human-in-the-loop setting where interactions with real users are available during training, which is
expensive and difficult to scale. As more large-scale high-quality dialogue corpus become available (e.g., MultiWOZ (Budzianowski et al., 2018)), recent years have seen a growing interest in learning reward function from expert demonstrations. Most recent approaches apply inverse reinforcement learning techniques for dialogue policy learning (Takanobu et al., 2019; Li et al., 2019b). These all require a complete state-action annotation for expert demonstrations. We aim to overcome this limitation in this study.
Semi-supervised learning aims to utilize unlabeled data to boost model performance, and is studied in computer vision (Iscen et al., 2019), item ranking (Park and Chang, 2019; Huang et al., 2019b), and multi-label classification (Miyato et al., 2015; Wang et al., 2018, 2019b). Many studies apply semi-supervised VAE (Kingma et al., 2014) for different classification tasks, e.g., sentiment analysis (Xu et al., 2017; Li et al., 2019a), text matching (Shen et al., 2018; Choi et al., 2019). While these work focus on prediction accuracies, we aim to enrich expert demonstrations via semi-supervised learning.
We study the problem of semi-supervised policy learning and propose Act-VRNN to provide more effective and stable rewards estimations. We formulate a generative model to jointly infer action labels and learn action embeddings. We design a novel reward function to first model dialogue progress, and estimate action rewards by determining whether the action leads to similar progress as expert dialogues. The experimental results confirm that Act-VRNN achieves better task completion compared with the state-of-the-art in two settings that consider partially labeled or unlabeled dialogues. For future work, we will explore the scenarios that annotations are absent for all expert dialogues.
We would like to thank Xiaojie Wang for his help. This work is supported by Australian Research Council (ARC) Discovery Project DP180102050, and China Scholarship Council (CSC).
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ra
madan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Conference on Empirical Methods in Natural Language Processing, pages 5016-5026.
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas. 2019. Learning action representations for reinforcement learning. In International Conference on Machine Learning, pages 941-950.
Runyu Chen, Qili Wang, and Wen I Xu. 2019. Mining user requirements to facilitate mobile app quality upgrades with big data. *Electron. Commer. Res. Appl.*, 38.
Jihun Choi, Taeuk Kim, and Sang-goo Lee. 2019. A cross-sentence latent variable model for semi-supervised text sequence matching. In Annual Meeting of the Association for Computational Linguistics, pages 4747-4761, Florence, Italy. Association for Computational Linguistics.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. 2015. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems, pages 2980-2988.
Justin Fu, Katie Luo, and Sergey Levine. 2017. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248.
Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural approaches to conversational ai. arXiv preprint arXiv:1809.08267.
Milica Gasic, Catherine Breslin, Matthew Henderson, Dongho Kim, Martin Szummer, Blaise Thomson, Pirros Tsiakoulis, and Steve Young. 2013. On-line policy optimisation of bayesian spoken dialogue systems via human interaction. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8367-8371.
Xinting Huang, Jianzhong Qi, Yu Sun, and Rui Zhang. 2019a. Mala: Cross-domain dialogue generation with action learning. arXiv preprint arXiv:1912.08442.
Xinting Huang, Jianzhong Qi, Yu Sun, Rui Zhang, and Hai-Tao Zheng. 2019b. Carl: Aggregated search with context-aware module embedding learning. In IJCNN, pages 101-108.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondej Chum. 2019. Label propagation for deep semi-supervised learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5065-5074.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, pages 3581-3589.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, pages 3294-3302.
Dong-Hyun Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, volume 3, page 2.
Sungjin Lee, Qi Zhu, Ryuichi Takanobu, Xiang Li, Yaoqin Zhang, Zheng Zhang, Jinchao Li, Baolin Peng, Xijun Li, Minlie Huang, and Jianfeng Gao. 2019. Convlab: Multi-domain end-to-end dialog system platform. In Annual Meeting of the Association for Computational Linguistics, page 6469.
Xuijun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017. End-to-end task-completion neural dialogue systems. In International Joint Conference on Natural Language Processing, pages 187-196.
Yitong Li, Timothy Baldwin, and Trevor Cohn. 2019a. Semi-supervised stochastic multi-domain learning using variational inference. In Annual Meeting of the Association for Computational Linguistics, pages 1923-1934, Florence, Italy. Association for Computational Linguistics.
Ziming Li, Julia Kiseleva, and Maarten de Rijke. 2019b. Dialogue generation: From imitation learning to inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, pages 6722-6729.
Bing Liu and Ian Lane. 2018. Adversarial learning of task-oriented neural dialog models. In Annual SIG- dial Meeting on Discourse and Dialogue, pages 350- 359.
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, and Honglak Lee. 2019. Unsupervised learning of object structure and dynamics from videos. arXiv preprint arXiv:1906.07889.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. 2015. Distributional smoothing with virtual adversarial training. arXiv preprint arXiv:1507.00677.
Dae Hoon Park and Yi Chang. 2019. Adversarial sampling and training for semi-supervised information retrieval. ArXiv, abs/1811.04155.
Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kam-Fai Wong. 2018. Deep dyna-q: Integrating planning for task-completion dialogue policy learning. In Annual Meeting of the Association for Computational Linguistics, pages 2182-2192.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
Dinghan Shen, Yizhe Zhang, Ricardo Henao, Qinliang Su, and Lawrence Carin. 2018. Deconvolutional latent-variable model for text sequence matching. In Thirty-Second AAAI Conference on Artificial Intelligence, pages 5438-5445.
Pei-Hao Su, Milica Gasic, Nikola Mrkšić, Lina M Rojas Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. On-line active reward learning for policy optimisation in spoken dialogue systems. In Annual Meeting of the Association for Computational Linguistics, pages 2431-2441.
Pei-Hao Su, David Vandyke, Milica Gasic, Nikola Mrkusic, Tsung-Hsien Wen, and Steve Young. 2015. Reward shaping with recurrent neural networks for speeding up on-line policy learning in spoken dialogue systems. In Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 417-421.
Chen Sun, Per Karlsson, Jiajun Wu, Joshua B Tenenbaum, and Kevin Murphy. 2019. Stochastic prediction of multi-agent interactions from partial observations. arXiv preprint arXiv:1902.09641.
Yu Sun, Nicholas Jing Yuan, Yingzi Wang, Xing Xie, Kieran McDonald, and Rui Zhang. 2016. Contextual intent tracking for personal assistants. In SIGKDD, pages 273-282. ACM.
Yu Sun, Nicholas Jing Yuan, Xing Xie, Kieran McDonald, and Rui Zhang. 2017. Collaborative intent prediction with real-time contextual data. ACM Transactions on Information Systems (TOIS), 35(4):1-33.
Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. 2019. Guided dialog policy learning: Reward estimation for multi-domain task-oriented dialog. In Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 100-110.
Stefan Ultes, Paweł Budzianowski, Iñigo Casanueva, Nikola Mrkšić, Lina M Rojas Barahona, Pei-Hao Su, Tsung-Hsien Wen, Milica Gasic, and Steve Young. 2017. Reward-balancing for statistical spoken dialogue systems using multi-objective reinforcement learning. In 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 65-70.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998-6008.
Qili Wang, Wei Xu, Xinting Huang, and Kunlin Yang. 2019a. Enhancing intraday stock price manipulation detection by leveraging recurrent neural networks with ensemble learning. Neurocomputing, 347:46-58.
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. 2018. Kdgan: knowledge distillation with generative adversarial networks. In Advances in Neural Information Processing Systems, pages 775-786.
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. 2019b. Adversarial distillation for learning with privileged provisions. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-12.
Weidi Xu, Haoze Sun, Chao Deng, and Ying Tan. 2017. Variational autoencoder for semi-supervised text classification. In AAAI Conference on Artificial Intelligence, pages 3358-3364.
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. 2018. Unsupervised text style transfer using language models as discriminators. In Advances in Neural Information Processing Systems, pages 7287-7298.
Steve Young, Milica Gasic, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160-1179.
Zhirui Zhang, Xiujun Li, Jianfeng Gao, and Enhong Chen. 2019. Budgeted policy learning for task-oriented dialogue systems. arXiv preprint arXiv:1906.00499.
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1208-1218.