SlowGuess/NAACL_2022_OCR huggingface dataset:
Model	Joint Accuracy
MinTL-T5-small (Lin et al., 2020)	51.2
SUMBT+LaRL (Lee et al., 2020)	51.5
MinTL-BART (Lin et al., 2020)	52.1
UBAR (Yang et al., 2021)	52.6
SOLOIST (Peng et al., 2020)	53.2
BORT	54.0
effectiveness of BORT. Figure 3 shows an example generated by MinTL and BORT, respectively. More examples are provided in Appendix A.5. MinTL generates the response to request for the preferred area about college since it generates an inaccurate dialog state 'attraction_type=college' rather than correct dialog state 'attraction_name=jesus college'. In contrast, BORT generates an accurate dialog state, achieving the appropriate response that provides the information of jesus college. These further demonstrate the effectiveness of our proposed reconstruction strategies.
Figure 3: An example of the task-oriented dialog systems in dialog session PMUL4025.
For human evaluation, we manually evaluate the quality of generated responses on 50 dialog sessions, which are randomly extracted from the MultiWOZ 2.0 testing set. We consider the fluency and appropriateness of the generated response based on scores ranging from 1 to 5. The fluency metric measures whether the generated response is fluent. The appropriateness metric measures whether the generated response is appropriate and the system understands the user's goal. Three fluent English speakers are asked to evaluate these generated responses. The average scores evaluated by them are shown in Table 6. The results are consistent with the automatic evaluation, indicating that BORT could improve the quality of generated response.
Table 5: The dialog state tracking performance of end-to-end task-oriented dialog systems on MultiWOZ 2.0.
Model	Fluency	Appropriateness
MinTL-T5-small	4.50	3.88
UBAR	4.50	3.81
BORT	4.55	3.98
4.9 Domain Adaptation Analysis
To investigate the domain adaptation ability of BORT to generalize to some unseen domains, we simulate zero-shot experiments by excluding one domain and training BORT on other domains. As shown in Table 7, the train and taxi domains achieve the highest combined scores because they have a high overlap in ontology with other domains. In addition, BORT and MinTL with an encoder-decoder-based pre-trained model achieve significantly better domain adaptation performance than DAMD without a pre-trained model and UBAR with a decoder-based pre-trained model, which demonstrates the encoder-decoder based pretrained model have better domain transferability. Moreover, our proposed reconstruction strategy could further improve combined scores in the zero-shot domain scenario.
Table 6: The human evaluation of the end-to-end task-oriented dialog systems on MultiWOZ 2.0.
Model	Attraction	Hotel	Restaurant	Taxi	Train
DAMD	28.7	26.9	24.4	52.3	51.4
UBAR	28.3	29.5	23.5	59.5	53.9
MinTL	33.4	37.3	31.5	60.4	77.1
BORT	33.6	38.7	32.0	62.7	85.6
Table 7: Comparison of combined scores on the MultiWOZ 2.0 in the zero-shot domain scenario.
4.10 Low Resource Scenario Analysis
To better assess the robustness of our proposed BORT, we choose  $5\%$ ,  $10\%$ ,  $20\%$ , and  $30\%$  of training dialog sessions to investigate the performance of task-oriented dialog systems in the low resource scenario. As shown in Table 8, BORT substantially outperforms other methods in these low-resource scenarios. This is because the error propagation problem in the low resource scenario is more serious, while BORT could effectively alleviate the error propagation problem. Moreover, our proposed BORT trained on the  $30\%$  dataset performs comparably to some baseline systems trained on all datasets as shown in Table 1. These further demonstrate that our proposed BORT is robust, alleviating poor performance in the low-resource
Model	5%	10%	20%	30%
Inform	Success	BLEU	Combined	Inform	Success	BLEU	Combined	Inform	Success	BLEU	Combined	Inform	Success	BLEU	Combined
DAMD	49.1	23.7	11.3	47.7	57.6	32.6	12.0	57.1	64.7	45.0	15.3	70.2	64.5	47.3	15.5	71.4
UBAR	35.7	21.2	11.0	39.5	62.4	43.6	12.7	65.7	76.2	58.3	14.1	81.4	81.2	65.4	14.7	88.0
MinTL	55.2	40.9	13.9	62.0	67.7	55.7	15.3	77.0	66.7	57.9	17.3	79.6	74.9	66.5	17.3	88.0
BORT	69.8	45.9	11.0	68.9	74.5	60.6	15.5	83.1	82.1	65.5	14.3	88.1	83.8	69.9	17.2	94.1
Table 8: Comparison of task-oriented dialog systems on the MultiWOZ 2.0 in the low resource scenarios.
scenario.
4.11 Error Propagation Analysis
To investigate the denoising capability of our proposed BORT, we perform the simulated experiments, where noise is added in the oracle dialog state for the policy optimization evaluation. In detail, we replace every token in the oracle dialog state with the masked token with a probability to generate synthetic noise. As shown in Figure 4, BORT performs substantially better than the baseline system in the noisy scenario. In particular, as the noise ratio in the oracle dialog state increases, the performance gap between the baseline system and BORT increases. When noise proportion is 0, BORT still performs better than the baseline system because BORT generates more appropriate response via the denoising reconstruction strategy. These demonstrate that our proposed BORT is robust and effective.
Figure 4: The policy optimization performance (combined score) of baseline and BORT as the noise in the oracle dialog state increases on the MultiWOZ 2.0.
End-to-end task-oriented dialog system has attracted much attention in the dialog community. A two-stage copynet framework was proposed to establish an end-to-end task-oriented dialog system based on a single sequence-to-sequence model (Lei et al., 2018). Zhang et al. (2020b) proposed a multi-action data augmentation framework to improve the diversity of dialog responses. Recently, large-scale language model pre-training has been effective for enhancing many natural language process
ing tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019). Decoder-based pre-trained language model such as GPT-2 (Radford et al., 2019) was used to improve the performance of end-to-end task-oriented dialog system (Budzianowski and Vulic, 2019; Hosseini-Asl et al., 2020; Peng et al., 2020; Yang et al., 2021). The Levenshtein dialog state instead of the dialog state was generated to reduce the inference latency (Lin et al., 2020). In addition, they used encoder-decoder-based pre-trained model such as T5 (Raffel et al., 2020) and BART (Lewis et al., 2020) to establish a dialog system. In contrast with previous work, in which system response was generated, Wu et al. (2020) used encoder-based pre-trained model such as BERT (Devlin et al., 2019) for task-oriented dialog system, aiming to retrieve the most relative system response from a candidate pool. Reinforcement learning could also be used to enable task-oriented dialog systems to achieve more successful task completion (Lubis et al., 2020; Lee et al., 2020).
Tu et al. (2017) proposed an encoder-decoder-reconstructor framework for neural machine translation to alleviate over-translation and under-translation problems. Reconstruction strategy was used to moderate dropped pronoun translation problems (Wang et al., 2018). In contrast, we considered the adequacy of semantic representations rather than natural language sentences to build the reconstruction model. Vincent et al. (2010) proposed a denoising autoencoder, in which random noise is added to enhance the robustness of the model, alleviating the overfitting problem of traditional auto-encoder. The denoising auto-encoder strategy was used as the language model to generate more fluent translation candidates for the unsupervised neural machine translation (Artetxe et al., 2018; Lample et al., 2018; Sun et al., 2019). In addition, a denoising auto-encoder was used to pretrain sequence-to-sequence models on the large scale corpus (Lewis et al., 2020; Liu et al., 2020). In contrast, we proposed a denoising reconstruction mechanism to alleviate the error propagation problem along the multi-turn conversation flow.
This paper proposes back and denoising reconstruction strategies for the end-to-end task-oriented dialog system. Back reconstruction strategy is proposed to mitigate the generation of inaccurate dialog states, achieving better task completion of the task-oriented dialog system. Denoising reconstruction is used to train a robust task-oriented dialog system, further alleviating the error propagation problem. Our extensive experiments and analysis on MultiWOZ 2.0 and CamRest676 demonstrate the effectiveness of our proposed reconstruction strategies.
We are grateful to the anonymous reviewers and the area chair for their insightful comments and suggestions. This work is supported by the National Key Research and Development Program of China under Grant No. 2020AAA0108600.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In Proceedings of the 6th International Conference on Learning Representations. OpenReview.net.
Paweł Budzianowski and Ivan Vulić. 2019. Hello, it's GPT-2 - how can I help you? towards the use of pretrained language models for task-oriented dialogue systems. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 15-22. Association for Computational Linguistics.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026. Association for Computational Linguistics.
Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, and William Yang Wang. 2019. Semantically conditioned dialog response generation via hierarchical disentangled self-attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3696–3709. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In Advances in Neural Information Processing Systems 33, pages 20179–20191. Curran Associates, Inc.
Hyunmin Jeon and Gary Geunbae Lee. 2021. Domain state tracking for a simplified dialogue system. CoRR, abs/2103.06648.
Jonás Kulhánek, Vojtech Hudecek, Tomás Nekvinda, and Ondrej Dusek. 2021. Augpt: Dialogue with pre-trained language models and data augmentation. CoRR, abs/2102.05126.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039-5049. Association for Computational Linguistics.
Hwaran Lee, Seokhwan Jo, HyungJun Kim, Sangkeun Jung, and Tae-Yoon Kim. 2020. Sumbt+larl: End-to-end neural task-oriented dialog system with reinforcement learning. CoRR, abs/2009.10447.
Hwaran Lee, Jinsik Lee, and Tae-Yoon Kim. 2019. SUMBT: Slot-utterance matching for universal and scalable belief tracking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5478-5483. Association for Computational Linguistics.
Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1437-1447. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880. Association for Computational Linguistics.
Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. 2020. MinTL: Minimalist transfer learning for task-oriented dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3391-3405. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of the 7th International Conference on Learning Representations. OpenReview.net.
Nurul Lubis, Christian Geishauser, Michael Heck, Hsien-chin Lin, Marco Moresi, Carel van Niekerk, and Milica Gasic. 2020. LAVA: Latent action spaces via variational auto-encoding for dialogue policy optimization. In Proceedings of the 28th International Conference on Computational Linguistics, pages 465-479. International Committee on Computational Linguistics.
Tomáš Nekvinda and Ondřej Dusek. 2021. Shades of BLEU, flavours of success: The case of MultiWOZ. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics, pages 34-46. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318. Association for Computational Linguistics.
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayan- deh, Lars Liden, and Jianfeng Gao. 2020. Soloist: Building task bots at scale with transfer learning and machine teaching. arXiv preprint arXiv:2005.05298.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws.com/openai-assetss/researchcovers/languageunsupervised/language understanding paper.pdf.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2019. Unsupervised bilingual word embedding agreement for unsupervised neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1235-1245. Association for Computational Linguistics.
Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pages 3097-3103. AAAI Press.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11:3371-3408.
Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. 2021. Modelling hierarchical structure between dialogue policy and natural language generator with option framework for task-oriented dialogue system. In Proceedings of the 9th International Conference on Learning Representations. OpenReview.net.
Longyue Wang, Zhaopeng Tu, Shuming Shi, Tong Zhang, Yvette Graham, and Qun Liu. 2018. Translating pro-drop languages with reconstruction models. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 4937-4945. AAAI Press.
Yifan Wang, Jing Zhao, Junwei Bao, Chaoqun Duan, Youzheng Wu, and Xiaodong He. 2022. Luna: Learning slot-turn alignment for dialogue state tracking. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics.
Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438-449. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45. Association for Computational Linguistics.
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 917-929. Association for Computational Linguistics.
Qingyang Wu, Yichi Zhang, Yu Li, and Zhou Yu. 2021. Alternating recurrent dialog model with large-scale pre-trained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1292-1301. Association for Computational Linguistics.
Yunyi Yang, Yunhao Li, and Xiaojun Quan. 2021. UBAR: towards fully end-to-end task-oriented dialog system with GPT-2. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, pages 14230-14238. AAAI Press.
Yichi Zhang, Zhijian Ou, Min Hu, and Junlan Feng. 2020a. A probabilistic end-to-end task-oriented dialog model with latent belief states towards semisupervised learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9207-9219. Association for Computational Linguistics.
Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020b. Task-oriented dialog systems that consider multiple appropriate responses under the same context. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 9604-9611. AAAI Press.
Zheng Zhang, Ryuichi Takanobu, Minlie Huang, and Xiaoyan Zhu. 2020c. Recent advances and challenges in task-oriented dialog system. CoRR, abs/2003.07490.
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1208-1218. Association for Computational Linguistics.
A.1 BORT_G Settings and Results
For the BORT_G backbone, we follow the settings of Zhang et al. (2020b). We use one layer bi-directional GRU for the encoder and the decoder. The dimension of hidden layers is set to 100. The batch size is 128. The AdamW optimizer (Loshchilov and Hutter, 2019) is used to optimize the model parameters, and the learning rate is 0.005.
Model	Inform	Success	BLEU	Combined
DAMD	76.3	60.4	16.6	85.0
MinTL-T5-small	80.0	72.7	19.1	95.5
BORT_G	87.3	75.8	18.4	100.0
Table 9: Comparison of end-to-end models evaluated on MultiWOZ 2.0.
As shown in Table 9, BORT_G performs better than DAMD without a pre-trained model, achieving the improvement of 15.0 combined scores, even though multi-action data augmentation is not used in BORT_G. Moreover, BORT_G outperforms MinTL (Lin et al., 2020), using the pre-trained model. This demonstrates the effectiveness and applicability of our proposed reconstruction strategies.
A.2 Hyper-parameter Analysis
In Figure 5, we empirically investigate how the hyper-parameters in Eq. 9 affects the dialog performance on the MultiWOZ 2.0 validation set. The selection of hyper-parameters  $\lambda_{1}$  and  $\lambda_{2}$  influence the role of the  $\mathcal{L}_{BR}$  and  $\mathcal{L}_{DR}$  across the entire end-to-end task-oriented dialog training process. Larger values of  $\lambda_{1}$  or  $\lambda_{2}$  cause the  $\mathcal{L}_{BR}$  or  $\mathcal{L}_{DR}$  to play a more important role than the original task-oriented dialog loss terms. The smaller the value of  $\lambda_{1}$  or  $\lambda_{2}$ , the less important is the  $\mathcal{L}_{BR}$  or  $\mathcal{L}_{DR}$ . As Figure 5 shows,  $\lambda_{1}$  ranging from 0.01 to 0.5 nearly all enhances task-oriented dialog performance, and when  $\lambda_{2}$  is larger than 0.3, the performance underperforms the baseline system. When  $\lambda_{1} = 0.05$  and  $\lambda_{2} = 0.03$ , our proposed BORT achieves the best performance on the validation set.
In addition, the influence of noise type and noise proportion on the performance of our proposed BORT on the MultiWOZ 2.0 validation set is empirically investigated, as shown in Figure 6. Both deletion and masking noise strategies could improve the dialog performance. In particular, their
Figure 5: BORT performance (combined score) with different levels of hyper-parameters on the MultiWOZ 2.0 validation set.
combination is further better than both of them. This demonstrates that both noise strategies can complement each other to further improve the dialog performance. As shown in Figure 6, when the noise proportion is 0.15, our proposed BORT achieves the best performance on the validation set.
Figure 6: BORT performance (combined score) with different levels of noise type and noise proportion on the MultiWOZ 2.0 validation set.
A.3 Policy Optimization Evaluation
The detailed inform rates, success rates, BLEU scores, and combined scores of policy optimization dialog models on the MultiWOZ 2.0 are presented in Table 10. The ground truth dialog state is used for the policy optimization setting to query the database entities and generate system responses. Our proposed BORT achieves performance comparable to the state-of-the-art LAVA in terms of inform rate. In addition, compared with previous policy optimization methods, BORT achieves bet
Model	Pre-trained	Inform	Success	BLEU	Combined
LaRL (Zhao et al., 2019)	n/a	82.8	79.2	12.8	93.8
SimpleTOD (Hosseini-Asl et al., 2020)	DistilGPT2	88.9	67.1	16.9	94.9
HDSA (Chen et al., 2019)	BERT-base	82.9	68.9	23.6	99.5
ARDM (Wu et al., 2021)	GPT-2	87.4	72.8	20.6	100.7
DAMD (Zhang et al., 2020b)	n/a	89.2	77.9	18.6	102.2
SOLOIST (Peng et al., 2020)	GPT-2	89.6	79.3	18.0	102.5
UBAR (Yang et al., 2021)	DistilGPT2	94.0	83.6	17.2	106.0
LAVA (Lubis et al., 2020)	n/a	97.5	94.8	12.1	108.3
HDNO (Wang et al., 2021)	n/a	96.4	84.7	18.9	109.5
BORT_G	n/a	89.6	80.5	19.1	104.2
BORT	T5-small	96.1	88.8	19.0	111.5
ter performance in terms of the combined score even though BORT has not modeled action learning.
Compared with previous works, BORT achieves much more significant improvement in the end-to-end setting rather than policy optimization setting because our proposed reconstruction strategies pay more attention to improving the quality of dialog state while the golden dialog state is used in the policy optimization setting.
A.4 Ablation Study