SlowGuess/ACL_2020_OCR huggingface dataset:
Table 3 shows the joint accuracy of our model and other baselines on the test set of MultiWOZ 2.0 and 2.1. Our model sets all baselines whether they are based on pre-defined ontology or open vocabulary, and achieves  $268\%$  and  $58.55\%$  joint accuracy with considerable improvements (1.24% and  $5.98\%$ ) over previous best results on MultiWOZ 2.0 and 2.1, respectively. Also, our model achieves  $97.69\%$  and  $98.14\%$  slot accuracy with  $0.36\%$  and  $0.58\%$  improvements over the previous best results on MultiWOZ 2.0 and 2.1, respectively. Similar to (Kim et al., 2019), we find that our model achieves much higher improvements on MultiWOZ 2.1 than
Table 3: Joint accuracy & slot accuracy on the test sets of MultiWOZ 2.0 and 2.1. The ontology column indicates if a model is based on predefined ontology or not.  $\dagger$  means the updated results on SUMBT's GitHub $^2$  and  $\ddagger$  means our reproduction results using source code of SUMBT $^2$ .  $\star$  means we borrow results from (Eric et al., 2019).
Model	M. WOZ 2.1
Our Model	58.5
- state transition prediction	57.36 (-0.69)
- adaptive object-fine tuning	57.45 (-1.10)
- above two closely C-N)†	57.00 (-1.55)
Our Model' FL (α=1,γ=1)‡	58.10 (-0.45)
Table 4: The ability of a study of the state transition prediction and the adaption objective on the MultiWOZ 2.1 test set with joint accuracy  $(\%)$ . † means removing above two modules and remaining CHAN only. ‡ means one-tuning with focal loss instead.
a MultiWOZ 2.0. This is probably because Multi-WOZ 2.1 fixes lots of notation errors in MultiWOZ 2.0 and our model can benefit more from more accurate relevant context.
As shown in Table 4, we estimate the effectiveness of the proposed state transition prediction and adaptive objective on the MultiWOZ 2.1 test set. The results show that both state transition prediction task and adaptive objective can boost the performance. Removing the state transition prediction task reduces joint accuracy by  $0.69\%$ , and the joint accuracy decreases by  $1.10\%$  without the adaptive objective fine-tuning. Moreover, when we remove the state transition prediction task and don't fine-tune our model with adaptive objective (only CHAN remains), the joint accuracy decreases by  $1.55\%$ . Also, to explore the importance of adjusting the  $\alpha_{s}$  adaptively, we replace the adaptive objective with original focal loss  $(\alpha = 1, \gamma = 2)$ , which leads to  $0.45\%$  drop.
To prove the effectiveness of each module of the proposed CHAN, we conduct ablation experiments
<sup>3</sup>It is published as bert-base-uncased model in https://github.com/huggingface/pytorch-transformers
6327
Dialogue Example
Turn 1, U: i am looking for a cheap restaurant in the center of the city.
Turn 2, R: do you have any specific type of food you would like? U: no, i' m not picky as long as the prices are low.
Turn 3, R: there is a cheap chinese restaurant called the dojo noodle bar located in the centre of town. would you like to book a table?
U: yes please, for 8 people at 18 : 30 on thursday.
Turn 4, R: i am sorry but dojo noodle bar is solidly booked at that time. i can try a different time or day for you.
U: can you try to book it at 17 : 30 .
Turn 5, R: all set. your reference number is k2bo09vq.
U: thanks. i 'm also looking for some entertainment close to the restaurant. any suggestions?
Slot-turn Attention
Slot-word Attention
Turn 3
Figure 2: The turn-level and word-level attention visualization of our model on an example from MultiWOZ 2.1 test set, which is predicting the value of slot "restaurant-name" at the  $s$ th turn. The columns "0,1,2,3" are the index of each head of multi-head attention. Although there is no direct information at 5th turn, our model still makes the correct prediction by attending to historiacal relevant words "do, noodle bar" and relevant turns
Turn 4
Turn 5
00 00
(doo noodle bar)
$\{3,4\}$ , which is highlighted in red. Best viewed in color.
on the MultiWOZ 2.1 test set as shown in Table 5. We observe that a slight joint accuracy drop of  $0.24\%$  after removing the global-local fusion gate, which proves the effectiveness of fusing global context and local utterances. Moreover, removing the slot-turn attention and context encoder results decreases by  $0.15\%$  and  $1.72\%$  respectively, which demonstrates that the turn-level relevant information and the contextual representations of word-level relevant information are effective to improve the performance. Moreover, after we remove the aforementioned three modules and sum the word-level relevant information  $\{1,\dots ,t\}$  turns as output, the just accuracy reduces by  $6.72\%$ , which is much higher than the sum of above three reductions. It demonstrates that effectively modeling interactions with word-level relevant information of dialogue history is crucial for DST.
4.3 Attention Visualization
Figure 2 shows the visualization of turn-level and word-level attention of the "restaurant-name" slot on a prediction example of our model at turn 5. The turn-level attention visualization indicates that our model attends to the turns  $\{3,4\}$  that are semantically related to the given slots "restaurant-name"
MultiWOZ 2.1
THAN	57.00
global-local fusion gate	56.76 (-0.24)
- slot-turn attention	56.85 (-0.15)
- context encoder	55.28 (-1.72)
- above three†	50.28 (-6.72)
Table 5: The ablation study of the CHAN on the MultiWOZ 2.1 test set with joint accuracy  $(\%)$ .  $\dagger$  means removing above three modules and summing the word-level relevant information of  $\{1, \dots, t\}$  turns as output.
while almost pays no attention to turns  $\{1,2\}$ . And from the word-level attention visualization, we can easily find that the "restaurant-name" slot attends to the "dojo noodle bar" with the highest weight in both turn 3 and turn 4. Although there is no slot-related information at turn 5, our model still makes the correct decision by exploiting relevant context from the historical dialogue.
4.4 Effects of Adaptive Obj. on Acc. per Slot
As Figure 3 shows, we draw the accuracy changes of each slot on MultiWOZ 2.1 test set after finetuning our model with adaptive objective. We sort all slots in ascending order according to their fre
6328
Figure 3: The accuracy changes (\%) of each slot on the MultiWOZ 2.1 test set after fine-tuning with adaptive objective. We sort all slots in ascending order according to their frequency (Please refer to Appendix for detailed accuracy results).
quency (The detailed accuracy results are in the Appendix). Thus, slots on the left side are relatively more difficult than slots on the right side. After fine-tuning with the adaptive objective, most slots on the left side achieve significant improvements, which proves the adaptive objective can encourage the learning of the hard slots. Although adaptive objective tends to decrease the weight of slots on the right side, they also benefit from the fine-tuning. We think that this is because encouraging the optimizing of hard slots enhances our model by tracking more complicated dialogue states. It proves that our adaptive objective can not only improve the performance of relatively hard slots but also boost the performance of relatively slots.
4.5 Qualitative Analysis
To explore the advantages of a model constructed to baseline models, we conduct a linear evaluation on a subset of the MultiWcZ 2.1 test where our model makes correct predictions while SUMBT (a previous strong baseline) fails. We predefine three types of improvement: historical information inference and improvement which means inferring historical information. It is necessary for correct decisions, current information inference improvement which means inferring current information is enough for correct decisions, and other improvements. As shown in Table 6,  $64.49\%$  improvements come from historical information inference, which demonstrates that our model can better exploit relevant context from the dialogue history.
Traditional statistical dialogue state tracking models combine semantics extracted by spoken lan
Improvement Type	Percentage
Historical Information Inference Improvement	64.49%
Current Information Inference Improvement	34.86%
Others	0.65%
Table 6: Qualitative analysis on the improvements of our model compared to a previous strong baseline SUMBT. It is evaluated by human on a subset of MultiWOZ 2.1 test set where our model makes correct predictions while SUMBT fails.
guage understanding modules to predict the current dialogue state (Williams and Young, 2007; Thomson and Young, 2010; Wang and Lemon, 2013; Williams, 2014) or to jointly understand speech understanding (Henderson et al., 2014b; Zara and Jurcicek, 2015; Wen et al., 2016). One drawback is that they rely on hand-crafted features and complex domain-specific lemmas, besides its ontology, and they are hard to extend and scale to new domains. Recent neural network models are proposed for further improvements (Mrksic et al., 2015; Hori et al., 2016; Mrki et al., 2017; Lei et al., 2018; Xu and Hu, 2018; Zhong et al., 2018; Nouri and Heinei-Asl, 2018; Wu et al., 2019; Ren et al., 2019; Balaraman and Magnini, 2019). Ren et al. (2018) and Lee et al. (2019) use an RNN to encode the related information of each turn, where slots can not attend to relevant information of past turns directly. Sharma et al. (2019) employ a heuristic rule to extract partial dialogue history and then integrate the historical information into prediction in a coarse manner. Goel et al. (2019) encode the dialogue history into a hidden state and then simply combine it with the slot to make decisions. These models are deficient in fully exploiting the relevant context in dialogue history.
Gao et al. (2019b) introduce a slot carryover model to decide whether the values from the previous turn should be used or not and Kim et al. (2019) introduce a state operation predictor to decide the operation with the previous state. Different from them, we consider the state transition prediction as an additional enhancement while they integrate it into their DST pipelines. Besides, Zhong et al. (2018) only employ local modules to model the slot-specific representations, which neglects the slot imbalance problem.
The general backbone of our model is a hierarchical attention network that can effectively aggregate query-related information at multiple levels (Yang
6329
et al., 2016; Ying et al., 2018; Wang et al., 2018; Xing et al., 2018; Aujogue and Aussem, 2019; Naik et al., 2018; Liu and Chen, 2019).
We introduce an effective model that consists of a contextual hierarchical attention network to fully exploit relevant context from dialogue history and an adaptive objective to alleviate the slot imbalance problem in dialogue state tracking. Experimental results show that our model achieves state-of-the-art performance of  $52.68\%$  and  $58.55\%$  joint accuracy with considerable improvements  $(+1.24\%$  and  $+5.98\%)$  over previous best results on MultiWOZ 2.0 and MultiWOZ2.1 datasets, respectively.
Although our model is based on predefined ontology, it is universal and scalable to unseen domains, slots and values. The main contributions of our model, CHAN and adaptive objective, can also be applied to open-vocabulary models. We will explore it in the future.
We thank the anonymous reviewers for their insightful comments. This work was supported by National Key R&D Program of China (NO. 2017YFE0192900).
Jean-Baptiste Aujogue and Alex Aussen 2011 hierarchical recurrent attention networks in context-aware education chatbots. In 2019 International Joint Conference on New AI Works (II-NN), pages 1-8. IEEE.
Vevake Balaraman and Fernando Magnin. 2019. Scalable neural dialogue: "e-tracki." arXiv preprint arXiv:1910.09042.
Pawel Budzynski, Tsung-Isien Wen, Bo-Hsiang Tseng, Inig Cao and Stefan Ultes, Osman Ramadan, and M. Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026.
Guan-Lin Chao and Ian Lane. 2019. Bert-dst: Scalable end-to-end dialogue state tracking with bidirectional encoder representations from transformer. Proc. Interspeech 2019, pages 1468-1472.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of
the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur. 2019. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint arXiv:1907.01669.
Jianfeng Gao, Michel Galley, Lihong Li, et al. 2019a. Neural approaches to conversational ai. Foundations and Trends in Information Retrieval, 13(2-3):127-298.
Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, Dilek Hakkani-Tur, and Amazon Alexa AI. 2019b. Dialog state tracking: A neural reading comprehension approach. In 20th Annual Meeting of the Special Interest Group on Discourse and Dialogue, page 4.
Rahul Goel, Shachi Pan, and Dilek Tunkhani-Tur. 2019. Hyst: A hybrid approach for flexible and accurate dialogue state tracking. ICIS speech 2019.
Matthew Hewson, Blaithomson, and Jason D Williams. 2013. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263-272.
Matthew Henderson, Blaise Thomson, and Steve You. 2014b. Word-based dialog state tracking with recurrent neural networks. In Proceedings of 15th Annual Meeting of the Special Interests Group on Discourse and Dialogue (SIGDIAL), pages 292-299.
Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Takaaki Hori, Hai Wang, Chiori Hori, Shinji Watanabe, Bret Harsham, Jonathan Le Roux, John R Hershey, Yusuke Koji, Yi Jing, Zhaocheng Zhu, et al. 2016. Dialog state tracking with attention-based sequence-to-sequence learning. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 552-558. IEEE.
Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sang-Woo Lee. 2019. Efficient dialogue state tracking by selectively overwriting memory. arXiv preprint arXiv:1911.03906.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Hung Le, Doyen Sahoo, Chenghao Liu, Nancy F. Chen, and Steven C.H. Hoi. 2020a. End-to-end multidomain task-oriented dialogue systems with multi-level neural belief tracker. In OpenReview.
Hung Le, Richard Socher, and Steven C.H. Hoi. 2020b. Non-autoregressive dialog state tracking. In International Conference on Learning Representations.
Hwaran Lee, Jinsik Lee, and Tae-Yoon Kim. 2019. Sumbt: Slot-utterance matching for universal and scalable belief tracking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5478-5483.
Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1437-1447.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dólar. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
Zhengyuan Liu and Nancy Chen. 2019. Reading turn by turn: Hierarchical attention architecture for spoken dialogue comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5460-5466.
Nikola Mrkšić, Diarmuid Řeşaghda, Blaise Thomson, Milica Gasic, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2015. Multidomain dialog state tracking using recurrent neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conferences on Natural Language Processing (Volume 2: Short Papers), pages 794-799.
Nikola Mrkšić, Diarmuid Žáček, Tsung Wen, Blaise Thomson, and Tevečný. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1-3 pages, 1777-1788.
Chetan Naik Arpit Gupta, Han Jeng Ge, Mathias Lambert, and K. S. Sastry 2018. Contextual slot carry-over for Discrete Schemas. Proc. Interspeech 2018, pages 596-600.
Elnaz Nouri and Ehsan Hosseini-Asl. 2018. Toward scalable neural dialogue state tracking model. In Proceedings of NeurIPS 2018, 2nd Conversational AI workshop.
Liliang Ren, Jianmo Ni, and Julian McAuley. 2019. Scalable and accurate dialogue state tracking via hierarchical sequence generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1876-1885.
Liliang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018. Towards universal dialogue state tracking. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2780-2786.
Sanuj Sharma, Prafulla Kumar Choubey, and Ruihong Huang. 2019. Improving dialogue state tracking by discerning the relevant context. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 576-581.
Blaise Thomson and Steve Young. 2010. Bayesian update of dialogue state: A pomdp framework for spoken dialogue systems. Computer Speech & Language, 24(4):562-588.
Ashish Vaswani, Noam Shazes, Nnejarmar, Jakob Uszkoreit, Llion Jones, Adan N Guzez, Lukasz Kaiser, and Illia Polosukhin 2017. Attention is all you need. In Advances in new information processing systems, pages 598-600.
Wei Wang, Ming Yan, and Chen Wu. 2018. Multi-granularity: hierarchical attention fusion networks for reading comprehension and question answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 1705-1714.
Zhou, Wang and Oliver Lemon. 2013. A simple and reliable belief tracking mechanism for the dialog state tracking challenge: On the believability of obse- sed information. In Proceedings of the SIGDIAL 2013 Conference, pages 423-432.
Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gasic, Lina M Rojas Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438-449.
Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference, pages 404-413.
Jason D Williams. 2014. Web-style ranking and slu combination for dialog state tracking. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 282-291.
Jason D Williams and Steve Young. 2007. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):393-422.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale
6331
Fung, 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 808-819.
Chen Xing, Yu Wu, Wei Wu, Yalou Huang, and Ming Zhou. 2018. Hierarchical recurrent attention network for response generation. In Thirty-Second AAAI Conference on Artificial Intelligence.
Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1448-1457.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480-1489.
Haochao Ying, Fuzhen Zhuang, Fuzheng Zhang, Yanchi Liu, Guandong Xu, Xing Xie, Hui Xiong, and Jian Wu. 2018. Sequential recommender system based on hierarchical attention networks. In the 27th International Joint Conference on Artificial Intelligence.
Steve Young. 2002. Talking to machines (statistical speaking). In Seventh International Conference of Spoken Language Processing.
Steve Young, Milica Gašić, Blaise Thomasić and Jason D Williams. 2013. Pomdp-based statistic, spoken dialog systems: A review. Proceedings of IEEE, 101(5):1160-1179.
Jian-Guo Zhang, Kazuma H shin, Chien-heng Wu, Yao Wan, Philip S Yu, Richard Socher, and Caiming Xiong. 2019. Find or classify dual strategy for slot-value predictive on multi-domain dialog state tracking. arXiv: preprint-Xiv:1910.03544.
Victor Zhong, Caring Xiond and Richard Socher. 2018. Causal self-attentive encoder for dialogue state learning: proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers), pages 1458-1467.
Li Zhou and Kevin Small. 2019. Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering. arXiv preprint arXiv:1911.06192.
Lukas Zilka and Filip Jurcicek. 2015. Incremental lstm-based dialog state tracker. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (Asru), pages 757-762. IEEE.
6332
A Slot Imbalance
Figure 4 shows the relationships between frequency and accuracy of slots (left) and slot-value pairs (right). Because the frequency will be the same for all slots if we consider "none" as well, we calculate accuracy with "none" value excluded for slots. Overall, the more the frequency, the higher the accuracy. It demonstrates that the slot imbalance problem results in different learning difficulties for different slots. Moreover, the slot imbalance problem makes some slots hard to learn and hence hurts the accuracy, which limits the overall performance.
Figure 4: The relationships between frequency and accuracy of slots (left) and slot-value pairs (right). Because the frequency will be the same for all slots in we consider "none" as well, we calculate accuracy with "none" value excluded for slots.