SlowGuess/ACL_2020_OCR huggingface dataset:
larities between the dialogue and the slot ontology to track the domain and the value of the slot jointly. GLAD relies on global modules to learn the general information and local modules to catch the slot-specific information (Zhong et al., 2018) from the dialogues. GCE efficiently improves and simplifies GLAD, while keeping the excellent performance of GLAD (Nouri and Hosseini-Asl, 2018). SpanPtr first introduces the pointer network (Vinyals et al., 2015) into the dialogue state tracking task to extract unknown slot values (Xu and Hu, 2018). And in that paper, they also apply an effective dropout technique for training. TRADE directly generates slot values from the dialogues by using the copy mechanism and gets rid of the predefined value list (Wu et al., 2019). It achieves the previous state-of-the-art performance.
We use the fix combination version of SAS in Table 1 with the integration ratio  $\alpha$  of 0.8 and the threshold  $\beta$  of 0.8. That's the best hyperparameters we find for MultiWOZ.
In this section, we first show the result of our model on MultiWoZ dataset, then on MultiWoZ(restaurant) and MultiWOZ (except hotel) dataset. After conducting the ablation experiment, we also display the improvement the slot attention and slot information sharing brings.
Our model achieves the best performance in the most important metric, joint goal accuracy. Our model outperformed the previous state-of-the-art model, TRADE by  $2.41\%$  absolute score on joint goal accuracy. We only observe slight increase of slot accuracy compared to TRADE. We suspect it is because TRADE was already achieving nearly  $97\%$  accuracy, which is close to the up-bound of the slot accuracy in this task. After carefully checking the error cases, we found these errors mainly come from the difficulty of generating name phrases.
To test SAS's ability on single domain dialogue tasks, we also evaluate our model on the a subset of MultiWOZ which contains only the restaurant search task. As displayed in Table 1, SAS achieved  $1.99\%$  improvement over TRADE on the joint goal accuracy as well, suggesting SAS's good performance generalize to single domain task.
Table 2 also shows how different choices of the hyperparameters influence the final results. On MultiWOZ, the integration ratio of 0.8 and the threshold of 0.8 are the best hyperparameters. But as
6371
illustrated in Table 2, the best integration ratio is no longer 0.8 on MultiWOZ (except hotel). The best values of the integration ratio and the threshold will vary with the ontology.
We also perform ablation study to quantify different modules' contribution. We observe in Table 3 that adding the slot attention improves the state tracking results by  $1.37\%$  on MultiWOZ. Such improvement suggests having slot attention that focuses on the key information of the history is useful. And the slot information sharing further enhances the performance by  $1.04\%$ . The reason behind this may be that the information sharing of the related slots makes the data-insufficient slot receive more information. This handles the rare or unknown slot-value problems to some extent. As illustrated in Table 3, a model with the fix combination sharing method performs better than the k-means sharing method. But the fix combination method has an obvious shortcoming. It is difficult to generalize to new ontology. We need search the hyperparameters for every new ontology and these efforts are usually costly and time-consuming. Results in Table 2 and Table 3 indicate that the k-means algorithm provides a more robust model with respect to different parameters.
Model	MultiWOZ	MultiWOZ(res)
Joint	Slot	Joint	Slot
SAS-att-shr	48.62	96.92	65.35	93.28
SAS-shr	49.99	97.10	66.89	93.62
SAS(RT shr)	51.03	97.20	67.34	93.83
SAS(KM shr)	50.46	97.15	66.65	93.78
SAS(HM shr)	50.27	97.13	66.89	93.62
Table 3: Performances of the models with different components on MultiWOZ dataset and MultiWOZ (restaurant) dataset. RT shr, KM shr, HM shr indicate the model is using the fix combination sharing method, k-means sharing method, and the human evaluated sharing method in the slot information sharing respectively.
To investigate whether the slot similarity matrices used by the two sharing methods really reflect the similarity among slots, we also compare them with a human constructed similarity matrix. We invite three volunteers to carefully rate (1 or 0) the relationship between every two slots and obtain the slot similarity matrix used in the human evaluated method. As shown in Table 2 and Table 3, the performance of the k-means sharing method is close to the one the human constructed method. This indicates human knowledge cannot further im
prove this task. Besides that, we also notice that the fix combination model usually outperforms the human constructed method, demonstrating that the fix combination model can automatically discover some hidden relationship among all slots that human cannot capture.
To better understand why our model improves the performance, we investigated some dialogue examples and shown them in Table 4.
In the first dialogue, by asking "Could you also find me a hotel with a moderate price that offers internet?", the user has briefly informed the agent that he/she is looking for a hotel "with internet". The previous model missed the "hotel-internet" in the tracked slots. Because the model is mislead by the long interaction history. Our model learns to focus on important information using the slot attention to track the correct internet slot.
In the second dialogue, although the previous model manages to capture the value "21:30". It still confused "arriveby" with "leaveat". While SAS can distinguish them. We suspect this is because our model can learn the differences between these slots by training on isolated key features per slot without seeing any irrelevant information.
In the third example, the user agrees to visit an attraction named "Christ's College" from many college-type choices the agent suggests. Previous model fetches a wrong message and fills the slot "attraction-name" with "Clare College". In contrast, SAS captures the correct attraction name and also deduces that the attraction type is college. Similar to the first dialogue, the slot attention helps model gain more clean information to detect slot values more accurately. And by sharing the information fetched from slot "attraction-name" with the slot "attraction-type", our model is more sensitive with the value "college".
We also investigate the limitation of our model by analyzing the state tracking errors. We noticed two types of errors. First, SAS can not effectively identify value "dontcare" for most slots. For example, when the agent asks the user about his/her requirement on the hotel rating, though he/she answers "that is not really important for me", the model fails to fill "dontcare" into the slot "hotelstar". We believe this is due to the fact that the meaning of "dontcare" has plenty of expressions, it is much harder for the model to learn the semantic
6372
No	Model	Context
1	I am looking for a train that leaves on saturday and arrives by 10:30. // Where are you traveling to and from? // … // Yes, that train sounds good. Please book it for me. Could you also find me a hotel with a moderate price that offers internet? // … // The north part of town please, preferably in a guesthouse.
True	‘hotel-area-north’, ‘train-day-saturday’, ‘hotel-internet-yes’, …, ‘hotel-pricerange-moderate’, ‘hotel-type-guest house’
TRADE	‘train-arriveby-10:30’, ‘train-day-saturday’, ‘train-departure-birmingham new street’, ‘train-destination-cambridge’, ‘hotel-pricerange-moderate’, ‘hotel-type-guest house’
SAS	‘train-destination-cambridge’, ‘train-departure-birmingham new street’, …, ‘hotel-internet-yes’, ‘train-arriveby-10:30’
2	I am looking for a Chinese restaurant in the centre of town. // … // All Saints Church is famous for its architecture. It's located on Jesus Lane, cb58bs. They can be reached at 01223452587. Is there anything else I can find for you? // Yes. I need a taxi to take me from the church to the restaurant at 21:30.
True	‘restaurant-food-chinese’, ‘attraction-area-centre’, …, ‘taxi-departure-all saints church’, ‘restaurant-area-centre’, ‘taxi-leaveat-21:30’
TRADE	‘restaurant-food-chinese’, ‘attraction-area-centre’, …, ‘taxi-arriveby-21:30’, ‘taxi-departure-all saints church’, ‘restaurant-area-centre’, ‘restaurant-pricerange-dontcare’, ‘taxi-leaveat-21:30’
SAS	‘taxi-destination-all saints church’, ‘restaurant-pricerange-dontcare’, ‘attraction-area-centre’, ‘taxi-leaveat-21:30’, ‘restaurant-food-chinese’, ‘taxi-departure-all saints church’, ‘attraction-name-all saints church’, ‘restaurant-area-centre’
3	I would like to get some information about colleges to visit? // There is Christs College, Churchill College, Clare College, Clare Hall, Corpus Christi, Downing College, Emmanuel College, and Huges Hall. Would you like me to list more? // … // Tr6359 leaves at 13:40 and arrives 16:23, will this 1 work for you ? // Yes i need 6 tickets.
True	‘attraction-type-college’, ‘train-departure-birmingham new street’, …, ‘attraction-name-christ s college’, ‘train-book people-6’, ‘train-day-friday’, ‘train-arriveby-16:30’
TRADE	‘attraction-name-clare college’, ‘train-departure-birmingham new street’, ‘train-destination-cambridge’, ‘train-book people-6’, ‘train-day-friday’, ‘train-arriveby-16:30’
SAS	‘train-destination-cambridge’, ‘train-departure-birmingham new street’, ‘attraction-name-christ s college’, ‘train-book people-6’, …, ‘attraction-type-college’
Table 4: Example dialogue state outputs from TRADE and SAS. "True" stands for ground truth dialogue states, "TRADE" and "SAS" are the generation results from TRADE and SAS respectively.
of "dontcare" than other slots. Besides that, we also notice that the tracking errors of departure or destination location are still common. The reason may be that location name words are usually rich in variations and have few grammatical feature.
We present SAS, an effective DST model which successfully extracts the key feature from the original information excessive dialogue. The slot attention of SAS enables it to isolate the key information for each slot, while the slot information sharing enhances the expressiveness of the infor
6373
mation passed to each slot by integrating the information from similar slots. The sharing allows SAS to generalize on rare slot-value pairs with few training data. Our model reaches the state-of-the-art performance compared with previous models.
We believe that SAS provides promising potential extensions, such as adapting our model on other tasks where are troubled by excessive information. Besides that, we also notice that it is hard for SAS to correctly extract names of hotel or attraction which have rich variations. Designing a new model to address these problems may be our future work.
This research is funded by the Science and Technology Commission of Shanghai Municipality (19511120200 & 18511105502) and by Xiaoi Research. The computation is performed in ECNU Multifunctional Platform for Innovation (001).
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2015. Generating sentences from a continuous space. arXiv preprint, arXiv:1511.06349. Version 4.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026.
Danqi Chen. 2018. Neural reading comprehension and beyond. Ph.D. thesis, Stanford University.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. arXiv preprint, arXiv:1704.00051. Version 2.
Tongfei Chen, Chetan Naik, Hua He, Pushpendre Rastogi, and Lambert Mathias. 2019. Improving long distance slot carryover in spoken dialogue systems. arXiv preprint, arXiv:1906.01149. Version 1.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint, arXiv:1412.3555. Version 1.
Franck Dernoncourt, Ji Young Lee, Trung H Bui, and Hung H Bui. 2017. Robust dialog state tracking for large ontologies. In Dialogues with Social Robots, pages 475-485. Springer.
Kazuma Hashimoto, Yoshimasa Tsuruoka, Richard Socher, et al. 2017. A joint many-task model: Growing a neural network for multiple nlp tasks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1923-1933.
Matthew Henderson, Blaise Thomson, and Steve Young. 2014. Word-based dialog state tracking with recurrent neural networks. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 292-299.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint, arXiv:1412.6980. Version 9.
Nikola Mrksic, Diarmuid Řeashdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1777-1788.
Elnaz Nouri and Ehsan Hosseini-Asl. 2018. Toward scalable neural dialogue state tracking model. arXiv preprint, arXiv:1812.00899. Version 1.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Julien Perez and Fei Liu. 2017. Dialog state tracking, a machine reading approach using memory network. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 305-314.
Osman Ramadan, Paweł Budzianowski, and Milica Gasic. 2018. Large-scale multi-domain belief tracking with knowledge sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 432-437.
Abhinav Rastogi, Dilek Hakkani-Tür, and Larry Heck. 2017. Scalable multi-domain dialogue state tracking. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 561-568. IEEE.
Sanuj Sharma, Prafulla Kumar Choubey, and Ruihong Huang. 2019. Improving dialogue state tracking by discerning the relevant context. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 576-581.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
6374
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692-2700.
Zhuoran Wang and Oliver Lemon. 2013. A simple and generic belief tracking mechanism for the dialog state tracking challenge: On the believability of observed information. In Proceedings of the SIGDIAL 2013 Conference, pages 423-432.
TH Wen, D Vandyke, N Mrksic, M Gaic, LM Rojas-Barahona, PH Su, S Ultes, and S Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017-Proceedings of Conference, volume 1, pages 438-449.
Jason D Williams. 2014. Web-style ranking and slu combination for dialog state tracking. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 282-291.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. arXiv preprint, arXiv:1905.08743. Version 1.
Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1448-1457.
Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1458-1467.
Lukas Zilka and Filip Jurcicek. 2015. Incremental lstm-based dialog state tracker. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (Asru), pages 757-762. IEEE.
6375